pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedWIP: Data at rest encryption
93+ messages / 24 participants
[nested] [flat]
* WIP: Data at rest encryption
@ 2016-06-07 13:56 Ants Aasma <ants.aasma@gmail.com>
2016-06-10 02:23 ` Re: WIP: Data at rest encryption Haribabu Kommi <kommi.haribabu@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-06-27 08:02 ` Re: [HACKERS] WIP: Data at rest encryption Antonin Houska <ah@cybertec.at>
2019-08-02 06:40 ` Re: WIP: Data at rest encryption Shawn Wang <shawn.wang@highgo.ca>
0 siblings, 4 replies; 93+ messages in thread
From: Ants Aasma @ 2016-06-07 13:56 UTC (permalink / raw)
To: pgsql-hackers
Hi all,
I have been working on data-at-rest encryption support for PostgreSQL.
In my experience this is a common request that customers make. The
short of the feature is that all PostgreSQL data files are encrypted
with a single master key and are decrypted when read from the OS. It
does not provide column level encryption which is an almost orthogonal
feature, arguably better done client side.
Similar things can be achieved with filesystem level encryption.
However this is not always optimal for various reasons. One of the
better reasons is the desire for HSM based encryption in a storage
area network based setup.
Attached to this mail is a work in progress patch that adds an
extensible encryption mechanism. There are some loose ends left to tie
up, but the general concept and architecture is at a point where it's
ready for some feedback, fresh ideas and bikeshedding.
Usage
=====
Set up database like so:
(read -sp "Postgres passphrase: " PGENCRYPTIONKEY; echo;
export PGENCRYPTIONKEY
initdb -k -K pgcrypto $PGDATA )
Start PostgreSQL:
(read -sp "Postgres passphrase: " PGENCRYPTIONKEY; echo;
export PGENCRYPTIONKEY
postgres $PGDATA )
Design
======
The patch adds a new GUC called encryption_library, when specified the
named library is loaded before shared_preload_libraries and is
expected to register its encryption routines. For now the API is
pretty narrow, one parameterless function that lets the extension do
key setup on its own terms, and two functions for
encrypting/decrypting an arbitrary sized block of data with tweak. The
tweak should alter the encryption function so that identical block
contents are encrypted differently based on their location. The GUC
needs to be set at bootstrap time, so it gets set by a new option for
initdb. During bootstrap an encryption sample gets stored in the
control file, enabling useful error messages.
The library name is not stored in controldata. I'm not quite sure
about this decision. On one hand it would be very useful to tell the
user what he needs to get at his data if the configuration somehow
goes missing and it would get rid of the extra GUC. On the other hand
I don't really want to bloat control data, and the same encryption
algorithm could be provided by different implementations.
For now the encryption is done for everything that goes through md,
xlog and slru. Based on a review of read/write/fread/fwrite calls this
list is missing:
* BufFile - needs refactoring
* Logical reorder buffer serialization - probably needs a stream mode
cipher API addition.
* logical_heap_rewrite - can be encrypted as one big block
* 2PC state data - ditto
* pg_stat_statements - query texts get appended so a stream mode
cipher might be needed here too.
copydir needed some changes too because tablespace and database oid
are included in the tweak and so copying also needs to decrypt and
encrypt with the new tweak value.
For demonstration purposes I imported Brian Gladman's AES-128-XTS mode
implementation into pgcrypto and used an environment variable for key
setup. This part is not really in any reviewable state, the XTS code
needs heavy cleanup to bring it up to PostgreSQL coding standards,
keysetup needs something secure, like PBKDF2 or scrypt.
Performance with current AES implementation is not great, but not
horrible either, I'm seeing around 2x slowdown for larger than
shared_buffers, smaller than free memory workloads. However the plan
is to fix this - I have a prototype AES-NI implementation that does
3GB/s per core on my Haswell based laptop (1.25 B/cycle).
Open questions
==============
The main questions is what to do about BufFile? It currently provides
both unaligned random access and a block based interface. I wonder if
it would be a good idea to refactor it to be fully block based under
the covers.
I would also like to incorporate some database identifier as a salt in
key setup. However, system identifier stored in control file doesn't
fit this role well. It gets initialized somewhat too late in the
bootstrap process, and more importantly, gets changed on pg_upgrade.
This will make link mode upgrades impossible, which seems like a no
go. I'm torn whether to add a new value for this purpose (perhaps
stored outside the control file) or allow setting of system identifier
via initdb. The first seems like a better idea, the file could double
as a place to store additional encryption parameters, like key length
or different cipher primitive.
Regards,
Ants Aasma
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
data-at-rest-encryption-wip-2016.06.07.patch (text/x-patch, 92.1K) download
diff --git a/contrib/pgcrypto/Makefile b/contrib/pgcrypto/Makefile
index 18bad1a..04ce887 100644
--- a/contrib/pgcrypto/Makefile
+++ b/contrib/pgcrypto/Makefile
@@ -20,7 +20,7 @@ SRCS = pgcrypto.c px.c px-hmac.c px-crypt.c \
mbuf.c pgp.c pgp-armor.c pgp-cfb.c pgp-compress.c \
pgp-decrypt.c pgp-encrypt.c pgp-info.c pgp-mpi.c \
pgp-pubdec.c pgp-pubenc.c pgp-pubkey.c pgp-s2k.c \
- pgp-pgsql.c
+ pgp-pgsql.c xts.c
MODULE_big = pgcrypto
OBJS = $(SRCS:.c=.o) $(WIN32RES)
diff --git a/contrib/pgcrypto/brg_endian.h b/contrib/pgcrypto/brg_endian.h
new file mode 100644
index 0000000..e3cf0d1
--- /dev/null
+++ b/contrib/pgcrypto/brg_endian.h
@@ -0,0 +1,133 @@
+/*
+ ---------------------------------------------------------------------------
+ Copyright (c) 1998-2008, Brian Gladman, Worcester, UK. All rights reserved.
+
+ LICENSE TERMS
+
+ The redistribution and use of this software (with or without changes)
+ is allowed without the payment of fees or royalties provided that:
+
+ 1. source code distributions include the above copyright notice, this
+ list of conditions and the following disclaimer;
+
+ 2. binary distributions include the above copyright notice, this list
+ of conditions and the following disclaimer in their documentation;
+
+ 3. the name of the copyright holder is not used to endorse products
+ built using this software without specific written permission.
+
+ DISCLAIMER
+
+ This software is provided 'as is' with no explicit or implied warranties
+ in respect of its properties, including, but not limited to, correctness
+ and/or fitness for purpose.
+ ---------------------------------------------------------------------------
+ Issue Date: 20/12/2007
+*/
+
+#ifndef _BRG_ENDIAN_H
+#define _BRG_ENDIAN_H
+
+#define IS_BIG_ENDIAN 4321 /* byte 0 is most significant (mc68k) */
+#define IS_LITTLE_ENDIAN 1234 /* byte 0 is least significant (i386) */
+
+/* Include files where endian defines and byteswap functions may reside */
+#if defined( __sun )
+# include <sys/isa_defs.h>
+#elif defined( __FreeBSD__ ) || defined( __OpenBSD__ ) || defined( __NetBSD__ )
+# include <sys/endian.h>
+#elif defined( BSD ) && ( BSD >= 199103 ) || defined( __APPLE__ ) || \
+ defined( __CYGWIN32__ ) || defined( __DJGPP__ ) || defined( __osf__ )
+# include <machine/endian.h>
+#elif defined( __linux__ ) || defined( __GNUC__ ) || defined( __GNU_LIBRARY__ )
+# if !defined( __MINGW32__ ) && !defined( _AIX )
+# include <endian.h>
+# if !defined( __BEOS__ )
+# include <byteswap.h>
+# endif
+# endif
+#endif
+
+/* Now attempt to set the define for platform byte order using any */
+/* of the four forms SYMBOL, _SYMBOL, __SYMBOL & __SYMBOL__, which */
+/* seem to encompass most endian symbol definitions */
+
+#if defined( BIG_ENDIAN ) && defined( LITTLE_ENDIAN )
+# if defined( BYTE_ORDER ) && BYTE_ORDER == BIG_ENDIAN
+# define PLATFORM_BYTE_ORDER IS_BIG_ENDIAN
+# elif defined( BYTE_ORDER ) && BYTE_ORDER == LITTLE_ENDIAN
+# define PLATFORM_BYTE_ORDER IS_LITTLE_ENDIAN
+# endif
+#elif defined( BIG_ENDIAN )
+# define PLATFORM_BYTE_ORDER IS_BIG_ENDIAN
+#elif defined( LITTLE_ENDIAN )
+# define PLATFORM_BYTE_ORDER IS_LITTLE_ENDIAN
+#endif
+
+#if defined( _BIG_ENDIAN ) && defined( _LITTLE_ENDIAN )
+# if defined( _BYTE_ORDER ) && _BYTE_ORDER == _BIG_ENDIAN
+# define PLATFORM_BYTE_ORDER IS_BIG_ENDIAN
+# elif defined( _BYTE_ORDER ) && _BYTE_ORDER == _LITTLE_ENDIAN
+# define PLATFORM_BYTE_ORDER IS_LITTLE_ENDIAN
+# endif
+#elif defined( _BIG_ENDIAN )
+# define PLATFORM_BYTE_ORDER IS_BIG_ENDIAN
+#elif defined( _LITTLE_ENDIAN )
+# define PLATFORM_BYTE_ORDER IS_LITTLE_ENDIAN
+#endif
+
+#if defined( __BIG_ENDIAN ) && defined( __LITTLE_ENDIAN )
+# if defined( __BYTE_ORDER ) && __BYTE_ORDER == __BIG_ENDIAN
+# define PLATFORM_BYTE_ORDER IS_BIG_ENDIAN
+# elif defined( __BYTE_ORDER ) && __BYTE_ORDER == __LITTLE_ENDIAN
+# define PLATFORM_BYTE_ORDER IS_LITTLE_ENDIAN
+# endif
+#elif defined( __BIG_ENDIAN )
+# define PLATFORM_BYTE_ORDER IS_BIG_ENDIAN
+#elif defined( __LITTLE_ENDIAN )
+# define PLATFORM_BYTE_ORDER IS_LITTLE_ENDIAN
+#endif
+
+#if defined( __BIG_ENDIAN__ ) && defined( __LITTLE_ENDIAN__ )
+# if defined( __BYTE_ORDER__ ) && __BYTE_ORDER__ == __BIG_ENDIAN__
+# define PLATFORM_BYTE_ORDER IS_BIG_ENDIAN
+# elif defined( __BYTE_ORDER__ ) && __BYTE_ORDER__ == __LITTLE_ENDIAN__
+# define PLATFORM_BYTE_ORDER IS_LITTLE_ENDIAN
+# endif
+#elif defined( __BIG_ENDIAN__ )
+# define PLATFORM_BYTE_ORDER IS_BIG_ENDIAN
+#elif defined( __LITTLE_ENDIAN__ )
+# define PLATFORM_BYTE_ORDER IS_LITTLE_ENDIAN
+#endif
+
+/* if the platform byte order could not be determined, then try to */
+/* set this define using common machine defines */
+#if !defined(PLATFORM_BYTE_ORDER)
+
+#if defined( __alpha__ ) || defined( __alpha ) || defined( i386 ) || \
+ defined( __i386__ ) || defined( _M_I86 ) || defined( _M_IX86 ) || \
+ defined( __OS2__ ) || defined( sun386 ) || defined( __TURBOC__ ) || \
+ defined( vax ) || defined( vms ) || defined( VMS ) || \
+ defined( __VMS ) || defined( _M_X64 )
+# define PLATFORM_BYTE_ORDER IS_LITTLE_ENDIAN
+
+#elif defined( AMIGA ) || defined( applec ) || defined( __AS400__ ) || \
+ defined( _CRAY ) || defined( __hppa ) || defined( __hp9000 ) || \
+ defined( ibm370 ) || defined( mc68000 ) || defined( m68k ) || \
+ defined( __MRC__ ) || defined( __MVS__ ) || defined( __MWERKS__ ) || \
+ defined( sparc ) || defined( __sparc) || defined( SYMANTEC_C ) || \
+ defined( __VOS__ ) || defined( __TIGCC__ ) || defined( __TANDEM ) || \
+ defined( THINK_C ) || defined( __VMCMS__ ) || defined( _AIX )
+# define PLATFORM_BYTE_ORDER IS_BIG_ENDIAN
+
+#elif 0 /* **** EDIT HERE IF NECESSARY **** */
+# define PLATFORM_BYTE_ORDER IS_LITTLE_ENDIAN
+#elif 0 /* **** EDIT HERE IF NECESSARY **** */
+# define PLATFORM_BYTE_ORDER IS_BIG_ENDIAN
+#else
+# error Please edit lines 126 or 128 in brg_endian.h to set the platform byte order
+#endif
+
+#endif
+
+#endif
diff --git a/contrib/pgcrypto/brg_types.h b/contrib/pgcrypto/brg_types.h
new file mode 100644
index 0000000..2675d2d
--- /dev/null
+++ b/contrib/pgcrypto/brg_types.h
@@ -0,0 +1,223 @@
+/*
+ ---------------------------------------------------------------------------
+ Copyright (c) 1998-2008, Brian Gladman, Worcester, UK. All rights reserved.
+
+ LICENSE TERMS
+
+ The redistribution and use of this software (with or without changes)
+ is allowed without the payment of fees or royalties provided that:
+
+ 1. source code distributions include the above copyright notice, this
+ list of conditions and the following disclaimer;
+
+ 2. binary distributions include the above copyright notice, this list
+ of conditions and the following disclaimer in their documentation;
+
+ 3. the name of the copyright holder is not used to endorse products
+ built using this software without specific written permission.
+
+ DISCLAIMER
+
+ This software is provided 'as is' with no explicit or implied warranties
+ in respect of its properties, including, but not limited to, correctness
+ and/or fitness for purpose.
+ ---------------------------------------------------------------------------
+ Issue Date: 20/12/2007
+
+ The unsigned integer types defined here are of the form uint_<nn>t where
+ <nn> is the length of the type; for example, the unsigned 32-bit type is
+ 'uint_32t'. These are NOT the same as the 'C99 integer types' that are
+ defined in the inttypes.h and stdint.h headers since attempts to use these
+ types have shown that support for them is still highly variable. However,
+ since the latter are of the form uint<nn>_t, a regular expression search
+ and replace (in VC++ search on 'uint_{:z}t' and replace with 'uint\1_t')
+ can be used to convert the types used here to the C99 standard types.
+*/
+
+#ifndef _BRG_TYPES_H
+#define _BRG_TYPES_H
+
+#if defined(__cplusplus)
+extern "C" {
+#endif
+
+#include <limits.h>
+
+#if defined( _MSC_VER ) && ( _MSC_VER >= 1300 )
+# include <stddef.h>
+# define ptrint_t intptr_t
+#elif defined( __GNUC__ ) && ( __GNUC__ >= 3 )
+# include <stdint.h>
+# define ptrint_t intptr_t
+#else
+# define ptrint_t int
+#endif
+
+#ifndef BRG_UI8
+# define BRG_UI8
+# if UCHAR_MAX == 255u
+ typedef unsigned char uint_8t;
+# else
+# error Please define uint_8t as an 8-bit unsigned integer type in brg_types.h
+# endif
+#endif
+
+#ifndef BRG_UI16
+# define BRG_UI16
+# if USHRT_MAX == 65535u
+ typedef unsigned short uint_16t;
+# else
+# error Please define uint_16t as a 16-bit unsigned short type in brg_types.h
+# endif
+#endif
+
+#ifndef BRG_UI32
+# define BRG_UI32
+# if UINT_MAX == 4294967295u
+# define li_32(h) 0x##h##u
+ typedef unsigned int uint_32t;
+# elif ULONG_MAX == 4294967295u
+# define li_32(h) 0x##h##ul
+ typedef unsigned long uint_32t;
+# elif defined( _CRAY )
+# error This code needs 32-bit data types, which Cray machines do not provide
+# else
+# error Please define uint_32t as a 32-bit unsigned integer type in brg_types.h
+# endif
+#endif
+
+#ifndef BRG_UI64
+# if defined( __BORLANDC__ ) && !defined( __MSDOS__ )
+# define BRG_UI64
+# define li_64(h) 0x##h##ui64
+ typedef unsigned __int64 uint_64t;
+# elif defined( _MSC_VER ) && ( _MSC_VER < 1300 ) /* 1300 == VC++ 7.0 */
+# define BRG_UI64
+# define li_64(h) 0x##h##ui64
+ typedef unsigned __int64 uint_64t;
+# elif defined( __sun ) && defined(ULONG_MAX) && ULONG_MAX == 0xfffffffful
+# define BRG_UI64
+# define li_64(h) 0x##h##ull
+ typedef unsigned long long uint_64t;
+# elif defined( __MVS__ )
+# define BRG_UI64
+# define li_64(h) 0x##h##ull
+ typedef unsigned int long long uint_64t;
+# elif defined( UINT_MAX ) && UINT_MAX > 4294967295u
+# if UINT_MAX == 18446744073709551615u
+# define BRG_UI64
+# define li_64(h) 0x##h##u
+ typedef unsigned int uint_64t;
+# endif
+# elif defined( ULONG_MAX ) && ULONG_MAX > 4294967295u
+# if ULONG_MAX == 18446744073709551615ul
+# define BRG_UI64
+# define li_64(h) 0x##h##ul
+ typedef unsigned long uint_64t;
+# endif
+# elif defined( ULLONG_MAX ) && ULLONG_MAX > 4294967295u
+# if ULLONG_MAX == 18446744073709551615ull
+# define BRG_UI64
+# define li_64(h) 0x##h##ull
+ typedef unsigned long long uint_64t;
+# endif
+# elif defined( ULONG_LONG_MAX ) && ULONG_LONG_MAX > 4294967295u
+# if ULONG_LONG_MAX == 18446744073709551615ull
+# define BRG_UI64
+# define li_64(h) 0x##h##ull
+ typedef unsigned long long uint_64t;
+# endif
+# endif
+#endif
+
+#if !defined( BRG_UI64 )
+# if defined( NEED_UINT_64T )
+# error Please define uint_64t as an unsigned 64 bit type in brg_types.h
+# endif
+#endif
+
+#ifndef RETURN_VALUES
+# define RETURN_VALUES
+# if defined( DLL_EXPORT )
+# if defined( _MSC_VER ) || defined ( __INTEL_COMPILER )
+# define VOID_RETURN __declspec( dllexport ) void __stdcall
+# define INT_RETURN __declspec( dllexport ) int __stdcall
+# elif defined( __GNUC__ )
+# define VOID_RETURN __declspec( __dllexport__ ) void
+# define INT_RETURN __declspec( __dllexport__ ) int
+# else
+# error Use of the DLL is only available on the Microsoft, Intel and GCC compilers
+# endif
+# elif defined( DLL_IMPORT )
+# if defined( _MSC_VER ) || defined ( __INTEL_COMPILER )
+# define VOID_RETURN __declspec( dllimport ) void __stdcall
+# define INT_RETURN __declspec( dllimport ) int __stdcall
+# elif defined( __GNUC__ )
+# define VOID_RETURN __declspec( __dllimport__ ) void
+# define INT_RETURN __declspec( __dllimport__ ) int
+# else
+# error Use of the DLL is only available on the Microsoft, Intel and GCC compilers
+# endif
+# elif defined( __WATCOMC__ )
+# define VOID_RETURN void __cdecl
+# define INT_RETURN int __cdecl
+# else
+# define VOID_RETURN void
+# define INT_RETURN int
+# endif
+#endif
+
+/* These defines are used to detect and set the memory alignment of pointers.
+ Note that offsets are in bytes.
+
+ ALIGN_OFFSET(x,n) return the positive or zero offset of
+ the memory addressed by the pointer 'x'
+ from an address that is aligned on an
+ 'n' byte boundary ('n' is a power of 2)
+
+ ALIGN_FLOOR(x,n) return a pointer that points to memory
+ that is aligned on an 'n' byte boundary
+ and is not higher than the memory address
+ pointed to by 'x' ('n' is a power of 2)
+
+ ALIGN_CEIL(x,n) return a pointer that points to memory
+ that is aligned on an 'n' byte boundary
+ and is not lower than the memory address
+ pointed to by 'x' ('n' is a power of 2)
+*/
+
+#define ALIGN_OFFSET(x,n) (((ptrint_t)(x)) & ((n) - 1))
+#define ALIGN_FLOOR(x,n) ((uint_8t*)(x) - ( ((ptrint_t)(x)) & ((n) - 1)))
+#define ALIGN_CEIL(x,n) ((uint_8t*)(x) + (-((ptrint_t)(x)) & ((n) - 1)))
+
+/* These defines are used to declare buffers in a way that allows
+ faster operations on longer variables to be used. In all these
+ defines 'size' must be a power of 2 and >= 8. NOTE that the
+ buffer size is in bytes but the type length is in bits
+
+ UNIT_TYPEDEF(x,size) declares a variable 'x' of length
+ 'size' bits
+
+ BUFR_TYPEDEF(x,size,bsize) declares a buffer 'x' of length 'bsize'
+ bytes defined as an array of variables
+ each of 'size' bits (bsize must be a
+ multiple of size / 8)
+
+ UNIT_CAST(x,size) casts a variable to a type of
+ length 'size' bits
+
+ UPTR_CAST(x,size) casts a pointer to a pointer to a
+ varaiable of length 'size' bits
+*/
+
+#define UI_TYPE(size) uint_##size##t
+#define UNIT_TYPEDEF(x,size) typedef UI_TYPE(size) x
+#define BUFR_TYPEDEF(x,size,bsize) typedef UI_TYPE(size) x[bsize / (size >> 3)]
+#define UNIT_CAST(x,size) ((UI_TYPE(size) )(x))
+#define UPTR_CAST(x,size) ((UI_TYPE(size)*)(x))
+
+#if defined(__cplusplus)
+}
+#endif
+
+#endif
diff --git a/contrib/pgcrypto/mode_hdr.h b/contrib/pgcrypto/mode_hdr.h
new file mode 100644
index 0000000..46bd94f
--- /dev/null
+++ b/contrib/pgcrypto/mode_hdr.h
@@ -0,0 +1,323 @@
+/*
+ ---------------------------------------------------------------------------
+ Copyright (c) 1998-2008, Brian Gladman, Worcester, UK. All rights reserved.
+
+ LICENSE TERMS
+
+ The redistribution and use of this software (with or without changes)
+ is allowed without the payment of fees or royalties provided that:
+
+ 1. source code distributions include the above copyright notice, this
+ list of conditions and the following disclaimer;
+
+ 2. binary distributions include the above copyright notice, this list
+ of conditions and the following disclaimer in their documentation;
+
+ 3. the name of the copyright holder is not used to endorse products
+ built using this software without specific written permission.
+
+ DISCLAIMER
+
+ This software is provided 'as is' with no explicit or implied warranties
+ in respect of its properties, including, but not limited to, correctness
+ and/or fitness for purpose.
+ ---------------------------------------------------------------------------
+ Issue Date: 20/12/2007
+
+ This header file is an INTERNAL file which supports mode implementation
+*/
+
+#ifndef _MODE_HDR_H
+#define _MODE_HDR_H
+
+#include <string.h>
+#include <limits.h>
+
+#include "brg_endian.h"
+
+/* This define sets the units in which buffers are processed. This code
+ can provide significant speed gains if buffers can be processed in
+ 32 or 64 bit chunks rather than in bytes. This define sets the units
+ in which buffers will be accessed if possible
+*/
+#if !defined( UNIT_BITS )
+# if 1
+# define UNIT_BITS 64
+# elif 0
+# define UNIT_BITS 32
+# else
+# define UNIT_BITS 8
+# endif
+#endif
+
+#if UNIT_BITS == 64 && !defined( NEED_UINT_64T )
+# define NEED_UINT_64T
+#endif
+
+#include "brg_types.h"
+
+/* Use of inlines is preferred but code blocks can also be expanded inline
+ using 'defines'. But the latter approach will typically generate a LOT
+ of code and is not recommended.
+*/
+#if 1 && !defined( USE_INLINING )
+# define USE_INLINING
+#endif
+
+#if defined( _MSC_VER )
+# if _MSC_VER >= 1400
+# include <stdlib.h>
+# include <intrin.h>
+# pragma intrinsic(memset)
+# pragma intrinsic(memcpy)
+# define rotl32 _rotl
+# define rotr32 _rotr
+# define rotl64 _rotl64
+# define rotr64 _rotl64
+# define bswap_16(x) _byteswap_ushort(x)
+# define bswap_32(x) _byteswap_ulong(x)
+# define bswap_64(x) _byteswap_uint64(x)
+# else
+# define rotl32 _lrotl
+# define rotr32 _lrotr
+# endif
+#endif
+
+#if defined( USE_INLINING )
+# if defined( _MSC_VER )
+# define mh_decl __inline
+# elif defined( __GNUC__ ) || defined( __GNU_LIBRARY__ )
+# define mh_decl static inline
+# else
+# define mh_decl static
+# endif
+#endif
+
+#if defined(__cplusplus)
+extern "C" {
+#endif
+
+#define UI8_PTR(x) UPTR_CAST(x, 8)
+#define UI16_PTR(x) UPTR_CAST(x, 16)
+#define UI32_PTR(x) UPTR_CAST(x, 32)
+#define UI64_PTR(x) UPTR_CAST(x, 64)
+#define UNIT_PTR(x) UPTR_CAST(x, UNIT_BITS)
+
+#define BUF_INC (UNIT_BITS >> 3)
+#define BUF_ADRMASK ((UNIT_BITS >> 3) - 1)
+
+#define rep2_u2(f,r,x) f( 0,r,x); f( 1,r,x)
+#define rep2_u4(f,r,x) f( 0,r,x); f( 1,r,x); f( 2,r,x); f( 3,r,x)
+#define rep2_u16(f,r,x) f( 0,r,x); f( 1,r,x); f( 2,r,x); f( 3,r,x); \
+ f( 4,r,x); f( 5,r,x); f( 6,r,x); f( 7,r,x); \
+ f( 8,r,x); f( 9,r,x); f(10,r,x); f(11,r,x); \
+ f(12,r,x); f(13,r,x); f(14,r,x); f(15,r,x)
+
+#define rep2_d2(f,r,x) f( 1,r,x); f( 0,r,x)
+#define rep2_d4(f,r,x) f( 3,r,x); f( 2,r,x); f( 1,r,x); f( 0,r,x)
+#define rep2_d16(f,r,x) f(15,r,x); f(14,r,x); f(13,r,x); f(12,r,x); \
+ f(11,r,x); f(10,r,x); f( 9,r,x); f( 8,r,x); \
+ f( 7,r,x); f( 6,r,x); f( 5,r,x); f( 4,r,x); \
+ f( 3,r,x); f( 2,r,x); f( 1,r,x); f( 0,r,x)
+
+#define rep3_u2(f,r,x,y) f( 0,r,x,y); f( 1,r,x,y)
+#define rep3_u4(f,r,x,y) f( 0,r,x,y); f( 1,r,x,y); f( 2,r,x,y); f( 3,r,x,y)
+#define rep3_u16(f,r,x,y) f( 0,r,x,y); f( 1,r,x,y); f( 2,r,x,y); f( 3,r,x,y); \
+ f( 4,r,x,y); f( 5,r,x,y); f( 6,r,x,y); f( 7,r,x,y); \
+ f( 8,r,x,y); f( 9,r,x,y); f(10,r,x,y); f(11,r,x,y); \
+ f(12,r,x,y); f(13,r,x,y); f(14,r,x,y); f(15,r,x,y)
+
+#define rep3_d2(f,r,x,y) f( 1,r,x,y); f( 0,r,x,y)
+#define rep3_d4(f,r,x,y) f( 3,r,x,y); f( 2,r,x,y); f( 1,r,x,y); f( 0,r,x,y)
+#define rep3_d16(f,r,x,y) f(15,r,x,y); f(14,r,x,y); f(13,r,x,y); f(12,r,x,y); \
+ f(11,r,x,y); f(10,r,x,y); f( 9,r,x,y); f( 8,r,x,y); \
+ f( 7,r,x,y); f( 6,r,x,y); f( 5,r,x,y); f( 4,r,x,y); \
+ f( 3,r,x,y); f( 2,r,x,y); f( 1,r,x,y); f( 0,r,x,y)
+
+/* function pointers might be used for fast XOR operations */
+
+typedef void (*xor_function)(void* r, const void* p, const void* q);
+
+/* left and right rotates on 32 and 64 bit variables */
+
+#if !defined( rotl32 ) // NOTE: 0 <= n <= 32 ASSUMED
+mh_decl uint_32t rotl32(uint_32t x, int n)
+{
+ return (((x) << n) | ((x) >> (32 - n)));
+}
+#endif
+
+#if !defined( rotr32 ) // NOTE: 0 <= n <= 32 ASSUMED
+mh_decl uint_32t rotr32(uint_32t x, int n)
+{
+ return (((x) >> n) | ((x) << (32 - n)));
+}
+#endif
+
+#if !defined( rotl64 ) // NOTE: 0 <= n <= 64 ASSUMED
+mh_decl uint_64t rotl64(uint_64t x, int n)
+{
+ return (((x) << n) | ((x) >> (64 - n)));
+}
+#endif
+
+#if !defined( rotr64 ) // NOTE: 0 <= n <= 64 ASSUMED
+mh_decl uint_64t rotr64(uint_64t x, int n)
+{
+ return (((x) >> n) | ((x) << (64 - n)));
+}
+#endif
+
+/* byte order inversions for 16, 32 and 64 bit variables */
+
+#if !defined(bswap_16)
+mh_decl uint_16t bswap_16(uint_16t x)
+{
+ return (x >> 8) | (x << 8);
+}
+#endif
+
+#if !defined(bswap_32)
+mh_decl uint_32t bswap_32(uint_32t x)
+{
+ return ((rotr32((x), 24) & 0x00ff00ff) | (rotr32((x), 8) & 0xff00ff00));
+}
+#endif
+
+#if !defined(bswap_64)
+mh_decl uint_64t bswap_64(uint_64t x)
+{
+ return bswap_32((uint_32t)(x >> 32)) | ((uint_64t)bswap_32((uint_32t)x) << 32);
+}
+#endif
+/* support for fast aligned buffer move, xor and byte swap operations -
+ source and destination buffers for move and xor operations must not
+ overlap, those for byte order revesal must either not overlap or
+ must be identical
+*/
+#define f_copy(n,p,q) p[n] = q[n]
+#define f_xor(n,r,p,q) r[n] = p[n] ^ q[n]
+
+mh_decl void copy_block(void* p, const void* q)
+{
+ memcpy(p, q, 16);
+}
+
+mh_decl void copy_block_aligned(void *p, const void *q)
+{
+#if UNIT_BITS == 8
+ memcpy(p, q, 16);
+#elif UNIT_BITS == 32
+ rep2_u4(f_copy,UNIT_PTR(p),UNIT_PTR(q));
+#else
+ rep2_u2(f_copy,UNIT_PTR(p),UNIT_PTR(q));
+#endif
+}
+
+mh_decl void xor_block(void *r, const void* p, const void* q)
+{
+ rep3_u16(f_xor, UI8_PTR(r), UI8_PTR(p), UI8_PTR(q));
+}
+
+mh_decl void xor_block_aligned(void *r, const void *p, const void *q)
+{
+#if UNIT_BITS == 8
+ rep3_u16(f_xor, UNIT_PTR(r), UNIT_PTR(p), UNIT_PTR(q));
+#elif UNIT_BITS == 32
+ rep3_u4(f_xor, UNIT_PTR(r), UNIT_PTR(p), UNIT_PTR(q));
+#else
+ rep3_u2(f_xor, UNIT_PTR(r), UNIT_PTR(p), UNIT_PTR(q));
+#endif
+}
+
+mh_decl void bswap32_block(void *d, const void* s)
+{
+#if UNIT_BITS == 8
+ uint_8t t;
+ t = UNIT_PTR(s)[ 0]; UNIT_PTR(d)[ 0] = UNIT_PTR(s)[ 3]; UNIT_PTR(d)[ 3] = t;
+ t = UNIT_PTR(s)[ 1]; UNIT_PTR(d)[ 1] = UNIT_PTR(s)[ 2]; UNIT_PTR(d)[ 2] = t;
+ t = UNIT_PTR(s)[ 4]; UNIT_PTR(d)[ 4] = UNIT_PTR(s)[ 7]; UNIT_PTR(d)[ 7] = t;
+ t = UNIT_PTR(s)[ 5]; UNIT_PTR(d)[ 5] = UNIT_PTR(s)[ 6]; UNIT_PTR(d) [6] = t;
+ t = UNIT_PTR(s)[ 8]; UNIT_PTR(d)[ 8] = UNIT_PTR(s)[11]; UNIT_PTR(d)[12] = t;
+ t = UNIT_PTR(s)[ 9]; UNIT_PTR(d)[ 9] = UNIT_PTR(s)[10]; UNIT_PTR(d)[10] = t;
+ t = UNIT_PTR(s)[12]; UNIT_PTR(d)[12] = UNIT_PTR(s)[15]; UNIT_PTR(d)[15] = t;
+ t = UNIT_PTR(s)[13]; UNIT_PTR(d)[ 3] = UNIT_PTR(s)[14]; UNIT_PTR(d)[14] = t;
+#elif UNIT_BITS == 32
+ UNIT_PTR(d)[0] = bswap_32(UNIT_PTR(s)[0]); UNIT_PTR(d)[1] = bswap_32(UNIT_PTR(s)[1]);
+ UNIT_PTR(d)[2] = bswap_32(UNIT_PTR(s)[2]); UNIT_PTR(d)[3] = bswap_32(UNIT_PTR(s)[3]);
+#else
+ UI32_PTR(d)[0] = bswap_32(UI32_PTR(s)[0]); UI32_PTR(d)[1] = bswap_32(UI32_PTR(s)[1]);
+ UI32_PTR(d)[2] = bswap_32(UI32_PTR(s)[2]); UI32_PTR(d)[3] = bswap_32(UI32_PTR(s)[3]);
+#endif
+}
+
+mh_decl void bswap64_block(void *d, const void* s)
+{
+#if UNIT_BITS == 8
+ uint_8t t;
+ t = UNIT_PTR(s)[ 0]; UNIT_PTR(d)[ 0] = UNIT_PTR(s)[ 7]; UNIT_PTR(d)[ 7] = t;
+ t = UNIT_PTR(s)[ 1]; UNIT_PTR(d)[ 1] = UNIT_PTR(s)[ 6]; UNIT_PTR(d)[ 6] = t;
+ t = UNIT_PTR(s)[ 2]; UNIT_PTR(d)[ 2] = UNIT_PTR(s)[ 5]; UNIT_PTR(d)[ 5] = t;
+ t = UNIT_PTR(s)[ 3]; UNIT_PTR(d)[ 3] = UNIT_PTR(s)[ 3]; UNIT_PTR(d) [3] = t;
+ t = UNIT_PTR(s)[ 8]; UNIT_PTR(d)[ 8] = UNIT_PTR(s)[15]; UNIT_PTR(d)[15] = t;
+ t = UNIT_PTR(s)[ 9]; UNIT_PTR(d)[ 9] = UNIT_PTR(s)[14]; UNIT_PTR(d)[14] = t;
+ t = UNIT_PTR(s)[10]; UNIT_PTR(d)[10] = UNIT_PTR(s)[13]; UNIT_PTR(d)[13] = t;
+ t = UNIT_PTR(s)[11]; UNIT_PTR(d)[11] = UNIT_PTR(s)[12]; UNIT_PTR(d)[12] = t;
+#elif UNIT_BITS == 32
+ uint_32t t;
+ t = bswap_32(UNIT_PTR(s)[0]); UNIT_PTR(d)[0] = bswap_32(UNIT_PTR(s)[1]); UNIT_PTR(d)[1] = t;
+ t = bswap_32(UNIT_PTR(s)[2]); UNIT_PTR(d)[2] = bswap_32(UNIT_PTR(s)[2]); UNIT_PTR(d)[3] = t;
+#else
+ UNIT_PTR(d)[0] = bswap_64(UNIT_PTR(s)[0]); UNIT_PTR(d)[1] = bswap_64(UNIT_PTR(s)[1]);
+#endif
+}
+
+mh_decl void bswap128_block(void *d, const void* s)
+{
+#if UNIT_BITS == 8
+ uint_8t t;
+ t = UNIT_PTR(s)[0]; UNIT_PTR(d)[0] = UNIT_PTR(s)[15]; UNIT_PTR(d)[15] = t;
+ t = UNIT_PTR(s)[1]; UNIT_PTR(d)[1] = UNIT_PTR(s)[14]; UNIT_PTR(d)[14] = t;
+ t = UNIT_PTR(s)[2]; UNIT_PTR(d)[2] = UNIT_PTR(s)[13]; UNIT_PTR(d)[13] = t;
+ t = UNIT_PTR(s)[3]; UNIT_PTR(d)[3] = UNIT_PTR(s)[12]; UNIT_PTR(d)[12] = t;
+ t = UNIT_PTR(s)[4]; UNIT_PTR(d)[4] = UNIT_PTR(s)[11]; UNIT_PTR(d)[11] = t;
+ t = UNIT_PTR(s)[5]; UNIT_PTR(d)[5] = UNIT_PTR(s)[10]; UNIT_PTR(d)[10] = t;
+ t = UNIT_PTR(s)[6]; UNIT_PTR(d)[6] = UNIT_PTR(s)[ 9]; UNIT_PTR(d)[ 9] = t;
+ t = UNIT_PTR(s)[7]; UNIT_PTR(d)[7] = UNIT_PTR(s)[ 8]; UNIT_PTR(d)[ 8] = t;
+#elif UNIT_BITS == 32
+ uint_32t t;
+ t = bswap_32(UNIT_PTR(s)[0]); UNIT_PTR(d)[0] = bswap_32(UNIT_PTR(s)[3]); UNIT_PTR(d)[3] = t;
+ t = bswap_32(UNIT_PTR(s)[1]); UNIT_PTR(d)[1] = bswap_32(UNIT_PTR(s)[2]); UNIT_PTR(d)[2] = t;
+#else
+ uint_64t t;
+ t = bswap_64(UNIT_PTR(s)[0]); UNIT_PTR(d)[0] = bswap_64(UNIT_PTR(s)[1]); UNIT_PTR(d)[1] = t;
+#endif
+}
+
+/* platform byte order to big or little endian order for 16, 32 and 64 bit variables */
+
+#if PLATFORM_BYTE_ORDER == IS_BIG_ENDIAN
+
+# define uint_16t_to_le(x) (x) = bswap_16((x))
+# define uint_32t_to_le(x) (x) = bswap_32((x))
+# define uint_64t_to_le(x) (x) = bswap_64((x))
+# define uint_16t_to_be(x)
+# define uint_32t_to_be(x)
+# define uint_64t_to_be(x)
+
+#else
+
+# define uint_16t_to_le(x)
+# define uint_32t_to_le(x)
+# define uint_64t_to_le(x)
+# define uint_16t_to_be(x) (x) = bswap_16((x))
+# define uint_32t_to_be(x) (x) = bswap_32((x))
+# define uint_64t_to_be(x) (x) = bswap_64((x))
+
+#endif
+
+#if defined(__cplusplus)
+}
+#endif
+
+#endif
diff --git a/contrib/pgcrypto/pgcrypto.c b/contrib/pgcrypto/pgcrypto.c
index 2d446d8..8599203 100644
--- a/contrib/pgcrypto/pgcrypto.c
+++ b/contrib/pgcrypto/pgcrypto.c
@@ -34,12 +34,15 @@
#include <ctype.h>
#include "parser/scansup.h"
+#include "storage/encryption.h"
#include "utils/builtins.h"
#include "utils/uuid.h"
#include "px.h"
#include "px-crypt.h"
#include "pgcrypto.h"
+#include "sha2.h"
+#include "xts.h"
PG_MODULE_MAGIC;
@@ -47,6 +50,23 @@ PG_MODULE_MAGIC;
typedef int (*PFN) (const char *name, void **res);
static void *find_provider(text *name, PFN pf, char *desc, int silent);
+static bool pgcrypto_encryption_setup();
+static void pgcrypto_encrypt_block(const char *input,
+ char *output, Size size, const char *tweak);
+static void pgcrypto_decrypt_block(const char *input,
+ char *output, Size size, const char *tweak);
+void _PG_init(void);
+
+/*
+ * Encryption and decryption keys for full database encryption support.
+ */
+typedef struct {
+ xts_encrypt_ctx enc_ctx[1];
+ xts_decrypt_ctx dec_ctx[1];
+} db_encryption_ctx;
+
+/* Full database encryption key, initialized by pgcrypto_encryption_setup. */
+static db_encryption_ctx db_key;
/* SQL function: hash(bytea, text) returns bytea */
PG_FUNCTION_INFO_V1(pg_digest);
@@ -494,3 +514,72 @@ find_provider(text *name,
return err ? NULL : res;
}
+
+/*
+ * Pgcrypto module does AES-128-XTS encryption.
+ */
+static bool
+pgcrypto_encryption_setup()
+{
+ uint8 key[32];
+ char *passphrase = getenv("PGENCRYPTIONKEY");
+
+ /* Empty or missing passphrase means that encryption is not configured */
+ if (passphrase == NULL || passphrase[0] == '\0')
+ {
+ ereport(LOG,
+ (errmsg("encryption key not provided"),
+ errdetail("The database cluster was initialized with encryption"
+ " but the server was started without an encryption key."),
+ errhint("Set the key using PGENCRYPTIONKEY environment variable.")));
+ return false;
+ }
+
+ /* TODO: replace with PBKDF2 or scrypt */
+ {
+ SHA256_CTX sha_ctx;
+ SHA256_Init(&sha_ctx);
+ SHA256_Update(&sha_ctx, (uint8*) passphrase, strlen(passphrase));
+ SHA256_Final(key, &sha_ctx);
+ }
+
+ if (xts_encrypt_key(key, 32, db_key.enc_ctx) != EXIT_SUCCESS ||
+ xts_decrypt_key(key, 32, db_key.dec_ctx) != EXIT_SUCCESS)
+ {
+ elog(ERROR, "Encryption key setup failed.");
+ return false;
+ }
+
+ return true;
+}
+
+static void
+pgcrypto_encrypt_block(const char *input, char *output, Size size,
+ const char *tweak)
+{
+ if (input != output)
+ memcpy(output, input, size);
+
+ xts_encrypt_block((uint8*) output, (const uint8*) tweak, size, db_key.enc_ctx);
+}
+
+static void
+pgcrypto_decrypt_block(const char *input, char *output, Size size,
+ const char *tweak)
+{
+ if (input != output)
+ memcpy(output, input, size);
+
+ xts_decrypt_block((uint8*) output, (const uint8*) tweak, size, db_key.dec_ctx);
+}
+
+void
+_PG_init(void)
+{
+ EncryptionRoutines routines;
+ routines.SetupEncryption = &pgcrypto_encryption_setup;
+ routines.EncryptBlock = &pgcrypto_encrypt_block;
+ routines.DecryptBlock = &pgcrypto_decrypt_block;
+
+ register_encryption_module("pgcrypto", &routines);
+}
diff --git a/contrib/pgcrypto/xts.c b/contrib/pgcrypto/xts.c
new file mode 100644
index 0000000..4efc2bc
--- /dev/null
+++ b/contrib/pgcrypto/xts.c
@@ -0,0 +1,220 @@
+/*
+ ---------------------------------------------------------------------------
+ Copyright (c) 1998-2008, Brian Gladman, Worcester, UK. All rights reserved.
+
+ LICENSE TERMS
+
+ The redistribution and use of this software (with or without changes)
+ is allowed without the payment of fees or royalties provided that:
+
+ 1. source code distributions include the above copyright notice, this
+ list of conditions and the following disclaimer;
+
+ 2. binary distributions include the above copyright notice, this list
+ of conditions and the following disclaimer in their documentation;
+
+ 3. the name of the copyright holder is not used to endorse products
+ built using this software without specific written permission.
+
+ DISCLAIMER
+
+ This software is provided 'as is' with no explicit or implied warranties
+ in respect of its properties, including, but not limited to, correctness
+ and/or fitness for purpose.
+ ---------------------------------------------------------------------------
+ Issue Date: 20/12/2007
+
+ My thanks to both Doug Whiting and Olaf Pors for their much appreciated
+ assistance in debugging and testing this code.
+*/
+
+#include "postgres.h"
+
+#include "mode_hdr.h"
+#include "xts.h"
+
+static void gf_mulx(void *x);
+
+UNIT_TYPEDEF(buf_unit, UNIT_BITS);
+BUFR_TYPEDEF(buf_type, UNIT_BITS, AES_BLOCK_SIZE);
+
+static void gf_mulx(void *x)
+{
+#if UNIT_BITS == 8
+
+ uint_8t i = 16, t = ((uint_8t*)x)[15];
+ while(--i)
+ ((uint_8t*)x)[i] = (((uint_8t*)x)[i] << 1) | (((uint_8t*)x)[i - 1] & 0x80 ? 1 : 0);
+ ((uint_8t*)x)[0] = (((uint_8t*)x)[0] << 1) ^ (t & 0x80 ? 0x87 : 0x00);
+
+#elif PLATFORM_BYTE_ORDER == IS_LITTLE_ENDIAN
+
+# if UNIT_BITS == 64
+
+# define GF_MASK li_64(8000000000000000)
+# define GF_XOR li_64(0000000000000087)
+ uint_64t _tt = ((UPTR_CAST(x,64)[1] & GF_MASK) ? GF_XOR : 0);
+ UPTR_CAST(x,64)[1] = (UPTR_CAST(x,64)[1] << 1) | (UPTR_CAST(x,64)[0] & GF_MASK ? 1 : 0);
+ UPTR_CAST(x,64)[0] = (UPTR_CAST(x,64)[0] << 1) ^ _tt;
+
+# else /* UNIT_BITS == 32 */
+
+# define GF_MASK li_32(80000000)
+# define GF_XOR li_32(00000087)
+ uint_32t _tt = ((UPTR_CAST(x,32)[3] & GF_MASK) ? GF_XOR : 0);;
+ UPTR_CAST(x,32)[3] = (UPTR_CAST(x,32)[3] << 1) | (UPTR_CAST(x,32)[2] & GF_MASK ? 1 : 0);
+ UPTR_CAST(x,32)[2] = (UPTR_CAST(x,32)[2] << 1) | (UPTR_CAST(x,32)[1] & GF_MASK ? 1 : 0);
+ UPTR_CAST(x,32)[1] = (UPTR_CAST(x,32)[1] << 1) | (UPTR_CAST(x,32)[0] & GF_MASK ? 1 : 0);
+ UPTR_CAST(x,32)[0] = (UPTR_CAST(x,32)[0] << 1) ^ _tt;
+
+# endif
+
+#else /* PLATFORM_BYTE_ORDER == IS_BIG_ENDIAN */
+
+# if UNIT_BITS == 64
+
+# define MASK_01 li_64(0101010101010101)
+# define GF_MASK li_64(0000000000000080)
+# define GF_XOR li_64(8700000000000000)
+ uint_64t _tt = ((UPTR_CAST(x,64)[1] & GF_MASK) ? GF_XOR : 0);
+ UPTR_CAST(x,64)[1] = ((UPTR_CAST(x,64)[1] << 1) & ~MASK_01)
+ | (((UPTR_CAST(x,64)[1] >> 15) | (UPTR_CAST(x,64)[0] << 49)) & MASK_01);
+ UPTR_CAST(x,64)[0] = (((UPTR_CAST(x,64)[0] << 1) & ~MASK_01)
+ | ((UPTR_CAST(x,64)[0] >> 15) & MASK_01)) ^ _tt;
+
+# else /* UNIT_BITS == 32 */
+
+# define MASK_01 li_32(01010101)
+# define GF_MASK li_32(00000080)
+# define GF_XOR li_32(87000000)
+ uint_32t _tt = ((UPTR_CAST(x,32)[3] & GF_MASK) ? GF_XOR : 0);
+ UPTR_CAST(x,32)[3] = ((UPTR_CAST(x,32)[3] << 1) & ~MASK_01)
+ | (((UPTR_CAST(x,32)[3] >> 15) | (UPTR_CAST(x,32)[2] << 17)) & MASK_01);
+ UPTR_CAST(x,32)[2] = ((UPTR_CAST(x,32)[2] << 1) & ~MASK_01)
+ | (((UPTR_CAST(x,32)[2] >> 15) | (UPTR_CAST(x,32)[1] << 17)) & MASK_01);
+ UPTR_CAST(x,32)[1] = ((UPTR_CAST(x,32)[1] << 1) & ~MASK_01)
+ | (((UPTR_CAST(x,32)[1] >> 15) | (UPTR_CAST(x,32)[0] << 17)) & MASK_01);
+ UPTR_CAST(x,32)[0] = (((UPTR_CAST(x,32)[0] << 1) & ~MASK_01)
+ | ((UPTR_CAST(x,32)[0] >> 15) & MASK_01)) ^ _tt;
+
+# endif
+
+#endif
+}
+
+INT_RETURN xts_encrypt_key( const unsigned char key[], int key_len, xts_encrypt_ctx ctx[1] )
+{ int aes_klen_by;
+
+ switch( key_len )
+ {
+ default: return EXIT_FAILURE;
+ case 32:
+ case 256: aes_klen_by = 16; break;
+ case 64:
+ case 512: aes_klen_by = 32; break;
+ }
+
+ rijndael_set_key(ctx->enc_ctx, UPTR_CAST(key, 32), aes_klen_by*8, 1);
+ rijndael_set_key(ctx->twk_ctx, UPTR_CAST(key + aes_klen_by, 32), aes_klen_by*8, 1);
+ return EXIT_SUCCESS;
+}
+
+INT_RETURN xts_decrypt_key( const unsigned char key[], int key_len, xts_decrypt_ctx ctx[1] )
+{ int aes_klen_by;
+
+ switch( key_len )
+ {
+ default: return EXIT_FAILURE;
+ case 32:
+ case 256: aes_klen_by = 16; break;
+ case 64:
+ case 512: aes_klen_by = 32; break;
+ }
+
+ rijndael_set_key(ctx->dec_ctx, UPTR_CAST(key, 32), aes_klen_by*8, 0);
+ rijndael_set_key(ctx->twk_ctx, UPTR_CAST(key + aes_klen_by, 32), aes_klen_by*8, 1);
+ return EXIT_SUCCESS;
+}
+
+INT_RETURN xts_encrypt_block( unsigned char sector[], const unsigned char tweak[],
+ unsigned int sector_len, xts_encrypt_ctx ctx[1] )
+{
+ buf_type hh;
+ uint_8t *pos = sector, *hi = sector + sector_len;
+
+ xor_function f_ptr = (!ALIGN_OFFSET(sector, UNIT_BITS >> 3) ? xor_block_aligned : xor_block );
+
+ if( sector_len < AES_BLOCK_SIZE )
+ return EXIT_FAILURE;
+
+ rijndael_encrypt(ctx->twk_ctx, UPTR_CAST(tweak, 32), UPTR_CAST(hh, 32));
+
+ while(pos + AES_BLOCK_SIZE <= hi)
+ {
+ f_ptr(pos, pos, hh);
+ rijndael_encrypt(ctx->enc_ctx, UPTR_CAST(pos, 32), UPTR_CAST(pos, 32));
+ f_ptr(pos, pos, hh);
+ pos += AES_BLOCK_SIZE;
+ gf_mulx(hh);
+ }
+
+ if(pos < hi)
+ {
+ uint_8t *tp = pos - AES_BLOCK_SIZE;
+ while(pos < hi)
+ {
+ uint_8t tt = *(pos - AES_BLOCK_SIZE);
+ *(pos - AES_BLOCK_SIZE) = *pos;
+ *pos++ = tt;
+ }
+ f_ptr(tp, tp, hh);
+ rijndael_encrypt(ctx->enc_ctx, UPTR_CAST(tp, 32), UPTR_CAST(tp, 32));
+ f_ptr(tp, tp, hh);
+ }
+ return EXIT_SUCCESS;
+}
+
+INT_RETURN xts_decrypt_block( unsigned char sector[], const unsigned char tweak[],
+ unsigned int sector_len, xts_decrypt_ctx ctx[1] )
+{
+ buf_type hh, hh2;
+ uint_8t *pos = sector, *hi = sector + sector_len;
+
+ xor_function f_ptr = (!ALIGN_OFFSET(sector, UNIT_BITS >> 3) ? xor_block_aligned : xor_block );
+
+ if( sector_len < AES_BLOCK_SIZE )
+ return EXIT_FAILURE;
+
+ rijndael_encrypt(ctx->twk_ctx, UPTR_CAST(tweak, 32), UPTR_CAST(hh, 32));
+
+ while(pos + AES_BLOCK_SIZE <= hi)
+ {
+ if(hi - pos > AES_BLOCK_SIZE && hi - pos < 2 * AES_BLOCK_SIZE)
+ {
+ memcpy(hh2, hh, AES_BLOCK_SIZE);
+ gf_mulx(hh);
+ }
+ f_ptr(pos, pos, hh);
+ rijndael_decrypt(ctx->dec_ctx, UPTR_CAST(pos, 32), UPTR_CAST(pos, 32));
+ f_ptr(pos, pos, hh);
+ pos += AES_BLOCK_SIZE;
+ gf_mulx(hh);
+ }
+
+ if(pos < hi)
+ {
+ uint_8t *tp = pos - AES_BLOCK_SIZE;
+ while(pos < hi)
+ {
+ uint_8t tt = *(pos - AES_BLOCK_SIZE);
+ *(pos - AES_BLOCK_SIZE) = *pos;
+ *pos++ = tt;
+ }
+ f_ptr(tp, tp, hh2);
+ rijndael_decrypt(ctx->dec_ctx, UPTR_CAST(tp, 32), UPTR_CAST(tp, 32));
+ f_ptr(tp, tp, hh2);
+ }
+
+ return EXIT_SUCCESS;
+}
+
diff --git a/contrib/pgcrypto/xts.h b/contrib/pgcrypto/xts.h
new file mode 100644
index 0000000..88b2b7b
--- /dev/null
+++ b/contrib/pgcrypto/xts.h
@@ -0,0 +1,62 @@
+/*
+ ---------------------------------------------------------------------------
+ Copyright (c) 1998-2008, Brian Gladman, Worcester, UK. All rights reserved.
+
+ LICENSE TERMS
+
+ The redistribution and use of this software (with or without changes)
+ is allowed without the payment of fees or royalties provided that:
+
+ 1. source code distributions include the above copyright notice, this
+ list of conditions and the following disclaimer;
+
+ 2. binary distributions include the above copyright notice, this list
+ of conditions and the following disclaimer in their documentation;
+
+ 3. the name of the copyright holder is not used to endorse products
+ built using this software without specific written permission.
+
+ DISCLAIMER
+
+ This software is provided 'as is' with no explicit or implied warranties
+ in respect of its properties, including, but not limited to, correctness
+ and/or fitness for purpose.
+ ---------------------------------------------------------------------------
+ Issue Date: 20/12/2007
+*/
+
+#ifndef _XTS_H
+#define _XTS_H
+
+#include "rijndael.h"
+
+/* lifted from aes.h */
+#define AES_BLOCK_SIZE 16 /* the AES block size in bytes */
+
+/* end */
+
+typedef struct
+{
+ rijndael_ctx twk_ctx[1];
+ rijndael_ctx enc_ctx[1];
+} xts_encrypt_ctx;
+
+typedef struct
+{
+ rijndael_ctx twk_ctx[1];
+ rijndael_ctx dec_ctx[1];
+} xts_decrypt_ctx;
+
+#define INT_RETURN int
+
+INT_RETURN xts_encrypt_key( const unsigned char key[], int key_len, xts_encrypt_ctx ctx[1] );
+
+INT_RETURN xts_decrypt_key( const unsigned char key[], int key_len, xts_decrypt_ctx ctx[1] );
+
+INT_RETURN xts_encrypt_block( unsigned char sector[], const unsigned char tweak[],
+ unsigned int sector_len, xts_encrypt_ctx ctx[1] );
+
+INT_RETURN xts_decrypt_block( unsigned char sector[], const unsigned char tweak[],
+ unsigned int sector_len, xts_decrypt_ctx ctx[1] );
+
+#endif
diff --git a/src/backend/access/transam/slru.c b/src/backend/access/transam/slru.c
index 36a011c..43da67d 100644
--- a/src/backend/access/transam/slru.c
+++ b/src/backend/access/transam/slru.c
@@ -54,6 +54,7 @@
#include "access/slru.h"
#include "access/transam.h"
#include "access/xlog.h"
+#include "storage/encryption.h"
#include "storage/fd.h"
#include "storage/shmem.h"
#include "miscadmin.h"
@@ -121,6 +122,8 @@ typedef enum
static SlruErrorCause slru_errcause;
static int slru_errno;
+static char slru_encryption_buf[BLCKSZ];
+static char slru_encryption_tweak[TWEAK_SIZE];
static void SimpleLruZeroLSNs(SlruCtl ctl, int slotno);
@@ -135,6 +138,7 @@ static int SlruSelectLRUPage(SlruCtl ctl, int pageno);
static bool SlruScanDirCbDeleteCutoff(SlruCtl ctl, char *filename,
int segpage, void *data);
static void SlruInternalDeleteSegment(SlruCtl ctl, char *filename);
+static void SlruEncryptionTweak(char *tweak, int pageno);
/*
* Initialization of shared memory
@@ -641,6 +645,7 @@ SlruPhysicalReadPage(SlruCtl ctl, int pageno, int slotno)
int offset = rpageno * BLCKSZ;
char path[MAXPGPATH];
int fd;
+ char *rbuf;
SlruFileName(ctl, path, segno);
@@ -676,8 +681,13 @@ SlruPhysicalReadPage(SlruCtl ctl, int pageno, int slotno)
return false;
}
+ if (encryption_enabled)
+ rbuf = slru_encryption_buf;
+ else
+ rbuf = shared->page_buffer[slotno];
+
errno = 0;
- if (read(fd, shared->page_buffer[slotno], BLCKSZ) != BLCKSZ)
+ if (read(fd, rbuf, BLCKSZ) != BLCKSZ)
{
slru_errcause = SLRU_READ_FAILED;
slru_errno = errno;
@@ -685,6 +695,14 @@ SlruPhysicalReadPage(SlruCtl ctl, int pageno, int slotno)
return false;
}
+
+ if (encryption_enabled)
+ {
+ SlruEncryptionTweak(slru_encryption_tweak, pageno);
+ decrypt_block(slru_encryption_buf, shared->page_buffer[slotno],
+ BLCKSZ, slru_encryption_tweak);
+ }
+
if (CloseTransientFile(fd))
{
slru_errcause = SLRU_CLOSE_FAILED;
@@ -718,6 +736,7 @@ SlruPhysicalWritePage(SlruCtl ctl, int pageno, int slotno, SlruFlush fdata)
int offset = rpageno * BLCKSZ;
char path[MAXPGPATH];
int fd = -1;
+ char *wbuf;
/*
* Honor the write-WAL-before-data rule, if appropriate, so that we do not
@@ -835,8 +854,16 @@ SlruPhysicalWritePage(SlruCtl ctl, int pageno, int slotno, SlruFlush fdata)
return false;
}
+ wbuf = shared->page_buffer[slotno];
+ if (encryption_enabled)
+ {
+ SlruEncryptionTweak(slru_encryption_tweak, pageno);
+ encrypt_block(wbuf, slru_encryption_buf, BLCKSZ, slru_encryption_tweak);
+ wbuf = slru_encryption_buf;
+ }
+
errno = 0;
- if (write(fd, shared->page_buffer[slotno], BLCKSZ) != BLCKSZ)
+ if (write(fd, wbuf, BLCKSZ) != BLCKSZ)
{
/* if write didn't set errno, assume problem is no disk space */
if (errno == 0)
@@ -1392,3 +1419,14 @@ SlruScanDirectory(SlruCtl ctl, SlruScanCallback callback, void *data)
return retval;
}
+
+/*
+ * SLRU data encryption is tweaked by page number.
+ */
+static void
+SlruEncryptionTweak(char *tweak, int pageno)
+{
+ /* TODO: would be nice to incorporate SLRU type in tweak */
+ memcpy(tweak, &pageno, sizeof(pageno));
+ memset(tweak + sizeof(pageno), 0, TWEAK_SIZE - sizeof(pageno));
+}
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index b473f19..640374d 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -53,6 +53,7 @@
#include "replication/walsender.h"
#include "storage/barrier.h"
#include "storage/bufmgr.h"
+#include "storage/encryption.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/large_object.h"
@@ -74,6 +75,8 @@
#include "pg_trace.h"
extern uint32 bootstrap_data_checksum_version;
+extern bool bootstrap_data_encrypted;
+extern char *bootstrap_encryption_sample;
/* File path names (all relative to $PGDATA) */
#define RECOVERY_COMMAND_FILE "recovery.conf"
@@ -803,6 +806,7 @@ static void LocalSetXLogInsertAllowed(void);
static void CreateEndOfRecoveryRecord(void);
static void CheckPointGuts(XLogRecPtr checkPointRedo, int flags);
static void KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo);
+static void XLogWritePages(char *from, int npages);
static XLogRecPtr XLogGetReplicationSlotMinimumLSN(void);
static void AdvanceXLInsertBuffer(XLogRecPtr upto, bool opportunistic);
@@ -2260,8 +2264,6 @@ XLogWrite(XLogwrtRqst WriteRqst, bool flexible)
{
char *from;
Size nbytes;
- Size nleft;
- int written;
/* Need to seek in the file? */
if (openLogOff != startoffset)
@@ -2278,28 +2280,28 @@ XLogWrite(XLogwrtRqst WriteRqst, bool flexible)
/* OK to write the page(s) */
from = XLogCtl->pages + startidx * (Size) XLOG_BLCKSZ;
nbytes = npages * (Size) XLOG_BLCKSZ;
- nleft = nbytes;
- do
- {
- errno = 0;
- written = write(openLogFile, from, nleft);
- if (written <= 0)
- {
- if (errno == EINTR)
- continue;
- ereport(PANIC,
- (errcode_for_file_access(),
- errmsg("could not write to log file %s "
- "at offset %u, length %zu: %m",
- XLogFileNameP(ThisTimeLineID, openLogSegNo),
- openLogOff, nbytes)));
- }
- nleft -= written;
- from += written;
- } while (nleft > 0);
+ if (encryption_enabled) {
+ int i;
+ /*
+ * XXX: use larger encryption buffer to enable larger writes
+ * and reduce number of syscalls?
+ */
+ for (i = 0; i < npages; i++) {
+ char buf[XLOG_BLCKSZ];
+ char tweak[TWEAK_SIZE];
+ XLogEncryptionTweak(tweak, ThisTimeLineID, openLogSegNo, openLogOff);
+ encrypt_block(from, buf, XLOG_BLCKSZ, tweak);
+ XLogWritePages(buf, 1);
+
+ from += XLOG_BLCKSZ;
+ openLogOff += XLOG_BLCKSZ;
+ }
+ } else {
+ XLogWritePages(from, npages);
+ openLogOff += nbytes;
+ }
/* Update state for write */
- openLogOff += nbytes;
npages = 0;
/*
@@ -2412,6 +2414,32 @@ XLogWrite(XLogwrtRqst WriteRqst, bool flexible)
}
}
+static void
+XLogWritePages(char *from, int npages)
+{
+ Size nleft = npages * (Size) XLOG_BLCKSZ;
+ Size written;
+
+ do
+ {
+ errno = 0;
+ written = write(openLogFile, from, nleft);
+ if (written <= 0)
+ {
+ if (errno == EINTR)
+ continue;
+ ereport(PANIC,
+ (errcode_for_file_access(),
+ errmsg("could not write to log file %s "
+ "at offset %u, length %zu: %m",
+ XLogFileNameP(ThisTimeLineID, openLogSegNo),
+ openLogOff, npages * (Size) XLOG_BLCKSZ)));
+ }
+ nleft -= written;
+ from += written;
+ } while (nleft > 0);
+}
+
/*
* Record the LSN for an asynchronous transaction commit/abort
* and nudge the WALWriter if there is work for it to do.
@@ -4454,6 +4482,24 @@ ReadControlFile(void)
errhint("It looks like you need to recompile or initdb.")));
#endif
+ if (ControlFile->data_encrypted && !encryption_enabled)
+ ereport(FATAL,
+ (errmsg("database files are encrypted"),
+ errdetail("The database cluster was initialized with encryption"
+ " but the server was started without an encryption module."),
+ errhint("Set the encryption module using "
+ "encryption_library configuration parameter.")));
+ else if (encryption_enabled)
+ {
+ char sample[ENCRYPTION_SAMPLE_SIZE];
+ sample_encryption(sample);
+ if (memcmp(ControlFile->encryption_verification, sample, ENCRYPTION_SAMPLE_SIZE))
+ ereport(FATAL,
+ (errmsg("invalid encryption key"),
+ errdetail("The key specified in PGENCRYPTIONKEY does not match"
+ " database encryption key.")));
+ }
+
/* Make the initdb settings visible as GUC variables, too */
SetConfigOption("data_checksums", DataChecksumsEnabled() ? "yes" : "no",
PGC_INTERNAL, PGC_S_OVERRIDE);
@@ -4863,6 +4909,13 @@ BootStrapXLOG(void)
use_existent = false;
openLogFile = XLogFileInit(1, &use_existent, false);
+ if (encryption_enabled)
+ {
+ char tweak[TWEAK_SIZE];
+ XLogEncryptionTweak(tweak, ThisTimeLineID, 1, 0);
+ encrypt_block((char*)page, (char*)page, XLOG_BLCKSZ, tweak);
+ }
+
/* Write the first page with the initial record */
errno = 0;
if (write(openLogFile, page, XLOG_BLCKSZ) != XLOG_BLCKSZ)
@@ -4907,6 +4960,11 @@ BootStrapXLOG(void)
ControlFile->wal_log_hints = wal_log_hints;
ControlFile->track_commit_timestamp = track_commit_timestamp;
ControlFile->data_checksum_version = bootstrap_data_checksum_version;
+ ControlFile->data_encrypted = bootstrap_data_encrypted;
+ if (bootstrap_data_encrypted)
+ memcpy(ControlFile->encryption_verification, bootstrap_encryption_sample, 16);
+ else
+ memset(ControlFile->encryption_verification, 0, 16);
/* some additional ControlFile fields are set in WriteControlFile() */
@@ -11080,6 +11138,13 @@ retry:
Assert(targetPageOff == readOff);
Assert(reqLen <= readLen);
+ if (encryption_enabled)
+ {
+ char tweak[TWEAK_SIZE];
+ XLogEncryptionTweak(tweak, curFileTLI, readSegNo, readOff);
+ decrypt_block(readBuf, readBuf, XLOG_BLCKSZ, tweak);
+ }
+
*readTLI = curFileTLI;
return readLen;
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index 51a8e8d..1f47bba 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -24,6 +24,7 @@
#include "access/xlogutils.h"
#include "catalog/catalog.h"
#include "miscadmin.h"
+#include "storage/encryption.h"
#include "storage/smgr.h"
#include "utils/guc.h"
#include "utils/hsearch.h"
@@ -653,18 +654,24 @@ XLogTruncateRelation(RelFileNode rnode, ForkNumber forkNum,
static void
XLogRead(char *buf, TimeLineID tli, XLogRecPtr startptr, Size count)
{
- char *p;
+ char *p, *decrypt_p;
XLogRecPtr recptr;
Size nbytes;
+ uint32 decryptOff;
/* state maintained across calls */
static int sendFile = -1;
static XLogSegNo sendSegNo = 0;
static uint32 sendOff = 0;
- p = buf;
+ /* We only support block aligned reads to support encryption */
+ Assert(startptr % XLOG_BLCKSZ == 0);
+ Assert(count % XLOG_BLCKSZ == 0);
+
+ decrypt_p = p = buf;
recptr = startptr;
nbytes = count;
+ decryptOff = startptr % XLogSegSize;
while (nbytes > 0)
{
@@ -746,6 +753,20 @@ XLogRead(char *buf, TimeLineID tli, XLogRecPtr startptr, Size count)
sendOff += readbytes;
nbytes -= readbytes;
p += readbytes;
+
+ /* Decrypt completed blocks */
+ if (encryption_enabled)
+ {
+ while (decrypt_p + XLOG_BLCKSZ <= p)
+ {
+ char tweak[TWEAK_SIZE];
+ XLogEncryptionTweak(tweak, tli, sendSegNo, decryptOff);
+ decrypt_block(decrypt_p, decrypt_p, XLOG_BLCKSZ, tweak);
+
+ decrypt_p += XLOG_BLCKSZ;
+ decryptOff += XLOG_BLCKSZ;
+ }
+ }
}
}
@@ -823,3 +844,15 @@ read_local_xlog_page(XLogReaderState *state, XLogRecPtr targetPagePtr,
/* number of valid bytes in the buffer */
return count;
}
+
+/*
+ * Xlog is encrypted page at a time. Each xlog page gets a unique tweak via
+ * timeline, segment and offset.
+ */
+void
+XLogEncryptionTweak(char *tweak, TimeLineID timeline, XLogSegNo segment, uint32 offset)
+{
+ memcpy(tweak, &segment, sizeof(XLogSegNo));
+ memcpy(tweak + sizeof(XLogSegNo), &offset, sizeof(offset));
+ memcpy(tweak + sizeof(XLogSegNo) + sizeof(uint32), &timeline, sizeof(timeline));
+}
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index e518e17..ab2a8e2 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -21,6 +21,7 @@
#include "bootstrap/bootstrap.h"
#include "catalog/index.h"
#include "catalog/pg_collation.h"
+#include "catalog/pg_control.h"
#include "catalog/pg_type.h"
#include "libpq/pqsignal.h"
#include "miscadmin.h"
@@ -33,6 +34,7 @@
#include "replication/walreceiver.h"
#include "storage/bufmgr.h"
#include "storage/bufpage.h"
+#include "storage/encryption.h"
#include "storage/ipc.h"
#include "storage/proc.h"
#include "tcop/tcopprot.h"
@@ -45,7 +47,8 @@
#include "utils/tqual.h"
uint32 bootstrap_data_checksum_version = 0; /* No checksum */
-
+bool bootstrap_data_encrypted = false;
+char *bootstrap_encryption_sample = NULL;
#define ALLOC(t, c) ((t *) calloc((unsigned)(c), sizeof(t)))
@@ -352,6 +355,17 @@ AuxiliaryProcessMain(int argc, char *argv[])
if (!IsUnderPostmaster)
InitializeMaxBackends();
+ if (!IsUnderPostmaster)
+ setup_encryption();
+
+ if (encryption_enabled)
+ {
+ bootstrap_data_encrypted = true;
+ bootstrap_encryption_sample = palloc0(ENCRYPTION_SAMPLE_SIZE);
+ sample_encryption(bootstrap_encryption_sample);
+ }
+
+
BaseInit();
/*
diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c
index c1c0223..d7054bc 100644
--- a/src/backend/commands/dbcommands.c
+++ b/src/backend/commands/dbcommands.c
@@ -615,7 +615,11 @@ createdb(const CreatedbStmt *stmt)
*
* We don't need to copy subdirectories
*/
- copydir(srcpath, dstpath, false);
+ {
+ RelFileNode fromNode = {srctablespace, src_dboid, InvalidOid};
+ RelFileNode toNode = {dsttablespace, dboid, InvalidOid};
+ copydir(srcpath, dstpath, &fromNode, &toNode);
+ }
/* Record the filesystem change in XLOG */
{
@@ -1221,7 +1225,11 @@ movedb(const char *dbname, const char *tblspcname)
/*
* Copy files from the old tablespace to the new one
*/
- copydir(src_dbpath, dst_dbpath, false);
+ {
+ RelFileNode fromNode = {src_tblspcoid, db_id, InvalidOid};
+ RelFileNode toNode = {dst_tblspcoid, db_id, InvalidOid};
+ copydir(src_dbpath, dst_dbpath, &fromNode, &toNode);
+ }
/*
* Record the filesystem change in XLOG
@@ -2084,7 +2092,11 @@ dbase_redo(XLogReaderState *record)
*
* We don't need to copy subdirectories
*/
- copydir(src_path, dst_path, false);
+ {
+ RelFileNode fromNode = {xlrec->src_tablespace_id, xlrec->src_db_id, InvalidOid};
+ RelFileNode toNode = {xlrec->tablespace_id, xlrec->db_id, InvalidOid};
+ copydir(src_path, dst_path, &fromNode, &toNode);
+ }
}
else if (info == XLOG_DBASE_DROP)
{
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 6cf51e1..c16f11c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -114,6 +114,7 @@
#include "postmaster/postmaster.h"
#include "postmaster/syslogger.h"
#include "replication/walsender.h"
+#include "storage/encryption.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/pg_shmem.h"
@@ -920,6 +921,8 @@ PostmasterMain(int argc, char *argv[])
secure_initialize();
#endif
+ setup_encryption();
+
/*
* process any libraries that should be preloaded at postmaster start
*/
diff --git a/src/backend/storage/file/copydir.c b/src/backend/storage/file/copydir.c
index a51ee81..4911c51 100644
--- a/src/backend/storage/file/copydir.c
+++ b/src/backend/storage/file/copydir.c
@@ -22,25 +22,31 @@
#include <unistd.h>
#include <sys/stat.h>
+#include "catalog/catalog.h"
#include "storage/copydir.h"
+#include "storage/encryption.h"
#include "storage/fd.h"
+#include "storage/smgr.h"
#include "miscadmin.h"
/*
* copydir: copy a directory
*
- * If recurse is false, subdirectories are ignored. Anything that's not
- * a directory or a regular file is ignored.
+ * RelFileNode values must specify tablespace and database oids for source
+ * and target to support re-encryption if necessary. relNode value in provided
+ * structs will be clobbered.
*/
void
-copydir(char *fromdir, char *todir, bool recurse)
+copydir(char *fromdir, char *todir, RelFileNode *fromNode, RelFileNode *toNode)
{
DIR *xldir;
struct dirent *xlde;
char fromfile[MAXPGPATH];
char tofile[MAXPGPATH];
+ Assert(!encryption_enabled || (fromNode != NULL && toNode != NULL));
+
if (mkdir(todir, S_IRWXU) != 0)
ereport(ERROR,
(errcode_for_file_access(),
@@ -71,14 +77,31 @@ copydir(char *fromdir, char *todir, bool recurse)
(errcode_for_file_access(),
errmsg("could not stat file \"%s\": %m", fromfile)));
- if (S_ISDIR(fst.st_mode))
+ if (S_ISREG(fst.st_mode))
{
- /* recurse to handle subdirectories */
- if (recurse)
- copydir(fromfile, tofile, true);
+ int oidchars;
+ ForkNumber forkNum;
+ int segment;
+
+ /*
+ * For encrypted databases we need to reencrypt files with new
+ * tweaks.
+ */
+ if (encryption_enabled &&
+ parse_filename_for_nontemp_relation(xlde->d_name,
+ &oidchars, &forkNum, &segment))
+ {
+ char oidbuf[OIDCHARS+1];
+ memcpy(oidbuf, xlde->d_name, oidchars);
+ oidbuf[oidchars] = '\0';
+
+ /* We scribble over the provided RelFileNodes here */
+ fromNode->relNode = toNode->relNode = atol(oidbuf);
+ copy_file(fromfile, tofile, fromNode, toNode, forkNum, forkNum, segment);
+ }
+ else
+ copy_file(fromfile, tofile, NULL, NULL, 0, 0, 0);
}
- else if (S_ISREG(fst.st_mode))
- copy_file(fromfile, tofile);
}
FreeDir(xldir);
@@ -129,15 +152,20 @@ copydir(char *fromdir, char *todir, bool recurse)
}
/*
- * copy one file
+ * copy one file. If decryption and reencryption is needed specify
+ * relfilenodes for source and target.
*/
void
-copy_file(char *fromfile, char *tofile)
+copy_file(char *fromfile, char *tofile, RelFileNode *fromNode,
+ RelFileNode *toNode, ForkNumber fromForkNum, ForkNumber toForkNum,
+ int segment)
{
char *buffer;
int srcfd;
int dstfd;
int nbytes;
+ int bytesread;
+ BlockNumber blockNum = segment*RELSEG_SIZE;
off_t offset;
/* Use palloc to ensure we get a maxaligned buffer */
@@ -169,14 +197,37 @@ copy_file(char *fromfile, char *tofile)
/* If we got a cancel signal during the copy of the file, quit */
CHECK_FOR_INTERRUPTS();
- nbytes = read(srcfd, buffer, COPY_BUF_SIZE);
- if (nbytes < 0)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not read file \"%s\": %m", fromfile)));
+ /*
+ * Try to read as much as we fit in the buffer so we can deal with
+ * complete blocks if we need to reencrypt.
+ */
+ nbytes = 0;
+ while (nbytes < COPY_BUF_SIZE)
+ {
+ bytesread = read(srcfd, buffer, COPY_BUF_SIZE);
+ if (bytesread < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m", fromfile)));
+ nbytes += bytesread;
+ if (bytesread == 0)
+ break;
+ }
if (nbytes == 0)
break;
errno = 0;
+
+ /*
+ * If the database is encrypted we need to decrypt the data here
+ * and reencrypt it to adjust the tweak values of blocks.
+ */
+ if (fromNode != NULL)
+ {
+ Assert(toNode != NULL);
+ blockNum = ReencryptBlock(buffer, nbytes/BLCKSZ,
+ fromNode, toNode, fromForkNum, toForkNum, blockNum);
+ }
+
if ((int) write(dstfd, buffer, nbytes) != nbytes)
{
/* if write didn't set errno, assume problem is no disk space */
@@ -204,3 +255,71 @@ copy_file(char *fromfile, char *tofile)
pfree(buffer);
}
+
+
+/*
+ * Basic parsing of putative relation filenames.
+ *
+ * This function returns true if the file appears to be in the correct format
+ * for a non-temporary relation and false otherwise.
+ *
+ * NB: If this function returns true, the caller is entitled to assume that
+ * *oidchars has been set to the a value no more than OIDCHARS, and thus
+ * that a buffer of OIDCHARS+1 characters is sufficient to hold the OID
+ * portion of the filename. This is critical to protect against a possible
+ * buffer overrun.
+ */
+bool
+parse_filename_for_nontemp_relation(const char *name, int *oidchars,
+ ForkNumber *fork, int *segment)
+{
+ int pos;
+ int segstart = 0;
+
+ /* Look for a non-empty string of digits (that isn't too long). */
+ for (pos = 0; isdigit((unsigned char) name[pos]); ++pos)
+ ;
+ if (pos == 0 || pos > OIDCHARS)
+ return false;
+ *oidchars = pos;
+
+ /* Check for a fork name. */
+ if (name[pos] != '_')
+ *fork = MAIN_FORKNUM;
+ else
+ {
+ int forkchar;
+
+ forkchar = forkname_chars(&name[pos + 1], fork);
+ if (forkchar <= 0)
+ return false;
+ pos += forkchar + 1;
+ }
+
+ /* Check for a segment number. */
+ if (name[pos] == '.')
+ {
+ int segchar;
+
+ segstart = pos + 1;
+ for (segchar = 1; isdigit((unsigned char) name[pos + segchar]); ++segchar)
+ ;
+ if (segchar <= 1)
+ return false;
+ pos += segchar;
+ }
+
+ /* Now we should be at the end. */
+ if (name[pos] != '\0')
+ return false;
+
+ if (segment != NULL)
+ {
+ if (segstart == 0)
+ *segment = 0;
+ else
+ *segment = atoi(name + segstart);
+ }
+
+ return true;
+}
diff --git a/src/backend/storage/file/reinit.c b/src/backend/storage/file/reinit.c
index 7e8138b..326036c 100644
--- a/src/backend/storage/file/reinit.c
+++ b/src/backend/storage/file/reinit.c
@@ -17,6 +17,7 @@
#include <unistd.h>
#include "catalog/catalog.h"
+#include "catalog/pg_tablespace.h"
#include "common/relpath.h"
#include "storage/copydir.h"
#include "storage/fd.h"
@@ -25,11 +26,9 @@
#include "utils/memutils.h"
static void ResetUnloggedRelationsInTablespaceDir(const char *tsdirname,
- int op);
+ int op, Oid spcOid);
static void ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname,
- int op);
-static bool parse_filename_for_nontemp_relation(const char *name,
- int *oidchars, ForkNumber *fork);
+ int op, Oid spcOid, Oid dbOid);
typedef struct
{
@@ -73,7 +72,7 @@ ResetUnloggedRelations(int op)
/*
* First process unlogged files in pg_default ($PGDATA/base)
*/
- ResetUnloggedRelationsInTablespaceDir("base", op);
+ ResetUnloggedRelationsInTablespaceDir("base", op, DEFAULTTABLESPACE_OID);
/*
* Cycle through directories for all non-default tablespaces.
@@ -82,13 +81,15 @@ ResetUnloggedRelations(int op)
while ((spc_de = ReadDir(spc_dir, "pg_tblspc")) != NULL)
{
+ Oid spcOid;
if (strcmp(spc_de->d_name, ".") == 0 ||
strcmp(spc_de->d_name, "..") == 0)
continue;
snprintf(temp_path, sizeof(temp_path), "pg_tblspc/%s/%s",
spc_de->d_name, TABLESPACE_VERSION_DIRECTORY);
- ResetUnloggedRelationsInTablespaceDir(temp_path, op);
+ spcOid = atoi(spc_de->d_name);
+ ResetUnloggedRelationsInTablespaceDir(temp_path, op, spcOid);
}
FreeDir(spc_dir);
@@ -102,7 +103,7 @@ ResetUnloggedRelations(int op)
/* Process one tablespace directory for ResetUnloggedRelations */
static void
-ResetUnloggedRelationsInTablespaceDir(const char *tsdirname, int op)
+ResetUnloggedRelationsInTablespaceDir(const char *tsdirname, int op, Oid spcOid)
{
DIR *ts_dir;
struct dirent *de;
@@ -122,6 +123,7 @@ ResetUnloggedRelationsInTablespaceDir(const char *tsdirname, int op)
while ((de = ReadDir(ts_dir, tsdirname)) != NULL)
{
int i = 0;
+ Oid dbOid;
/*
* We're only interested in the per-database directories, which have
@@ -133,9 +135,10 @@ ResetUnloggedRelationsInTablespaceDir(const char *tsdirname, int op)
if (de->d_name[i] != '\0' || i == 0)
continue;
+ dbOid = atoi(de->d_name);
snprintf(dbspace_path, sizeof(dbspace_path), "%s/%s",
tsdirname, de->d_name);
- ResetUnloggedRelationsInDbspaceDir(dbspace_path, op);
+ ResetUnloggedRelationsInDbspaceDir(dbspace_path, op, spcOid, dbOid);
}
FreeDir(ts_dir);
@@ -143,7 +146,8 @@ ResetUnloggedRelationsInTablespaceDir(const char *tsdirname, int op)
/* Process one per-dbspace directory for ResetUnloggedRelations */
static void
-ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op)
+ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op,
+ Oid spcOid, Oid dbOid)
{
DIR *dbspace_dir;
struct dirent *de;
@@ -192,7 +196,7 @@ ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op)
/* Skip anything that doesn't look like a relation data file. */
if (!parse_filename_for_nontemp_relation(de->d_name, &oidchars,
- &forkNum))
+ &forkNum, NULL))
continue;
/* Also skip it unless this is the init fork. */
@@ -245,7 +249,7 @@ ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op)
/* Skip anything that doesn't look like a relation data file. */
if (!parse_filename_for_nontemp_relation(de->d_name, &oidchars,
- &forkNum))
+ &forkNum, NULL))
continue;
/* We never remove the init fork. */
@@ -309,13 +313,14 @@ ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op)
{
ForkNumber forkNum;
int oidchars;
+ int segment;
char oidbuf[OIDCHARS + 1];
char srcpath[MAXPGPATH];
char dstpath[MAXPGPATH];
/* Skip anything that doesn't look like a relation data file. */
if (!parse_filename_for_nontemp_relation(de->d_name, &oidchars,
- &forkNum))
+ &forkNum, &segment))
continue;
/* Also skip it unless this is the init fork. */
@@ -335,7 +340,12 @@ ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op)
/* OK, we're ready to perform the actual copy. */
elog(DEBUG2, "copying %s to %s", srcpath, dstpath);
- copy_file(srcpath, dstpath);
+ {
+ RelFileNode srcNode = {spcOid, dbOid, atol(oidbuf)};
+ RelFileNode dstNode = srcNode;
+ copy_file(srcpath, dstpath, &srcNode, &dstNode,
+ INIT_FORKNUM, MAIN_FORKNUM, segment);
+ }
}
FreeDir(dbspace_dir);
@@ -366,7 +376,7 @@ ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op)
/* Skip anything that doesn't look like a relation data file. */
if (!parse_filename_for_nontemp_relation(de->d_name, &oidchars,
- &forkNum))
+ &forkNum, NULL))
continue;
/* Also skip it unless this is the init fork. */
@@ -388,59 +398,3 @@ ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op)
fsync_fname(dbspacedirname, true);
}
}
-
-/*
- * Basic parsing of putative relation filenames.
- *
- * This function returns true if the file appears to be in the correct format
- * for a non-temporary relation and false otherwise.
- *
- * NB: If this function returns true, the caller is entitled to assume that
- * *oidchars has been set to the a value no more than OIDCHARS, and thus
- * that a buffer of OIDCHARS+1 characters is sufficient to hold the OID
- * portion of the filename. This is critical to protect against a possible
- * buffer overrun.
- */
-static bool
-parse_filename_for_nontemp_relation(const char *name, int *oidchars,
- ForkNumber *fork)
-{
- int pos;
-
- /* Look for a non-empty string of digits (that isn't too long). */
- for (pos = 0; isdigit((unsigned char) name[pos]); ++pos)
- ;
- if (pos == 0 || pos > OIDCHARS)
- return false;
- *oidchars = pos;
-
- /* Check for a fork name. */
- if (name[pos] != '_')
- *fork = MAIN_FORKNUM;
- else
- {
- int forkchar;
-
- forkchar = forkname_chars(&name[pos + 1], fork);
- if (forkchar <= 0)
- return false;
- pos += forkchar + 1;
- }
-
- /* Check for a segment number. */
- if (name[pos] == '.')
- {
- int segchar;
-
- for (segchar = 1; isdigit((unsigned char) name[pos + segchar]); ++segchar)
- ;
- if (segchar <= 1)
- return false;
- pos += segchar;
- }
-
- /* Now we should be at the end. */
- if (name[pos] != '\0')
- return false;
- return true;
-}
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index f2a07f2..423034e 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -119,18 +119,7 @@ PageIsVerified(Page page, BlockNumber blkno)
}
/* Check all-zeroes case */
- all_zeroes = true;
- pagebytes = (char *) page;
- for (i = 0; i < BLCKSZ; i++)
- {
- if (pagebytes[i] != 0)
- {
- all_zeroes = false;
- break;
- }
- }
-
- if (all_zeroes)
+ if (IsAllZero((char *) page, BLCKSZ))
return true;
/*
@@ -1134,3 +1123,38 @@ PageSetChecksumInplace(Page page, BlockNumber blkno)
((PageHeader) page)->pd_checksum = pg_checksum_page((char *) page, blkno);
}
+
+/*
+ * Helper function to check if a page is completely empty.
+ */
+bool
+IsAllZero(const char *input, Size size)
+{
+ const char *pos = input;
+ const char *aligned_start = (char*) MAXALIGN64(input);
+ const char *end = input + size;
+
+ /* Check 1 byte at a time until pos is 8 byte aligned */
+ while (pos < aligned_start)
+ if (*pos++ != 0)
+ return false;
+
+ /*
+ * Run 8 parallel 8 byte checks in one iteration. On 2016 hardware
+ * slightly faster than 4 parallel checks.
+ **/
+ while (pos + 8*sizeof(uint64) <= end)
+ {
+ uint64 *p = (uint64*) pos;
+ if ((p[0] | p[1] | p[2] | p[3] | p[4] | p[5] | p[6] | p[7]) != 0)
+ return false;
+ pos += 8*sizeof(uint64);
+ }
+
+ /* Handle unaligned tail. */
+ while (pos < end)
+ if (*pos++ != 0)
+ return false;
+
+ return true;
+}
diff --git a/src/backend/storage/smgr/Makefile b/src/backend/storage/smgr/Makefile
index 2b95cb0..0601ea7 100644
--- a/src/backend/storage/smgr/Makefile
+++ b/src/backend/storage/smgr/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/storage/smgr
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
-OBJS = md.o smgr.o smgrtype.o
+OBJS = encryption.o md.o smgr.o smgrtype.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/smgr/encryption.c b/src/backend/storage/smgr/encryption.c
new file mode 100644
index 0000000..766487a
--- /dev/null
+++ b/src/backend/storage/smgr/encryption.c
@@ -0,0 +1,145 @@
+/*-------------------------------------------------------------------------
+ *
+ * encryption.c
+ * This code handles encryption and decryption of data.
+ *
+ * Encryption is done by extension modules loaded by encryption_library GUC.
+ * The extension module must register itself and provide a cryptography
+ * implementation. Key setup is left to the extension module.
+ *
+ *
+ * Copyright (c) 2016, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/storage/smgr/encryption.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "catalog/pg_control.h"
+#include "storage/bufpage.h"
+#include "storage/encryption.h"
+#include "miscadmin.h"
+#include "fmgr.h"
+#include "port.h"
+
+bool encryption_enabled = false;
+bool have_encryption_provider = false;
+EncryptionRoutines encryption_hooks;
+
+/*
+ * Hook function for encryption providers. The first library to call this
+ * function gets to provide encryption capability.
+ */
+void
+register_encryption_module(char *name, EncryptionRoutines *enc)
+{
+ if (!have_encryption_provider)
+ {
+ elog(DEBUG1, "Registering encryption module %s", name);
+ encryption_hooks = *enc;
+ have_encryption_provider = true;
+ }
+}
+
+/*
+ * Encrypts a fixed value into *buf to verify that encryption key is correct.
+ * Caller provided buf needs to be able to hold at least ENCRYPTION_SAMPLE_SIZE
+ * bytes.
+ */
+void
+sample_encryption(char *buf)
+{
+ char tweak[TWEAK_SIZE];
+ int i;
+ for (i = 0; i < TWEAK_SIZE; i++)
+ tweak[i] = i;
+
+ encrypt_block("postgresqlcrypt", buf, ENCRYPTION_SAMPLE_SIZE, tweak);
+}
+
+/*
+ * Encrypts one block of data with a specified tweak value. Input and output
+ * buffer may point to the same location. Size of input must be at least
+ * ENCRYPTION_BLOCK bytes. Tweak value must be TWEAK_SIZE bytes.
+ *
+ * All zero blocks are not encrypted or decrypted to correctly handle relation
+ * extension.
+ *
+ * Must only be called when encryption_enabled is true.
+ */
+void
+encrypt_block(const char *input, char *output, Size size, const char *tweak)
+{
+ Assert(size >= ENCRYPTION_BLOCK);
+ Assert(encryption_enabled);
+
+ if (IsAllZero(input, size))
+ {
+ if (input != output)
+ memset(output, 0, size);
+ }
+ else
+ encryption_hooks.EncryptBlock(input, output, size, tweak);
+}
+
+/*
+ * Decrypts one block of data with a specified tweak value. Input and output
+ * buffer may point to the same location. Tweak value must match the one used
+ * when encrypting.
+ *
+ * Must only be called when encryption_enabled is true.
+ */
+void
+decrypt_block(const char *input, char *output, Size size, const char *tweak)
+{
+ Assert(size >= ENCRYPTION_BLOCK);
+ Assert(encryption_enabled);
+
+ if (IsAllZero(input, size))
+ {
+ if (input != output)
+ memset(output, 0, size);
+ }
+ else
+ encryption_hooks.DecryptBlock(input, output, size, tweak);
+}
+
+/*
+ * Initialize encryption subsystem for use. Must be called before any
+ * encryptable data is read from or written to data directory.
+ */
+void
+setup_encryption()
+{
+ char *filename;
+
+ if (encryption_library_string == NULL || encryption_library_string[0] == '\0')
+ return;
+
+ /* Try to load encryption library */
+ filename = pstrdup(encryption_library_string);
+
+ canonicalize_path(filename);
+ load_file(filename, false);
+ ereport(DEBUG1,
+ (errmsg("loaded library \"%s\" for encryption", filename)));
+ pfree(filename);
+
+ if (have_encryption_provider)
+ {
+ encryption_enabled = encryption_hooks.SetupEncryption();
+ if (encryption_enabled)
+ {
+ if (!IsBootstrapProcessingMode())
+ elog(LOG, "data encryption performed by %s", encryption_library_string);
+ }
+ else
+ elog(FATAL, "data encryption could not be initialized");
+ }
+ else
+ elog(ERROR, "Specified encryption library %s did not provide encryption hooks.", encryption_library_string);
+}
+
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index f329d15..82b8328 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -28,10 +28,12 @@
#include "miscadmin.h"
#include "access/xlog.h"
#include "catalog/catalog.h"
+#include "libpq/md5.h"
#include "portability/instr_time.h"
#include "postmaster/bgwriter.h"
#include "storage/fd.h"
#include "storage/bufmgr.h"
+#include "storage/encryption.h"
#include "storage/relfilenode.h"
#include "storage/smgr.h"
#include "utils/hsearch.h"
@@ -116,7 +118,8 @@ typedef struct _MdfdVec
} MdfdVec;
static MemoryContext MdCxt; /* context for all MdfdVec objects */
-
+static char *md_encryption_buffer;
+static char *md_encryption_tweak;
/*
* In some contexts (currently, standalone backends and the checkpointer)
@@ -198,7 +201,9 @@ static MdfdVec *_mdfd_getseg(SMgrRelation reln, ForkNumber forkno,
BlockNumber blkno, bool skipFsync, int behavior);
static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum,
MdfdVec *seg);
-
+static void mdtweak(char *tweak, RelFileNode *relnode, ForkNumber forknum, BlockNumber blocknum);
+static void mdencrypt(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, char *buffer);
+static void mddecrypt(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, char *buffer);
/*
* mdinit() -- Initialize private state for magnetic disk storage manager.
@@ -247,6 +252,9 @@ mdinit(void)
HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
pendingUnlinks = NIL;
}
+
+ md_encryption_buffer = MemoryContextAllocZero(MdCxt, BLCKSZ);
+ md_encryption_tweak = MemoryContextAllocZero(MdCxt, TWEAK_SIZE);
}
/*
@@ -542,7 +550,10 @@ mdextend(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
errmsg("could not seek to block %u in file \"%s\": %m",
blocknum, FilePathName(v->mdfd_vfd))));
- if ((nbytes = FileWrite(v->mdfd_vfd, buffer, BLCKSZ)) != BLCKSZ)
+ if (encryption_enabled)
+ mdencrypt(reln, forknum, blocknum, buffer);
+
+ if ((nbytes = FileWrite(v->mdfd_vfd, encryption_enabled ? md_encryption_buffer : buffer, BLCKSZ)) != BLCKSZ)
{
if (nbytes < 0)
ereport(ERROR,
@@ -755,7 +766,7 @@ mdread(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
errmsg("could not seek to block %u in file \"%s\": %m",
blocknum, FilePathName(v->mdfd_vfd))));
- nbytes = FileRead(v->mdfd_vfd, buffer, BLCKSZ);
+ nbytes = FileRead(v->mdfd_vfd, encryption_enabled ? md_encryption_buffer : buffer, BLCKSZ);
TRACE_POSTGRESQL_SMGR_MD_READ_DONE(forknum, blocknum,
reln->smgr_rnode.node.spcNode,
@@ -790,6 +801,8 @@ mdread(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
blocknum, FilePathName(v->mdfd_vfd),
nbytes, BLCKSZ)));
}
+ else if (encryption_enabled)
+ mddecrypt(reln, forknum, blocknum, buffer);
}
/*
@@ -831,7 +844,9 @@ mdwrite(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
errmsg("could not seek to block %u in file \"%s\": %m",
blocknum, FilePathName(v->mdfd_vfd))));
- nbytes = FileWrite(v->mdfd_vfd, buffer, BLCKSZ);
+ if (encryption_enabled)
+ mdencrypt(reln, forknum, blocknum, buffer);
+ nbytes = FileWrite(v->mdfd_vfd, encryption_enabled ? md_encryption_buffer : buffer, BLCKSZ);
TRACE_POSTGRESQL_SMGR_MD_WRITE_DONE(forknum, blocknum,
reln->smgr_rnode.node.spcNode,
@@ -1904,3 +1919,54 @@ _mdnblocks(SMgrRelation reln, ForkNumber forknum, MdfdVec *seg)
/* note that this calculation will ignore any partial block at EOF */
return (BlockNumber) (len / BLCKSZ);
}
+
+/*
+ * md files are encrypted block at a time. Tweak will alias higher numbered
+ * forks for huge tables.
+ */
+static void
+mdtweak(char *tweak, RelFileNode *relnode, ForkNumber forknum, BlockNumber blocknum)
+{
+ uint32 fork_and_block = (forknum << 24) ^ blocknum;
+ memcpy(tweak, relnode, 12);
+ memcpy(tweak+12, &fork_and_block, 4);
+}
+
+static void
+mdencrypt(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, char *buffer)
+{
+ mdtweak(md_encryption_tweak, &(reln->smgr_rnode.node), forknum, blocknum);
+ encrypt_block(buffer, md_encryption_buffer, BLCKSZ, md_encryption_tweak);
+}
+
+static void
+mddecrypt(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, char *dest)
+{
+ mdtweak(md_encryption_tweak, &(reln->smgr_rnode.node), forknum, blocknum);
+ decrypt_block(md_encryption_buffer, dest, BLCKSZ, md_encryption_tweak);
+}
+
+/*
+ * Copying relations between tablespaces/databases means that the tweak values
+ * of each block will change. This function transcodes a series of blocks with
+ * new tweak values. Returns the new block number for convenience.
+ */
+BlockNumber
+ReencryptBlock(char *buffer, int blocks,
+ RelFileNode *srcNode, RelFileNode *dstNode,
+ ForkNumber srcForkNum, ForkNumber dstForkNum,
+ BlockNumber blockNum)
+{
+ char *cur;
+ char srcTweak[16];
+ char dstTweak[16];
+ for (cur = buffer; cur < buffer + blocks * BLCKSZ; cur += BLCKSZ)
+ {
+ mdtweak(srcTweak, srcNode, srcForkNum, blockNum);
+ mdtweak(dstTweak, dstNode, dstForkNum, blockNum);
+ decrypt_block(cur, cur, BLCKSZ, srcTweak);
+ encrypt_block(cur, cur, BLCKSZ, dstTweak);
+ blockNum++;
+ }
+ return blockNum;
+}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index b185c1b..4215d5f 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -59,6 +59,7 @@
#include "replication/walsender.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/encryption.h"
#include "storage/ipc.h"
#include "storage/proc.h"
#include "storage/procsignal.h"
@@ -3692,6 +3693,8 @@ PostgresMain(int argc, char *argv[],
/* Change into DataDir (if under postmaster, was done already) */
ChangeToDataDir();
+ setup_encryption();
+
/*
* Create lockfile for data directory.
*/
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index d13355b..87d8dcf 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -1392,6 +1392,7 @@ ValidatePgVersion(const char *path)
* GUC variables: lists of library names to be preloaded at postmaster
* start and at backend start
*/
+char *encryption_library_string = NULL;
char *session_preload_libraries_string = NULL;
char *shared_preload_libraries_string = NULL;
char *local_preload_libraries_string = NULL;
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index e246a9c..61c42bb 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -3151,6 +3151,17 @@ static struct config_string ConfigureNamesString[] =
},
{
+ {"encryption_library", PGC_POSTMASTER, CLIENT_CONN_PRELOAD,
+ gettext_noop("Encryption library to provide transparent data encryption."),
+ NULL,
+ GUC_SUPERUSER_ONLY
+ },
+ &encryption_library_string,
+ "",
+ NULL, NULL, NULL
+ },
+
+ {
{"session_preload_libraries", PGC_SUSET, CLIENT_CONN_PRELOAD,
gettext_noop("Lists shared libraries to preload into each backend."),
NULL,
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 2d2db7e..924e78a 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -105,6 +105,7 @@
#tcp_keepalives_count = 0 # TCP_KEEPCNT;
# 0 selects the system default
+#encryption_library = ''
#------------------------------------------------------------------------------
# RESOURCE USAGE (except WAL)
@@ -173,7 +174,6 @@
#backend_flush_after = 0 # 0 disables,
# default is 128kb on linux, 0 otherwise
-
#------------------------------------------------------------------------------
# WRITE AHEAD LOG
#------------------------------------------------------------------------------
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index d4a5e7c..318b114 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -142,6 +142,8 @@ static bool do_sync = true;
static bool sync_only = false;
static bool show_setting = false;
static bool data_checksums = false;
+static char *data_encryption_module = NULL;
+static char *data_encryption_key = NULL;
static char *xlog_dir = "";
@@ -279,6 +281,7 @@ static bool check_locale_encoding(const char *locale, int encoding);
static void setlocales(void);
static void usage(const char *progname);
void setup_pgdata(void);
+void setup_encryption(void);
void setup_bin_paths(const char *argv0);
void setup_data_file_paths(void);
void setup_locale_encoding(void);
@@ -1218,6 +1221,13 @@ setup_config(void)
n_buffers * (BLCKSZ / 1024));
conflines = replace_token(conflines, "#shared_buffers = 32MB", repltok);
+ if (data_encryption_module != NULL)
+ {
+ snprintf(repltok, sizeof(repltok), "encryption_library = '%s'",
+ data_encryption_module);
+ conflines = replace_token(conflines, "#encryption_library = ''", repltok);
+ }
+
#ifdef HAVE_UNIX_SOCKETS
snprintf(repltok, sizeof(repltok), "#unix_socket_directories = '%s'",
DEFAULT_PGSOCKET_DIR);
@@ -2783,6 +2793,20 @@ setup_pgdata(void)
putenv(pgdata_set_env);
}
+void
+setup_encryption(void)
+{
+ char *key = getenv("PGENCRYPTIONKEY");
+ if (key != NULL && strlen(key) != 0)
+ data_encryption_key = pg_strdup(key);
+
+ if (data_encryption_key != NULL)
+ {
+ char *pgencryptionkey_set_env;
+ pgencryptionkey_set_env = psprintf("PGENCRYPTIONKEY=%s", data_encryption_key);
+ putenv(pgencryptionkey_set_env);
+ }
+}
void
setup_bin_paths(const char *argv0)
@@ -3333,7 +3357,6 @@ initialize_data_directory(void)
check_ok();
}
-
int
main(int argc, char *argv[])
{
@@ -3364,6 +3387,7 @@ main(int argc, char *argv[])
{"sync-only", no_argument, NULL, 'S'},
{"xlogdir", required_argument, NULL, 'X'},
{"data-checksums", no_argument, NULL, 'k'},
+ {"data-encryption", required_argument, NULL, 'K'},
{NULL, 0, NULL, 0}
};
@@ -3404,7 +3428,7 @@ main(int argc, char *argv[])
/* process command-line options */
- while ((c = getopt_long(argc, argv, "dD:E:kL:nNU:WA:sST:X:", long_options, &option_index)) != -1)
+ while ((c = getopt_long(argc, argv, "dD:E:kK:L:nNU:WA:sST:X:", long_options, &option_index)) != -1)
{
switch (c)
{
@@ -3456,6 +3480,10 @@ main(int argc, char *argv[])
case 'k':
data_checksums = true;
break;
+ case 'K':
+ if (strlen(optarg) > 0)
+ data_encryption_module = pg_strdup(optarg);
+ break;
case 'L':
share_path = pg_strdup(optarg);
break;
@@ -3560,6 +3588,7 @@ main(int argc, char *argv[])
setup_bin_paths(argv[0]);
+
effective_user = get_id();
if (strlen(username) == 0)
username = effective_user;
@@ -3590,6 +3619,13 @@ main(int argc, char *argv[])
else
printf(_("Data page checksums are disabled.\n"));
+ setup_encryption();
+
+ if (data_encryption_module != NULL)
+ printf(_("Using %s for data encryption.\n"), data_encryption_module);
+ else
+ printf(_("Data encryption is disabled.\n"));
+
printf("\n");
initialize_data_directory();
diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c
index 96619a2..3188688 100644
--- a/src/bin/pg_controldata/pg_controldata.c
+++ b/src/bin/pg_controldata/pg_controldata.c
@@ -296,5 +296,14 @@ main(int argc, char *argv[])
(ControlFile->float8ByVal ? _("by value") : _("by reference")));
printf(_("Data page checksum version: %u\n"),
ControlFile->data_checksum_version);
+ printf(_("Data encryption: %s\n"),
+ ControlFile->data_encrypted ? _("on") : _("off"));
+ if (ControlFile->data_encrypted)
+ printf(_("Data encryption fingerprint: %08X%08X%08X%08X\n"),
+ htonl(((uint32*)ControlFile->encryption_verification)[0]),
+ htonl(((uint32*)ControlFile->encryption_verification)[1]),
+ htonl(((uint32*)ControlFile->encryption_verification)[2]),
+ htonl(((uint32*)ControlFile->encryption_verification)[3])
+ );
return 0;
}
diff --git a/src/bin/pg_upgrade/controldata.c b/src/bin/pg_upgrade/controldata.c
index 2601827..6486fc2 100644
--- a/src/bin/pg_upgrade/controldata.c
+++ b/src/bin/pg_upgrade/controldata.c
@@ -58,6 +58,7 @@ get_control_data(ClusterInfo *cluster, bool live_check)
bool got_large_object = false;
bool got_date_is_int = false;
bool got_data_checksum_version = false;
+ bool got_data_encrypted = false;
char *lc_collate = NULL;
char *lc_ctype = NULL;
char *lc_monetary = NULL;
@@ -129,6 +130,13 @@ get_control_data(ClusterInfo *cluster, bool live_check)
got_data_checksum_version = true;
}
+ /* Only in <= 9.6 */
+ if (GET_MAJOR_VERSION(cluster->major_version) <= 906)
+ {
+ cluster->controldata.data_encrypted = false;
+ got_data_encrypted = true;
+ }
+
/* we have the result of cmd in "output". so parse it line by line now */
while (fgets(bufin, sizeof(bufin), output))
{
@@ -412,6 +420,33 @@ get_control_data(ClusterInfo *cluster, bool live_check)
cluster->controldata.data_checksum_version = str2uint(p);
got_data_checksum_version = true;
}
+ else if ((p = strstr(bufin, "encryption fingerprint")) != NULL)
+ {
+ int i;
+ p = strchr(p, ':');
+
+ if (p == NULL || strlen(p) <= 1)
+ pg_fatal("%d: controldata retrieval problem\n", __LINE__);
+
+ cluster->controldata.data_encrypted = true;
+
+ /* Skip the colon and any whitespace after it */
+ p = strchr(p, ':');
+ if (p == NULL || strlen(p) <= 1)
+ pg_fatal("%d: controldata retrieval problem\n", __LINE__);
+ p = strpbrk(p, "01234567890ABCDEF");
+ if (p == NULL || strlen(p) <= 1)
+ pg_fatal("%d: controldata retrieval problem\n", __LINE__);
+
+ /* Make sure it looks like a valid finerprint */
+ if (strspn(p, "0123456789ABCDEF") != 32)
+ pg_fatal("%d: controldata retrieval problem\n", __LINE__);
+
+ for(i = 0; i < 16; i++)
+ sscanf(p + 2*i, "%2hhx",
+ cluster->controldata.encryption_verification + i);
+ got_data_encrypted = true;
+ }
}
pclose(output);
@@ -466,7 +501,7 @@ get_control_data(ClusterInfo *cluster, bool live_check)
!got_index || !got_toast ||
(!got_large_object &&
cluster->controldata.ctrl_ver >= LARGE_OBJECT_SIZE_PG_CONTROL_VER) ||
- !got_date_is_int || !got_data_checksum_version)
+ !got_date_is_int || !got_data_checksum_version || !got_data_encrypted)
{
pg_log(PG_REPORT,
"The %s cluster lacks some required control information:\n",
@@ -529,6 +564,10 @@ get_control_data(ClusterInfo *cluster, bool live_check)
if (!got_data_checksum_version)
pg_log(PG_REPORT, " data checksum version\n");
+ /* value added in Postgres 10 */
+ if (!got_data_encrypted)
+ pg_log(PG_REPORT, " data encryption status\n");
+
pg_fatal("Cannot continue without required control information, terminating\n");
}
}
@@ -593,6 +632,18 @@ check_control_data(ControlData *oldctrl,
pg_fatal("old cluster uses data checksums but the new one does not\n");
else if (oldctrl->data_checksum_version != newctrl->data_checksum_version)
pg_fatal("old and new cluster pg_controldata checksum versions do not match\n");
+
+ if (oldctrl->data_encrypted && !newctrl->data_encrypted)
+ pg_fatal("old cluster is encrypted, but the new one is not\n");
+ else if (!oldctrl->data_encrypted && newctrl->data_encrypted)
+ pg_fatal("old cluster is not encrypted, but the new one is\n");
+ else if (oldctrl->data_encrypted && newctrl->data_encrypted)
+ {
+ if (oldctrl->encryption_verification != newctrl->encryption_verification)
+ pg_fatal("encryption keys do not match between old and new cluster\n");
+ else
+ pg_fatal("upgrading encrypted databases is not implemented yet\n"); /* TODO */
+ }
}
diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h
index 10182de..965881f 100644
--- a/src/bin/pg_upgrade/pg_upgrade.h
+++ b/src/bin/pg_upgrade/pg_upgrade.h
@@ -227,6 +227,8 @@ typedef struct
bool date_is_int;
bool float8_pass_by_value;
bool data_checksum_version;
+ bool data_encrypted;
+ uint8 encryption_verification[16];
} ControlData;
/*
diff --git a/src/include/access/xlogutils.h b/src/include/access/xlogutils.h
index d027ea1..5e4d3e2 100644
--- a/src/include/access/xlogutils.h
+++ b/src/include/access/xlogutils.h
@@ -51,5 +51,7 @@ extern int read_local_xlog_page(XLogReaderState *state,
XLogRecPtr targetPagePtr, int reqLen,
XLogRecPtr targetRecPtr, char *cur_page,
TimeLineID *pageTLI);
+extern void XLogEncryptionTweak(char *tweak, TimeLineID timeline,
+ XLogSegNo segment, uint32 offset);
#endif
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index 7ba396d..1e21318 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -91,6 +91,11 @@ typedef enum DBState
} DBState;
/*
+ * Number of bytes reserved to store encryption sample in ControlFileData.
+ */
+#define ENCRYPTION_SAMPLE_SIZE 16
+
+/*
* Contents of pg_control.
*
* NOTE: try to keep this under 512 bytes so that it will fit on one physical
@@ -225,6 +230,11 @@ typedef struct ControlFileData
/* Are data pages protected by checksums? Zero if no checksum version */
uint32 data_checksum_version;
+ /* Is data directory encrypted? */
+ bool data_encrypted;
+ /* Sample value for encryption key verification */
+ uint8 encryption_verification[ENCRYPTION_SAMPLE_SIZE];
+
/* CRC of all above ... MUST BE LAST! */
pg_crc32c crc;
} ControlFileData;
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 78545da..d8057f7 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -421,6 +421,7 @@ extern void BaseInit(void);
/* in utils/init/miscinit.c */
extern bool IgnoreSystemIndexes;
extern PGDLLIMPORT bool process_shared_preload_libraries_in_progress;
+extern char *encryption_library_string;
extern char *session_preload_libraries_string;
extern char *shared_preload_libraries_string;
extern char *local_preload_libraries_string;
diff --git a/src/include/storage/bufpage.h b/src/include/storage/bufpage.h
index 15cebfc..bdceb81 100644
--- a/src/include/storage/bufpage.h
+++ b/src/include/storage/bufpage.h
@@ -432,4 +432,7 @@ extern void PageIndexDeleteNoCompact(Page page, OffsetNumber *itemnos,
extern char *PageSetChecksumCopy(Page page, BlockNumber blkno);
extern void PageSetChecksumInplace(Page page, BlockNumber blkno);
+/* TODO: Is this the best location for this definition? */
+extern bool IsAllZero(const char *input, Size size);
+
#endif /* BUFPAGE_H */
diff --git a/src/include/storage/copydir.h b/src/include/storage/copydir.h
index 4b7b813..d28df73 100644
--- a/src/include/storage/copydir.h
+++ b/src/include/storage/copydir.h
@@ -13,7 +13,11 @@
#ifndef COPYDIR_H
#define COPYDIR_H
-extern void copydir(char *fromdir, char *todir, bool recurse);
-extern void copy_file(char *fromfile, char *tofile);
+#include "storage/relfilenode.h"
+
+extern void copydir(char *fromdir, char *todir, RelFileNode *fromNode, RelFileNode *toNode);
+extern void copy_file(char *fromfile, char *tofile, RelFileNode *fromNode, RelFileNode *toNode, ForkNumber fromForkNum, ForkNumber toForkNum, int segment);
+extern bool parse_filename_for_nontemp_relation(const char *name,
+ int *oidchars, ForkNumber *fork, int *segment);
#endif /* COPYDIR_H */
diff --git a/src/include/storage/encryption.h b/src/include/storage/encryption.h
new file mode 100644
index 0000000..bab6412
--- /dev/null
+++ b/src/include/storage/encryption.h
@@ -0,0 +1,60 @@
+/*-------------------------------------------------------------------------
+ *
+ * encryption.h
+ * Full database encryption support
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/encryption.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef ENCRYPTION_H
+#define ENCRYPTION_H
+
+#include "lib/ilist.h"
+
+#define ENCRYPTION_BLOCK 16
+#define TWEAK_SIZE 16
+
+extern PGDLLIMPORT bool encryption_enabled;
+
+
+void setup_encryption(void);
+void sample_encryption(char *buf);
+void encrypt_block(const char *input, char *output, Size size,
+ const char *tweak);
+void decrypt_block(const char *input, char *output, Size size,
+ const char *tweak);
+
+typedef bool (*SetupEncryption_function) ();
+typedef void (*EncryptBlock_function) (const char *input, char *output,
+ Size size, const char *tweak);
+typedef void (*DecryptBlock_function) (const char *input, char *output,
+ Size size, const char *tweak);
+
+/*
+ * Hook functions to register an encryption provider.
+ */
+typedef struct {
+ dlist_node node;
+ /*
+ * Will be called at system initialization time immediately after loading
+ * the encryption module. Return value indicates if encryption is
+ * successfully initialized. Returning false will result in a FATAL error.
+ */
+ SetupEncryption_function SetupEncryption;
+ /*
+ * Encrypt/decrypt one block of data. Input and output buffers may point
+ * to the same buffer. Buffer alignment is not guaranteed. Buffer size
+ * will be at least 16 bytes, but is not guaranteed to be a multiple of 16.
+ */
+ EncryptBlock_function EncryptBlock;
+ DecryptBlock_function DecryptBlock;
+} EncryptionRoutines;
+
+void register_encryption_module(char *name, EncryptionRoutines *provider);
+
+#endif /* ENCRYPTION_H */
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index a8e7877..7e69289 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -140,6 +140,12 @@ extern void RememberFsyncRequest(RelFileNode rnode, ForkNumber forknum,
extern void ForgetRelationFsyncRequests(RelFileNode rnode, ForkNumber forknum);
extern void ForgetDatabaseFsyncRequests(Oid dbid);
+
+extern BlockNumber ReencryptBlock(char *buffer, int blocks,
+ RelFileNode *srcNode, RelFileNode *dstNode,
+ ForkNumber srcForkNum, ForkNumber dstForkNum,
+ BlockNumber blockNum);
+
/* smgrtype.c */
extern Datum smgrout(PG_FUNCTION_ARGS);
extern Datum smgrin(PG_FUNCTION_ARGS);
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
@ 2016-06-10 02:23 ` Haribabu Kommi <kommi.haribabu@gmail.com>
2016-06-12 07:13 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
3 siblings, 1 reply; 93+ messages in thread
From: Haribabu Kommi @ 2016-06-10 02:23 UTC (permalink / raw)
To: Ants Aasma <ants.aasma@gmail.com>; +Cc: pgsql-hackers
On Tue, Jun 7, 2016 at 11:56 PM, Ants Aasma <ants.aasma@gmail.com> wrote:
> Hi all,
>
> I have been working on data-at-rest encryption support for PostgreSQL.
> In my experience this is a common request that customers make. The
> short of the feature is that all PostgreSQL data files are encrypted
> with a single master key and are decrypted when read from the OS. It
> does not provide column level encryption which is an almost orthogonal
> feature, arguably better done client side.
>
> Similar things can be achieved with filesystem level encryption.
> However this is not always optimal for various reasons. One of the
> better reasons is the desire for HSM based encryption in a storage
> area network based setup.
>
> Attached to this mail is a work in progress patch that adds an
> extensible encryption mechanism. There are some loose ends left to tie
> up, but the general concept and architecture is at a point where it's
> ready for some feedback, fresh ideas and bikeshedding.
Yes, encryption is really a nice and wanted feature.
Following are my thoughts regarding the approach.
1. Instead of doing the entire database files encryption, how about
providing user an option to protect only some particular tables that
wants the encryption at table/tablespace level. This not only provides
an option to the user, it reduces the performance impact on tables
that doesn't need any encryption. The problem with this approach
is that every xlog record needs to validate to handle the encryption
/decryption, instead of at page level.
2. Instead of depending on a contrib module for the encryption, how
about integrating pgcrypto contrib in to the core and add that as a
default encryption method. And also provide an option to the user
to use a different encryption methods if needs.
3. Currently entire xlog pages are encrypted and stored in the file.
can pg_xlogdump works with those files?
4. For logical decoding, how about the adding a decoding behavior
based on the module to decide whether data to be encrypted/decrypted.
5. Instead of providing passphrase through environmental variable,
better to provide some options to pg_ctl etc.
6. I don't have any idea whether is it possible to integrate the checksum
and encryption in a single shot to avoid performance penalty.
> I would also like to incorporate some database identifier as a salt in
> key setup. However, system identifier stored in control file doesn't
> fit this role well. It gets initialized somewhat too late in the
> bootstrap process, and more importantly, gets changed on pg_upgrade.
> This will make link mode upgrades impossible, which seems like a no
> go. I'm torn whether to add a new value for this purpose (perhaps
> stored outside the control file) or allow setting of system identifier
> via initdb. The first seems like a better idea, the file could double
> as a place to store additional encryption parameters, like key length
> or different cipher primitive.
I feel separate file is better to include the key data instead of pg_control
file.
Regards,
Hari Babu
Fujitsu Australia
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-10 02:23 ` Re: WIP: Data at rest encryption Haribabu Kommi <kommi.haribabu@gmail.com>
@ 2016-06-12 07:13 ` Ants Aasma <ants.aasma@gmail.com>
2016-06-13 05:17 ` Re: WIP: Data at rest encryption Michael Paquier <michael.paquier@gmail.com>
2016-06-13 15:18 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2016-06-14 02:20 ` Re: WIP: Data at rest encryption Haribabu Kommi <kommi.haribabu@gmail.com>
2016-06-14 19:59 ` Re: WIP: Data at rest encryption Jim Nasby <Jim.Nasby@BlueTreble.com>
0 siblings, 4 replies; 93+ messages in thread
From: Ants Aasma @ 2016-06-12 07:13 UTC (permalink / raw)
To: Haribabu Kommi <kommi.haribabu@gmail.com>; +Cc: pgsql-hackers
On Fri, Jun 10, 2016 at 5:23 AM, Haribabu Kommi
<kommi.haribabu@gmail.com> wrote:
> 1. Instead of doing the entire database files encryption, how about
> providing user an option to protect only some particular tables that
> wants the encryption at table/tablespace level. This not only provides
> an option to the user, it reduces the performance impact on tables
> that doesn't need any encryption. The problem with this approach
> is that every xlog record needs to validate to handle the encryption
> /decryption, instead of at page level.
Is there a real need for this? The customers I have talked to want to
encrypt the whole database and my goal is to make the feature fast
enough to make that feasible for pretty much everyone. I guess
switching encryption off per table would be feasible, but the key
setup would still need to be done at server startup. Per record
encryption would result in some additional information leakage though.
Overall I thought it would not be worth it, but I'm willing to have my
mind changed on this.
> 2. Instead of depending on a contrib module for the encryption, how
> about integrating pgcrypto contrib in to the core and add that as a
> default encryption method. And also provide an option to the user
> to use a different encryption methods if needs.
Technically that would be simple enough, this is more of a policy
decision. I think having builtin encryption provided by pgcrypto is
completely fine. If a consensus emerges that it needs to be
integrated, it would need to be a separate patch anyway.
> 3. Currently entire xlog pages are encrypted and stored in the file.
> can pg_xlogdump works with those files?
Technically yes, with the patch as it stands, no. Added this to my todo list.
> 4. For logical decoding, how about the adding a decoding behavior
> based on the module to decide whether data to be encrypted/decrypted.
The data to be encrypted does not depend on the module used, so I
don't think it should be module controlled. The reorder buffer
contains pretty much the same stuff as the xlog, so not encrypting it
does not look like a valid choice. For logical heap rewrites it could
be argued that nothing useful is leaked in most cases, but encrypting
it is not hard. Just a small matter of programming.
> 5. Instead of providing passphrase through environmental variable,
> better to provide some options to pg_ctl etc.
That looks like it would be worse from a security perspective.
Integrating a passphrase prompt would be an option, but a way for
scripts to provide passphrases would still be needed.
> 6. I don't have any idea whether is it possible to integrate the checksum
> and encryption in a single shot to avoid performance penalty.
Currently no, the checksum gets stored in the page header and for any
decent cipher mode the encryption of the rest of the page will depend
on it. However, the performance difference should be negligible
because both algorithms are compute bound for cached data. The data is
very likely to be completely in L1 cache as the operations are done in
quick succession.
The non-cryptographic checksum algorithm could actually be an attack
vector for an adversary that can trigger repeated encryption by
tweaking a couple of bytes at the end of the page to see when the
checksum matches and try to infer the data from that. Similarly to the
CRIME attack. However the LSN stored at the beginning of the page
header basically provides a nonce that makes this impossible.
This also means that encryption needs to imply wal_log_hints. Will
include this in the next version of the patch.
>> I would also like to incorporate some database identifier as a salt in
>> key setup. However, system identifier stored in control file doesn't
>> fit this role well. It gets initialized somewhat too late in the
>> bootstrap process, and more importantly, gets changed on pg_upgrade.
>> This will make link mode upgrades impossible, which seems like a no
>> go. I'm torn whether to add a new value for this purpose (perhaps
>> stored outside the control file) or allow setting of system identifier
>> via initdb. The first seems like a better idea, the file could double
>> as a place to store additional encryption parameters, like key length
>> or different cipher primitive.
>
> I feel separate file is better to include the key data instead of pg_control
> file.
I guess that would be more flexible. However I think at least the fact
that the database is encrypted should remain in the control file to
provide useful error messages for faulty backup procedures.
Thanks for your input.
Regards,
Ants Aasma
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-10 02:23 ` Re: WIP: Data at rest encryption Haribabu Kommi <kommi.haribabu@gmail.com>
2016-06-12 07:13 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
@ 2016-06-13 05:17 ` Michael Paquier <michael.paquier@gmail.com>
2016-06-13 13:10 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
3 siblings, 1 reply; 93+ messages in thread
From: Michael Paquier @ 2016-06-13 05:17 UTC (permalink / raw)
To: Ants Aasma <ants.aasma@gmail.com>; +Cc: Haribabu Kommi <kommi.haribabu@gmail.com>; pgsql-hackers
On Sun, Jun 12, 2016 at 4:13 PM, Ants Aasma <ants.aasma@gmail.com> wrote:
>> I feel separate file is better to include the key data instead of pg_control
>> file.
>
> I guess that would be more flexible. However I think at least the fact
> that the database is encrypted should remain in the control file to
> provide useful error messages for faulty backup procedures.
Another possibility could be always to do some encryption at data-type
level for text data. For example I recalled the following thing while
going through this thread:
https://github.com/nec-postgres/tdeforpg
Though I don't quite understand the use for encrypt.enable in this
code... This has the advantage to not patch upstream.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-10 02:23 ` Re: WIP: Data at rest encryption Haribabu Kommi <kommi.haribabu@gmail.com>
2016-06-12 07:13 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 05:17 ` Re: WIP: Data at rest encryption Michael Paquier <michael.paquier@gmail.com>
@ 2016-06-13 13:10 ` Ants Aasma <ants.aasma@eesti.ee>
0 siblings, 0 replies; 93+ messages in thread
From: Ants Aasma @ 2016-06-13 13:10 UTC (permalink / raw)
To: Michael Paquier <michael.paquier@gmail.com>; +Cc: Haribabu Kommi <kommi.haribabu@gmail.com>; pgsql-hackers
On Mon, Jun 13, 2016 at 5:17 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:
> On Sun, Jun 12, 2016 at 4:13 PM, Ants Aasma <ants.aasma@gmail.com> wrote:
>>> I feel separate file is better to include the key data instead of pg_control
>>> file.
>>
>> I guess that would be more flexible. However I think at least the fact
>> that the database is encrypted should remain in the control file to
>> provide useful error messages for faulty backup procedures.
>
> Another possibility could be always to do some encryption at data-type
> level for text data. For example I recalled the following thing while
> going through this thread:
> https://github.com/nec-postgres/tdeforpg
> Though I don't quite understand the use for encrypt.enable in this
> code... This has the advantage to not patch upstream.
While certainly possible, this does not cover the requirements I want
to satisfy - user data never gets stored on disk unencrypted without
making changes to the application or schema. This seems to be mostly
about separating administrator roles, specifically that centralised
storage and backup administrators should not have access to database
contents. I see this as orthogonal to per column encryption, which in
my opinion is better done in the application.
Regards,
Ants Aasma
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-10 02:23 ` Re: WIP: Data at rest encryption Haribabu Kommi <kommi.haribabu@gmail.com>
2016-06-12 07:13 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
@ 2016-06-13 15:18 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
3 siblings, 0 replies; 93+ messages in thread
From: Peter Eisentraut @ 2016-06-13 15:18 UTC (permalink / raw)
To: Ants Aasma <ants.aasma@gmail.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; +Cc: pgsql-hackers
On 6/12/16 3:13 AM, Ants Aasma wrote:
>> 5. Instead of providing passphrase through environmental variable,
>> > better to provide some options to pg_ctl etc.
> That looks like it would be worse from a security perspective.
> Integrating a passphrase prompt would be an option, but a way for
> scripts to provide passphrases would still be needed.
Environment variables and command-line options are visible to other
processes on the machine, so neither of these approaches is really going
to work. We would need some kind of integration with secure
password-entry mechanisms, such as pinentry.
Also note that all tools that work directly on the data directory would
need password-entry and encryption/decryption support, including
pg_basebackup, pg_controldata, pg_ctl, pg_receivexlog, pg_resetxlog,
pg_rewind, pg_upgrade, pg_xlogdump.
It seems that your implementation doesn't encrypt pg_control, thus
avoiding some of that. But that doesn't seem right.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-10 02:23 ` Re: WIP: Data at rest encryption Haribabu Kommi <kommi.haribabu@gmail.com>
2016-06-12 07:13 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
@ 2016-06-14 02:20 ` Haribabu Kommi <kommi.haribabu@gmail.com>
3 siblings, 0 replies; 93+ messages in thread
From: Haribabu Kommi @ 2016-06-14 02:20 UTC (permalink / raw)
To: Ants Aasma <ants.aasma@gmail.com>; +Cc: pgsql-hackers
On Sun, Jun 12, 2016 at 5:13 PM, Ants Aasma <ants.aasma@gmail.com> wrote:
> On Fri, Jun 10, 2016 at 5:23 AM, Haribabu Kommi
> <kommi.haribabu@gmail.com> wrote:
>
>> 2. Instead of depending on a contrib module for the encryption, how
>> about integrating pgcrypto contrib in to the core and add that as a
>> default encryption method. And also provide an option to the user
>> to use a different encryption methods if needs.
>
> Technically that would be simple enough, this is more of a policy
> decision. I think having builtin encryption provided by pgcrypto is
> completely fine. If a consensus emerges that it needs to be
> integrated, it would need to be a separate patch anyway.
In our proprietary database, we are using the encryption methods
provided by openSSL [1]. May be we can have a look at those
methods provided by openSSL for the use of encryption for builds
under USE_SSL. Ignore it if you have already validated.
>> 5. Instead of providing passphrase through environmental variable,
>> better to provide some options to pg_ctl etc.
>
> That looks like it would be worse from a security perspective.
> Integrating a passphrase prompt would be an option, but a way for
> scripts to provide passphrases would still be needed.
What I felt was, if we store the passphrase in an environmental variable,
a person who is having an access to the system can get the details
and using that it may be possible to decrypt the data files.
[1] - https://www.openssl.org/docs/manmaster/crypto/EVP_EncryptInit.html
Regards,
Hari Babu
Fujitsu Australia
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-10 02:23 ` Re: WIP: Data at rest encryption Haribabu Kommi <kommi.haribabu@gmail.com>
2016-06-12 07:13 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
@ 2016-06-14 19:59 ` Jim Nasby <Jim.Nasby@BlueTreble.com>
2016-06-15 20:05 ` Re: WIP: Data at rest encryption PostgreSQL - Hans-Jürgen Schönig <postgres@cybertec.at>
3 siblings, 1 reply; 93+ messages in thread
From: Jim Nasby @ 2016-06-14 19:59 UTC (permalink / raw)
To: Ants Aasma <ants.aasma@gmail.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; +Cc: pgsql-hackers
On 6/12/16 2:13 AM, Ants Aasma wrote:
> On Fri, Jun 10, 2016 at 5:23 AM, Haribabu Kommi
> <kommi.haribabu@gmail.com> wrote:
>> > 1. Instead of doing the entire database files encryption, how about
>> > providing user an option to protect only some particular tables that
>> > wants the encryption at table/tablespace level. This not only provides
>> > an option to the user, it reduces the performance impact on tables
>> > that doesn't need any encryption. The problem with this approach
>> > is that every xlog record needs to validate to handle the encryption
>> > /decryption, instead of at page level.
> Is there a real need for this? The customers I have talked to want to
> encrypt the whole database and my goal is to make the feature fast
> enough to make that feasible for pretty much everyone. I guess
> switching encryption off per table would be feasible, but the key
> setup would still need to be done at server startup. Per record
> encryption would result in some additional information leakage though.
> Overall I thought it would not be worth it, but I'm willing to have my
> mind changed on this.
I actually design with this in mind. Tables that contain sensitive info
go into designated schemas, partly so that you can blanket move all of
those to an encrypted tablespace (or safer would be to move things not
in those schemas to an unencrypted tablespace). Since that can be done
with an encrypted filesystem maybe that's good enough. (It's not really
clear to me what this buys us over an encrypted FS, other than a feature
comparison checkmark...)
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532) mobile: 512-569-9461
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-10 02:23 ` Re: WIP: Data at rest encryption Haribabu Kommi <kommi.haribabu@gmail.com>
2016-06-12 07:13 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-14 19:59 ` Re: WIP: Data at rest encryption Jim Nasby <Jim.Nasby@BlueTreble.com>
@ 2016-06-15 20:05 ` PostgreSQL - Hans-Jürgen Schönig <postgres@cybertec.at>
0 siblings, 0 replies; 93+ messages in thread
From: PostgreSQL - Hans-Jürgen Schönig @ 2016-06-15 20:05 UTC (permalink / raw)
To: pgsql-hackers
On 06/14/2016 09:59 PM, Jim Nasby wrote:
> On 6/12/16 2:13 AM, Ants Aasma wrote:
>> On Fri, Jun 10, 2016 at 5:23 AM, Haribabu Kommi
>> <kommi.haribabu@gmail.com> wrote:
>>> > 1. Instead of doing the entire database files encryption, how about
>>> > providing user an option to protect only some particular tables that
>>> > wants the encryption at table/tablespace level. This not only
>>> provides
>>> > an option to the user, it reduces the performance impact on tables
>>> > that doesn't need any encryption. The problem with this approach
>>> > is that every xlog record needs to validate to handle the encryption
>>> > /decryption, instead of at page level.
>> Is there a real need for this? The customers I have talked to want to
>> encrypt the whole database and my goal is to make the feature fast
>> enough to make that feasible for pretty much everyone. I guess
>> switching encryption off per table would be feasible, but the key
>> setup would still need to be done at server startup. Per record
>> encryption would result in some additional information leakage though.
>> Overall I thought it would not be worth it, but I'm willing to have my
>> mind changed on this.
>
> I actually design with this in mind. Tables that contain sensitive
> info go into designated schemas, partly so that you can blanket move
> all of those to an encrypted tablespace (or safer would be to move
> things not in those schemas to an unencrypted tablespace). Since that
> can be done with an encrypted filesystem maybe that's good enough.
> (It's not really clear to me what this buys us over an encrypted FS,
> other than a feature comparison checkmark...)
the reason why this is needed is actually very simple: security
guidelines and legal requirements ...
we have dealt with a couple of companies recently, who explicitly
demanded PostgreSQL level encryption in a transparent way to fulfill
some internal or legal requirements. this is especially true for
financial stuff. and yes, sure ... you can do a lot of stuff with
filesystem encryption.
the core idea of this entire thing is however to have a counterpart on
the database level. if you don't have the key you cannot start the
instance and if you happen to get access to the filesystem you are still
not able to fire up the DB.
as it said: requirements by ever bigger companies.
as far as benchmarking is concerned: i did a quick test yesterday (not
with the final AES implementation yet) and i got pretty good results.
with a reasonably well cached database in a typical application I expect
to loose around 10-20%. if everything fits in memory there is 0 loss of
course. the worst I got with the standard AES (no hardware support used
yet) I lost around 45% or so. but this requires a value as low as 32 MB
of shared buffers or so.
many thanks,
hans
--
Hans-Jürgen Schönig
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt
Web: http://www.postgresql-support.de, http://www.cybertec.at
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
@ 2016-06-13 15:07 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
3 siblings, 1 reply; 93+ messages in thread
From: Peter Eisentraut @ 2016-06-13 15:07 UTC (permalink / raw)
To: Ants Aasma <ants.aasma@gmail.com>; pgsql-hackers
On 6/7/16 9:56 AM, Ants Aasma wrote:
> Similar things can be achieved with filesystem level encryption.
> However this is not always optimal for various reasons. One of the
> better reasons is the desire for HSM based encryption in a storage
> area network based setup.
Could you explain this in more detail?
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
@ 2017-06-12 19:38 ` Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-18 22:11 ` Re: WIP: Data at rest encryption Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 2 replies; 93+ messages in thread
From: Robert Haas @ 2017-06-12 19:38 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; +Cc: Ants Aasma <ants.aasma@gmail.com>; pgsql-hackers
On Mon, Jun 13, 2016 at 11:07 AM, Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:
> On 6/7/16 9:56 AM, Ants Aasma wrote:
>>
>> Similar things can be achieved with filesystem level encryption.
>> However this is not always optimal for various reasons. One of the
>> better reasons is the desire for HSM based encryption in a storage
>> area network based setup.
>
> Could you explain this in more detail?
I don't think Ants ever responded to this point.
I'm curious whether this is something that is likely to be pursued for
PostgreSQL 11.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
@ 2017-06-12 21:11 ` Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 15:35 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
1 sibling, 3 replies; 93+ messages in thread
From: Ants Aasma @ 2017-06-12 21:11 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Jun 12, 2017 at 10:38 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Mon, Jun 13, 2016 at 11:07 AM, Peter Eisentraut
> <peter.eisentraut@2ndquadrant.com> wrote:
>> On 6/7/16 9:56 AM, Ants Aasma wrote:
>>>
>>> Similar things can be achieved with filesystem level encryption.
>>> However this is not always optimal for various reasons. One of the
>>> better reasons is the desire for HSM based encryption in a storage
>>> area network based setup.
>>
>> Could you explain this in more detail?
>
> I don't think Ants ever responded to this point.
>
> I'm curious whether this is something that is likely to be pursued for
> PostgreSQL 11.
Yes, the plan is to pick it up again, Real Soon Now(tm). There are a
couple of loose ends for stuff that should be encrypted, but in the
current state of the patch aren't yet (from the top of my head,
logical decoding and pg_stat_statements write some files). The code
handling keys could really take better precautions as Peter pointed
out in another e-mail. And I expect there to be a bunch of polishing
work to make the APIs as good as they can be.
To answer Peter's question about HSMs, many enterprise deployments are
on top of shared storage systems. For regulatory reasons or to limit
security clearance of storage administrators, the data on shared
storage should be encrypted. Now for there to be any point to this
endeavor, the key needs to be stored somewhere else. This is where
hardware security modules come in. They are basically hardware key
storage appliances that can either output the key when requested, or
for higher security hold onto the key and perform
encryption/decryption on behalf of the user. The patch enables the
user to use a custom shell command to go and fetch the key from the
HSM, for example using the KMIP protocol. Or a motivated person could
write an extension that implements the encryption hooks to delegate
encryption/decryption of blocks to an HSM.
Fundamentally there doesn't seem to be a big benefit of implementing
the encryption at PostgreSQL level instead of the filesystem. The
patch doesn't take any real advantage from the higher level knowledge
of the system, nor do I see much possibility for it to do that. The
main benefit for us is that it's much easier to get a PostgreSQL based
solution deployed.
I'm curious if the community thinks this is a feature worth having?
Even considering that security experts would classify this kind of
encryption as a checkbox feature.
Regards,
Ants Aasma
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
@ 2017-06-13 13:07 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 09:04 ` Re: WIP: Data at rest encryption Aleksander Alekseev <a.alekseev@postgrespro.ru>
2 siblings, 1 reply; 93+ messages in thread
From: Peter Eisentraut @ 2017-06-13 13:07 UTC (permalink / raw)
To: Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; +Cc: pgsql-hackers
On 6/12/17 17:11, Ants Aasma wrote:
> I'm curious if the community thinks this is a feature worth having?
> Even considering that security experts would classify this kind of
> encryption as a checkbox feature.
File system encryption already exists and is well-tested. I don't see
any big advantages in re-implementing all of this one level up. You
would have to touch every single place in PostgreSQL backend and tool
code where a file is being read or written. Yikes.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
@ 2017-06-14 09:04 ` Aleksander Alekseev <a.alekseev@postgrespro.ru>
2017-06-14 12:46 ` Re: WIP: Data at rest encryption Kenneth Marshall <ktm@rice.edu>
2017-06-14 21:32 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
0 siblings, 2 replies; 93+ messages in thread
From: Aleksander Alekseev @ 2017-06-14 09:04 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; +Cc: Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
Hi Ants,
On Tue, Jun 13, 2017 at 09:07:49AM -0400, Peter Eisentraut wrote:
> On 6/12/17 17:11, Ants Aasma wrote:
> > I'm curious if the community thinks this is a feature worth having?
> > Even considering that security experts would classify this kind of
> > encryption as a checkbox feature.
>
> File system encryption already exists and is well-tested. I don't see
> any big advantages in re-implementing all of this one level up. You
> would have to touch every single place in PostgreSQL backend and tool
> code where a file is being read or written. Yikes.
I appreciate your work, but unfortunately I must agree with Peter.
On Linux you can configure the full disc encryption using LUKS /
dm-crypt in like 5 minutes [1]. On FreeBSD you can do the same using
geli [2]. In my personal opinion PostgreSQL is already complicated
enough. A few companies that hired system administrators that are too
lazy to read two or three man pages is not a reason to re-implement file
system encryption (or compression, or mirroring if that matters) in any
open source RDBMS.
[1] http://eax.me/dm-crypt/
[2] http://eax.me/freebsd-geli/
--
Best regards,
Aleksander Alekseev
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 09:04 ` Re: WIP: Data at rest encryption Aleksander Alekseev <a.alekseev@postgrespro.ru>
@ 2017-06-14 12:46 ` Kenneth Marshall <ktm@rice.edu>
2017-06-14 13:05 ` Re: WIP: Data at rest encryption Aleksander Alekseev <a.alekseev@postgrespro.ru>
1 sibling, 1 reply; 93+ messages in thread
From: Kenneth Marshall @ 2017-06-14 12:46 UTC (permalink / raw)
To: Aleksander Alekseev <a.alekseev@postgrespro.ru>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
On Wed, Jun 14, 2017 at 12:04:26PM +0300, Aleksander Alekseev wrote:
> Hi Ants,
>
> On Tue, Jun 13, 2017 at 09:07:49AM -0400, Peter Eisentraut wrote:
> > On 6/12/17 17:11, Ants Aasma wrote:
> > > I'm curious if the community thinks this is a feature worth having?
> > > Even considering that security experts would classify this kind of
> > > encryption as a checkbox feature.
> >
> > File system encryption already exists and is well-tested. I don't see
> > any big advantages in re-implementing all of this one level up. You
> > would have to touch every single place in PostgreSQL backend and tool
> > code where a file is being read or written. Yikes.
>
> I appreciate your work, but unfortunately I must agree with Peter.
>
> On Linux you can configure the full disc encryption using LUKS /
> dm-crypt in like 5 minutes [1]. On FreeBSD you can do the same using
> geli [2]. In my personal opinion PostgreSQL is already complicated
> enough. A few companies that hired system administrators that are too
> lazy to read two or three man pages is not a reason to re-implement file
> system encryption (or compression, or mirroring if that matters) in any
> open source RDBMS.
>
Hi Aleksander,
While I agree that configuring full disk encryption is not technically
difficult, it requires much more privileged access to the system and
basically requires the support of a system administrator. In addition,
if a volume is not available for encryption, PostgreSQL support for
encryption would still allow for its data to be encrypted and as others
have mentioned can be enabled by the DBA alone.
Regards,
Ken
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 09:04 ` Re: WIP: Data at rest encryption Aleksander Alekseev <a.alekseev@postgrespro.ru>
2017-06-14 12:46 ` Re: WIP: Data at rest encryption Kenneth Marshall <ktm@rice.edu>
@ 2017-06-14 13:05 ` Aleksander Alekseev <a.alekseev@postgrespro.ru>
2017-06-14 13:13 ` Re: WIP: Data at rest encryption Aleksander Alekseev <a.alekseev@postgrespro.ru>
0 siblings, 1 reply; 93+ messages in thread
From: Aleksander Alekseev @ 2017-06-14 13:05 UTC (permalink / raw)
To: Kenneth Marshall <ktm@rice.edu>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
Hi Kenneth,
> > > File system encryption already exists and is well-tested. I don't see
> > > any big advantages in re-implementing all of this one level up. You
> > > would have to touch every single place in PostgreSQL backend and tool
> > > code where a file is being read or written. Yikes.
> >
> > I appreciate your work, but unfortunately I must agree with Peter.
> >
> > On Linux you can configure the full disc encryption using LUKS /
> > dm-crypt in like 5 minutes [1]. On FreeBSD you can do the same using
> > geli [2]. In my personal opinion PostgreSQL is already complicated
> > enough. A few companies that hired system administrators that are too
> > lazy to read two or three man pages is not a reason to re-implement file
> > system encryption (or compression, or mirroring if that matters) in any
> > open source RDBMS.
>
> While I agree that configuring full disk encryption is not technically
> difficult, it requires much more privileged access to the system and
> basically requires the support of a system administrator. In addition,
> if a volume is not available for encryption, PostgreSQL support for
> encryption would still allow for its data to be encrypted and as others
> have mentioned can be enabled by the DBA alone.
Frankly I'm having difficulties imagining when it could be a real
problem. It doesn't seem to be such a burden to ask a colleague for
assistance in case you don't have sufficient permissions to do
something. And I got a strong feeling that solving bureaucracy issues of
specific organizations by changing PostgreSQL core in very invasive way
(keeping in mind testing, maintaining, etc) is misguided.
--
Best regards,
Aleksander Alekseev
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 09:04 ` Re: WIP: Data at rest encryption Aleksander Alekseev <a.alekseev@postgrespro.ru>
2017-06-14 12:46 ` Re: WIP: Data at rest encryption Kenneth Marshall <ktm@rice.edu>
2017-06-14 13:05 ` Re: WIP: Data at rest encryption Aleksander Alekseev <a.alekseev@postgrespro.ru>
@ 2017-06-14 13:13 ` Aleksander Alekseev <a.alekseev@postgrespro.ru>
2017-06-14 14:31 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 93+ messages in thread
From: Aleksander Alekseev @ 2017-06-14 13:13 UTC (permalink / raw)
To: Kenneth Marshall <ktm@rice.edu>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
> > While I agree that configuring full disk encryption is not technically
> > difficult, it requires much more privileged access to the system and
> > basically requires the support of a system administrator. In addition,
> > if a volume is not available for encryption, PostgreSQL support for
> > encryption would still allow for its data to be encrypted and as others
> > have mentioned can be enabled by the DBA alone.
>
> Frankly I'm having difficulties imagining when it could be a real
> problem. It doesn't seem to be such a burden to ask a colleague for
> assistance in case you don't have sufficient permissions to do
> something. And I got a strong feeling that solving bureaucracy issues of
> specific organizations by changing PostgreSQL core in very invasive way
> (keeping in mind testing, maintaining, etc) is misguided.
In the same time implementing a plugable storage API and then implementing
encrypted / compressed / whatever storage in a standalone extension using
this API seems to be a reasonable thing to do.
--
Best regards,
Aleksander Alekseev
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 09:04 ` Re: WIP: Data at rest encryption Aleksander Alekseev <a.alekseev@postgrespro.ru>
2017-06-14 12:46 ` Re: WIP: Data at rest encryption Kenneth Marshall <ktm@rice.edu>
2017-06-14 13:05 ` Re: WIP: Data at rest encryption Aleksander Alekseev <a.alekseev@postgrespro.ru>
2017-06-14 13:13 ` Re: WIP: Data at rest encryption Aleksander Alekseev <a.alekseev@postgrespro.ru>
@ 2017-06-14 14:31 ` Bruce Momjian <bruce@momjian.us>
0 siblings, 0 replies; 93+ messages in thread
From: Bruce Momjian @ 2017-06-14 14:31 UTC (permalink / raw)
To: Aleksander Alekseev <a.alekseev@postgrespro.ru>; +Cc: Kenneth Marshall <ktm@rice.edu>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
On Wed, Jun 14, 2017 at 04:13:57PM +0300, Aleksander Alekseev wrote:
> > > While I agree that configuring full disk encryption is not technically
> > > difficult, it requires much more privileged access to the system and
> > > basically requires the support of a system administrator. In addition,
> > > if a volume is not available for encryption, PostgreSQL support for
> > > encryption would still allow for its data to be encrypted and as others
> > > have mentioned can be enabled by the DBA alone.
> >
> > Frankly I'm having difficulties imagining when it could be a real
> > problem. It doesn't seem to be such a burden to ask a colleague for
> > assistance in case you don't have sufficient permissions to do
> > something. And I got a strong feeling that solving bureaucracy issues of
> > specific organizations by changing PostgreSQL core in very invasive way
> > (keeping in mind testing, maintaining, etc) is misguided.
>
> In the same time implementing a plugable storage API and then implementing
> encrypted / compressed / whatever storage in a standalone extension using
> this API seems to be a reasonable thing to do.
Agreed, good point.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 09:04 ` Re: WIP: Data at rest encryption Aleksander Alekseev <a.alekseev@postgrespro.ru>
@ 2017-06-14 21:32 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
1 sibling, 0 replies; 93+ messages in thread
From: Peter Eisentraut @ 2017-06-14 21:32 UTC (permalink / raw)
To: Aleksander Alekseev <a.alekseev@postgrespro.ru>; +Cc: Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
On 6/14/17 05:04, Aleksander Alekseev wrote:
> A few companies that hired system administrators that are too
> lazy to read two or three man pages is not a reason to re-implement file
> system encryption (or compression, or mirroring if that matters) in any
> open source RDBMS.
To be fair, we did implement our own compression (TOAST) and mirroring
(compared to, say, DRBD) because there were clear advantages in
simplicity and performance. Checksumming is another area where we moved
forward in spite of arguments that the file system should do it. So we
will need to see the whole picture.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
@ 2017-06-13 13:24 ` Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2 siblings, 1 reply; 93+ messages in thread
From: Stephen Frost @ 2017-06-13 13:24 UTC (permalink / raw)
To: Ants Aasma <ants.aasma@eesti.ee>; +Cc: Robert Haas <robertmhaas@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
Ants, all,
* Ants Aasma (ants.aasma@eesti.ee) wrote:
> Yes, the plan is to pick it up again, Real Soon Now(tm). There are a
> couple of loose ends for stuff that should be encrypted, but in the
> current state of the patch aren't yet (from the top of my head,
> logical decoding and pg_stat_statements write some files). The code
> handling keys could really take better precautions as Peter pointed
> out in another e-mail. And I expect there to be a bunch of polishing
> work to make the APIs as good as they can be.
Very glad to hear that you're going to be continuing to work on this
effort.
> To answer Peter's question about HSMs, many enterprise deployments are
> on top of shared storage systems. For regulatory reasons or to limit
> security clearance of storage administrators, the data on shared
> storage should be encrypted. Now for there to be any point to this
> endeavor, the key needs to be stored somewhere else. This is where
> hardware security modules come in. They are basically hardware key
> storage appliances that can either output the key when requested, or
> for higher security hold onto the key and perform
> encryption/decryption on behalf of the user. The patch enables the
> user to use a custom shell command to go and fetch the key from the
> HSM, for example using the KMIP protocol. Or a motivated person could
> write an extension that implements the encryption hooks to delegate
> encryption/decryption of blocks to an HSM.
An extension, or perhaps even something built-in, would certainly be
good here but I don't think it's necessary in an initial implementation
as long as it's something we can do later.
> Fundamentally there doesn't seem to be a big benefit of implementing
> the encryption at PostgreSQL level instead of the filesystem. The
> patch doesn't take any real advantage from the higher level knowledge
> of the system, nor do I see much possibility for it to do that. The
> main benefit for us is that it's much easier to get a PostgreSQL based
> solution deployed.
Making it easier to get a PostgreSQL solution deployed is certainly a
very worthwhile goal.
> I'm curious if the community thinks this is a feature worth having?
> Even considering that security experts would classify this kind of
> encryption as a checkbox feature.
I would say that some security experts would consider it a 'checkbox'
feature, while others would say that it's actually a quite useful
capability for a database to have and isn't just for being able to check
a given box. I tended to lean towards the 'checkbox' camp and
encouraged people to use filesystem encryption also, but there are
use-cases where it'd be really nice to be able to have PG doing the
encryption instead of the filesystem because then you can do things like
backup the database, copy it somewhere else directly, and then restore
it using the regular PG mechanisms, as long as you have access to the
key. That's not something you can directly do with filesystem-level
encryption (unless you happen to be lucky enough to be able to use ZFS,
which can do exporting, or you can do a block-level exact copy to an
exactly identically sized partition on the remote side or similar..),
and while you could encrypt the PG files during the backup, that
requires that you make sure both sides agree on how that encryption is
done and have the same tools for performing the encryption/decryption.
Possible, certainly, but not nearly as convenient.
+1 for having this capability.
Thanks!
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
@ 2017-06-13 14:28 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 20:52 ` Re: WIP: Data at rest encryption David Fetter <david@fetter.org>
0 siblings, 2 replies; 93+ messages in thread
From: Peter Eisentraut @ 2017-06-13 14:28 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; Ants Aasma <ants.aasma@eesti.ee>; +Cc: Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
On 6/13/17 09:24, Stephen Frost wrote:
> but there are
> use-cases where it'd be really nice to be able to have PG doing the
> encryption instead of the filesystem because then you can do things like
> backup the database, copy it somewhere else directly, and then restore
> it using the regular PG mechanisms, as long as you have access to the
> key. That's not something you can directly do with filesystem-level
> encryption
Interesting point.
I wonder what the proper extent of "encryption at rest" should be. If
you encrypt just on a file or block level, then someone looking at the
data directory or a backup can still learn a number of things about the
number of tables, transaction rates, various configuration settings, and
so on. In the scenario of a sensitive application hosted on a shared
SAN, I don't think that is good enough.
Also, in the use case you describe, if you use pg_basebackup to make a
direct encrypted copy of a data directory, I think that would mean you'd
have to keep using the same key for all copies.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
@ 2017-06-13 15:04 ` Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
1 sibling, 1 reply; 93+ messages in thread
From: Stephen Frost @ 2017-06-13 15:04 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; +Cc: Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
Peter,
* Peter Eisentraut (peter.eisentraut@2ndquadrant.com) wrote:
> I wonder what the proper extent of "encryption at rest" should be. If
> you encrypt just on a file or block level, then someone looking at the
> data directory or a backup can still learn a number of things about the
> number of tables, transaction rates, various configuration settings, and
> so on. In the scenario of a sensitive application hosted on a shared
> SAN, I don't think that is good enough.
If someone has access to the SAN, it'd be very difficult to avoid
revealing some information about transaction rates or I/O throughput.
Being able to have the configuration files encrypted would be good
(thinking particularly about pg_hba.conf/pg_ident.conf) but I don't
know that it's strictly necessary or that it would have to be done in an
initial version.
Certainly, there is a trade-off here when it comes to the information
which someone can learn about the system by looking at the number and
sizes of files from using PG-based encryption vs. what information
someone can learn from being able to look at only an encrypted
filesystem, but that's a trade-off which security experts are good at
making a determination on and will be case-by-case, based on how easy
setting up filesystem-encryption is in a particular environment and what
the use-cases are for the system.
> Also, in the use case you describe, if you use pg_basebackup to make a
> direct encrypted copy of a data directory, I think that would mean you'd
> have to keep using the same key for all copies.
That's true, but that might be acceptable and possibly even desirable in
certain cases. On the other hand, it would certainly be a useful
feature to have a way to migrate from one key to another. Perhaps that
would start out as an off-line tool, but maybe we'd be able to work out
a way to support having it done on-line in the future (certainly
non-trivial, but if we supported multiple keys concurrently with a
preference for which key is used to write data back out, and required
that checksums be in place to allow us to test if decrypting with a
specific key worked ... lots more hand-waving here... ).
Thanks!
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
@ 2017-06-13 16:04 ` Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 93+ messages in thread
From: Bruce Momjian @ 2017-06-13 16:04 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
On Tue, Jun 13, 2017 at 11:04:21AM -0400, Stephen Frost wrote:
> > Also, in the use case you describe, if you use pg_basebackup to make a
> > direct encrypted copy of a data directory, I think that would mean you'd
> > have to keep using the same key for all copies.
>
> That's true, but that might be acceptable and possibly even desirable in
> certain cases. On the other hand, it would certainly be a useful
> feature to have a way to migrate from one key to another. Perhaps that
> would start out as an off-line tool, but maybe we'd be able to work out
> a way to support having it done on-line in the future (certainly
> non-trivial, but if we supported multiple keys concurrently with a
> preference for which key is used to write data back out, and required
> that checksums be in place to allow us to test if decrypting with a
> specific key worked ... lots more hand-waving here... ).
As I understand it, having encryption in the database means the key is
stored in the database, while having encryption in the file system means
the key is stored in the operating system somewhere. Of course, if the
key stored in the database is visible to someone using the operating
system, we really haven't added much/any security --- I guess my point
is that the OS easily can hide the key from the database, but the
database can't easily hide the key from the operating system.
Of course, if the storage is split from the database server then having
the key on the database server seems like a win. However, I think a db
server could easily encrypt blocks before sending them to the SAN
server. This would not work for NAS, of course, since it is file-based.
I have to admit we tend to avoid heavy-API solutions that are designed
just to work around deployment challenges. Commercial databases are
fine in doing that, but it leads to very complex products.
I think the larger issue is where to store the key. I would love for us
to come up with a unified solution to that and then build encryption on
that, including all-cluster encryption.
One cool idea I have is using public encryption to store the encryption
key by users who don't know the decryption key, e.g. RSA. It would be a
write-only encryption option. Not sure how useful that is, but it
easily possible, and doesn't require us to keep the _encryption_ key
secret, just the decryption one.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
@ 2017-06-13 16:23 ` Stephen Frost <sfrost@snowman.net>
2017-06-13 16:28 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
0 siblings, 2 replies; 93+ messages in thread
From: Stephen Frost @ 2017-06-13 16:23 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
Bruce,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Tue, Jun 13, 2017 at 11:04:21AM -0400, Stephen Frost wrote:
> > > Also, in the use case you describe, if you use pg_basebackup to make a
> > > direct encrypted copy of a data directory, I think that would mean you'd
> > > have to keep using the same key for all copies.
> >
> > That's true, but that might be acceptable and possibly even desirable in
> > certain cases. On the other hand, it would certainly be a useful
> > feature to have a way to migrate from one key to another. Perhaps that
> > would start out as an off-line tool, but maybe we'd be able to work out
> > a way to support having it done on-line in the future (certainly
> > non-trivial, but if we supported multiple keys concurrently with a
> > preference for which key is used to write data back out, and required
> > that checksums be in place to allow us to test if decrypting with a
> > specific key worked ... lots more hand-waving here... ).
>
> As I understand it, having encryption in the database means the key is
> stored in the database, while having encryption in the file system means
> the key is stored in the operating system somewhere.
Key management is an entirely independent discussion from this and the
proposal from Ants, as I understand it, is that the key would *not* be
in the database but could be anywhere that a shell command could get it
from, including possibly a HSM (hardware device).
Having the data encrypted by PostgreSQL does not mean the key is stored
in the database.
> Of course, if the
> key stored in the database is visible to someone using the operating
> system, we really haven't added much/any security --- I guess my point
> is that the OS easily can hide the key from the database, but the
> database can't easily hide the key from the operating system.
This is correct- the key must be available to the PostgreSQL process
and therefore someone with privileged access to the OS would be able to
retrieve the key, but that's also true of filesystem encryption.
Basically, if the server is doing the encryption and you have the
ability to read all memory on the server then you can get the key. Of
course, if you can read all memory then you can just look at shared
buffers and you don't really need to bother yourself with the key or
the encryption, and it doesn't make any difference if you're encrypting
in the database or in the filesystem. That attack vector is not one
which this is intending to address.
> Of course, if the storage is split from the database server then having
> the key on the database server seems like a win. However, I think a db
> server could easily encrypt blocks before sending them to the SAN
> server. This would not work for NAS, of course, since it is file-based.
The key doesn't necessairly have to be stored anywhere on the server- it
just needs to be kept in memory while the database process is running
and made available to the database at startup, unless an external system
is used to perform the encryption, which might be possible with an
extension, as discussed. In some environments, it might be acceptable
to have the key stored on the database server, of course, but there's no
requirement for the key to be stored on the database server or in the
database at all.
> I have to admit we tend to avoid heavy-API solutions that are designed
> just to work around deployment challenges. Commercial databases are
> fine in doing that, but it leads to very complex products.
I'm not following what you mean here.
> I think the larger issue is where to store the key. I would love for us
> to come up with a unified solution to that and then build encryption on
> that, including all-cluster encryption.
Honestly, key management is something that I'd rather we *not* worry
about in an initial implementation, which is one reason that I liked the
approach discussed here of having a command which runs to provide the
key. We could certainly look into improving that in the future, but key
management really is a largely independent issue from encryption and
it's much more difficult and complicated and whatever we come up with
would still almost certainly be usable with the approach proposed here.
> One cool idea I have is using public encryption to store the encryption
> key by users who don't know the decryption key, e.g. RSA. It would be a
> write-only encryption option. Not sure how useful that is, but it
> easily possible, and doesn't require us to keep the _encryption_ key
> secret, just the decryption one.
The downside here is that asymmetric encryption is much more expensive
than symmetric encryption and that probably makes it a non-starter. I
do think we'll want to support multiple encryption methods and perhaps
we can have an option where asymmetric encryption is used, but that's
not what I expect will be typically used.
Thanks!
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
@ 2017-06-13 16:28 ` Robert Haas <robertmhaas@gmail.com>
2017-06-13 16:55 ` Re: WIP: Data at rest encryption Joe Conway <mail@joeconway.com>
1 sibling, 1 reply; 93+ messages in thread
From: Robert Haas @ 2017-06-13 16:28 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Bruce Momjian <bruce@momjian.us>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; pgsql-hackers
On Tue, Jun 13, 2017 at 12:23 PM, Stephen Frost <sfrost@snowman.net> wrote:
> Key management is an entirely independent discussion from this and the
> proposal from Ants, as I understand it, is that the key would *not* be
> in the database but could be anywhere that a shell command could get it
> from, including possibly a HSM (hardware device).
Yes. I think the right way to implement this is something like:
1. Have a GUC that runs a shell command to get the key.
2. If the command successfully gets the key, it prints it to stdout
and returns 0.
3. If it doesn't get successfully get the key, it returns 1. The
database can retry or give up, whatever we decide to do.
That way, if the user wants to store the key in an unencrypted text
file, they can set the encryption_key_command = 'cat /not/very/secure'
and call it a day. If they want to prompt the user on the console or
request the key from an HSM or get it in any other way, they just have
to write the appropriate shell script. We just provide mechanism, not
policy, and the user can adopt any policy they like, from an extremely
insecure policy to one suitable for Fort Knox.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:28 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
@ 2017-06-13 16:55 ` Joe Conway <mail@joeconway.com>
2017-06-13 16:58 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 93+ messages in thread
From: Joe Conway @ 2017-06-13 16:55 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; Stephen Frost <sfrost@snowman.net>; +Cc: Bruce Momjian <bruce@momjian.us>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; pgsql-hackers
On 06/13/2017 09:28 AM, Robert Haas wrote:
> On Tue, Jun 13, 2017 at 12:23 PM, Stephen Frost <sfrost@snowman.net> wrote:
>> Key management is an entirely independent discussion from this and the
>> proposal from Ants, as I understand it, is that the key would *not* be
>> in the database but could be anywhere that a shell command could get it
>> from, including possibly a HSM (hardware device).
>
> Yes. I think the right way to implement this is something like:
>
> 1. Have a GUC that runs a shell command to get the key.
>
> 2. If the command successfully gets the key, it prints it to stdout
> and returns 0.
>
> 3. If it doesn't get successfully get the key, it returns 1. The
> database can retry or give up, whatever we decide to do.
>
> That way, if the user wants to store the key in an unencrypted text
> file, they can set the encryption_key_command = 'cat /not/very/secure'
> and call it a day. If they want to prompt the user on the console or
> request the key from an HSM or get it in any other way, they just have
> to write the appropriate shell script. We just provide mechanism, not
> policy, and the user can adopt any policy they like, from an extremely
> insecure policy to one suitable for Fort Knox.
Agreed, but as Bruce alluded to, we want this to be a master key, which
is in turn used to encrypt the actual key, or keys, that are used to
encrypt the data. The actual data encryption keys could be very long
randomly generated binary, and there could be more than one of them
(e.g. one per tablespace) in a file which is encrypted with the master
key. This is more secure and allows, for example the master key to be
changed without having to decrypt/re-encrypt the entire database.
Joe
--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:28 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-13 16:55 ` Re: WIP: Data at rest encryption Joe Conway <mail@joeconway.com>
@ 2017-06-13 16:58 ` Bruce Momjian <bruce@momjian.us>
2017-06-13 17:05 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 93+ messages in thread
From: Bruce Momjian @ 2017-06-13 16:58 UTC (permalink / raw)
To: Joe Conway <mail@joeconway.com>; +Cc: Robert Haas <robertmhaas@gmail.com>; Stephen Frost <sfrost@snowman.net>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; pgsql-hackers
On Tue, Jun 13, 2017 at 09:55:10AM -0700, Joe Conway wrote:
> > That way, if the user wants to store the key in an unencrypted text
> > file, they can set the encryption_key_command = 'cat /not/very/secure'
> > and call it a day. If they want to prompt the user on the console or
> > request the key from an HSM or get it in any other way, they just have
> > to write the appropriate shell script. We just provide mechanism, not
> > policy, and the user can adopt any policy they like, from an extremely
> > insecure policy to one suitable for Fort Knox.
>
> Agreed, but as Bruce alluded to, we want this to be a master key, which
> is in turn used to encrypt the actual key, or keys, that are used to
> encrypt the data. The actual data encryption keys could be very long
> randomly generated binary, and there could be more than one of them
> (e.g. one per tablespace) in a file which is encrypted with the master
> key. This is more secure and allows, for example the master key to be
> changed without having to decrypt/re-encrypt the entire database.
Yes, thank you. Also, you can make multiple RSA-encrypted copies of the
symetric key, one for each role you want to view the data. And good
point on the ability to change the RSA key/password without having to
reencrypt the data.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:28 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-13 16:55 ` Re: WIP: Data at rest encryption Joe Conway <mail@joeconway.com>
2017-06-13 16:58 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
@ 2017-06-13 17:05 ` Stephen Frost <sfrost@snowman.net>
2017-06-13 17:09 ` Re: WIP: Data at rest encryption Joe Conway <mail@joeconway.com>
0 siblings, 1 reply; 93+ messages in thread
From: Stephen Frost @ 2017-06-13 17:05 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Joe Conway <mail@joeconway.com>; Robert Haas <robertmhaas@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; pgsql-hackers
Bruce, Joe,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Tue, Jun 13, 2017 at 09:55:10AM -0700, Joe Conway wrote:
> > > That way, if the user wants to store the key in an unencrypted text
> > > file, they can set the encryption_key_command = 'cat /not/very/secure'
> > > and call it a day. If they want to prompt the user on the console or
> > > request the key from an HSM or get it in any other way, they just have
> > > to write the appropriate shell script. We just provide mechanism, not
> > > policy, and the user can adopt any policy they like, from an extremely
> > > insecure policy to one suitable for Fort Knox.
> >
> > Agreed, but as Bruce alluded to, we want this to be a master key, which
> > is in turn used to encrypt the actual key, or keys, that are used to
> > encrypt the data. The actual data encryption keys could be very long
> > randomly generated binary, and there could be more than one of them
> > (e.g. one per tablespace) in a file which is encrypted with the master
> > key. This is more secure and allows, for example the master key to be
> > changed without having to decrypt/re-encrypt the entire database.
>
> Yes, thank you. Also, you can make multiple RSA-encrypted copies of the
> symetric key, one for each role you want to view the data. And good
> point on the ability to change the RSA key/password without having to
> reencrypt the data.
There's nothing in this proposal that prevents the user from using a
very long randomly generated binary key. We aren't talking about
prompting the user for a password unless that's what they decide the
shell script should do, unless the user decides to do that and if they
do then that's their choice.
Let us, please, stop stressing over the right way to do key management
as part of this discussion about providing encryption. The two are
different things and we do not need to solve both at once.
Further, yes, we will definitely want to get to a point where we can
encrypt subsets of the system in different ways, but that doesn't have
to be done in the first implementation either.
Thanks!
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:28 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-13 16:55 ` Re: WIP: Data at rest encryption Joe Conway <mail@joeconway.com>
2017-06-13 16:58 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:05 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
@ 2017-06-13 17:09 ` Joe Conway <mail@joeconway.com>
2017-06-13 17:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 93+ messages in thread
From: Joe Conway @ 2017-06-13 17:09 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; Bruce Momjian <bruce@momjian.us>; +Cc: Robert Haas <robertmhaas@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; pgsql-hackers
On 06/13/2017 10:05 AM, Stephen Frost wrote:
> Bruce, Joe,
>
> * Bruce Momjian (bruce@momjian.us) wrote:
>> On Tue, Jun 13, 2017 at 09:55:10AM -0700, Joe Conway wrote:
>> > > That way, if the user wants to store the key in an unencrypted text
>> > > file, they can set the encryption_key_command = 'cat /not/very/secure'
>> > > and call it a day. If they want to prompt the user on the console or
>> > > request the key from an HSM or get it in any other way, they just have
>> > > to write the appropriate shell script. We just provide mechanism, not
>> > > policy, and the user can adopt any policy they like, from an extremely
>> > > insecure policy to one suitable for Fort Knox.
>> >
>> > Agreed, but as Bruce alluded to, we want this to be a master key, which
>> > is in turn used to encrypt the actual key, or keys, that are used to
>> > encrypt the data. The actual data encryption keys could be very long
>> > randomly generated binary, and there could be more than one of them
>> > (e.g. one per tablespace) in a file which is encrypted with the master
>> > key. This is more secure and allows, for example the master key to be
>> > changed without having to decrypt/re-encrypt the entire database.
>>
>> Yes, thank you. Also, you can make multiple RSA-encrypted copies of the
>> symetric key, one for each role you want to view the data. And good
>> point on the ability to change the RSA key/password without having to
>> reencrypt the data.
>
> There's nothing in this proposal that prevents the user from using a
> very long randomly generated binary key. We aren't talking about
> prompting the user for a password unless that's what they decide the
> shell script should do, unless the user decides to do that and if they
> do then that's their choice.
Except shell escaping issues, etc, etc
> Let us, please, stop stressing over the right way to do key management
> as part of this discussion about providing encryption. The two are
> different things and we do not need to solve both at once.
Not stressing, but this is an important part of the design and should be
done correctly. It is also very simple, so should not be hard to add.
> Further, yes, we will definitely want to get to a point where we can
> encrypt subsets of the system in different ways, but that doesn't have
> to be done in the first implementation either.
No, it doesn't, but that doesn't change the utility of doing it this way
from the start.
Joe
--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:28 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-13 16:55 ` Re: WIP: Data at rest encryption Joe Conway <mail@joeconway.com>
2017-06-13 16:58 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:05 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:09 ` Re: WIP: Data at rest encryption Joe Conway <mail@joeconway.com>
@ 2017-06-13 17:20 ` Stephen Frost <sfrost@snowman.net>
2017-06-13 17:26 ` Re: WIP: Data at rest encryption Joe Conway <mail@joeconway.com>
0 siblings, 1 reply; 93+ messages in thread
From: Stephen Frost @ 2017-06-13 17:20 UTC (permalink / raw)
To: Joe Conway <mail@joeconway.com>; +Cc: Bruce Momjian <bruce@momjian.us>; Robert Haas <robertmhaas@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; pgsql-hackers
Joe,
* Joe Conway (mail@joeconway.com) wrote:
> Except shell escaping issues, etc, etc
That's not an issue- we're talking about reading the stdout of some
other process, there's no shell escaping that has to be done there.
> > Let us, please, stop stressing over the right way to do key management
> > as part of this discussion about providing encryption. The two are
> > different things and we do not need to solve both at once.
>
> Not stressing, but this is an important part of the design and should be
> done correctly. It is also very simple, so should not be hard to add.
I disagree that proper key management is "simple". If we really get to
a point where we think we have a simple answer to it then perhaps that
can be implemented in addition to the encryption piece in the same
release cycle- but they certainly don't need to be in the same patch,
nor do we need to make good key management a requirement for adding
encryption support.
> > Further, yes, we will definitely want to get to a point where we can
> > encrypt subsets of the system in different ways, but that doesn't have
> > to be done in the first implementation either.
>
> No, it doesn't, but that doesn't change the utility of doing it this way
> from the start.
No, but it seriously changes the level of complexity. I feel like we're
trying to go from zero to light speed here because there's an idea that
it's "simple" to add X, Y or Z additional requirement beyond the basic
feature, but we don't have anything yet. I continue to be of the
feeling that we should start simple and keep it to the basic feature
first and make sure that we can actually get that right before we start
looking into adding on additional bits.
Thanks!
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:28 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-13 16:55 ` Re: WIP: Data at rest encryption Joe Conway <mail@joeconway.com>
2017-06-13 16:58 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:05 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:09 ` Re: WIP: Data at rest encryption Joe Conway <mail@joeconway.com>
2017-06-13 17:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
@ 2017-06-13 17:26 ` Joe Conway <mail@joeconway.com>
2017-06-13 17:44 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 93+ messages in thread
From: Joe Conway @ 2017-06-13 17:26 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Bruce Momjian <bruce@momjian.us>; Robert Haas <robertmhaas@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; pgsql-hackers
On 06/13/2017 10:20 AM, Stephen Frost wrote:
> * Joe Conway (mail@joeconway.com) wrote:
>> Except shell escaping issues, etc, etc
>
> That's not an issue- we're talking about reading the stdout of some
> other process, there's no shell escaping that has to be done there.
It could be an issue depending on how the user stores their master key.
> I disagree that proper key management is "simple". If we really get to
> a point where we think we have a simple answer to it then perhaps that
> can be implemented in addition to the encryption piece in the same
> release cycle- but they certainly don't need to be in the same patch,
> nor do we need to make good key management a requirement for adding
> encryption support.
I never said key management was simple. Indeed it is the most complex
and hazardous part of all this as you said earlier. What is simple is
implementing a master key encrypting actual keys scheme. Keeping the
user's master key management out of this design is unchanged by what I
proposed, and what I proposed is a superior yet simple method. Yes, it
can be done separately but what is the point? We should at least discuss
it as part of the design.
> No, but it seriously changes the level of complexity. I feel like we're
> trying to go from zero to light speed here because there's an idea that
> it's "simple" to add X, Y or Z additional requirement beyond the basic
> feature, but we don't have anything yet.
I think that is hyperbole. It does not significantly add to the
complexity of what is being discussed.
Joe
--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:28 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-13 16:55 ` Re: WIP: Data at rest encryption Joe Conway <mail@joeconway.com>
2017-06-13 16:58 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:05 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:09 ` Re: WIP: Data at rest encryption Joe Conway <mail@joeconway.com>
2017-06-13 17:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:26 ` Re: WIP: Data at rest encryption Joe Conway <mail@joeconway.com>
@ 2017-06-13 17:44 ` Stephen Frost <sfrost@snowman.net>
2017-06-13 17:48 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 93+ messages in thread
From: Stephen Frost @ 2017-06-13 17:44 UTC (permalink / raw)
To: Joe Conway <mail@joeconway.com>; +Cc: Bruce Momjian <bruce@momjian.us>; Robert Haas <robertmhaas@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; pgsql-hackers
Joe,
* Joe Conway (mail@joeconway.com) wrote:
> On 06/13/2017 10:20 AM, Stephen Frost wrote:
> > * Joe Conway (mail@joeconway.com) wrote:
> >> Except shell escaping issues, etc, etc
> >
> > That's not an issue- we're talking about reading the stdout of some
> > other process, there's no shell escaping that has to be done there.
>
> It could be an issue depending on how the user stores their master key.
... eh? The user gives us a command to run, we run it, it spits out
some binary blob to stdout which we read in and use as the key. I don't
see where in that there's any need to be concerned about shell escaping
issues.
> > I disagree that proper key management is "simple". If we really get to
> > a point where we think we have a simple answer to it then perhaps that
> > can be implemented in addition to the encryption piece in the same
> > release cycle- but they certainly don't need to be in the same patch,
> > nor do we need to make good key management a requirement for adding
> > encryption support.
>
> I never said key management was simple. Indeed it is the most complex
> and hazardous part of all this as you said earlier. What is simple is
> implementing a master key encrypting actual keys scheme. Keeping the
> user's master key management out of this design is unchanged by what I
> proposed, and what I proposed is a superior yet simple method. Yes, it
> can be done separately but what is the point? We should at least discuss
> it as part of the design.
The point is that we haven't got any encryption of any kind and you're
suggesting we introduce key management which you agree isn't simple.
That you're trying to argue that it actually is simple because it's just
<<description of a key management system>> is a bit bizarre to me.
> > No, but it seriously changes the level of complexity. I feel like we're
> > trying to go from zero to light speed here because there's an idea that
> > it's "simple" to add X, Y or Z additional requirement beyond the basic
> > feature, but we don't have anything yet.
>
> I think that is hyperbole. It does not significantly add to the
> complexity of what is being discussed.
If I stipulate that it's, indeed, simple to implement a system where we
have a master key and other keys- where are those other keys going to
kept (even if they're encrypted)? How many extra keys are we talking
about? When are those keys going to be used and how do we know what key
to use when? If we're going to do per-tablespace or per-table
encryption, how are we going to handle the WAL for that? Will we have
an independent key for WAL (in which case, what's the point of using
different keys for tables, et al, when all the data is in the WAL?)?
Having a single key which is used cluster-wide is extremely simple and
lets us get some form of encryption first before we try to tackle the
more complicated multi-key/partial-encryption system.
Just to be clear, I don't have any issue with discussing the idea that
we want to get to a point where we can work with multiple keys and
encrypt different tables with different keys (or not encrypt certain
tables, et al) with the goal of implementing the single-key approach in
a way that allows us to expand on it down the road easily, I just don't
think we need to have it all done in the very first patch which adds the
ability to encrypt the data files. Maybe you're not saying that it has
to be included in the first implementation, in which case we seem to
just be talking past each other, but that isn't the impression I got..
Thanks!
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:28 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-13 16:55 ` Re: WIP: Data at rest encryption Joe Conway <mail@joeconway.com>
2017-06-13 16:58 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:05 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:09 ` Re: WIP: Data at rest encryption Joe Conway <mail@joeconway.com>
2017-06-13 17:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:26 ` Re: WIP: Data at rest encryption Joe Conway <mail@joeconway.com>
2017-06-13 17:44 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
@ 2017-06-13 17:48 ` Bruce Momjian <bruce@momjian.us>
2017-06-13 18:06 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 93+ messages in thread
From: Bruce Momjian @ 2017-06-13 17:48 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Joe Conway <mail@joeconway.com>; Robert Haas <robertmhaas@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; pgsql-hackers
On Tue, Jun 13, 2017 at 01:44:51PM -0400, Stephen Frost wrote:
> Just to be clear, I don't have any issue with discussing the idea that
> we want to get to a point where we can work with multiple keys and
> encrypt different tables with different keys (or not encrypt certain
> tables, et al) with the goal of implementing the single-key approach in
> a way that allows us to expand on it down the road easily, I just don't
> think we need to have it all done in the very first patch which adds the
> ability to encrypt the data files. Maybe you're not saying that it has
> to be included in the first implementation, in which case we seem to
> just be talking past each other, but that isn't the impression I got..
We don't want to implement all-cluster encryption with a simple user API
and then realize we need another API for later encryption features, do
we? And we are not going to know that if we don't talk about it, but
hey, this is just an email thread and I can marshal opposition to the
feature later when it appears, and point this all out again.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:28 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-13 16:55 ` Re: WIP: Data at rest encryption Joe Conway <mail@joeconway.com>
2017-06-13 16:58 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:05 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:09 ` Re: WIP: Data at rest encryption Joe Conway <mail@joeconway.com>
2017-06-13 17:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:26 ` Re: WIP: Data at rest encryption Joe Conway <mail@joeconway.com>
2017-06-13 17:44 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:48 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
@ 2017-06-13 18:06 ` Stephen Frost <sfrost@snowman.net>
0 siblings, 0 replies; 93+ messages in thread
From: Stephen Frost @ 2017-06-13 18:06 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Joe Conway <mail@joeconway.com>; Robert Haas <robertmhaas@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; pgsql-hackers
Bruce,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Tue, Jun 13, 2017 at 01:44:51PM -0400, Stephen Frost wrote:
> > Just to be clear, I don't have any issue with discussing the idea that
> > we want to get to a point where we can work with multiple keys and
> > encrypt different tables with different keys (or not encrypt certain
> > tables, et al) with the goal of implementing the single-key approach in
> > a way that allows us to expand on it down the road easily, I just don't
> > think we need to have it all done in the very first patch which adds the
> > ability to encrypt the data files. Maybe you're not saying that it has
> > to be included in the first implementation, in which case we seem to
> > just be talking past each other, but that isn't the impression I got..
>
> We don't want to implement all-cluster encryption with a simple user API
> and then realize we need another API for later encryption features, do
> we?
I actually suspect that's exactly where we'll end up- but due to
necessity rather than because there's a way to avoid it. We are going
to want to encrypt cluster-wide components of the system (shared
catalogs, WAL, etc) and that means that we have to have a key provided
very early on. That's a very different thing from being able to do
something like encrypt specific tables, tablespaces, etc, where the key
can be provided much later and we'll want to allow users to use SQL or
the libpq protocol to be able to specify what to encrypt and possibly
even provide the encryption keys.
That said, the approach outlined here could be used for both by
expanding on the command string, perhaps passing it a keyid which is
what we store in the catalog to indicate what key a table is encrypted
with and then the keyid is "%k" in the command string and the command
has to return the specified key for us to decrypt the table. That would
involve adding a new catalog table to identify keys and their keyids,
I'd think, and an additional column in pg_class which specifies the key
(or perhaps we'd just have a new catalog table that says what tables are
encrypted in what way).
Thanks!
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
@ 2017-06-13 16:44 ` Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
1 sibling, 1 reply; 93+ messages in thread
From: Bruce Momjian @ 2017-06-13 16:44 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
On Tue, Jun 13, 2017 at 12:23:01PM -0400, Stephen Frost wrote:
> > As I understand it, having encryption in the database means the key is
> > stored in the database, while having encryption in the file system means
> > the key is stored in the operating system somewhere.
>
> Key management is an entirely independent discussion from this and the
> proposal from Ants, as I understand it, is that the key would *not* be
> in the database but could be anywhere that a shell command could get it
> from, including possibly a HSM (hardware device).
>
> Having the data encrypted by PostgreSQL does not mean the key is stored
> in the database.
Yes, I was just simplifying.
> > Of course, if the
> > key stored in the database is visible to someone using the operating
> > system, we really haven't added much/any security --- I guess my point
> > is that the OS easily can hide the key from the database, but the
> > database can't easily hide the key from the operating system.
>
> This is correct- the key must be available to the PostgreSQL process
> and therefore someone with privileged access to the OS would be able to
> retrieve the key, but that's also true of filesystem encryption.
>
> Basically, if the server is doing the encryption and you have the
> ability to read all memory on the server then you can get the key. Of
> course, if you can read all memory then you can just look at shared
> buffers and you don't really need to bother yourself with the key or
> the encryption, and it doesn't make any difference if you're encrypting
> in the database or in the filesystem. That attack vector is not one
> which this is intending to address.
My point is that if you have the key accessible to the database server,
both the database server and OS have access to it. If you store it in
the OS, only the OS has access to it.
> > I have to admit we tend to avoid heavy-API solutions that are designed
> > just to work around deployment challenges. Commercial databases are
> > fine in doing that, but it leads to very complex products.
>
> I'm not following what you mean here.
By adding all-cluster encryption, we are re-implementing something the
OS does just fine, in most cases. We are going to have API overhead to
do it in the database, and historically we have avoided that.
> > One cool idea I have is using public encryption to store the encryption
> > key by users who don't know the decryption key, e.g. RSA. It would be a
> > write-only encryption option. Not sure how useful that is, but it
> > easily possible, and doesn't require us to keep the _encryption_ key
> > secret, just the decryption one.
>
> The downside here is that asymmetric encryption is much more expensive
> than symmetric encryption and that probably makes it a non-starter. I
> do think we'll want to support multiple encryption methods and perhaps
> we can have an option where asymmetric encryption is used, but that's
> not what I expect will be typically used.
Well, usually the symetric key is stored using RSA and a symetric
cipher is used to encrypt/decrypt the data. I was thinking of a case
where you encrypt a row using a symetric key, then store RSA-encrypted
versions of the symetric key encrypted that only specific users could
decrypt and get the key to decrypt the data.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
@ 2017-06-13 17:01 ` Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 93+ messages in thread
From: Stephen Frost @ 2017-06-13 17:01 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
Bruce,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Tue, Jun 13, 2017 at 12:23:01PM -0400, Stephen Frost wrote:
> > > Of course, if the
> > > key stored in the database is visible to someone using the operating
> > > system, we really haven't added much/any security --- I guess my point
> > > is that the OS easily can hide the key from the database, but the
> > > database can't easily hide the key from the operating system.
> >
> > This is correct- the key must be available to the PostgreSQL process
> > and therefore someone with privileged access to the OS would be able to
> > retrieve the key, but that's also true of filesystem encryption.
> >
> > Basically, if the server is doing the encryption and you have the
> > ability to read all memory on the server then you can get the key. Of
> > course, if you can read all memory then you can just look at shared
> > buffers and you don't really need to bother yourself with the key or
> > the encryption, and it doesn't make any difference if you're encrypting
> > in the database or in the filesystem. That attack vector is not one
> > which this is intending to address.
>
> My point is that if you have the key accessible to the database server,
> both the database server and OS have access to it. If you store it in
> the OS, only the OS has access to it.
I understand that, but that's not a particularly interesting distinction
as the database server must, necessairly, have access to the data in the
database.
> > > I have to admit we tend to avoid heavy-API solutions that are designed
> > > just to work around deployment challenges. Commercial databases are
> > > fine in doing that, but it leads to very complex products.
> >
> > I'm not following what you mean here.
>
> By adding all-cluster encryption, we are re-implementing something the
> OS does just fine, in most cases. We are going to have API overhead to
> do it in the database, and historically we have avoided that.
We do try to avoid the overhead of additional function calls, in places
where it matters. As this is all about reading and writing data, the
overhead for the additional check to see if we're doing encryption or
not is unlikely to be interesting.
> > > One cool idea I have is using public encryption to store the encryption
> > > key by users who don't know the decryption key, e.g. RSA. It would be a
> > > write-only encryption option. Not sure how useful that is, but it
> > > easily possible, and doesn't require us to keep the _encryption_ key
> > > secret, just the decryption one.
> >
> > The downside here is that asymmetric encryption is much more expensive
> > than symmetric encryption and that probably makes it a non-starter. I
> > do think we'll want to support multiple encryption methods and perhaps
> > we can have an option where asymmetric encryption is used, but that's
> > not what I expect will be typically used.
>
> Well, usually the symetric key is stored using RSA and a symetric
> cipher is used to encrypt/decrypt the data. I was thinking of a case
> where you encrypt a row using a symetric key, then store RSA-encrypted
> versions of the symetric key encrypted that only specific users could
> decrypt and get the key to decrypt the data.
This goes back to key management and I agree that it often makes sense
to use RSA or similar to encrypt the symmetric key, and this approach
would allow the user to do so. That doesn't actually give you a
"write-only" encryption option though, since any user who can decrypt
the symmetric key is able to use the symmetric key for both encryption
and decryption, and someone who only has access to the RSA encryption
key can't actually encrypt the data since they can't access the
symmetric key.
Thanks!
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
@ 2017-06-13 17:13 ` Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 93+ messages in thread
From: Bruce Momjian @ 2017-06-13 17:13 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
On Tue, Jun 13, 2017 at 01:01:32PM -0400, Stephen Frost wrote:
> > Well, usually the symetric key is stored using RSA and a symetric
> > cipher is used to encrypt/decrypt the data. I was thinking of a case
> > where you encrypt a row using a symetric key, then store RSA-encrypted
> > versions of the symetric key encrypted that only specific users could
> > decrypt and get the key to decrypt the data.
>
> This goes back to key management and I agree that it often makes sense
> to use RSA or similar to encrypt the symmetric key, and this approach
> would allow the user to do so. That doesn't actually give you a
> "write-only" encryption option though, since any user who can decrypt
> the symmetric key is able to use the symmetric key for both encryption
> and decryption, and someone who only has access to the RSA encryption
> key can't actually encrypt the data since they can't access the
> symmetric key.
I think the big win of Postgres doing the encryption is that the
user-visible file system is no longer a target (assuming OS permissions
are bypassed), while for file system encryption it is the storage device
that is encrypted.
My big question is how many times are the OS permissions bypassed in a
way that would also not expose the db clusters key or db data?
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
@ 2017-06-13 17:25 ` Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 93+ messages in thread
From: Stephen Frost @ 2017-06-13 17:25 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
Bruce,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Tue, Jun 13, 2017 at 01:01:32PM -0400, Stephen Frost wrote:
> > > Well, usually the symetric key is stored using RSA and a symetric
> > > cipher is used to encrypt/decrypt the data. I was thinking of a case
> > > where you encrypt a row using a symetric key, then store RSA-encrypted
> > > versions of the symetric key encrypted that only specific users could
> > > decrypt and get the key to decrypt the data.
> >
> > This goes back to key management and I agree that it often makes sense
> > to use RSA or similar to encrypt the symmetric key, and this approach
> > would allow the user to do so. That doesn't actually give you a
> > "write-only" encryption option though, since any user who can decrypt
> > the symmetric key is able to use the symmetric key for both encryption
> > and decryption, and someone who only has access to the RSA encryption
> > key can't actually encrypt the data since they can't access the
> > symmetric key.
>
> I think the big win of Postgres doing the encryption is that the
> user-visible file system is no longer a target (assuming OS permissions
> are bypassed), while for file system encryption it is the storage device
> that is encrypted.
If OS permissions are bypassed then the encryption isn't going to help
because the attacker can just access shared memory.
The big wins for doing the encryption in PostgreSQL are, as Robert and I
have both mentioned on this thread already, that it provides
data-at-rest encryption in an easier to deploy fashion which will work
the same across different systems and allows the encrypted cluster to be
transferred more easily between systems. There are almsot certainly
other wins from having PG do the encryption, but the above strikes me as
the big ones, and those are certainly valuable enough on their own for
us to seriously consider adding this capability.
> My big question is how many times are the OS permissions bypassed in a
> way that would also not expose the db clusters key or db data?
This is not the attack vector that this solution is attempting to
address, so there really isn't much point in discussing it on this
thread.
Thanks!
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
@ 2017-06-13 17:43 ` Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 93+ messages in thread
From: Bruce Momjian @ 2017-06-13 17:43 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
On Tue, Jun 13, 2017 at 01:25:00PM -0400, Stephen Frost wrote:
> > I think the big win of Postgres doing the encryption is that the
> > user-visible file system is no longer a target (assuming OS permissions
> > are bypassed), while for file system encryption it is the storage device
> > that is encrypted.
>
> If OS permissions are bypassed then the encryption isn't going to help
> because the attacker can just access shared memory.
>
> The big wins for doing the encryption in PostgreSQL are, as Robert and I
> have both mentioned on this thread already, that it provides
> data-at-rest encryption in an easier to deploy fashion which will work
> the same across different systems and allows the encrypted cluster to be
> transferred more easily between systems. There are almsot certainly
> other wins from having PG do the encryption, but the above strikes me as
> the big ones, and those are certainly valuable enough on their own for
> us to seriously consider adding this capability.
Since you seem to be trying to shut down discussion, I will simply say I
am unimpressed that this use-case is sufficient justification to add the
feature.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
@ 2017-06-13 18:23 ` Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 93+ messages in thread
From: Stephen Frost @ 2017-06-13 18:23 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
Bruce,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Tue, Jun 13, 2017 at 01:25:00PM -0400, Stephen Frost wrote:
> > > I think the big win of Postgres doing the encryption is that the
> > > user-visible file system is no longer a target (assuming OS permissions
> > > are bypassed), while for file system encryption it is the storage device
> > > that is encrypted.
> >
> > If OS permissions are bypassed then the encryption isn't going to help
> > because the attacker can just access shared memory.
> >
> > The big wins for doing the encryption in PostgreSQL are, as Robert and I
> > have both mentioned on this thread already, that it provides
> > data-at-rest encryption in an easier to deploy fashion which will work
> > the same across different systems and allows the encrypted cluster to be
> > transferred more easily between systems. There are almsot certainly
> > other wins from having PG do the encryption, but the above strikes me as
> > the big ones, and those are certainly valuable enough on their own for
> > us to seriously consider adding this capability.
>
> Since you seem to be trying to shut down discussion, I will simply say I
> am unimpressed that this use-case is sufficient justification to add the
> feature.
I'm not trying to shut down discussion, I'm simply pointing out where
this feature will be helpful and where it won't be. If there's a way to
make it better and able to address an attack where the OS permission
system is bypassed, that'd be great, but I certainly don't know of any
way to do that and we don't want to claim that this feature will protect
against an attack vector that it won't.
If the lack of that means you don't support the feature, that's
unfortunate as it seems to imply, to me at least, that we'll never have
any kind of encryption because there's no way for it to prevent attacks
where the OS permission system is able to be bypassed.
Thanks!
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
@ 2017-06-13 18:29 ` Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 93+ messages in thread
From: Bruce Momjian @ 2017-06-13 18:29 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
On Tue, Jun 13, 2017 at 02:23:39PM -0400, Stephen Frost wrote:
> I'm not trying to shut down discussion, I'm simply pointing out where
> this feature will be helpful and where it won't be. If there's a way to
> make it better and able to address an attack where the OS permission
> system is bypassed, that'd be great, but I certainly don't know of any
> way to do that and we don't want to claim that this feature will protect
> against an attack vector that it won't.
>
> If the lack of that means you don't support the feature, that's
> unfortunate as it seems to imply, to me at least, that we'll never have
> any kind of encryption because there's no way for it to prevent attacks
> where the OS permission system is able to be bypassed.
It means if we can't discuss the actual benefits that this feature
brings, and doesn't bring, and how it will deal with future feature
additions, then you are right we will never have it.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
@ 2017-06-13 18:38 ` Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 93+ messages in thread
From: Stephen Frost @ 2017-06-13 18:38 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
Bruce,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Tue, Jun 13, 2017 at 02:23:39PM -0400, Stephen Frost wrote:
> > I'm not trying to shut down discussion, I'm simply pointing out where
> > this feature will be helpful and where it won't be. If there's a way to
> > make it better and able to address an attack where the OS permission
> > system is bypassed, that'd be great, but I certainly don't know of any
> > way to do that and we don't want to claim that this feature will protect
> > against an attack vector that it won't.
> >
> > If the lack of that means you don't support the feature, that's
> > unfortunate as it seems to imply, to me at least, that we'll never have
> > any kind of encryption because there's no way for it to prevent attacks
> > where the OS permission system is able to be bypassed.
>
> It means if we can't discuss the actual benefits that this feature
> brings, and doesn't bring, and how it will deal with future feature
> additions, then you are right we will never have it.
I apologize for having come across as trying to shut down discussion,
that was not my intent.
It's good to discuss what the feature would bring and what cases it
doesn't cover, as well as discussing how it can be designed to make sure
that later improvements are able to be done without having to change it
around. I do think it's a good idea for us to consider taking an
incremental approach where we're adding pieces and building things up as
we go. I'm concerned that if we try to do too much in the initial
implementation that we'll end up not having anything.
As it relates to the different attack vectors that this would address,
it's primairly the same ones which filesystem-level encryption also
addresses, but it's an improvement when it comes to ease of use.
Unfortunately, it won't address cases where the OS is compromised.
Thanks!
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
@ 2017-06-13 18:56 ` Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 93+ messages in thread
From: Bruce Momjian @ 2017-06-13 18:56 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
On Tue, Jun 13, 2017 at 02:38:58PM -0400, Stephen Frost wrote:
> It's good to discuss what the feature would bring and what cases it
> doesn't cover, as well as discussing how it can be designed to make sure
> that later improvements are able to be done without having to change it
> around. I do think it's a good idea for us to consider taking an
> incremental approach where we're adding pieces and building things up as
> we go. I'm concerned that if we try to do too much in the initial
> implementation that we'll end up not having anything.
>
> As it relates to the different attack vectors that this would address,
> it's primairly the same ones which filesystem-level encryption also
> addresses, but it's an improvement when it comes to ease of use.
> Unfortunately, it won't address cases where the OS is compromised.
OK, so let's go back. You are saying there are no security benefits to
this vs. file system encryption. The benefit is allowing configuration
in the database rather than the OS? You stated you can transfer
db-level encrypted files between servers, but can't you do that anyway?
Is the problem that you have to encrypt before sending and decrypt on
arrival, if you don't trust the transmission link? Is that used a lot?
Is having the db encrypt every write a reasonable solution to that?
As far as future features, we don't have to add the all features at this
time, but if someone has a good idea for an API and we can make it work
easily while adding this feature, why wouldn't we do that?
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
@ 2017-06-13 19:20 ` Stephen Frost <sfrost@snowman.net>
2017-06-13 19:43 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 19:45 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 19:51 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
0 siblings, 3 replies; 93+ messages in thread
From: Stephen Frost @ 2017-06-13 19:20 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
Bruce,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Tue, Jun 13, 2017 at 02:38:58PM -0400, Stephen Frost wrote:
> > It's good to discuss what the feature would bring and what cases it
> > doesn't cover, as well as discussing how it can be designed to make sure
> > that later improvements are able to be done without having to change it
> > around. I do think it's a good idea for us to consider taking an
> > incremental approach where we're adding pieces and building things up as
> > we go. I'm concerned that if we try to do too much in the initial
> > implementation that we'll end up not having anything.
> >
> > As it relates to the different attack vectors that this would address,
> > it's primairly the same ones which filesystem-level encryption also
> > addresses, but it's an improvement when it comes to ease of use.
> > Unfortunately, it won't address cases where the OS is compromised.
>
> OK, so let's go back. You are saying there are no security benefits to
> this vs. file system encryption.
I'm not sure that I can see any, myself.. Perhaps I'm wrong there, but
it seems unlikely that this would be an improvement over filesystem
level encryption in the general sense. I'm not sure that I see it as
really worse than filesystem-level encryption either, to be clear.
There's a bit of increased information leakage, as Peter mentioned and I
agreed with, but it's not a lot and I expect in most cases that
information leak would be acceptable. That seems like something which
would need to be considered on a case-by-case basis.
> The benefit is allowing configuration
> in the database rather than the OS?
No, the benefit is that the database administrator can configure it and
set it up and not have to get an OS-level administrator involved. There
may also be other reasons why filesystem-level encryption is difficult
to set up or use in a certain environment, but this wouldn't depend on
anything OS-related and therefore could be done.
> You stated you can transfer
> db-level encrypted files between servers, but can't you do that anyway?
If the filesystem is encrypted and you wanted to transfer the entire
cluster from one system to another, keeping it encrypted with the same
key, you'd have to transfer the entire filesystem at a block level.
That's not typically very easy to do (ZFS, specifically, has this
capability where you can export a filesystem and send it from one
machine to another, but I don't know of any other filesystems which do
and ZFS isn't always an option..).
You could go through a process of re-encrypting the files prior to
transferring them, or deciding that simply having the transport
mechanism encrypted is sufficient (eg: SSH), but if what you really want
to do is keep the existing encryption of the database and transfer it to
another system, this allows that pretty easily.
For example, you could simply do:
cp -a /path/to/PG /mnt/usb
and you're done. If you're using filesystem level encryption then you'd
have to re-encrypt the data, using something like:
tar -cf - /path/to/PG | openssl -key private.key > /mnt/usb/encrypted_cluster.tar
And then you would need openssl on the other system to decrypt it.
Of course, either way you'd have to provide for a way to get the key
from one system to the other.
Also, tools like pg_basebackup could be used, with nearly zero changes,
I think, to get an encrypted copy of the database for backup purposes,
removing the need to work out a way to handle encrypting backups.
> Is the problem that you have to encrypt before sending and decrypt on
> arrival, if you don't trust the transmission link? Is that used a lot?
> Is having the db encrypt every write a reasonable solution to that?
There's multiple use-cases here. Making it easier to copy the database
is just one of them and it isn't the biggest one. The biggest benefit
is that there's cases where filesystem-level encryption isn't really an
option or, even if it is, it's not desirable for other reasons.
> As far as future features, we don't have to add the all features at this
> time, but if someone has a good idea for an API and we can make it work
> easily while adding this feature, why wouldn't we do that?
Sure, I'm all for specific suggestions about how to improve the API, or
just in general recommendations of how to improve the patch. The
suggestions which have been made about key management don't really come
across to me as specific API-level recommendations but rather "this
would also be nice to have" kind of comments, which isn't really the
same.
Thanks!
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
@ 2017-06-13 19:43 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 22:11 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2 siblings, 1 reply; 93+ messages in thread
From: Peter Eisentraut @ 2017-06-13 19:43 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; Bruce Momjian <bruce@momjian.us>; +Cc: Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
On 6/13/17 15:20, Stephen Frost wrote:
> No, the benefit is that the database administrator can configure it and
> set it up and not have to get an OS-level administrator involved. There
> may also be other reasons why filesystem-level encryption is difficult
> to set up or use in a certain environment, but this wouldn't depend on
> anything OS-related and therefore could be done.
Let's see a proposal in those terms then. How easy can you make it,
compared to existing OS-level solutions, and will that justify the
maintenance overhead?
Considering how ubiquitous file-system encryption is, I have my doubts
that the trade-offs will come out right, but let's see.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:43 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
@ 2017-06-13 22:11 ` Stephen Frost <sfrost@snowman.net>
2017-06-14 21:36 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
0 siblings, 1 reply; 93+ messages in thread
From: Stephen Frost @ 2017-06-13 22:11 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; +Cc: Bruce Momjian <bruce@momjian.us>; Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
Peter,
* Peter Eisentraut (peter.eisentraut@2ndquadrant.com) wrote:
> On 6/13/17 15:20, Stephen Frost wrote:
> > No, the benefit is that the database administrator can configure it and
> > set it up and not have to get an OS-level administrator involved. There
> > may also be other reasons why filesystem-level encryption is difficult
> > to set up or use in a certain environment, but this wouldn't depend on
> > anything OS-related and therefore could be done.
>
> Let's see a proposal in those terms then. How easy can you make it,
> compared to existing OS-level solutions, and will that justify the
> maintenance overhead?
From the original post on this thread, which included a WIP patch:
----------------------------------
Usage
=====
Set up database like so:
(read -sp "Postgres passphrase: " PGENCRYPTIONKEY; echo;
export PGENCRYPTIONKEY
initdb -k -K pgcrypto $PGDATA )
Start PostgreSQL:
(read -sp "Postgres passphrase: " PGENCRYPTIONKEY; echo;
export PGENCRYPTIONKEY
postgres $PGDATA )
----------------------------------
That certainly seems very straight-forward to me, though I expect that
packagers would probably improve upon it further.
> Considering how ubiquitous file-system encryption is, I have my doubts
> that the trade-offs will come out right, but let's see.
There's definitely environments out there where DBAs aren't able to have
root privileges and that limits what they're able to do. I'm not really
sure how to objectively weigh "you don't need to be root to encrypt the
database" vs. maintenance overhead of this feature. Subjectively, for
my 2c anyway, it seems well worth it, but that's naturally subjective.
:)
Thanks!
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:43 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 22:11 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
@ 2017-06-14 21:36 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 21:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 93+ messages in thread
From: Peter Eisentraut @ 2017-06-14 21:36 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Bruce Momjian <bruce@momjian.us>; Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
On 6/13/17 18:11, Stephen Frost wrote:
>> Let's see a proposal in those terms then. How easy can you make it,
>> compared to existing OS-level solutions, and will that justify the
>> maintenance overhead?
> From the original post on this thread, which included a WIP patch:
>
> ----------------------------------
> Usage
> =====
>
> Set up database like so:
>
> (read -sp "Postgres passphrase: " PGENCRYPTIONKEY; echo;
> export PGENCRYPTIONKEY
> initdb -k -K pgcrypto $PGDATA )
>
> Start PostgreSQL:
>
> (read -sp "Postgres passphrase: " PGENCRYPTIONKEY; echo;
> export PGENCRYPTIONKEY
> postgres $PGDATA )
> ----------------------------------
Relying on environment variables is clearly pretty crappy. So if that's
the proposal, then I think it needs to be better.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:43 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 22:11 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-14 21:36 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
@ 2017-06-14 21:41 ` Stephen Frost <sfrost@snowman.net>
2017-06-15 15:53 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 16:06 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
0 siblings, 2 replies; 93+ messages in thread
From: Stephen Frost @ 2017-06-14 21:41 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; +Cc: Bruce Momjian <bruce@momjian.us>; Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
Peter,
* Peter Eisentraut (peter.eisentraut@2ndquadrant.com) wrote:
> On 6/13/17 18:11, Stephen Frost wrote:
> >> Let's see a proposal in those terms then. How easy can you make it,
> >> compared to existing OS-level solutions, and will that justify the
> >> maintenance overhead?
> > From the original post on this thread, which included a WIP patch:
> >
> > ----------------------------------
> > Usage
> > =====
> >
> > Set up database like so:
> >
> > (read -sp "Postgres passphrase: " PGENCRYPTIONKEY; echo;
> > export PGENCRYPTIONKEY
> > initdb -k -K pgcrypto $PGDATA )
> >
> > Start PostgreSQL:
> >
> > (read -sp "Postgres passphrase: " PGENCRYPTIONKEY; echo;
> > export PGENCRYPTIONKEY
> > postgres $PGDATA )
> > ----------------------------------
>
> Relying on environment variables is clearly pretty crappy. So if that's
> the proposal, then I think it needs to be better.
I don't believe that was ever intended to be the final solution, I was
just pointing out that it's what the WIP patch did.
The discussion had moved into having a command called which provided the
key on stdout, as I recall, allowing it to be whatever the user wished,
including binary of any kind.
If you have other suggestions, I'm sure they would be well received. As
to the question of complexity, it certainly looks like it'll probably be
quite straight-forward for users to use.
Thanks!
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:43 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 22:11 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-14 21:36 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 21:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
@ 2017-06-15 15:53 ` Robert Haas <robertmhaas@gmail.com>
1 sibling, 0 replies; 93+ messages in thread
From: Robert Haas @ 2017-06-15 15:53 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Bruce Momjian <bruce@momjian.us>; Ants Aasma <ants.aasma@eesti.ee>; pgsql-hackers
On Wed, Jun 14, 2017 at 5:41 PM, Stephen Frost <sfrost@snowman.net> wrote:
> I don't believe that was ever intended to be the final solution, I was
> just pointing out that it's what the WIP patch did.
>
> The discussion had moved into having a command called which provided the
> key on stdout, as I recall, allowing it to be whatever the user wished,
> including binary of any kind.
>
> If you have other suggestions, I'm sure they would be well received. As
> to the question of complexity, it certainly looks like it'll probably be
> quite straight-forward for users to use.
To me, this reads a bit like you're still trying to shut down the
discussion here. Perhaps I am misreading it.
Upthread, you basically said that we shouldn't talk about key
management (specifically, you said, "Key management is an entirely
independent discussion from this") which I think is a ridiculous
statement. We have to have some kind of simple key management in
order to have the feature at all. It does not have to be crazy
complicated, but it has to exist.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:43 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 22:11 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-14 21:36 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 21:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
@ 2017-06-15 16:06 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-15 19:09 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
1 sibling, 1 reply; 93+ messages in thread
From: Peter Eisentraut @ 2017-06-15 16:06 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Bruce Momjian <bruce@momjian.us>; Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
On 6/14/17 17:41, Stephen Frost wrote:
>> Relying on environment variables is clearly pretty crappy. So if that's
>> the proposal, then I think it needs to be better.
> I don't believe that was ever intended to be the final solution, I was
> just pointing out that it's what the WIP patch did.
>
> The discussion had moved into having a command called which provided the
> key on stdout, as I recall, allowing it to be whatever the user wished,
> including binary of any kind.
>
> If you have other suggestions, I'm sure they would be well received. As
> to the question of complexity, it certainly looks like it'll probably be
> quite straight-forward for users to use.
I think the passphrase entry part of the problem is actually a bit
harder than it appears.
Making this work well would be a major part of the usability story that
this is being sold on. If the proposed solution is that you can cobble
together a few bits of shell, then not only is that not very
user-friendly, it also won't work consistently across platforms, won't
work under systemd (launchd? Windows service?), and might behave
awkwardly under restricted environments where there is no terminal or
only a limited OS environment. Moreover, it leaves the security aspects
of that part of the solution (keys lingering in memory or in swap) up to
the user.
There was a discussion a while ago about how to handle passphrase entry
for SSL keys. The conclusion was that it works pretty crappily right
now, and several suggestions for improvement were discussed. I suggest
that fixing that properly and with flexibility could also yield a
solution for encryption key entry.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:43 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 22:11 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-14 21:36 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 21:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 16:06 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
@ 2017-06-15 19:09 ` Robert Haas <robertmhaas@gmail.com>
2017-06-15 20:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 93+ messages in thread
From: Robert Haas @ 2017-06-15 19:09 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; +Cc: Stephen Frost <sfrost@snowman.net>; Bruce Momjian <bruce@momjian.us>; Ants Aasma <ants.aasma@eesti.ee>; pgsql-hackers
On Thu, Jun 15, 2017 at 12:06 PM, Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:
> Making this work well would be a major part of the usability story that
> this is being sold on. If the proposed solution is that you can cobble
> together a few bits of shell, then not only is that not very
> user-friendly, it also won't work consistently across platforms, won't
> work under systemd (launchd? Windows service?), and might behave
> awkwardly under restricted environments where there is no terminal or
> only a limited OS environment. Moreover, it leaves the security aspects
> of that part of the solution (keys lingering in memory or in swap) up to
> the user.
>
> There was a discussion a while ago about how to handle passphrase entry
> for SSL keys. The conclusion was that it works pretty crappily right
> now, and several suggestions for improvement were discussed. I suggest
> that fixing that properly and with flexibility could also yield a
> solution for encryption key entry.
That sounds like a good idea to me.
However, I'd like to disagree with the idea that key management is the
primary way in which this feature would improve usability. To me, the
big advantage is that you don't need to be root (also, we can have
more consistency in behavior across operating systems). I disagree
vigorously with the idea that anyone who wants to encrypt their
PostgreSQL database should just get root privileges on the system and
use an encrypted filesystem. In some environments, "just get root
privileges" is not something that is very easy to do; but even if you
*have* root privileges, you don't necessarily want to have to use them
just to install and configure your database.
Right now, I can compile and install PostgreSQL from my own user
account, run initdb, and start it up. Surely everyone PostgreSQL
developer in the world - and a good number of users - would agree that
if I suddenly needed to run initdb as root, that would be a huge
usability regression. You wouldn't even be able to run 'make check'
without root privileges, which would suck. In the same way, when we
removed (for most users) the need to tune System V shared memory
parameters (b0fc0df9364d2d2d17c0162cf3b8b59f6cb09f67), a lot of users
were very happy precisely because it eliminated a setup step that
formerly had to be done as root. It is very reasonable to suppose
that users who need encryption will similarly be happy if they no
longer need to be root to get an encrypted PostgreSQL working. Sure,
that's only a subset of users rather than all of them, but it's the
same kind of issue.
Also, I don't think we should be presenting filesystem encryption and
built-in encryption as if they were two methods of solving the exact
same problem. In some scenarios, they are in fact solving the same
problem. However, it's entirely reasonable to want to use both. For
example, the hard disk in my laptop is encrypted, because that's a
thing Apple does. If somebody steals the hard disk out of my laptop,
they may have some difficulty recovering the contents. However, as
Stephen also mentioned, that has not deterred me from putting a
passphrase on my SSH keys. There are situations in which the
passphrase provides protection that the whole-drive encryption won't.
For example, if I copy my home directory onto a USB stick and then
copy it from there to a new laptop, somebody might steal the USB
stick. If they manage to do that, they will get most of my files, but
they won't get my SSH keys, or at least not without guessing my
passphrase. Similarly, if I foolishly walk away from my laptop in the
presence of some nefarious person (say, Magnus) without locking it,
that person can't steal my keys. That person might be able to
impersonate me for as long as I'm away from the laptop, if the keys
are loaded into my SSH agent, but not afterwards. The issues for the
database are similar. You might want one of these things or the other
or both depending on the exact situation.
To be honest, I find the hostility toward this feature a bit baffling.
The argument seems to be essentially that we shouldn't have this
feature because we'd have to maintain the code and many of the same
goals could be accomplished by using facilities that already exist
outside the database server. But that's also true for parallel query
(cf. Stado), logical replication (cf. Slony, Bucardo, Londiste),
physical replication (cf. DRBD), partitioning (cf. pg_partman), RLS
(cf. veil), and anything that could be written as application logic
(eg. psql's \if ... \endif, every procedural language we have,
user-defined functions themselves, database-enforced constraints,
FDWs). Yet, in every one of those cases, we find it worthwhile to
have the feature because it works better and is easier to use when
it's built in. I don't think that a patch for this feature is likely
to be bigger than (or even as large as) the patches for logical
replication or parallel query, and it will probably be less work to
maintain going forward than either.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:43 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 22:11 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-14 21:36 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 21:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 16:06 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-15 19:09 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
@ 2017-06-15 20:29 ` Bruce Momjian <bruce@momjian.us>
2017-06-15 21:04 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
0 siblings, 1 reply; 93+ messages in thread
From: Bruce Momjian @ 2017-06-15 20:29 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Stephen Frost <sfrost@snowman.net>; Ants Aasma <ants.aasma@eesti.ee>; pgsql-hackers
On Thu, Jun 15, 2017 at 03:09:32PM -0400, Robert Haas wrote:
> To be honest, I find the hostility toward this feature a bit baffling.
> The argument seems to be essentially that we shouldn't have this
> feature because we'd have to maintain the code and many of the same
> goals could be accomplished by using facilities that already exist
> outside the database server. But that's also true for parallel query
> (cf. Stado), logical replication (cf. Slony, Bucardo, Londiste),
> physical replication (cf. DRBD), partitioning (cf. pg_partman), RLS
> (cf. veil), and anything that could be written as application logic
> (eg. psql's \if ... \endif, every procedural language we have,
> user-defined functions themselves, database-enforced constraints,
> FDWs). Yet, in every one of those cases, we find it worthwhile to
> have the feature because it works better and is easier to use when
> it's built in. I don't think that a patch for this feature is likely
> to be bigger than (or even as large as) the patches for logical
> replication or parallel query, and it will probably be less work to
> maintain going forward than either.
I think the big win for having OS features in the database is
selectivity --- the ability to selectively apply a feature to part of
the database. This is what you are doing by putting a password on your
SSH key, and my idea about row encryption.
It is also a question of convenience. If SSH told users they have to
create an encrypted volume to store their SSH keys with a password, it
would be silly, since the files are so small compared to a file system.
I think the assumption is that any security-concerned deployment of
Postgres will already have Postgres on its own partition and have the
root administrator involved. I think it is this assumption that drives
the idea that requiring root to run Postgres doesn't make sense, but it
does to do encryption. Also, there is the sense that security requires
trust of the root user, while using Postgres doesn't require the root
user to also use Postgres.
One serious difference between in-database-encryption and SSH keys is
that the use of passwords for SSH is well understood and reasonable to
use, while I think we all admit that use of passwords for database
objects like SSL keys is murky. Use of keys for OS-level encryption is
a little better handled, but not as clean as SSH keys.
I admit there is no hard line here, so I guess we will have to see what
the final patch looks like. I am basing my statements on what I guess
the complexity will be. Complexity has a cost so we will have to weigh
it when we see it. When SSH added password access, it was probably an
easy decision because the use-case was high and the complexity was low.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:43 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 22:11 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-14 21:36 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 21:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 16:06 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-15 19:09 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 20:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
@ 2017-06-15 21:04 ` Robert Haas <robertmhaas@gmail.com>
2017-06-15 21:12 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 93+ messages in thread
From: Robert Haas @ 2017-06-15 21:04 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Stephen Frost <sfrost@snowman.net>; Ants Aasma <ants.aasma@eesti.ee>; pgsql-hackers
On Thu, Jun 15, 2017 at 4:29 PM, Bruce Momjian <bruce@momjian.us> wrote:
> I think the big win for having OS features in the database is
> selectivity --- the ability to selectively apply a feature to part of
> the database. This is what you are doing by putting a password on your
> SSH key, and my idea about row encryption.
I agree. I think we will eventually want to be able to apply
encryption selectively, although I don't think we need to have that
feature in a first patch. One problem is that if you don't encrypt
the WAL, there's not much point in encrypting the table data, so it
becomes tricky to encrypt some things and not others. However, I am
sure we can eventually solve those problems, given enough time and
development effort.
> It is also a question of convenience. If SSH told users they have to
> create an encrypted volume to store their SSH keys with a password, it
> would be silly, since the files are so small compared to a file system.
> I think the assumption is that any security-concerned deployment of
> Postgres will already have Postgres on its own partition and have the
> root administrator involved. I think it is this assumption that drives
> the idea that requiring root to run Postgres doesn't make sense, but it
> does to do encryption.
I don't think that's a particularly good assumption, though,
especially with the proliferation of virtual and containerized
environments where access to root privileges tends to be more
circumscribed. Also, there are a lot of small databases out there
that you might want to be able to encrypt without encrypting
everything on the filesystem. For example, there are products that
embed PostgreSQL. If a particular PostgreSQL configuration requires
root access, then using that configuration means that the installing
the product which contains it also requires root access. Installing
the product means changing /etc/fstab, and uninstalling it means
reversing those changes. That's very awkward. I agree that if you've
got a terabyte of sensitive data, you probably want to encrypt the
filesystem and involve the DBA, but there are still people who have a
gigabyte of sensitive data. For those people, a separate filesystem
likely doesn't make sense, but they may still want encryption.
> Also, there is the sense that security requires
> trust of the root user, while using Postgres doesn't require the root
> user to also use Postgres.
I don't understand this. It is certainly true that you're running
binaries owned by root, the root user could Trojan the binaries and
break any security you think you have. But that problem is no better
or worse for PostgreSQL than anything else.
> One serious difference between in-database-encryption and SSH keys is
> that the use of passwords for SSH is well understood and reasonable to
> use, while I think we all admit that use of passwords for database
> objects like SSL keys is murky. Use of keys for OS-level encryption is
> a little better handled, but not as clean as SSH keys.
Peter pointed out upthread that our handling of SSL passphrases leaves
a lot to be desired, and that maybe we should fix that problem first;
I agree. But I don't think this is any kind of intrinsic limitation
of PostgreSQL vs. encrypted filesystems vs. SSH; it's just a
quality-of-implementation issue.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:43 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 22:11 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-14 21:36 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 21:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 16:06 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-15 19:09 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 20:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 21:04 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
@ 2017-06-15 21:12 ` Bruce Momjian <bruce@momjian.us>
2017-06-15 22:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 93+ messages in thread
From: Bruce Momjian @ 2017-06-15 21:12 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Stephen Frost <sfrost@snowman.net>; Ants Aasma <ants.aasma@eesti.ee>; pgsql-hackers
On Thu, Jun 15, 2017 at 05:04:17PM -0400, Robert Haas wrote:
> > Also, there is the sense that security requires
> > trust of the root user, while using Postgres doesn't require the root
> > user to also use Postgres.
>
> I don't understand this. It is certainly true that you're running
> binaries owned by root, the root user could Trojan the binaries and
> break any security you think you have. But that problem is no better
> or worse for PostgreSQL than anything else.
I couldn't find a cleaner way to see it --- it is that database use
doesn't involve the root user using it, while database security requires
the root user to also be security-conscious.
> > One serious difference between in-database-encryption and SSH keys is
> > that the use of passwords for SSH is well understood and reasonable to
> > use, while I think we all admit that use of passwords for database
> > objects like SSL keys is murky. Use of keys for OS-level encryption is
> > a little better handled, but not as clean as SSH keys.
>
> Peter pointed out upthread that our handling of SSL passphrases leaves
> a lot to be desired, and that maybe we should fix that problem first;
> I agree. But I don't think this is any kind of intrinsic limitation
> of PostgreSQL vs. encrypted filesystems vs. SSH; it's just a
> quality-of-implementation issue.
I think there are environmental issues that make password use on SSH
easier than the other cases --- it isn't just code quality. However, it
would be good to research how SSH handles it to see if we can get any
ideas.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:43 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 22:11 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-14 21:36 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 21:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 16:06 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-15 19:09 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 20:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 21:04 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 21:12 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
@ 2017-06-15 22:41 ` Stephen Frost <sfrost@snowman.net>
2017-06-15 23:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 93+ messages in thread
From: Stephen Frost @ 2017-06-15 22:41 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Robert Haas <robertmhaas@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; pgsql-hackers
Bruce,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Thu, Jun 15, 2017 at 05:04:17PM -0400, Robert Haas wrote:
> > > Also, there is the sense that security requires
> > > trust of the root user, while using Postgres doesn't require the root
> > > user to also use Postgres.
> >
> > I don't understand this. It is certainly true that you're running
> > binaries owned by root, the root user could Trojan the binaries and
> > break any security you think you have. But that problem is no better
> > or worse for PostgreSQL than anything else.
>
> I couldn't find a cleaner way to see it --- it is that database use
> doesn't involve the root user using it, while database security requires
> the root user to also be security-conscious.
I tend to agree with Robert here (in general). Further, there are
certainly environments where the administrator with root access is
absolutely security conscious, but that doesn't mean that the DBA has
easy access to the root account or to folks who have that level of
access and it's much easier for the DBA if they're able to address all
of their requirements by building PG themselves, installing it, and,
ideally, encrypting the DB.
> > > One serious difference between in-database-encryption and SSH keys is
> > > that the use of passwords for SSH is well understood and reasonable to
> > > use, while I think we all admit that use of passwords for database
> > > objects like SSL keys is murky. Use of keys for OS-level encryption is
> > > a little better handled, but not as clean as SSH keys.
> >
> > Peter pointed out upthread that our handling of SSL passphrases leaves
> > a lot to be desired, and that maybe we should fix that problem first;
> > I agree. But I don't think this is any kind of intrinsic limitation
> > of PostgreSQL vs. encrypted filesystems vs. SSH; it's just a
> > quality-of-implementation issue.
I'm not thrilled with asking Ants to implement a solution to SSL
passphrases, and generalizing it to work for this, to get this feature
accepted. I assume that the reason for asking for that work to be done
now is because we decided that the current approach for SSL sucks but we
couldn't actually drop support for it, but we don't want to add other
features which work in a similar way because, well, it sucks.
I get that. I'm not thrilled with it, but I get it. I'm hopeful it
ends up not being too bad, but if it ends up meaning we don't get this
feature, then I'll reconsider my position about agreeing that it's an
acceptable requirement.
> I think there are environmental issues that make password use on SSH
> easier than the other cases --- it isn't just code quality. However, it
> would be good to research how SSH handles it to see if we can get any
> ideas.
Actually, the approach SSH uses is a really good one, imv, and one which
we might be able to leverage.. I'm not sure though. I will say that,
in general, I like the idea of leveraging the external libraries which
handle keys and deal with encryption to make this happen as those allow
things like hardware devices to hold the key and possibly perform the
encryption/decryption, etc.
Thanks!
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:43 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 22:11 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-14 21:36 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 21:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 16:06 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-15 19:09 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 20:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 21:04 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 21:12 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 22:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
@ 2017-06-15 23:13 ` Bruce Momjian <bruce@momjian.us>
2017-06-15 23:27 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 93+ messages in thread
From: Bruce Momjian @ 2017-06-15 23:13 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Robert Haas <robertmhaas@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; pgsql-hackers
On Thu, Jun 15, 2017 at 06:41:08PM -0400, Stephen Frost wrote:
> > > > One serious difference between in-database-encryption and SSH keys is
> > > > that the use of passwords for SSH is well understood and reasonable to
> > > > use, while I think we all admit that use of passwords for database
> > > > objects like SSL keys is murky. Use of keys for OS-level encryption is
> > > > a little better handled, but not as clean as SSH keys.
> > >
> > > Peter pointed out upthread that our handling of SSL passphrases leaves
> > > a lot to be desired, and that maybe we should fix that problem first;
> > > I agree. But I don't think this is any kind of intrinsic limitation
> > > of PostgreSQL vs. encrypted filesystems vs. SSH; it's just a
> > > quality-of-implementation issue.
>
> I'm not thrilled with asking Ants to implement a solution to SSL
> passphrases, and generalizing it to work for this, to get this feature
> accepted. I assume that the reason for asking for that work to be done
> now is because we decided that the current approach for SSL sucks but we
> couldn't actually drop support for it, but we don't want to add other
> features which work in a similar way because, well, it sucks.
My point is that if our support for db-level encryption is as bad as SSL
key passwords, then it will be nearly useless, so we might as well not
have it. Isn't that obvious?
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:43 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 22:11 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-14 21:36 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 21:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 16:06 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-15 19:09 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 20:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 21:04 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 21:12 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 22:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 23:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
@ 2017-06-15 23:27 ` Stephen Frost <sfrost@snowman.net>
2017-06-15 23:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 93+ messages in thread
From: Stephen Frost @ 2017-06-15 23:27 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Robert Haas <robertmhaas@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; pgsql-hackers
Bruce,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Thu, Jun 15, 2017 at 06:41:08PM -0400, Stephen Frost wrote:
> > > > > One serious difference between in-database-encryption and SSH keys is
> > > > > that the use of passwords for SSH is well understood and reasonable to
> > > > > use, while I think we all admit that use of passwords for database
> > > > > objects like SSL keys is murky. Use of keys for OS-level encryption is
> > > > > a little better handled, but not as clean as SSH keys.
> > > >
> > > > Peter pointed out upthread that our handling of SSL passphrases leaves
> > > > a lot to be desired, and that maybe we should fix that problem first;
> > > > I agree. But I don't think this is any kind of intrinsic limitation
> > > > of PostgreSQL vs. encrypted filesystems vs. SSH; it's just a
> > > > quality-of-implementation issue.
> >
> > I'm not thrilled with asking Ants to implement a solution to SSL
> > passphrases, and generalizing it to work for this, to get this feature
> > accepted. I assume that the reason for asking for that work to be done
> > now is because we decided that the current approach for SSL sucks but we
> > couldn't actually drop support for it, but we don't want to add other
> > features which work in a similar way because, well, it sucks.
>
> My point is that if our support for db-level encryption is as bad as SSL
> key passwords, then it will be nearly useless, so we might as well not
> have it. Isn't that obvious?
Well, no, because the reason we even have an approach at all for SSL key
passwords is because multiple people (myself and Magnus, at least, as I
recall) complained as we are aware of installations which are actively
using that approach.
That doesn't mean it's a great solution or that it doesn't suck- really,
I tend to agree that it does, but it's necessary because we need a
solution, it works, and users are using it. Having a better solution
would be great and something agent-based might be the right answer (or
perhaps something where we have support for using hardware accellerators
through an existing library...).
I expect the same would happen with the shell-command approach suggested
up-thread and the prompt-on-stdin approach too, they aren't great but I
expect users would still use the feature. As Robert and I have
mentioned, there is a good bit of value to having this feature simply
because it avoids the need to get someone with root privileges to set up
an encrypted volume and I don't think having to use a shell command or
providing the password on stdin at startup really changes that very
much.
Thanks!
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:43 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 22:11 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-14 21:36 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 21:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 16:06 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-15 19:09 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 20:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 21:04 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 21:12 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 22:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 23:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 23:27 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
@ 2017-06-15 23:44 ` Bruce Momjian <bruce@momjian.us>
2017-06-15 23:51 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 23:51 ` Re: WIP: Data at rest encryption Alvaro Herrera <alvherre@2ndquadrant.com>
2017-06-15 23:56 ` Re: WIP: Data at rest encryption Andres Freund <andres@anarazel.de>
0 siblings, 3 replies; 93+ messages in thread
From: Bruce Momjian @ 2017-06-15 23:44 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Robert Haas <robertmhaas@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; pgsql-hackers
On Thu, Jun 15, 2017 at 07:27:55PM -0400, Stephen Frost wrote:
> I expect the same would happen with the shell-command approach suggested
> up-thread and the prompt-on-stdin approach too, they aren't great but I
> expect users would still use the feature. As Robert and I have
> mentioned, there is a good bit of value to having this feature simply
> because it avoids the need to get someone with root privileges to set up
> an encrypted volume and I don't think having to use a shell command or
> providing the password on stdin at startup really changes that very
> much.
Understood, but now you are promoting a feature with an admittedly-poor
API, duplication of an OS feature, and perhaps an invasive change to the
code. Those are high hurdles.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:43 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 22:11 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-14 21:36 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 21:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 16:06 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-15 19:09 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 20:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 21:04 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 21:12 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 22:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 23:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 23:27 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 23:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
@ 2017-06-15 23:51 ` Stephen Frost <sfrost@snowman.net>
2 siblings, 0 replies; 93+ messages in thread
From: Stephen Frost @ 2017-06-15 23:51 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Robert Haas <robertmhaas@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; pgsql-hackers
Bruce,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Thu, Jun 15, 2017 at 07:27:55PM -0400, Stephen Frost wrote:
> > I expect the same would happen with the shell-command approach suggested
> > up-thread and the prompt-on-stdin approach too, they aren't great but I
> > expect users would still use the feature. As Robert and I have
> > mentioned, there is a good bit of value to having this feature simply
> > because it avoids the need to get someone with root privileges to set up
> > an encrypted volume and I don't think having to use a shell command or
> > providing the password on stdin at startup really changes that very
> > much.
>
> Understood, but now you are promoting a feature with an admittedly-poor
> API, duplication of an OS feature, and perhaps an invasive change to the
> code. Those are high hurdles.
Of those, the only one that worries me, at least, is that it might be an
invasive and difficult to maintain code change. As Robert said, and I
agree with, "duplication of an OS feature" is something we pretty
routinly, and justifiably, do. The poor interface is unfortunate, but
if it's consistent with what we have today for a similar feature then
I'm really not too upset with it. If we can do better, great, I'm all
for that, but if not, then I'd rather have the feature with the poor
interface than not have it at all.
If it's an invasive code change or one which ends up being difficult to
maintain, then that's a problem. Getting some focus on that aspect
would be great and I certainly appreciate Robert's initial review and
commentary on it.
Thanks!
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:43 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 22:11 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-14 21:36 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 21:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 16:06 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-15 19:09 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 20:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 21:04 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 21:12 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 22:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 23:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 23:27 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 23:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
@ 2017-06-15 23:51 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2017-06-15 23:55 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-19 16:30 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2 siblings, 2 replies; 93+ messages in thread
From: Alvaro Herrera @ 2017-06-15 23:51 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Stephen Frost <sfrost@snowman.net>; Robert Haas <robertmhaas@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; pgsql-hackers
Bruce Momjian wrote:
> On Thu, Jun 15, 2017 at 07:27:55PM -0400, Stephen Frost wrote:
> > I expect the same would happen with the shell-command approach suggested
> > up-thread and the prompt-on-stdin approach too, they aren't great but I
> > expect users would still use the feature. As Robert and I have
> > mentioned, there is a good bit of value to having this feature simply
> > because it avoids the need to get someone with root privileges to set up
> > an encrypted volume and I don't think having to use a shell command or
> > providing the password on stdin at startup really changes that very
> > much.
>
> Understood, but now you are promoting a feature with an admittedly-poor
> API, duplication of an OS feature, and perhaps an invasive change to the
> code. Those are high hurdles.
I thought we called it "incremental development". From the opposite
point of view, would you say we should ban use of passphrase-protected
SSL key files because the current user interface for them is bad?
I have no use for data-at-rest encryption myself, but I wouldn't stop
development just because the initial design proposal doesn't include
top-notch key management.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:43 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 22:11 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-14 21:36 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 21:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 16:06 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-15 19:09 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 20:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 21:04 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 21:12 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 22:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 23:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 23:27 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 23:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 23:51 ` Re: WIP: Data at rest encryption Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2017-06-15 23:55 ` Bruce Momjian <bruce@momjian.us>
1 sibling, 0 replies; 93+ messages in thread
From: Bruce Momjian @ 2017-06-15 23:55 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Stephen Frost <sfrost@snowman.net>; Robert Haas <robertmhaas@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; pgsql-hackers
On Thu, Jun 15, 2017 at 07:51:36PM -0400, Alvaro Herrera wrote:
> Bruce Momjian wrote:
> > On Thu, Jun 15, 2017 at 07:27:55PM -0400, Stephen Frost wrote:
> > > I expect the same would happen with the shell-command approach suggested
> > > up-thread and the prompt-on-stdin approach too, they aren't great but I
> > > expect users would still use the feature. As Robert and I have
> > > mentioned, there is a good bit of value to having this feature simply
> > > because it avoids the need to get someone with root privileges to set up
> > > an encrypted volume and I don't think having to use a shell command or
> > > providing the password on stdin at startup really changes that very
> > > much.
> >
> > Understood, but now you are promoting a feature with an admittedly-poor
> > API, duplication of an OS feature, and perhaps an invasive change to the
> > code. Those are high hurdles.
>
> I thought we called it "incremental development". From the opposite
> point of view, would you say we should ban use of passphrase-protected
> SSL key files because the current user interface for them is bad?
>
> I have no use for data-at-rest encryption myself, but I wouldn't stop
> development just because the initial design proposal doesn't include
> top-notch key management.
Yes, but we have to have a plan on how to improve it. Why add a feature
that is hard to maintain, and hard to use.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:43 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 22:11 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-14 21:36 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 21:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 16:06 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-15 19:09 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 20:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 21:04 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 21:12 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 22:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 23:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 23:27 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 23:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 23:51 ` Re: WIP: Data at rest encryption Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2017-06-19 16:30 ` Robert Haas <robertmhaas@gmail.com>
2017-06-19 16:30 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
1 sibling, 1 reply; 93+ messages in thread
From: Robert Haas @ 2017-06-19 16:30 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Bruce Momjian <bruce@momjian.us>; Stephen Frost <sfrost@snowman.net>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; pgsql-hackers
On Thu, Jun 15, 2017 at 7:51 PM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:
> I thought we called it "incremental development". From the opposite
> point of view, would you say we should ban use of passphrase-protected
> SSL key files because the current user interface for them is bad?
I think that we've got a number of features which exist in the tree
today only because either (a) our standards were lower at the time
that those features were committed than they are today or (b) we
didn't realize how much trouble those features were going to create.
Just because we don't want to hose the people already using those
features does not mean that we want more features engineered to that
same quality level. Obviously, there's room for debate in any
particular case about how reasonable it is to expect someone who wants
to implement A to also improve B, and, well, maybe handling thing as
we do SSL certificates is good enough for this feature, too. I find
myself a bit skeptical about that, though. It preclude as lot of
things we might want to do. You're not going to be able to interface
with some external key management server that way, nor do encryption
of only part of the database, nor have multiple keys for different
parts of the database using that kind of setup.
One could argue that can all be added later, but I think there's a
question about how much that's going to affect the code structure.
Surely we don't want to install encryption v1 and then find that, by
not considering key management, we've made it really hard to get to
v2, and that it basically can't be done without ripping the whole
implementation out and replacing it. Maybe the database needs, at
some rather low level, a concept of whether the encryption key (or an
encryption key) is available yet, and maybe you get out of considering
that by deciding you're just going to prompt very early in startup,
but now when someone comes along and wants to improve things later,
and they've got to try to go find all of the code that depends on the
assumption that the key is always available and fix it. That could be
pretty annoying to validate. I think it's better to give at least
some consideration to these key management questions from the
beginning, lest we back ourselves into a corner. Whether or not the
SSL-passphrase style implementation is above or below the level we'd
consider a minimally viable feature, it's surely not where we want to
end up, and we shouldn't do anything that makes it likely that we'll
get stuck at precisely that point.
Also, practically, I think that type of solution is going to be
extremely difficult to use for most people. It means that the
database can't be started by the system startup scripts; you'll have
to log into the PG OS user account and launch the database from there.
IIUC, that means it won't be able to be controlled by things like
systemd, that just know about start and stop, but not about ask for a
password in the middle. Maybe I'm wrong about that, though. And
certainly, there will be some users for whom starting the database
manually and prompting for a password will be good enough, if not
ideal. But for people who want to fetch the encryption key from a key
management server, which I bet is a lot of people, that's not really
going to be good enough. I'm not really sure that rushing a first
patch that "works" for sufficiently small values of "works" is
actually going
> I have no use for data-at-rest encryption myself, but I wouldn't stop
> development just because the initial design proposal doesn't include
> top-notch key management.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:43 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 22:11 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-14 21:36 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 21:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 16:06 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-15 19:09 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 20:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 21:04 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 21:12 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 22:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 23:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 23:27 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 23:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 23:51 ` Re: WIP: Data at rest encryption Alvaro Herrera <alvherre@2ndquadrant.com>
2017-06-19 16:30 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
@ 2017-06-19 16:30 ` Robert Haas <robertmhaas@gmail.com>
0 siblings, 0 replies; 93+ messages in thread
From: Robert Haas @ 2017-06-19 16:30 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Bruce Momjian <bruce@momjian.us>; Stephen Frost <sfrost@snowman.net>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; pgsql-hackers
On Mon, Jun 19, 2017 at 12:30 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Thu, Jun 15, 2017 at 7:51 PM, Alvaro Herrera
> <alvherre@2ndquadrant.com> wrote:
>> I thought we called it "incremental development". From the opposite
>> point of view, would you say we should ban use of passphrase-protected
>> SSL key files because the current user interface for them is bad?
>
> I think that we've got a number of features which exist in the tree
> today only because either (a) our standards were lower at the time
> that those features were committed than they are today or (b) we
> didn't realize how much trouble those features were going to create.
> Just because we don't want to hose the people already using those
> features does not mean that we want more features engineered to that
> same quality level. Obviously, there's room for debate in any
> particular case about how reasonable it is to expect someone who wants
> to implement A to also improve B, and, well, maybe handling thing as
> we do SSL certificates is good enough for this feature, too. I find
> myself a bit skeptical about that, though. It preclude as lot of
> things we might want to do. You're not going to be able to interface
> with some external key management server that way, nor do encryption
> of only part of the database, nor have multiple keys for different
> parts of the database using that kind of setup.
>
> One could argue that can all be added later, but I think there's a
> question about how much that's going to affect the code structure.
> Surely we don't want to install encryption v1 and then find that, by
> not considering key management, we've made it really hard to get to
> v2, and that it basically can't be done without ripping the whole
> implementation out and replacing it. Maybe the database needs, at
> some rather low level, a concept of whether the encryption key (or an
> encryption key) is available yet, and maybe you get out of considering
> that by deciding you're just going to prompt very early in startup,
> but now when someone comes along and wants to improve things later,
> and they've got to try to go find all of the code that depends on the
> assumption that the key is always available and fix it. That could be
> pretty annoying to validate. I think it's better to give at least
> some consideration to these key management questions from the
> beginning, lest we back ourselves into a corner. Whether or not the
> SSL-passphrase style implementation is above or below the level we'd
> consider a minimally viable feature, it's surely not where we want to
> end up, and we shouldn't do anything that makes it likely that we'll
> get stuck at precisely that point.
>
> Also, practically, I think that type of solution is going to be
> extremely difficult to use for most people. It means that the
> database can't be started by the system startup scripts; you'll have
> to log into the PG OS user account and launch the database from there.
> IIUC, that means it won't be able to be controlled by things like
> systemd, that just know about start and stop, but not about ask for a
> password in the middle. Maybe I'm wrong about that, though. And
> certainly, there will be some users for whom starting the database
> manually and prompting for a password will be good enough, if not
> ideal. But for people who want to fetch the encryption key from a key
> management server, which I bet is a lot of people, that's not really
> going to be good enough. I'm not really sure that rushing a first
> patch that "works" for sufficiently small values of "works" is
> actually going
...to move us forward very much.
>> I have no use for data-at-rest encryption myself, but I wouldn't stop
>> development just because the initial design proposal doesn't include
>> top-notch key management.
I agree with that, but there's a difference between "not top-notch"
and "pretty bad".
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:43 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 22:11 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-14 21:36 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 21:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 16:06 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-15 19:09 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 20:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 21:04 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 21:12 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 22:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 23:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 23:27 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 23:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
@ 2017-06-15 23:56 ` Andres Freund <andres@anarazel.de>
2017-06-16 00:08 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2 siblings, 1 reply; 93+ messages in thread
From: Andres Freund @ 2017-06-15 23:56 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Stephen Frost <sfrost@snowman.net>; Robert Haas <robertmhaas@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; pgsql-hackers
On 2017-06-15 19:44:43 -0400, Bruce Momjian wrote:
> Understood, but now you are promoting a feature with an admittedly-poor
> API, duplication of an OS feature, and perhaps an invasive change to the
> code.
*Perhaps* an invasive change to the code? To me it's pretty evident
that this'll be a pretty costly feature from that angle. We've quite a
few places that manipulate on-disk files, and they'll all have to be
manipulated. Several of those are essentially critical sections, adding
memory allocations to them wouldn't be good, so we'll need
pre-allocation APIs.
I've only skimmed the discussion, but based on that I'm very surprised
how few concerns about this feature's complexity / maintainability
impact have been raised.
- Andres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:43 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 22:11 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-14 21:36 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 21:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 16:06 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-15 19:09 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 20:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 21:04 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 21:12 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 22:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 23:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 23:27 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 23:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 23:56 ` Re: WIP: Data at rest encryption Andres Freund <andres@anarazel.de>
@ 2017-06-16 00:08 ` Bruce Momjian <bruce@momjian.us>
2017-06-16 02:17 ` Re: WIP: Data at rest encryption Tatsuo Ishii <ishii@sraoss.co.jp>
2017-06-16 08:06 ` Re: WIP: Data at rest encryption Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
2017-06-16 15:41 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
0 siblings, 3 replies; 93+ messages in thread
From: Bruce Momjian @ 2017-06-16 00:08 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Stephen Frost <sfrost@snowman.net>; Robert Haas <robertmhaas@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; pgsql-hackers
On Thu, Jun 15, 2017 at 04:56:36PM -0700, Andres Freund wrote:
> On 2017-06-15 19:44:43 -0400, Bruce Momjian wrote:
> > Understood, but now you are promoting a feature with an admittedly-poor
> > API, duplication of an OS feature, and perhaps an invasive change to the
> > code.
>
> *Perhaps* an invasive change to the code? To me it's pretty evident
> that this'll be a pretty costly feature from that angle. We've quite a
> few places that manipulate on-disk files, and they'll all have to be
> manipulated. Several of those are essentially critical sections, adding
> memory allocations to them wouldn't be good, so we'll need
> pre-allocation APIs.
>
> I've only skimmed the discussion, but based on that I'm very surprised
> how few concerns about this feature's complexity / maintainability
> impact have been raised.
Yeah, I guess we will just have to wait to see it since other people are
excited about it. My concern is code complexity and usability
challenges, vs punting the problem to the operating system, though
admittedly there are some cases where that is not possible.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:43 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 22:11 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-14 21:36 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 21:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 16:06 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-15 19:09 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 20:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 21:04 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 21:12 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 22:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 23:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 23:27 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 23:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 23:56 ` Re: WIP: Data at rest encryption Andres Freund <andres@anarazel.de>
2017-06-16 00:08 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
@ 2017-06-16 02:17 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2 siblings, 0 replies; 93+ messages in thread
From: Tatsuo Ishii @ 2017-06-16 02:17 UTC (permalink / raw)
To: bruce@momjian.us; +Cc: andres@anarazel.de; sfrost@snowman.net; robertmhaas@gmail.com; peter.eisentraut@2ndquadrant.com; ants.aasma@eesti.ee; pgsql-hackers
> Yeah, I guess we will just have to wait to see it since other people are
> excited about it. My concern is code complexity and usability
> challenges, vs punting the problem to the operating system, though
> admittedly there are some cases where that is not possible.
Oracle sells this feature only with the expensive enterprise
edition. And people actually buy it. I guess the feature is pretty
important for some users.
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:43 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 22:11 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-14 21:36 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 21:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 16:06 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-15 19:09 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 20:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 21:04 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 21:12 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 22:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 23:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 23:27 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 23:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 23:56 ` Re: WIP: Data at rest encryption Andres Freund <andres@anarazel.de>
2017-06-16 00:08 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
@ 2017-06-16 08:06 ` Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
2017-06-16 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2 siblings, 1 reply; 93+ messages in thread
From: Konstantin Knizhnik @ 2017-06-16 08:06 UTC (permalink / raw)
To: pgsql-hackers
On 16.06.2017 03:08, Bruce Momjian wrote:
>
> Yeah, I guess we will just have to wait to see it since other people are
> excited about it. My concern is code complexity and usability
> challenges, vs punting the problem to the operating system, though
> admittedly there are some cases where that is not possible.
>
Let me also share my opinion about encryption and compression support at
database level.
PostgresPro Enterprise does support both. I have made presentation about
it at PgConn 2016 in Tallinn.
I was a little bit surprised that there were more questions about
encryption than about compression.
But right now we have several customers which are using compression and
none of them use encryption (just because them do not need
to protect their databases). But I absolutely sure that there are many
Postgres users which first of all need to protect their data.
Encryption is much easier to implement than compression, because it is
not changing page size. So I do not see any "complexity and flexibility
challenges" here.
Just for reference I attached to this mail our own encryption patch. I
do not want to propose it as alternative to Aasmas patch: it is less
flexible and doesn't support encryption of WAL, just encryption of
relation data. Also it doesn't allow custom encryption libraries: AES
implementation is embedded. Encryption cipher is taken from environment
variable. At Tallin's conferences I was informed about possible security
issue with passing key through environment variable: it is possible to
inspect server's environment variables using plpython/plperl stored
procedure.
This is why we unset this environment variable after reading. I am not
expect in security, but I do not know other issues with such solution.
Concerning the question whether to implement compression/encryption on
database level or rely on OS, my opinion is that there are many
scenarios where it is not possible or is not desirable to use OS level
encryption/protection. It first of all includes cloud installations and
embedded applications. I do not want to repeat arguments already
mentioned in this thread.
But the fact is that there are many people which really need
compression/encryption support and them can not or do not want to
redirect this aspects to OS. Almost all DBMSes are supporting
compression encryption, so lack of this features in Postgres definitely
can not be considered as Postgres advantage.
Postgres buffer manager interface significantly simplifies integration
of encryption and compression. There is actually single path through
which data is fetched/stored to the disk.
It is most obvious and natural solution to decompress/decrypt data when
it is read from the disk to page pool and compress/encrypt it when it is
written back. Taken in account that memory is cheap now and many
databases can completely fit in memory, storing pages in the buffer
cache in plain (decompressed/decrypted) format allows to minimize
overhead of compression/encryption and its influence on performance. For
read only queries working with cached data performance will be exactly
the same as without encryption/compression.
Write speed for encrypted pages will be certainly slightly worse, but
still encryption speed is much higher than disk IO speed.
So I do not think that it is really necessary to support encryption of
some particular tables, storing "non-secrete" data in plain format
without encryption. It should not cause noticeable improve of
performance, but may complicate implementation and increase possibility
of leaking secure data.
I do not think that pluggable storage API is right approach to integrate
compression and especially encryption. It is better to plugin encryption
between buffer manager and storage device,
allowing to use it with any storage implementation. Also it is not
clear to me whether encryption of WAL can be provided using pluggable
storage API.
The last discussed question is whether it is necessary to encrypt
temporary data (BufFile). In our solution we encrypt only main fork of
non-system relations and do no encrypt temporary relations. It may cause
that some secrete data will be stored at this disk in non-encrypted
format. But accessing this data is not trivial. You can not just
copy/stole disk, open database and do "select * from SecreteTable": you
will have to extract data from raw file yourself. So looks like it is
better to allow user to make choice whether to encrypt temporary data or
not.
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
aes.patch (text/x-patch, 80.8K) download
diff --git a/src/backend/storage/file/Makefile b/src/backend/storage/file/Makefile
index d2198f2..9492662 100644
--- a/src/backend/storage/file/Makefile
+++ b/src/backend/storage/file/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/storage/file
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
-OBJS = fd.o buffile.o copydir.o reinit.o
+OBJS = fd.o buffile.o copydir.o reinit.o rijndael.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/file/rijndael.c b/src/backend/storage/file/rijndael.c
new file mode 100644
index 0000000..31b2a60
--- /dev/null
+++ b/src/backend/storage/file/rijndael.c
@@ -0,0 +1,677 @@
+/* $OpenBSD: rijndael.c,v 1.6 2000/12/09 18:51:34 markus Exp $ */
+
+/* contrib/pgcrypto/rijndael.c */
+
+/* This is an independent implementation of the encryption algorithm: */
+/* */
+/* RIJNDAEL by Joan Daemen and Vincent Rijmen */
+/* */
+/* which is a candidate algorithm in the Advanced Encryption Standard */
+/* programme of the US National Institute of Standards and Technology. */
+/* */
+/* Copyright in this implementation is held by Dr B R Gladman but I */
+/* hereby give permission for its free direct or derivative use subject */
+/* to acknowledgment of its origin and compliance with any conditions */
+/* that the originators of the algorithm place on its exploitation. */
+/* */
+/* Dr Brian Gladman (gladman@seven77.demon.co.uk) 14th January 1999 */
+
+/* Timing data for Rijndael (rijndael.c)
+
+Algorithm: rijndael (rijndael.c)
+
+128 bit key:
+Key Setup: 305/1389 cycles (encrypt/decrypt)
+Encrypt: 374 cycles = 68.4 mbits/sec
+Decrypt: 352 cycles = 72.7 mbits/sec
+Mean: 363 cycles = 70.5 mbits/sec
+
+192 bit key:
+Key Setup: 277/1595 cycles (encrypt/decrypt)
+Encrypt: 439 cycles = 58.3 mbits/sec
+Decrypt: 425 cycles = 60.2 mbits/sec
+Mean: 432 cycles = 59.3 mbits/sec
+
+256 bit key:
+Key Setup: 374/1960 cycles (encrypt/decrypt)
+Encrypt: 502 cycles = 51.0 mbits/sec
+Decrypt: 498 cycles = 51.4 mbits/sec
+Mean: 500 cycles = 51.2 mbits/sec
+
+*/
+
+#include "postgres.h"
+
+#include <sys/param.h>
+#include <string.h>
+
+#include "storage/rijndael.h"
+
+#define PRE_CALC_TABLES
+#define LARGE_TABLES
+
+static void gen_tabs(void);
+
+/* 3. Basic macros for speeding up generic operations */
+
+/* Circular rotate of 32 bit values */
+
+#define rotr(x,n) (((x) >> ((int)(n))) | ((x) << (32 - (int)(n))))
+#define rotl(x,n) (((x) << ((int)(n))) | ((x) >> (32 - (int)(n))))
+
+/* Invert byte order in a 32 bit variable */
+
+#define bswap(x) ((rotl((x), 8) & 0x00ff00ff) | (rotr((x), 8) & 0xff00ff00))
+
+/* Extract byte from a 32 bit quantity (little endian notation) */
+
+#define byte(x,n) ((u1byte)((x) >> (8 * (n))))
+
+#ifdef WORDS_BIGENDIAN
+#define io_swap(x) bswap(x)
+#else
+#define io_swap(x) (x)
+#endif
+
+#ifdef PRINT_TABS
+#undef PRE_CALC_TABLES
+#endif
+
+#ifdef PRE_CALC_TABLES
+
+#include "rijndael.tbl"
+#define tab_gen 1
+#else /* !PRE_CALC_TABLES */
+
+static u1byte pow_tab[256];
+static u1byte log_tab[256];
+static u1byte sbx_tab[256];
+static u1byte isb_tab[256];
+static u4byte rco_tab[10];
+static u4byte ft_tab[4][256];
+static u4byte it_tab[4][256];
+
+#ifdef LARGE_TABLES
+static u4byte fl_tab[4][256];
+static u4byte il_tab[4][256];
+#endif
+
+static u4byte tab_gen = 0;
+#endif /* !PRE_CALC_TABLES */
+
+#define ff_mult(a,b) ((a) && (b) ? pow_tab[(log_tab[a] + log_tab[b]) % 255] : 0)
+
+#define f_rn(bo, bi, n, k) \
+ (bo)[n] = ft_tab[0][byte((bi)[n],0)] ^ \
+ ft_tab[1][byte((bi)[((n) + 1) & 3],1)] ^ \
+ ft_tab[2][byte((bi)[((n) + 2) & 3],2)] ^ \
+ ft_tab[3][byte((bi)[((n) + 3) & 3],3)] ^ *((k) + (n))
+
+#define i_rn(bo, bi, n, k) \
+ (bo)[n] = it_tab[0][byte((bi)[n],0)] ^ \
+ it_tab[1][byte((bi)[((n) + 3) & 3],1)] ^ \
+ it_tab[2][byte((bi)[((n) + 2) & 3],2)] ^ \
+ it_tab[3][byte((bi)[((n) + 1) & 3],3)] ^ *((k) + (n))
+
+#ifdef LARGE_TABLES
+
+#define ls_box(x) \
+ ( fl_tab[0][byte(x, 0)] ^ \
+ fl_tab[1][byte(x, 1)] ^ \
+ fl_tab[2][byte(x, 2)] ^ \
+ fl_tab[3][byte(x, 3)] )
+
+#define f_rl(bo, bi, n, k) \
+ (bo)[n] = fl_tab[0][byte((bi)[n],0)] ^ \
+ fl_tab[1][byte((bi)[((n) + 1) & 3],1)] ^ \
+ fl_tab[2][byte((bi)[((n) + 2) & 3],2)] ^ \
+ fl_tab[3][byte((bi)[((n) + 3) & 3],3)] ^ *((k) + (n))
+
+#define i_rl(bo, bi, n, k) \
+ (bo)[n] = il_tab[0][byte((bi)[n],0)] ^ \
+ il_tab[1][byte((bi)[((n) + 3) & 3],1)] ^ \
+ il_tab[2][byte((bi)[((n) + 2) & 3],2)] ^ \
+ il_tab[3][byte((bi)[((n) + 1) & 3],3)] ^ *((k) + (n))
+#else
+
+#define ls_box(x) \
+ ((u4byte)sbx_tab[byte(x, 0)] << 0) ^ \
+ ((u4byte)sbx_tab[byte(x, 1)] << 8) ^ \
+ ((u4byte)sbx_tab[byte(x, 2)] << 16) ^ \
+ ((u4byte)sbx_tab[byte(x, 3)] << 24)
+
+#define f_rl(bo, bi, n, k) \
+ (bo)[n] = (u4byte)sbx_tab[byte((bi)[n],0)] ^ \
+ rotl(((u4byte)sbx_tab[byte((bi)[((n) + 1) & 3],1)]), 8) ^ \
+ rotl(((u4byte)sbx_tab[byte((bi)[((n) + 2) & 3],2)]), 16) ^ \
+ rotl(((u4byte)sbx_tab[byte((bi)[((n) + 3) & 3],3)]), 24) ^ *((k) + (n))
+
+#define i_rl(bo, bi, n, k) \
+ (bo)[n] = (u4byte)isb_tab[byte((bi)[n],0)] ^ \
+ rotl(((u4byte)isb_tab[byte((bi)[((n) + 3) & 3],1)]), 8) ^ \
+ rotl(((u4byte)isb_tab[byte((bi)[((n) + 2) & 3],2)]), 16) ^ \
+ rotl(((u4byte)isb_tab[byte((bi)[((n) + 1) & 3],3)]), 24) ^ *((k) + (n))
+#endif
+
+static void
+gen_tabs(void)
+{
+#ifndef PRE_CALC_TABLES
+ u4byte i,
+ t;
+ u1byte p,
+ q;
+
+ /* log and power tables for GF(2**8) finite field with */
+ /* 0x11b as modular polynomial - the simplest prmitive */
+ /* root is 0x11, used here to generate the tables */
+
+ for (i = 0, p = 1; i < 256; ++i)
+ {
+ pow_tab[i] = (u1byte) p;
+ log_tab[p] = (u1byte) i;
+
+ p = p ^ (p << 1) ^ (p & 0x80 ? 0x01b : 0);
+ }
+
+ log_tab[1] = 0;
+ p = 1;
+
+ for (i = 0; i < 10; ++i)
+ {
+ rco_tab[i] = p;
+
+ p = (p << 1) ^ (p & 0x80 ? 0x1b : 0);
+ }
+
+ /* note that the affine byte transformation matrix in */
+ /* rijndael specification is in big endian format with */
+ /* bit 0 as the most significant bit. In the remainder */
+ /* of the specification the bits are numbered from the */
+ /* least significant end of a byte. */
+
+ for (i = 0; i < 256; ++i)
+ {
+ p = (i ? pow_tab[255 - log_tab[i]] : 0);
+ q = p;
+ q = (q >> 7) | (q << 1);
+ p ^= q;
+ q = (q >> 7) | (q << 1);
+ p ^= q;
+ q = (q >> 7) | (q << 1);
+ p ^= q;
+ q = (q >> 7) | (q << 1);
+ p ^= q ^ 0x63;
+ sbx_tab[i] = (u1byte) p;
+ isb_tab[p] = (u1byte) i;
+ }
+
+ for (i = 0; i < 256; ++i)
+ {
+ p = sbx_tab[i];
+
+#ifdef LARGE_TABLES
+
+ t = p;
+ fl_tab[0][i] = t;
+ fl_tab[1][i] = rotl(t, 8);
+ fl_tab[2][i] = rotl(t, 16);
+ fl_tab[3][i] = rotl(t, 24);
+#endif
+ t = ((u4byte) ff_mult(2, p)) |
+ ((u4byte) p << 8) |
+ ((u4byte) p << 16) |
+ ((u4byte) ff_mult(3, p) << 24);
+
+ ft_tab[0][i] = t;
+ ft_tab[1][i] = rotl(t, 8);
+ ft_tab[2][i] = rotl(t, 16);
+ ft_tab[3][i] = rotl(t, 24);
+
+ p = isb_tab[i];
+
+#ifdef LARGE_TABLES
+
+ t = p;
+ il_tab[0][i] = t;
+ il_tab[1][i] = rotl(t, 8);
+ il_tab[2][i] = rotl(t, 16);
+ il_tab[3][i] = rotl(t, 24);
+#endif
+ t = ((u4byte) ff_mult(14, p)) |
+ ((u4byte) ff_mult(9, p) << 8) |
+ ((u4byte) ff_mult(13, p) << 16) |
+ ((u4byte) ff_mult(11, p) << 24);
+
+ it_tab[0][i] = t;
+ it_tab[1][i] = rotl(t, 8);
+ it_tab[2][i] = rotl(t, 16);
+ it_tab[3][i] = rotl(t, 24);
+ }
+
+ tab_gen = 1;
+#endif /* !PRE_CALC_TABLES */
+}
+
+
+#define star_x(x) (((x) & 0x7f7f7f7f) << 1) ^ ((((x) & 0x80808080) >> 7) * 0x1b)
+
+#define imix_col(y,x) \
+do { \
+ u = star_x(x); \
+ v = star_x(u); \
+ w = star_x(v); \
+ t = w ^ (x); \
+ (y) = u ^ v ^ w; \
+ (y) ^= rotr(u ^ t, 8) ^ \
+ rotr(v ^ t, 16) ^ \
+ rotr(t,24); \
+} while (0)
+
+/* initialise the key schedule from the user supplied key */
+
+#define loop4(i) \
+do { t = ls_box(rotr(t, 8)) ^ rco_tab[i]; \
+ t ^= e_key[4 * i]; e_key[4 * i + 4] = t; \
+ t ^= e_key[4 * i + 1]; e_key[4 * i + 5] = t; \
+ t ^= e_key[4 * i + 2]; e_key[4 * i + 6] = t; \
+ t ^= e_key[4 * i + 3]; e_key[4 * i + 7] = t; \
+} while (0)
+
+#define loop6(i) \
+do { t = ls_box(rotr(t, 8)) ^ rco_tab[i]; \
+ t ^= e_key[6 * (i)]; e_key[6 * (i) + 6] = t; \
+ t ^= e_key[6 * (i) + 1]; e_key[6 * (i) + 7] = t; \
+ t ^= e_key[6 * (i) + 2]; e_key[6 * (i) + 8] = t; \
+ t ^= e_key[6 * (i) + 3]; e_key[6 * (i) + 9] = t; \
+ t ^= e_key[6 * (i) + 4]; e_key[6 * (i) + 10] = t; \
+ t ^= e_key[6 * (i) + 5]; e_key[6 * (i) + 11] = t; \
+} while (0)
+
+#define loop8(i) \
+do { t = ls_box(rotr(t, 8)) ^ rco_tab[i]; \
+ t ^= e_key[8 * (i)]; e_key[8 * (i) + 8] = t; \
+ t ^= e_key[8 * (i) + 1]; e_key[8 * (i) + 9] = t; \
+ t ^= e_key[8 * (i) + 2]; e_key[8 * (i) + 10] = t; \
+ t ^= e_key[8 * (i) + 3]; e_key[8 * (i) + 11] = t; \
+ t = e_key[8 * (i) + 4] ^ ls_box(t); \
+ e_key[8 * (i) + 12] = t; \
+ t ^= e_key[8 * (i) + 5]; e_key[8 * (i) + 13] = t; \
+ t ^= e_key[8 * (i) + 6]; e_key[8 * (i) + 14] = t; \
+ t ^= e_key[8 * (i) + 7]; e_key[8 * (i) + 15] = t; \
+} while (0)
+
+rijndael_ctx *
+rijndael_set_key(rijndael_ctx *ctx, const u4byte *in_key, const u4byte key_len,
+ int encrypt)
+{
+ u4byte i,
+ t,
+ u,
+ v,
+ w;
+ u4byte *e_key = ctx->e_key;
+ u4byte *d_key = ctx->d_key;
+
+ ctx->decrypt = !encrypt;
+
+ if (!tab_gen)
+ gen_tabs();
+
+ ctx->k_len = (key_len + 31) / 32;
+
+ e_key[0] = io_swap(in_key[0]);
+ e_key[1] = io_swap(in_key[1]);
+ e_key[2] = io_swap(in_key[2]);
+ e_key[3] = io_swap(in_key[3]);
+
+ switch (ctx->k_len)
+ {
+ case 4:
+ t = e_key[3];
+ for (i = 0; i < 10; ++i)
+ loop4(i);
+ break;
+
+ case 6:
+ e_key[4] = io_swap(in_key[4]);
+ t = e_key[5] = io_swap(in_key[5]);
+ for (i = 0; i < 8; ++i)
+ loop6(i);
+ break;
+
+ case 8:
+ e_key[4] = io_swap(in_key[4]);
+ e_key[5] = io_swap(in_key[5]);
+ e_key[6] = io_swap(in_key[6]);
+ t = e_key[7] = io_swap(in_key[7]);
+ for (i = 0; i < 7; ++i)
+ loop8(i);
+ break;
+ }
+
+ if (!encrypt)
+ {
+ d_key[0] = e_key[0];
+ d_key[1] = e_key[1];
+ d_key[2] = e_key[2];
+ d_key[3] = e_key[3];
+
+ for (i = 4; i < 4 * ctx->k_len + 24; ++i)
+ imix_col(d_key[i], e_key[i]);
+ }
+
+ return ctx;
+}
+
+/* encrypt a block of text */
+
+#define f_nround(bo, bi, k) \
+do { \
+ f_rn(bo, bi, 0, k); \
+ f_rn(bo, bi, 1, k); \
+ f_rn(bo, bi, 2, k); \
+ f_rn(bo, bi, 3, k); \
+ k += 4; \
+} while (0)
+
+#define f_lround(bo, bi, k) \
+do { \
+ f_rl(bo, bi, 0, k); \
+ f_rl(bo, bi, 1, k); \
+ f_rl(bo, bi, 2, k); \
+ f_rl(bo, bi, 3, k); \
+} while (0)
+
+void
+rijndael_encrypt(rijndael_ctx *ctx, const u4byte *in_blk, u4byte *out_blk)
+{
+ u4byte k_len = ctx->k_len;
+ u4byte *e_key = ctx->e_key;
+ u4byte b0[4],
+ b1[4],
+ *kp;
+
+ b0[0] = io_swap(in_blk[0]) ^ e_key[0];
+ b0[1] = io_swap(in_blk[1]) ^ e_key[1];
+ b0[2] = io_swap(in_blk[2]) ^ e_key[2];
+ b0[3] = io_swap(in_blk[3]) ^ e_key[3];
+
+ kp = e_key + 4;
+
+ if (k_len > 6)
+ {
+ f_nround(b1, b0, kp);
+ f_nround(b0, b1, kp);
+ }
+
+ if (k_len > 4)
+ {
+ f_nround(b1, b0, kp);
+ f_nround(b0, b1, kp);
+ }
+
+ f_nround(b1, b0, kp);
+ f_nround(b0, b1, kp);
+ f_nround(b1, b0, kp);
+ f_nround(b0, b1, kp);
+ f_nround(b1, b0, kp);
+ f_nround(b0, b1, kp);
+ f_nround(b1, b0, kp);
+ f_nround(b0, b1, kp);
+ f_nround(b1, b0, kp);
+ f_lround(b0, b1, kp);
+
+ out_blk[0] = io_swap(b0[0]);
+ out_blk[1] = io_swap(b0[1]);
+ out_blk[2] = io_swap(b0[2]);
+ out_blk[3] = io_swap(b0[3]);
+}
+
+/* decrypt a block of text */
+
+#define i_nround(bo, bi, k) \
+do { \
+ i_rn(bo, bi, 0, k); \
+ i_rn(bo, bi, 1, k); \
+ i_rn(bo, bi, 2, k); \
+ i_rn(bo, bi, 3, k); \
+ k -= 4; \
+} while (0)
+
+#define i_lround(bo, bi, k) \
+do { \
+ i_rl(bo, bi, 0, k); \
+ i_rl(bo, bi, 1, k); \
+ i_rl(bo, bi, 2, k); \
+ i_rl(bo, bi, 3, k); \
+} while (0)
+
+void
+rijndael_decrypt(rijndael_ctx *ctx, const u4byte *in_blk, u4byte *out_blk)
+{
+ u4byte b0[4],
+ b1[4],
+ *kp;
+ u4byte k_len = ctx->k_len;
+ u4byte *e_key = ctx->e_key;
+ u4byte *d_key = ctx->d_key;
+
+ b0[0] = io_swap(in_blk[0]) ^ e_key[4 * k_len + 24];
+ b0[1] = io_swap(in_blk[1]) ^ e_key[4 * k_len + 25];
+ b0[2] = io_swap(in_blk[2]) ^ e_key[4 * k_len + 26];
+ b0[3] = io_swap(in_blk[3]) ^ e_key[4 * k_len + 27];
+
+ kp = d_key + 4 * (k_len + 5);
+
+ if (k_len > 6)
+ {
+ i_nround(b1, b0, kp);
+ i_nround(b0, b1, kp);
+ }
+
+ if (k_len > 4)
+ {
+ i_nround(b1, b0, kp);
+ i_nround(b0, b1, kp);
+ }
+
+ i_nround(b1, b0, kp);
+ i_nround(b0, b1, kp);
+ i_nround(b1, b0, kp);
+ i_nround(b0, b1, kp);
+ i_nround(b1, b0, kp);
+ i_nround(b0, b1, kp);
+ i_nround(b1, b0, kp);
+ i_nround(b0, b1, kp);
+ i_nround(b1, b0, kp);
+ i_lround(b0, b1, kp);
+
+ out_blk[0] = io_swap(b0[0]);
+ out_blk[1] = io_swap(b0[1]);
+ out_blk[2] = io_swap(b0[2]);
+ out_blk[3] = io_swap(b0[3]);
+}
+
+/*
+ * conventional interface
+ *
+ * ATM it hopes all data is 4-byte aligned - which
+ * should be true for PX. -marko
+ */
+
+void
+aes_set_key(rijndael_ctx *ctx, const uint8 *key, unsigned keybits, int enc)
+{
+ uint32 *k;
+
+ k = (uint32 *) key;
+ rijndael_set_key(ctx, k, keybits, enc);
+}
+
+void
+aes_ecb_encrypt(rijndael_ctx *ctx, uint8 *data, unsigned len)
+{
+ unsigned bs = 16;
+ uint32 *d;
+
+ while (len >= bs)
+ {
+ d = (uint32 *) data;
+ rijndael_encrypt(ctx, d, d);
+
+ len -= bs;
+ data += bs;
+ }
+}
+
+void
+aes_ecb_decrypt(rijndael_ctx *ctx, uint8 *data, unsigned len)
+{
+ unsigned bs = 16;
+ uint32 *d;
+
+ while (len >= bs)
+ {
+ d = (uint32 *) data;
+ rijndael_decrypt(ctx, d, d);
+
+ len -= bs;
+ data += bs;
+ }
+}
+
+void
+aes_cbc_encrypt(rijndael_ctx *ctx, uint8 *iva, uint8 *data, unsigned len)
+{
+ uint32 *iv = (uint32 *) iva;
+ uint32 *d = (uint32 *) data;
+ unsigned bs = 16;
+
+ while (len >= bs)
+ {
+ d[0] ^= iv[0];
+ d[1] ^= iv[1];
+ d[2] ^= iv[2];
+ d[3] ^= iv[3];
+
+ rijndael_encrypt(ctx, d, d);
+
+ iv = d;
+ d += bs / 4;
+ len -= bs;
+ }
+}
+
+void
+aes_cbc_decrypt(rijndael_ctx *ctx, uint8 *iva, uint8 *data, unsigned len)
+{
+ uint32 *d = (uint32 *) data;
+ unsigned bs = 16;
+ uint32 buf[4],
+ iv[4];
+
+ memcpy(iv, iva, bs);
+ while (len >= bs)
+ {
+ buf[0] = d[0];
+ buf[1] = d[1];
+ buf[2] = d[2];
+ buf[3] = d[3];
+
+ rijndael_decrypt(ctx, buf, d);
+
+ d[0] ^= iv[0];
+ d[1] ^= iv[1];
+ d[2] ^= iv[2];
+ d[3] ^= iv[3];
+
+ iv[0] = buf[0];
+ iv[1] = buf[1];
+ iv[2] = buf[2];
+ iv[3] = buf[3];
+ d += 4;
+ len -= bs;
+ }
+}
+
+/*
+ * pre-calculate tables.
+ *
+ * On i386 lifts 17k from .bss to .rodata
+ * and avoids 1k code and setup time.
+ * -marko
+ */
+#ifdef PRINT_TABS
+
+static void
+show256u8(char *name, uint8 *data)
+{
+ int i;
+
+ printf("static const u1byte %s[256] = {\n ", name);
+ for (i = 0; i < 256;)
+ {
+ printf("%u", pow_tab[i++]);
+ if (i < 256)
+ printf(i % 16 ? ", " : ",\n ");
+ }
+ printf("\n};\n\n");
+}
+
+
+static void
+show4x256u32(char *name, uint32 data[4][256])
+{
+ int i,
+ j;
+
+ printf("static const u4byte %s[4][256] = {\n{\n ", name);
+ for (i = 0; i < 4; i++)
+ {
+ for (j = 0; j < 256;)
+ {
+ printf("0x%08x", data[i][j]);
+ j++;
+ if (j < 256)
+ printf(j % 4 ? ", " : ",\n ");
+ }
+ printf(i < 3 ? "\n}, {\n " : "\n}\n");
+ }
+ printf("};\n\n");
+}
+
+int
+main()
+{
+ int i;
+ char *hdr = "/* Generated by rijndael.c */\n\n";
+
+ gen_tabs();
+
+ printf(hdr);
+ show256u8("pow_tab", pow_tab);
+ show256u8("log_tab", log_tab);
+ show256u8("sbx_tab", sbx_tab);
+ show256u8("isb_tab", isb_tab);
+
+ show4x256u32("ft_tab", ft_tab);
+ show4x256u32("it_tab", it_tab);
+#ifdef LARGE_TABLES
+ show4x256u32("fl_tab", fl_tab);
+ show4x256u32("il_tab", il_tab);
+#endif
+ printf("static const u4byte rco_tab[10] = {\n ");
+ for (i = 0; i < 10; i++)
+ {
+ printf("0x%08x", rco_tab[i]);
+ if (i < 9)
+ printf(", ");
+ if (i == 4)
+ printf("\n ");
+ }
+ printf("\n};\n\n");
+ return 0;
+}
+
+#endif
diff --git a/src/backend/storage/file/rijndael.tbl b/src/backend/storage/file/rijndael.tbl
new file mode 100644
index 0000000..c7610c0
--- /dev/null
+++ b/src/backend/storage/file/rijndael.tbl
@@ -0,0 +1,1138 @@
+/* Generated by rijndael.c */
+
+static const u1byte pow_tab[256] = {
+ 1, 3, 5, 15, 17, 51, 85, 255, 26, 46, 114, 150, 161, 248, 19, 53,
+ 95, 225, 56, 72, 216, 115, 149, 164, 247, 2, 6, 10, 30, 34, 102, 170,
+ 229, 52, 92, 228, 55, 89, 235, 38, 106, 190, 217, 112, 144, 171, 230, 49,
+ 83, 245, 4, 12, 20, 60, 68, 204, 79, 209, 104, 184, 211, 110, 178, 205,
+ 76, 212, 103, 169, 224, 59, 77, 215, 98, 166, 241, 8, 24, 40, 120, 136,
+ 131, 158, 185, 208, 107, 189, 220, 127, 129, 152, 179, 206, 73, 219, 118, 154,
+ 181, 196, 87, 249, 16, 48, 80, 240, 11, 29, 39, 105, 187, 214, 97, 163,
+ 254, 25, 43, 125, 135, 146, 173, 236, 47, 113, 147, 174, 233, 32, 96, 160,
+ 251, 22, 58, 78, 210, 109, 183, 194, 93, 231, 50, 86, 250, 21, 63, 65,
+ 195, 94, 226, 61, 71, 201, 64, 192, 91, 237, 44, 116, 156, 191, 218, 117,
+ 159, 186, 213, 100, 172, 239, 42, 126, 130, 157, 188, 223, 122, 142, 137, 128,
+ 155, 182, 193, 88, 232, 35, 101, 175, 234, 37, 111, 177, 200, 67, 197, 84,
+ 252, 31, 33, 99, 165, 244, 7, 9, 27, 45, 119, 153, 176, 203, 70, 202,
+ 69, 207, 74, 222, 121, 139, 134, 145, 168, 227, 62, 66, 198, 81, 243, 14,
+ 18, 54, 90, 238, 41, 123, 141, 140, 143, 138, 133, 148, 167, 242, 13, 23,
+ 57, 75, 221, 124, 132, 151, 162, 253, 28, 36, 108, 180, 199, 82, 246, 1
+};
+
+static const u1byte log_tab[256] = {
+ 1, 3, 5, 15, 17, 51, 85, 255, 26, 46, 114, 150, 161, 248, 19, 53,
+ 95, 225, 56, 72, 216, 115, 149, 164, 247, 2, 6, 10, 30, 34, 102, 170,
+ 229, 52, 92, 228, 55, 89, 235, 38, 106, 190, 217, 112, 144, 171, 230, 49,
+ 83, 245, 4, 12, 20, 60, 68, 204, 79, 209, 104, 184, 211, 110, 178, 205,
+ 76, 212, 103, 169, 224, 59, 77, 215, 98, 166, 241, 8, 24, 40, 120, 136,
+ 131, 158, 185, 208, 107, 189, 220, 127, 129, 152, 179, 206, 73, 219, 118, 154,
+ 181, 196, 87, 249, 16, 48, 80, 240, 11, 29, 39, 105, 187, 214, 97, 163,
+ 254, 25, 43, 125, 135, 146, 173, 236, 47, 113, 147, 174, 233, 32, 96, 160,
+ 251, 22, 58, 78, 210, 109, 183, 194, 93, 231, 50, 86, 250, 21, 63, 65,
+ 195, 94, 226, 61, 71, 201, 64, 192, 91, 237, 44, 116, 156, 191, 218, 117,
+ 159, 186, 213, 100, 172, 239, 42, 126, 130, 157, 188, 223, 122, 142, 137, 128,
+ 155, 182, 193, 88, 232, 35, 101, 175, 234, 37, 111, 177, 200, 67, 197, 84,
+ 252, 31, 33, 99, 165, 244, 7, 9, 27, 45, 119, 153, 176, 203, 70, 202,
+ 69, 207, 74, 222, 121, 139, 134, 145, 168, 227, 62, 66, 198, 81, 243, 14,
+ 18, 54, 90, 238, 41, 123, 141, 140, 143, 138, 133, 148, 167, 242, 13, 23,
+ 57, 75, 221, 124, 132, 151, 162, 253, 28, 36, 108, 180, 199, 82, 246, 1
+};
+
+static const u1byte sbx_tab[256] = {
+ 1, 3, 5, 15, 17, 51, 85, 255, 26, 46, 114, 150, 161, 248, 19, 53,
+ 95, 225, 56, 72, 216, 115, 149, 164, 247, 2, 6, 10, 30, 34, 102, 170,
+ 229, 52, 92, 228, 55, 89, 235, 38, 106, 190, 217, 112, 144, 171, 230, 49,
+ 83, 245, 4, 12, 20, 60, 68, 204, 79, 209, 104, 184, 211, 110, 178, 205,
+ 76, 212, 103, 169, 224, 59, 77, 215, 98, 166, 241, 8, 24, 40, 120, 136,
+ 131, 158, 185, 208, 107, 189, 220, 127, 129, 152, 179, 206, 73, 219, 118, 154,
+ 181, 196, 87, 249, 16, 48, 80, 240, 11, 29, 39, 105, 187, 214, 97, 163,
+ 254, 25, 43, 125, 135, 146, 173, 236, 47, 113, 147, 174, 233, 32, 96, 160,
+ 251, 22, 58, 78, 210, 109, 183, 194, 93, 231, 50, 86, 250, 21, 63, 65,
+ 195, 94, 226, 61, 71, 201, 64, 192, 91, 237, 44, 116, 156, 191, 218, 117,
+ 159, 186, 213, 100, 172, 239, 42, 126, 130, 157, 188, 223, 122, 142, 137, 128,
+ 155, 182, 193, 88, 232, 35, 101, 175, 234, 37, 111, 177, 200, 67, 197, 84,
+ 252, 31, 33, 99, 165, 244, 7, 9, 27, 45, 119, 153, 176, 203, 70, 202,
+ 69, 207, 74, 222, 121, 139, 134, 145, 168, 227, 62, 66, 198, 81, 243, 14,
+ 18, 54, 90, 238, 41, 123, 141, 140, 143, 138, 133, 148, 167, 242, 13, 23,
+ 57, 75, 221, 124, 132, 151, 162, 253, 28, 36, 108, 180, 199, 82, 246, 1
+};
+
+static const u1byte isb_tab[256] = {
+ 1, 3, 5, 15, 17, 51, 85, 255, 26, 46, 114, 150, 161, 248, 19, 53,
+ 95, 225, 56, 72, 216, 115, 149, 164, 247, 2, 6, 10, 30, 34, 102, 170,
+ 229, 52, 92, 228, 55, 89, 235, 38, 106, 190, 217, 112, 144, 171, 230, 49,
+ 83, 245, 4, 12, 20, 60, 68, 204, 79, 209, 104, 184, 211, 110, 178, 205,
+ 76, 212, 103, 169, 224, 59, 77, 215, 98, 166, 241, 8, 24, 40, 120, 136,
+ 131, 158, 185, 208, 107, 189, 220, 127, 129, 152, 179, 206, 73, 219, 118, 154,
+ 181, 196, 87, 249, 16, 48, 80, 240, 11, 29, 39, 105, 187, 214, 97, 163,
+ 254, 25, 43, 125, 135, 146, 173, 236, 47, 113, 147, 174, 233, 32, 96, 160,
+ 251, 22, 58, 78, 210, 109, 183, 194, 93, 231, 50, 86, 250, 21, 63, 65,
+ 195, 94, 226, 61, 71, 201, 64, 192, 91, 237, 44, 116, 156, 191, 218, 117,
+ 159, 186, 213, 100, 172, 239, 42, 126, 130, 157, 188, 223, 122, 142, 137, 128,
+ 155, 182, 193, 88, 232, 35, 101, 175, 234, 37, 111, 177, 200, 67, 197, 84,
+ 252, 31, 33, 99, 165, 244, 7, 9, 27, 45, 119, 153, 176, 203, 70, 202,
+ 69, 207, 74, 222, 121, 139, 134, 145, 168, 227, 62, 66, 198, 81, 243, 14,
+ 18, 54, 90, 238, 41, 123, 141, 140, 143, 138, 133, 148, 167, 242, 13, 23,
+ 57, 75, 221, 124, 132, 151, 162, 253, 28, 36, 108, 180, 199, 82, 246, 1
+};
+
+static const u4byte ft_tab[4][256] = {
+{
+ 0xa56363c6, 0x847c7cf8, 0x997777ee, 0x8d7b7bf6,
+ 0x0df2f2ff, 0xbd6b6bd6, 0xb16f6fde, 0x54c5c591,
+ 0x50303060, 0x03010102, 0xa96767ce, 0x7d2b2b56,
+ 0x19fefee7, 0x62d7d7b5, 0xe6abab4d, 0x9a7676ec,
+ 0x45caca8f, 0x9d82821f, 0x40c9c989, 0x877d7dfa,
+ 0x15fafaef, 0xeb5959b2, 0xc947478e, 0x0bf0f0fb,
+ 0xecadad41, 0x67d4d4b3, 0xfda2a25f, 0xeaafaf45,
+ 0xbf9c9c23, 0xf7a4a453, 0x967272e4, 0x5bc0c09b,
+ 0xc2b7b775, 0x1cfdfde1, 0xae93933d, 0x6a26264c,
+ 0x5a36366c, 0x413f3f7e, 0x02f7f7f5, 0x4fcccc83,
+ 0x5c343468, 0xf4a5a551, 0x34e5e5d1, 0x08f1f1f9,
+ 0x937171e2, 0x73d8d8ab, 0x53313162, 0x3f15152a,
+ 0x0c040408, 0x52c7c795, 0x65232346, 0x5ec3c39d,
+ 0x28181830, 0xa1969637, 0x0f05050a, 0xb59a9a2f,
+ 0x0907070e, 0x36121224, 0x9b80801b, 0x3de2e2df,
+ 0x26ebebcd, 0x6927274e, 0xcdb2b27f, 0x9f7575ea,
+ 0x1b090912, 0x9e83831d, 0x742c2c58, 0x2e1a1a34,
+ 0x2d1b1b36, 0xb26e6edc, 0xee5a5ab4, 0xfba0a05b,
+ 0xf65252a4, 0x4d3b3b76, 0x61d6d6b7, 0xceb3b37d,
+ 0x7b292952, 0x3ee3e3dd, 0x712f2f5e, 0x97848413,
+ 0xf55353a6, 0x68d1d1b9, 0x00000000, 0x2cededc1,
+ 0x60202040, 0x1ffcfce3, 0xc8b1b179, 0xed5b5bb6,
+ 0xbe6a6ad4, 0x46cbcb8d, 0xd9bebe67, 0x4b393972,
+ 0xde4a4a94, 0xd44c4c98, 0xe85858b0, 0x4acfcf85,
+ 0x6bd0d0bb, 0x2aefefc5, 0xe5aaaa4f, 0x16fbfbed,
+ 0xc5434386, 0xd74d4d9a, 0x55333366, 0x94858511,
+ 0xcf45458a, 0x10f9f9e9, 0x06020204, 0x817f7ffe,
+ 0xf05050a0, 0x443c3c78, 0xba9f9f25, 0xe3a8a84b,
+ 0xf35151a2, 0xfea3a35d, 0xc0404080, 0x8a8f8f05,
+ 0xad92923f, 0xbc9d9d21, 0x48383870, 0x04f5f5f1,
+ 0xdfbcbc63, 0xc1b6b677, 0x75dadaaf, 0x63212142,
+ 0x30101020, 0x1affffe5, 0x0ef3f3fd, 0x6dd2d2bf,
+ 0x4ccdcd81, 0x140c0c18, 0x35131326, 0x2fececc3,
+ 0xe15f5fbe, 0xa2979735, 0xcc444488, 0x3917172e,
+ 0x57c4c493, 0xf2a7a755, 0x827e7efc, 0x473d3d7a,
+ 0xac6464c8, 0xe75d5dba, 0x2b191932, 0x957373e6,
+ 0xa06060c0, 0x98818119, 0xd14f4f9e, 0x7fdcdca3,
+ 0x66222244, 0x7e2a2a54, 0xab90903b, 0x8388880b,
+ 0xca46468c, 0x29eeeec7, 0xd3b8b86b, 0x3c141428,
+ 0x79dedea7, 0xe25e5ebc, 0x1d0b0b16, 0x76dbdbad,
+ 0x3be0e0db, 0x56323264, 0x4e3a3a74, 0x1e0a0a14,
+ 0xdb494992, 0x0a06060c, 0x6c242448, 0xe45c5cb8,
+ 0x5dc2c29f, 0x6ed3d3bd, 0xefacac43, 0xa66262c4,
+ 0xa8919139, 0xa4959531, 0x37e4e4d3, 0x8b7979f2,
+ 0x32e7e7d5, 0x43c8c88b, 0x5937376e, 0xb76d6dda,
+ 0x8c8d8d01, 0x64d5d5b1, 0xd24e4e9c, 0xe0a9a949,
+ 0xb46c6cd8, 0xfa5656ac, 0x07f4f4f3, 0x25eaeacf,
+ 0xaf6565ca, 0x8e7a7af4, 0xe9aeae47, 0x18080810,
+ 0xd5baba6f, 0x887878f0, 0x6f25254a, 0x722e2e5c,
+ 0x241c1c38, 0xf1a6a657, 0xc7b4b473, 0x51c6c697,
+ 0x23e8e8cb, 0x7cdddda1, 0x9c7474e8, 0x211f1f3e,
+ 0xdd4b4b96, 0xdcbdbd61, 0x868b8b0d, 0x858a8a0f,
+ 0x907070e0, 0x423e3e7c, 0xc4b5b571, 0xaa6666cc,
+ 0xd8484890, 0x05030306, 0x01f6f6f7, 0x120e0e1c,
+ 0xa36161c2, 0x5f35356a, 0xf95757ae, 0xd0b9b969,
+ 0x91868617, 0x58c1c199, 0x271d1d3a, 0xb99e9e27,
+ 0x38e1e1d9, 0x13f8f8eb, 0xb398982b, 0x33111122,
+ 0xbb6969d2, 0x70d9d9a9, 0x898e8e07, 0xa7949433,
+ 0xb69b9b2d, 0x221e1e3c, 0x92878715, 0x20e9e9c9,
+ 0x49cece87, 0xff5555aa, 0x78282850, 0x7adfdfa5,
+ 0x8f8c8c03, 0xf8a1a159, 0x80898909, 0x170d0d1a,
+ 0xdabfbf65, 0x31e6e6d7, 0xc6424284, 0xb86868d0,
+ 0xc3414182, 0xb0999929, 0x772d2d5a, 0x110f0f1e,
+ 0xcbb0b07b, 0xfc5454a8, 0xd6bbbb6d, 0x3a16162c
+}, {
+ 0x6363c6a5, 0x7c7cf884, 0x7777ee99, 0x7b7bf68d,
+ 0xf2f2ff0d, 0x6b6bd6bd, 0x6f6fdeb1, 0xc5c59154,
+ 0x30306050, 0x01010203, 0x6767cea9, 0x2b2b567d,
+ 0xfefee719, 0xd7d7b562, 0xabab4de6, 0x7676ec9a,
+ 0xcaca8f45, 0x82821f9d, 0xc9c98940, 0x7d7dfa87,
+ 0xfafaef15, 0x5959b2eb, 0x47478ec9, 0xf0f0fb0b,
+ 0xadad41ec, 0xd4d4b367, 0xa2a25ffd, 0xafaf45ea,
+ 0x9c9c23bf, 0xa4a453f7, 0x7272e496, 0xc0c09b5b,
+ 0xb7b775c2, 0xfdfde11c, 0x93933dae, 0x26264c6a,
+ 0x36366c5a, 0x3f3f7e41, 0xf7f7f502, 0xcccc834f,
+ 0x3434685c, 0xa5a551f4, 0xe5e5d134, 0xf1f1f908,
+ 0x7171e293, 0xd8d8ab73, 0x31316253, 0x15152a3f,
+ 0x0404080c, 0xc7c79552, 0x23234665, 0xc3c39d5e,
+ 0x18183028, 0x969637a1, 0x05050a0f, 0x9a9a2fb5,
+ 0x07070e09, 0x12122436, 0x80801b9b, 0xe2e2df3d,
+ 0xebebcd26, 0x27274e69, 0xb2b27fcd, 0x7575ea9f,
+ 0x0909121b, 0x83831d9e, 0x2c2c5874, 0x1a1a342e,
+ 0x1b1b362d, 0x6e6edcb2, 0x5a5ab4ee, 0xa0a05bfb,
+ 0x5252a4f6, 0x3b3b764d, 0xd6d6b761, 0xb3b37dce,
+ 0x2929527b, 0xe3e3dd3e, 0x2f2f5e71, 0x84841397,
+ 0x5353a6f5, 0xd1d1b968, 0x00000000, 0xededc12c,
+ 0x20204060, 0xfcfce31f, 0xb1b179c8, 0x5b5bb6ed,
+ 0x6a6ad4be, 0xcbcb8d46, 0xbebe67d9, 0x3939724b,
+ 0x4a4a94de, 0x4c4c98d4, 0x5858b0e8, 0xcfcf854a,
+ 0xd0d0bb6b, 0xefefc52a, 0xaaaa4fe5, 0xfbfbed16,
+ 0x434386c5, 0x4d4d9ad7, 0x33336655, 0x85851194,
+ 0x45458acf, 0xf9f9e910, 0x02020406, 0x7f7ffe81,
+ 0x5050a0f0, 0x3c3c7844, 0x9f9f25ba, 0xa8a84be3,
+ 0x5151a2f3, 0xa3a35dfe, 0x404080c0, 0x8f8f058a,
+ 0x92923fad, 0x9d9d21bc, 0x38387048, 0xf5f5f104,
+ 0xbcbc63df, 0xb6b677c1, 0xdadaaf75, 0x21214263,
+ 0x10102030, 0xffffe51a, 0xf3f3fd0e, 0xd2d2bf6d,
+ 0xcdcd814c, 0x0c0c1814, 0x13132635, 0xececc32f,
+ 0x5f5fbee1, 0x979735a2, 0x444488cc, 0x17172e39,
+ 0xc4c49357, 0xa7a755f2, 0x7e7efc82, 0x3d3d7a47,
+ 0x6464c8ac, 0x5d5dbae7, 0x1919322b, 0x7373e695,
+ 0x6060c0a0, 0x81811998, 0x4f4f9ed1, 0xdcdca37f,
+ 0x22224466, 0x2a2a547e, 0x90903bab, 0x88880b83,
+ 0x46468cca, 0xeeeec729, 0xb8b86bd3, 0x1414283c,
+ 0xdedea779, 0x5e5ebce2, 0x0b0b161d, 0xdbdbad76,
+ 0xe0e0db3b, 0x32326456, 0x3a3a744e, 0x0a0a141e,
+ 0x494992db, 0x06060c0a, 0x2424486c, 0x5c5cb8e4,
+ 0xc2c29f5d, 0xd3d3bd6e, 0xacac43ef, 0x6262c4a6,
+ 0x919139a8, 0x959531a4, 0xe4e4d337, 0x7979f28b,
+ 0xe7e7d532, 0xc8c88b43, 0x37376e59, 0x6d6ddab7,
+ 0x8d8d018c, 0xd5d5b164, 0x4e4e9cd2, 0xa9a949e0,
+ 0x6c6cd8b4, 0x5656acfa, 0xf4f4f307, 0xeaeacf25,
+ 0x6565caaf, 0x7a7af48e, 0xaeae47e9, 0x08081018,
+ 0xbaba6fd5, 0x7878f088, 0x25254a6f, 0x2e2e5c72,
+ 0x1c1c3824, 0xa6a657f1, 0xb4b473c7, 0xc6c69751,
+ 0xe8e8cb23, 0xdddda17c, 0x7474e89c, 0x1f1f3e21,
+ 0x4b4b96dd, 0xbdbd61dc, 0x8b8b0d86, 0x8a8a0f85,
+ 0x7070e090, 0x3e3e7c42, 0xb5b571c4, 0x6666ccaa,
+ 0x484890d8, 0x03030605, 0xf6f6f701, 0x0e0e1c12,
+ 0x6161c2a3, 0x35356a5f, 0x5757aef9, 0xb9b969d0,
+ 0x86861791, 0xc1c19958, 0x1d1d3a27, 0x9e9e27b9,
+ 0xe1e1d938, 0xf8f8eb13, 0x98982bb3, 0x11112233,
+ 0x6969d2bb, 0xd9d9a970, 0x8e8e0789, 0x949433a7,
+ 0x9b9b2db6, 0x1e1e3c22, 0x87871592, 0xe9e9c920,
+ 0xcece8749, 0x5555aaff, 0x28285078, 0xdfdfa57a,
+ 0x8c8c038f, 0xa1a159f8, 0x89890980, 0x0d0d1a17,
+ 0xbfbf65da, 0xe6e6d731, 0x424284c6, 0x6868d0b8,
+ 0x414182c3, 0x999929b0, 0x2d2d5a77, 0x0f0f1e11,
+ 0xb0b07bcb, 0x5454a8fc, 0xbbbb6dd6, 0x16162c3a
+}, {
+ 0x63c6a563, 0x7cf8847c, 0x77ee9977, 0x7bf68d7b,
+ 0xf2ff0df2, 0x6bd6bd6b, 0x6fdeb16f, 0xc59154c5,
+ 0x30605030, 0x01020301, 0x67cea967, 0x2b567d2b,
+ 0xfee719fe, 0xd7b562d7, 0xab4de6ab, 0x76ec9a76,
+ 0xca8f45ca, 0x821f9d82, 0xc98940c9, 0x7dfa877d,
+ 0xfaef15fa, 0x59b2eb59, 0x478ec947, 0xf0fb0bf0,
+ 0xad41ecad, 0xd4b367d4, 0xa25ffda2, 0xaf45eaaf,
+ 0x9c23bf9c, 0xa453f7a4, 0x72e49672, 0xc09b5bc0,
+ 0xb775c2b7, 0xfde11cfd, 0x933dae93, 0x264c6a26,
+ 0x366c5a36, 0x3f7e413f, 0xf7f502f7, 0xcc834fcc,
+ 0x34685c34, 0xa551f4a5, 0xe5d134e5, 0xf1f908f1,
+ 0x71e29371, 0xd8ab73d8, 0x31625331, 0x152a3f15,
+ 0x04080c04, 0xc79552c7, 0x23466523, 0xc39d5ec3,
+ 0x18302818, 0x9637a196, 0x050a0f05, 0x9a2fb59a,
+ 0x070e0907, 0x12243612, 0x801b9b80, 0xe2df3de2,
+ 0xebcd26eb, 0x274e6927, 0xb27fcdb2, 0x75ea9f75,
+ 0x09121b09, 0x831d9e83, 0x2c58742c, 0x1a342e1a,
+ 0x1b362d1b, 0x6edcb26e, 0x5ab4ee5a, 0xa05bfba0,
+ 0x52a4f652, 0x3b764d3b, 0xd6b761d6, 0xb37dceb3,
+ 0x29527b29, 0xe3dd3ee3, 0x2f5e712f, 0x84139784,
+ 0x53a6f553, 0xd1b968d1, 0x00000000, 0xedc12ced,
+ 0x20406020, 0xfce31ffc, 0xb179c8b1, 0x5bb6ed5b,
+ 0x6ad4be6a, 0xcb8d46cb, 0xbe67d9be, 0x39724b39,
+ 0x4a94de4a, 0x4c98d44c, 0x58b0e858, 0xcf854acf,
+ 0xd0bb6bd0, 0xefc52aef, 0xaa4fe5aa, 0xfbed16fb,
+ 0x4386c543, 0x4d9ad74d, 0x33665533, 0x85119485,
+ 0x458acf45, 0xf9e910f9, 0x02040602, 0x7ffe817f,
+ 0x50a0f050, 0x3c78443c, 0x9f25ba9f, 0xa84be3a8,
+ 0x51a2f351, 0xa35dfea3, 0x4080c040, 0x8f058a8f,
+ 0x923fad92, 0x9d21bc9d, 0x38704838, 0xf5f104f5,
+ 0xbc63dfbc, 0xb677c1b6, 0xdaaf75da, 0x21426321,
+ 0x10203010, 0xffe51aff, 0xf3fd0ef3, 0xd2bf6dd2,
+ 0xcd814ccd, 0x0c18140c, 0x13263513, 0xecc32fec,
+ 0x5fbee15f, 0x9735a297, 0x4488cc44, 0x172e3917,
+ 0xc49357c4, 0xa755f2a7, 0x7efc827e, 0x3d7a473d,
+ 0x64c8ac64, 0x5dbae75d, 0x19322b19, 0x73e69573,
+ 0x60c0a060, 0x81199881, 0x4f9ed14f, 0xdca37fdc,
+ 0x22446622, 0x2a547e2a, 0x903bab90, 0x880b8388,
+ 0x468cca46, 0xeec729ee, 0xb86bd3b8, 0x14283c14,
+ 0xdea779de, 0x5ebce25e, 0x0b161d0b, 0xdbad76db,
+ 0xe0db3be0, 0x32645632, 0x3a744e3a, 0x0a141e0a,
+ 0x4992db49, 0x060c0a06, 0x24486c24, 0x5cb8e45c,
+ 0xc29f5dc2, 0xd3bd6ed3, 0xac43efac, 0x62c4a662,
+ 0x9139a891, 0x9531a495, 0xe4d337e4, 0x79f28b79,
+ 0xe7d532e7, 0xc88b43c8, 0x376e5937, 0x6ddab76d,
+ 0x8d018c8d, 0xd5b164d5, 0x4e9cd24e, 0xa949e0a9,
+ 0x6cd8b46c, 0x56acfa56, 0xf4f307f4, 0xeacf25ea,
+ 0x65caaf65, 0x7af48e7a, 0xae47e9ae, 0x08101808,
+ 0xba6fd5ba, 0x78f08878, 0x254a6f25, 0x2e5c722e,
+ 0x1c38241c, 0xa657f1a6, 0xb473c7b4, 0xc69751c6,
+ 0xe8cb23e8, 0xdda17cdd, 0x74e89c74, 0x1f3e211f,
+ 0x4b96dd4b, 0xbd61dcbd, 0x8b0d868b, 0x8a0f858a,
+ 0x70e09070, 0x3e7c423e, 0xb571c4b5, 0x66ccaa66,
+ 0x4890d848, 0x03060503, 0xf6f701f6, 0x0e1c120e,
+ 0x61c2a361, 0x356a5f35, 0x57aef957, 0xb969d0b9,
+ 0x86179186, 0xc19958c1, 0x1d3a271d, 0x9e27b99e,
+ 0xe1d938e1, 0xf8eb13f8, 0x982bb398, 0x11223311,
+ 0x69d2bb69, 0xd9a970d9, 0x8e07898e, 0x9433a794,
+ 0x9b2db69b, 0x1e3c221e, 0x87159287, 0xe9c920e9,
+ 0xce8749ce, 0x55aaff55, 0x28507828, 0xdfa57adf,
+ 0x8c038f8c, 0xa159f8a1, 0x89098089, 0x0d1a170d,
+ 0xbf65dabf, 0xe6d731e6, 0x4284c642, 0x68d0b868,
+ 0x4182c341, 0x9929b099, 0x2d5a772d, 0x0f1e110f,
+ 0xb07bcbb0, 0x54a8fc54, 0xbb6dd6bb, 0x162c3a16
+}, {
+ 0xc6a56363, 0xf8847c7c, 0xee997777, 0xf68d7b7b,
+ 0xff0df2f2, 0xd6bd6b6b, 0xdeb16f6f, 0x9154c5c5,
+ 0x60503030, 0x02030101, 0xcea96767, 0x567d2b2b,
+ 0xe719fefe, 0xb562d7d7, 0x4de6abab, 0xec9a7676,
+ 0x8f45caca, 0x1f9d8282, 0x8940c9c9, 0xfa877d7d,
+ 0xef15fafa, 0xb2eb5959, 0x8ec94747, 0xfb0bf0f0,
+ 0x41ecadad, 0xb367d4d4, 0x5ffda2a2, 0x45eaafaf,
+ 0x23bf9c9c, 0x53f7a4a4, 0xe4967272, 0x9b5bc0c0,
+ 0x75c2b7b7, 0xe11cfdfd, 0x3dae9393, 0x4c6a2626,
+ 0x6c5a3636, 0x7e413f3f, 0xf502f7f7, 0x834fcccc,
+ 0x685c3434, 0x51f4a5a5, 0xd134e5e5, 0xf908f1f1,
+ 0xe2937171, 0xab73d8d8, 0x62533131, 0x2a3f1515,
+ 0x080c0404, 0x9552c7c7, 0x46652323, 0x9d5ec3c3,
+ 0x30281818, 0x37a19696, 0x0a0f0505, 0x2fb59a9a,
+ 0x0e090707, 0x24361212, 0x1b9b8080, 0xdf3de2e2,
+ 0xcd26ebeb, 0x4e692727, 0x7fcdb2b2, 0xea9f7575,
+ 0x121b0909, 0x1d9e8383, 0x58742c2c, 0x342e1a1a,
+ 0x362d1b1b, 0xdcb26e6e, 0xb4ee5a5a, 0x5bfba0a0,
+ 0xa4f65252, 0x764d3b3b, 0xb761d6d6, 0x7dceb3b3,
+ 0x527b2929, 0xdd3ee3e3, 0x5e712f2f, 0x13978484,
+ 0xa6f55353, 0xb968d1d1, 0x00000000, 0xc12ceded,
+ 0x40602020, 0xe31ffcfc, 0x79c8b1b1, 0xb6ed5b5b,
+ 0xd4be6a6a, 0x8d46cbcb, 0x67d9bebe, 0x724b3939,
+ 0x94de4a4a, 0x98d44c4c, 0xb0e85858, 0x854acfcf,
+ 0xbb6bd0d0, 0xc52aefef, 0x4fe5aaaa, 0xed16fbfb,
+ 0x86c54343, 0x9ad74d4d, 0x66553333, 0x11948585,
+ 0x8acf4545, 0xe910f9f9, 0x04060202, 0xfe817f7f,
+ 0xa0f05050, 0x78443c3c, 0x25ba9f9f, 0x4be3a8a8,
+ 0xa2f35151, 0x5dfea3a3, 0x80c04040, 0x058a8f8f,
+ 0x3fad9292, 0x21bc9d9d, 0x70483838, 0xf104f5f5,
+ 0x63dfbcbc, 0x77c1b6b6, 0xaf75dada, 0x42632121,
+ 0x20301010, 0xe51affff, 0xfd0ef3f3, 0xbf6dd2d2,
+ 0x814ccdcd, 0x18140c0c, 0x26351313, 0xc32fecec,
+ 0xbee15f5f, 0x35a29797, 0x88cc4444, 0x2e391717,
+ 0x9357c4c4, 0x55f2a7a7, 0xfc827e7e, 0x7a473d3d,
+ 0xc8ac6464, 0xbae75d5d, 0x322b1919, 0xe6957373,
+ 0xc0a06060, 0x19988181, 0x9ed14f4f, 0xa37fdcdc,
+ 0x44662222, 0x547e2a2a, 0x3bab9090, 0x0b838888,
+ 0x8cca4646, 0xc729eeee, 0x6bd3b8b8, 0x283c1414,
+ 0xa779dede, 0xbce25e5e, 0x161d0b0b, 0xad76dbdb,
+ 0xdb3be0e0, 0x64563232, 0x744e3a3a, 0x141e0a0a,
+ 0x92db4949, 0x0c0a0606, 0x486c2424, 0xb8e45c5c,
+ 0x9f5dc2c2, 0xbd6ed3d3, 0x43efacac, 0xc4a66262,
+ 0x39a89191, 0x31a49595, 0xd337e4e4, 0xf28b7979,
+ 0xd532e7e7, 0x8b43c8c8, 0x6e593737, 0xdab76d6d,
+ 0x018c8d8d, 0xb164d5d5, 0x9cd24e4e, 0x49e0a9a9,
+ 0xd8b46c6c, 0xacfa5656, 0xf307f4f4, 0xcf25eaea,
+ 0xcaaf6565, 0xf48e7a7a, 0x47e9aeae, 0x10180808,
+ 0x6fd5baba, 0xf0887878, 0x4a6f2525, 0x5c722e2e,
+ 0x38241c1c, 0x57f1a6a6, 0x73c7b4b4, 0x9751c6c6,
+ 0xcb23e8e8, 0xa17cdddd, 0xe89c7474, 0x3e211f1f,
+ 0x96dd4b4b, 0x61dcbdbd, 0x0d868b8b, 0x0f858a8a,
+ 0xe0907070, 0x7c423e3e, 0x71c4b5b5, 0xccaa6666,
+ 0x90d84848, 0x06050303, 0xf701f6f6, 0x1c120e0e,
+ 0xc2a36161, 0x6a5f3535, 0xaef95757, 0x69d0b9b9,
+ 0x17918686, 0x9958c1c1, 0x3a271d1d, 0x27b99e9e,
+ 0xd938e1e1, 0xeb13f8f8, 0x2bb39898, 0x22331111,
+ 0xd2bb6969, 0xa970d9d9, 0x07898e8e, 0x33a79494,
+ 0x2db69b9b, 0x3c221e1e, 0x15928787, 0xc920e9e9,
+ 0x8749cece, 0xaaff5555, 0x50782828, 0xa57adfdf,
+ 0x038f8c8c, 0x59f8a1a1, 0x09808989, 0x1a170d0d,
+ 0x65dabfbf, 0xd731e6e6, 0x84c64242, 0xd0b86868,
+ 0x82c34141, 0x29b09999, 0x5a772d2d, 0x1e110f0f,
+ 0x7bcbb0b0, 0xa8fc5454, 0x6dd6bbbb, 0x2c3a1616
+}
+};
+
+static const u4byte it_tab[4][256] = {
+{
+ 0x50a7f451, 0x5365417e, 0xc3a4171a, 0x965e273a,
+ 0xcb6bab3b, 0xf1459d1f, 0xab58faac, 0x9303e34b,
+ 0x55fa3020, 0xf66d76ad, 0x9176cc88, 0x254c02f5,
+ 0xfcd7e54f, 0xd7cb2ac5, 0x80443526, 0x8fa362b5,
+ 0x495ab1de, 0x671bba25, 0x980eea45, 0xe1c0fe5d,
+ 0x02752fc3, 0x12f04c81, 0xa397468d, 0xc6f9d36b,
+ 0xe75f8f03, 0x959c9215, 0xeb7a6dbf, 0xda595295,
+ 0x2d83bed4, 0xd3217458, 0x2969e049, 0x44c8c98e,
+ 0x6a89c275, 0x78798ef4, 0x6b3e5899, 0xdd71b927,
+ 0xb64fe1be, 0x17ad88f0, 0x66ac20c9, 0xb43ace7d,
+ 0x184adf63, 0x82311ae5, 0x60335197, 0x457f5362,
+ 0xe07764b1, 0x84ae6bbb, 0x1ca081fe, 0x942b08f9,
+ 0x58684870, 0x19fd458f, 0x876cde94, 0xb7f87b52,
+ 0x23d373ab, 0xe2024b72, 0x578f1fe3, 0x2aab5566,
+ 0x0728ebb2, 0x03c2b52f, 0x9a7bc586, 0xa50837d3,
+ 0xf2872830, 0xb2a5bf23, 0xba6a0302, 0x5c8216ed,
+ 0x2b1ccf8a, 0x92b479a7, 0xf0f207f3, 0xa1e2694e,
+ 0xcdf4da65, 0xd5be0506, 0x1f6234d1, 0x8afea6c4,
+ 0x9d532e34, 0xa055f3a2, 0x32e18a05, 0x75ebf6a4,
+ 0x39ec830b, 0xaaef6040, 0x069f715e, 0x51106ebd,
+ 0xf98a213e, 0x3d06dd96, 0xae053edd, 0x46bde64d,
+ 0xb58d5491, 0x055dc471, 0x6fd40604, 0xff155060,
+ 0x24fb9819, 0x97e9bdd6, 0xcc434089, 0x779ed967,
+ 0xbd42e8b0, 0x888b8907, 0x385b19e7, 0xdbeec879,
+ 0x470a7ca1, 0xe90f427c, 0xc91e84f8, 0x00000000,
+ 0x83868009, 0x48ed2b32, 0xac70111e, 0x4e725a6c,
+ 0xfbff0efd, 0x5638850f, 0x1ed5ae3d, 0x27392d36,
+ 0x64d90f0a, 0x21a65c68, 0xd1545b9b, 0x3a2e3624,
+ 0xb1670a0c, 0x0fe75793, 0xd296eeb4, 0x9e919b1b,
+ 0x4fc5c080, 0xa220dc61, 0x694b775a, 0x161a121c,
+ 0x0aba93e2, 0xe52aa0c0, 0x43e0223c, 0x1d171b12,
+ 0x0b0d090e, 0xadc78bf2, 0xb9a8b62d, 0xc8a91e14,
+ 0x8519f157, 0x4c0775af, 0xbbdd99ee, 0xfd607fa3,
+ 0x9f2601f7, 0xbcf5725c, 0xc53b6644, 0x347efb5b,
+ 0x7629438b, 0xdcc623cb, 0x68fcedb6, 0x63f1e4b8,
+ 0xcadc31d7, 0x10856342, 0x40229713, 0x2011c684,
+ 0x7d244a85, 0xf83dbbd2, 0x1132f9ae, 0x6da129c7,
+ 0x4b2f9e1d, 0xf330b2dc, 0xec52860d, 0xd0e3c177,
+ 0x6c16b32b, 0x99b970a9, 0xfa489411, 0x2264e947,
+ 0xc48cfca8, 0x1a3ff0a0, 0xd82c7d56, 0xef903322,
+ 0xc74e4987, 0xc1d138d9, 0xfea2ca8c, 0x360bd498,
+ 0xcf81f5a6, 0x28de7aa5, 0x268eb7da, 0xa4bfad3f,
+ 0xe49d3a2c, 0x0d927850, 0x9bcc5f6a, 0x62467e54,
+ 0xc2138df6, 0xe8b8d890, 0x5ef7392e, 0xf5afc382,
+ 0xbe805d9f, 0x7c93d069, 0xa92dd56f, 0xb31225cf,
+ 0x3b99acc8, 0xa77d1810, 0x6e639ce8, 0x7bbb3bdb,
+ 0x097826cd, 0xf418596e, 0x01b79aec, 0xa89a4f83,
+ 0x656e95e6, 0x7ee6ffaa, 0x08cfbc21, 0xe6e815ef,
+ 0xd99be7ba, 0xce366f4a, 0xd4099fea, 0xd67cb029,
+ 0xafb2a431, 0x31233f2a, 0x3094a5c6, 0xc066a235,
+ 0x37bc4e74, 0xa6ca82fc, 0xb0d090e0, 0x15d8a733,
+ 0x4a9804f1, 0xf7daec41, 0x0e50cd7f, 0x2ff69117,
+ 0x8dd64d76, 0x4db0ef43, 0x544daacc, 0xdf0496e4,
+ 0xe3b5d19e, 0x1b886a4c, 0xb81f2cc1, 0x7f516546,
+ 0x04ea5e9d, 0x5d358c01, 0x737487fa, 0x2e410bfb,
+ 0x5a1d67b3, 0x52d2db92, 0x335610e9, 0x1347d66d,
+ 0x8c61d79a, 0x7a0ca137, 0x8e14f859, 0x893c13eb,
+ 0xee27a9ce, 0x35c961b7, 0xede51ce1, 0x3cb1477a,
+ 0x59dfd29c, 0x3f73f255, 0x79ce1418, 0xbf37c773,
+ 0xeacdf753, 0x5baafd5f, 0x146f3ddf, 0x86db4478,
+ 0x81f3afca, 0x3ec468b9, 0x2c342438, 0x5f40a3c2,
+ 0x72c31d16, 0x0c25e2bc, 0x8b493c28, 0x41950dff,
+ 0x7101a839, 0xdeb30c08, 0x9ce4b4d8, 0x90c15664,
+ 0x6184cb7b, 0x70b632d5, 0x745c6c48, 0x4257b8d0
+}, {
+ 0xa7f45150, 0x65417e53, 0xa4171ac3, 0x5e273a96,
+ 0x6bab3bcb, 0x459d1ff1, 0x58faacab, 0x03e34b93,
+ 0xfa302055, 0x6d76adf6, 0x76cc8891, 0x4c02f525,
+ 0xd7e54ffc, 0xcb2ac5d7, 0x44352680, 0xa362b58f,
+ 0x5ab1de49, 0x1bba2567, 0x0eea4598, 0xc0fe5de1,
+ 0x752fc302, 0xf04c8112, 0x97468da3, 0xf9d36bc6,
+ 0x5f8f03e7, 0x9c921595, 0x7a6dbfeb, 0x595295da,
+ 0x83bed42d, 0x217458d3, 0x69e04929, 0xc8c98e44,
+ 0x89c2756a, 0x798ef478, 0x3e58996b, 0x71b927dd,
+ 0x4fe1beb6, 0xad88f017, 0xac20c966, 0x3ace7db4,
+ 0x4adf6318, 0x311ae582, 0x33519760, 0x7f536245,
+ 0x7764b1e0, 0xae6bbb84, 0xa081fe1c, 0x2b08f994,
+ 0x68487058, 0xfd458f19, 0x6cde9487, 0xf87b52b7,
+ 0xd373ab23, 0x024b72e2, 0x8f1fe357, 0xab55662a,
+ 0x28ebb207, 0xc2b52f03, 0x7bc5869a, 0x0837d3a5,
+ 0x872830f2, 0xa5bf23b2, 0x6a0302ba, 0x8216ed5c,
+ 0x1ccf8a2b, 0xb479a792, 0xf207f3f0, 0xe2694ea1,
+ 0xf4da65cd, 0xbe0506d5, 0x6234d11f, 0xfea6c48a,
+ 0x532e349d, 0x55f3a2a0, 0xe18a0532, 0xebf6a475,
+ 0xec830b39, 0xef6040aa, 0x9f715e06, 0x106ebd51,
+ 0x8a213ef9, 0x06dd963d, 0x053eddae, 0xbde64d46,
+ 0x8d5491b5, 0x5dc47105, 0xd406046f, 0x155060ff,
+ 0xfb981924, 0xe9bdd697, 0x434089cc, 0x9ed96777,
+ 0x42e8b0bd, 0x8b890788, 0x5b19e738, 0xeec879db,
+ 0x0a7ca147, 0x0f427ce9, 0x1e84f8c9, 0x00000000,
+ 0x86800983, 0xed2b3248, 0x70111eac, 0x725a6c4e,
+ 0xff0efdfb, 0x38850f56, 0xd5ae3d1e, 0x392d3627,
+ 0xd90f0a64, 0xa65c6821, 0x545b9bd1, 0x2e36243a,
+ 0x670a0cb1, 0xe757930f, 0x96eeb4d2, 0x919b1b9e,
+ 0xc5c0804f, 0x20dc61a2, 0x4b775a69, 0x1a121c16,
+ 0xba93e20a, 0x2aa0c0e5, 0xe0223c43, 0x171b121d,
+ 0x0d090e0b, 0xc78bf2ad, 0xa8b62db9, 0xa91e14c8,
+ 0x19f15785, 0x0775af4c, 0xdd99eebb, 0x607fa3fd,
+ 0x2601f79f, 0xf5725cbc, 0x3b6644c5, 0x7efb5b34,
+ 0x29438b76, 0xc623cbdc, 0xfcedb668, 0xf1e4b863,
+ 0xdc31d7ca, 0x85634210, 0x22971340, 0x11c68420,
+ 0x244a857d, 0x3dbbd2f8, 0x32f9ae11, 0xa129c76d,
+ 0x2f9e1d4b, 0x30b2dcf3, 0x52860dec, 0xe3c177d0,
+ 0x16b32b6c, 0xb970a999, 0x489411fa, 0x64e94722,
+ 0x8cfca8c4, 0x3ff0a01a, 0x2c7d56d8, 0x903322ef,
+ 0x4e4987c7, 0xd138d9c1, 0xa2ca8cfe, 0x0bd49836,
+ 0x81f5a6cf, 0xde7aa528, 0x8eb7da26, 0xbfad3fa4,
+ 0x9d3a2ce4, 0x9278500d, 0xcc5f6a9b, 0x467e5462,
+ 0x138df6c2, 0xb8d890e8, 0xf7392e5e, 0xafc382f5,
+ 0x805d9fbe, 0x93d0697c, 0x2dd56fa9, 0x1225cfb3,
+ 0x99acc83b, 0x7d1810a7, 0x639ce86e, 0xbb3bdb7b,
+ 0x7826cd09, 0x18596ef4, 0xb79aec01, 0x9a4f83a8,
+ 0x6e95e665, 0xe6ffaa7e, 0xcfbc2108, 0xe815efe6,
+ 0x9be7bad9, 0x366f4ace, 0x099fead4, 0x7cb029d6,
+ 0xb2a431af, 0x233f2a31, 0x94a5c630, 0x66a235c0,
+ 0xbc4e7437, 0xca82fca6, 0xd090e0b0, 0xd8a73315,
+ 0x9804f14a, 0xdaec41f7, 0x50cd7f0e, 0xf691172f,
+ 0xd64d768d, 0xb0ef434d, 0x4daacc54, 0x0496e4df,
+ 0xb5d19ee3, 0x886a4c1b, 0x1f2cc1b8, 0x5165467f,
+ 0xea5e9d04, 0x358c015d, 0x7487fa73, 0x410bfb2e,
+ 0x1d67b35a, 0xd2db9252, 0x5610e933, 0x47d66d13,
+ 0x61d79a8c, 0x0ca1377a, 0x14f8598e, 0x3c13eb89,
+ 0x27a9ceee, 0xc961b735, 0xe51ce1ed, 0xb1477a3c,
+ 0xdfd29c59, 0x73f2553f, 0xce141879, 0x37c773bf,
+ 0xcdf753ea, 0xaafd5f5b, 0x6f3ddf14, 0xdb447886,
+ 0xf3afca81, 0xc468b93e, 0x3424382c, 0x40a3c25f,
+ 0xc31d1672, 0x25e2bc0c, 0x493c288b, 0x950dff41,
+ 0x01a83971, 0xb30c08de, 0xe4b4d89c, 0xc1566490,
+ 0x84cb7b61, 0xb632d570, 0x5c6c4874, 0x57b8d042
+}, {
+ 0xf45150a7, 0x417e5365, 0x171ac3a4, 0x273a965e,
+ 0xab3bcb6b, 0x9d1ff145, 0xfaacab58, 0xe34b9303,
+ 0x302055fa, 0x76adf66d, 0xcc889176, 0x02f5254c,
+ 0xe54ffcd7, 0x2ac5d7cb, 0x35268044, 0x62b58fa3,
+ 0xb1de495a, 0xba25671b, 0xea45980e, 0xfe5de1c0,
+ 0x2fc30275, 0x4c8112f0, 0x468da397, 0xd36bc6f9,
+ 0x8f03e75f, 0x9215959c, 0x6dbfeb7a, 0x5295da59,
+ 0xbed42d83, 0x7458d321, 0xe0492969, 0xc98e44c8,
+ 0xc2756a89, 0x8ef47879, 0x58996b3e, 0xb927dd71,
+ 0xe1beb64f, 0x88f017ad, 0x20c966ac, 0xce7db43a,
+ 0xdf63184a, 0x1ae58231, 0x51976033, 0x5362457f,
+ 0x64b1e077, 0x6bbb84ae, 0x81fe1ca0, 0x08f9942b,
+ 0x48705868, 0x458f19fd, 0xde94876c, 0x7b52b7f8,
+ 0x73ab23d3, 0x4b72e202, 0x1fe3578f, 0x55662aab,
+ 0xebb20728, 0xb52f03c2, 0xc5869a7b, 0x37d3a508,
+ 0x2830f287, 0xbf23b2a5, 0x0302ba6a, 0x16ed5c82,
+ 0xcf8a2b1c, 0x79a792b4, 0x07f3f0f2, 0x694ea1e2,
+ 0xda65cdf4, 0x0506d5be, 0x34d11f62, 0xa6c48afe,
+ 0x2e349d53, 0xf3a2a055, 0x8a0532e1, 0xf6a475eb,
+ 0x830b39ec, 0x6040aaef, 0x715e069f, 0x6ebd5110,
+ 0x213ef98a, 0xdd963d06, 0x3eddae05, 0xe64d46bd,
+ 0x5491b58d, 0xc471055d, 0x06046fd4, 0x5060ff15,
+ 0x981924fb, 0xbdd697e9, 0x4089cc43, 0xd967779e,
+ 0xe8b0bd42, 0x8907888b, 0x19e7385b, 0xc879dbee,
+ 0x7ca1470a, 0x427ce90f, 0x84f8c91e, 0x00000000,
+ 0x80098386, 0x2b3248ed, 0x111eac70, 0x5a6c4e72,
+ 0x0efdfbff, 0x850f5638, 0xae3d1ed5, 0x2d362739,
+ 0x0f0a64d9, 0x5c6821a6, 0x5b9bd154, 0x36243a2e,
+ 0x0a0cb167, 0x57930fe7, 0xeeb4d296, 0x9b1b9e91,
+ 0xc0804fc5, 0xdc61a220, 0x775a694b, 0x121c161a,
+ 0x93e20aba, 0xa0c0e52a, 0x223c43e0, 0x1b121d17,
+ 0x090e0b0d, 0x8bf2adc7, 0xb62db9a8, 0x1e14c8a9,
+ 0xf1578519, 0x75af4c07, 0x99eebbdd, 0x7fa3fd60,
+ 0x01f79f26, 0x725cbcf5, 0x6644c53b, 0xfb5b347e,
+ 0x438b7629, 0x23cbdcc6, 0xedb668fc, 0xe4b863f1,
+ 0x31d7cadc, 0x63421085, 0x97134022, 0xc6842011,
+ 0x4a857d24, 0xbbd2f83d, 0xf9ae1132, 0x29c76da1,
+ 0x9e1d4b2f, 0xb2dcf330, 0x860dec52, 0xc177d0e3,
+ 0xb32b6c16, 0x70a999b9, 0x9411fa48, 0xe9472264,
+ 0xfca8c48c, 0xf0a01a3f, 0x7d56d82c, 0x3322ef90,
+ 0x4987c74e, 0x38d9c1d1, 0xca8cfea2, 0xd498360b,
+ 0xf5a6cf81, 0x7aa528de, 0xb7da268e, 0xad3fa4bf,
+ 0x3a2ce49d, 0x78500d92, 0x5f6a9bcc, 0x7e546246,
+ 0x8df6c213, 0xd890e8b8, 0x392e5ef7, 0xc382f5af,
+ 0x5d9fbe80, 0xd0697c93, 0xd56fa92d, 0x25cfb312,
+ 0xacc83b99, 0x1810a77d, 0x9ce86e63, 0x3bdb7bbb,
+ 0x26cd0978, 0x596ef418, 0x9aec01b7, 0x4f83a89a,
+ 0x95e6656e, 0xffaa7ee6, 0xbc2108cf, 0x15efe6e8,
+ 0xe7bad99b, 0x6f4ace36, 0x9fead409, 0xb029d67c,
+ 0xa431afb2, 0x3f2a3123, 0xa5c63094, 0xa235c066,
+ 0x4e7437bc, 0x82fca6ca, 0x90e0b0d0, 0xa73315d8,
+ 0x04f14a98, 0xec41f7da, 0xcd7f0e50, 0x91172ff6,
+ 0x4d768dd6, 0xef434db0, 0xaacc544d, 0x96e4df04,
+ 0xd19ee3b5, 0x6a4c1b88, 0x2cc1b81f, 0x65467f51,
+ 0x5e9d04ea, 0x8c015d35, 0x87fa7374, 0x0bfb2e41,
+ 0x67b35a1d, 0xdb9252d2, 0x10e93356, 0xd66d1347,
+ 0xd79a8c61, 0xa1377a0c, 0xf8598e14, 0x13eb893c,
+ 0xa9ceee27, 0x61b735c9, 0x1ce1ede5, 0x477a3cb1,
+ 0xd29c59df, 0xf2553f73, 0x141879ce, 0xc773bf37,
+ 0xf753eacd, 0xfd5f5baa, 0x3ddf146f, 0x447886db,
+ 0xafca81f3, 0x68b93ec4, 0x24382c34, 0xa3c25f40,
+ 0x1d1672c3, 0xe2bc0c25, 0x3c288b49, 0x0dff4195,
+ 0xa8397101, 0x0c08deb3, 0xb4d89ce4, 0x566490c1,
+ 0xcb7b6184, 0x32d570b6, 0x6c48745c, 0xb8d04257
+}, {
+ 0x5150a7f4, 0x7e536541, 0x1ac3a417, 0x3a965e27,
+ 0x3bcb6bab, 0x1ff1459d, 0xacab58fa, 0x4b9303e3,
+ 0x2055fa30, 0xadf66d76, 0x889176cc, 0xf5254c02,
+ 0x4ffcd7e5, 0xc5d7cb2a, 0x26804435, 0xb58fa362,
+ 0xde495ab1, 0x25671bba, 0x45980eea, 0x5de1c0fe,
+ 0xc302752f, 0x8112f04c, 0x8da39746, 0x6bc6f9d3,
+ 0x03e75f8f, 0x15959c92, 0xbfeb7a6d, 0x95da5952,
+ 0xd42d83be, 0x58d32174, 0x492969e0, 0x8e44c8c9,
+ 0x756a89c2, 0xf478798e, 0x996b3e58, 0x27dd71b9,
+ 0xbeb64fe1, 0xf017ad88, 0xc966ac20, 0x7db43ace,
+ 0x63184adf, 0xe582311a, 0x97603351, 0x62457f53,
+ 0xb1e07764, 0xbb84ae6b, 0xfe1ca081, 0xf9942b08,
+ 0x70586848, 0x8f19fd45, 0x94876cde, 0x52b7f87b,
+ 0xab23d373, 0x72e2024b, 0xe3578f1f, 0x662aab55,
+ 0xb20728eb, 0x2f03c2b5, 0x869a7bc5, 0xd3a50837,
+ 0x30f28728, 0x23b2a5bf, 0x02ba6a03, 0xed5c8216,
+ 0x8a2b1ccf, 0xa792b479, 0xf3f0f207, 0x4ea1e269,
+ 0x65cdf4da, 0x06d5be05, 0xd11f6234, 0xc48afea6,
+ 0x349d532e, 0xa2a055f3, 0x0532e18a, 0xa475ebf6,
+ 0x0b39ec83, 0x40aaef60, 0x5e069f71, 0xbd51106e,
+ 0x3ef98a21, 0x963d06dd, 0xddae053e, 0x4d46bde6,
+ 0x91b58d54, 0x71055dc4, 0x046fd406, 0x60ff1550,
+ 0x1924fb98, 0xd697e9bd, 0x89cc4340, 0x67779ed9,
+ 0xb0bd42e8, 0x07888b89, 0xe7385b19, 0x79dbeec8,
+ 0xa1470a7c, 0x7ce90f42, 0xf8c91e84, 0x00000000,
+ 0x09838680, 0x3248ed2b, 0x1eac7011, 0x6c4e725a,
+ 0xfdfbff0e, 0x0f563885, 0x3d1ed5ae, 0x3627392d,
+ 0x0a64d90f, 0x6821a65c, 0x9bd1545b, 0x243a2e36,
+ 0x0cb1670a, 0x930fe757, 0xb4d296ee, 0x1b9e919b,
+ 0x804fc5c0, 0x61a220dc, 0x5a694b77, 0x1c161a12,
+ 0xe20aba93, 0xc0e52aa0, 0x3c43e022, 0x121d171b,
+ 0x0e0b0d09, 0xf2adc78b, 0x2db9a8b6, 0x14c8a91e,
+ 0x578519f1, 0xaf4c0775, 0xeebbdd99, 0xa3fd607f,
+ 0xf79f2601, 0x5cbcf572, 0x44c53b66, 0x5b347efb,
+ 0x8b762943, 0xcbdcc623, 0xb668fced, 0xb863f1e4,
+ 0xd7cadc31, 0x42108563, 0x13402297, 0x842011c6,
+ 0x857d244a, 0xd2f83dbb, 0xae1132f9, 0xc76da129,
+ 0x1d4b2f9e, 0xdcf330b2, 0x0dec5286, 0x77d0e3c1,
+ 0x2b6c16b3, 0xa999b970, 0x11fa4894, 0x472264e9,
+ 0xa8c48cfc, 0xa01a3ff0, 0x56d82c7d, 0x22ef9033,
+ 0x87c74e49, 0xd9c1d138, 0x8cfea2ca, 0x98360bd4,
+ 0xa6cf81f5, 0xa528de7a, 0xda268eb7, 0x3fa4bfad,
+ 0x2ce49d3a, 0x500d9278, 0x6a9bcc5f, 0x5462467e,
+ 0xf6c2138d, 0x90e8b8d8, 0x2e5ef739, 0x82f5afc3,
+ 0x9fbe805d, 0x697c93d0, 0x6fa92dd5, 0xcfb31225,
+ 0xc83b99ac, 0x10a77d18, 0xe86e639c, 0xdb7bbb3b,
+ 0xcd097826, 0x6ef41859, 0xec01b79a, 0x83a89a4f,
+ 0xe6656e95, 0xaa7ee6ff, 0x2108cfbc, 0xefe6e815,
+ 0xbad99be7, 0x4ace366f, 0xead4099f, 0x29d67cb0,
+ 0x31afb2a4, 0x2a31233f, 0xc63094a5, 0x35c066a2,
+ 0x7437bc4e, 0xfca6ca82, 0xe0b0d090, 0x3315d8a7,
+ 0xf14a9804, 0x41f7daec, 0x7f0e50cd, 0x172ff691,
+ 0x768dd64d, 0x434db0ef, 0xcc544daa, 0xe4df0496,
+ 0x9ee3b5d1, 0x4c1b886a, 0xc1b81f2c, 0x467f5165,
+ 0x9d04ea5e, 0x015d358c, 0xfa737487, 0xfb2e410b,
+ 0xb35a1d67, 0x9252d2db, 0xe9335610, 0x6d1347d6,
+ 0x9a8c61d7, 0x377a0ca1, 0x598e14f8, 0xeb893c13,
+ 0xceee27a9, 0xb735c961, 0xe1ede51c, 0x7a3cb147,
+ 0x9c59dfd2, 0x553f73f2, 0x1879ce14, 0x73bf37c7,
+ 0x53eacdf7, 0x5f5baafd, 0xdf146f3d, 0x7886db44,
+ 0xca81f3af, 0xb93ec468, 0x382c3424, 0xc25f40a3,
+ 0x1672c31d, 0xbc0c25e2, 0x288b493c, 0xff41950d,
+ 0x397101a8, 0x08deb30c, 0xd89ce4b4, 0x6490c156,
+ 0x7b6184cb, 0xd570b632, 0x48745c6c, 0xd04257b8
+}
+};
+
+static const u4byte fl_tab[4][256] = {
+{
+ 0x00000063, 0x0000007c, 0x00000077, 0x0000007b,
+ 0x000000f2, 0x0000006b, 0x0000006f, 0x000000c5,
+ 0x00000030, 0x00000001, 0x00000067, 0x0000002b,
+ 0x000000fe, 0x000000d7, 0x000000ab, 0x00000076,
+ 0x000000ca, 0x00000082, 0x000000c9, 0x0000007d,
+ 0x000000fa, 0x00000059, 0x00000047, 0x000000f0,
+ 0x000000ad, 0x000000d4, 0x000000a2, 0x000000af,
+ 0x0000009c, 0x000000a4, 0x00000072, 0x000000c0,
+ 0x000000b7, 0x000000fd, 0x00000093, 0x00000026,
+ 0x00000036, 0x0000003f, 0x000000f7, 0x000000cc,
+ 0x00000034, 0x000000a5, 0x000000e5, 0x000000f1,
+ 0x00000071, 0x000000d8, 0x00000031, 0x00000015,
+ 0x00000004, 0x000000c7, 0x00000023, 0x000000c3,
+ 0x00000018, 0x00000096, 0x00000005, 0x0000009a,
+ 0x00000007, 0x00000012, 0x00000080, 0x000000e2,
+ 0x000000eb, 0x00000027, 0x000000b2, 0x00000075,
+ 0x00000009, 0x00000083, 0x0000002c, 0x0000001a,
+ 0x0000001b, 0x0000006e, 0x0000005a, 0x000000a0,
+ 0x00000052, 0x0000003b, 0x000000d6, 0x000000b3,
+ 0x00000029, 0x000000e3, 0x0000002f, 0x00000084,
+ 0x00000053, 0x000000d1, 0x00000000, 0x000000ed,
+ 0x00000020, 0x000000fc, 0x000000b1, 0x0000005b,
+ 0x0000006a, 0x000000cb, 0x000000be, 0x00000039,
+ 0x0000004a, 0x0000004c, 0x00000058, 0x000000cf,
+ 0x000000d0, 0x000000ef, 0x000000aa, 0x000000fb,
+ 0x00000043, 0x0000004d, 0x00000033, 0x00000085,
+ 0x00000045, 0x000000f9, 0x00000002, 0x0000007f,
+ 0x00000050, 0x0000003c, 0x0000009f, 0x000000a8,
+ 0x00000051, 0x000000a3, 0x00000040, 0x0000008f,
+ 0x00000092, 0x0000009d, 0x00000038, 0x000000f5,
+ 0x000000bc, 0x000000b6, 0x000000da, 0x00000021,
+ 0x00000010, 0x000000ff, 0x000000f3, 0x000000d2,
+ 0x000000cd, 0x0000000c, 0x00000013, 0x000000ec,
+ 0x0000005f, 0x00000097, 0x00000044, 0x00000017,
+ 0x000000c4, 0x000000a7, 0x0000007e, 0x0000003d,
+ 0x00000064, 0x0000005d, 0x00000019, 0x00000073,
+ 0x00000060, 0x00000081, 0x0000004f, 0x000000dc,
+ 0x00000022, 0x0000002a, 0x00000090, 0x00000088,
+ 0x00000046, 0x000000ee, 0x000000b8, 0x00000014,
+ 0x000000de, 0x0000005e, 0x0000000b, 0x000000db,
+ 0x000000e0, 0x00000032, 0x0000003a, 0x0000000a,
+ 0x00000049, 0x00000006, 0x00000024, 0x0000005c,
+ 0x000000c2, 0x000000d3, 0x000000ac, 0x00000062,
+ 0x00000091, 0x00000095, 0x000000e4, 0x00000079,
+ 0x000000e7, 0x000000c8, 0x00000037, 0x0000006d,
+ 0x0000008d, 0x000000d5, 0x0000004e, 0x000000a9,
+ 0x0000006c, 0x00000056, 0x000000f4, 0x000000ea,
+ 0x00000065, 0x0000007a, 0x000000ae, 0x00000008,
+ 0x000000ba, 0x00000078, 0x00000025, 0x0000002e,
+ 0x0000001c, 0x000000a6, 0x000000b4, 0x000000c6,
+ 0x000000e8, 0x000000dd, 0x00000074, 0x0000001f,
+ 0x0000004b, 0x000000bd, 0x0000008b, 0x0000008a,
+ 0x00000070, 0x0000003e, 0x000000b5, 0x00000066,
+ 0x00000048, 0x00000003, 0x000000f6, 0x0000000e,
+ 0x00000061, 0x00000035, 0x00000057, 0x000000b9,
+ 0x00000086, 0x000000c1, 0x0000001d, 0x0000009e,
+ 0x000000e1, 0x000000f8, 0x00000098, 0x00000011,
+ 0x00000069, 0x000000d9, 0x0000008e, 0x00000094,
+ 0x0000009b, 0x0000001e, 0x00000087, 0x000000e9,
+ 0x000000ce, 0x00000055, 0x00000028, 0x000000df,
+ 0x0000008c, 0x000000a1, 0x00000089, 0x0000000d,
+ 0x000000bf, 0x000000e6, 0x00000042, 0x00000068,
+ 0x00000041, 0x00000099, 0x0000002d, 0x0000000f,
+ 0x000000b0, 0x00000054, 0x000000bb, 0x00000016
+}, {
+ 0x00006300, 0x00007c00, 0x00007700, 0x00007b00,
+ 0x0000f200, 0x00006b00, 0x00006f00, 0x0000c500,
+ 0x00003000, 0x00000100, 0x00006700, 0x00002b00,
+ 0x0000fe00, 0x0000d700, 0x0000ab00, 0x00007600,
+ 0x0000ca00, 0x00008200, 0x0000c900, 0x00007d00,
+ 0x0000fa00, 0x00005900, 0x00004700, 0x0000f000,
+ 0x0000ad00, 0x0000d400, 0x0000a200, 0x0000af00,
+ 0x00009c00, 0x0000a400, 0x00007200, 0x0000c000,
+ 0x0000b700, 0x0000fd00, 0x00009300, 0x00002600,
+ 0x00003600, 0x00003f00, 0x0000f700, 0x0000cc00,
+ 0x00003400, 0x0000a500, 0x0000e500, 0x0000f100,
+ 0x00007100, 0x0000d800, 0x00003100, 0x00001500,
+ 0x00000400, 0x0000c700, 0x00002300, 0x0000c300,
+ 0x00001800, 0x00009600, 0x00000500, 0x00009a00,
+ 0x00000700, 0x00001200, 0x00008000, 0x0000e200,
+ 0x0000eb00, 0x00002700, 0x0000b200, 0x00007500,
+ 0x00000900, 0x00008300, 0x00002c00, 0x00001a00,
+ 0x00001b00, 0x00006e00, 0x00005a00, 0x0000a000,
+ 0x00005200, 0x00003b00, 0x0000d600, 0x0000b300,
+ 0x00002900, 0x0000e300, 0x00002f00, 0x00008400,
+ 0x00005300, 0x0000d100, 0x00000000, 0x0000ed00,
+ 0x00002000, 0x0000fc00, 0x0000b100, 0x00005b00,
+ 0x00006a00, 0x0000cb00, 0x0000be00, 0x00003900,
+ 0x00004a00, 0x00004c00, 0x00005800, 0x0000cf00,
+ 0x0000d000, 0x0000ef00, 0x0000aa00, 0x0000fb00,
+ 0x00004300, 0x00004d00, 0x00003300, 0x00008500,
+ 0x00004500, 0x0000f900, 0x00000200, 0x00007f00,
+ 0x00005000, 0x00003c00, 0x00009f00, 0x0000a800,
+ 0x00005100, 0x0000a300, 0x00004000, 0x00008f00,
+ 0x00009200, 0x00009d00, 0x00003800, 0x0000f500,
+ 0x0000bc00, 0x0000b600, 0x0000da00, 0x00002100,
+ 0x00001000, 0x0000ff00, 0x0000f300, 0x0000d200,
+ 0x0000cd00, 0x00000c00, 0x00001300, 0x0000ec00,
+ 0x00005f00, 0x00009700, 0x00004400, 0x00001700,
+ 0x0000c400, 0x0000a700, 0x00007e00, 0x00003d00,
+ 0x00006400, 0x00005d00, 0x00001900, 0x00007300,
+ 0x00006000, 0x00008100, 0x00004f00, 0x0000dc00,
+ 0x00002200, 0x00002a00, 0x00009000, 0x00008800,
+ 0x00004600, 0x0000ee00, 0x0000b800, 0x00001400,
+ 0x0000de00, 0x00005e00, 0x00000b00, 0x0000db00,
+ 0x0000e000, 0x00003200, 0x00003a00, 0x00000a00,
+ 0x00004900, 0x00000600, 0x00002400, 0x00005c00,
+ 0x0000c200, 0x0000d300, 0x0000ac00, 0x00006200,
+ 0x00009100, 0x00009500, 0x0000e400, 0x00007900,
+ 0x0000e700, 0x0000c800, 0x00003700, 0x00006d00,
+ 0x00008d00, 0x0000d500, 0x00004e00, 0x0000a900,
+ 0x00006c00, 0x00005600, 0x0000f400, 0x0000ea00,
+ 0x00006500, 0x00007a00, 0x0000ae00, 0x00000800,
+ 0x0000ba00, 0x00007800, 0x00002500, 0x00002e00,
+ 0x00001c00, 0x0000a600, 0x0000b400, 0x0000c600,
+ 0x0000e800, 0x0000dd00, 0x00007400, 0x00001f00,
+ 0x00004b00, 0x0000bd00, 0x00008b00, 0x00008a00,
+ 0x00007000, 0x00003e00, 0x0000b500, 0x00006600,
+ 0x00004800, 0x00000300, 0x0000f600, 0x00000e00,
+ 0x00006100, 0x00003500, 0x00005700, 0x0000b900,
+ 0x00008600, 0x0000c100, 0x00001d00, 0x00009e00,
+ 0x0000e100, 0x0000f800, 0x00009800, 0x00001100,
+ 0x00006900, 0x0000d900, 0x00008e00, 0x00009400,
+ 0x00009b00, 0x00001e00, 0x00008700, 0x0000e900,
+ 0x0000ce00, 0x00005500, 0x00002800, 0x0000df00,
+ 0x00008c00, 0x0000a100, 0x00008900, 0x00000d00,
+ 0x0000bf00, 0x0000e600, 0x00004200, 0x00006800,
+ 0x00004100, 0x00009900, 0x00002d00, 0x00000f00,
+ 0x0000b000, 0x00005400, 0x0000bb00, 0x00001600
+}, {
+ 0x00630000, 0x007c0000, 0x00770000, 0x007b0000,
+ 0x00f20000, 0x006b0000, 0x006f0000, 0x00c50000,
+ 0x00300000, 0x00010000, 0x00670000, 0x002b0000,
+ 0x00fe0000, 0x00d70000, 0x00ab0000, 0x00760000,
+ 0x00ca0000, 0x00820000, 0x00c90000, 0x007d0000,
+ 0x00fa0000, 0x00590000, 0x00470000, 0x00f00000,
+ 0x00ad0000, 0x00d40000, 0x00a20000, 0x00af0000,
+ 0x009c0000, 0x00a40000, 0x00720000, 0x00c00000,
+ 0x00b70000, 0x00fd0000, 0x00930000, 0x00260000,
+ 0x00360000, 0x003f0000, 0x00f70000, 0x00cc0000,
+ 0x00340000, 0x00a50000, 0x00e50000, 0x00f10000,
+ 0x00710000, 0x00d80000, 0x00310000, 0x00150000,
+ 0x00040000, 0x00c70000, 0x00230000, 0x00c30000,
+ 0x00180000, 0x00960000, 0x00050000, 0x009a0000,
+ 0x00070000, 0x00120000, 0x00800000, 0x00e20000,
+ 0x00eb0000, 0x00270000, 0x00b20000, 0x00750000,
+ 0x00090000, 0x00830000, 0x002c0000, 0x001a0000,
+ 0x001b0000, 0x006e0000, 0x005a0000, 0x00a00000,
+ 0x00520000, 0x003b0000, 0x00d60000, 0x00b30000,
+ 0x00290000, 0x00e30000, 0x002f0000, 0x00840000,
+ 0x00530000, 0x00d10000, 0x00000000, 0x00ed0000,
+ 0x00200000, 0x00fc0000, 0x00b10000, 0x005b0000,
+ 0x006a0000, 0x00cb0000, 0x00be0000, 0x00390000,
+ 0x004a0000, 0x004c0000, 0x00580000, 0x00cf0000,
+ 0x00d00000, 0x00ef0000, 0x00aa0000, 0x00fb0000,
+ 0x00430000, 0x004d0000, 0x00330000, 0x00850000,
+ 0x00450000, 0x00f90000, 0x00020000, 0x007f0000,
+ 0x00500000, 0x003c0000, 0x009f0000, 0x00a80000,
+ 0x00510000, 0x00a30000, 0x00400000, 0x008f0000,
+ 0x00920000, 0x009d0000, 0x00380000, 0x00f50000,
+ 0x00bc0000, 0x00b60000, 0x00da0000, 0x00210000,
+ 0x00100000, 0x00ff0000, 0x00f30000, 0x00d20000,
+ 0x00cd0000, 0x000c0000, 0x00130000, 0x00ec0000,
+ 0x005f0000, 0x00970000, 0x00440000, 0x00170000,
+ 0x00c40000, 0x00a70000, 0x007e0000, 0x003d0000,
+ 0x00640000, 0x005d0000, 0x00190000, 0x00730000,
+ 0x00600000, 0x00810000, 0x004f0000, 0x00dc0000,
+ 0x00220000, 0x002a0000, 0x00900000, 0x00880000,
+ 0x00460000, 0x00ee0000, 0x00b80000, 0x00140000,
+ 0x00de0000, 0x005e0000, 0x000b0000, 0x00db0000,
+ 0x00e00000, 0x00320000, 0x003a0000, 0x000a0000,
+ 0x00490000, 0x00060000, 0x00240000, 0x005c0000,
+ 0x00c20000, 0x00d30000, 0x00ac0000, 0x00620000,
+ 0x00910000, 0x00950000, 0x00e40000, 0x00790000,
+ 0x00e70000, 0x00c80000, 0x00370000, 0x006d0000,
+ 0x008d0000, 0x00d50000, 0x004e0000, 0x00a90000,
+ 0x006c0000, 0x00560000, 0x00f40000, 0x00ea0000,
+ 0x00650000, 0x007a0000, 0x00ae0000, 0x00080000,
+ 0x00ba0000, 0x00780000, 0x00250000, 0x002e0000,
+ 0x001c0000, 0x00a60000, 0x00b40000, 0x00c60000,
+ 0x00e80000, 0x00dd0000, 0x00740000, 0x001f0000,
+ 0x004b0000, 0x00bd0000, 0x008b0000, 0x008a0000,
+ 0x00700000, 0x003e0000, 0x00b50000, 0x00660000,
+ 0x00480000, 0x00030000, 0x00f60000, 0x000e0000,
+ 0x00610000, 0x00350000, 0x00570000, 0x00b90000,
+ 0x00860000, 0x00c10000, 0x001d0000, 0x009e0000,
+ 0x00e10000, 0x00f80000, 0x00980000, 0x00110000,
+ 0x00690000, 0x00d90000, 0x008e0000, 0x00940000,
+ 0x009b0000, 0x001e0000, 0x00870000, 0x00e90000,
+ 0x00ce0000, 0x00550000, 0x00280000, 0x00df0000,
+ 0x008c0000, 0x00a10000, 0x00890000, 0x000d0000,
+ 0x00bf0000, 0x00e60000, 0x00420000, 0x00680000,
+ 0x00410000, 0x00990000, 0x002d0000, 0x000f0000,
+ 0x00b00000, 0x00540000, 0x00bb0000, 0x00160000
+}, {
+ 0x63000000, 0x7c000000, 0x77000000, 0x7b000000,
+ 0xf2000000, 0x6b000000, 0x6f000000, 0xc5000000,
+ 0x30000000, 0x01000000, 0x67000000, 0x2b000000,
+ 0xfe000000, 0xd7000000, 0xab000000, 0x76000000,
+ 0xca000000, 0x82000000, 0xc9000000, 0x7d000000,
+ 0xfa000000, 0x59000000, 0x47000000, 0xf0000000,
+ 0xad000000, 0xd4000000, 0xa2000000, 0xaf000000,
+ 0x9c000000, 0xa4000000, 0x72000000, 0xc0000000,
+ 0xb7000000, 0xfd000000, 0x93000000, 0x26000000,
+ 0x36000000, 0x3f000000, 0xf7000000, 0xcc000000,
+ 0x34000000, 0xa5000000, 0xe5000000, 0xf1000000,
+ 0x71000000, 0xd8000000, 0x31000000, 0x15000000,
+ 0x04000000, 0xc7000000, 0x23000000, 0xc3000000,
+ 0x18000000, 0x96000000, 0x05000000, 0x9a000000,
+ 0x07000000, 0x12000000, 0x80000000, 0xe2000000,
+ 0xeb000000, 0x27000000, 0xb2000000, 0x75000000,
+ 0x09000000, 0x83000000, 0x2c000000, 0x1a000000,
+ 0x1b000000, 0x6e000000, 0x5a000000, 0xa0000000,
+ 0x52000000, 0x3b000000, 0xd6000000, 0xb3000000,
+ 0x29000000, 0xe3000000, 0x2f000000, 0x84000000,
+ 0x53000000, 0xd1000000, 0x00000000, 0xed000000,
+ 0x20000000, 0xfc000000, 0xb1000000, 0x5b000000,
+ 0x6a000000, 0xcb000000, 0xbe000000, 0x39000000,
+ 0x4a000000, 0x4c000000, 0x58000000, 0xcf000000,
+ 0xd0000000, 0xef000000, 0xaa000000, 0xfb000000,
+ 0x43000000, 0x4d000000, 0x33000000, 0x85000000,
+ 0x45000000, 0xf9000000, 0x02000000, 0x7f000000,
+ 0x50000000, 0x3c000000, 0x9f000000, 0xa8000000,
+ 0x51000000, 0xa3000000, 0x40000000, 0x8f000000,
+ 0x92000000, 0x9d000000, 0x38000000, 0xf5000000,
+ 0xbc000000, 0xb6000000, 0xda000000, 0x21000000,
+ 0x10000000, 0xff000000, 0xf3000000, 0xd2000000,
+ 0xcd000000, 0x0c000000, 0x13000000, 0xec000000,
+ 0x5f000000, 0x97000000, 0x44000000, 0x17000000,
+ 0xc4000000, 0xa7000000, 0x7e000000, 0x3d000000,
+ 0x64000000, 0x5d000000, 0x19000000, 0x73000000,
+ 0x60000000, 0x81000000, 0x4f000000, 0xdc000000,
+ 0x22000000, 0x2a000000, 0x90000000, 0x88000000,
+ 0x46000000, 0xee000000, 0xb8000000, 0x14000000,
+ 0xde000000, 0x5e000000, 0x0b000000, 0xdb000000,
+ 0xe0000000, 0x32000000, 0x3a000000, 0x0a000000,
+ 0x49000000, 0x06000000, 0x24000000, 0x5c000000,
+ 0xc2000000, 0xd3000000, 0xac000000, 0x62000000,
+ 0x91000000, 0x95000000, 0xe4000000, 0x79000000,
+ 0xe7000000, 0xc8000000, 0x37000000, 0x6d000000,
+ 0x8d000000, 0xd5000000, 0x4e000000, 0xa9000000,
+ 0x6c000000, 0x56000000, 0xf4000000, 0xea000000,
+ 0x65000000, 0x7a000000, 0xae000000, 0x08000000,
+ 0xba000000, 0x78000000, 0x25000000, 0x2e000000,
+ 0x1c000000, 0xa6000000, 0xb4000000, 0xc6000000,
+ 0xe8000000, 0xdd000000, 0x74000000, 0x1f000000,
+ 0x4b000000, 0xbd000000, 0x8b000000, 0x8a000000,
+ 0x70000000, 0x3e000000, 0xb5000000, 0x66000000,
+ 0x48000000, 0x03000000, 0xf6000000, 0x0e000000,
+ 0x61000000, 0x35000000, 0x57000000, 0xb9000000,
+ 0x86000000, 0xc1000000, 0x1d000000, 0x9e000000,
+ 0xe1000000, 0xf8000000, 0x98000000, 0x11000000,
+ 0x69000000, 0xd9000000, 0x8e000000, 0x94000000,
+ 0x9b000000, 0x1e000000, 0x87000000, 0xe9000000,
+ 0xce000000, 0x55000000, 0x28000000, 0xdf000000,
+ 0x8c000000, 0xa1000000, 0x89000000, 0x0d000000,
+ 0xbf000000, 0xe6000000, 0x42000000, 0x68000000,
+ 0x41000000, 0x99000000, 0x2d000000, 0x0f000000,
+ 0xb0000000, 0x54000000, 0xbb000000, 0x16000000
+}
+};
+
+static const u4byte il_tab[4][256] = {
+{
+ 0x00000052, 0x00000009, 0x0000006a, 0x000000d5,
+ 0x00000030, 0x00000036, 0x000000a5, 0x00000038,
+ 0x000000bf, 0x00000040, 0x000000a3, 0x0000009e,
+ 0x00000081, 0x000000f3, 0x000000d7, 0x000000fb,
+ 0x0000007c, 0x000000e3, 0x00000039, 0x00000082,
+ 0x0000009b, 0x0000002f, 0x000000ff, 0x00000087,
+ 0x00000034, 0x0000008e, 0x00000043, 0x00000044,
+ 0x000000c4, 0x000000de, 0x000000e9, 0x000000cb,
+ 0x00000054, 0x0000007b, 0x00000094, 0x00000032,
+ 0x000000a6, 0x000000c2, 0x00000023, 0x0000003d,
+ 0x000000ee, 0x0000004c, 0x00000095, 0x0000000b,
+ 0x00000042, 0x000000fa, 0x000000c3, 0x0000004e,
+ 0x00000008, 0x0000002e, 0x000000a1, 0x00000066,
+ 0x00000028, 0x000000d9, 0x00000024, 0x000000b2,
+ 0x00000076, 0x0000005b, 0x000000a2, 0x00000049,
+ 0x0000006d, 0x0000008b, 0x000000d1, 0x00000025,
+ 0x00000072, 0x000000f8, 0x000000f6, 0x00000064,
+ 0x00000086, 0x00000068, 0x00000098, 0x00000016,
+ 0x000000d4, 0x000000a4, 0x0000005c, 0x000000cc,
+ 0x0000005d, 0x00000065, 0x000000b6, 0x00000092,
+ 0x0000006c, 0x00000070, 0x00000048, 0x00000050,
+ 0x000000fd, 0x000000ed, 0x000000b9, 0x000000da,
+ 0x0000005e, 0x00000015, 0x00000046, 0x00000057,
+ 0x000000a7, 0x0000008d, 0x0000009d, 0x00000084,
+ 0x00000090, 0x000000d8, 0x000000ab, 0x00000000,
+ 0x0000008c, 0x000000bc, 0x000000d3, 0x0000000a,
+ 0x000000f7, 0x000000e4, 0x00000058, 0x00000005,
+ 0x000000b8, 0x000000b3, 0x00000045, 0x00000006,
+ 0x000000d0, 0x0000002c, 0x0000001e, 0x0000008f,
+ 0x000000ca, 0x0000003f, 0x0000000f, 0x00000002,
+ 0x000000c1, 0x000000af, 0x000000bd, 0x00000003,
+ 0x00000001, 0x00000013, 0x0000008a, 0x0000006b,
+ 0x0000003a, 0x00000091, 0x00000011, 0x00000041,
+ 0x0000004f, 0x00000067, 0x000000dc, 0x000000ea,
+ 0x00000097, 0x000000f2, 0x000000cf, 0x000000ce,
+ 0x000000f0, 0x000000b4, 0x000000e6, 0x00000073,
+ 0x00000096, 0x000000ac, 0x00000074, 0x00000022,
+ 0x000000e7, 0x000000ad, 0x00000035, 0x00000085,
+ 0x000000e2, 0x000000f9, 0x00000037, 0x000000e8,
+ 0x0000001c, 0x00000075, 0x000000df, 0x0000006e,
+ 0x00000047, 0x000000f1, 0x0000001a, 0x00000071,
+ 0x0000001d, 0x00000029, 0x000000c5, 0x00000089,
+ 0x0000006f, 0x000000b7, 0x00000062, 0x0000000e,
+ 0x000000aa, 0x00000018, 0x000000be, 0x0000001b,
+ 0x000000fc, 0x00000056, 0x0000003e, 0x0000004b,
+ 0x000000c6, 0x000000d2, 0x00000079, 0x00000020,
+ 0x0000009a, 0x000000db, 0x000000c0, 0x000000fe,
+ 0x00000078, 0x000000cd, 0x0000005a, 0x000000f4,
+ 0x0000001f, 0x000000dd, 0x000000a8, 0x00000033,
+ 0x00000088, 0x00000007, 0x000000c7, 0x00000031,
+ 0x000000b1, 0x00000012, 0x00000010, 0x00000059,
+ 0x00000027, 0x00000080, 0x000000ec, 0x0000005f,
+ 0x00000060, 0x00000051, 0x0000007f, 0x000000a9,
+ 0x00000019, 0x000000b5, 0x0000004a, 0x0000000d,
+ 0x0000002d, 0x000000e5, 0x0000007a, 0x0000009f,
+ 0x00000093, 0x000000c9, 0x0000009c, 0x000000ef,
+ 0x000000a0, 0x000000e0, 0x0000003b, 0x0000004d,
+ 0x000000ae, 0x0000002a, 0x000000f5, 0x000000b0,
+ 0x000000c8, 0x000000eb, 0x000000bb, 0x0000003c,
+ 0x00000083, 0x00000053, 0x00000099, 0x00000061,
+ 0x00000017, 0x0000002b, 0x00000004, 0x0000007e,
+ 0x000000ba, 0x00000077, 0x000000d6, 0x00000026,
+ 0x000000e1, 0x00000069, 0x00000014, 0x00000063,
+ 0x00000055, 0x00000021, 0x0000000c, 0x0000007d
+}, {
+ 0x00005200, 0x00000900, 0x00006a00, 0x0000d500,
+ 0x00003000, 0x00003600, 0x0000a500, 0x00003800,
+ 0x0000bf00, 0x00004000, 0x0000a300, 0x00009e00,
+ 0x00008100, 0x0000f300, 0x0000d700, 0x0000fb00,
+ 0x00007c00, 0x0000e300, 0x00003900, 0x00008200,
+ 0x00009b00, 0x00002f00, 0x0000ff00, 0x00008700,
+ 0x00003400, 0x00008e00, 0x00004300, 0x00004400,
+ 0x0000c400, 0x0000de00, 0x0000e900, 0x0000cb00,
+ 0x00005400, 0x00007b00, 0x00009400, 0x00003200,
+ 0x0000a600, 0x0000c200, 0x00002300, 0x00003d00,
+ 0x0000ee00, 0x00004c00, 0x00009500, 0x00000b00,
+ 0x00004200, 0x0000fa00, 0x0000c300, 0x00004e00,
+ 0x00000800, 0x00002e00, 0x0000a100, 0x00006600,
+ 0x00002800, 0x0000d900, 0x00002400, 0x0000b200,
+ 0x00007600, 0x00005b00, 0x0000a200, 0x00004900,
+ 0x00006d00, 0x00008b00, 0x0000d100, 0x00002500,
+ 0x00007200, 0x0000f800, 0x0000f600, 0x00006400,
+ 0x00008600, 0x00006800, 0x00009800, 0x00001600,
+ 0x0000d400, 0x0000a400, 0x00005c00, 0x0000cc00,
+ 0x00005d00, 0x00006500, 0x0000b600, 0x00009200,
+ 0x00006c00, 0x00007000, 0x00004800, 0x00005000,
+ 0x0000fd00, 0x0000ed00, 0x0000b900, 0x0000da00,
+ 0x00005e00, 0x00001500, 0x00004600, 0x00005700,
+ 0x0000a700, 0x00008d00, 0x00009d00, 0x00008400,
+ 0x00009000, 0x0000d800, 0x0000ab00, 0x00000000,
+ 0x00008c00, 0x0000bc00, 0x0000d300, 0x00000a00,
+ 0x0000f700, 0x0000e400, 0x00005800, 0x00000500,
+ 0x0000b800, 0x0000b300, 0x00004500, 0x00000600,
+ 0x0000d000, 0x00002c00, 0x00001e00, 0x00008f00,
+ 0x0000ca00, 0x00003f00, 0x00000f00, 0x00000200,
+ 0x0000c100, 0x0000af00, 0x0000bd00, 0x00000300,
+ 0x00000100, 0x00001300, 0x00008a00, 0x00006b00,
+ 0x00003a00, 0x00009100, 0x00001100, 0x00004100,
+ 0x00004f00, 0x00006700, 0x0000dc00, 0x0000ea00,
+ 0x00009700, 0x0000f200, 0x0000cf00, 0x0000ce00,
+ 0x0000f000, 0x0000b400, 0x0000e600, 0x00007300,
+ 0x00009600, 0x0000ac00, 0x00007400, 0x00002200,
+ 0x0000e700, 0x0000ad00, 0x00003500, 0x00008500,
+ 0x0000e200, 0x0000f900, 0x00003700, 0x0000e800,
+ 0x00001c00, 0x00007500, 0x0000df00, 0x00006e00,
+ 0x00004700, 0x0000f100, 0x00001a00, 0x00007100,
+ 0x00001d00, 0x00002900, 0x0000c500, 0x00008900,
+ 0x00006f00, 0x0000b700, 0x00006200, 0x00000e00,
+ 0x0000aa00, 0x00001800, 0x0000be00, 0x00001b00,
+ 0x0000fc00, 0x00005600, 0x00003e00, 0x00004b00,
+ 0x0000c600, 0x0000d200, 0x00007900, 0x00002000,
+ 0x00009a00, 0x0000db00, 0x0000c000, 0x0000fe00,
+ 0x00007800, 0x0000cd00, 0x00005a00, 0x0000f400,
+ 0x00001f00, 0x0000dd00, 0x0000a800, 0x00003300,
+ 0x00008800, 0x00000700, 0x0000c700, 0x00003100,
+ 0x0000b100, 0x00001200, 0x00001000, 0x00005900,
+ 0x00002700, 0x00008000, 0x0000ec00, 0x00005f00,
+ 0x00006000, 0x00005100, 0x00007f00, 0x0000a900,
+ 0x00001900, 0x0000b500, 0x00004a00, 0x00000d00,
+ 0x00002d00, 0x0000e500, 0x00007a00, 0x00009f00,
+ 0x00009300, 0x0000c900, 0x00009c00, 0x0000ef00,
+ 0x0000a000, 0x0000e000, 0x00003b00, 0x00004d00,
+ 0x0000ae00, 0x00002a00, 0x0000f500, 0x0000b000,
+ 0x0000c800, 0x0000eb00, 0x0000bb00, 0x00003c00,
+ 0x00008300, 0x00005300, 0x00009900, 0x00006100,
+ 0x00001700, 0x00002b00, 0x00000400, 0x00007e00,
+ 0x0000ba00, 0x00007700, 0x0000d600, 0x00002600,
+ 0x0000e100, 0x00006900, 0x00001400, 0x00006300,
+ 0x00005500, 0x00002100, 0x00000c00, 0x00007d00
+}, {
+ 0x00520000, 0x00090000, 0x006a0000, 0x00d50000,
+ 0x00300000, 0x00360000, 0x00a50000, 0x00380000,
+ 0x00bf0000, 0x00400000, 0x00a30000, 0x009e0000,
+ 0x00810000, 0x00f30000, 0x00d70000, 0x00fb0000,
+ 0x007c0000, 0x00e30000, 0x00390000, 0x00820000,
+ 0x009b0000, 0x002f0000, 0x00ff0000, 0x00870000,
+ 0x00340000, 0x008e0000, 0x00430000, 0x00440000,
+ 0x00c40000, 0x00de0000, 0x00e90000, 0x00cb0000,
+ 0x00540000, 0x007b0000, 0x00940000, 0x00320000,
+ 0x00a60000, 0x00c20000, 0x00230000, 0x003d0000,
+ 0x00ee0000, 0x004c0000, 0x00950000, 0x000b0000,
+ 0x00420000, 0x00fa0000, 0x00c30000, 0x004e0000,
+ 0x00080000, 0x002e0000, 0x00a10000, 0x00660000,
+ 0x00280000, 0x00d90000, 0x00240000, 0x00b20000,
+ 0x00760000, 0x005b0000, 0x00a20000, 0x00490000,
+ 0x006d0000, 0x008b0000, 0x00d10000, 0x00250000,
+ 0x00720000, 0x00f80000, 0x00f60000, 0x00640000,
+ 0x00860000, 0x00680000, 0x00980000, 0x00160000,
+ 0x00d40000, 0x00a40000, 0x005c0000, 0x00cc0000,
+ 0x005d0000, 0x00650000, 0x00b60000, 0x00920000,
+ 0x006c0000, 0x00700000, 0x00480000, 0x00500000,
+ 0x00fd0000, 0x00ed0000, 0x00b90000, 0x00da0000,
+ 0x005e0000, 0x00150000, 0x00460000, 0x00570000,
+ 0x00a70000, 0x008d0000, 0x009d0000, 0x00840000,
+ 0x00900000, 0x00d80000, 0x00ab0000, 0x00000000,
+ 0x008c0000, 0x00bc0000, 0x00d30000, 0x000a0000,
+ 0x00f70000, 0x00e40000, 0x00580000, 0x00050000,
+ 0x00b80000, 0x00b30000, 0x00450000, 0x00060000,
+ 0x00d00000, 0x002c0000, 0x001e0000, 0x008f0000,
+ 0x00ca0000, 0x003f0000, 0x000f0000, 0x00020000,
+ 0x00c10000, 0x00af0000, 0x00bd0000, 0x00030000,
+ 0x00010000, 0x00130000, 0x008a0000, 0x006b0000,
+ 0x003a0000, 0x00910000, 0x00110000, 0x00410000,
+ 0x004f0000, 0x00670000, 0x00dc0000, 0x00ea0000,
+ 0x00970000, 0x00f20000, 0x00cf0000, 0x00ce0000,
+ 0x00f00000, 0x00b40000, 0x00e60000, 0x00730000,
+ 0x00960000, 0x00ac0000, 0x00740000, 0x00220000,
+ 0x00e70000, 0x00ad0000, 0x00350000, 0x00850000,
+ 0x00e20000, 0x00f90000, 0x00370000, 0x00e80000,
+ 0x001c0000, 0x00750000, 0x00df0000, 0x006e0000,
+ 0x00470000, 0x00f10000, 0x001a0000, 0x00710000,
+ 0x001d0000, 0x00290000, 0x00c50000, 0x00890000,
+ 0x006f0000, 0x00b70000, 0x00620000, 0x000e0000,
+ 0x00aa0000, 0x00180000, 0x00be0000, 0x001b0000,
+ 0x00fc0000, 0x00560000, 0x003e0000, 0x004b0000,
+ 0x00c60000, 0x00d20000, 0x00790000, 0x00200000,
+ 0x009a0000, 0x00db0000, 0x00c00000, 0x00fe0000,
+ 0x00780000, 0x00cd0000, 0x005a0000, 0x00f40000,
+ 0x001f0000, 0x00dd0000, 0x00a80000, 0x00330000,
+ 0x00880000, 0x00070000, 0x00c70000, 0x00310000,
+ 0x00b10000, 0x00120000, 0x00100000, 0x00590000,
+ 0x00270000, 0x00800000, 0x00ec0000, 0x005f0000,
+ 0x00600000, 0x00510000, 0x007f0000, 0x00a90000,
+ 0x00190000, 0x00b50000, 0x004a0000, 0x000d0000,
+ 0x002d0000, 0x00e50000, 0x007a0000, 0x009f0000,
+ 0x00930000, 0x00c90000, 0x009c0000, 0x00ef0000,
+ 0x00a00000, 0x00e00000, 0x003b0000, 0x004d0000,
+ 0x00ae0000, 0x002a0000, 0x00f50000, 0x00b00000,
+ 0x00c80000, 0x00eb0000, 0x00bb0000, 0x003c0000,
+ 0x00830000, 0x00530000, 0x00990000, 0x00610000,
+ 0x00170000, 0x002b0000, 0x00040000, 0x007e0000,
+ 0x00ba0000, 0x00770000, 0x00d60000, 0x00260000,
+ 0x00e10000, 0x00690000, 0x00140000, 0x00630000,
+ 0x00550000, 0x00210000, 0x000c0000, 0x007d0000
+}, {
+ 0x52000000, 0x09000000, 0x6a000000, 0xd5000000,
+ 0x30000000, 0x36000000, 0xa5000000, 0x38000000,
+ 0xbf000000, 0x40000000, 0xa3000000, 0x9e000000,
+ 0x81000000, 0xf3000000, 0xd7000000, 0xfb000000,
+ 0x7c000000, 0xe3000000, 0x39000000, 0x82000000,
+ 0x9b000000, 0x2f000000, 0xff000000, 0x87000000,
+ 0x34000000, 0x8e000000, 0x43000000, 0x44000000,
+ 0xc4000000, 0xde000000, 0xe9000000, 0xcb000000,
+ 0x54000000, 0x7b000000, 0x94000000, 0x32000000,
+ 0xa6000000, 0xc2000000, 0x23000000, 0x3d000000,
+ 0xee000000, 0x4c000000, 0x95000000, 0x0b000000,
+ 0x42000000, 0xfa000000, 0xc3000000, 0x4e000000,
+ 0x08000000, 0x2e000000, 0xa1000000, 0x66000000,
+ 0x28000000, 0xd9000000, 0x24000000, 0xb2000000,
+ 0x76000000, 0x5b000000, 0xa2000000, 0x49000000,
+ 0x6d000000, 0x8b000000, 0xd1000000, 0x25000000,
+ 0x72000000, 0xf8000000, 0xf6000000, 0x64000000,
+ 0x86000000, 0x68000000, 0x98000000, 0x16000000,
+ 0xd4000000, 0xa4000000, 0x5c000000, 0xcc000000,
+ 0x5d000000, 0x65000000, 0xb6000000, 0x92000000,
+ 0x6c000000, 0x70000000, 0x48000000, 0x50000000,
+ 0xfd000000, 0xed000000, 0xb9000000, 0xda000000,
+ 0x5e000000, 0x15000000, 0x46000000, 0x57000000,
+ 0xa7000000, 0x8d000000, 0x9d000000, 0x84000000,
+ 0x90000000, 0xd8000000, 0xab000000, 0x00000000,
+ 0x8c000000, 0xbc000000, 0xd3000000, 0x0a000000,
+ 0xf7000000, 0xe4000000, 0x58000000, 0x05000000,
+ 0xb8000000, 0xb3000000, 0x45000000, 0x06000000,
+ 0xd0000000, 0x2c000000, 0x1e000000, 0x8f000000,
+ 0xca000000, 0x3f000000, 0x0f000000, 0x02000000,
+ 0xc1000000, 0xaf000000, 0xbd000000, 0x03000000,
+ 0x01000000, 0x13000000, 0x8a000000, 0x6b000000,
+ 0x3a000000, 0x91000000, 0x11000000, 0x41000000,
+ 0x4f000000, 0x67000000, 0xdc000000, 0xea000000,
+ 0x97000000, 0xf2000000, 0xcf000000, 0xce000000,
+ 0xf0000000, 0xb4000000, 0xe6000000, 0x73000000,
+ 0x96000000, 0xac000000, 0x74000000, 0x22000000,
+ 0xe7000000, 0xad000000, 0x35000000, 0x85000000,
+ 0xe2000000, 0xf9000000, 0x37000000, 0xe8000000,
+ 0x1c000000, 0x75000000, 0xdf000000, 0x6e000000,
+ 0x47000000, 0xf1000000, 0x1a000000, 0x71000000,
+ 0x1d000000, 0x29000000, 0xc5000000, 0x89000000,
+ 0x6f000000, 0xb7000000, 0x62000000, 0x0e000000,
+ 0xaa000000, 0x18000000, 0xbe000000, 0x1b000000,
+ 0xfc000000, 0x56000000, 0x3e000000, 0x4b000000,
+ 0xc6000000, 0xd2000000, 0x79000000, 0x20000000,
+ 0x9a000000, 0xdb000000, 0xc0000000, 0xfe000000,
+ 0x78000000, 0xcd000000, 0x5a000000, 0xf4000000,
+ 0x1f000000, 0xdd000000, 0xa8000000, 0x33000000,
+ 0x88000000, 0x07000000, 0xc7000000, 0x31000000,
+ 0xb1000000, 0x12000000, 0x10000000, 0x59000000,
+ 0x27000000, 0x80000000, 0xec000000, 0x5f000000,
+ 0x60000000, 0x51000000, 0x7f000000, 0xa9000000,
+ 0x19000000, 0xb5000000, 0x4a000000, 0x0d000000,
+ 0x2d000000, 0xe5000000, 0x7a000000, 0x9f000000,
+ 0x93000000, 0xc9000000, 0x9c000000, 0xef000000,
+ 0xa0000000, 0xe0000000, 0x3b000000, 0x4d000000,
+ 0xae000000, 0x2a000000, 0xf5000000, 0xb0000000,
+ 0xc8000000, 0xeb000000, 0xbb000000, 0x3c000000,
+ 0x83000000, 0x53000000, 0x99000000, 0x61000000,
+ 0x17000000, 0x2b000000, 0x04000000, 0x7e000000,
+ 0xba000000, 0x77000000, 0xd6000000, 0x26000000,
+ 0xe1000000, 0x69000000, 0x14000000, 0x63000000,
+ 0x55000000, 0x21000000, 0x0c000000, 0x7d000000
+}
+};
+
+static const u4byte rco_tab[10] = {
+ 0x00000001, 0x00000002, 0x00000004, 0x00000008, 0x00000010,
+ 0x00000020, 0x00000040, 0x00000080, 0x0000001b, 0x00000036
+};
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 2d1ed14..96a87f8 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -149,6 +149,7 @@ CreateSharedMemoryAndSemaphores(bool makePrivate, int port)
size = add_size(size, BTreeShmemSize());
size = add_size(size, SyncScanShmemSize());
size = add_size(size, AsyncShmemSize());
+ size = add_size(size, CryptShmemSize());
size = add_size(size, BackendRandomShmemSize());
#ifdef EXEC_BACKEND
size = add_size(size, ShmemBackendArraySize());
@@ -269,6 +270,7 @@ CreateSharedMemoryAndSemaphores(bool makePrivate, int port)
BTreeShmemInit();
SyncScanShmemInit();
AsyncShmemInit();
+ CryptShmemInit();
BackendRandomShmemInit();
#ifdef EXEC_BACKEND
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index 9bc00b6..fa57d24 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -35,6 +35,7 @@
#include "storage/bufmgr.h"
#include "storage/relfilenode.h"
#include "storage/smgr.h"
+#include "storage/rijndael.h"
#include "utils/hsearch.h"
#include "utils/memutils.h"
#include "pg_trace.h"
@@ -198,6 +199,83 @@ static MdfdVec *_mdfd_getseg(SMgrRelation reln, ForkNumber forkno,
static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum,
MdfdVec *seg);
+static rijndael_ctx* aes_context;
+
+int CryptShmemSize(void)
+{
+ return sizeof(rijndael_ctx);
+}
+
+void CryptShmemInit(void)
+{
+ char* cipher_key = getenv("PG_CIPHER_KEY");
+ if (cipher_key != NULL)
+ {
+ bool found;
+ aes_context = (rijndael_ctx*)ShmemInitStruct("AES encryption context", sizeof(rijndael_ctx), &found);
+ if (!found)
+ {
+ int key_length;
+ uint8 aes_key[32] = {0}; /* at most 256 bits */
+
+ key_length = strlen(cipher_key);
+
+ memcpy(&aes_key, cipher_key, key_length > sizeof(aes_key) ? sizeof(aes_key) : key_length);
+ rijndael_set_key(
+ aes_context, /* context */
+ (u4byte*)&aes_key, /* key */
+ sizeof(aes_key) * 8 /* key size in bits */,
+ 1 /* for CTR mode we need only encryption */
+ );
+ }
+ unsetenv("PG_CIPHER_KEY"); /* disable inspection of this environment variable */
+ }
+}
+
+/* Encryption and decryption using AES in CTR mode */
+static void aes_crypt_block(SMgrRelation reln, void* block, BlockNumber blocknum)
+{
+/*
+#define AES_DEBUG 1
+*/
+ uint32 aes_in[4]; /* 16 bytes, 128 bits */
+ uint32 aes_out[4];
+ uint8* plaintext = (uint8*)block;
+ uint8* gamma = (uint8*)&aes_out;
+ uint32 offs = 0;
+ uint32 i;
+
+ aes_in[0] = reln->smgr_rnode.node.spcNode;
+ aes_in[1] = reln->smgr_rnode.node.relNode;
+ aes_in[2] = blocknum;
+ aes_in[3] = 0;
+ rijndael_encrypt(aes_context, (u4byte*)&aes_in, (u4byte*)&aes_out);
+
+#ifdef AES_DEBUG
+ elog(LOG, "aes_crypt_block, in = %08X %08X %08X %08X, out = %08X %08X %08X %08X",
+ aes_in[0], aes_in[1], aes_in[2], aes_in[3],
+ aes_out[0], aes_out[1], aes_out[2], aes_out[3]);
+#endif
+
+ for(i = 0; i < BLCKSZ; i++)
+ {
+ plaintext[i] ^= gamma[offs & 0xF];
+ offs++;
+ if((offs & 0xF) == 0)
+ {
+ /* Prepare next gamma part */
+ aes_in[3] = offs;
+ rijndael_encrypt(aes_context, (u4byte*)&aes_in, (u4byte*)&aes_out);
+
+#ifdef AES_DEBUG
+ elog(LOG, "aes_crypt_block, in = %08X %08X %08X %08X, out = %08X %08X %08X %08X",
+ aes_in[0], aes_in[1], aes_in[2], aes_in[3],
+ aes_out[0], aes_out[1], aes_out[2], aes_out[3]);
+#endif
+ }
+ }
+}
+
/*
* mdinit() -- Initialize private state for magnetic disk storage manager.
@@ -498,6 +576,7 @@ mdextend(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
off_t seekpos;
int nbytes;
MdfdVec *v;
+ char cryptbuf[BLCKSZ];
/* This assert is too expensive to have on normally ... */
#ifdef CHECK_WRITE_VS_EXTEND
@@ -537,6 +616,13 @@ mdextend(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
errmsg("could not seek to block %u in file \"%s\": %m",
blocknum, FilePathName(v->mdfd_vfd))));
+ if (aes_context && forknum == MAIN_FORKNUM)
+ {
+ memcpy(cryptbuf, buffer, BLCKSZ);
+ aes_crypt_block(reln, cryptbuf, blocknum);
+ buffer = cryptbuf;
+ }
+
if ((nbytes = FileWrite(v->mdfd_vfd, buffer, BLCKSZ, WAIT_EVENT_DATA_FILE_EXTEND)) != BLCKSZ)
{
if (nbytes < 0)
@@ -789,6 +875,11 @@ mdread(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
blocknum, FilePathName(v->mdfd_vfd),
nbytes, BLCKSZ)));
}
+ else if (aes_context && forknum == MAIN_FORKNUM)
+ {
+ if (*(long*)buffer != 0)
+ aes_crypt_block(reln, buffer, blocknum);
+ }
}
/*
@@ -805,6 +896,7 @@ mdwrite(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
off_t seekpos;
int nbytes;
MdfdVec *v;
+ char cryptbuf[BLCKSZ];
/* This assert is too expensive to have on normally ... */
#ifdef CHECK_WRITE_VS_EXTEND
@@ -830,6 +922,13 @@ mdwrite(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
errmsg("could not seek to block %u in file \"%s\": %m",
blocknum, FilePathName(v->mdfd_vfd))));
+ if (aes_context && forknum == MAIN_FORKNUM)
+ {
+ memcpy(cryptbuf, buffer, BLCKSZ);
+ aes_crypt_block(reln, cryptbuf, blocknum);
+ buffer = cryptbuf;
+ }
+
nbytes = FileWrite(v->mdfd_vfd, buffer, BLCKSZ, WAIT_EVENT_DATA_FILE_WRITE);
TRACE_POSTGRESQL_SMGR_MD_WRITE_DONE(forknum, blocknum,
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index 0568049..7b36fb5 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -59,7 +59,6 @@ extern int max_files_per_process;
*/
extern int max_safe_fds;
-
/*
* prototypes for functions in fd.c
*/
diff --git a/src/include/storage/rijndael.h b/src/include/storage/rijndael.h
new file mode 100644
index 0000000..e536c61
--- /dev/null
+++ b/src/include/storage/rijndael.h
@@ -0,0 +1,60 @@
+/*
+ * contrib/pgcrypto/rijndael.h
+ *
+ * $OpenBSD: rijndael.h,v 1.3 2001/05/09 23:01:32 markus Exp $ */
+
+/* This is an independent implementation of the encryption algorithm: */
+/* */
+/* RIJNDAEL by Joan Daemen and Vincent Rijmen */
+/* */
+/* which is a candidate algorithm in the Advanced Encryption Standard */
+/* programme of the US National Institute of Standards and Technology. */
+/* */
+/* Copyright in this implementation is held by Dr B R Gladman but I */
+/* hereby give permission for its free direct or derivative use subject */
+/* to acknowledgment of its origin and compliance with any conditions */
+/* that the originators of the algorithm place on its exploitation. */
+/* */
+/* Dr Brian Gladman (gladman@seven77.demon.co.uk) 14th January 1999 */
+
+#ifndef _RIJNDAEL_H_
+#define _RIJNDAEL_H_
+
+/* 1. Standard types for AES cryptography source code */
+
+typedef uint8 u1byte; /* an 8 bit unsigned character type */
+typedef uint16 u2byte; /* a 16 bit unsigned integer type */
+typedef uint32 u4byte; /* a 32 bit unsigned integer type */
+
+typedef int8 s1byte; /* an 8 bit signed character type */
+typedef int16 s2byte; /* a 16 bit signed integer type */
+typedef int32 s4byte; /* a 32 bit signed integer type */
+
+typedef struct _rijndael_ctx
+{
+ u4byte k_len;
+ int decrypt;
+ u4byte e_key[64];
+ u4byte d_key[64];
+} rijndael_ctx;
+
+
+/* 2. Standard interface for AES cryptographic routines */
+
+/* These are all based on 32 bit unsigned values and will therefore */
+/* require endian conversions for big-endian architectures */
+
+rijndael_ctx *
+ rijndael_set_key(rijndael_ctx *, const u4byte *, const u4byte, int);
+void rijndael_encrypt(rijndael_ctx *, const u4byte *, u4byte *);
+void rijndael_decrypt(rijndael_ctx *, const u4byte *, u4byte *);
+
+/* conventional interface */
+
+void aes_set_key(rijndael_ctx *ctx, const uint8 *key, unsigned keybits, int enc);
+void aes_ecb_encrypt(rijndael_ctx *ctx, uint8 *data, unsigned len);
+void aes_ecb_decrypt(rijndael_ctx *ctx, uint8 *data, unsigned len);
+void aes_cbc_encrypt(rijndael_ctx *ctx, uint8 *iva, uint8 *data, unsigned len);
+void aes_cbc_decrypt(rijndael_ctx *ctx, uint8 *iva, uint8 *data, unsigned len);
+
+#endif /* _RIJNDAEL_H_ */
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index 9ce6829..9eb0042 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -144,4 +144,7 @@ extern void RememberFsyncRequest(RelFileNode rnode, ForkNumber forknum,
extern void ForgetRelationFsyncRequests(RelFileNode rnode, ForkNumber forknum);
extern void ForgetDatabaseFsyncRequests(Oid dbid);
+extern int CryptShmemSize(void);
+extern void CryptShmemInit(void);
+
#endif /* SMGR_H */
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:43 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 22:11 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-14 21:36 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 21:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 16:06 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-15 19:09 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 20:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 21:04 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 21:12 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 22:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 23:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 23:27 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 23:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 23:56 ` Re: WIP: Data at rest encryption Andres Freund <andres@anarazel.de>
2017-06-16 00:08 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-16 08:06 ` Re: WIP: Data at rest encryption Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
@ 2017-06-16 16:04 ` Bruce Momjian <bruce@momjian.us>
0 siblings, 0 replies; 93+ messages in thread
From: Bruce Momjian @ 2017-06-16 16:04 UTC (permalink / raw)
To: Konstantin Knizhnik <k.knizhnik@postgrespro.ru>; +Cc: pgsql-hackers
On Fri, Jun 16, 2017 at 11:06:39AM +0300, Konstantin Knizhnik wrote:
> Encryption is much easier to implement than compression, because it is not
> changing page size. So I do not see any "complexity and flexibility
> challenges" here.
> Just for reference I attached to this mail our own encryption patch. I do
I didn't see you using CPU AES instructions, which can improve
performance by 3-10x. Is there a reason?
> Postgres buffer manager interface significantly simplifies integration of
> encryption and compression. There is actually single path through which data
> is fetched/stored to the disk.
> It is most obvious and natural solution to decompress/decrypt data when it
> is read from the disk to page pool and compress/encrypt it when it is
> written back. Taken in account that memory is cheap now and many databases
> can completely fit in memory, storing pages in the buffer cache in plain
> (decompressed/decrypted) format allows to minimize overhead of
> compression/encryption and its influence on performance. For read only
> queries working with cached data performance will be exactly the same as
> without encryption/compression.
> Write speed for encrypted pages will be certainly slightly worse, but still
> encryption speed is much higher than disk IO speed.
Good point.
> I do not think that pluggable storage API is right approach to integrate
> compression and especially encryption. It is better to plugin encryption
> between buffer manager and storage device,
> allowing to use it with any storage implementation. Also it is not clear to
> me whether encryption of WAL can be provided using pluggable storage API.
Yes, you are completely correct. I withdraw my suggestion of doing it
as plugin storage.
> The last discussed question is whether it is necessary to encrypt temporary
> data (BufFile). In our solution we encrypt only main fork of non-system
> relations and do no encrypt temporary relations. It may cause that some
> secrete data will be stored at this disk in non-encrypted format. But
> accessing this data is not trivial. You can not just copy/stole disk, open
> database and do "select * from SecreteTable": you will have to extract data
> from raw file yourself. So looks like it is better to allow user to make
> choice whether to encrypt temporary data or not.
If we go forward with in-db encryption, I think we are going to have to
have a discussion about what parts of PGDATA need to be encrypted,
i.e., I don't think pg_clog needs encryption.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:43 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 22:11 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-14 21:36 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 21:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 16:06 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-15 19:09 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 20:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 21:04 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-15 21:12 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 22:41 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 23:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 23:27 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-15 23:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-15 23:56 ` Re: WIP: Data at rest encryption Andres Freund <andres@anarazel.de>
2017-06-16 00:08 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
@ 2017-06-16 15:41 ` Bruce Momjian <bruce@momjian.us>
2 siblings, 0 replies; 93+ messages in thread
From: Bruce Momjian @ 2017-06-16 15:41 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Stephen Frost <sfrost@snowman.net>; Robert Haas <robertmhaas@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; pgsql-hackers
On Thu, Jun 15, 2017 at 08:08:05PM -0400, Bruce Momjian wrote:
> On Thu, Jun 15, 2017 at 04:56:36PM -0700, Andres Freund wrote:
> > how few concerns about this feature's complexity / maintainability
> > impact have been raised.
>
> Yeah, I guess we will just have to wait to see it since other people are
> excited about it. My concern is code complexity and usability
> challenges, vs punting the problem to the operating system, though
> admittedly there are some cases where that is not possible.
I know some OS's can create file systems inside files. Can you encrypt
such file storage as non-root? I assume that is just too odd.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
@ 2017-06-13 19:45 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 22:13 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2 siblings, 1 reply; 93+ messages in thread
From: Peter Eisentraut @ 2017-06-13 19:45 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; Bruce Momjian <bruce@momjian.us>; +Cc: Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
On 6/13/17 15:20, Stephen Frost wrote:
> For example, you could simply do:
>
> cp -a /path/to/PG /mnt/usb
>
> and you're done. If you're using filesystem level encryption then you'd
> have to re-encrypt the data, using something like:
>
> tar -cf - /path/to/PG | openssl -key private.key > /mnt/usb/encrypted_cluster.tar
>
> And then you would need openssl on the other system to decrypt it.
Or make the USB file system encrypted as well? If you're in that kind
of environment, that would surely be feasible, if not required.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:45 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
@ 2017-06-13 22:13 ` Stephen Frost <sfrost@snowman.net>
0 siblings, 0 replies; 93+ messages in thread
From: Stephen Frost @ 2017-06-13 22:13 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; +Cc: Bruce Momjian <bruce@momjian.us>; Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
Peter,
* Peter Eisentraut (peter.eisentraut@2ndquadrant.com) wrote:
> On 6/13/17 15:20, Stephen Frost wrote:
> > And then you would need openssl on the other system to decrypt it.
>
> Or make the USB file system encrypted as well? If you're in that kind
> of environment, that would surely be feasible, if not required.
Right, but requiring file system encryption to work on a USB stick
across different types of systems strikes me as actually a higher bar
than requiring openssl to exist on both the source and destination
sides.
Naturally, if the environment you're in has already solved that problem
across the enterprise then it's a good approach, although you might want
to use a different encryption key, perhaps, though hopefully that's
something you'd be able to do pretty easily too.
Thanks!
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
@ 2017-06-13 19:51 ` Bruce Momjian <bruce@momjian.us>
2017-06-13 20:08 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 22:29 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2 siblings, 2 replies; 93+ messages in thread
From: Bruce Momjian @ 2017-06-13 19:51 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
On Tue, Jun 13, 2017 at 03:20:12PM -0400, Stephen Frost wrote:
> Bruce,
>
> * Bruce Momjian (bruce@momjian.us) wrote:
> > On Tue, Jun 13, 2017 at 02:38:58PM -0400, Stephen Frost wrote:
> > > It's good to discuss what the feature would bring and what cases it
> > > doesn't cover, as well as discussing how it can be designed to make sure
> > > that later improvements are able to be done without having to change it
> > > around. I do think it's a good idea for us to consider taking an
> > > incremental approach where we're adding pieces and building things up as
> > > we go. I'm concerned that if we try to do too much in the initial
> > > implementation that we'll end up not having anything.
> > >
> > > As it relates to the different attack vectors that this would address,
> > > it's primairly the same ones which filesystem-level encryption also
> > > addresses, but it's an improvement when it comes to ease of use.
> > > Unfortunately, it won't address cases where the OS is compromised.
> >
> > OK, so let's go back. You are saying there are no security benefits to
> > this vs. file system encryption.
>
> I'm not sure that I can see any, myself.. Perhaps I'm wrong there, but
> it seems unlikely that this would be an improvement over filesystem
> level encryption in the general sense. I'm not sure that I see it as
> really worse than filesystem-level encryption either, to be clear.
> There's a bit of increased information leakage, as Peter mentioned and I
> agreed with, but it's not a lot and I expect in most cases that
> information leak would be acceptable. That seems like something which
> would need to be considered on a case-by-case basis.
Isn't the leakage controlled by OS permissions, so is it really leakage,
i.e., if you can see the leakage, you probably have bypassed the OS
permissions and see the key and data anyway.
> > The benefit is allowing configuration
> > in the database rather than the OS?
>
> No, the benefit is that the database administrator can configure it and
> set it up and not have to get an OS-level administrator involved. There
> may also be other reasons why filesystem-level encryption is difficult
> to set up or use in a certain environment, but this wouldn't depend on
> anything OS-related and therefore could be done.
OK, my only point here is that we are going down a slippery slope of
implementing OS things in the database. There is nothing wrong with
that but it has often been something we have avoided, because of the
added complexity required in the db server.
As a counter-example, we only added an external collation library to
Postgres when we clearly saw a benefit, e.g. detecting collation
changes.
> > You stated you can transfer
> > db-level encrypted files between servers, but can't you do that anyway?
>
> If the filesystem is encrypted and you wanted to transfer the entire
> cluster from one system to another, keeping it encrypted with the same
> key, you'd have to transfer the entire filesystem at a block level.
> That's not typically very easy to do (ZFS, specifically, has this
> capability where you can export a filesystem and send it from one
> machine to another, but I don't know of any other filesystems which do
> and ZFS isn't always an option..).
>
> You could go through a process of re-encrypting the files prior to
> transferring them, or deciding that simply having the transport
> mechanism encrypted is sufficient (eg: SSH), but if what you really want
> to do is keep the existing encryption of the database and transfer it to
> another system, this allows that pretty easily.
>
> For example, you could simply do:
>
> cp -a /path/to/PG /mnt/usb
>
> and you're done. If you're using filesystem level encryption then you'd
> have to re-encrypt the data, using something like:
>
> tar -cf - /path/to/PG | openssl -key private.key > /mnt/usb/encrypted_cluster.tar
>
> And then you would need openssl on the other system to decrypt it.
>
> Of course, either way you'd have to provide for a way to get the key
> from one system to the other.
Uh, doesn't scp do this? I have trouble seeing how avoiding calling
openssl justifies changes to our database server.
> Also, tools like pg_basebackup could be used, with nearly zero changes,
> I think, to get an encrypted copy of the database for backup purposes,
> removing the need to work out a way to handle encrypting backups.
I do think we need much more documentation on how to encrypt things,
though that is a separate issue. It might help to document how you
_should_ do things now to see the limitations we have.
> > Is the problem that you have to encrypt before sending and decrypt on
> > arrival, if you don't trust the transmission link? Is that used a lot?
> > Is having the db encrypt every write a reasonable solution to that?
>
> There's multiple use-cases here. Making it easier to copy the database
> is just one of them and it isn't the biggest one. The biggest benefit
> is that there's cases where filesystem-level encryption isn't really an
> option or, even if it is, it's not desirable for other reasons.
I am thinking NAS storage you don't trust, though there is the leakage
there.
> > As far as future features, we don't have to add the all features at this
> > time, but if someone has a good idea for an API and we can make it work
> > easily while adding this feature, why wouldn't we do that?
>
> Sure, I'm all for specific suggestions about how to improve the API, or
> just in general recommendations of how to improve the patch. The
> suggestions which have been made about key management don't really come
> across to me as specific API-level recommendations but rather "this
> would also be nice to have" kind of comments, which isn't really the
> same.
They are related, or you can't even know that because you are/were
preventing the discussion from happening. As an example, full-cluster
encryption doesn't really add any new capabilities, maybe just easier
deployment for some users. Where things get useful is fine-grained
encryption, which file system-level encryption can't do.
Also, has anyone asked users if they would find db-encryption better
than file system encryption?
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:51 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
@ 2017-06-13 20:08 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 20:10 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
1 sibling, 1 reply; 93+ messages in thread
From: Peter Eisentraut @ 2017-06-13 20:08 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; Stephen Frost <sfrost@snowman.net>; +Cc: Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
On 6/13/17 15:51, Bruce Momjian wrote:
> Isn't the leakage controlled by OS permissions, so is it really leakage,
> i.e., if you can see the leakage, you probably have bypassed the OS
> permissions and see the key and data anyway.
One scenario (among many) is when you're done with the disk. If the
content was fully encrypted, then you can just throw it into the trash
or have your provider dispose of it or reuse it. If not, then,
depending on policy, you will have to physically obtain it and burn it.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:51 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 20:08 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
@ 2017-06-13 20:10 ` Bruce Momjian <bruce@momjian.us>
0 siblings, 0 replies; 93+ messages in thread
From: Bruce Momjian @ 2017-06-13 20:10 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; +Cc: Stephen Frost <sfrost@snowman.net>; Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
On Tue, Jun 13, 2017 at 04:08:29PM -0400, Peter Eisentraut wrote:
> On 6/13/17 15:51, Bruce Momjian wrote:
> > Isn't the leakage controlled by OS permissions, so is it really leakage,
> > i.e., if you can see the leakage, you probably have bypassed the OS
> > permissions and see the key and data anyway.
>
> One scenario (among many) is when you're done with the disk. If the
> content was fully encrypted, then you can just throw it into the trash
> or have your provider dispose of it or reuse it. If not, then,
> depending on policy, you will have to physically obtain it and burn it.
Oh, I see your point --- db-level encryption stores the file system as
mountable on the device, while it is not with storage-level encryption
--- got it.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:51 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
@ 2017-06-13 22:29 ` Stephen Frost <sfrost@snowman.net>
2017-06-14 14:53 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
1 sibling, 1 reply; 93+ messages in thread
From: Stephen Frost @ 2017-06-13 22:29 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
Bruce,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Tue, Jun 13, 2017 at 03:20:12PM -0400, Stephen Frost wrote:
> > > OK, so let's go back. You are saying there are no security benefits to
> > > this vs. file system encryption.
> >
> > I'm not sure that I can see any, myself.. Perhaps I'm wrong there, but
> > it seems unlikely that this would be an improvement over filesystem
> > level encryption in the general sense. I'm not sure that I see it as
> > really worse than filesystem-level encryption either, to be clear.
> > There's a bit of increased information leakage, as Peter mentioned and I
> > agreed with, but it's not a lot and I expect in most cases that
> > information leak would be acceptable. That seems like something which
> > would need to be considered on a case-by-case basis.
>
> Isn't the leakage controlled by OS permissions, so is it really leakage,
> i.e., if you can see the leakage, you probably have bypassed the OS
> permissions and see the key and data anyway.
The case I'm mainly considering is if you somehow lose control over the
medium in which the encrypted database resides- that is, someone steals
the hard drive, or perhaps the hard drive is sold without properly being
wiped first, things like that.
In such a case, there's no OS permissions to bypass because the OS is
now controlled by the attacker. In that case, if the key wasn't stored
on the hard drive, then the attacker would be able to see the contents
of the filesystem and the associated metadata, but not the contents of
the cluster.
In that case, the distinction between filesystem-level encryption and
PG-level encryption is that with filesystem-level encryption the
attacker wouldn't be able to even see what files exist or any metadata
about them, whereas with PG-level encryption that information would be
available to the attacker.
In terms of an online attack where an attacker has gained access to the
system then, in general, you're right that if the attacker is able to
see into the PG data directory at all then they've figured out a way to
bypass the OS-level permissions and would then be able to see the data
directly anyway. That's a different scenario which would most likely be
helped by something like SELinux being used, which could prevent the
attacker from being able to look at the PG data directory because the
attacker has connected to the system from a network which isn't allowed
to directly access those files.
> > > The benefit is allowing configuration
> > > in the database rather than the OS?
> >
> > No, the benefit is that the database administrator can configure it and
> > set it up and not have to get an OS-level administrator involved. There
> > may also be other reasons why filesystem-level encryption is difficult
> > to set up or use in a certain environment, but this wouldn't depend on
> > anything OS-related and therefore could be done.
>
> OK, my only point here is that we are going down a slippery slope of
> implementing OS things in the database. There is nothing wrong with
> that but it has often been something we have avoided, because of the
> added complexity required in the db server.
I'm not sure that I actually agree that encryption is really solely an
OS-level function, or even that encryption at rest is solely the OS's
job. As a counter-example, I encrypt my SSH keys and GPG keys
routinely, even when I'm using OS-level filesystem encryption. Perhaps
that's excessive of me, but, well, I don't find it so. :)
> As a counter-example, we only added an external collation library to
> Postgres when we clearly saw a benefit, e.g. detecting collation
> changes.
Right, and there's also the potential for adding more flexibility down
the road, which I'm certainly all for, but I see value in having even
this initial version of the feature too.
> > Of course, either way you'd have to provide for a way to get the key
> > from one system to the other.
>
> Uh, doesn't scp do this? I have trouble seeing how avoiding calling
> openssl justifies changes to our database server.
scp may not be an option as it requires network connectivity between the
systems. This is also just one of the use-cases, and not the main
reason, at least in my view, to add this feature.
> > Also, tools like pg_basebackup could be used, with nearly zero changes,
> > I think, to get an encrypted copy of the database for backup purposes,
> > removing the need to work out a way to handle encrypting backups.
>
> I do think we need much more documentation on how to encrypt things,
> though that is a separate issue. It might help to document how you
> _should_ do things now to see the limitations we have.
Improving our documentation would certainly be good, but I'm not sure
that we can really recommend specific ways of doing things like
filesystem-level encryption, as that's really OS-dependent and there
could be trade-offs in different ways a given OS might provide that
capability. I'm not sure that having our documentation generically say
"you should use filesystem-level encryption" would really be very
helpful.
Perhaps I misunderstood your suggestion here though..?
> > > Is the problem that you have to encrypt before sending and decrypt on
> > > arrival, if you don't trust the transmission link? Is that used a lot?
> > > Is having the db encrypt every write a reasonable solution to that?
> >
> > There's multiple use-cases here. Making it easier to copy the database
> > is just one of them and it isn't the biggest one. The biggest benefit
> > is that there's cases where filesystem-level encryption isn't really an
> > option or, even if it is, it's not desirable for other reasons.
>
> I am thinking NAS storage you don't trust, though there is the leakage
> there.
Yes, NAS or SAN storage where you don't want the NAS/SAN administrators
to have access to the data is a good example of where encryption would
be useful. Of course, either filesystem-level or PG-level encryption
would address that, but with the trade-off that PG-level encryption
would allow the NAS/SAN administrators to see the file metadata, as
discussed above.
> > > As far as future features, we don't have to add the all features at this
> > > time, but if someone has a good idea for an API and we can make it work
> > > easily while adding this feature, why wouldn't we do that?
> >
> > Sure, I'm all for specific suggestions about how to improve the API, or
> > just in general recommendations of how to improve the patch. The
> > suggestions which have been made about key management don't really come
> > across to me as specific API-level recommendations but rather "this
> > would also be nice to have" kind of comments, which isn't really the
> > same.
>
> They are related, or you can't even know that because you are/were
> preventing the discussion from happening. As an example, full-cluster
> encryption doesn't really add any new capabilities, maybe just easier
> deployment for some users. Where things get useful is fine-grained
> encryption, which file system-level encryption can't do.
I agree that having fine-grained control would be helpful, but as I was
trying to get at, that's going to involve new catalog tables and SQL
syntax, and we'd still need to address how to encrypt the cluster-wide
files anyway, which is what this patch is working to do.
> Also, has anyone asked users if they would find db-encryption better
> than file system encryption?
I've been asked for this capability multiple times from our users and
have generally pushed back and encouraged filesystem-level encryption.
That hasn't always been an acceptable solution, unfortunately.
Thanks!
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 19:51 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-13 22:29 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
@ 2017-06-14 14:53 ` Bruce Momjian <bruce@momjian.us>
0 siblings, 0 replies; 93+ messages in thread
From: Bruce Momjian @ 2017-06-14 14:53 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
On Tue, Jun 13, 2017 at 06:29:20PM -0400, Stephen Frost wrote:
> > Isn't the leakage controlled by OS permissions, so is it really leakage,
> > i.e., if you can see the leakage, you probably have bypassed the OS
> > permissions and see the key and data anyway.
>
> The case I'm mainly considering is if you somehow lose control over the
> medium in which the encrypted database resides- that is, someone steals
> the hard drive, or perhaps the hard drive is sold without properly being
> wiped first, things like that.
>
> In such a case, there's no OS permissions to bypass because the OS is
> now controlled by the attacker. In that case, if the key wasn't stored
> on the hard drive, then the attacker would be able to see the contents
> of the filesystem and the associated metadata, but not the contents of
> the cluster.
>
> In that case, the distinction between filesystem-level encryption and
> PG-level encryption is that with filesystem-level encryption the
> attacker wouldn't be able to even see what files exist or any metadata
> about them, whereas with PG-level encryption that information would be
> available to the attacker.
Yes, Peter Eisentraut pointed that out in an earlier email in this
thread. Thanks.
> > > > The benefit is allowing configuration
> > > > in the database rather than the OS?
> > >
> > > No, the benefit is that the database administrator can configure it and
> > > set it up and not have to get an OS-level administrator involved. There
> > > may also be other reasons why filesystem-level encryption is difficult
> > > to set up or use in a certain environment, but this wouldn't depend on
> > > anything OS-related and therefore could be done.
> >
> > OK, my only point here is that we are going down a slippery slope of
> > implementing OS things in the database. There is nothing wrong with
> > that but it has often been something we have avoided, because of the
> > added complexity required in the db server.
>
> I'm not sure that I actually agree that encryption is really solely an
> OS-level function, or even that encryption at rest is solely the OS's
> job. As a counter-example, I encrypt my SSH keys and GPG keys
> routinely, even when I'm using OS-level filesystem encryption. Perhaps
> that's excessive of me, but, well, I don't find it so. :)
Well, I think SSH and GPG keys are a case of selective encryption, which
is where I said database encryption could really be a win, because you
can't do that outside the database. Just to go crazy, here is something
I think would be cool for a fully or partially encrypted data row:
row data encrypted with symmetric key k
k encrypted with public key of user 1
k encrypted with public key of user 2
hash of previous fields signed by insert user
The would allow the insert user complete control over who sees the row.
The database administrator could corrupt the row or add/remove users,
but that would be detected by the hash signature being invalid. This is
kind of like TLS bundled in the database. I think this is actually all
possible now too. :-)
> > As a counter-example, we only added an external collation library to
> > Postgres when we clearly saw a benefit, e.g. detecting collation
> > changes.
>
> Right, and there's also the potential for adding more flexibility down
> the road, which I'm certainly all for, but I see value in having even
> this initial version of the feature too.
Understood.
> > > Also, tools like pg_basebackup could be used, with nearly zero changes,
> > > I think, to get an encrypted copy of the database for backup purposes,
> > > removing the need to work out a way to handle encrypting backups.
> >
> > I do think we need much more documentation on how to encrypt things,
> > though that is a separate issue. It might help to document how you
> > _should_ do things now to see the limitations we have.
>
> Improving our documentation would certainly be good, but I'm not sure
> that we can really recommend specific ways of doing things like
> filesystem-level encryption, as that's really OS-dependent and there
> could be trade-offs in different ways a given OS might provide that
> capability. I'm not sure that having our documentation generically say
> "you should use filesystem-level encryption" would really be very
> helpful.
>
> Perhaps I misunderstood your suggestion here though..?
I was just throwing out the idea that sometimes writing things down
shows the gaps in our feature-set --- it might not apply here.
> > > > Is the problem that you have to encrypt before sending and decrypt on
> > > > arrival, if you don't trust the transmission link? Is that used a lot?
> > > > Is having the db encrypt every write a reasonable solution to that?
> > >
> > > There's multiple use-cases here. Making it easier to copy the database
> > > is just one of them and it isn't the biggest one. The biggest benefit
> > > is that there's cases where filesystem-level encryption isn't really an
> > > option or, even if it is, it's not desirable for other reasons.
> >
> > I am thinking NAS storage you don't trust, though there is the leakage
> > there.
>
> Yes, NAS or SAN storage where you don't want the NAS/SAN administrators
> to have access to the data is a good example of where encryption would
> be useful. Of course, either filesystem-level or PG-level encryption
> would address that, but with the trade-off that PG-level encryption
> would allow the NAS/SAN administrators to see the file metadata, as
> discussed above.
Uh, as far as I understand it, SAN could technically use encryption
because you are sending blocks to the network storage and you could
encrypt the blocks before transfer. However, I don't think that would
work for NAS/NFS.
> > Also, has anyone asked users if they would find db-encryption better
> > than file system encryption?
>
> I've been asked for this capability multiple times from our users and
> have generally pushed back and encouraged filesystem-level encryption.
> That hasn't always been an acceptable solution, unfortunately.
Yes, it would be good to know how often that happens, and whether we
should be adjusting Postgres to address it. The idea posted of using
plugin storage for encryption seems like a cool idea, and compression
and stuff could be added. (However, I realize encryption and compression
doesn't work well because of information leakage.)
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:24 ` Re: WIP: Data at rest encryption Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
@ 2017-06-13 20:52 ` David Fetter <david@fetter.org>
1 sibling, 0 replies; 93+ messages in thread
From: David Fetter @ 2017-06-13 20:52 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; +Cc: Stephen Frost <sfrost@snowman.net>; Ants Aasma <ants.aasma@eesti.ee>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
On Tue, Jun 13, 2017 at 10:28:14AM -0400, Peter Eisentraut wrote:
> On 6/13/17 09:24, Stephen Frost wrote:
> > but there are use-cases where it'd be really nice to be able to
> > have PG doing the encryption instead of the filesystem because
> > then you can do things like backup the database, copy it somewhere
> > else directly, and then restore it using the regular PG
> > mechanisms, as long as you have access to the key. That's not
> > something you can directly do with filesystem-level encryption
>
> Interesting point.
>
> I wonder what the proper extent of "encryption at rest" should be.
> If you encrypt just on a file or block level, then someone looking
> at the data directory or a backup can still learn a number of things
> about the number of tables, transaction rates, various configuration
> settings, and so on.
In the end, information leaks at a strictly positive baud rate because
physics (cf. Claude Shannon, et al).
Encryption at rest is one technique whereby people can slow this rate,
but there's no such thing as getting it to zero. Let's not creep this
feature in the ultimately futile attempt to do so.
> In the scenario of a sensitive application hosted on a shared
> SAN, I don't think that is good enough.
>
> Also, in the use case you describe, if you use pg_basebackup to make a
> direct encrypted copy of a data directory, I think that would mean you'd
> have to keep using the same key for all copies.
Right.
Best,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com
Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
@ 2017-06-13 15:35 ` Robert Haas <robertmhaas@gmail.com>
2017-06-13 15:51 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-14 15:10 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2 siblings, 2 replies; 93+ messages in thread
From: Robert Haas @ 2017-06-13 15:35 UTC (permalink / raw)
To: Ants Aasma <ants.aasma@eesti.ee>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Jun 12, 2017 at 5:11 PM, Ants Aasma <ants.aasma@eesti.ee> wrote:
> Fundamentally there doesn't seem to be a big benefit of implementing
> the encryption at PostgreSQL level instead of the filesystem. The
> patch doesn't take any real advantage from the higher level knowledge
> of the system, nor do I see much possibility for it to do that. The
> main benefit for us is that it's much easier to get a PostgreSQL based
> solution deployed.
I agree with all of that, but ease of deployment has some value unto
itself. I think pretty much every modern operating system has some
way of encrypting a filesystem, but it's different on Linux vs.
Windows vs. macOS vs. BSD, and you probably need to be the system
administrator on any of those systems in order to set it up.
Something built into PostgreSQL could run without administrator
privileges and work the same way on every platform we support. That
would be useful.
Of course, what would be even more useful is fine-grained encryption -
encrypt these tables (and the corresponding indexes, toast tables, and
WAL records related to any of that) with this key, encrypt these other
tables (and the same list of associated stuff) with this other key,
and leave the rest unencrypted. The problem with that is that you
probably can't run recovery without all of the keys, and even on a
clean startup there would be a good deal of engineering work involved
in refusing access to tables whose key hadn't been provided yet. I
don't think we should wait to have this feature until all of those
problems are solved. In my opinion, something coarse-grained that
just encrypts the whole cluster would be a pretty useful place to
start and would meet the needs of enough people to be worthwhile all
on its own. Performance is likely to be poor on large databases,
because every time a page transits between shared_buffers and the
buffer cache we've got to en/decrypt, but as long as it's only poor
for the people who opt into the feature I don't see a big problem with
that.
I anticipate that one of the trickier problems here will be handling
encryption of the write-ahead log. Suppose you encrypt WAL a block at
a time. In the current system, once you've written and flushed a
block, you can consider it durably committed, but if that block is
encrypted, this is no longer true. A crash might tear the block,
making it impossible to decrypt. Replay will therefore stop at the
end of the previous block, not at the last record actually flushed as
would happen today. So, your synchronous_commit suddenly isn't. A
similar problem will occur any other page where we choose not to
protect against torn pages using full page writes. For instance,
unless checksums are enabled or wal_log_hints=on, we'll write a data
page where a single bit has been flipped and assume that the bit will
either make it to disk or not; the page can't really be torn in any
way that hurts us. But with encryption that's no longer true, because
the hint bit will turn into much more than a single bit flip, and
rereading that page with half old and half new contents will be the
end of the world (TM). I don't know off-hand whether we're
protecting, say, CLOG page writes with FPWs.: because setting a couple
of bits is idempotent and doesn't depend on the existing page
contents, we might not need it currently, but with encryption, every
bit in the page depends on every other bit in the page, so we
certainly would. I don't know how many places we've got assumptions
like this baked into the system, but I'm guessing there are a bunch.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 15:35 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
@ 2017-06-13 15:51 ` Bruce Momjian <bruce@momjian.us>
1 sibling, 0 replies; 93+ messages in thread
From: Bruce Momjian @ 2017-06-13 15:51 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Ants Aasma <ants.aasma@eesti.ee>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Jun 13, 2017 at 11:35:03AM -0400, Robert Haas wrote:
> I anticipate that one of the trickier problems here will be handling
> encryption of the write-ahead log. Suppose you encrypt WAL a block at
> a time. In the current system, once you've written and flushed a
> block, you can consider it durably committed, but if that block is
> encrypted, this is no longer true. A crash might tear the block,
> making it impossible to decrypt. Replay will therefore stop at the
> end of the previous block, not at the last record actually flushed as
> would happen today. So, your synchronous_commit suddenly isn't. A
> similar problem will occur any other page where we choose not to
> protect against torn pages using full page writes. For instance,
> unless checksums are enabled or wal_log_hints=on, we'll write a data
> page where a single bit has been flipped and assume that the bit will
> either make it to disk or not; the page can't really be torn in any
> way that hurts us. But with encryption that's no longer true, because
> the hint bit will turn into much more than a single bit flip, and
> rereading that page with half old and half new contents will be the
> end of the world (TM). I don't know off-hand whether we're
> protecting, say, CLOG page writes with FPWs.: because setting a couple
> of bits is idempotent and doesn't depend on the existing page
> contents, we might not need it currently, but with encryption, every
> bit in the page depends on every other bit in the page, so we
> certainly would. I don't know how many places we've got assumptions
> like this baked into the system, but I'm guessing there are a bunch.
That is not necessary true. You are describing a cipher mode where the
user data goes through the cipher, e.g. AES in CBC mode. However, if
you are using a stream cipher based on a block cipher, e.g. CTR, GCM,
you XOR the user data with a random bit stream, and in that case one bit
change in user data would be one bit change in the cipher output.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 15:35 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
@ 2017-06-14 15:10 ` Ants Aasma <ants.aasma@eesti.ee>
2017-06-14 15:26 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
1 sibling, 1 reply; 93+ messages in thread
From: Ants Aasma @ 2017-06-14 15:10 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Jun 13, 2017 at 6:35 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> Of course, what would be even more useful is fine-grained encryption -
> encrypt these tables (and the corresponding indexes, toast tables, and
> WAL records related to any of that) with this key, encrypt these other
> tables (and the same list of associated stuff) with this other key,
> and leave the rest unencrypted. The problem with that is that you
> probably can't run recovery without all of the keys, and even on a
> clean startup there would be a good deal of engineering work involved
> in refusing access to tables whose key hadn't been provided yet.
That's pretty much the reason why I decided to skip anything more
complicated for now. Anything that would be able to run recovery
without knowing the encryption key looks like an order of magnitude
more complicated to implement.
> Performance is likely to be poor on large databases,
> because every time a page transits between shared_buffers and the
> buffer cache we've got to en/decrypt, but as long as it's only poor
> for the people who opt into the feature I don't see a big problem with
> that.
It would make sense to tune the database with large shared buffers if
encryption is enabled. That should make sure that most shared buffers
traffic is going to disk anyway. As for performance, I have a
prototype assembly implementation of AES that does 3GB/s/core on my
laptop. If we add that behind a CPUID check the overhead should be
quite reasonable.
> I anticipate that one of the trickier problems here will be handling
> encryption of the write-ahead log. Suppose you encrypt WAL a block at
> a time. In the current system, once you've written and flushed a
> block, you can consider it durably committed, but if that block is
> encrypted, this is no longer true. A crash might tear the block,
> making it impossible to decrypt. Replay will therefore stop at the
> end of the previous block, not at the last record actually flushed as
> would happen today.
My patch is currenly doing a block at a time for WAL. The XTS mode
used to encrypt has the useful property that blocks that share
identical prefix unencrypted also have identical prefix when
encrypted. It requires that the tearing is 16B aligned, but I think
that is true for pretty much all storage systems. That property of
course has security downsides, but for table/index storage we have a
nonce in the form of LSN in the page header eliminating the issue.
> So, your synchronous_commit suddenly isn't. A
> similar problem will occur any other page where we choose not to
> protect against torn pages using full page writes. For instance,
> unless checksums are enabled or wal_log_hints=on, we'll write a data
> page where a single bit has been flipped and assume that the bit will
> either make it to disk or not; the page can't really be torn in any
> way that hurts us. But with encryption that's no longer true, because
> the hint bit will turn into much more than a single bit flip, and
> rereading that page with half old and half new contents will be the
> end of the world (TM). I don't know off-hand whether we're
> protecting, say, CLOG page writes with FPWs.: because setting a couple
> of bits is idempotent and doesn't depend on the existing page
> contents, we might not need it currently, but with encryption, every
> bit in the page depends on every other bit in the page, so we
> certainly would. I don't know how many places we've got assumptions
> like this baked into the system, but I'm guessing there are a bunch.
I think we need to require wal_log_hints=on when encryption is
enabled. Currently I have not considered tearing on CLOG bits. Other
SLRUs probably have similar issues. I need to think a bit about how to
solve that.
Regards,
Ants Aasma
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 15:35 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-14 15:10 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
@ 2017-06-14 15:26 ` Bruce Momjian <bruce@momjian.us>
2017-06-14 15:41 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
0 siblings, 1 reply; 93+ messages in thread
From: Bruce Momjian @ 2017-06-14 15:26 UTC (permalink / raw)
To: Ants Aasma <ants.aasma@eesti.ee>; +Cc: Robert Haas <robertmhaas@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Jun 14, 2017 at 06:10:32PM +0300, Ants Aasma wrote:
> On Tue, Jun 13, 2017 at 6:35 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> > Performance is likely to be poor on large databases,
> > because every time a page transits between shared_buffers and the
> > buffer cache we've got to en/decrypt, but as long as it's only poor
> > for the people who opt into the feature I don't see a big problem with
> > that.
>
> It would make sense to tune the database with large shared buffers if
> encryption is enabled. That should make sure that most shared buffers
> traffic is going to disk anyway. As for performance, I have a
> prototype assembly implementation of AES that does 3GB/s/core on my
> laptop. If we add that behind a CPUID check the overhead should be
> quite reasonable.
Are you checking the CPU type or if AES instructions are enabled on the
CPU? I ask this because I just realized in researching my new TLS talk
that my BIOS defaults to AES instructions disabled, and I had to
manually enable it.
> > I anticipate that one of the trickier problems here will be handling
> > encryption of the write-ahead log. Suppose you encrypt WAL a block at
> > a time. In the current system, once you've written and flushed a
> > block, you can consider it durably committed, but if that block is
> > encrypted, this is no longer true. A crash might tear the block,
> > making it impossible to decrypt. Replay will therefore stop at the
> > end of the previous block, not at the last record actually flushed as
> > would happen today.
>
> My patch is currenly doing a block at a time for WAL. The XTS mode
Uh, how are you writing partial writes to the WAL. I assume you are
doing a streaming cipher so you can write in increments, right?
> used to encrypt has the useful property that blocks that share
> identical prefix unencrypted also have identical prefix when
> encrypted. It requires that the tearing is 16B aligned, but I think
> that is true for pretty much all storage systems. That property of
> course has security downsides, but for table/index storage we have a
> nonce in the form of LSN in the page header eliminating the issue.
>
> > So, your synchronous_commit suddenly isn't. A
> > similar problem will occur any other page where we choose not to
> > protect against torn pages using full page writes. For instance,
> > unless checksums are enabled or wal_log_hints=on, we'll write a data
> > page where a single bit has been flipped and assume that the bit will
> > either make it to disk or not; the page can't really be torn in any
> > way that hurts us. But with encryption that's no longer true, because
> > the hint bit will turn into much more than a single bit flip, and
> > rereading that page with half old and half new contents will be the
> > end of the world (TM). I don't know off-hand whether we're
> > protecting, say, CLOG page writes with FPWs.: because setting a couple
> > of bits is idempotent and doesn't depend on the existing page
> > contents, we might not need it currently, but with encryption, every
> > bit in the page depends on every other bit in the page, so we
> > certainly would. I don't know how many places we've got assumptions
> > like this baked into the system, but I'm guessing there are a bunch.
>
> I think we need to require wal_log_hints=on when encryption is
> enabled. Currently I have not considered tearing on CLOG bits. Other
> SLRUs probably have similar issues. I need to think a bit about how to
> solve that.
I am not sure if clog even needs to be encrypted.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 15:35 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-14 15:10 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-14 15:26 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
@ 2017-06-14 15:41 ` Ants Aasma <ants.aasma@eesti.ee>
2017-06-14 15:57 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 93+ messages in thread
From: Ants Aasma @ 2017-06-14 15:41 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Robert Haas <robertmhaas@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Jun 14, 2017 at 6:26 PM, Bruce Momjian <bruce@momjian.us> wrote:
> Are you checking the CPU type or if AES instructions are enabled on the
> CPU? I ask this because I just realized in researching my new TLS talk
> that my BIOS defaults to AES instructions disabled, and I had to
> manually enable it.
There is zero code for that now, but the plan was to check the CPUID
instruction. My understanding is that it should report what is
currently enabled on the CPU. Will double check when actually writing
the code for the check.
>> > I anticipate that one of the trickier problems here will be handling
>> > encryption of the write-ahead log. Suppose you encrypt WAL a block at
>> > a time. In the current system, once you've written and flushed a
>> > block, you can consider it durably committed, but if that block is
>> > encrypted, this is no longer true. A crash might tear the block,
>> > making it impossible to decrypt. Replay will therefore stop at the
>> > end of the previous block, not at the last record actually flushed as
>> > would happen today.
>>
>> My patch is currenly doing a block at a time for WAL. The XTS mode
>
> Uh, how are you writing partial writes to the WAL. I assume you are
> doing a streaming cipher so you can write in increments, right?
We were doing 8kB page aligned writes to WAL anyway. I just encrypt
the block before it gets written out.
>> I think we need to require wal_log_hints=on when encryption is
>> enabled. Currently I have not considered tearing on CLOG bits. Other
>> SLRUs probably have similar issues. I need to think a bit about how to
>> solve that.
>
> I am not sure if clog even needs to be encrypted.
Me neither, but it currently is, and it looks like that's broken in a
"silently corrupts your data" way in face of torn writes. Using OFB
mode (xor plaintext with pseudorandom stream for cipher) looks like it
might help here, if other approaches fail.
Regards,
Ants Aasma
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 15:35 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
2017-06-14 15:10 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
2017-06-14 15:26 ` Re: WIP: Data at rest encryption Bruce Momjian <bruce@momjian.us>
2017-06-14 15:41 ` Re: WIP: Data at rest encryption Ants Aasma <ants.aasma@eesti.ee>
@ 2017-06-14 15:57 ` Bruce Momjian <bruce@momjian.us>
0 siblings, 0 replies; 93+ messages in thread
From: Bruce Momjian @ 2017-06-14 15:57 UTC (permalink / raw)
To: Ants Aasma <ants.aasma@eesti.ee>; +Cc: Robert Haas <robertmhaas@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Jun 14, 2017 at 06:41:43PM +0300, Ants Aasma wrote:
> On Wed, Jun 14, 2017 at 6:26 PM, Bruce Momjian <bruce@momjian.us> wrote:
> > Are you checking the CPU type or if AES instructions are enabled on the
> > CPU? I ask this because I just realized in researching my new TLS talk
> > that my BIOS defaults to AES instructions disabled, and I had to
> > manually enable it.
>
> There is zero code for that now, but the plan was to check the CPUID
> instruction. My understanding is that it should report what is
> currently enabled on the CPU. Will double check when actually writing
> the code for the check.
Just for specifics, I have two Intel Xeon CPU E5620, but the AES
instructions were disabled for this CPU since 2012 when I bought it.
:-( The good news is that only recently have I forced https in some
pages so this is the first time I heavily need it. I now have a boot
test, which returns 16:
grep -c '\<aes\>' /proc/cpuinfo
> >> > I anticipate that one of the trickier problems here will be handling
> >> > encryption of the write-ahead log. Suppose you encrypt WAL a block at
> >> > a time. In the current system, once you've written and flushed a
> >> > block, you can consider it durably committed, but if that block is
> >> > encrypted, this is no longer true. A crash might tear the block,
> >> > making it impossible to decrypt. Replay will therefore stop at the
> >> > end of the previous block, not at the last record actually flushed as
> >> > would happen today.
> >>
> >> My patch is currently doing a block at a time for WAL. The XTS mode
> >
> > Uh, how are you writing partial writes to the WAL. I assume you are
> > doing a streaming cipher so you can write in increments, right?
>
> We were doing 8kB page aligned writes to WAL anyway. I just encrypt
> the block before it gets written out.
Oh, we do. The beauty of streaming ciphers built on block ciphers is
that you can pre-compute the cipher to be XOR'ed with the data because
the block cipher output doesn't depend on the user data. This is used
for SSH, for example.
> >> I think we need to require wal_log_hints=on when encryption is
> >> enabled. Currently I have not considered tearing on CLOG bits. Other
> >> SLRUs probably have similar issues. I need to think a bit about how to
> >> solve that.
> >
> > I am not sure if clog even needs to be encrypted.
>
> Me neither, but it currently is, and it looks like that's broken in a
> "silently corrupts your data" way in face of torn writes. Using OFB
> mode (xor plaintext with pseudorandom stream for cipher) looks like it
> might help here, if other approaches fail.
I would just document the limitation and move on.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-13 15:07 ` Re: WIP: Data at rest encryption Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Re: WIP: Data at rest encryption Robert Haas <robertmhaas@gmail.com>
@ 2017-06-18 22:11 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 0 replies; 93+ messages in thread
From: Tomas Vondra @ 2017-06-18 22:11 UTC (permalink / raw)
To: pgsql-hackers
Hi all,
I've noticed this thread got resurrected a few days ago, but I haven't
managed to read all the messages until today. I do have a bunch of
comments, but let me share them as a single consistent message instead
of sending a thousand responses to individual messages.
1) Threat model
---------------
Firstly, I think the thread would seriously benefit from an explanation
and discussion of the threat model - what types of attacks it's meant to
address, and what attacks it can't defend against.
My understanding is that data-at-rest encryption generally addresses
only the "someone stole the disk" case and pretty much nothing else.
Moreover, I claim that encryption implemented at the database-level is
strictly weaker compared to LUKS or encrypted disks, because it simply
reveals a lot more information even without decryption (file sizes,
timestamps, etc.)
That is a serious issue in practice, and researches have been proving
that for a long time now. I do recommend this paper from Cornell Tech as
a great starting point (it cites many papers relevant to this thread):
Why Your Encrypted Database Is Not Secure
Paul Grubbs, Thomas Ristenpart, Vitaly Schmatikov
http://eprint.iacr.org/2017/468.pdf
The paper explains how encryption schemes on general-purpose databases
fail, due to exactly such side-channels. MVCC, logging and other side
channels turn all attackers into "persistent passive attackers".
Now, this does not mean the feature is useless - nothing is perfect, and
security is not a binary feature. It certainly makes attacks mode
difficult compared to plaintext database. But it's untrue that it's
basically LUKS, just implemented at the database level.
I'm not suggesting that we should not pursue this idea, but the threat
model is a crucial piece of information missing in this thread.
2) How do other databases do it?
--------------------------------
It was repeatedly mentioned that other databases support this type of
encryption. So how do they deal with the hard parts? For example how do
they get and protect the encryption key?
I agree with Stephen that we should not require a full key management
from v1 of the patch, that's an incredibly difficult thing. And it
largely depends on the hardware (e.g. it should be possible to move the
key to TrustZone on ARM / SGX on Intel).
3) Why do users prefer this to FDE?
-----------------------------------
I suppose we're discussing this feature because we've been asked about
it by users/customers who can't use FDE. Can we reach to them and ask
them about the reasons? Why can't they use FDE?
It was mentioned in the thread that the FDE solutions are not portable
between different systems, but honestly - is that an issue? You probably
can't copy the datadir anyway due locale differences anyway. If you're
running multiple operating systems, FDE is just one of many differences.
4) Other solutions?
-------------------
Clearly, FDE (at the block device level) and DB-level encryption are not
the only solutions. There are also filesystems-level solutions now, for
example.
ext4 (since kernel 4.1) and f2fs (since kernel 4.2) allow encryption at
directory level, are transparent to the user space, and include things
like key management (well, that's provided by kernel). NTFS can do
something quite similar using EFS.
https://lwn.net/Articles/639427/
https://blog.quarkslab.com/a-glimpse-of-ext4-filesystem-level-encryption.html
Of course, if you happen to use another filesystem (e.g. XFS), this
won't work for you. But then there's eCryptfs, for example:
https://en.wikipedia.org/wiki/ECryptfs
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: [HACKERS] WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
@ 2018-06-27 08:02 ` Antonin Houska <ah@cybertec.at>
2018-07-13 07:56 ` Re: [HACKERS] WIP: Data at rest encryption Toshi Harada <harada.toshi@po.ntt-tx.co.jp>
3 siblings, 1 reply; 93+ messages in thread
From: Antonin Houska @ 2018-06-27 08:02 UTC (permalink / raw)
To: pgsql-hackers
Ants Aasma <ants.aasma@gmail.com> wrote:
> Attached to this mail is a work in progress patch that adds an
> extensible encryption mechanism. There are some loose ends left to tie
> up, but the general concept and architecture is at a point where it's
> ready for some feedback, fresh ideas and bikeshedding.
Rebased patch is attached here, in case it helps to achieve (some of) the
goals mentioned in the related thread [1].
Besides encrypting table and WAL pages, it encrypts the temporary files
(buffile.c), data stored during logical decoding (reorderbuffer.c) and
statistics temporary files (pgstat.c). Unlike the previous version, SLRU files
(e.g. CLOG) are not encrypted (it does not seem critical and the encryption
makes torn page write quite difficult to handle).
Another difference is that we use the OpenSSL of the (tweaked) AES XTS cipher
now.
Binary upgrade from unencrypted to encrypted cluster is not implemented yet.
[1] https://www.postgresql.org/message-id/031401d3f41d$5c70ed90$1552c8b0$@lab.ntt.co.jp
--
Antonin Houska
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26, A-2700 Wiener Neustadt
Web: https://www.cybertec-postgresql.com
Attachments:
data-at-rest-encryption-wip-2018.06.27.patch (text/x-diff, 204.9K) download
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index dcfef36591..8a9f14d4b2 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -54,6 +54,7 @@
#include "replication/walreceiver.h"
#include "replication/walsender.h"
#include "storage/bufmgr.h"
+#include "storage/encryption.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/large_object.h"
@@ -77,6 +78,7 @@
#include "pg_trace.h"
extern uint32 bootstrap_data_checksum_version;
+extern char *bootstrap_encryption_sample;
/* File path names (all relative to $PGDATA) */
#define RECOVERY_COMMAND_FILE "recovery.conf"
@@ -862,6 +864,7 @@ static void LocalSetXLogInsertAllowed(void);
static void CreateEndOfRecoveryRecord(void);
static void CheckPointGuts(XLogRecPtr checkPointRedo, int flags);
static void KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo);
+static void XLogWritePages(char *from, int npages);
static XLogRecPtr XLogGetReplicationSlotMinimumLSN(void);
static void AdvanceXLInsertBuffer(XLogRecPtr upto, bool opportunistic);
@@ -1233,7 +1236,7 @@ ReserveXLogInsertLocation(int size, XLogRecPtr *StartPos, XLogRecPtr *EndPos,
uint64 endbytepos;
uint64 prevbytepos;
- size = MAXALIGN(size);
+ size = XLOG_REC_ALIGN(size);
/* All (non xlog-switch) records should contain data. */
Assert(size > SizeOfXLogRecord);
@@ -1287,7 +1290,7 @@ ReserveXLogSwitch(XLogRecPtr *StartPos, XLogRecPtr *EndPos, XLogRecPtr *PrevPtr)
uint64 startbytepos;
uint64 endbytepos;
uint64 prevbytepos;
- uint32 size = MAXALIGN(SizeOfXLogRecord);
+ uint32 size = XLOG_REC_ALIGN(SizeOfXLogRecord);
XLogRecPtr ptr;
uint32 segleft;
@@ -1583,7 +1586,7 @@ CopyXLogRecordToWAL(int write_len, bool isLogSwitch, XLogRecData *rdata,
else
{
/* Align the end position, so that the next record starts aligned */
- CurrPos = MAXALIGN64(CurrPos);
+ CurrPos = XLOG_REC_ALIGN(CurrPos);
}
if (CurrPos != EndPos)
@@ -2467,8 +2470,6 @@ XLogWrite(XLogwrtRqst WriteRqst, bool flexible)
{
char *from;
Size nbytes;
- Size nleft;
- int written;
/* Need to seek in the file? */
if (openLogOff != startoffset)
@@ -2485,30 +2486,40 @@ XLogWrite(XLogwrtRqst WriteRqst, bool flexible)
/* OK to write the page(s) */
from = XLogCtl->pages + startidx * (Size) XLOG_BLCKSZ;
nbytes = npages * (Size) XLOG_BLCKSZ;
- nleft = nbytes;
- do
+ if (data_encrypted)
{
- errno = 0;
- pgstat_report_wait_start(WAIT_EVENT_WAL_WRITE);
- written = write(openLogFile, from, nleft);
- pgstat_report_wait_end();
- if (written <= 0)
+#ifdef USE_ENCRYPTION
+ int i;
+
+ /*
+ * XXX: use larger encryption buffer to enable larger writes
+ * and reduce number of syscalls?
+ */
+ for (i = 0; i < npages; i++)
{
- if (errno == EINTR)
- continue;
- ereport(PANIC,
- (errcode_for_file_access(),
- errmsg("could not write to log file %s "
- "at offset %u, length %zu: %m",
- XLogFileNameP(ThisTimeLineID, openLogSegNo),
- openLogOff, nbytes)));
- }
- nleft -= written;
- from += written;
- } while (nleft > 0);
+ char buf[XLOG_BLCKSZ];
+ char tweak[TWEAK_SIZE];
+
+ XLogEncryptionTweak(tweak, openLogSegNo, openLogOff);
+ encrypt_block(from, buf, XLOG_BLCKSZ, tweak);
+ pgstat_report_wait_start(WAIT_EVENT_WAL_WRITE);
+ XLogWritePages(buf, 1);
+ pgstat_report_wait_end();
+
+ from += XLOG_BLCKSZ;
+ openLogOff += XLOG_BLCKSZ;
+ }
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+ else
+ {
+ XLogWritePages(from, npages);
+ openLogOff += nbytes;
+ }
/* Update state for write */
- openLogOff += nbytes;
npages = 0;
/*
@@ -2624,6 +2635,32 @@ XLogWrite(XLogwrtRqst WriteRqst, bool flexible)
}
}
+static void
+XLogWritePages(char *from, int npages)
+{
+ Size nleft = npages * (Size) XLOG_BLCKSZ;
+ Size written;
+
+ do
+ {
+ errno = 0;
+ written = write(openLogFile, from, nleft);
+ if (written <= 0)
+ {
+ if (errno == EINTR)
+ continue;
+ ereport(PANIC,
+ (errcode_for_file_access(),
+ errmsg("could not write to log file %s "
+ "at offset %u, length %zu: %m",
+ XLogFileNameP(ThisTimeLineID, openLogSegNo),
+ openLogOff, npages * (Size) XLOG_BLCKSZ)));
+ }
+ nleft -= written;
+ from += written;
+ } while (nleft > 0);
+}
+
/*
* Record the LSN for an asynchronous transaction commit/abort
* and nudge the WALWriter if there is work for it to do.
@@ -4681,6 +4718,31 @@ ReadControlFile(void)
ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("\"max_wal_size\" must be at least twice \"wal_segment_size\".")));
+ /*
+ * Initialize encryption, but not if the current backend has already done
+ * that.
+ */
+ if (ControlFile->data_encrypted && !data_encrypted)
+ {
+ char sample[ENCRYPTION_SAMPLE_SIZE];
+
+ setup_encryption(false);
+
+ memset(sample, 0, ENCRYPTION_SAMPLE_SIZE);
+ sample_encryption(sample);
+ if (memcmp(ControlFile->encryption_verification, sample, ENCRYPTION_SAMPLE_SIZE))
+ ereport(FATAL,
+ (errmsg("invalid encryption key"),
+ errdetail("The passed encryption key does not match"
+ " database encryption key.")));
+ }
+ SetConfigOption("data_encryption", DataEncryptionEnabled() ? "yes" : "no",
+ PGC_INTERNAL, PGC_S_OVERRIDE);
+
+ /*
+ * This calculation relies on data_encryption (in particular the header
+ * sizes do), so we could not do it earlier.
+ */
UsableBytesInSegment =
(wal_segment_size / XLOG_BLCKSZ * UsableBytesInPage) -
(SizeOfXLogLongPHD - SizeOfXLogShortPHD);
@@ -4768,6 +4830,16 @@ DataChecksumsEnabled(void)
}
/*
+ * Is this cluster encrypted?
+ */
+bool
+DataEncryptionEnabled(void)
+{
+ Assert(ControlFile != NULL);
+ return (ControlFile->data_encrypted);
+}
+
+/*
* Returns a fake LSN for unlogged relations.
*
* Each call generates an LSN that is greater than any previous value
@@ -5146,6 +5218,18 @@ BootStrapXLOG(void)
use_existent = false;
openLogFile = XLogFileInit(1, &use_existent, false);
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ char tweak[TWEAK_SIZE];
+
+ XLogEncryptionTweak(tweak, 1, 0);
+ encrypt_block((char *) page, (char *) page, XLOG_BLCKSZ, tweak);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
/* Write the first page with the initial record */
errno = 0;
pgstat_report_wait_start(WAIT_EVENT_WAL_BOOTSTRAP_WRITE);
@@ -5195,6 +5279,22 @@ BootStrapXLOG(void)
ControlFile->wal_log_hints = wal_log_hints;
ControlFile->track_commit_timestamp = track_commit_timestamp;
ControlFile->data_checksum_version = bootstrap_data_checksum_version;
+ ControlFile->data_encrypted = data_encrypted;
+
+ if (data_encrypted)
+ {
+ char *sample;
+
+ sample = palloc0(ENCRYPTION_SAMPLE_SIZE);
+ sample_encryption(sample);
+
+ memcpy(ControlFile->encryption_verification, sample,
+ ENCRYPTION_SAMPLE_SIZE);
+ pfree(sample);
+ }
+ else
+ memset(ControlFile->encryption_verification, 0,
+ ENCRYPTION_SAMPLE_SIZE);
/* some additional ControlFile fields are set in WriteControlFile() */
@@ -11710,6 +11810,18 @@ retry:
Assert(targetPageOff == readOff);
Assert(reqLen <= readLen);
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ char tweak[TWEAK_SIZE];
+
+ XLogEncryptionTweak(tweak, readSegNo, readOff);
+ decrypt_block(readBuf, readBuf, XLOG_BLCKSZ, tweak);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
*readTLI = curFileTLI;
/*
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 5bea073a2b..148d436632 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -28,6 +28,7 @@
#include "miscadmin.h"
#include "replication/origin.h"
#include "storage/bufmgr.h"
+#include "storage/encryption.h"
#include "storage/proc.h"
#include "utils/memutils.h"
#include "pg_trace.h"
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index dd96cef8f0..be7a420b96 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -24,6 +24,7 @@
#include "catalog/pg_control.h"
#include "common/pg_lzcompress.h"
#include "replication/origin.h"
+#include "storage/encryption.h"
#ifndef FRONTEND
#include "utils/memutils.h"
@@ -317,9 +318,9 @@ XLogReadRecord(XLogReaderState *state, XLogRecPtr RecPtr, char **errormsg)
*
* NB: Even though we use an XLogRecord pointer here, the whole record
* header might not fit on this page. xl_tot_len is the first field of the
- * struct, so it must be on this page (the records are MAXALIGNed), but we
- * cannot access any other fields until we've verified that we got the
- * whole header.
+ * struct, so it must be on this page (the records are XLOG_REC_ALIGNed)
+ * but we cannot access any other fields until we've verified that we got
+ * the whole header.
*/
record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ);
total_len = record->xl_tot_len;
@@ -461,8 +462,8 @@ XLogReadRecord(XLogReaderState *state, XLogRecPtr RecPtr, char **errormsg)
pageHeaderSize = XLogPageHeaderSize((XLogPageHeader) state->readBuf);
state->ReadRecPtr = RecPtr;
- state->EndRecPtr = targetPagePtr + pageHeaderSize
- + MAXALIGN(pageHeader->xlp_rem_len);
+ state->EndRecPtr = targetPagePtr + pageHeaderSize;
+ state->EndRecPtr += XLOG_REC_ALIGN(pageHeader->xlp_rem_len);
}
else
{
@@ -476,7 +477,7 @@ XLogReadRecord(XLogReaderState *state, XLogRecPtr RecPtr, char **errormsg)
if (!ValidXLogRecord(state, record, RecPtr))
goto err;
- state->EndRecPtr = RecPtr + MAXALIGN(total_len);
+ state->EndRecPtr = RecPtr + XLOG_REC_ALIGN(total_len);
state->ReadRecPtr = RecPtr;
memcpy(state->readRecordBuf, record, total_len);
@@ -532,6 +533,15 @@ ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen)
Assert((pageptr % XLOG_BLCKSZ) == 0);
+ /*
+ * Make sure that we only return data that can be decrypted in a sensible
+ * way. If the valid data ended in the middle of encryption block, then
+ * decryption of that last block would turn the contained data into
+ * garbage.
+ */
+ if (data_encrypted)
+ reqLen = XLOG_REC_ALIGN(reqLen);
+
XLByteToSeg(pageptr, targetSegNo, state->wal_segment_size);
targetPageOff = XLogSegmentOffset(pageptr, state->wal_segment_size);
@@ -954,9 +964,10 @@ XLogFindNextRecord(XLogReaderState *state, XLogRecPtr RecPtr)
* next page header will contain the remaining length of the
* continuation data
*
- * Note that record headers are MAXALIGN'ed
+ * Note that record headers are XLOG_REC_ALIGN'ed
*/
- if (MAXALIGN(header->xlp_rem_len) > (XLOG_BLCKSZ - pageHeaderSize))
+ if (XLOG_REC_ALIGN(header->xlp_rem_len) >
+ (XLOG_BLCKSZ - pageHeaderSize))
tmpRecPtr = targetPagePtr + XLOG_BLCKSZ;
else
{
@@ -965,7 +976,7 @@ XLogFindNextRecord(XLogReaderState *state, XLogRecPtr RecPtr)
* tmpRecPtr to point to the first valid record
*/
tmpRecPtr = targetPagePtr + pageHeaderSize
- + MAXALIGN(header->xlp_rem_len);
+ + XLOG_REC_ALIGN(header->xlp_rem_len);
break;
}
}
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index 4ecdc9220f..736059531d 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -25,6 +25,7 @@
#include "access/xlogutils.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "storage/encryption.h"
#include "storage/smgr.h"
#include "utils/guc.h"
#include "utils/hsearch.h"
@@ -653,12 +654,15 @@ XLogTruncateRelation(RelFileNode rnode, ForkNumber forkNum,
* frontend). Probably these should be merged at some point.
*/
static void
-XLogRead(char *buf, int segsize, TimeLineID tli, XLogRecPtr startptr,
- Size count)
+XLogRead(char *buf, int segsize, TimeLineID tli, XLogRecPtr startptr, Size count)
{
- char *p;
XLogRecPtr recptr;
Size nbytes;
+#ifdef USE_ENCRYPTION
+ char *decrypt_p;
+ uint32 decryptOff;
+#endif
+ char *p;
/* state maintained across calls */
static int sendFile = -1;
@@ -679,6 +683,9 @@ XLogRead(char *buf, int segsize, TimeLineID tli, XLogRecPtr startptr,
int readbytes;
startoff = XLogSegmentOffset(recptr, segsize);
+#ifdef USE_ENCRYPTION
+ decryptOff = startoff;
+#endif
/* Do we need to switch to a different xlog segment? */
if (sendFile < 0 || !XLByteInSeg(recptr, sendSegNo, segsize) ||
@@ -758,6 +765,25 @@ XLogRead(char *buf, int segsize, TimeLineID tli, XLogRecPtr startptr,
sendOff += readbytes;
nbytes -= readbytes;
p += readbytes;
+
+ /* Decrypt completed blocks */
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ while (decrypt_p + XLOG_BLCKSZ <= p)
+ {
+ char tweak[TWEAK_SIZE];
+
+ XLogEncryptionTweak(tweak, sendSegNo, decryptOff);
+ decrypt_block(decrypt_p, decrypt_p, XLOG_BLCKSZ, tweak);
+
+ decrypt_p += XLOG_BLCKSZ;
+ decryptOff += XLOG_BLCKSZ;
+ }
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
}
}
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 7e34bee63e..cf8df2f81d 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -23,6 +23,7 @@
#include "bootstrap/bootstrap.h"
#include "catalog/index.h"
#include "catalog/pg_collation.h"
+#include "catalog/pg_control.h"
#include "catalog/pg_type.h"
#include "libpq/pqsignal.h"
#include "miscadmin.h"
@@ -36,6 +37,7 @@
#include "storage/bufmgr.h"
#include "storage/bufpage.h"
#include "storage/condition_variable.h"
+#include "storage/encryption.h"
#include "storage/ipc.h"
#include "storage/proc.h"
#include "tcop/tcopprot.h"
@@ -49,7 +51,6 @@
uint32 bootstrap_data_checksum_version = 0; /* No checksum */
-
#define ALLOC(t, c) \
((t *) MemoryContextAllocZero(TopMemoryContext, (unsigned)(c) * sizeof(t)))
@@ -223,7 +224,7 @@ AuxiliaryProcessMain(int argc, char *argv[])
/* If no -x argument, we are a CheckerProcess */
MyAuxProcType = CheckerProcess;
- while ((flag = getopt(argc, argv, "B:c:d:D:Fkr:x:X:-:")) != -1)
+ while ((flag = getopt(argc, argv, "B:c:d:D:FkK:r:x:X:-:")) != -1)
{
switch (flag)
{
@@ -249,6 +250,25 @@ AuxiliaryProcessMain(int argc, char *argv[])
case 'F':
SetConfigOption("fsync", "false", PGC_POSTMASTER, PGC_S_ARGV);
break;
+#ifdef USE_OPENSSL
+ case 'K':
+ encryption_key_command = strdup(optarg);
+
+ /*
+ * When auxiliary process (typically bootstrap) starts, the
+ * control file might not exist yet. In this case we also use
+ * encryption_key_command to indicate that the encryption is
+ * enabled.
+ *
+ * Postmaster should not set this variable. Instead, it just
+ * sets data_encrypted according to the control file and child
+ * processes inherit that.
+ */
+ Assert(!IsUnderPostmaster);
+ data_encrypted = true;
+
+ break;
+#endif /* USE_OPENSSL */
case 'k':
bootstrap_data_checksum_version = PG_DATA_CHECKSUM_VERSION;
break;
@@ -370,6 +390,20 @@ AuxiliaryProcessMain(int argc, char *argv[])
if (!IsUnderPostmaster)
InitializeMaxBackends();
+ /*
+ * If data_encryption is set because of command line argument, do the
+ * setup now. (If set by postmaster, postmaster should have performed the
+ * setup.)
+ *
+ * This should only be useful for the bootstrap process. Anyone else
+ * initializes the encryption via ReadControlFile().
+ */
+ if (data_encrypted && MyAuxProcType == BootstrapProcess)
+ {
+ Assert(!IsUnderPostmaster);
+ setup_encryption(true);
+ }
+
BaseInit();
/*
diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c
index 5342f217c0..0645ac00ba 100644
--- a/src/backend/commands/dbcommands.c
+++ b/src/backend/commands/dbcommands.c
@@ -625,7 +625,12 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
*
* We don't need to copy subdirectories
*/
- copydir(srcpath, dstpath, false);
+ {
+ RelFileNode fromNode = {srctablespace, src_dboid, InvalidOid};
+ RelFileNode toNode = {dsttablespace, dboid, InvalidOid};
+
+ copydir(srcpath, dstpath, &fromNode, &toNode);
+ }
/* Record the filesystem change in XLOG */
{
@@ -1254,7 +1259,12 @@ movedb(const char *dbname, const char *tblspcname)
/*
* Copy files from the old tablespace to the new one
*/
- copydir(src_dbpath, dst_dbpath, false);
+ {
+ RelFileNode fromNode = {src_tblspcoid, db_id, InvalidOid};
+ RelFileNode toNode = {dst_tblspcoid, db_id, InvalidOid};
+
+ copydir(src_dbpath, dst_dbpath, &fromNode, &toNode);
+ }
/*
* Record the filesystem change in XLOG
@@ -2116,7 +2126,12 @@ dbase_redo(XLogReaderState *record)
*
* We don't need to copy subdirectories
*/
- copydir(src_path, dst_path, false);
+ {
+ RelFileNode fromNode = {xlrec->src_tablespace_id, xlrec->src_db_id, InvalidOid};
+ RelFileNode toNode = {xlrec->tablespace_id, xlrec->db_id, InvalidOid};
+
+ copydir(src_path, dst_path, &fromNode, &toNode);
+ }
}
else if (info == XLOG_DBASE_DROP)
{
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 084573e77c..9254c855af 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -42,6 +42,7 @@
#include "catalog/pg_database.h"
#include "catalog/pg_proc.h"
#include "common/ip.h"
+#include "lib/stringinfo.h"
#include "libpq/libpq.h"
#include "libpq/pqsignal.h"
#include "mb/pg_wchar.h"
@@ -52,6 +53,7 @@
#include "postmaster/postmaster.h"
#include "replication/walsender.h"
#include "storage/backendid.h"
+#include "storage/encryption.h"
#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
@@ -105,6 +107,10 @@
#define PGSTAT_TAB_HASH_SIZE 512
#define PGSTAT_FUNCTION_HASH_SIZE 512
+/*
+ * Size of a buffer used to read stats files from file.
+ */
+#define PGSTAT_FILE_BUFFER_SIZE 1024
/* ----------
* Total number of backends including auxiliary
@@ -309,6 +315,9 @@ static HTAB *pgstat_collect_oids(Oid catalogid);
static PgStat_TableStatus *get_tabstat_entry(Oid rel_id, bool isshared);
static void pgstat_setup_memcxt(void);
+static StringInfo pgstat_setup_serial_buffer(MemoryContext *context);
+static bool pgstat_read_file(StringInfo buf, const char *statfile,
+ FILE *fpin);
static const char *pgstat_get_wait_activity(WaitEventActivity w);
static const char *pgstat_get_wait_client(WaitEventClient w);
@@ -337,6 +346,12 @@ static void pgstat_recv_recoveryconflict(PgStat_MsgRecoveryConflict *msg, int le
static void pgstat_recv_deadlock(PgStat_MsgDeadlock *msg, int len);
static void pgstat_recv_tempfile(PgStat_MsgTempFile *msg, int len);
+static bool readFromStringInfo(StringInfo str, void *data, Size size);
+
+#ifdef USE_ENCRYPTION
+static void pgstat_encryption_tweak(char *tweak, Oid dbid, bool permament);
+#endif
+
/* ------------------------------------------------------------
* Public functions called from postmaster follow
* ------------------------------------------------------------
@@ -3784,6 +3799,15 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_DSM_FILL_ZERO_WRITE:
event_name = "DSMFillZeroWrite";
break;
+ case WAIT_EVENT_KDF_FILE_READ:
+ event_name = "KDFFileRead";
+ break;
+ case WAIT_EVENT_KDF_FILE_SYNC:
+ event_name = "KDFFileSync";
+ break;
+ case WAIT_EVENT_KDF_FILE_WRITE:
+ event_name = "KDFFileWrite";
+ break;
case WAIT_EVENT_LOCK_FILE_ADDTODATADIR_READ:
event_name = "LockFileAddToDataDirRead";
break;
@@ -4680,6 +4704,8 @@ pgstat_write_statsfiles(bool permanent, bool allDbs)
const char *tmpfile = permanent ? PGSTAT_STAT_PERMANENT_TMPFILE : pgstat_stat_tmpname;
const char *statfile = permanent ? PGSTAT_STAT_PERMANENT_FILENAME : pgstat_stat_filename;
int rc;
+ StringInfo buf;
+ MemoryContext serialContext;
elog(DEBUG2, "writing stats file \"%s\"", statfile);
@@ -4702,23 +4728,25 @@ pgstat_write_statsfiles(bool permanent, bool allDbs)
globalStats.stats_timestamp = GetCurrentTimestamp();
/*
+ * Memory for data serialization.
+ */
+ buf = pgstat_setup_serial_buffer(&serialContext);
+
+ /*
* Write the file header --- currently just a format ID.
*/
format_id = PGSTAT_FILE_FORMAT_ID;
- rc = fwrite(&format_id, sizeof(format_id), 1, fpout);
- (void) rc; /* we'll check for error with ferror */
+ appendBinaryStringInfo(buf, (char *) &format_id, sizeof(format_id));
/*
* Write global stats struct
*/
- rc = fwrite(&globalStats, sizeof(globalStats), 1, fpout);
- (void) rc; /* we'll check for error with ferror */
+ appendBinaryStringInfo(buf, (char *) &globalStats, sizeof(globalStats));
/*
* Write archiver stats struct
*/
- rc = fwrite(&archiverStats, sizeof(archiverStats), 1, fpout);
- (void) rc; /* we'll check for error with ferror */
+ appendBinaryStringInfo(buf, (char *) &archiverStats, sizeof(archiverStats));
/*
* Walk through the database table.
@@ -4742,18 +4770,53 @@ pgstat_write_statsfiles(bool permanent, bool allDbs)
* Write out the DB entry. We don't write the tables or functions
* pointers, since they're of no use to any other process.
*/
- fputc('D', fpout);
- rc = fwrite(dbentry, offsetof(PgStat_StatDBEntry, tables), 1, fpout);
- (void) rc; /* we'll check for error with ferror */
+ appendBinaryStringInfo(buf, (char *) "D", 1);
+ appendBinaryStringInfo(buf, (char *) dbentry,
+ offsetof(PgStat_StatDBEntry, tables));
+ }
+
+ /*
+ * No more output to be done.
+ */
+ appendBinaryStringInfo(buf, (char *) "E", 1);
+
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ char tweak[TWEAK_SIZE];
+
+ /*
+ * Make sure the data is aligned to ENCRYPTION_BLOCK.
+ */
+ if (buf->len % ENCRYPTION_BLOCK > 0)
+ {
+ char zerobuf[ENCRYPTION_BLOCK];
+
+ memset(zerobuf, 0, ENCRYPTION_BLOCK);
+ appendBinaryStringInfo(buf,
+ zerobuf,
+ ENCRYPTION_BLOCK - buf->len % ENCRYPTION_BLOCK);
+ }
+
+ pgstat_encryption_tweak(tweak, InvalidOid, permanent);
+ encrypt_block(buf->data, buf->data, buf->len, tweak);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
}
/*
- * No more output to be done. Close the temp file and replace the old
- * pgstat.stat with it. The ferror() check replaces testing for error
- * after each individual fputc or fwrite above.
+ * Write the data to the file.
*/
- fputc('E', fpout);
+ rc = fwrite(buf->data, buf->len, 1, fpout);
+ (void) rc; /* we'll check for error with ferror */
+
+ /* Free the temporary storage. */
+ MemoryContextResetOnly(serialContext);
+ /*
+ * Close the temp file and replace the old pgstat.stat with it.
+ */
if (ferror(fpout))
{
ereport(LOG,
@@ -4834,6 +4897,8 @@ pgstat_write_db_statsfile(PgStat_StatDBEntry *dbentry, bool permanent)
int rc;
char tmpfile[MAXPGPATH];
char statfile[MAXPGPATH];
+ StringInfo buf;
+ MemoryContext serialContext;
get_dbstat_filename(permanent, true, dbid, tmpfile, MAXPGPATH);
get_dbstat_filename(permanent, false, dbid, statfile, MAXPGPATH);
@@ -4854,11 +4919,15 @@ pgstat_write_db_statsfile(PgStat_StatDBEntry *dbentry, bool permanent)
}
/*
+ * Memory for data serialization.
+ */
+ buf = pgstat_setup_serial_buffer(&serialContext);
+
+ /*
* Write the file header --- currently just a format ID.
*/
format_id = PGSTAT_FILE_FORMAT_ID;
- rc = fwrite(&format_id, sizeof(format_id), 1, fpout);
- (void) rc; /* we'll check for error with ferror */
+ appendBinaryStringInfo(buf, (char *) &format_id, sizeof(format_id));
/*
* Walk through the database's access stats per table.
@@ -4866,9 +4935,8 @@ pgstat_write_db_statsfile(PgStat_StatDBEntry *dbentry, bool permanent)
hash_seq_init(&tstat, dbentry->tables);
while ((tabentry = (PgStat_StatTabEntry *) hash_seq_search(&tstat)) != NULL)
{
- fputc('T', fpout);
- rc = fwrite(tabentry, sizeof(PgStat_StatTabEntry), 1, fpout);
- (void) rc; /* we'll check for error with ferror */
+ appendBinaryStringInfo(buf, (char *) "T", 1);
+ appendBinaryStringInfo(buf, (char *) tabentry, sizeof(PgStat_StatTabEntry));
}
/*
@@ -4877,18 +4945,52 @@ pgstat_write_db_statsfile(PgStat_StatDBEntry *dbentry, bool permanent)
hash_seq_init(&fstat, dbentry->functions);
while ((funcentry = (PgStat_StatFuncEntry *) hash_seq_search(&fstat)) != NULL)
{
- fputc('F', fpout);
- rc = fwrite(funcentry, sizeof(PgStat_StatFuncEntry), 1, fpout);
- (void) rc; /* we'll check for error with ferror */
+ appendBinaryStringInfo(buf, (char *) "F", 1);
+ appendBinaryStringInfo(buf, (char *) funcentry, sizeof(PgStat_StatFuncEntry));
}
/*
- * No more output to be done. Close the temp file and replace the old
- * pgstat.stat with it. The ferror() check replaces testing for error
- * after each individual fputc or fwrite above.
+ * No more output to be done.
*/
- fputc('E', fpout);
+ appendBinaryStringInfo(buf, (char *) "E", 1);
+
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ char tweak[TWEAK_SIZE];
+
+ /*
+ * Make sure the data is aligned to ENCRYPTION_BLOCK.
+ */
+ if (buf->len % ENCRYPTION_BLOCK > 0)
+ {
+ char zerobuf[ENCRYPTION_BLOCK];
+ memset(zerobuf, 0, ENCRYPTION_BLOCK);
+ appendBinaryStringInfo(buf,
+ zerobuf,
+ ENCRYPTION_BLOCK - buf->len % ENCRYPTION_BLOCK);
+ }
+
+ pgstat_encryption_tweak(tweak, dbid, permanent);
+ encrypt_block(buf->data, buf->data, buf->len, tweak);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
+ /*
+ * Write the data to the file.
+ */
+ rc = fwrite(buf->data, buf->len, 1, fpout);
+ (void) rc; /* we'll check for error with ferror */
+
+ /* Free the temporary storage. */
+ MemoryContextResetOnly(serialContext);
+
+ /*
+ * Close the temp file and replace the old pgstat.stat with it.
+ */
if (ferror(fpout))
{
ereport(LOG,
@@ -4955,6 +5057,8 @@ pgstat_read_statsfiles(Oid onlydb, bool permanent, bool deep)
int32 format_id;
bool found;
const char *statfile = permanent ? PGSTAT_STAT_PERMANENT_FILENAME : pgstat_stat_filename;
+ StringInfo buf;
+ MemoryContext serialContext;
/*
* The tables will live in pgStatLocalContext.
@@ -5005,9 +5109,38 @@ pgstat_read_statsfiles(Oid onlydb, bool permanent, bool deep)
}
/*
+ * Memory for data serialization.
+ */
+ buf = pgstat_setup_serial_buffer(&serialContext);
+
+ /*
+ * Read the data into memory.
+ */
+ if (!pgstat_read_file(buf, statfile, fpin))
+ goto done;
+
+ /*
+ * Decrypt the data if it's encrypted.
+ */
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ char tweak[TWEAK_SIZE];
+
+ pgstat_encryption_tweak(tweak, InvalidOid, permanent);
+ decrypt_block(buf->data, buf->data, buf->len, tweak);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
+ /* Prepare for reading. */
+ buf->cursor = 0;
+
+ /*
* Verify it's of the expected format.
*/
- if (fread(&format_id, 1, sizeof(format_id), fpin) != sizeof(format_id) ||
+ if (!readFromStringInfo(buf, &format_id, sizeof(format_id)) ||
format_id != PGSTAT_FILE_FORMAT_ID)
{
ereport(pgStatRunningInCollector ? LOG : WARNING,
@@ -5018,7 +5151,7 @@ pgstat_read_statsfiles(Oid onlydb, bool permanent, bool deep)
/*
* Read global stats struct
*/
- if (fread(&globalStats, 1, sizeof(globalStats), fpin) != sizeof(globalStats))
+ if (!readFromStringInfo(buf, &globalStats, sizeof(globalStats)))
{
ereport(pgStatRunningInCollector ? LOG : WARNING,
(errmsg("corrupted statistics file \"%s\"", statfile)));
@@ -5039,7 +5172,7 @@ pgstat_read_statsfiles(Oid onlydb, bool permanent, bool deep)
/*
* Read archiver stats struct
*/
- if (fread(&archiverStats, 1, sizeof(archiverStats), fpin) != sizeof(archiverStats))
+ if (!readFromStringInfo(buf, &archiverStats, sizeof(archiverStats)))
{
ereport(pgStatRunningInCollector ? LOG : WARNING,
(errmsg("corrupted statistics file \"%s\"", statfile)));
@@ -5053,15 +5186,24 @@ pgstat_read_statsfiles(Oid onlydb, bool permanent, bool deep)
*/
for (;;)
{
- switch (fgetc(fpin))
+ char kind;
+
+ if (!readFromStringInfo(buf, &kind, 1))
+ {
+ ereport(pgStatRunningInCollector ? LOG : WARNING,
+ (errmsg("corrupted statistics file \"%s\"", statfile)));
+ goto done;
+ }
+
+ switch (kind)
{
/*
* 'D' A PgStat_StatDBEntry struct describing a database
* follows.
*/
case 'D':
- if (fread(&dbbuf, 1, offsetof(PgStat_StatDBEntry, tables),
- fpin) != offsetof(PgStat_StatDBEntry, tables))
+ if (!readFromStringInfo(buf, &dbbuf,
+ offsetof(PgStat_StatDBEntry, tables)))
{
ereport(pgStatRunningInCollector ? LOG : WARNING,
(errmsg("corrupted statistics file \"%s\"",
@@ -5151,6 +5293,9 @@ pgstat_read_statsfiles(Oid onlydb, bool permanent, bool deep)
done:
FreeFile(fpin);
+ /* Free the temporary storage. */
+ MemoryContextResetOnly(serialContext);
+
/* If requested to read the permanent file, also get rid of it. */
if (permanent)
{
@@ -5188,6 +5333,8 @@ pgstat_read_db_statsfile(Oid databaseid, HTAB *tabhash, HTAB *funchash,
int32 format_id;
bool found;
char statfile[MAXPGPATH];
+ StringInfo buf;
+ MemoryContext serialContext;
get_dbstat_filename(permanent, false, databaseid, statfile, MAXPGPATH);
@@ -5211,9 +5358,39 @@ pgstat_read_db_statsfile(Oid databaseid, HTAB *tabhash, HTAB *funchash,
}
/*
+ * Memory for data serialization.
+ */
+ buf = pgstat_setup_serial_buffer(&serialContext);
+
+ /*
+ * Read the data into memory.
+ */
+
+ if (!pgstat_read_file(buf, statfile, fpin))
+ goto done;
+
+ /*
+ * Decrypt the data if it's encrypted.
+ */
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ char tweak[TWEAK_SIZE];
+
+ pgstat_encryption_tweak(tweak, databaseid, permanent);
+ decrypt_block(buf->data, buf->data, buf->len, tweak);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
+ /* Prepare for reading. */
+ buf->cursor = 0;
+
+ /*
* Verify it's of the expected format.
*/
- if (fread(&format_id, 1, sizeof(format_id), fpin) != sizeof(format_id) ||
+ if (!readFromStringInfo(buf, &format_id, sizeof(format_id)) ||
format_id != PGSTAT_FILE_FORMAT_ID)
{
ereport(pgStatRunningInCollector ? LOG : WARNING,
@@ -5227,14 +5404,22 @@ pgstat_read_db_statsfile(Oid databaseid, HTAB *tabhash, HTAB *funchash,
*/
for (;;)
{
- switch (fgetc(fpin))
+ char kind;
+
+ if (!readFromStringInfo(buf, &kind, 1))
+ {
+ ereport(pgStatRunningInCollector ? LOG : WARNING,
+ (errmsg("corrupted statistics file \"%s\"", statfile)));
+ goto done;
+ }
+
+ switch (kind)
{
/*
* 'T' A PgStat_StatTabEntry follows.
*/
case 'T':
- if (fread(&tabbuf, 1, sizeof(PgStat_StatTabEntry),
- fpin) != sizeof(PgStat_StatTabEntry))
+ if (!readFromStringInfo(buf, &tabbuf, sizeof(PgStat_StatTabEntry)))
{
ereport(pgStatRunningInCollector ? LOG : WARNING,
(errmsg("corrupted statistics file \"%s\"",
@@ -5267,8 +5452,8 @@ pgstat_read_db_statsfile(Oid databaseid, HTAB *tabhash, HTAB *funchash,
* 'F' A PgStat_StatFuncEntry follows.
*/
case 'F':
- if (fread(&funcbuf, 1, sizeof(PgStat_StatFuncEntry),
- fpin) != sizeof(PgStat_StatFuncEntry))
+ if (!readFromStringInfo(buf, &funcbuf,
+ sizeof(PgStat_StatFuncEntry)))
{
ereport(pgStatRunningInCollector ? LOG : WARNING,
(errmsg("corrupted statistics file \"%s\"",
@@ -5314,6 +5499,9 @@ pgstat_read_db_statsfile(Oid databaseid, HTAB *tabhash, HTAB *funchash,
done:
FreeFile(fpin);
+ /* Free the temporary storage. */
+ MemoryContextResetOnly(serialContext);
+
if (permanent)
{
elog(DEBUG2, "removing permanent stats file \"%s\"", statfile);
@@ -5348,6 +5536,8 @@ pgstat_read_db_statsfile_timestamp(Oid databaseid, bool permanent,
FILE *fpin;
int32 format_id;
const char *statfile = permanent ? PGSTAT_STAT_PERMANENT_FILENAME : pgstat_stat_filename;
+ StringInfo buf;
+ MemoryContext serialContext;
/*
* Try to open the stats file. As above, anything but ENOENT is worthy of
@@ -5364,38 +5554,68 @@ pgstat_read_db_statsfile_timestamp(Oid databaseid, bool permanent,
}
/*
+ * Memory for data serialization.
+ */
+ buf = pgstat_setup_serial_buffer(&serialContext);
+
+ /*
+ * Read the data into memory.
+ */
+ if (!pgstat_read_file(buf, statfile, fpin))
+ goto done;
+
+ /*
+ * Decrypt the data if it's encrypted.
+ */
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ char tweak[TWEAK_SIZE];
+
+ pgstat_encryption_tweak(tweak, InvalidOid, permanent);
+ decrypt_block(buf->data, buf->data, buf->len, tweak);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
+ /* Prepare for reading. */
+ buf->cursor = 0;
+
+ /*
* Verify it's of the expected format.
*/
- if (fread(&format_id, 1, sizeof(format_id), fpin) != sizeof(format_id) ||
+ if (!readFromStringInfo(buf, &format_id, sizeof(format_id)) ||
format_id != PGSTAT_FILE_FORMAT_ID)
{
ereport(pgStatRunningInCollector ? LOG : WARNING,
(errmsg("corrupted statistics file \"%s\"", statfile)));
FreeFile(fpin);
+ MemoryContextResetOnly(serialContext);
return false;
}
/*
* Read global stats struct
*/
- if (fread(&myGlobalStats, 1, sizeof(myGlobalStats),
- fpin) != sizeof(myGlobalStats))
+ if (!readFromStringInfo(buf, &myGlobalStats, sizeof(myGlobalStats)))
{
ereport(pgStatRunningInCollector ? LOG : WARNING,
(errmsg("corrupted statistics file \"%s\"", statfile)));
FreeFile(fpin);
+ MemoryContextResetOnly(serialContext);
return false;
}
/*
* Read archiver stats struct
*/
- if (fread(&myArchiverStats, 1, sizeof(myArchiverStats),
- fpin) != sizeof(myArchiverStats))
+ if (!readFromStringInfo(buf, &myArchiverStats, sizeof(myArchiverStats)))
{
ereport(pgStatRunningInCollector ? LOG : WARNING,
(errmsg("corrupted statistics file \"%s\"", statfile)));
FreeFile(fpin);
+ MemoryContextResetOnly(serialContext);
return false;
}
@@ -5408,15 +5628,24 @@ pgstat_read_db_statsfile_timestamp(Oid databaseid, bool permanent,
*/
for (;;)
{
- switch (fgetc(fpin))
+ char kind;
+
+ if (!readFromStringInfo(buf, &kind, 1))
+ {
+ ereport(pgStatRunningInCollector ? LOG : WARNING,
+ (errmsg("corrupted statistics file \"%s\"", statfile)));
+ goto done;
+ }
+
+ switch (kind)
{
/*
* 'D' A PgStat_StatDBEntry struct describing a database
* follows.
*/
case 'D':
- if (fread(&dbentry, 1, offsetof(PgStat_StatDBEntry, tables),
- fpin) != offsetof(PgStat_StatDBEntry, tables))
+ if (!readFromStringInfo(buf, &dbentry,
+ offsetof(PgStat_StatDBEntry, tables)))
{
ereport(pgStatRunningInCollector ? LOG : WARNING,
(errmsg("corrupted statistics file \"%s\"",
@@ -5449,6 +5678,7 @@ pgstat_read_db_statsfile_timestamp(Oid databaseid, bool permanent,
done:
FreeFile(fpin);
+ MemoryContextResetOnly(serialContext);
return true;
}
@@ -5600,6 +5830,61 @@ pgstat_setup_memcxt(void)
ALLOCSET_SMALL_SIZES);
}
+/* ----------
+ * pgstat_setup_serial_buffer() -
+ *
+ * Allocate memory for data serialization and return the containing memory
+ * context.
+ *
+ * ----------
+ */
+static StringInfo
+pgstat_setup_serial_buffer(MemoryContext *context)
+{
+ MemoryContext oldCxt;
+ StringInfo result;
+
+ *context = AllocSetContextCreate(TopMemoryContext,
+ "Statistics encryption",
+ ALLOCSET_DEFAULT_SIZES);
+ oldCxt = MemoryContextSwitchTo(*context);
+ result = makeStringInfo();
+ MemoryContextSwitchTo(oldCxt);
+
+ return result;
+}
+
+/*
+ * Read statistics data from file into memory.
+ */
+static bool
+pgstat_read_file(StringInfo buf, const char *statfile, FILE *fpin)
+{
+ for (;;)
+ {
+ char bufread[PGSTAT_FILE_BUFFER_SIZE];
+ Size bytesread;
+
+ bytesread = fread(bufread, 1, PGSTAT_FILE_BUFFER_SIZE, fpin);
+ appendBinaryStringInfo(buf, bufread, bytesread);
+
+ if (bytesread < PGSTAT_FILE_BUFFER_SIZE)
+ break;
+ }
+
+ /*
+ * Check if the amount of data makes sense.
+ */
+ if (buf->len == 0 || (data_encrypted && buf->len % ENCRYPTION_BLOCK > 0))
+ {
+ ereport(pgStatRunningInCollector ? LOG : WARNING,
+ (errmsg("corrupted statistics file \"%s\"", statfile)));
+ return false;
+ }
+
+ return true;
+}
+
/* ----------
* pgstat_clear_snapshot() -
@@ -6309,6 +6594,35 @@ pgstat_write_statsfile_needed(void)
return false;
}
+/*
+ * Read "size" bytes into "data" from StringInfo.
+ *
+ * Return true iff there was enough data available.
+ *
+ * XXX Consider moving this function to stringinfo.c.
+ */
+static bool
+readFromStringInfo(StringInfo str, void *data, Size size)
+{
+ if (str->cursor + size > str->len)
+ return false;
+
+ memcpy(data, str->data + str->cursor, size);
+ str->cursor += size;
+
+ return true;
+}
+
+#ifdef USE_ENCRYPTION
+static void
+pgstat_encryption_tweak(char *tweak, Oid dbid, bool permanent)
+{
+ memset(tweak, 0, TWEAK_SIZE);
+ memcpy(tweak, &dbid, sizeof(Oid));
+ memcpy(tweak + sizeof(Oid), &permanent, sizeof(bool));
+}
+#endif /* USE_ENCRYPTION */
+
/* ----------
* pgstat_db_requested() -
*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index a4b53b33cd..907a2f5cf0 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -116,6 +116,7 @@
#include "postmaster/syslogger.h"
#include "replication/logicallauncher.h"
#include "replication/walsender.h"
+#include "storage/encryption.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/pg_shmem.h"
@@ -674,7 +675,7 @@ PostmasterMain(int argc, char *argv[])
* tcop/postgres.c (the option sets should not conflict) and with the
* common help() function in main/main.c.
*/
- while ((opt = getopt(argc, argv, "B:bc:C:D:d:EeFf:h:ijk:lN:nOo:Pp:r:S:sTt:W:-:")) != -1)
+ while ((opt = getopt(argc, argv, "B:bc:C:D:d:EeFf:h:ijK:k:lN:nOo:Pp:r:S:sTt:W:-:")) != -1)
{
switch (opt)
{
@@ -732,6 +733,12 @@ PostmasterMain(int argc, char *argv[])
/* only used by interactive backend */
break;
+#ifdef USE_OPENSSL
+ case 'K':
+ encryption_key_command = strdup(optarg);
+ break;
+#endif /* USE_OPENSSL */
+
case 'k':
SetConfigOption("unix_socket_directories", optarg, PGC_POSTMASTER, PGC_S_ARGV);
break;
@@ -1175,6 +1182,11 @@ PostmasterMain(int argc, char *argv[])
/*
* Set up shared memory and semaphores.
+ *
+ * This includes call of setup_encryption() as soon as we realize that
+ * "data_encrypted" field of the control file is set. The encryption must
+ * be initialized at the point so that "encryption_verification" field of
+ * the control file can be checked.
*/
reset_shared(PostPortNumber);
diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c
index 3f1eae38a9..c4fcb1e33f 100644
--- a/src/backend/replication/basebackup.c
+++ b/src/backend/replication/basebackup.c
@@ -1102,7 +1102,7 @@ sendDir(const char *path, int basepathlen, bool sizeonly, List *tablespaces,
/* Exclude all forks for unlogged tables except the init fork */
if (isDbDir &&
parse_filename_for_nontemp_relation(de->d_name, &relOidChars,
- &relForkNum))
+ &relForkNum, NULL))
{
/* Never exclude init forks */
if (relForkNum != INIT_FORKNUM)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 3799ad4011..61bebca84a 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -70,6 +70,7 @@
#include "replication/slot.h"
#include "replication/snapbuild.h" /* just for SnapBuildSnapDecRefcount */
#include "storage/bufmgr.h"
+#include "storage/encryption.h"
#include "storage/fd.h"
#include "storage/sinval.h"
#include "utils/builtins.h"
@@ -190,7 +191,7 @@ static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTX
static void ReorderBufferCheckSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
- int fd, ReorderBufferChange *change);
+ int fd, XLogSegNo segno, ReorderBufferChange *change);
static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn,
int *fd, XLogSegNo *segno);
static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
@@ -216,6 +217,20 @@ static void ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *t
Relation relation, ReorderBufferChange *change);
+/* ----------------------------------------------
+ * encryption / decryption of serialized changes
+ * ----------------------------------------------
+ */
+
+/*
+ * Compute encryption tweak for buffer change.
+ */
+#define REORDER_BUFFER_CHANGE_TWEAK(tweak, segno) \
+ StaticAssertStmt(sizeof(XLogSegNo) <= TWEAK_SIZE, \
+ "XLogSegNo does not fit into encryption tweak"); \
+ memset((tweak), 0, TWEAK_SIZE); \
+ memcpy((tweak), (char *) &segno, sizeof(XLogSegNo))
+
/*
* Allocate a new ReorderBuffer and clean out any old serialized state from
* prior ReorderBuffer instances for the same slot.
@@ -2134,7 +2149,7 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
errmsg("could not open file \"%s\": %m", path)));
}
- ReorderBufferSerializeChange(rb, txn, fd, change);
+ ReorderBufferSerializeChange(rb, txn, fd, curOpenSegNo, change);
dlist_delete(&change->node);
ReorderBufferReturnChange(rb, change);
@@ -2155,12 +2170,26 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
*/
static void
ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
- int fd, ReorderBufferChange *change)
+ int fd, XLogSegNo segno,
+ ReorderBufferChange *change)
{
ReorderBufferDiskChange *ondisk;
- Size sz = sizeof(ReorderBufferDiskChange);
+ Size sz,
+ sz_hdr;
+ char *outbuf;
+
+ sz = sizeof(ReorderBufferDiskChange);
- ReorderBufferSerializeReserve(rb, sz);
+ /*
+ * As the on-disk change has variable length, the header will have to be
+ * de-serialized separate, see ReorderBufferRestoreChanges(). Therefore we
+ * also need to encrypt it separate. Make sure the size is appropriate.
+ */
+ if (data_encrypted)
+ sz = TYPEALIGN(ENCRYPTION_BLOCK, sz);
+
+ sz_hdr = sz;
+ ReorderBufferSerializeReserve(rb, sz_hdr);
ondisk = (ReorderBufferDiskChange *) rb->outbuf;
memcpy(&ondisk->change, change, sizeof(ReorderBufferChange));
@@ -2197,6 +2226,15 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
}
/* make sure we have enough space */
+ if (data_encrypted)
+ {
+ /*
+ * Encryption works with blocks. As the changes are stored
+ * and retrieved separately, we also have to decrypt them
+ * alone.
+ */
+ sz = TYPEALIGN(ENCRYPTION_BLOCK, sz);
+ }
ReorderBufferSerializeReserve(rb, sz);
data = ((char *) rb->outbuf) + sizeof(ReorderBufferDiskChange);
@@ -2227,12 +2265,15 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
char *data;
Size prefix_size = strlen(change->data.msg.prefix) + 1;
+
sz += prefix_size + change->data.msg.message_size +
sizeof(Size) + sizeof(Size);
+
+ if (data_encrypted)
+ sz = TYPEALIGN(ENCRYPTION_BLOCK, sz);
ReorderBufferSerializeReserve(rb, sz);
data = ((char *) rb->outbuf) + sizeof(ReorderBufferDiskChange);
-
/* might have been reallocated above */
ondisk = (ReorderBufferDiskChange *) rb->outbuf;
@@ -2265,11 +2306,13 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
;
/* make sure we have enough space */
+ if (data_encrypted)
+ sz = TYPEALIGN(ENCRYPTION_BLOCK, sz);
ReorderBufferSerializeReserve(rb, sz);
+
data = ((char *) rb->outbuf) + sizeof(ReorderBufferDiskChange);
/* might have been reallocated above */
ondisk = (ReorderBufferDiskChange *) rb->outbuf;
-
memcpy(data, snap, sizeof(SnapshotData));
data += sizeof(SnapshotData);
@@ -2298,8 +2341,50 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
ondisk->size = sz;
+ /*
+ * Encrypt the change if encryption is required.
+ */
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ char tweak[TWEAK_SIZE];
+
+ REORDER_BUFFER_CHANGE_TWEAK(tweak, segno);
+
+ enlarge_encryption_buffer(ondisk->size);
+
+ /*
+ * Encrypt the header and the payload separate as explained at the top
+ * of the function.
+ */
+ encrypt_block((char *) rb->outbuf, encryption_buffer, sz_hdr,
+ tweak);
+
+ if (ondisk->size > sz_hdr)
+ encrypt_block((char *) rb->outbuf + sz_hdr,
+ encryption_buffer + sz_hdr,
+ ondisk->size - sz_hdr, tweak);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
+ /*
+ * Make sure the correct buffer is written to disk.
+ */
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ outbuf = encryption_buffer;
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+ else
+ outbuf = rb->outbuf;
+
pgstat_report_wait_start(WAIT_EVENT_REORDER_BUFFER_WRITE);
- if (write(fd, rb->outbuf, ondisk->size) != ondisk->size)
+ if (write(fd, outbuf, ondisk->size) != ondisk->size)
{
int save_errno = errno;
@@ -2327,6 +2412,10 @@ ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn,
Size restored = 0;
XLogSegNo last_segno;
dlist_mutable_iter cleanup_iter;
+ Size sz_hdr;
+#ifdef USE_ENCRYPTION
+ char encryption_tweak[TWEAK_SIZE];
+#endif
Assert(txn->first_lsn != InvalidXLogRecPtr);
Assert(txn->final_lsn != InvalidXLogRecPtr);
@@ -2345,6 +2434,17 @@ ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn,
XLByteToSeg(txn->final_lsn, last_segno, wal_segment_size);
+ /*
+ * See ReorderBufferSerializeChange for explanation.
+ */
+ sz_hdr = sizeof(ReorderBufferDiskChange);
+ if (data_encrypted)
+#ifdef USE_ENCRYPTION
+ sz_hdr = TYPEALIGN(ENCRYPTION_BLOCK, sz_hdr);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+
while (restored < max_changes_in_memory && *segno <= last_segno)
{
int readBytes;
@@ -2386,9 +2486,9 @@ ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn,
* about the total size. If we couldn't read a record, we're at the
* end of this file.
*/
- ReorderBufferSerializeReserve(rb, sizeof(ReorderBufferDiskChange));
+ ReorderBufferSerializeReserve(rb, sz_hdr);
pgstat_report_wait_start(WAIT_EVENT_REORDER_BUFFER_READ);
- readBytes = read(*fd, rb->outbuf, sizeof(ReorderBufferDiskChange));
+ readBytes = read(*fd, rb->outbuf, sz_hdr);
pgstat_report_wait_end();
/* eof */
@@ -2403,38 +2503,64 @@ ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn,
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not read from reorderbuffer spill file: %m")));
- else if (readBytes != sizeof(ReorderBufferDiskChange))
+ else if (readBytes != sz_hdr)
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not read from reorderbuffer spill file: read %d instead of %u bytes",
readBytes,
- (uint32) sizeof(ReorderBufferDiskChange))));
+ (uint32) sz_hdr)));
+
+ /*
+ * Decrypt the header if the change is decrypted, so that we know the
+ * change size.
+ */
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ REORDER_BUFFER_CHANGE_TWEAK(encryption_tweak, *segno);
+ decrypt_block(rb->outbuf, rb->outbuf, sz_hdr, encryption_tweak);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
ondisk = (ReorderBufferDiskChange *) rb->outbuf;
- ReorderBufferSerializeReserve(rb,
- sizeof(ReorderBufferDiskChange) + ondisk->size);
+ ReorderBufferSerializeReserve(rb, ondisk->size);
ondisk = (ReorderBufferDiskChange *) rb->outbuf;
pgstat_report_wait_start(WAIT_EVENT_REORDER_BUFFER_READ);
- readBytes = read(*fd, rb->outbuf + sizeof(ReorderBufferDiskChange),
- ondisk->size - sizeof(ReorderBufferDiskChange));
+ readBytes = read(*fd, rb->outbuf + sz_hdr, ondisk->size - sz_hdr);
pgstat_report_wait_end();
if (readBytes < 0)
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not read from reorderbuffer spill file: %m")));
- else if (readBytes != ondisk->size - sizeof(ReorderBufferDiskChange))
+ else if (readBytes != ondisk->size - sz_hdr)
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not read from reorderbuffer spill file: read %d instead of %u bytes",
readBytes,
- (uint32) (ondisk->size - sizeof(ReorderBufferDiskChange)))));
+ (uint32) (ondisk->size - sz_hdr))));
+
+ /*
+ * ok, read a full change from disk, now decrypt it if it's encrypted
+ * and if there's some data beyond the header.
+ */
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ if (ondisk->size > sz_hdr)
+ decrypt_block(rb->outbuf + sz_hdr, rb->outbuf + sz_hdr,
+ ondisk->size - sz_hdr, encryption_tweak);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
/*
- * ok, read a full change from disk, now restore it into proper
- * in-memory format
+ * Restore the change into proper in-memory format
*/
ReorderBufferRestoreChange(rb, txn, rb->outbuf);
restored++;
diff --git a/src/backend/storage/file/buffile.c b/src/backend/storage/file/buffile.c
index efbede7629..ca0848dd20 100644
--- a/src/backend/storage/file/buffile.c
+++ b/src/backend/storage/file/buffile.c
@@ -47,12 +47,17 @@
#include "storage/fd.h"
#include "storage/buffile.h"
#include "storage/buf_internals.h"
+#include "storage/encryption.h"
+#include "utils/datetime.h"
#include "utils/resowner.h"
/*
* We break BufFiles into gigabyte-sized segments, regardless of RELSEG_SIZE.
* The reason is that we'd like large BufFiles to be spread across multiple
* tablespaces when available.
+ *
+ * The number of useful bytes is appended to each segment of shared encrypted
+ * file, see BufFile.useful.
*/
#define MAX_PHYSICAL_FILESIZE 0x40000000
#define BUFFILE_SEG_SIZE (MAX_PHYSICAL_FILESIZE / BLCKSZ)
@@ -68,6 +73,35 @@ struct BufFile
/* all files except the last have length exactly MAX_PHYSICAL_FILESIZE */
File *files; /* palloc'd array with numFiles entries */
off_t *offsets; /* palloc'd array with numFiles entries */
+#ifdef USE_ENCRYPTION
+
+ /*
+ * If the file is encrypted, only the whole buffer can be loaded / dumped
+ * --- see BufFileLoadBuffer() for more info --- whether it's space is
+ * used up or not. Therefore we need to keep track of the actual on-disk
+ * size buffer of each component file, as it would be if there was no
+ * encryption.
+ *
+ * List would make coding simpler, however would not contribute to
+ * performance. Random access is important here.
+ */
+ off_t *useful;
+
+ /*
+ * The array may need to be expanded independent from extendBufFile()
+ * (i.e. earlier than the buffer gets dumped), so store the number of
+ * elements separate from numFiles.
+ */
+ int nuseful;
+
+ /*
+ * Segment number is used to compute encryption tweak so we must remember
+ * the original numbers of segments if the file is encrypted and if it was
+ * passed as target to BufFileAppend() at least once. If this field is
+ * NULL, ->curFile is used to compute the tweak.
+ */
+ off_t *segnos;
+#endif
/*
* offsets[i] is the current seek position of files[i]. We use this to
@@ -106,6 +140,10 @@ static void BufFileLoadBuffer(BufFile *file);
static void BufFileDumpBuffer(BufFile *file);
static int BufFileFlush(BufFile *file);
static File MakeNewSharedSegment(BufFile *file, int segment);
+#ifdef USE_ENCRYPTION
+static void BufFileTweak(char *tweak, BufFile *file);
+static void ensureUsefulArraySize(BufFile *file, int required);
+#endif
/*
* Create BufFile and perform the common initialization.
@@ -125,6 +163,28 @@ makeBufFileCommon(int nfiles)
file->pos = 0;
file->nbytes = 0;
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ file->useful = (off_t *) palloc0(sizeof(off_t) * nfiles);
+ file->nuseful = nfiles;
+ file->segnos = NULL;
+
+ /*
+ * The unused (trailing) part of the buffer should not contain
+ * undefined data: if we encrypt such a buffer and flush it to disk,
+ * the encrypted form of that "undefined part" can get zeroed due to
+ * seek and write beyond EOF. If such a buffer gets loaded and
+ * decrypted, the change of the undefined part to zeroes can affect
+ * the valid part if it does not end at block boundary. By setting the
+ * whole buffer to zeroes we ensure that the unused part always
+ * contains zeroes.
+ */
+ MemSet(file->buffer, 0, BLCKSZ);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
return file;
}
@@ -172,6 +232,22 @@ extendBufFile(BufFile *file)
(file->numFiles + 1) * sizeof(File));
file->offsets = (off_t *) repalloc(file->offsets,
(file->numFiles + 1) * sizeof(off_t));
+
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ ensureUsefulArraySize(file, file->numFiles + 1);
+
+ if (file->segnos)
+ {
+ file->segnos = (off_t *) repalloc(file->segnos,
+ (file->numFiles + 1) * sizeof(off_t));
+ file->segnos[file->numFiles] = file->numFiles;
+ }
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
file->files[file->numFiles] = pfile;
file->offsets[file->numFiles] = 0L;
file->numFiles++;
@@ -302,6 +378,7 @@ BufFileOpenShared(SharedFileSet *fileset, const char *name)
files[nfiles] = SharedFileSetOpen(fileset, segment_name);
if (files[nfiles] <= 0)
break;
+
++nfiles;
CHECK_FOR_INTERRUPTS();
@@ -317,7 +394,64 @@ BufFileOpenShared(SharedFileSet *fileset, const char *name)
errmsg("could not open BufFile \"%s\"", name)));
file = makeBufFileCommon(nfiles);
+
+ /*
+ * Shared encrypted segment should contain information on the number of
+ * useful bytes at the end.
+ */
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ off_t pos;
+ int i;
+
+ for (i = 0; i < nfiles; i++)
+ {
+ int nbytes;
+ File segment = files[i];
+
+ pos = FileSeek(segment, -sizeof(off_t), SEEK_END);
+
+ /*
+ * The word must immediately follow the last buffer of the
+ * segment.
+ */
+ if (pos <= 0 || pos % BLCKSZ != 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not find padding info in BufFile \"%s\"",
+ name)));
+
+ nbytes = FileRead(segment, (char *) &file->useful[i],
+ sizeof(off_t), WAIT_EVENT_BUFFILE_READ);
+ if (nbytes != sizeof(off_t))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read padding info from BufFile \"%s\"",
+ name)));
+ Assert(file->useful[i] > 0);
+
+ /*
+ * Keep ->offsets up-to-date.
+ */
+ file->offsets[i] = FileTell(segment);
+
+ CHECK_FOR_INTERRUPTS();
+ }
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
file->files = files;
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ file->nuseful = nfiles;
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
file->readOnly = true; /* Can't write to files opened this way */
file->fileset = fileset;
file->name = pstrdup(name);
@@ -397,6 +531,17 @@ BufFileClose(BufFile *file)
/* release the buffer space */
pfree(file->files);
pfree(file->offsets);
+
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ if (file->segnos)
+ pfree(file->segnos);
+ pfree(file->useful);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
pfree(file);
}
@@ -404,7 +549,7 @@ BufFileClose(BufFile *file)
* BufFileLoadBuffer
*
* Load some data into buffer, if possible, starting from curOffset.
- * At call, must have dirty = false, pos and nbytes = 0.
+ * At call, must have dirty = false, nbytes = 0.
* On exit, nbytes is number of bytes loaded.
*/
static void
@@ -413,6 +558,22 @@ BufFileLoadBuffer(BufFile *file)
File thisfile;
/*
+ * Only whole multiple of ENCRYPTION_BLOCK can be encrypted / decrypted,
+ * but we choose to use BLCKSZ (i.e. BufFile buffer) as the unit. The
+ * point is that curOffset is a component of the encryption tweak, and all
+ * data within particular call of encrypt_block() / decrypt_block() must
+ * have the same tweak. So whichever unit we choose we must stick on it
+ * and never encrypt / decrypt multiple units at a time.
+ *
+ * BLCKSZ also seems better choice than ENCRYPTION_BLOCK for performance
+ * purposes. We assume that alignment to BLCKSZ implies alignment to
+ * ENCRYPTION_BLOCK.
+ */
+ Assert((file->curOffset % BLCKSZ == 0 &&
+ file->curOffset % ENCRYPTION_BLOCK == 0) ||
+ !data_encrypted);
+
+ /*
* Advance to next component file if necessary and possible.
*/
if (file->curOffset >= MAX_PHYSICAL_FILESIZE &&
@@ -423,6 +584,20 @@ BufFileLoadBuffer(BufFile *file)
}
/*
+ * See makeBufFile().
+ *
+ * Actually here we only handle the case of FileRead() returning zero
+ * bytes below. In contrast, if the buffer contains any data but it's not
+ * full, it should already have the trailing zeroes (encrypted) on disk.
+ * And as the encrypted buffer is always loaded in its entirety (i.e. EOF
+ * should only appear at buffer boundary if the data is encrypted), all
+ * unused bytes of the buffer should eventually be zeroes after the
+ * decryption.
+ */
+ if (data_encrypted)
+ MemSet(file->buffer, 0, BLCKSZ);
+
+ /*
* May need to reposition physical file.
*/
thisfile = file->files[file->curFile];
@@ -445,6 +620,64 @@ BufFileLoadBuffer(BufFile *file)
file->offsets[file->curFile] += file->nbytes;
/* we choose not to advance curOffset here */
+ if (data_encrypted && file->nbytes > 0)
+ {
+#ifdef USE_ENCRYPTION
+ char tweak[TWEAK_SIZE];
+ int nbytes = file->nbytes;
+
+ /*
+ * The encrypted component file can only consist of whole number of
+ * our encryption units. (Only the whole buffers are dumped / loaded.)
+ * The only exception is that we're at the end of segment file and
+ * found the word indicating the number of useful bytes in the
+ * segment. This can only happen for shared file.
+ */
+ if (nbytes % BLCKSZ != 0)
+ {
+ Assert(nbytes == sizeof(off_t) && file->fileset != NULL);
+
+ /*
+ * This special word cannot start within the useful data.
+ */
+ Assert(file->offsets[file->curFile] - nbytes >=
+ file->useful[file->curFile]);
+
+ /*
+ * This metadata his hidden to caller, so all he needs to know
+ * that there's no real data at the end of the file.
+ */
+ file->nbytes = 0;
+ return;
+ }
+
+ BufFileTweak(tweak, file);
+
+ /*
+ * The whole block is encrypted / decrypted at once as explained
+ * above.
+ */
+ decrypt_block(file->buffer, file->buffer, BLCKSZ, tweak);
+
+#ifdef USE_ASSERT_CHECKING
+
+ /*
+ * The unused part of the buffer which we've read from disk and
+ * decrypted should only contain zeroes, as explained in front of the
+ * MemSet() call.
+ */
+ {
+ int i;
+
+ for (i = file->nbytes; i < BLCKSZ; i++)
+ Assert(file->buffer[i] == 0);
+ }
+#endif /* USE_ASSERT_CHECKING */
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
if (file->nbytes > 0)
pgBufferUsage.temp_blks_read++;
}
@@ -455,6 +688,9 @@ BufFileLoadBuffer(BufFile *file)
* Dump buffer contents starting at curOffset.
* At call, should have dirty = true, nbytes > 0.
* On exit, dirty is cleared if successful write, and curOffset is advanced.
+ *
+ * XXX Consider separate function for encrypted buffer. (No loop is needed to
+ * dump the encrypted buffer.)
*/
static void
BufFileDumpBuffer(BufFile *file)
@@ -462,6 +698,52 @@ BufFileDumpBuffer(BufFile *file)
int wpos = 0;
int bytestowrite;
File thisfile;
+ char *write_ptr;
+
+ /*
+ * See comments in BufFileLoadBuffer();
+ */
+ Assert((file->curOffset % BLCKSZ == 0 &&
+ file->curOffset % ENCRYPTION_BLOCK == 0) ||
+ !data_encrypted);
+
+ /*
+ * Caller's responsibility.
+ */
+ Assert(file->pos <= file->nbytes);
+
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ char tweak[TWEAK_SIZE];
+
+ BufFileTweak(tweak, file);
+
+ /*
+ * The amount of data encrypted must be a multiple of
+ * ENCRYPTION_BLOCK. We meet this condition simply by encrypting the
+ * whole buffer.
+ *
+ * XXX Alternatively we could get the encrypted chunk length by
+ * rounding file->nbytes up to the nearest multiple of
+ * ENCRYPTION_BLOCK, and for decryption use the
+ * file->useful[file->curFile] value to find out how many blocks
+ * should be decrypted. That would reduce I/O if the buffer is mostly
+ * empty, but (BLCKSZ / ENCRYPTION_BLOCK) calls of encrypt_block()
+ * would be needed for full buffers. See also BufFileLoadBuffer() for
+ * explanation why we must stick on the unit of data amount encrypted
+ * / decrypted.
+ */
+ if (encryption_buf_size < BLCKSZ)
+ enlarge_encryption_buffer(BLCKSZ);
+ encrypt_block(file->buffer, encryption_buffer, BLCKSZ, tweak);
+ write_ptr = encryption_buffer;
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+ else
+ write_ptr = file->buffer;
/*
* Unlike BufFileLoadBuffer, we must dump the whole buffer even if it
@@ -469,8 +751,6 @@ BufFileDumpBuffer(BufFile *file)
*/
while (wpos < file->nbytes)
{
- off_t availbytes;
-
/*
* Advance to next component file if necessary and possible.
*/
@@ -482,31 +762,56 @@ BufFileDumpBuffer(BufFile *file)
file->curOffset = 0L;
}
- /*
- * Determine how much we need to write into this file.
- */
- bytestowrite = file->nbytes - wpos;
- availbytes = MAX_PHYSICAL_FILESIZE - file->curOffset;
+ if (!data_encrypted)
+ {
+ off_t availbytes;
+
+ bytestowrite = file->nbytes - wpos;
+ availbytes = MAX_PHYSICAL_FILESIZE - file->curOffset;
- if ((off_t) bytestowrite > availbytes)
- bytestowrite = (int) availbytes;
+ if ((off_t) bytestowrite > availbytes)
+ bytestowrite = (int) availbytes;
+ }
+ else
+ {
+ /*
+ * This condition plus the alignment of curOffset to BLCKSZ
+ * (checked above) ensure that the encrypted buffer never crosses
+ * component file boundary.
+ */
+ StaticAssertStmt((MAX_PHYSICAL_FILESIZE % BLCKSZ) == 0,
+ "BLCKSZ is not whole multiple of MAX_PHYSICAL_FILESIZE");
+
+ /*
+ * Encrypted data is dumped all at once.
+ *
+ * Here we don't have to check availbytes because --- according to
+ * the assertions above --- currOffset should be lower than
+ * MAX_PHYSICAL_FILESIZE by non-zero multiple of BLCKSZ.
+ */
+ bytestowrite = BLCKSZ;
+ }
/*
* May need to reposition physical file.
*/
thisfile = file->files[file->curFile];
+
if (file->curOffset != file->offsets[file->curFile])
{
if (FileSeek(thisfile, file->curOffset, SEEK_SET) != file->curOffset)
return; /* seek failed, give up */
+
file->offsets[file->curFile] = file->curOffset;
}
bytestowrite = FileWrite(thisfile,
- file->buffer + wpos,
+ write_ptr + wpos,
bytestowrite,
WAIT_EVENT_BUFFILE_WRITE);
- if (bytestowrite <= 0)
+ if (bytestowrite <= 0 ||
+ (data_encrypted && bytestowrite != BLCKSZ))
return; /* failed to write */
+
file->offsets[file->curFile] += bytestowrite;
file->curOffset += bytestowrite;
wpos += bytestowrite;
@@ -515,25 +820,130 @@ BufFileDumpBuffer(BufFile *file)
}
file->dirty = false;
- /*
- * At this point, curOffset has been advanced to the end of the buffer,
- * ie, its original value + nbytes. We need to make it point to the
- * logical file position, ie, original value + pos, in case that is less
- * (as could happen due to a small backwards seek in a dirty buffer!)
- */
- file->curOffset -= (file->nbytes - file->pos);
- if (file->curOffset < 0) /* handle possible segment crossing */
+ if (!data_encrypted)
{
- file->curFile--;
- Assert(file->curFile >= 0);
- file->curOffset += MAX_PHYSICAL_FILESIZE;
+ /*
+ * At this point, curOffset has been advanced to the end of the
+ * buffer, ie, its original value + nbytes. We need to make it point
+ * to the logical file position, ie, original value + pos, in case
+ * that is less (as could happen due to a small backwards seek in a
+ * dirty buffer!)
+ */
+ file->curOffset -= (file->nbytes - file->pos);
+ if (file->curOffset < 0) /* handle possible segment crossing */
+ {
+ file->curFile--;
+ Assert(file->curFile >= 0);
+ file->curOffset += MAX_PHYSICAL_FILESIZE;
+ }
+
+ /*
+ * Now we can set the buffer empty without changing the logical
+ * position
+ */
+ file->pos = 0;
+ file->nbytes = 0;
}
+ else
+ {
+ /*
+ * curOffset should be at buffer boundary and buffer is the smallest
+ * I/O unit for encrypted data.
+ */
+ Assert(file->curOffset % BLCKSZ == 0);
- /*
- * Now we can set the buffer empty without changing the logical position
- */
- file->pos = 0;
- file->nbytes = 0;
+ /*
+ * The number of useful bytes needs to be written at the end of each
+ * encrypted segment of a shared file so that the other backends know
+ * how many bytes of the last buffer are useful.
+ */
+ if (file->fileset != NULL)
+ {
+ off_t useful;
+
+#ifdef USE_ENCRYPTION
+
+ /*
+ * nuseful may be increased earlier than numFiles but not later,
+ * so the corresponding entry should always exist in ->useful.
+ */
+ Assert(file->curFile < file->nuseful);
+
+ /*
+ * The number of useful bytes in the current segment file.
+ */
+ useful = file->useful[file->curFile];
+#endif
+
+ /*
+ * Have we dumped the last buffer of the segment, i.e. the one
+ * that can contain padding?
+ */
+ if (file->curOffset >= useful)
+ {
+ int bytes_extra;
+
+ /*
+ * Write the number of useful bytes in the segment.
+ */
+ bytes_extra = FileWrite(thisfile,
+ (char *) &useful,
+ sizeof(useful),
+ WAIT_EVENT_BUFFILE_WRITE);
+ if (bytes_extra != sizeof(useful))
+ return; /* failed to write */
+
+ /*
+ * Remember the file position at OS level, but do not increase
+ * curOffset. That ensures that the next buffer appended will
+ * overwrite the "useful" value just written, instead of being
+ * appended to it.
+ *
+ * XXX The additional seek backwards at OS level when the next
+ * buffer is gonna be written is hard to avoid by writing this
+ * info higher in the stack (e.g. in BufFileWrite() or
+ * BufFileFlush()). Once we've cleared the "dirty" flag, we
+ * cannot recognize easily later that the number of useful
+ * bytes might need to be updated on disk because of this
+ * buffer. Is it worth introducing another bool field and
+ * making the logic trickier?
+ */
+ file->offsets[file->curFile] += bytes_extra;
+ }
+ }
+
+ if (file->pos >= BLCKSZ)
+ {
+ Assert(file->pos == BLCKSZ);
+
+ /*
+ * curOffset points to the beginning of the next buffer, so just
+ * reset pos and nbytes.
+ */
+ file->pos = 0;
+ file->nbytes = 0;
+
+ /* See makeBufFile() */
+ if (data_encrypted)
+ MemSet(file->buffer, 0, BLCKSZ);
+ }
+ else
+ {
+ /*
+ * Move curOffset to the beginning of the just-written buffer and
+ * preserve pos.
+ */
+ file->curOffset -= BLCKSZ;
+
+ /*
+ * At least pos bytes should be written even if the first change
+ * since now appears at pos == nbytes, but in fact the whole
+ * buffer will be written regardless pos. This is the price we pay
+ * for the choosing BLCKSZ as the I/O unit for encrypted data.
+ */
+ file->nbytes = BLCKSZ;
+ }
+ }
}
/*
@@ -556,18 +966,107 @@ BufFileRead(BufFile *file, void *ptr, size_t size)
while (size > 0)
{
+ int avail;
+
if (file->pos >= file->nbytes)
{
+ /*
+ * Neither read nor write nor seek should leave pos greater than
+ * nbytes, regardless the data is encrypted or not. pos can only
+ * be greater if nbytes is zero --- this situation can be caused
+ * by BufFileSeek().
+ */
+ Assert(file->pos == file->nbytes || file->nbytes == 0);
+
+ /*
+ * The Assert() above implies that pos is a whole multiple of
+ * BLCKSZ, so curOffset has meet the same encryption-specific
+ * requirement too.
+ */
+ Assert(file->curOffset % BLCKSZ == 0 || !data_encrypted);
+
/* Try to load more data into buffer. */
- file->curOffset += file->pos;
- file->pos = 0;
- file->nbytes = 0;
- BufFileLoadBuffer(file);
- if (file->nbytes <= 0)
- break; /* no more data available */
+ if (!data_encrypted || file->pos % BLCKSZ == 0)
+ {
+ file->curOffset += file->pos;
+ file->pos = 0;
+ file->nbytes = 0;
+ BufFileLoadBuffer(file);
+ if (file->nbytes <= 0)
+ break; /* no more data available */
+ }
+ else
+ {
+ int nbytes_orig = file->nbytes;
+
+ /*
+ * Given that BLCKSZ is the I/O unit for encrypted data (see
+ * comments in BufFileLoadBuffer()), we cannot add pos to
+ * curOffset because that would make it point outside block
+ * boundary. The only thing we can do is to reload the whole
+ * buffer and see if more data is eventually there than the
+ * previous load has fetched.
+ */
+ BufFileLoadBuffer(file);
+ if (file->nbytes <= nbytes_orig)
+ break; /* no more data available */
+ }
+ }
+
+ avail = file->nbytes;
+ nthistime = avail - file->pos;
+
+ /*
+ * The buffer can contain trailing zeroes because BLCKSZ is the I/O
+ * unit for encrypted data. These are not available for reading.
+ */
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ off_t useful = file->useful[file->curFile];
+
+ /*
+ * The criterion is whether the useful data end within the
+ * currently loaded buffer.
+ */
+ if (useful < file->curOffset + BLCKSZ)
+ {
+ /*
+ * Compute the number of bytes available in the current
+ * buffer.
+ */
+ avail = useful - file->curOffset;
+ Assert(avail >= 0);
+
+ /*
+ * An empty buffer can exist, e.g. after a seek to the end of
+ * the last component file.
+ */
+ if (avail == 0)
+ break;
+
+ /*
+ * Seek beyond the current EOF, which was not followed by
+ * write, could have resulted in position outside the useful
+ * data
+ */
+ if (file->pos > avail)
+ break;
+
+ nthistime = avail - file->pos;
+ Assert(nthistime >= 0);
+
+ /*
+ * Have we reached the end of the valid data?
+ */
+ if (nthistime == 0)
+ break;
+ }
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
}
- nthistime = file->nbytes - file->pos;
if (nthistime > size)
nthistime = size;
Assert(nthistime > 0);
@@ -609,11 +1108,28 @@ BufFileWrite(BufFile *file, void *ptr, size_t size)
}
else
{
- /* Hmm, went directly from reading to writing? */
+ /*
+ * Hmm, went directly from reading to writing?
+ *
+ * As pos should be exactly BLCKSZ, there is nothing special
+ * to do about data_encrypted. Except for zeroing the buffer.
+ */
+ Assert(file->pos == BLCKSZ);
+
file->curOffset += file->pos;
file->pos = 0;
file->nbytes = 0;
+
+ /* See makeBufFile() */
+ if (data_encrypted)
+ MemSet(file->buffer, 0, BLCKSZ);
}
+
+ /*
+ * If curOffset changed above, it should still meet the assumption
+ * that buffer is the I/O unit for encrypted data.
+ */
+ Assert(file->curOffset % BLCKSZ == 0 || !data_encrypted);
}
nthistime = BLCKSZ - file->pos;
@@ -627,6 +1143,65 @@ BufFileWrite(BufFile *file, void *ptr, size_t size)
file->pos += nthistime;
if (file->nbytes < file->pos)
file->nbytes = file->pos;
+
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ off_t new_offset;
+ int fileno = file->curFile;
+
+ /*
+ * curFile does not necessarily correspond to the offset: it can
+ * still have the initial value if BufFileSeek() skipped the
+ * previous file w/o dumping anything of it. While curFile will be
+ * fixed during the next dump, we need valid fileno now.
+ */
+ if (file->curOffset >= MAX_PHYSICAL_FILESIZE)
+ {
+ /*
+ * Even BufFileSeek() should not allow curOffset to become
+ * more than MAX_PHYSICAL_FILESIZE (if caller passes higher
+ * offset, curFile gets increased instead).
+ */
+ Assert(file->curOffset == MAX_PHYSICAL_FILESIZE);
+
+ fileno++;
+ }
+
+ /*
+ * fileno can point to a segment that does not exist on disk yet.
+ */
+ ensureUsefulArraySize(file, fileno + 1);
+
+ /*
+ * Update the "useful offset" of the underlying component file if
+ * we've added any useful data.
+ */
+ new_offset = file->curOffset + file->pos;
+
+ /*
+ * Make sure the offset is relative to the correct component file.
+ * It should have been adjusted during sequential write, but if
+ * we've just used BufFileSeek() to jump to segment boundary w/o
+ * writing, the value is relative to the start of the *previous*
+ * segment.
+ */
+ if (file->curOffset % MAX_PHYSICAL_FILESIZE == 0)
+ new_offset %= MAX_PHYSICAL_FILESIZE;
+
+ /*
+ * Adjust the number of useful bytes in the file if needed. This
+ * has to happen immediately, independent from
+ * BufFileDumpBuffer(), so that BufFileRead() works correctly
+ * anytime.
+ */
+ if (new_offset > file->useful[fileno])
+ file->useful[fileno] = new_offset;
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
ptr = (void *) ((char *) ptr + nthistime);
size -= nthistime;
nwritten += nthistime;
@@ -741,9 +1316,43 @@ BufFileSeek(BufFile *file, int fileno, off_t offset, int whence)
return EOF;
/* Seek is OK! */
file->curFile = newFile;
- file->curOffset = newOffset;
- file->pos = 0;
- file->nbytes = 0;
+ if (!data_encrypted)
+ {
+ file->curOffset = newOffset;
+ file->pos = 0;
+ file->nbytes = 0;
+ }
+ else
+ {
+ /*
+ * Offset of an encrypted buffer must be a multiple of BLCKSZ.
+ */
+ file->pos = newOffset % BLCKSZ;
+ file->curOffset = newOffset - file->pos;
+
+ /*
+ * BufFileLoadBuffer() will set nbytes iff it can read something.
+ */
+ file->nbytes = 0;
+
+ /*
+ * Load and decrypt the existing part of the buffer.
+ */
+ BufFileLoadBuffer(file);
+ if (file->nbytes == 0)
+ {
+ /*
+ * The data requested is not in the file, but this is not an
+ * error.
+ */
+ return 0;
+ }
+
+ /*
+ * The whole buffer should have been loaded.
+ */
+ Assert(file->nbytes == BLCKSZ);
+ }
return 0;
}
@@ -774,6 +1383,55 @@ BufFileSeekBlock(BufFile *file, long blknum)
SEEK_SET);
}
+#ifdef USE_ENCRYPTION
+static void
+BufFileTweak(char *tweak, BufFile *file)
+{
+ off_t block;
+ int curFile = file->curFile;
+
+ /*
+ * If the file was produced by BufFileAppend(), we need the original
+ * curFile, as it was used originally for encryption.
+ */
+ if (file->segnos)
+ curFile = file->segnos[curFile];
+
+ block = curFile * BUFFILE_SEG_SIZE + file->curOffset / BLCKSZ;
+
+ /*
+ * The unused bytes should always be defined.
+ */
+ memset(tweak, 0, TWEAK_SIZE);
+ StaticAssertStmt(sizeof(block) <= TWEAK_SIZE,
+ "block number does not fit into encryption tweak");
+ memcpy(tweak, &block, sizeof(off_t));
+}
+
+/*
+ * Make sure that BufFile.useful array has the required size.
+ */
+static void
+ensureUsefulArraySize(BufFile *file, int required)
+{
+ /*
+ * Does the array already have enough space?
+ */
+ if (required <= file->nuseful)
+ return;
+
+ /*
+ * It shouldn't be possible to jump beyond the end of the last segment,
+ * i.e. skip more than 1 segment.
+ */
+ Assert(file->nuseful + 1 == required);
+
+ file->useful = (off_t *) repalloc(file->useful, required * sizeof(off_t));
+ file->useful[file->nuseful] = 0L;
+ file->nuseful++;
+}
+#endif
+
#ifdef NOT_USED
/*
* BufFileTellBlock --- block-oriented tell
@@ -803,11 +1461,33 @@ BufFileSize(BufFile *file)
{
off_t lastFileSize;
- /* Get the size of the last physical file by seeking to end. */
- lastFileSize = FileSeek(file->files[file->numFiles - 1], 0, SEEK_END);
- if (lastFileSize < 0)
- return -1;
- file->offsets[file->numFiles - 1] = lastFileSize;
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ /*
+ * "useful" should be initialized even for shared file, see
+ * BufFileOpenShared().
+ */
+ Assert(file->useful != NULL && file->nuseful >= file->numFiles);
+
+ /*
+ * The number of useful bytes in the segment is what caller is
+ * interested in.
+ */
+ lastFileSize = file->useful[file->nuseful - 1];
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+ else
+ {
+ /* Get the size of the last physical file by seeking to end. */
+ lastFileSize = FileSeek(file->files[file->numFiles - 1], 0, SEEK_END);
+ if (lastFileSize < 0)
+ return -1;
+
+ file->offsets[file->numFiles - 1] = lastFileSize;
+ }
return ((file->numFiles - 1) * (off_t) MAX_PHYSICAL_FILESIZE) +
lastFileSize;
@@ -837,6 +1517,9 @@ BufFileAppend(BufFile *target, BufFile *source)
{
long startBlock = target->numFiles * BUFFILE_SEG_SIZE;
int newNumFiles = target->numFiles + source->numFiles;
+#ifdef USE_ENCRYPTION
+ int newNUseful = target->nuseful + source->nuseful;
+#endif
int i;
Assert(target->fileset != NULL);
@@ -851,11 +1534,75 @@ BufFileAppend(BufFile *target, BufFile *source)
repalloc(target->files, sizeof(File) * newNumFiles);
target->offsets = (off_t *)
repalloc(target->offsets, sizeof(off_t) * newNumFiles);
+
for (i = target->numFiles; i < newNumFiles; i++)
{
- target->files[i] = source->files[i - target->numFiles];
- target->offsets[i] = source->offsets[i - target->numFiles];
+ int n = i - target->numFiles;
+
+ target->files[i] = source->files[n];
+ target->offsets[i] = source->offsets[n];
}
+
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ /*
+ * XXX As the typical use case is that parallel workers expose file to
+ * the leader, can we expect both target and source to have been
+ * exported, i.e. flushed? In such a case "nuseful" would have to be
+ * equal to "numFiles" for both input files and the code could get a
+ * bit simpler. It seems that at least source should be flushed, as
+ * source->readOnly is expected to be true above.
+ */
+ target->useful = (off_t *)
+ repalloc(target->useful, sizeof(off_t) * newNUseful);
+
+ for (i = target->nuseful; i < newNUseful; i++)
+ target->useful[i] = source->useful[i - target->nuseful];
+ target->nuseful = newNUseful;
+
+ /*
+ * File segments can appear at different position due to
+ * concatenation, so make sure we remember the original positions for
+ * the sake of encryption tweak.
+ */
+ if (target->segnos == NULL)
+ {
+ /*
+ * If the target does not have the array yet, allocate it for both
+ * target and source and initialize the target part.
+ */
+ target->segnos = (off_t *) palloc(newNumFiles * sizeof(off_t));
+ for (i = 0; i < target->numFiles; i++)
+ target->segnos[i] = i;
+ }
+ else
+ {
+ /*
+ * Use the existing target part and add space for the source part.
+ */
+ target->segnos = (off_t *) repalloc(target->segnos,
+ newNumFiles * sizeof(off_t));
+ }
+
+ /*
+ * The source segment number either equals to (0-based) index of the
+ * segment, or to an element of an already existing array.
+ */
+ for (i = target->numFiles; i < newNumFiles; i++)
+ {
+ off_t segno = i - target->numFiles;
+
+ if (source->segnos == NULL)
+ target->segnos[i] = segno;
+ else
+ target->segnos[i] = source->segnos[segno];
+ }
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
target->numFiles = newNumFiles;
return startBlock;
diff --git a/src/backend/storage/file/copydir.c b/src/backend/storage/file/copydir.c
index 4a0d23b11e..eb4ad205e1 100644
--- a/src/backend/storage/file/copydir.c
+++ b/src/backend/storage/file/copydir.c
@@ -23,24 +23,30 @@
#include <sys/stat.h>
#include "storage/copydir.h"
+#include "storage/encryption.h"
#include "storage/fd.h"
+#include "storage/reinit.h"
+#include "storage/smgr.h"
#include "miscadmin.h"
#include "pgstat.h"
/*
* copydir: copy a directory
*
- * If recurse is false, subdirectories are ignored. Anything that's not
- * a directory or a regular file is ignored.
+ * RelFileNode values must specify tablespace and database oids for source
+ * and target to support re-encryption if necessary. relNode value in provided
+ * structs will be clobbered.
*/
void
-copydir(char *fromdir, char *todir, bool recurse)
+copydir(char *fromdir, char *todir, RelFileNode *fromNode, RelFileNode *toNode)
{
DIR *xldir;
struct dirent *xlde;
char fromfile[MAXPGPATH * 2];
char tofile[MAXPGPATH * 2];
+ Assert(!data_encrypted || (fromNode != NULL && toNode != NULL));
+
if (MakePGDirectory(todir) != 0)
ereport(ERROR,
(errcode_for_file_access(),
@@ -67,14 +73,32 @@ copydir(char *fromdir, char *todir, bool recurse)
(errcode_for_file_access(),
errmsg("could not stat file \"%s\": %m", fromfile)));
- if (S_ISDIR(fst.st_mode))
+ if (S_ISREG(fst.st_mode))
{
- /* recurse to handle subdirectories */
- if (recurse)
- copydir(fromfile, tofile, true);
+ int oidchars;
+ ForkNumber forkNum;
+ int segment;
+
+ /*
+ * For encrypted databases we need to reencrypt files with new
+ * tweaks.
+ */
+ if (data_encrypted &&
+ parse_filename_for_nontemp_relation(xlde->d_name,
+ &oidchars, &forkNum, &segment))
+ {
+ char oidbuf[OIDCHARS + 1];
+
+ memcpy(oidbuf, xlde->d_name, oidchars);
+ oidbuf[oidchars] = '\0';
+
+ /* We scribble over the provided RelFileNodes here */
+ fromNode->relNode = toNode->relNode = atol(oidbuf);
+ copy_file(fromfile, tofile, fromNode, toNode, forkNum, forkNum, segment);
+ }
+ else
+ copy_file(fromfile, tofile, NULL, NULL, 0, 0, 0);
}
- else if (S_ISREG(fst.st_mode))
- copy_file(fromfile, tofile);
}
FreeDir(xldir);
@@ -121,17 +145,24 @@ copydir(char *fromdir, char *todir, bool recurse)
}
/*
- * copy one file
+ * copy one file. If decryption and reencryption may be needed specify
+ * relfilenodes for source and target.
*/
void
-copy_file(char *fromfile, char *tofile)
+copy_file(char *fromfile, char *tofile, RelFileNode *fromNode,
+ RelFileNode *toNode, ForkNumber fromForkNum, ForkNumber toForkNum,
+ int segment)
{
char *buffer;
int srcfd;
int dstfd;
int nbytes;
+ int bytesread;
off_t offset;
off_t flush_offset;
+#ifdef USE_ENCRYPTION
+ BlockNumber blockNum = segment * RELSEG_SIZE;
+#endif
/* Size of copy buffer (read and write requests) */
#define COPY_BUF_SIZE (8 * BLCKSZ)
@@ -186,16 +217,55 @@ copy_file(char *fromfile, char *tofile)
flush_offset = offset;
}
- pgstat_report_wait_start(WAIT_EVENT_COPY_FILE_READ);
- nbytes = read(srcfd, buffer, COPY_BUF_SIZE);
- pgstat_report_wait_end();
- if (nbytes < 0)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not read file \"%s\": %m", fromfile)));
+ /*
+ * Try to read as much as we fit in the buffer so we can deal with
+ * complete blocks if we need to reencrypt.
+ */
+ nbytes = 0;
+ while (nbytes < COPY_BUF_SIZE)
+ {
+ pgstat_report_wait_start(WAIT_EVENT_COPY_FILE_READ);
+ bytesread = read(srcfd, buffer + nbytes, COPY_BUF_SIZE - nbytes);
+ pgstat_report_wait_end();
+ if (bytesread < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m", fromfile)));
+ nbytes += bytesread;
+ if (bytesread == 0)
+ break;
+ }
if (nbytes == 0)
break;
+
+ /*
+ * If encryption is in place, only complete blocks should have been
+ * read. Thus we do not have to care whether any encryption block is
+ * affected by the previous buffer contents.
+ */
+ Assert(nbytes % BLCKSZ == 0 || (fromNode == NULL && toNode == NULL));
+
errno = 0;
+
+ /*
+ * If the database is encrypted we need to decrypt the data here and
+ * reencrypt it to adjust the tweak values of blocks.
+ */
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ if (fromNode != NULL)
+ {
+ Assert(toNode != NULL);
+ blockNum = ReencryptBlock(buffer, nbytes / BLCKSZ,
+ fromNode, toNode, fromForkNum,
+ toForkNum, blockNum);
+ }
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
pgstat_report_wait_start(WAIT_EVENT_COPY_FILE_WRITE);
if ((int) write(dstfd, buffer, nbytes) != nbytes)
{
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 8dd51f1767..b3e3a50d30 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -2163,10 +2163,6 @@ FileSeek(File file, off_t offset, int whence)
return vfdP->seekPos;
}
-/*
- * XXX not actually used but here for completeness
- */
-#ifdef NOT_USED
off_t
FileTell(File file)
{
@@ -2175,7 +2171,6 @@ FileTell(File file)
file, VfdCache[file].fileName));
return VfdCache[file].seekPos;
}
-#endif
int
FileTruncate(File file, off_t offset, uint32 wait_event_info)
diff --git a/src/backend/storage/file/reinit.c b/src/backend/storage/file/reinit.c
index 74ff6c359b..70e138503e 100644
--- a/src/backend/storage/file/reinit.c
+++ b/src/backend/storage/file/reinit.c
@@ -16,6 +16,7 @@
#include <unistd.h>
+#include "catalog/pg_tablespace.h"
#include "common/relpath.h"
#include "storage/copydir.h"
#include "storage/fd.h"
@@ -24,9 +25,9 @@
#include "utils/memutils.h"
static void ResetUnloggedRelationsInTablespaceDir(const char *tsdirname,
- int op);
+ int op, Oid spcOid);
static void ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname,
- int op);
+ int op, Oid spcOid, Oid dbOid);
typedef struct
{
@@ -68,7 +69,7 @@ ResetUnloggedRelations(int op)
/*
* First process unlogged files in pg_default ($PGDATA/base)
*/
- ResetUnloggedRelationsInTablespaceDir("base", op);
+ ResetUnloggedRelationsInTablespaceDir("base", op, DEFAULTTABLESPACE_OID);
/*
* Cycle through directories for all non-default tablespaces.
@@ -77,13 +78,16 @@ ResetUnloggedRelations(int op)
while ((spc_de = ReadDir(spc_dir, "pg_tblspc")) != NULL)
{
+ Oid spcOid;
+
if (strcmp(spc_de->d_name, ".") == 0 ||
strcmp(spc_de->d_name, "..") == 0)
continue;
snprintf(temp_path, sizeof(temp_path), "pg_tblspc/%s/%s",
spc_de->d_name, TABLESPACE_VERSION_DIRECTORY);
- ResetUnloggedRelationsInTablespaceDir(temp_path, op);
+ spcOid = atoi(spc_de->d_name);
+ ResetUnloggedRelationsInTablespaceDir(temp_path, op, spcOid);
}
FreeDir(spc_dir);
@@ -99,7 +103,8 @@ ResetUnloggedRelations(int op)
* Process one tablespace directory for ResetUnloggedRelations
*/
static void
-ResetUnloggedRelationsInTablespaceDir(const char *tsdirname, int op)
+ResetUnloggedRelationsInTablespaceDir(const char *tsdirname, int op,
+ Oid spcOid)
{
DIR *ts_dir;
struct dirent *de;
@@ -126,6 +131,8 @@ ResetUnloggedRelationsInTablespaceDir(const char *tsdirname, int op)
while ((de = ReadDir(ts_dir, tsdirname)) != NULL)
{
+ Oid dbOid;
+
/*
* We're only interested in the per-database directories, which have
* numeric names. Note that this code will also (properly) ignore "."
@@ -134,9 +141,10 @@ ResetUnloggedRelationsInTablespaceDir(const char *tsdirname, int op)
if (strspn(de->d_name, "0123456789") != strlen(de->d_name))
continue;
+ dbOid = atoi(de->d_name);
snprintf(dbspace_path, sizeof(dbspace_path), "%s/%s",
tsdirname, de->d_name);
- ResetUnloggedRelationsInDbspaceDir(dbspace_path, op);
+ ResetUnloggedRelationsInDbspaceDir(dbspace_path, op, spcOid, dbOid);
}
FreeDir(ts_dir);
@@ -146,7 +154,8 @@ ResetUnloggedRelationsInTablespaceDir(const char *tsdirname, int op)
* Process one per-dbspace directory for ResetUnloggedRelations
*/
static void
-ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op)
+ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op,
+ Oid spcOid, Oid dbOid)
{
DIR *dbspace_dir;
struct dirent *de;
@@ -187,7 +196,7 @@ ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op)
/* Skip anything that doesn't look like a relation data file. */
if (!parse_filename_for_nontemp_relation(de->d_name, &oidchars,
- &forkNum))
+ &forkNum, NULL))
continue;
/* Also skip it unless this is the init fork. */
@@ -229,7 +238,7 @@ ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op)
/* Skip anything that doesn't look like a relation data file. */
if (!parse_filename_for_nontemp_relation(de->d_name, &oidchars,
- &forkNum))
+ &forkNum, NULL))
continue;
/* We never remove the init fork. */
@@ -279,13 +288,14 @@ ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op)
{
ForkNumber forkNum;
int oidchars;
+ int segment;
char oidbuf[OIDCHARS + 1];
char srcpath[MAXPGPATH * 2];
char dstpath[MAXPGPATH];
/* Skip anything that doesn't look like a relation data file. */
if (!parse_filename_for_nontemp_relation(de->d_name, &oidchars,
- &forkNum))
+ &forkNum, &segment))
continue;
/* Also skip it unless this is the init fork. */
@@ -305,7 +315,13 @@ ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op)
/* OK, we're ready to perform the actual copy. */
elog(DEBUG2, "copying %s to %s", srcpath, dstpath);
- copy_file(srcpath, dstpath);
+ {
+ RelFileNode srcNode = {spcOid, dbOid, atol(oidbuf)};
+ RelFileNode dstNode = srcNode;
+
+ copy_file(srcpath, dstpath, &srcNode, &dstNode,
+ INIT_FORKNUM, MAIN_FORKNUM, segment);
+ }
}
FreeDir(dbspace_dir);
@@ -327,7 +343,7 @@ ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op)
/* Skip anything that doesn't look like a relation data file. */
if (!parse_filename_for_nontemp_relation(de->d_name, &oidchars,
- &forkNum))
+ &forkNum, NULL))
continue;
/* Also skip it unless this is the init fork. */
@@ -372,9 +388,10 @@ ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op)
*/
bool
parse_filename_for_nontemp_relation(const char *name, int *oidchars,
- ForkNumber *fork)
+ ForkNumber *fork, int *segment)
{
int pos;
+ int segstart = 0;
/* Look for a non-empty string of digits (that isn't too long). */
for (pos = 0; isdigit((unsigned char) name[pos]); ++pos)
@@ -401,6 +418,7 @@ parse_filename_for_nontemp_relation(const char *name, int *oidchars,
{
int segchar;
+ segstart = pos + 1;
for (segchar = 1; isdigit((unsigned char) name[pos + segchar]); ++segchar)
;
if (segchar <= 1)
@@ -411,5 +429,14 @@ parse_filename_for_nontemp_relation(const char *name, int *oidchars,
/* Now we should be at the end. */
if (name[pos] != '\0')
return false;
+
+ if (segment != NULL)
+ {
+ if (segstart == 0)
+ *segment = 0;
+ else
+ *segment = atoi(name + segstart);
+ }
+
return true;
}
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index dfbda5458f..1c3d9463d4 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -17,6 +17,7 @@
#include "access/htup_details.h"
#include "access/itup.h"
#include "access/xlog.h"
+#include "common/string.h"
#include "storage/checksum.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
@@ -81,11 +82,8 @@ bool
PageIsVerified(Page page, BlockNumber blkno)
{
PageHeader p = (PageHeader) page;
- size_t *pagebytes;
- int i;
bool checksum_failure = false;
bool header_sane = false;
- bool all_zeroes = false;
uint16 checksum = 0;
/*
@@ -118,26 +116,8 @@ PageIsVerified(Page page, BlockNumber blkno)
return true;
}
- /*
- * Check all-zeroes case. Luckily BLCKSZ is guaranteed to always be a
- * multiple of size_t - and it's much faster to compare memory using the
- * native word size.
- */
- StaticAssertStmt(BLCKSZ == (BLCKSZ / sizeof(size_t)) * sizeof(size_t),
- "BLCKSZ has to be a multiple of sizeof(size_t)");
-
- all_zeroes = true;
- pagebytes = (size_t *) page;
- for (i = 0; i < (BLCKSZ / sizeof(size_t)); i++)
- {
- if (pagebytes[i] != 0)
- {
- all_zeroes = false;
- break;
- }
- }
-
- if (all_zeroes)
+ /* Check all-zeroes case */
+ if (IsAllZero((char *) page, BLCKSZ))
return true;
/*
diff --git a/src/backend/storage/smgr/Makefile b/src/backend/storage/smgr/Makefile
index 2b95cb0df1..0601ea7894 100644
--- a/src/backend/storage/smgr/Makefile
+++ b/src/backend/storage/smgr/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/storage/smgr
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
-OBJS = md.o smgr.o smgrtype.o
+OBJS = encryption.o md.o smgr.o smgrtype.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/smgr/encryption.c b/src/backend/storage/smgr/encryption.c
new file mode 100644
index 0000000000..cef36b1f5a
--- /dev/null
+++ b/src/backend/storage/smgr/encryption.c
@@ -0,0 +1,814 @@
+/*-------------------------------------------------------------------------
+ *
+ * encryption.c
+ * This code handles encryption and decryption of data.
+ *
+ * Encryption is done by extension modules loaded by encryption_library GUC.
+ * The extension module must register itself and provide a cryptography
+ * implementation. Key setup is left to the extension module.
+ *
+ *
+ * Copyright (c) 2016, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/storage/smgr/encryption.c
+ *
+ * NOTES
+ * This file is compiled as both front-end and backend code, so it
+ * may not use ereport, server-defined static variables, etc.
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include <unistd.h>
+
+#include "common/fe_memutils.h"
+#include "common/sha2.h"
+#include "common/string.h"
+#include "catalog/pg_control.h"
+#ifndef FRONTEND
+#include "pgstat.h"
+#endif /* FRONTEND */
+#include "storage/bufpage.h"
+#include "storage/encryption.h"
+#include "storage/fd.h"
+#include "utils/memutils.h"
+#include "miscadmin.h"
+#include "fmgr.h"
+#include "port.h"
+
+#ifdef USE_ENCRYPTION
+#include <openssl/conf.h>
+#include <openssl/evp.h>
+#include <openssl/err.h>
+#endif
+
+#ifdef USE_ENCRYPTION
+unsigned char *encryption_key = NULL;
+const char *encryption_key_prefix = "encryption_key=";
+const char *encryption_pwd_prefix = "encryption_password=";
+#endif
+
+bool data_encrypted = false;
+
+#ifdef USE_ENCRYPTION
+char *encryption_key_command = NULL;
+char *encryption_buffer = NULL;
+Size encryption_buf_size = 0;
+
+static bool initialized = false;
+
+static void setup_encryption_internal(void);
+static char *run_encryption_key_command(bool *is_key_p, size_t *len_p);
+static void evp_error(void);
+
+/*
+ * Pointer to the KDF parameters.
+ *
+ * XXX Rename this and the write / read functions so they contain the
+ * 'keysetup' string?
+ */
+KDFParamsData *KDFParams = NULL;
+
+/*
+ * Initialize KDFParamsData and write it to a file.
+ *
+ * This is very similar to WriteControlFile().
+ */
+#ifndef FRONTEND
+extern void
+write_kdf_file(void)
+{
+ KDFParamsPBKDF2 *params;
+ int i,
+ fd;
+
+ StaticAssertStmt(sizeof(KDFParamsData) <= KDF_PARAMS_FILE_SIZE,
+ "kdf file is too large for atomic disk writes");
+
+ /*
+ * The initialization should not be repeated.
+ */
+ Assert(KDFParams == NULL);
+
+ KDFParams = MemoryContextAllocZero(TopMemoryContext,
+ KDF_PARAMS_FILE_SIZE);
+ KDFParams->function = KDF_OPENSSL_PKCS5_PBKDF2_HMAC_SHA;
+ params = &KDFParams->data.pbkdf2;
+ params->niter = ENCRYPTION_KDF_NITER;
+ for (i = 0; i < ENCRYPTION_KDF_SALT_LEN; i++)
+ params->salt[i] = (unsigned char) random();
+
+ /* Contents are protected with a CRC */
+ INIT_CRC32C(KDFParams->crc);
+ COMP_CRC32C(KDFParams->crc,
+ (char *) KDFParams,
+ offsetof(KDFParamsData, crc));
+ FIN_CRC32C(KDFParams->crc);
+
+ fd = BasicOpenFile(KDF_PARAMS_FILE,
+ O_RDWR | O_CREAT | O_EXCL | PG_BINARY);
+ if (fd < 0)
+ ereport(PANIC,
+ (errcode_for_file_access(),
+ errmsg("could not create key setup file \"%s\": %m",
+ KDF_PARAMS_FILE)));
+
+ pgstat_report_wait_start(WAIT_EVENT_KDF_FILE_WRITE);
+ if (write(fd, KDFParams, KDF_PARAMS_FILE_SIZE) != KDF_PARAMS_FILE_SIZE)
+ {
+ /* if write didn't set errno, assume problem is no disk space */
+ if (errno == 0)
+ errno = ENOSPC;
+ ereport(PANIC,
+ (errcode_for_file_access(),
+ errmsg("could not write to key setup file: %m")));
+ }
+ pgstat_report_wait_end();
+
+ pgstat_report_wait_start(WAIT_EVENT_KDF_FILE_SYNC);
+ if (pg_fsync(fd) != 0)
+ ereport(PANIC,
+ (errcode_for_file_access(),
+ errmsg("could not fsync key setup file: %m")));
+ pgstat_report_wait_end();
+
+ if (close(fd))
+ ereport(PANIC,
+ (errcode_for_file_access(),
+ errmsg("could not close key setup file: %m")));
+}
+
+/*
+ * Read KDFParamsData from file and store it in local memory.
+ *
+ * postmaster should call the function early enough for any other process to
+ * inherit valid pointer to the data.
+ */
+extern void
+read_kdf_file(void)
+{
+ pg_crc32c crc;
+ int fd;
+
+ KDFParams = MemoryContextAllocZero(TopMemoryContext,
+ KDF_PARAMS_FILE_SIZE);
+
+ fd = BasicOpenFile(KDF_PARAMS_FILE, O_RDONLY | PG_BINARY);
+
+ if (fd < 0)
+ {
+ ereport(PANIC,
+ (errcode_for_file_access(),
+ errmsg("could not open key setup file \"%s\": %m",
+ KDF_PARAMS_FILE)));
+ }
+
+ pgstat_report_wait_start(WAIT_EVENT_KDF_FILE_READ);
+
+ if (read(fd, KDFParams, sizeof(KDFParamsData)) != sizeof(KDFParamsData))
+ {
+ ereport(PANIC,
+ (errcode_for_file_access(),
+ errmsg("could not read from key setup file: %m")));
+ }
+
+ pgstat_report_wait_end();
+
+ close(fd);
+
+ /* Now check the CRC. */
+ INIT_CRC32C(crc);
+ COMP_CRC32C(crc,
+ (char *) KDFParams,
+ offsetof(KDFParamsData, crc));
+ FIN_CRC32C(crc);
+
+ if (!EQ_CRC32C(crc, KDFParams->crc))
+ ereport(FATAL,
+ (errmsg("incorrect checksum in key setup file")));
+
+ if (KDFParams->function != KDF_OPENSSL_PKCS5_PBKDF2_HMAC_SHA)
+ ereport(FATAL,
+ (errmsg("unsupported KDF function %d", KDFParams->function)));
+}
+#endif /* FRONTEND */
+#endif /* USE_ENCRYPTION */
+
+/*
+ * Encrypts a fixed value into *buf to verify that encryption key is correct.
+ * Caller provided buf needs to be able to hold at least ENCRYPTION_SAMPLE_SIZE
+ * bytes.
+ */
+void
+sample_encryption(char *buf)
+{
+#ifdef USE_ENCRYPTION
+ char tweak[TWEAK_SIZE];
+ int i;
+
+ for (i = 0; i < TWEAK_SIZE; i++)
+ tweak[i] = i;
+
+ encrypt_block("postgresqlcrypt", buf, ENCRYPTION_SAMPLE_SIZE, tweak);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+}
+
+/*
+ * Encrypts one block of data with a specified tweak value. May only be called
+ * when encryption_enabled is true.
+ *
+ * Input and output buffer may point to the same location.
+ *
+ * "size" must be a (non-zero) multiple of ENCRYPTION_BLOCK.
+ *
+ * "tweak" value must be TWEAK_SIZE bytes long.
+ *
+ * All-zero blocks are not encrypted to correctly handle relation extension,
+ * and also to simplify handling of holes created by seek past EOF and
+ * consequent write (see buffile.c).
+ */
+void
+encrypt_block(const char *input, char *output, Size size, const char *tweak)
+{
+#ifdef USE_ENCRYPTION
+ Assert(size >= ENCRYPTION_BLOCK && size % ENCRYPTION_BLOCK == 0);
+ Assert(initialized);
+
+ /*
+ * The EVP API does not seem to expect the output buffer to be equal to
+ * the input. Ensure that we pass separate pointers.
+ */
+ if (input == output)
+ {
+ if (size > encryption_buf_size)
+ enlarge_encryption_buffer(size);
+
+ memcpy(encryption_buffer, input, size);
+ input = encryption_buffer;
+ }
+
+ if (IsAllZero(input, size))
+ memset(output, 0, size);
+ else
+ {
+ int out_size;
+ EVP_CIPHER_CTX *ctx;
+
+ if ((ctx = EVP_CIPHER_CTX_new()) == NULL)
+ evp_error();
+
+ if (EVP_EncryptInit_ex(ctx, EVP_aes_256_xts(), NULL, encryption_key,
+ (unsigned char *) tweak) != 1)
+ evp_error();
+
+ /*
+ * No padding is needed, the input block size should already be a
+ * multiple of ENCRYPTION_BLOCK_OPENSSL.
+ */
+ EVP_CIPHER_CTX_set_padding(ctx, 0);
+
+ Assert(EVP_CIPHER_CTX_block_size(ctx) == ENCRYPTION_BLOCK_OPENSSL);
+ Assert(EVP_CIPHER_CTX_iv_length(ctx) == TWEAK_SIZE);
+ Assert(EVP_CIPHER_CTX_key_length(ctx) == ENCRYPTION_KEY_LENGTH);
+
+ /*
+ * Do the actual encryption. As the padding is disabled,
+ * EVP_EncryptFinal_ex() won't be needed.
+ */
+ if (EVP_EncryptUpdate(ctx, (unsigned char *) output, &out_size,
+ (unsigned char *) input, size) != 1)
+ evp_error();
+
+ /*
+ * The input size is a multiple of ENCRYPTION_BLOCK_OPENSSL, so the
+ * output of AES-XTS should meet this condition.
+ */
+ Assert(out_size == size);
+
+ if (EVP_CIPHER_CTX_cleanup(ctx) != 1)
+ evp_error();
+ }
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+}
+
+/*
+ * Decrypts one block of data with a specified tweak value. May only be called
+ * when encryption_enabled is true.
+ *
+ * Input and output buffer may point to the same location.
+ *
+ * "size" must be a (non-zero) multiple of ENCRYPTION_BLOCK.
+ *
+ * "tweak" value must be TWEAK_SIZE bytes long.
+ *
+ * All-zero blocks are not decrypted to correctly handle relation extension,
+ * and also to simplify handling of holes created by seek past EOF and
+ * consequent write (see buffile.c).
+ */
+void
+decrypt_block(const char *input, char *output, Size size, const char *tweak)
+{
+#ifdef USE_ENCRYPTION
+ Assert(size >= ENCRYPTION_BLOCK && size % ENCRYPTION_BLOCK == 0);
+ Assert(initialized);
+
+ /*
+ * The EVP API does not seem to expect the output buffer to be equal to
+ * the input. Ensure that we pass separate pointers.
+ */
+ if (input == output)
+ {
+ if (size > encryption_buf_size)
+ enlarge_encryption_buffer(size);
+
+ memcpy(encryption_buffer, input, size);
+ input = encryption_buffer;
+ }
+
+ if (IsAllZero(input, size))
+ memset(output, 0, size);
+ else
+ {
+ int out_size;
+ EVP_CIPHER_CTX *ctx;
+
+ if ((ctx = EVP_CIPHER_CTX_new()) == NULL)
+ evp_error();
+
+ if (EVP_DecryptInit_ex(ctx, EVP_aes_256_xts(), NULL, encryption_key,
+ (unsigned char *) tweak) != 1)
+ evp_error();
+
+ /* The same considerations apply below as those in encrypt_block(). */
+ EVP_CIPHER_CTX_set_padding(ctx, 0);
+ Assert(EVP_CIPHER_CTX_block_size(ctx) == ENCRYPTION_BLOCK_OPENSSL);
+ Assert(EVP_CIPHER_CTX_iv_length(ctx) == TWEAK_SIZE);
+ Assert(EVP_CIPHER_CTX_key_length(ctx) == ENCRYPTION_KEY_LENGTH);
+
+ if (EVP_DecryptUpdate(ctx, (unsigned char *) output, &out_size,
+ (unsigned char *) input, size) != 1)
+ evp_error();
+
+ Assert(out_size == size);
+
+ if (EVP_CIPHER_CTX_cleanup(ctx) != 1)
+ evp_error();
+ }
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+}
+
+/*
+ * Report an error in an universal way so that caller does not have to care
+ * whether it executes in backend or front-end.
+ */
+void
+encryption_error(bool fatal, char *message)
+{
+#ifndef FRONTEND
+ elog(fatal ? FATAL : INFO, "%s", message);
+#else
+ fprintf(stderr, "%s\n", message);
+ if (fatal)
+ exit(EXIT_FAILURE);
+#endif
+}
+
+/*
+ * Initialize encryption subsystem for use. Must be called before any
+ * encryptable data is read from or written to data directory.
+ */
+void
+setup_encryption(bool bootstrap)
+{
+#ifdef USE_ENCRYPTION
+ char *credentials;
+ bool is_key;
+ size_t len;
+
+ credentials = run_encryption_key_command(&is_key, &len);
+
+ /*
+ * Setup KDF if we need to derive the key from a password.
+ *
+ * Front-ends always need the key because some of them are not aware of
+ * the data directory and thus they'd need one more command line option to
+ * find the key setup file.
+ */
+#ifndef FRONTEND
+ if (!is_key)
+ {
+ if (bootstrap)
+ {
+ write_kdf_file();
+ }
+ else
+ read_kdf_file();
+ }
+#endif /* FRONTEND */
+
+ setup_encryption_key(credentials, is_key, len);
+ pfree(credentials);
+ setup_encryption_internal();
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+}
+
+#ifdef USE_ENCRYPTION
+static void
+setup_encryption_internal(void)
+{
+ /*
+ * Setup OpenSSL.
+ *
+ * None of these functions should return a value or raise error.
+ */
+ ERR_load_crypto_strings();
+ OpenSSL_add_all_algorithms();
+ OPENSSL_config(NULL);
+
+ /*
+ * It makes no sense to initialize the encryption multiple times.
+ */
+ Assert(!initialized);
+
+ initialized = true;
+}
+#endif /* USE_ENCRYPTION */
+
+/*
+ * If credentials is a key, just copy it to encryption_key. If it's a
+ * password, derive the key from it.
+ */
+void
+setup_encryption_key(char *credentials, bool is_key, size_t len)
+{
+#ifdef USE_ENCRYPTION
+ Assert(credentials != NULL);
+ Assert(encryption_key == NULL);
+
+ /*
+ * Although the function should be called in the TopMemoryContext, we
+ * should be pretty sure that the key does not become garbage due to
+ * pfree(). The cluster should crash in such a case, but if it did not
+ * happen immediately, some data could be encrypted using an invalid key
+ * and therefore become lost.
+ */
+#ifndef FRONTEND
+ encryption_key = (unsigned char *) MemoryContextAllocZero(TopMemoryContext,
+ ENCRYPTION_KEY_LENGTH);
+#else
+ encryption_key = (unsigned char *) palloc(ENCRYPTION_KEY_LENGTH);
+#endif /* FRONTEND */
+ if (is_key)
+ {
+ Assert(len = ENCRYPTION_KEY_LENGTH);
+ memcpy(encryption_key, credentials, len);
+ }
+ else
+ {
+ KDFParamsPBKDF2 *params;
+ int rc;
+
+ /*
+ * The file contains password so we need the KDF parameters to turn it
+ * to key.
+ */
+ if (KDFParams == NULL)
+ {
+#ifndef FRONTEND
+ ereport(FATAL,
+ (errmsg("this instance does not accept encryption password"),
+ errdetail("Encryption key was probably used to initialize the instance.")));
+#else
+ encryption_error(true,
+ "this instance does not accept encryption password.\n"
+ "Encryption key was probably used to initialize the instance.\n");
+#endif /* FRONTEND */
+ }
+
+ /*
+ * Turn the password into the encryption key.
+ */
+ params = &KDFParams->data.pbkdf2;
+ rc = PKCS5_PBKDF2_HMAC(credentials,
+ len,
+ params->salt,
+ ENCRYPTION_KDF_SALT_LEN,
+ params->niter,
+ EVP_sha(),
+ ENCRYPTION_KEY_LENGTH,
+ encryption_key);
+
+ if (rc != 1)
+ {
+#ifndef FRONTEND
+ ereport(FATAL,
+ (errmsg("failed to derive key from password")));
+#else
+ encryption_error(true, "failed to derive key from password");
+#endif /* FRONTEND */
+ }
+ }
+
+#else /* USE_ENCRYPTION */
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+}
+
+#ifdef USE_ENCRYPTION
+void
+enlarge_encryption_buffer(Size new_size)
+{
+ Assert(new_size > 0);
+
+ /*
+ * Shrinkage is not the use case for this routine.
+ */
+ if (new_size <= encryption_buf_size)
+ return;
+
+ /*
+ * Allocate a new chunk if nothing is there yet, else reallocate the
+ * existing one.
+ */
+ if (encryption_buf_size == 0)
+#ifndef FRONTEND
+ encryption_buffer = (char *) MemoryContextAlloc(TopMemoryContext,
+ new_size);
+#else
+ encryption_buffer = (char *) palloc(new_size);
+#endif /* FRONTEND */
+ else
+ encryption_buffer = (char *) repalloc(encryption_buffer, new_size);
+ encryption_buf_size = new_size;
+}
+
+/*
+ * Run the command stored in encryption_key_command and return the key or
+ * password.
+ *
+ * *is_key_p receives true if the command returns key, false if it's
+ * password. *len_p receives length of the data.
+ */
+static char *
+run_encryption_key_command(bool *is_key_p, size_t *len_p)
+{
+ FILE *fp;
+ char *buf,
+ *result;
+ bool is_key;
+ size_t key_pref_len,
+ pwd_pref_len,
+ read_len,
+ bytes_read;
+ size_t buf_size,
+ result_size;
+ int i;
+
+ if (encryption_key_command == NULL ||
+ !strlen(encryption_key_command))
+ {
+ /*
+ * encryption_key_command should have been set by initdb. It's weird
+ * if it was not, but there's no better recommendation we can give the
+ * user.
+ */
+#ifndef FRONTEND
+ ereport(FATAL,
+ (errmsg("encryption key not provided"),
+ errdetail("The database cluster was initialized with encryption"
+ " but the server was started without an encryption key."),
+ errhint("Set the encryption_key_command configuration variable.")));
+#else /* FRONTEND */
+ encryption_error(true,
+ "The database cluster was initialized with encryption"
+ " but the server was started without an encryption key. "
+ "Set the encryption_key_command configuration variable.\n");
+#endif /* FRONTEND */
+ }
+
+ encryption_error(false,
+ psprintf("Executing \"%s\" to set up encryption key",
+ encryption_key_command));
+
+ fp = popen(encryption_key_command, "r");
+ if (fp == NULL)
+ encryption_error(true,
+ psprintf("Failed to execute encryption_key_command \"%s\"",
+ encryption_key_command));
+
+ /*
+ * Check which prefix the file starts with.
+ *
+ * The prefixes probably won't change after the release but they might
+ * change during development. The reading logic should be generic so that
+ * change of prefix length requires no additional coding.
+ */
+ key_pref_len = strlen(encryption_key_prefix);
+ pwd_pref_len = strlen(encryption_pwd_prefix);
+
+ /*
+ * The buffer must accommodate either prefix.
+ */
+ buf_size = Max(key_pref_len, pwd_pref_len);
+ buf = (char *) palloc(buf_size);
+
+ /*
+ * Read as few bytes as necessary so that we don't have to move back in
+ * the buffer if the first comparison does not match.
+ */
+ read_len = Min(key_pref_len, pwd_pref_len);
+
+ if (fread(buf, 1, read_len, fp) != read_len)
+ encryption_error(true, "Not enough data received from encryption_key_command");
+
+ if (read_len == key_pref_len &&
+ strncmp(buf, encryption_key_prefix, key_pref_len) == 0)
+ is_key = true;
+ else if (read_len == pwd_pref_len &&
+ strncmp(buf, encryption_pwd_prefix, pwd_pref_len) == 0)
+ is_key = false;
+ else if (buf_size > read_len)
+ {
+ size_t len_diff;
+
+ /*
+ * Read enough data so that one of the prefixes must match.
+ */
+ len_diff = buf_size - read_len;
+ if (fread(buf + read_len, 1, len_diff, fp) != len_diff)
+ encryption_error(true,
+ "Not enough data received from encryption_key_command");
+ read_len += len_diff;
+
+ /*
+ * Try to match the prefixes again.
+ */
+ if (read_len == key_pref_len &&
+ strncmp(buf, encryption_key_prefix, key_pref_len) == 0)
+ is_key = true;
+ else if (read_len == pwd_pref_len &&
+ strncmp(buf, encryption_pwd_prefix, pwd_pref_len) == 0)
+ is_key = false;
+ else
+ encryption_error(true,
+ "Unknown data received from encryption_key_command");
+ }
+
+ *is_key_p = is_key;
+
+ /*
+ * Read the actual credentials.
+ */
+ read_len = is_key ? ENCRYPTION_KEY_LENGTH * 2 : ENCRYPTION_PWD_MAX_LENGTH;
+
+ /*
+ * One extra character can be read and this should be nothing but line
+ * delimiter, see below.
+ */
+ read_len++;
+
+ if (read_len > buf_size)
+ {
+ buf = (char *) repalloc(buf, read_len);
+ buf_size = read_len;
+ }
+
+ bytes_read = fread(buf, 1, read_len, fp);
+
+ if ((is_key && bytes_read < read_len) ||
+ (!is_key && bytes_read < (ENCRYPTION_PWD_MIN_LENGTH + 1)))
+ {
+ if (feof(fp))
+ encryption_error(true,
+ "Not enough data provided by encryption_key_command");
+ else
+ encryption_error(true,
+ psprintf("encryption_key_command returned error code %d",
+ ferror(fp)));
+ }
+ Assert(bytes_read > 0);
+
+ if (buf[bytes_read - 1] != '\n')
+ encryption_error(true, "Encryption key is too long.");
+
+ /*
+ * The credentials must not contain line delimiter.
+ *
+ * If that was allowed, the line delimiter would have to terminate
+ * password even if more characters followed, however those extra
+ * characters would be useless. Since distinguishing of the special (still
+ * legal) case "password\n<EOF>" would make the checks less simple and
+ * since the delimiter itself is useless in the case of a key (because the
+ * key has constant length), let's prohibit the line delimiters anywhere
+ * except for the last position.
+ */
+ for (i = 0; i < bytes_read; i++)
+ if (buf[i] == '\n' && i < (bytes_read - 1))
+ encryption_error(true,
+ "Neither password nor key may contain line break.");
+
+ /*
+ * For a key the result size is different from the amount of data read.
+ */
+ result_size = is_key ? ENCRYPTION_KEY_LENGTH : ENCRYPTION_PWD_MAX_LENGTH;
+
+ result = (char *) palloc(result_size);
+
+ if (is_key)
+ {
+ int i;
+
+ for (i = 0; i < ENCRYPTION_KEY_LENGTH; i++)
+ {
+ if (sscanf(buf + 2 * i, "%2hhx", result + i) == 0)
+ encryption_error(true,
+ psprintf("Invalid character in encryption key at position %d",
+ 2 * i));
+ }
+ *len_p = ENCRYPTION_KEY_LENGTH;
+ }
+ else
+ {
+ /*
+ * Ignore the line delimiter.
+ */
+ *len_p = bytes_read - 1;
+ memcpy(result, buf, *len_p);
+
+ }
+
+ /*
+ * No extra data is allowed.
+ */
+ if (fread(buf, 1, 1, fp) > 0)
+ encryption_error(true,
+ "Credentials are followed by useless data");
+
+
+ pfree(buf);
+ pclose(fp);
+
+ return result;
+}
+#endif /* USE_ENCRYPTION */
+
+/*
+ * Error callback for openssl.
+ */
+#ifdef USE_ENCRYPTION
+static void
+evp_error(void)
+{
+ ERR_print_errors_fp(stderr);
+#ifndef FRONTEND
+
+ /*
+ * FATAL is the appropriate level because backend can hardly fix anything
+ * if encryption / decryption has failed.
+ *
+ * XXX Do we yet need EVP_CIPHER_CTX_cleanup() here?
+ */
+ elog(FATAL, "OpenSSL encountered error during encryption or decryption.");
+#else
+ fprintf(stderr,
+ "OpenSSL encountered error during encryption or decryption.");
+ exit(EXIT_FAILURE);
+#endif /* FRONTEND */
+}
+#endif /* USE_ENCRYPTION */
+
+/*
+ * Xlog is encrypted page at a time. Each xlog page gets a unique tweak via
+ * segment and offset. Unfortunately we can't include timeline because
+ * exitArchiveRecovery() can copy part of the last segment of the old timeline
+ * into the first segment of the new timeline.
+ *
+ * TODO Consider teaching exitArchiveRecovery() to decrypt the copied pages
+ * and encrypt them using a tweak that mentions the new timeline.
+ *
+ * The function is located here rather than some of the xlog*.c modules so
+ * that front-end applications can easily use it too.
+ */
+void
+XLogEncryptionTweak(char *tweak, XLogSegNo segment, uint32 offset)
+{
+#ifdef USE_ENCRYPTION
+ memset(tweak, 0, TWEAK_SIZE);
+ memcpy(tweak, &segment, sizeof(XLogSegNo));
+ memcpy(tweak + sizeof(XLogSegNo), &offset, sizeof(offset));
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+}
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index 2ec103e604..f831a52e18 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -32,6 +32,7 @@
#include "postmaster/bgwriter.h"
#include "storage/fd.h"
#include "storage/bufmgr.h"
+#include "storage/encryption.h"
#include "storage/relfilenode.h"
#include "storage/smgr.h"
#include "utils/hsearch.h"
@@ -112,7 +113,14 @@ typedef struct _MdfdVec
} MdfdVec;
static MemoryContext MdCxt; /* context for all MdfdVec objects */
-
+#ifdef USE_ENCRYPTION
+/*
+ * encryption_buffer from encryption.h is not used here because of the special
+ * memory context.
+ */
+static char *md_encryption_buffer;
+static char *md_encryption_tweak;
+#endif
/*
* In some contexts (currently, standalone backends and the checkpointer)
@@ -197,6 +205,11 @@ static MdfdVec *_mdfd_getseg(SMgrRelation reln, ForkNumber forkno,
static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum,
MdfdVec *seg);
+#ifdef USE_ENCRYPTION
+static void mdtweak(char *tweak, RelFileNode *relnode, ForkNumber forknum, BlockNumber blocknum);
+static void mdencrypt(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, char *buffer);
+static void mddecrypt(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, char *buffer);
+#endif
/*
* mdinit() -- Initialize private state for magnetic disk storage manager.
@@ -241,6 +254,11 @@ mdinit(void)
HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
pendingUnlinks = NIL;
}
+
+#ifdef USE_ENCRYPTION
+ md_encryption_buffer = MemoryContextAllocZero(MdCxt, BLCKSZ);
+ md_encryption_tweak = MemoryContextAllocZero(MdCxt, TWEAK_SIZE);
+#endif
}
/*
@@ -536,6 +554,16 @@ mdextend(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
errmsg("could not seek to block %u in file \"%s\": %m",
blocknum, FilePathName(v->mdfd_vfd))));
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ mdencrypt(reln, forknum, blocknum, buffer);
+ buffer = md_encryption_buffer;
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
if ((nbytes = FileWrite(v->mdfd_vfd, buffer, BLCKSZ, WAIT_EVENT_DATA_FILE_EXTEND)) != BLCKSZ)
{
if (nbytes < 0)
@@ -733,6 +761,7 @@ mdread(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
off_t seekpos;
int nbytes;
MdfdVec *v;
+ char *buffer_read = buffer;
TRACE_POSTGRESQL_SMGR_MD_READ_START(forknum, blocknum,
reln->smgr_rnode.node.spcNode,
@@ -753,7 +782,15 @@ mdread(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
errmsg("could not seek to block %u in file \"%s\": %m",
blocknum, FilePathName(v->mdfd_vfd))));
- nbytes = FileRead(v->mdfd_vfd, buffer, BLCKSZ, WAIT_EVENT_DATA_FILE_READ);
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ buffer_read = md_encryption_buffer;
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+ nbytes = FileRead(v->mdfd_vfd, buffer_read, BLCKSZ, WAIT_EVENT_DATA_FILE_READ);
TRACE_POSTGRESQL_SMGR_MD_READ_DONE(forknum, blocknum,
reln->smgr_rnode.node.spcNode,
@@ -788,6 +825,14 @@ mdread(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
blocknum, FilePathName(v->mdfd_vfd),
nbytes, BLCKSZ)));
}
+ else if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ mddecrypt(reln, forknum, blocknum, buffer);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
}
/*
@@ -829,6 +874,15 @@ mdwrite(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
errmsg("could not seek to block %u in file \"%s\": %m",
blocknum, FilePathName(v->mdfd_vfd))));
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ mdencrypt(reln, forknum, blocknum, buffer);
+ buffer = md_encryption_buffer;
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
nbytes = FileWrite(v->mdfd_vfd, buffer, BLCKSZ, WAIT_EVENT_DATA_FILE_WRITE);
TRACE_POSTGRESQL_SMGR_MD_WRITE_DONE(forknum, blocknum,
@@ -1950,3 +2004,58 @@ _mdnblocks(SMgrRelation reln, ForkNumber forknum, MdfdVec *seg)
/* note that this calculation will ignore any partial block at EOF */
return (BlockNumber) (len / BLCKSZ);
}
+
+#ifdef USE_ENCRYPTION
+/*
+ * md files are encrypted block at a time. Tweak will alias higher numbered
+ * forks for huge tables.
+ */
+static void
+mdtweak(char *tweak, RelFileNode *relnode, ForkNumber forknum, BlockNumber blocknum)
+{
+ uint32 fork_and_block = (forknum << 24) ^ blocknum;
+
+ memcpy(tweak, relnode, sizeof(RelFileNode));
+ memcpy(tweak + sizeof(RelFileNode), &fork_and_block, 4);
+}
+
+static void
+mdencrypt(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, char *buffer)
+{
+ mdtweak(md_encryption_tweak, &(reln->smgr_rnode.node), forknum, blocknum);
+ encrypt_block(buffer, md_encryption_buffer, BLCKSZ, md_encryption_tweak);
+}
+
+static void
+mddecrypt(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, char *dest)
+{
+ mdtweak(md_encryption_tweak, &(reln->smgr_rnode.node), forknum, blocknum);
+ decrypt_block(md_encryption_buffer, dest, BLCKSZ, md_encryption_tweak);
+}
+
+/*
+ * Copying relations between tablespaces/databases means that the tweak values
+ * of each block will change. This function transcodes a series of blocks with
+ * new tweak values. Returns the new block number for convenience.
+ */
+BlockNumber
+ReencryptBlock(char *buffer, int blocks,
+ RelFileNode *srcNode, RelFileNode *dstNode,
+ ForkNumber srcForkNum, ForkNumber dstForkNum,
+ BlockNumber blockNum)
+{
+ char *cur;
+ char srcTweak[TWEAK_SIZE];
+ char dstTweak[TWEAK_SIZE];
+
+ for (cur = buffer; cur < buffer + blocks * BLCKSZ; cur += BLCKSZ)
+ {
+ mdtweak(srcTweak, srcNode, srcForkNum, blockNum);
+ mdtweak(dstTweak, dstNode, dstForkNum, blockNum);
+ decrypt_block(cur, cur, BLCKSZ, srcTweak);
+ encrypt_block(cur, cur, BLCKSZ, dstTweak);
+ blockNum++;
+ }
+ return blockNum;
+}
+#endif /* USE_ENCRYPTION */
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index f4133953be..d946729d17 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -63,6 +63,7 @@
#include "replication/walsender.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/encryption.h"
#include "storage/ipc.h"
#include "storage/proc.h"
#include "storage/procsignal.h"
@@ -3401,7 +3402,7 @@ process_postgres_switches(int argc, char *argv[], GucContext ctx,
* postmaster/postmaster.c (the option sets should not conflict) and with
* the common help() function in main/main.c.
*/
- while ((flag = getopt(argc, argv, "B:bc:C:D:d:EeFf:h:ijk:lN:nOo:Pp:r:S:sTt:v:W:-:")) != -1)
+ while ((flag = getopt(argc, argv, "B:bc:C:D:d:EeFf:h:ijK:k:lN:nOo:Pp:r:S:sTt:v:W:-:")) != -1)
{
switch (flag)
{
@@ -3459,6 +3460,12 @@ process_postgres_switches(int argc, char *argv[], GucContext ctx,
UseSemiNewlineNewline = true;
break;
+#ifdef USE_OPENSSL
+ case 'K':
+ encryption_key_command = strdup(optarg);
+ break;
+#endif /* USE_OPENSSL */
+
case 'k':
SetConfigOption("unix_socket_directories", optarg, ctx, gucsource);
break;
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 03b28c3604..1c878baf88 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -1510,6 +1510,7 @@ ValidatePgVersion(const char *path)
* GUC variables: lists of library names to be preloaded at postmaster
* start and at backend start
*/
+char *encryption_library_string = NULL;
char *session_preload_libraries_string = NULL;
char *shared_preload_libraries_string = NULL;
char *local_preload_libraries_string = NULL;
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 859ef931e7..a9d86c6ea2 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -69,6 +69,7 @@
#include "replication/walsender.h"
#include "storage/bufmgr.h"
#include "storage/dsm_impl.h"
+#include "storage/encryption.h"
#include "storage/standby.h"
#include "storage/fd.h"
#include "storage/large_object.h"
@@ -193,6 +194,9 @@ static bool check_cluster_name(char **newval, void **extra, GucSource source);
static const char *show_unix_socket_permissions(void);
static const char *show_log_file_mode(void);
static const char *show_data_directory_mode(void);
+#ifdef USE_OPENSSL
+static const char *show_encryption_key_command(void);
+#endif /* USE_OPENSSL */
/* Private functions in guc-file.l that need to be called from guc.c */
static ConfigVariable *ProcessConfigFileInternal(GucContext context,
@@ -1717,6 +1721,17 @@ static struct config_bool ConfigureNamesBool[] =
},
{
+ {"data_encryption", PGC_INTERNAL, PRESET_OPTIONS,
+ gettext_noop("Shows whether data encryption is turned on for this cluster."),
+ NULL,
+ GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE
+ },
+ &data_encrypted,
+ false,
+ NULL, NULL, NULL
+ },
+
+ {
{"syslog_sequence_numbers", PGC_SIGHUP, LOGGING_WHERE,
gettext_noop("Add sequence number to syslog messages to avoid duplicate suppression."),
NULL
@@ -3882,6 +3897,19 @@ static struct config_string ConfigureNamesString[] =
NULL, NULL, NULL
},
+#ifdef USE_OPENSSL
+ {
+ {"encryption_key_command", PGC_POSTMASTER, 0,
+ gettext_noop("Sets the shell command that will be called to fetch database encryption key."),
+ NULL,
+ GUC_IS_NAME | GUC_NOT_IN_SAMPLE
+ },
+ &encryption_key_command,
+ NULL,
+ NULL, NULL, show_encryption_key_command
+ },
+#endif /* USE_OPENSSL */
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, NULL, NULL, NULL, NULL
@@ -10784,4 +10812,15 @@ show_data_directory_mode(void)
return buf;
}
+#ifdef USE_OPENSSL
+static const char *
+show_encryption_key_command(void)
+{
+ if (encryption_key_command)
+ return encryption_key_command;
+ else
+ return "(disabled)";
+}
+#endif /* USE_OPENSSL */
+
#include "guc-file.c"
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 9e39baf466..50a22b15a9 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -88,6 +88,7 @@
#authentication_timeout = 1min # 1s-600s
#password_encryption = md5 # md5 or scram-sha-256
#db_user_namespace = off
+#encryption_key_command = ''
# GSSAPI using Kerberos
#krb_server_keyfile = ''
diff --git a/src/bin/Makefile b/src/bin/Makefile
index 8c11060a2f..431c406a65 100644
--- a/src/bin/Makefile
+++ b/src/bin/Makefile
@@ -21,6 +21,7 @@ SUBDIRS = \
pg_controldata \
pg_ctl \
pg_dump \
+ pg_keysetup \
pg_resetwal \
pg_rewind \
pg_test_fsync \
diff --git a/src/bin/initdb/Makefile b/src/bin/initdb/Makefile
index 8c23941930..ea7af9144f 100644
--- a/src/bin/initdb/Makefile
+++ b/src/bin/initdb/Makefile
@@ -26,7 +26,7 @@ ifneq (,$(with_system_tzdata))
override CPPFLAGS += '-DSYSTEMTZDIR="$(with_system_tzdata)"'
endif
-OBJS= initdb.o findtimezone.o localtime.o encnames.o $(WIN32RES)
+OBJS= initdb.o findtimezone.o localtime.o encnames.o encryption.o $(WIN32RES)
all: initdb
@@ -45,6 +45,9 @@ encnames.c: % : $(top_srcdir)/src/backend/utils/mb/%
localtime.c: % : $(top_srcdir)/src/timezone/%
rm -f $@ && $(LN_S) $< .
+encryption.c: % : $(top_srcdir)/src/backend/storage/smgr/%
+ rm -f $@ && $(LN_S) $< .
+
install: all installdirs
$(INSTALL_PROGRAM) initdb$(X) '$(DESTDIR)$(bindir)/initdb$(X)'
@@ -55,7 +58,7 @@ uninstall:
rm -f '$(DESTDIR)$(bindir)/initdb$(X)'
clean distclean maintainer-clean:
- rm -f initdb$(X) $(OBJS) encnames.c localtime.c
+ rm -f initdb$(X) $(OBJS) encnames.c localtime.c encryption.c
rm -rf tmp_check
# ensure that changes in datadir propagate into object file
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index ae22e7d9fb..e828f27bc3 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -68,6 +68,9 @@
#include "common/restricted_token.h"
#include "common/username.h"
#include "fe_utils/string_utils.h"
+#include "storage/encryption.h"
+#include "lib/ilist.h"
+#include "mb/pg_wchar.h"
#include "getaddrinfo.h"
#include "getopt_long.h"
#include "mb/pg_wchar.h"
@@ -77,6 +80,12 @@
/* Ideally this would be in a .h file, but it hardly seems worth the trouble */
extern const char *select_default_timezone(const char *share_path);
+typedef struct
+{
+ dlist_node list_node;
+ char *value;
+} extra_option;
+
static const char *const auth_methods_host[] = {
"trust", "reject", "scram-sha-256", "md5", "password", "ident", "radius",
#ifdef ENABLE_GSS
@@ -141,9 +150,11 @@ static bool do_sync = true;
static bool sync_only = false;
static bool show_setting = false;
static bool data_checksums = false;
+static char *encr_key_cmd_str = NULL;
static char *xlog_dir = NULL;
static char *str_wal_segment_size_mb = NULL;
static int wal_segment_size_mb;
+static dlist_head extra_options = DLIST_STATIC_INIT(extra_options);
/* internal vars */
@@ -529,6 +540,7 @@ readfile(const char *path)
fclose(infile);
free(buffer);
+
result[n] = NULL;
return result;
@@ -1070,6 +1082,39 @@ pretty_wal_size(int segment_count)
return result;
}
+static void
+append_extra_options(char ***conflines)
+{
+ dlist_iter iter;
+ int n_extra = 0;
+ int n_current = 0;
+ int i = 0;
+ char **new_conflines;
+
+ dlist_foreach(iter, &extra_options)
+ {
+ n_extra++;
+ }
+ while ((*conflines)[i++] != NULL)
+ n_current++;
+
+ new_conflines = (char **) pg_malloc((n_current + n_extra + 1) * sizeof(char *));
+ for (i = 0; i < n_current; i++)
+ new_conflines[i] = (*conflines)[i];
+
+ dlist_foreach(iter, &extra_options)
+ {
+ extra_option *opt = dlist_container(extra_option, list_node, iter.cur);
+
+ new_conflines[i++] = opt->value;
+ }
+
+ new_conflines[i] = NULL;
+ pg_free(*conflines);
+ *conflines = new_conflines;
+}
+
+
/*
* set up all the config files
*/
@@ -1089,6 +1134,8 @@ setup_config(void)
conflines = readfile(conf_file);
+ append_extra_options(&conflines);
+
snprintf(repltok, sizeof(repltok), "max_connections = %d", n_connections);
conflines = replace_token(conflines, "#max_connections = 100", repltok);
@@ -1231,6 +1278,17 @@ setup_config(void)
"log_file_mode = 0640");
}
+#ifdef USE_OPENSSL
+ if (encryption_key_command)
+ {
+ snprintf(repltok, sizeof(repltok), "encryption_key_command = '%s'",
+ encryption_key_command);
+ conflines = replace_token(conflines,
+ "#encryption_key_command = ''", repltok);
+ }
+#endif /* USE_OPENSSL */
+
+
snprintf(path, sizeof(path), "%s/postgresql.conf", pg_data);
writefile(path, conflines);
@@ -1448,11 +1506,29 @@ bootstrap_template1(void)
/* Also ensure backend isn't confused by this environment var: */
unsetenv("PGCLIENTENCODING");
+#ifdef USE_OPENSSL
+ /* Prepare the -K option for the backend. */
+ if (encryption_key_command)
+ {
+ size_t len;
+
+ len = 3 + strlen(encryption_key_command) + 1;
+ encr_key_cmd_str = (char *) pg_malloc(len);
+ snprintf(encr_key_cmd_str, len, "-K %s", encryption_key_command);
+ }
+ else
+ {
+ encr_key_cmd_str = (char *) pg_malloc(1);
+ encr_key_cmd_str[0] = '\0';
+ }
+#endif /* USE_OPENSSL */
+
snprintf(cmd, sizeof(cmd),
- "\"%s\" --boot -x1 -X %u %s %s %s",
+ "\"%s\" --boot -x1 -X %u %s %s %s %s",
backend_exec,
wal_segment_size_mb * (1024 * 1024),
data_checksums ? "-k" : "",
+ encr_key_cmd_str,
boot_options,
debug ? "-d 5" : "");
@@ -2403,6 +2479,10 @@ usage(const char *progname)
printf(_("\nLess commonly used options:\n"));
printf(_(" -d, --debug generate lots of debugging output\n"));
printf(_(" -k, --data-checksums use data page checksums\n"));
+#ifdef USE_OPENSSL
+ printf(_(" -K, --encryption-key-command\n"
+ " command that returns encryption key\n"));
+#endif /* USE_OPENSSL */
printf(_(" -L DIRECTORY where to find the input files\n"));
printf(_(" -n, --no-clean do not clean up after errors\n"));
printf(_(" -N, --no-sync do not wait for changes to be written safely to disk\n"));
@@ -2509,7 +2589,6 @@ setup_pgdata(void)
putenv(pgdata_set_env);
}
-
void
setup_bin_paths(const char *argv0)
{
@@ -3022,8 +3101,8 @@ initialize_data_directory(void)
fflush(stdout);
snprintf(cmd, sizeof(cmd),
- "\"%s\" %s template1 >%s",
- backend_exec, backend_options,
+ "\"%s\" %s %s template1 >%s",
+ backend_exec, backend_options, encr_key_cmd_str,
DEVNULL);
PG_CMD_OPEN;
@@ -3064,6 +3143,23 @@ initialize_data_directory(void)
check_ok();
}
+static void
+parse_extra_option_arg(char *optarg)
+{
+ extra_option *opt;
+
+ if (!strchr(optarg, '='))
+ {
+ fprintf(stderr, _("Option value is not in key=value format"));
+ exit(1);
+ }
+
+ opt = malloc(sizeof(extra_option));
+ if (asprintf(&opt->value, "%s\n", optarg) < 0)
+ exit(1);
+
+ dlist_push_tail(&extra_options, &opt->list_node);
+}
int
main(int argc, char *argv[])
@@ -3098,6 +3194,9 @@ main(int argc, char *argv[])
{"waldir", required_argument, NULL, 'X'},
{"wal-segsize", required_argument, NULL, 12},
{"data-checksums", no_argument, NULL, 'k'},
+#ifdef USE_OPENSSL
+ {"encryption-key-command", required_argument, NULL, 'K'},
+#endif /* USE_OPENSSL */
{"allow-group-access", no_argument, NULL, 'g'},
{NULL, 0, NULL, 0}
};
@@ -3140,7 +3239,7 @@ main(int argc, char *argv[])
/* process command-line options */
- while ((c = getopt_long(argc, argv, "dD:E:kL:nNU:WA:sST:X:g", long_options, &option_index)) != -1)
+ while ((c = getopt_long(argc, argv, "dD:E:kK:L:nNU:WA:sST:X:g", long_options, &option_index)) != -1)
{
switch (c)
{
@@ -3192,6 +3291,14 @@ main(int argc, char *argv[])
case 'k':
data_checksums = true;
break;
+#ifdef USE_OPENSSL
+ case 'K':
+ encryption_key_command = pg_strdup(optarg);
+ break;
+#endif /* USE_OPENSSL */
+ case 'c':
+ parse_extra_option_arg(optarg);
+ break;
case 'L':
share_path = pg_strdup(optarg);
break;
@@ -3365,7 +3472,12 @@ main(int argc, char *argv[])
if (pwprompt || pwfilename)
get_su_pwd();
- printf("\n");
+#ifdef USE_OPENSSL
+ if (encryption_key_command)
+ printf(_("Data encryption is enabled.\n"));
+ else
+ printf(_("Data encryption is disabled.\n"));
+#endif /* USE_OPENSSL */
initialize_data_directory();
diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c
index 895a51f89d..32c702a978 100644
--- a/src/bin/pg_controldata/pg_controldata.c
+++ b/src/bin/pg_controldata/pg_controldata.c
@@ -340,5 +340,14 @@ main(int argc, char *argv[])
ControlFile->data_checksum_version);
printf(_("Mock authentication nonce: %s\n"),
mock_auth_nonce_str);
+ printf(_("Data encryption: %s\n"),
+ ControlFile->data_encrypted ? _("on") : _("off"));
+ if (ControlFile->data_encrypted)
+ printf(_("Data encryption fingerprint: %08X%08X%08X%08X\n"),
+ htonl(((uint32 *) ControlFile->encryption_verification)[0]),
+ htonl(((uint32 *) ControlFile->encryption_verification)[1]),
+ htonl(((uint32 *) ControlFile->encryption_verification)[2]),
+ htonl(((uint32 *) ControlFile->encryption_verification)[3])
+ );
return 0;
}
diff --git a/src/bin/pg_keysetup/Makefile b/src/bin/pg_keysetup/Makefile
new file mode 100644
index 0000000000..c9fe1064d0
--- /dev/null
+++ b/src/bin/pg_keysetup/Makefile
@@ -0,0 +1,32 @@
+# src/bin/pg_keysetup/Makefile
+
+PGFILEDESC = "pg_keysetup - generate encryption key from a password"
+PGAPPICON=win32
+
+subdir = src/bin/pg_keysetup
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+OBJS = pg_keysetup.o $(RMGRDESCOBJS) encryption.o $(WIN32RES)
+
+override CPPFLAGS := -DFRONTEND $(CPPFLAGS)
+
+all: pg_keysetup
+
+pg_keysetup: $(OBJS) | submake-libpgport
+ $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+
+encryption.c: % : $(top_srcdir)/src/backend/storage/smgr/%
+ rm -f $@ && $(LN_S) $< .
+
+install: all installdirs
+ $(INSTALL_PROGRAM) pg_keysetup$(X) '$(DESTDIR)$(bindir)/pg_keysetup$(X)'
+
+installdirs:
+ $(MKDIR_P) '$(DESTDIR)$(bindir)'
+
+uninstall:
+ rm -f '$(DESTDIR)$(bindir)/pg_keysetup$(X)'
+
+clean distclean maintainer-clean:
+ rm -f pg_keysetup$(X) $(OBJS) encryption.c
diff --git a/src/bin/pg_keysetup/pg_keysetup.c b/src/bin/pg_keysetup/pg_keysetup.c
new file mode 100644
index 0000000000..68c33b26f7
--- /dev/null
+++ b/src/bin/pg_keysetup/pg_keysetup.c
@@ -0,0 +1,208 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_keysetup.c - Turn password into encryption key.
+ *
+ * Copyright (c) 2013-2017, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/bin/pg_keysetup/pg_keysetup.c
+ *-------------------------------------------------------------------------
+ */
+
+#define FRONTEND 1
+#include "postgres.h"
+
+#include <dirent.h>
+#include <unistd.h>
+
+#include "port/pg_crc32c.h"
+#include "storage/encryption.h"
+#include "getopt_long.h"
+
+#ifdef USE_ENCRYPTION
+static const char *progname;
+
+static void
+usage(const char *progname)
+{
+ printf(_("%s derives encryption key from a password.\n\n"),
+ progname);
+ printf(_("Usage:\n"));
+ printf(_(" %s [OPTION]...\n"), progname);
+ printf(_("\nOptions:\n"));
+ printf(_(" [-D] DATADIR data directory\n"));
+ printf(_(" -?, --help show this help, then exit\n\n"));
+ printf(_("Password is read from stdin and the key is sent to stdout\n"));
+}
+
+static void fatal_error(const char *fmt,...) pg_attribute_printf(1, 2);
+
+/*
+ * Big red button to push when things go horribly wrong.
+ */
+static void
+fatal_error(const char *fmt,...)
+{
+ va_list args;
+
+ fflush(stdout);
+
+ fprintf(stderr, _("%s: FATAL: "), progname);
+ va_start(args, fmt);
+ vfprintf(stderr, _(fmt), args);
+ va_end(args);
+ fputc('\n', stderr);
+
+ exit(EXIT_FAILURE);
+}
+
+/*
+ * This function does the same as read_kdf_file() in the backend. However most
+ * of the calls look different on frontend side, it seems better to write it
+ * here from scratch than to use too many #ifdef-#else-#endif constructs.
+ */
+static void
+read_kdf_file(char *path)
+{
+ pg_crc32c crc;
+ int fd;
+
+ KDFParams = palloc(KDF_PARAMS_FILE_SIZE);
+
+ fd = open(path, O_RDONLY | PG_BINARY, S_IRUSR);
+
+ if (fd < 0)
+ fatal_error("could not open key setup file \"%s\": %m",
+ KDF_PARAMS_FILE);
+
+ if (read(fd, KDFParams, sizeof(KDFParamsData)) != sizeof(KDFParamsData))
+ fatal_error("could not read from key setup file: %m");
+
+ close(fd);
+
+ /* Now check the CRC. */
+ INIT_CRC32C(crc);
+ COMP_CRC32C(crc,
+ (char *) KDFParams,
+ offsetof(KDFParamsData, crc));
+ FIN_CRC32C(crc);
+
+ if (!EQ_CRC32C(crc, KDFParams->crc))
+ fatal_error("incorrect checksum in key setup file");
+
+ if (KDFParams->function != KDF_OPENSSL_PKCS5_PBKDF2_HMAC_SHA)
+ fatal_error("unsupported KDF function %d", KDFParams->function);
+}
+#endif /* USE_ENCRYPTION */
+
+int
+main(int argc, char **argv)
+{
+#ifdef USE_ENCRYPTION
+ int c;
+ char *DataDir = NULL;
+ char fpath[MAXPGPATH];
+ int fd;
+ char password[ENCRYPTION_PWD_MAX_LENGTH];
+ size_t pwd_len,
+ i;
+
+ progname = get_progname(argv[0]);
+
+ if (argc > 1)
+ {
+ if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0)
+ {
+ usage(progname);
+ exit(0);
+ }
+ if (strcmp(argv[1], "--version") == 0 || strcmp(argv[1], "-V") == 0)
+ {
+ puts("pg_controldata (PostgreSQL) " PG_VERSION);
+ exit(0);
+ }
+ }
+
+ while ((c = getopt(argc, argv, "D:")) != -1)
+ {
+ switch (c)
+ {
+ case 'D':
+ DataDir = optarg;
+ break;
+
+ default:
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+ exit(1);
+ }
+ }
+
+ if (DataDir == NULL)
+ {
+ if (optind < argc)
+ DataDir = argv[optind++];
+ else
+ DataDir = getenv("PGDATA");
+ }
+
+ /* Complain if any arguments remain */
+ if (optind < argc)
+ {
+ fprintf(stderr, _("%s: too many command-line arguments (first is \"%s\")\n"),
+ progname, argv[optind]);
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+ progname);
+ exit(1);
+ }
+
+ if (DataDir == NULL)
+ {
+ fprintf(stderr, _("%s: no data directory specified\n"), progname);
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+ exit(1);
+ }
+
+ snprintf(fpath, MAXPGPATH, "%s/%s", DataDir, KDF_PARAMS_FILE);
+ fd = open(fpath, O_RDONLY | PG_BINARY, 0);
+ if (fd < 0)
+ fatal_error("could not open file \"%s\"", fpath);
+
+ read_kdf_file(fpath);
+ close(fd);
+
+ /*
+ * Read the password.
+ */
+ pwd_len = 0;
+ while (true)
+ {
+ int c = getchar();
+
+ if (c == EOF || c == '\n')
+ break;
+
+ if (pwd_len >= ENCRYPTION_PWD_MAX_LENGTH)
+ fatal_error("The password is too long");
+
+ password[pwd_len++] = c;
+ }
+
+ if (pwd_len < ENCRYPTION_PWD_MIN_LENGTH)
+ fatal_error("The password is too short");
+
+ /*
+ * Run the key derivation function.
+ */
+ setup_encryption_key(password, false, pwd_len);
+
+ /*
+ * Finally print the encryption key.
+ */
+ for (i = 0; i < ENCRYPTION_KEY_LENGTH; i++)
+ printf("%.2x", encryption_key[i]);
+ printf("\n");
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ return 0;
+}
diff --git a/src/bin/pg_resetwal/Makefile b/src/bin/pg_resetwal/Makefile
index e49d52b928..6ea9c515f7 100644
--- a/src/bin/pg_resetwal/Makefile
+++ b/src/bin/pg_resetwal/Makefile
@@ -15,13 +15,18 @@ subdir = src/bin/pg_resetwal
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-OBJS= pg_resetwal.o $(WIN32RES)
+override CPPFLAGS := -DFRONTEND $(CPPFLAGS)
+
+OBJS= pg_resetwal.o encryption.o $(WIN32RES)
all: pg_resetwal
pg_resetwal: $(OBJS) | submake-libpgport
$(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+encryption.c: % : $(top_srcdir)/src/backend/storage/smgr/%
+ rm -f $@ && $(LN_S) $< .
+
install: all installdirs
$(INSTALL_PROGRAM) pg_resetwal$(X) '$(DESTDIR)$(bindir)/pg_resetwal$(X)'
@@ -32,7 +37,7 @@ uninstall:
rm -f '$(DESTDIR)$(bindir)/pg_resetwal$(X)'
clean distclean maintainer-clean:
- rm -f pg_resetwal$(X) $(OBJS)
+ rm -f pg_resetwal$(X) $(OBJS) encryption.c
rm -rf tmp_check
check:
diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c
index 8cff535692..601ba83532 100644
--- a/src/bin/pg_resetwal/pg_resetwal.c
+++ b/src/bin/pg_resetwal/pg_resetwal.c
@@ -54,11 +54,11 @@
#include "common/fe_memutils.h"
#include "common/file_perm.h"
#include "common/restricted_token.h"
+#include "storage/encryption.h"
#include "storage/large_object.h"
#include "pg_getopt.h"
#include "getopt_long.h"
-
static ControlFileData ControlFile; /* pg_control values */
static XLogSegNo newXlogSegNo; /* new XLOG segment # */
static bool guessed = false; /* T if we had to guess at any values */
@@ -135,7 +135,7 @@ main(int argc, char *argv[])
}
- while ((c = getopt_long(argc, argv, "c:D:e:fl:m:no:O:x:", long_options, NULL)) != -1)
+ while ((c = getopt_long(argc, argv, "c:D:e:fK:l:m:no:O:x:", long_options, NULL)) != -1)
{
switch (c)
{
@@ -278,6 +278,12 @@ main(int argc, char *argv[])
}
break;
+#ifdef USE_ENCRYPTION
+ case 'K':
+ encryption_key_command = strdup(optarg);
+ break;
+#endif /* USE_ENCRYPTION */
+
case 'l':
if (strspn(optarg, "01234567890ABCDEFabcdef") != XLOG_FNAME_LEN)
{
@@ -415,6 +421,16 @@ main(int argc, char *argv[])
XLogFromFileName(log_fname, &minXlogTli, &minXlogSegNo, WalSegSz);
/*
+ * If the data is encrypted, we'll also have to encrypt the XLOG record
+ * below.
+ */
+ if (ControlFile.data_encrypted)
+ {
+ setup_encryption(false);
+ data_encrypted = true;
+ }
+
+ /*
* Also look at existing segment files to set up newXlogSegNo
*/
FindEndOfXLOG();
@@ -1255,6 +1271,18 @@ WriteEmptyXLOG(void)
FIN_CRC32C(crc);
record->xl_crc = crc;
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ char tweak[TWEAK_SIZE];
+
+ XLogEncryptionTweak(tweak, newXlogSegNo, 0);
+ encrypt_block(buffer, buffer, XLOG_BLCKSZ, tweak);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
/* Write the first page */
XLogFilePath(path, ControlFile.checkPointCopy.ThisTimeLineID,
newXlogSegNo, WalSegSz);
@@ -1318,6 +1346,10 @@ usage(void)
printf(_(" [-D, --pgdata=]DATADIR data directory\n"));
printf(_(" -e, --epoch=XIDEPOCH set next transaction ID epoch\n"));
printf(_(" -f, --force force update to be done\n"));
+#ifdef USE_ENCRYPTION
+ printf(_(" -K, --encryption-key-command\n"
+ " command that returns encryption key\n"));
+#endif /* USE_ENCRYPTION */
printf(_(" -l, --next-wal-file=WALFILE set minimum starting location for new WAL\n"));
printf(_(" -m, --multixact-ids=MXID,MXID set next and oldest multitransaction ID\n"));
printf(_(" -n, --dry-run no update, just show what would be done\n"));
diff --git a/src/bin/pg_rewind/Makefile b/src/bin/pg_rewind/Makefile
index 2bcfcc61af..50263c0438 100644
--- a/src/bin/pg_rewind/Makefile
+++ b/src/bin/pg_rewind/Makefile
@@ -20,7 +20,7 @@ LDFLAGS_INTERNAL += $(libpq_pgport)
OBJS = pg_rewind.o parsexlog.o xlogreader.o datapagemap.o timeline.o \
fetch.o file_ops.o copy_fetch.o libpq_fetch.o filemap.o logging.o \
- $(WIN32RES)
+ encryption.o $(WIN32RES)
EXTRA_CLEAN = xlogreader.c
@@ -32,6 +32,9 @@ pg_rewind: $(OBJS) | submake-libpq submake-libpgport
xlogreader.c: % : $(top_srcdir)/src/backend/access/transam/%
rm -f $@ && $(LN_S) $< .
+encryption.c: % : $(top_srcdir)/src/backend/storage/smgr/%
+ rm -f $@ && $(LN_S) $< .
+
install: all installdirs
$(INSTALL_PROGRAM) pg_rewind$(X) '$(DESTDIR)$(bindir)/pg_rewind$(X)'
@@ -42,7 +45,7 @@ uninstall:
rm -f '$(DESTDIR)$(bindir)/pg_rewind$(X)'
clean distclean maintainer-clean:
- rm -f pg_rewind$(X) $(OBJS) xlogreader.c
+ rm -f pg_rewind$(X) $(OBJS) xlogreader.c encryption.c
rm -rf tmp_check
check:
diff --git a/src/bin/pg_rewind/parsexlog.c b/src/bin/pg_rewind/parsexlog.c
index b4c1f827a6..057287748d 100644
--- a/src/bin/pg_rewind/parsexlog.c
+++ b/src/bin/pg_rewind/parsexlog.c
@@ -23,7 +23,7 @@
#include "catalog/pg_control.h"
#include "catalog/storage_xlog.h"
#include "commands/dbcommands_xlog.h"
-
+#include "storage/encryption.h"
/*
* RmgrNames is an array of resource manager names, to make error messages
@@ -316,6 +316,18 @@ SimpleXLogPageRead(XLogReaderState *xlogreader, XLogRecPtr targetPagePtr,
return -1;
}
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ char tweak[TWEAK_SIZE];
+
+ XLogEncryptionTweak(tweak, xlogreadsegno, targetPageOff);
+ decrypt_block(readBuf, readBuf, XLOG_BLCKSZ, tweak);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
Assert(targetSegNo == xlogreadsegno);
*pageTLI = targetHistory[private->tliIndex].tli;
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index b0fd3f66ac..c003ed072a 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -28,6 +28,7 @@
#include "common/restricted_token.h"
#include "getopt_long.h"
#include "storage/bufpage.h"
+#include "storage/encryption.h"
static void usage(const char *progname);
@@ -69,6 +70,10 @@ usage(const char *progname)
printf(_(" -D, --target-pgdata=DIRECTORY existing data directory to modify\n"));
printf(_(" --source-pgdata=DIRECTORY source data directory to synchronize with\n"));
printf(_(" --source-server=CONNSTR source server to synchronize with\n"));
+#ifdef USE_OPENSSL
+ printf(_(" -K, --encryption-key-command=COMMAND\n"
+ " command that returns encryption key\n"));
+#endif /* USE_OPENSSL */
printf(_(" -n, --dry-run stop before modifying anything\n"));
printf(_(" -P, --progress write progress messages\n"));
printf(_(" --debug write a lot of debug messages\n"));
@@ -90,6 +95,9 @@ main(int argc, char **argv)
{"dry-run", no_argument, NULL, 'n'},
{"progress", no_argument, NULL, 'P'},
{"debug", no_argument, NULL, 3},
+#ifdef USE_OPENSSL
+ {"encryption-key-command", required_argument, NULL, 'K'},
+#endif /* USE_OPENSSL */
{NULL, 0, NULL, 0}
};
int option_index;
@@ -124,7 +132,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "D:nP", long_options, &option_index)) != -1)
+ while ((c = getopt_long(argc, argv, "D:K:nP", long_options, &option_index)) != -1)
{
switch (c)
{
@@ -154,6 +162,12 @@ main(int argc, char **argv)
case 2: /* --source-server */
connstr_source = pg_strdup(optarg);
break;
+#ifdef USE_OPENSSL
+ case 4: /* --encryption-key-command */
+ case 'K':
+ encryption_key_command = strdup(optarg);
+ break;
+#endif /* USE_OPENSSL */
}
}
@@ -233,6 +247,16 @@ main(int argc, char **argv)
sanityChecks();
/*
+ * Setup encryption if it's obvious that we'll have to deal with encrypted
+ * XLOG.
+ */
+ if (ControlFile_target.data_encrypted)
+ {
+ setup_encryption(false);
+ data_encrypted = true;
+ }
+
+ /*
* If both clusters are already on the same timeline, there's nothing to
* do.
*/
@@ -425,6 +449,24 @@ sanityChecks(void)
ControlFile_source.state != DB_SHUTDOWNED &&
ControlFile_source.state != DB_SHUTDOWNED_IN_RECOVERY)
pg_fatal("source data directory must be shut down cleanly\n");
+
+ /*
+ * Since slave receives XLOG stream encrypted by master, handling
+ * differently encrypted clusters is not the typical use case for
+ * pg_rewind. Yet we should check the encryption.
+ */
+ if (ControlFile_source.data_encrypted ||
+ ControlFile_target.data_encrypted)
+ {
+ if (ControlFile_source.data_encrypted !=
+ ControlFile_target.data_encrypted)
+ pg_fatal("source and target server must be both unencrypted or both encrypted\n");
+
+ if (memcmp(ControlFile_source.encryption_verification,
+ ControlFile_target.encryption_verification,
+ ENCRYPTION_SAMPLE_SIZE))
+ pg_fatal("both source and target server must use the same encryption key");
+ }
}
/*
diff --git a/src/bin/pg_upgrade/check.c b/src/bin/pg_upgrade/check.c
index 577db73f10..f698a51701 100644
--- a/src/bin/pg_upgrade/check.c
+++ b/src/bin/pg_upgrade/check.c
@@ -267,6 +267,23 @@ check_cluster_versions(void)
GET_MAJOR_VERSION(new_cluster.bin_version))
pg_fatal("New cluster data and binary directories are from different major versions.\n");
+ /*
+ * As this is the first version that supports encryption, we actually do
+ * not expect the old cluster to be encrypted.
+ *
+ * TODO Change this logic as soon as we merge the encryption into the
+ * next, even minor version (pg_upgrade seems to accept clusters that
+ * differ only in the minor version), or for the PG core patch. Eventually
+ * we'll support the upgrade if either both clusters are unencrypted, or
+ * if both are encrypted using the same key. In the latter case, we need
+ * to ensure that the PGENCRYPTIONKEY environment variable is passed to
+ * both clusters.
+ */
+ if (old_cluster.controldata.data_encrypted)
+ pg_fatal("Old cluster is encrypted.\n");
+ if (new_cluster.controldata.data_encrypted)
+ pg_fatal("New cluster is encrypted.\n");
+
check_ok();
}
diff --git a/src/bin/pg_upgrade/controldata.c b/src/bin/pg_upgrade/controldata.c
index 0fe98a550e..d9fa8c63d1 100644
--- a/src/bin/pg_upgrade/controldata.c
+++ b/src/bin/pg_upgrade/controldata.c
@@ -58,6 +58,7 @@ get_control_data(ClusterInfo *cluster, bool live_check)
bool got_large_object = false;
bool got_date_is_int = false;
bool got_data_checksum_version = false;
+ bool got_data_encrypted = false;
char *lc_collate = NULL;
char *lc_ctype = NULL;
char *lc_monetary = NULL;
@@ -135,6 +136,13 @@ get_control_data(ClusterInfo *cluster, bool live_check)
got_data_checksum_version = true;
}
+ /* Only in <= 9.6 */
+ if (GET_MAJOR_VERSION(cluster->major_version) <= 906)
+ {
+ cluster->controldata.data_encrypted = false;
+ got_data_encrypted = true;
+ }
+
/* we have the result of cmd in "output". so parse it line by line now */
while (fgets(bufin, sizeof(bufin), output))
{
@@ -418,6 +426,34 @@ get_control_data(ClusterInfo *cluster, bool live_check)
cluster->controldata.data_checksum_version = str2uint(p);
got_data_checksum_version = true;
}
+ else if ((p = strstr(bufin, "encryption fingerprint")) != NULL)
+ {
+ int i;
+
+ p = strchr(p, ':');
+
+ if (p == NULL || strlen(p) <= 1)
+ pg_fatal("%d: controldata retrieval problem\n", __LINE__);
+
+ cluster->controldata.data_encrypted = true;
+
+ /* Skip the colon and any whitespace after it */
+ p = strchr(p, ':');
+ if (p == NULL || strlen(p) <= 1)
+ pg_fatal("%d: controldata retrieval problem\n", __LINE__);
+ p = strpbrk(p, "01234567890ABCDEF");
+ if (p == NULL || strlen(p) <= 1)
+ pg_fatal("%d: controldata retrieval problem\n", __LINE__);
+
+ /* Make sure it looks like a valid finerprint */
+ if (strspn(p, "0123456789ABCDEF") != 32)
+ pg_fatal("%d: controldata retrieval problem\n", __LINE__);
+
+ for (i = 0; i < 16; i++)
+ sscanf(p + 2 * i, "%2hhx",
+ cluster->controldata.encryption_verification + i);
+ got_data_encrypted = true;
+ }
}
pclose(output);
@@ -472,7 +508,7 @@ get_control_data(ClusterInfo *cluster, bool live_check)
!got_index || !got_toast ||
(!got_large_object &&
cluster->controldata.ctrl_ver >= LARGE_OBJECT_SIZE_PG_CONTROL_VER) ||
- !got_date_is_int || !got_data_checksum_version)
+ !got_date_is_int || !got_data_checksum_version || !got_data_encrypted)
{
if (cluster == &old_cluster)
pg_log(PG_REPORT,
@@ -538,6 +574,10 @@ get_control_data(ClusterInfo *cluster, bool live_check)
if (!got_data_checksum_version)
pg_log(PG_REPORT, " data checksum version\n");
+ /* value added in Postgres 10 */
+ if (!got_data_encrypted)
+ pg_log(PG_REPORT, " data encryption status\n");
+
pg_fatal("Cannot continue without required control information, terminating\n");
}
}
@@ -602,6 +642,18 @@ check_control_data(ControlData *oldctrl,
pg_fatal("old cluster uses data checksums but the new one does not\n");
else if (oldctrl->data_checksum_version != newctrl->data_checksum_version)
pg_fatal("old and new cluster pg_controldata checksum versions do not match\n");
+
+ if (oldctrl->data_encrypted && !newctrl->data_encrypted)
+ pg_fatal("old cluster is encrypted, but the new one is not\n");
+ else if (!oldctrl->data_encrypted && newctrl->data_encrypted)
+ pg_fatal("old cluster is not encrypted, but the new one is\n");
+ else if (oldctrl->data_encrypted && newctrl->data_encrypted)
+ {
+ if (oldctrl->encryption_verification != newctrl->encryption_verification)
+ pg_fatal("encryption keys do not match between old and new cluster\n");
+ else
+ pg_fatal("upgrading encrypted databases is not implemented yet\n"); /* TODO */
+ }
}
diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h
index 7e5e971294..66291cbf47 100644
--- a/src/bin/pg_upgrade/pg_upgrade.h
+++ b/src/bin/pg_upgrade/pg_upgrade.h
@@ -227,6 +227,8 @@ typedef struct
bool date_is_int;
bool float8_pass_by_value;
bool data_checksum_version;
+ bool data_encrypted;
+ uint8 encryption_verification[16];
} ControlData;
/*
diff --git a/src/bin/pg_waldump/Makefile b/src/bin/pg_waldump/Makefile
index f5957bd75a..5a8bd1fba9 100644
--- a/src/bin/pg_waldump/Makefile
+++ b/src/bin/pg_waldump/Makefile
@@ -7,8 +7,8 @@ subdir = src/bin/pg_waldump
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-OBJS = pg_waldump.o compat.o xlogreader.o rmgrdesc.o \
- $(RMGRDESCOBJS) $(WIN32RES)
+OBJS = pg_waldump.o compat.o xlogreader.o rmgrdesc.o encryption.o \
+ $(RMGRDESCOBJS) encryption.o $(WIN32RES)
override CPPFLAGS := -DFRONTEND $(CPPFLAGS)
@@ -27,6 +27,9 @@ xlogreader.c: % : $(top_srcdir)/src/backend/access/transam/%
$(RMGRDESCSOURCES): % : $(top_srcdir)/src/backend/access/rmgrdesc/%
rm -f $@ && $(LN_S) $< .
+encryption.c: % : $(top_srcdir)/src/backend/storage/smgr/%
+ rm -f $@ && $(LN_S) $< .
+
install: all installdirs
$(INSTALL_PROGRAM) pg_waldump$(X) '$(DESTDIR)$(bindir)/pg_waldump$(X)'
@@ -37,4 +40,5 @@ uninstall:
rm -f '$(DESTDIR)$(bindir)/pg_waldump$(X)'
clean distclean maintainer-clean:
- rm -f pg_waldump$(X) $(OBJS) $(RMGRDESCSOURCES) xlogreader.c
+ rm -f pg_waldump$(X) $(OBJS) $(RMGRDESCSOURCES) xlogreader.c \
+encryption.c
diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c
index 5c4f38e597..0013d20785 100644
--- a/src/bin/pg_waldump/pg_waldump.c
+++ b/src/bin/pg_waldump/pg_waldump.c
@@ -21,12 +21,15 @@
#include "access/xlog_internal.h"
#include "access/transam.h"
#include "common/fe_memutils.h"
+#include "storage/encryption.h"
#include "getopt_long.h"
#include "rmgrdesc.h"
static const char *progname;
+static char *str_wal_segment_size_mb = NULL;
+static bool encr_inited = false;
static int WalSegSz;
typedef struct XLogDumpPrivate
@@ -213,8 +216,85 @@ search_directory(const char *directory, const char *fname)
if (read(fd, buf, XLOG_BLCKSZ) == XLOG_BLCKSZ)
{
- XLogLongPageHeader longhdr = (XLogLongPageHeader) buf;
+ XLogLongPageHeader longhdr;
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ uint32 tli;
+ char tweak[TWEAK_SIZE];
+ XLogSegNo segNo;
+ int wal_segment_size;
+ XLogPageHeader hdr;
+
+ /*
+ * Segment size is contained in the encryption tweak so we
+ * cannot get the size from the header until the page is
+ * decrypted. User needs to pass the size if he uses
+ * non-default value.
+ *
+ * TODO The processing of wal-segsize is almost copy & pasted
+ * from initdb.c. Reuse in a better way.
+ */
+ /* set wal segment size */
+ if (str_wal_segment_size_mb == NULL)
+ wal_segment_size = DEFAULT_XLOG_SEG_SIZE;
+ else
+ {
+ char *endptr;
+
+ /* check that the argument is a number */
+ wal_segment_size = strtol(str_wal_segment_size_mb, &endptr, 10);
+
+ /* verify that wal segment size is valid */
+ if (endptr == str_wal_segment_size_mb || *endptr != '\0')
+ {
+ fprintf(stderr,
+ _("%s: argument of --wal-segsize must be a number\n"),
+ progname);
+ exit(1);
+ }
+
+ wal_segment_size *= 1024 * 1024;
+ if (!IsValidWalSegSize(wal_segment_size))
+ {
+ fprintf(stderr,
+ _("%s: argument of --wal-segsize must be a power of 2 between 1 and 1024\n"),
+ progname);
+ exit(1);
+ }
+ }
+
+ XLogFromFileName(fname, &tli, &segNo, wal_segment_size);
+ XLogEncryptionTweak(tweak, segNo, 0);
+ setup_encryption(false);
+ encr_inited = true;
+ decrypt_block(buf, buf, XLOG_BLCKSZ, tweak);
+
+ hdr = (XLogPageHeader) buf;
+ if (hdr->xlp_magic != XLOG_PAGE_MAGIC)
+ {
+ fprintf(stderr,
+ _("invalid magic number %04X. Make sure you've passed correct argument for --wal-segsize"),
+ hdr->xlp_magic);
+ exit(1);
+ }
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+ else
+ {
+ if (str_wal_segment_size_mb != NULL)
+ {
+ fprintf(stderr,
+ _("%s: --wal-segsize is only accepted for encrypted WAL\n"),
+ progname);
+ exit(1);
+ }
+ }
+
+ longhdr = (XLogLongPageHeader) buf;
WalSegSz = longhdr->xlp_seg_size;
if (!IsValidWalSegSize(WalSegSz))
@@ -453,6 +533,23 @@ XLogDumpReadPage(XLogReaderState *state, XLogRecPtr targetPagePtr, int reqLen,
XLogDumpXLogRead(private->inpath, private->timeline, targetPagePtr,
readBuff, count);
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ char tweak[TWEAK_SIZE];
+ XLogSegNo readSegNo;
+ uint32 readSegOff;
+
+ XLByteToSeg(targetPagePtr, readSegNo, WalSegSz);
+ readSegOff = targetPagePtr % WalSegSz;
+
+ XLogEncryptionTweak(tweak, readSegNo, readSegOff);
+ decrypt_block(readBuff, readBuff, count, tweak);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
return count;
}
@@ -801,6 +898,12 @@ usage(void)
printf(_(" -b, --bkp-details output detailed information about backup blocks\n"));
printf(_(" -e, --end=RECPTR stop reading at WAL location RECPTR\n"));
printf(_(" -f, --follow keep retrying after reaching end of WAL\n"));
+#ifdef USE_ENCRYPTION
+ printf(_(" -K, --encryption-key-command=COMMAND\n"
+ " command that returns encryption key\n"));
+ printf(_(" -w, --wal-segsize=SIZE\n"
+ " size of WAL segments, in megabytes\n"));
+#endif /* USE_ENCRYPTION */
printf(_(" -n, --limit=N number of records to display\n"));
printf(_(" -p, --path=PATH directory in which to find log segment files or a\n"
" directory with a ./pg_wal that contains such files\n"
@@ -835,6 +938,7 @@ main(int argc, char **argv)
{"end", required_argument, NULL, 'e'},
{"follow", no_argument, NULL, 'f'},
{"help", no_argument, NULL, '?'},
+ {"encryption-key-command", required_argument, NULL, 'K'},
{"limit", required_argument, NULL, 'n'},
{"path", required_argument, NULL, 'p'},
{"rmgr", required_argument, NULL, 'r'},
@@ -842,6 +946,7 @@ main(int argc, char **argv)
{"timeline", required_argument, NULL, 't'},
{"xid", required_argument, NULL, 'x'},
{"version", no_argument, NULL, 'V'},
+ {"wal-segsize", required_argument, NULL, 'w'},
{"stats", optional_argument, NULL, 'z'},
{NULL, 0, NULL, 0}
};
@@ -877,7 +982,7 @@ main(int argc, char **argv)
goto bad_argument;
}
- while ((option = getopt_long(argc, argv, "be:?fn:p:r:s:t:Vx:z",
+ while ((option = getopt_long(argc, argv, "be:?fK:n:p:r:s:t:Vx:z",
long_options, &optindex)) != -1)
{
switch (option)
@@ -901,6 +1006,12 @@ main(int argc, char **argv)
usage();
exit(EXIT_SUCCESS);
break;
+#ifdef USE_ENCRYPTION
+ case 'K':
+ encryption_key_command = pg_strdup(optarg);
+ data_encrypted = true;
+ break;
+#endif /* USE_ENCRYPTION */
case 'n':
if (sscanf(optarg, "%d", &config.stop_after_records) != 1)
{
@@ -961,6 +1072,9 @@ main(int argc, char **argv)
puts("pg_waldump (PostgreSQL) " PG_VERSION);
exit(EXIT_SUCCESS);
break;
+ case 'w':
+ str_wal_segment_size_mb = pg_strdup(optarg);
+ break;
case 'x':
if (sscanf(optarg, "%u", &config.filter_by_xid) != 1)
{
@@ -1108,6 +1222,12 @@ main(int argc, char **argv)
/* done with argument parsing, do the actual work */
+ if (data_encrypted && !encr_inited)
+ {
+ setup_encryption(false);
+ encr_inited = false;
+ }
+
/* we have everything we need, start reading */
xlogreader_state = XLogReaderAllocate(WalSegSz, XLogDumpReadPage,
&private);
diff --git a/src/common/string.c b/src/common/string.c
index 3260d37a84..57ede1dd14 100644
--- a/src/common/string.c
+++ b/src/common/string.c
@@ -42,6 +42,45 @@ pg_str_endswith(const char *str, const char *end)
return strcmp(str, end) == 0;
}
+/*
+ * Helper function to check if a page is completely empty.
+ *
+ * TODO Invent name that is more consistent with that of the other function(s)
+ * in this module.
+ */
+bool
+IsAllZero(const char *input, Size size)
+{
+ const char *pos = input;
+ const char *aligned_start = (char *) MAXALIGN64(input);
+ const char *end = input + size;
+
+ /* Check 1 byte at a time until pos is 8 byte aligned */
+ while (pos < aligned_start)
+ if (*pos++ != 0)
+ return false;
+
+ /*
+ * Run 8 parallel 8 byte checks in one iteration. On 2016 hardware
+ * slightly faster than 4 parallel checks.
+ */
+ while (pos + 8 * sizeof(uint64) <= end)
+ {
+ uint64 *p = (uint64 *) pos;
+
+ if ((p[0] | p[1] | p[2] | p[3] | p[4] | p[5] | p[6] | p[7]) != 0)
+ return false;
+ pos += 8 * sizeof(uint64);
+ }
+
+ /* Handle unaligned tail. */
+ while (pos < end)
+ if (*pos++ != 0)
+ return false;
+
+ return true;
+}
+
/*
* strtoint --- just like strtol, but returns int not long
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 421ba6d775..c7fea0f6cb 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -258,6 +258,7 @@ extern void UpdateControlFile(void);
extern uint64 GetSystemIdentifier(void);
extern char *GetMockAuthenticationNonce(void);
extern bool DataChecksumsEnabled(void);
+extern bool DataEncryptionEnabled(void);
extern XLogRecPtr GetFakeLSNForUnloggedRel(void);
extern Size XLOGShmemSize(void);
extern void XLOGShmemInit(void);
diff --git a/src/include/access/xlog_internal.h b/src/include/access/xlog_internal.h
index 7c766836db..7499bd0ca9 100644
--- a/src/include/access/xlog_internal.h
+++ b/src/include/access/xlog_internal.h
@@ -52,7 +52,7 @@ typedef struct XLogPageHeaderData
uint32 xlp_rem_len; /* total len of remaining data for record */
} XLogPageHeaderData;
-#define SizeOfXLogShortPHD MAXALIGN(sizeof(XLogPageHeaderData))
+#define SizeOfXLogShortPHD XLOG_REC_ALIGN(sizeof(XLogPageHeaderData))
typedef XLogPageHeaderData *XLogPageHeader;
@@ -69,7 +69,7 @@ typedef struct XLogLongPageHeaderData
uint32 xlp_xlog_blcksz; /* just as a cross-check */
} XLogLongPageHeaderData;
-#define SizeOfXLogLongPHD MAXALIGN(sizeof(XLogLongPageHeaderData))
+#define SizeOfXLogLongPHD XLOG_REC_ALIGN(sizeof(XLogLongPageHeaderData))
typedef XLogLongPageHeaderData *XLogLongPageHeader;
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index 773d9e6eba..ce4386d592 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -94,6 +94,11 @@ typedef enum DBState
} DBState;
/*
+ * Number of bytes reserved to store encryption sample in ControlFileData.
+ */
+#define ENCRYPTION_SAMPLE_SIZE 16
+
+/*
* Contents of pg_control.
*/
@@ -227,6 +232,11 @@ typedef struct ControlFileData
*/
char mock_authentication_nonce[MOCK_AUTH_NONCE_LEN];
+ /* Is data directory encrypted? */
+ bool data_encrypted;
+ /* Sample value for encryption key verification */
+ uint8 encryption_verification[ENCRYPTION_SAMPLE_SIZE];
+
/* CRC of all above ... MUST BE LAST! */
pg_crc32c crc;
} ControlFileData;
diff --git a/src/include/common/string.h b/src/include/common/string.h
index 78a450192e..26fbfdc51e 100644
--- a/src/include/common/string.h
+++ b/src/include/common/string.h
@@ -13,5 +13,5 @@
extern bool pg_str_endswith(const char *str, const char *end);
extern int strtoint(const char *pg_restrict str, char **pg_restrict endptr,
int base);
-
+extern bool IsAllZero(const char *input, Size size);
#endif /* COMMON_STRING_H */
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index e167ee8fcb..bc01ace87a 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -429,6 +429,7 @@ extern void BaseInit(void);
/* in utils/init/miscinit.c */
extern bool IgnoreSystemIndexes;
extern PGDLLIMPORT bool process_shared_preload_libraries_in_progress;
+extern char *encryption_library_string;
extern char *session_preload_libraries_string;
extern char *shared_preload_libraries_string;
extern char *local_preload_libraries_string;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index be2f59239b..b6948fce83 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -874,6 +874,9 @@ typedef enum
WAIT_EVENT_DATA_FILE_TRUNCATE,
WAIT_EVENT_DATA_FILE_WRITE,
WAIT_EVENT_DSM_FILL_ZERO_WRITE,
+ WAIT_EVENT_KDF_FILE_READ,
+ WAIT_EVENT_KDF_FILE_SYNC,
+ WAIT_EVENT_KDF_FILE_WRITE,
WAIT_EVENT_LOCK_FILE_ADDTODATADIR_READ,
WAIT_EVENT_LOCK_FILE_ADDTODATADIR_SYNC,
WAIT_EVENT_LOCK_FILE_ADDTODATADIR_WRITE,
diff --git a/src/include/storage/copydir.h b/src/include/storage/copydir.h
index 4fef3e2107..8aea3eb2ed 100644
--- a/src/include/storage/copydir.h
+++ b/src/include/storage/copydir.h
@@ -13,7 +13,9 @@
#ifndef COPYDIR_H
#define COPYDIR_H
-extern void copydir(char *fromdir, char *todir, bool recurse);
-extern void copy_file(char *fromfile, char *tofile);
+#include "storage/relfilenode.h"
+
+extern void copydir(char *fromdir, char *todir, RelFileNode *fromNode, RelFileNode *toNode);
+extern void copy_file(char *fromfile, char *tofile, RelFileNode *fromNode, RelFileNode *toNode, ForkNumber fromForkNum, ForkNumber toForkNum, int segment);
#endif /* COPYDIR_H */
diff --git a/src/include/storage/encryption.h b/src/include/storage/encryption.h
new file mode 100644
index 0000000000..dab1bea600
--- /dev/null
+++ b/src/include/storage/encryption.h
@@ -0,0 +1,182 @@
+/*-------------------------------------------------------------------------
+ *
+ * encryption.h
+ * Full database encryption support
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/encryption.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef ENCRYPTION_H
+#define ENCRYPTION_H
+
+#include "access/xlogdefs.h"
+#include "port/pg_crc32c.h"
+#include "lib/ilist.h"
+
+/*
+ * OpenSSL is currently the only implementation of encryption we use.
+ */
+#ifdef USE_OPENSSL
+#define USE_ENCRYPTION
+#endif
+
+/*
+ * Common error message issued when particular code path cannot be executed
+ * due to absence of the OpenSSL library.
+ */
+#define ENCRYPTION_NOT_SUPPORTED_MSG \
+ encryption_error(true, \
+ "data encryption cannot be used because SSL is not supported by this build\n" \
+ "Compile with --with-openssl to use SSL connections.")
+#ifdef USE_ENCRYPTION
+/*
+ * Full database encryption key.
+ *
+ * EVP_aes_256_xts() needs the key twice as long as AES would do in general.
+ */
+#define ENCRYPTION_KEY_LENGTH 64
+
+/*
+ * TODO Tune these values.
+ */
+#define ENCRYPTION_PWD_MIN_LENGTH 8
+#define ENCRYPTION_PWD_MAX_LENGTH 16
+#define ENCRYPTION_KDF_NITER 1048576
+#define ENCRYPTION_KDF_SALT_LEN 8
+
+extern unsigned char *encryption_key;
+#endif /* USE_ENCRYPTION */
+
+/*
+ * The encrypted data is a series of blocks of size
+ * ENCRYPTION_BLOCK. Currently we use the EVP_aes_256_xts implementation. Make
+ * sure the following constants match if adopting another algorithm.
+ */
+#define ENCRYPTION_BLOCK 16
+
+#ifdef USE_ENCRYPTION
+/*
+ * The openssl EVP API refers to a block in terms of padding of the output
+ * chunk. That's the purpose of this constant. However the openssl
+ * implementation of AES XTS still uses the 16-byte block internally, as
+ * defined by ENCRYPTION_BLOCK.
+ */
+#define ENCRYPTION_BLOCK_OPENSSL 1
+
+#define TWEAK_SIZE 16
+
+#define KDF_PARAMS_FILE "global/pg_keysetup"
+#define KDF_PARAMS_FILE_SIZE 512
+
+/*
+ * Key derivation function.
+ */
+typedef enum KDFKind
+{
+ KDF_OPENSSL_PKCS5_PBKDF2_HMAC_SHA = 0
+} KFDKind;
+
+typedef struct KDFParamsPBKDF2
+{
+ unsigned long int niter;
+ unsigned char salt[ENCRYPTION_KDF_SALT_LEN];
+} KDFParamsPBKDF2;
+
+/*
+ * Parameters of the key derivation function.
+ *
+ * The parameters are generated by initdb and stored into a file, which is
+ * then read during PG startup. This is similar to storing various settings in
+ * pg_control. However an existing KDF file is read only, so it does not have
+ * to be stored in shared memory.
+ */
+typedef struct KDFParamsData
+{
+ KFDKind function;
+
+ /*
+ * Function-specific parameters.
+ */
+ union
+ {
+ KDFParamsPBKDF2 pbkdf2;
+ } data;
+
+ /* CRC of all above ... MUST BE LAST! */
+ pg_crc32c crc;
+} KDFParamsData;
+
+extern KDFParamsData *KDFParams;
+#endif /* USE_ENCRYPTION */
+
+extern PGDLLIMPORT bool data_encrypted;
+
+#ifdef USE_ENCRYPTION
+#ifndef FRONTEND
+extern void write_kdf_file(void);
+extern void read_kdf_file(void);
+#endif /* FRONTEND */
+
+extern PGDLLIMPORT char *encryption_key_command;
+
+extern char *encryption_buffer;
+extern Size encryption_buf_size;
+
+extern void enlarge_encryption_buffer(Size new_size);
+#endif /* USE_ENCRYPTION */
+
+/*
+ * If one XLOG record ended and the following one started in the same block,
+ * we'd have to either encrypt and decrypt both records together, or encrypt
+ * (after having zeroed the part of the block occupied by the other record)
+ * and decrypt them separate. Neither approach is compatible with streaming
+ * replication. In the first case we can't ask standby not to decrypt the
+ * first record until the second has been streamed. The second approach would
+ * imply streaming of two different versions of the same block two times.
+ *
+ * We avoid this problem by aligning XLOG records to the encryption block
+ * size. This way no adjacent XLOG records should appear in the same block.
+ *
+ * For similar reasons, the alignment to ENCRYPTION_BLOCK also has to be
+ * applied when storing changes to disk in reorderbuffer.c. Another module
+ * that takes the block into account is buffile.c.
+ *
+ * TODO If the configuration allows walsender to decrypt the XLOG stream
+ * before sending it, adjust this expression so that the additional padding is
+ * not added to XLOG records. (Since the XLOG alignment cannot change without
+ * initdb, the same would apply to the configuration variable that makes
+ * walsender perform the decryption. Does such a variable make sense?)
+ */
+#define DO_ENCRYPTION_BLOCK_ALIGN data_encrypted
+
+/*
+ * Use TYPEALIGN64 since besides record size we also need to align XLogRecPtr.
+ */
+#define ENCRYPTION_BLOCK_ALIGN(LEN) TYPEALIGN64(ENCRYPTION_BLOCK, (LEN))
+
+/*
+ * Universal computation of XLOG record alignment.
+ */
+#define XLOG_REC_ALIGN(LEN) ((DO_ENCRYPTION_BLOCK_ALIGN) ?\
+ ENCRYPTION_BLOCK_ALIGN(LEN) : MAXALIGN64(LEN))
+
+extern void setup_encryption(bool bootstrap);
+extern void setup_encryption_key(char *credentials, bool is_key, size_t len);
+extern void sample_encryption(char *buf);
+extern void encrypt_block(const char *input, char *output, Size size,
+ const char *tweak);
+extern void decrypt_block(const char *input, char *output, Size size,
+ const char *tweak);
+#ifdef USE_ENCRYPTION
+extern void encryption_error(bool fatal, char *message);
+#endif
+
+extern void XLogEncryptionTweak(char *tweak, XLogSegNo segment,
+ uint32 offset);
+
+#endif /* ENCRYPTION_H */
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index 8e7c9728f4..0567c68c0c 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -72,6 +72,7 @@ extern int FileRead(File file, char *buffer, int amount, uint32 wait_event_info)
extern int FileWrite(File file, char *buffer, int amount, uint32 wait_event_info);
extern int FileSync(File file, uint32 wait_event_info);
extern off_t FileSeek(File file, off_t offset, int whence);
+extern off_t FileTell(File file);
extern int FileTruncate(File file, off_t offset, uint32 wait_event_info);
extern void FileWriteback(File file, off_t offset, off_t nbytes, uint32 wait_event_info);
extern char *FilePathName(File file);
diff --git a/src/include/storage/reinit.h b/src/include/storage/reinit.h
index a62703c647..5efd101d03 100644
--- a/src/include/storage/reinit.h
+++ b/src/include/storage/reinit.h
@@ -20,7 +20,8 @@
extern void ResetUnloggedRelations(int op);
extern bool parse_filename_for_nontemp_relation(const char *name,
- int *oidchars, ForkNumber *fork);
+ int *oidchars, ForkNumber *fork,
+ int *segment);
#define UNLOGGED_RELATION_CLEANUP 0x0001
#define UNLOGGED_RELATION_INIT 0x0002
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index 558e4d8518..4866821897 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -144,4 +144,17 @@ extern void RememberFsyncRequest(RelFileNode rnode, ForkNumber forknum,
extern void ForgetRelationFsyncRequests(RelFileNode rnode, ForkNumber forknum);
extern void ForgetDatabaseFsyncRequests(Oid dbid);
+#ifdef USE_OPENSSL
+extern BlockNumber ReencryptBlock(char *buffer, int blocks,
+ RelFileNode *srcNode, RelFileNode *dstNode,
+ ForkNumber srcForkNum, ForkNumber dstForkNum,
+ BlockNumber blockNum);
+#endif
+
+/* smgrtype.c */
+extern Datum smgrout(PG_FUNCTION_ARGS);
+extern Datum smgrin(PG_FUNCTION_ARGS);
+extern Datum smgreq(PG_FUNCTION_ARGS);
+extern Datum smgrne(PG_FUNCTION_ARGS);
+
#endif /* SMGR_H */
diff --git a/src/test/modules/buffile/Makefile b/src/test/modules/buffile/Makefile
new file mode 100644
index 0000000000..b658c0dd8d
--- /dev/null
+++ b/src/test/modules/buffile/Makefile
@@ -0,0 +1,21 @@
+PG_CONFIG ?= pg_config
+MODULE_big = buffile_test
+OBJS = buffile.o $(WIN32RES)
+PGFILEDESC = "buffile_test"
+
+EXTENSION = buffile
+DATA = buffile--1.0.sql
+
+REGRESS = test_00 test_01 test_02 test_03 test_04 test_05 test_06 test_07 \
+test_08 test_09 test_10 test_11 test_12
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/buffile
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/buffile/README b/src/test/modules/buffile/README
new file mode 100644
index 0000000000..9c80787797
--- /dev/null
+++ b/src/test/modules/buffile/README
@@ -0,0 +1,11 @@
+This extension was written to check if changes introduced due to cluster
+encryption do not break buffile.c.
+
+Caution: To make the test cheaper, it was decided to adjust the segment sizeq,
+see
+
+#define MAX_PHYSICAL_FILESIZE (4 * BLCKSZ)
+
+in buffile.c. BLCKSZ is 8192 (the default). All the tests rely on this
+value. So if you haven't compiled Postgres with this value, the tests will
+create 1 GB files and they will fail.
diff --git a/src/test/modules/buffile/buffile--1.0.sql b/src/test/modules/buffile/buffile--1.0.sql
new file mode 100644
index 0000000000..82baa043f7
--- /dev/null
+++ b/src/test/modules/buffile/buffile--1.0.sql
@@ -0,0 +1,39 @@
+CREATE FUNCTION buffile_create()
+RETURNS void
+AS 'MODULE_PATHNAME', 'buffile_create'
+LANGUAGE C;
+
+CREATE FUNCTION buffile_close()
+RETURNS void
+AS 'MODULE_PATHNAME', 'buffile_close'
+LANGUAGE C;
+
+CREATE FUNCTION buffile_write(text)
+RETURNS bigint
+AS 'MODULE_PATHNAME', 'buffile_write'
+LANGUAGE C;
+
+CREATE FUNCTION buffile_read(bigint)
+RETURNS bytea
+AS 'MODULE_PATHNAME', 'buffile_read'
+LANGUAGE C;
+
+CREATE FUNCTION buffile_seek(int, bigint)
+RETURNS int
+AS 'MODULE_PATHNAME', 'buffile_seek'
+LANGUAGE C;
+
+CREATE FUNCTION buffile_assert_fileno(int)
+RETURNS void
+AS 'MODULE_PATHNAME', 'buffile_assert_fileno'
+LANGUAGE C;
+
+CREATE FUNCTION buffile_test_shared()
+RETURNS void
+AS 'MODULE_PATHNAME', 'buffile_test_shared'
+LANGUAGE C;
+
+CREATE FUNCTION buffile_test_shared_append()
+RETURNS void
+AS 'MODULE_PATHNAME', 'buffile_test_shared_append'
+LANGUAGE C;
diff --git a/src/test/modules/buffile/buffile.c b/src/test/modules/buffile/buffile.c
new file mode 100644
index 0000000000..45ad6778ab
--- /dev/null
+++ b/src/test/modules/buffile/buffile.c
@@ -0,0 +1,353 @@
+#include "postgres.h"
+#include "fmgr.h"
+#include "lib/stringinfo.h"
+#include "storage/buffile.h"
+#include "utils/array.h"
+#include "utils/builtins.h"
+#include "utils/lsyscache.h"
+#include "utils/memutils.h"
+#include "utils/resowner.h"
+
+PG_MODULE_MAGIC;
+
+/*
+ * To cover various corner cases, the tests assume MAX_PHYSICAL_FILESIZE to be
+ * exactly MAX_PHYSICAL_FILESIZE_TEST.
+ */
+#define MAX_PHYSICAL_FILESIZE_TEST (4 * BLCKSZ)
+
+static BufFile *bf = NULL;
+
+static void check_file(void);
+
+extern Datum buffile_create(PG_FUNCTION_ARGS);
+
+PG_FUNCTION_INFO_V1(buffile_create);
+Datum
+buffile_create(PG_FUNCTION_ARGS)
+{
+ MemoryContext old_cxt;
+ ResourceOwner old_ro;
+
+ if (bf != NULL)
+ elog(ERROR, "file already exists");
+
+ old_cxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ /*
+ * Make sure the file is not deleted across function calls.
+ */
+ old_ro = CurrentResourceOwner;
+ CurrentResourceOwner = TopTransactionResourceOwner;
+
+ bf = BufFileCreateTemp(false);
+
+ CurrentResourceOwner = old_ro;
+ MemoryContextSwitchTo(old_cxt);
+
+ PG_RETURN_VOID();
+}
+
+extern Datum buffile_close(PG_FUNCTION_ARGS);
+
+PG_FUNCTION_INFO_V1(buffile_close);
+Datum
+buffile_close(PG_FUNCTION_ARGS)
+{
+ if (bf == NULL)
+ elog(ERROR, "there's no file to close");
+
+ BufFileClose(bf);
+ bf = NULL;
+
+ PG_RETURN_VOID();
+}
+
+extern Datum buffile_write(PG_FUNCTION_ARGS);
+
+PG_FUNCTION_INFO_V1(buffile_write);
+Datum
+buffile_write(PG_FUNCTION_ARGS)
+{
+ Datum d = PG_GETARG_DATUM(0);
+ char *s = TextDatumGetCString(d);
+ size_t res;
+
+ check_file();
+ res = BufFileWrite(bf, s, strlen(s));
+
+ PG_RETURN_INT64(res);
+}
+
+extern Datum buffile_read(PG_FUNCTION_ARGS);
+
+PG_FUNCTION_INFO_V1(buffile_read);
+Datum
+buffile_read(PG_FUNCTION_ARGS)
+{
+ int64 size = PG_GETARG_INT64(0);
+ StringInfo buf = makeStringInfo();
+ size_t res_size;
+ bytea *result;
+
+ check_file();
+
+ enlargeStringInfo(buf, size);
+ res_size = BufFileRead(bf, buf->data, size);
+ buf->len = res_size;
+
+ result = DatumGetByteaPP(DirectFunctionCall1(bytearecv,
+ PointerGetDatum(buf)));
+ PG_RETURN_BYTEA_P(result);
+}
+
+extern Datum buffile_seek(PG_FUNCTION_ARGS);
+
+PG_FUNCTION_INFO_V1(buffile_seek);
+Datum
+buffile_seek(PG_FUNCTION_ARGS)
+{
+ int32 fileno = PG_GETARG_INT32(0);
+ int64 offset = PG_GETARG_INT64(1);
+ int32 res;
+
+ check_file();
+ res = BufFileSeek(bf, fileno, offset, SEEK_SET);
+
+ PG_RETURN_INT32(res);
+}
+
+extern Datum buffile_assert_fileno(PG_FUNCTION_ARGS);
+
+PG_FUNCTION_INFO_V1(buffile_assert_fileno);
+Datum
+buffile_assert_fileno(PG_FUNCTION_ARGS)
+{
+ int32 fileno_expected = PG_GETARG_INT32(0);
+ int32 fileno;
+ off_t offset;
+
+ check_file();
+ BufFileTell(bf, &fileno, &offset);
+
+ if (fileno != fileno_expected)
+ {
+ /*
+ * Bring the backend down so that the following tests have no chance
+ * to create the 1GB files.
+ */
+ elog(FATAL, "file number does not match");
+ }
+
+ PG_RETURN_VOID();
+}
+
+static void
+check_file(void)
+{
+ if (bf == NULL)
+ elog(ERROR, "the file is not opened");
+}
+
+/*
+ * This test is especially important for shared encrypted files, see the
+ * comments below.
+ */
+extern Datum buffile_test_shared(PG_FUNCTION_ARGS);
+
+PG_FUNCTION_INFO_V1(buffile_test_shared);
+Datum
+buffile_test_shared(PG_FUNCTION_ARGS)
+{
+ dsm_segment *seg;
+ SharedFileSet *fileset;
+ BufFile *bf_1,
+ *bf_2;
+ char *data_1,
+ *data_2,
+ *data;
+ Size chunk_size_1,
+ chunk_size_2;
+ int fileno,
+ i;
+ off_t offset,
+ res,
+ total_size;
+
+ /*
+ * The size is not important, we actually do not need the shared memory.
+ * The segment is only needed to initialize the fileset.
+ */
+ seg = dsm_create(1024, 0);
+
+ /*
+ * The fileset must survive error handling, so that dsm_detach works fine.
+ * (The typical use case is that the fileset is in shared memory.)
+ */
+ fileset = (SharedFileSet *) MemoryContextAlloc(TopTransactionContext,
+ sizeof(SharedFileSet));
+ SharedFileSetInit(fileset, seg);
+
+ bf_1 = BufFileCreateShared(fileset, "file_1");
+
+ /*
+ * Write more data than the buffer size, so that we can check that the
+ * number of "useful bytes" word is only appended at the end of the
+ * segment, not after each buffer.
+ */
+ chunk_size_1 = BLCKSZ + 256;
+ data_1 = (char *) palloc(chunk_size_1);
+ memset(data_1, 1, chunk_size_1);
+ if (BufFileWrite(bf_1, data_1, chunk_size_1) != chunk_size_1)
+ elog(ERROR, "Failed to write data");
+ pfree(data_1);
+
+ /*
+ * Enforce buffer flush (The BufFileFlush() function is not exported).
+ * Thus the "useful bytes" metadata should appear at the current end the
+ * first file segment. The next write will have to seek back to overwrite
+ * the metadata.
+ */
+ BufFileTell(bf_1, &fileno, &offset);
+ if (BufFileSeek(bf_1, 0, 0, SEEK_SET) != 0)
+ elog(ERROR, "seek failed");
+ if (BufFileSeek(bf_1, fileno, offset, SEEK_SET) != 0)
+ elog(ERROR, "seek failed");
+
+ /*
+ * Write another chunk that does not fit into the first segment file. Thus
+ * the "useful bytes" metadata should appear at the end of both segments.
+ */
+ chunk_size_2 = 3 * BLCKSZ;
+ data_2 = (char *) palloc(chunk_size_2);
+ memset(data_2, 1, chunk_size_2);
+ if (BufFileWrite(bf_1, data_2, chunk_size_2) != chunk_size_2)
+ elog(ERROR, "Failed to write data");
+ pfree(data_2);
+ BufFileClose(bf_1);
+
+ /*
+ * The word indicating the number of "useful bytes" (i.e. the actual data
+ * w/o padding to buffer size) is stored at the end of each segment file.
+ * Check that this metadata is read correctly.
+ */
+ bf_2 = BufFileOpenShared(fileset, "file_1");
+ total_size = BufFileSize(bf_2);
+ if (total_size != (chunk_size_1 + chunk_size_2))
+ elog(ERROR, "Incorrect file size: %zu", total_size);
+
+ data = (char *) palloc(total_size);
+ res = BufFileRead(bf_2, data, total_size);
+ if (res != total_size)
+ elog(ERROR, "Incorrect chunk size read: %zu", res);
+ for (i = 0; i < total_size; i++)
+ if (data[i] != 1)
+ elog(ERROR, "Unexpected data read from the file");
+ pfree(data);
+ BufFileClose(bf_2);
+
+ dsm_detach(seg);
+
+ PG_RETURN_VOID();
+}
+
+
+/*
+ * Test BufFileAppend().
+ */
+extern Datum buffile_test_shared_append(PG_FUNCTION_ARGS);
+
+PG_FUNCTION_INFO_V1(buffile_test_shared_append);
+Datum
+buffile_test_shared_append(PG_FUNCTION_ARGS)
+{
+ dsm_segment *seg;
+ SharedFileSet *fileset;
+ BufFile *bf_1,
+ *bf_2,
+ *bf_3;
+ char *data;
+ Size chunk_size;
+ int fileno,
+ i;
+ off_t offset,
+ res,
+ total_size;
+
+ seg = dsm_create(1024, 0);
+
+ fileset = (SharedFileSet *) MemoryContextAlloc(TopTransactionContext,
+ sizeof(SharedFileSet));
+ SharedFileSetInit(fileset, seg);
+
+ /*
+ * XXX Does the chunk size matter much?
+ */
+ chunk_size = 8;
+ data = (char *) palloc(chunk_size);
+ memset(data, 1, chunk_size);
+
+ bf_1 = BufFileCreateShared(fileset, "file_1");
+ if (BufFileWrite(bf_1, data, chunk_size) != chunk_size)
+ elog(ERROR, "Failed to write data");
+
+ bf_2 = BufFileCreateShared(fileset, "file_2");
+ if (BufFileWrite(bf_2, data, chunk_size) != chunk_size)
+ elog(ERROR, "Failed to write data");
+
+ /*
+ * Make sure it's read-only so that BufFileAppend() can accept it as
+ * source.
+ */
+ BufFileClose(bf_2);
+ bf_2 = BufFileOpenShared(fileset, "file_2");
+
+ bf_3 = BufFileCreateShared(fileset, "file_3");
+ if (BufFileWrite(bf_3, data, chunk_size) != chunk_size)
+ elog(ERROR, "Failed to write data");
+ BufFileClose(bf_3);
+ bf_3 = BufFileOpenShared(fileset, "file_3");
+
+ BufFileAppend(bf_1, bf_2);
+ BufFileAppend(bf_1, bf_3);
+
+ total_size = BufFileSize(bf_1);
+
+ /*
+ * The result should contain complete segments of bf_1 and bf_2 and the
+ * valid part of bf_3.
+ */
+ if (total_size != (2 * MAX_PHYSICAL_FILESIZE_TEST + chunk_size))
+ elog(ERROR, "Incorrect total size of the appended data: %zu",
+ total_size);
+
+ /*
+ * Check that data of the 2nd segment was decrypted correctly.
+ */
+ if (BufFileSeek(bf_1, 1, 0, SEEK_SET) != 0)
+ elog(ERROR, "seek failed");
+ res = BufFileRead(bf_1, data, chunk_size);
+ if (res != chunk_size)
+ elog(ERROR, "Incorrect chunk size read: %zu", res);
+ for (i = 0; i < chunk_size; i++)
+ if (data[i] != 1)
+ elog(ERROR, "Unexpected data read from the file");
+
+ /*
+ * And the same for the 3rd segment.
+ *
+ * TODO Reuse the code above by putting it into a function.
+ */
+ if (BufFileSeek(bf_1, 2, 0, SEEK_SET) != 0)
+ elog(ERROR, "seek failed");
+ res = BufFileRead(bf_1, data, chunk_size);
+ if (res != chunk_size)
+ elog(ERROR, "Incorrect chunk size read: %zu", res);
+ for (i = 0; i < chunk_size; i++)
+ if (data[i] != 1)
+ elog(ERROR, "Unexpected data read from the file");
+
+ BufFileClose(bf_1);
+ dsm_detach(seg);
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/buffile/buffile.control b/src/test/modules/buffile/buffile.control
new file mode 100644
index 0000000000..8472c5a348
--- /dev/null
+++ b/src/test/modules/buffile/buffile.control
@@ -0,0 +1,5 @@
+# buffile_test extension
+comment = 'buffile_test'
+default_version = '1.0'
+module_pathname = '$libdir/buffile_test'
+relocatable = true
diff --git a/src/test/modules/buffile/expected/test_00.out b/src/test/modules/buffile/expected/test_00.out
new file mode 100644
index 0000000000..637c33689f
--- /dev/null
+++ b/src/test/modules/buffile/expected/test_00.out
@@ -0,0 +1,47 @@
+CREATE EXTENSION IF NOT EXISTS buffile;
+-- This test only verifies that PG is compiled with a component file size of
+-- 32 kB (i.e. 4 buffers of 8 kB) instead of 1 GB. That seems appropriate for
+-- testing. Some other tests may rely on it.
+BEGIN;
+SELECT buffile_create();
+ buffile_create
+----------------
+
+(1 row)
+
+-- Skip the first component file.
+SELECT buffile_seek(0, 32768);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+-- Write the first byte of the second component file. We can't simply
+-- buffile_seek() beyond the position 32768 as it would return EOF.
+SELECT buffile_write('a');
+ buffile_write
+---------------
+ 1
+(1 row)
+
+-- Enforce BufFileFlush(), which actually adds the component file.
+SELECT buffile_read(1);
+ buffile_read
+--------------
+ \x
+(1 row)
+
+-- Check that we're in the 2nd file, i.e. the file size is as expected.
+SELECT buffile_assert_fileno(1);
+ buffile_assert_fileno
+-----------------------
+
+(1 row)
+
+SELECT buffile_close();
+ buffile_close
+---------------
+
+(1 row)
+
+COMMIT;
diff --git a/src/test/modules/buffile/expected/test_01.out b/src/test/modules/buffile/expected/test_01.out
new file mode 100644
index 0000000000..448bac171c
--- /dev/null
+++ b/src/test/modules/buffile/expected/test_01.out
@@ -0,0 +1,59 @@
+BEGIN;
+SELECT buffile_create();
+ buffile_create
+----------------
+
+(1 row)
+
+SELECT buffile_seek(0, 1);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_write('abc');
+ buffile_write
+---------------
+ 3
+(1 row)
+
+SELECT buffile_seek(0, 0);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+-- Check that the trailing zeroes are not fetched.
+SELECT buffile_read(16);
+ buffile_read
+--------------
+ \x00616263
+(1 row)
+
+-- Adjust the number of useful bytes.
+SELECT buffile_write('abc');
+ buffile_write
+---------------
+ 3
+(1 row)
+
+-- ... and check again.
+SELECT buffile_seek(0, 0);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_read(16);
+ buffile_read
+------------------
+ \x00616263616263
+(1 row)
+
+SELECT buffile_close();
+ buffile_close
+---------------
+
+(1 row)
+
+COMMIT;
diff --git a/src/test/modules/buffile/expected/test_02.out b/src/test/modules/buffile/expected/test_02.out
new file mode 100644
index 0000000000..f783d0cb24
--- /dev/null
+++ b/src/test/modules/buffile/expected/test_02.out
@@ -0,0 +1,48 @@
+BEGIN;
+SELECT buffile_create();
+ buffile_create
+----------------
+
+(1 row)
+
+SELECT buffile_seek(0, 8189);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+-- Initialize the last 3 positions of the first buffer and the initial 3
+-- positions of the 2nd buffer.
+SELECT buffile_write('abcdef');
+ buffile_write
+---------------
+ 6
+(1 row)
+
+SELECT buffile_seek(0, 0);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+-- Read the first buffer.
+SELECT length(buffile_read(8192));
+ length
+--------
+ 8192
+(1 row)
+
+-- Only 3 bytes of the 2nd buffer should be fetched.
+SELECT length(buffile_read(8192));
+ length
+--------
+ 3
+(1 row)
+
+SELECT buffile_close();
+ buffile_close
+---------------
+
+(1 row)
+
+COMMIT;
diff --git a/src/test/modules/buffile/expected/test_03.out b/src/test/modules/buffile/expected/test_03.out
new file mode 100644
index 0000000000..e899fa3b38
--- /dev/null
+++ b/src/test/modules/buffile/expected/test_03.out
@@ -0,0 +1,27 @@
+BEGIN;
+SELECT buffile_create();
+ buffile_create
+----------------
+
+(1 row)
+
+-- Read from an empty file.
+SELECT buffile_seek(0, 8);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_read(16);
+ buffile_read
+--------------
+ \x
+(1 row)
+
+SELECT buffile_close();
+ buffile_close
+---------------
+
+(1 row)
+
+COMMIT;
diff --git a/src/test/modules/buffile/expected/test_04.out b/src/test/modules/buffile/expected/test_04.out
new file mode 100644
index 0000000000..1f8eeabe48
--- /dev/null
+++ b/src/test/modules/buffile/expected/test_04.out
@@ -0,0 +1,84 @@
+BEGIN;
+SELECT buffile_create();
+ buffile_create
+----------------
+
+(1 row)
+
+-- Write something near the end of the first buffer, but leave some trailing
+-- space.
+SELECT buffile_seek(0, 8184);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_write('abcd');
+ buffile_write
+---------------
+ 4
+(1 row)
+
+-- Leave the 2nd buffer empty, as well as a few leading bytes. Thus we should
+-- get a hole that spans the whole 2nd buffer as well as a few adjacent bytes
+-- on each side.
+SELECT buffile_seek(0, 2 * 8192 + 4);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_write('efgh');
+ buffile_write
+---------------
+ 4
+(1 row)
+
+-- Check the initial part of the hole, which crosses the boundary of the 1st
+-- and the 2nd buffer.
+SELECT buffile_seek(0, 8184);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_read(16);
+ buffile_read
+------------------------------------
+ \x61626364000000000000000000000000
+(1 row)
+
+-- Check the trailing part of the whole, which crosses the boundary of the 2nd
+-- and the 3rd buffer.
+SELECT buffile_seek(0, 2 * 8192 - 8);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_read(16);
+ buffile_read
+------------------------------------
+ \x00000000000000000000000065666768
+(1 row)
+
+-- Check that the hole contains nothing but zeroes.
+SELECT buffile_seek(0, 8192 - 4);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT btrim(buffile_read(8192 + 8), '\x00');
+ btrim
+-------
+ \x
+(1 row)
+
+SELECT buffile_close();
+ buffile_close
+---------------
+
+(1 row)
+
+COMMIT;
diff --git a/src/test/modules/buffile/expected/test_05.out b/src/test/modules/buffile/expected/test_05.out
new file mode 100644
index 0000000000..6a73147711
--- /dev/null
+++ b/src/test/modules/buffile/expected/test_05.out
@@ -0,0 +1,33 @@
+BEGIN;
+SELECT buffile_create();
+ buffile_create
+----------------
+
+(1 row)
+
+-- Seek does not extend the file if it's not followed by write.
+SELECT buffile_seek(0, 1);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_seek(0, 0);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_read(2);
+ buffile_read
+--------------
+ \x
+(1 row)
+
+SELECT buffile_close();
+ buffile_close
+---------------
+
+(1 row)
+
+COMMIT;
diff --git a/src/test/modules/buffile/expected/test_06.out b/src/test/modules/buffile/expected/test_06.out
new file mode 100644
index 0000000000..def9af5112
--- /dev/null
+++ b/src/test/modules/buffile/expected/test_06.out
@@ -0,0 +1,41 @@
+-- This test shows that the first component file (segment) stays empty, read
+-- stops prematurely even if it starts on that segment, even though it'd
+-- otherwise receive some data from the following one.
+BEGIN;
+SELECT buffile_create();
+ buffile_create
+----------------
+
+(1 row)
+
+SELECT buffile_seek(0, 32768);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_write('a');
+ buffile_write
+---------------
+ 1
+(1 row)
+
+SELECT buffile_seek(0, 32767);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_read(2);
+ buffile_read
+--------------
+ \x
+(1 row)
+
+SELECT buffile_close();
+ buffile_close
+---------------
+
+(1 row)
+
+COMMIT;
diff --git a/src/test/modules/buffile/expected/test_07.out b/src/test/modules/buffile/expected/test_07.out
new file mode 100644
index 0000000000..215280f0b0
--- /dev/null
+++ b/src/test/modules/buffile/expected/test_07.out
@@ -0,0 +1,39 @@
+BEGIN;
+SELECT buffile_create();
+ buffile_create
+----------------
+
+(1 row)
+
+-- Write data at component file boundary and try to read it.
+SELECT buffile_seek(0, 32768);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_write('abcd');
+ buffile_write
+---------------
+ 4
+(1 row)
+
+SELECT buffile_seek(0, 32768);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_read(8);
+ buffile_read
+--------------
+ \x61626364
+(1 row)
+
+SELECT buffile_close();
+ buffile_close
+---------------
+
+(1 row)
+
+COMMIT;
diff --git a/src/test/modules/buffile/expected/test_08.out b/src/test/modules/buffile/expected/test_08.out
new file mode 100644
index 0000000000..7c1069930c
--- /dev/null
+++ b/src/test/modules/buffile/expected/test_08.out
@@ -0,0 +1,39 @@
+BEGIN;
+SELECT buffile_create();
+ buffile_create
+----------------
+
+(1 row)
+
+-- Write data across component file boundary and try to read it.
+SELECT buffile_seek(0, 32766);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_write('abcd');
+ buffile_write
+---------------
+ 4
+(1 row)
+
+SELECT buffile_seek(0, 32766);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_read(8);
+ buffile_read
+--------------
+ \x61626364
+(1 row)
+
+SELECT buffile_close();
+ buffile_close
+---------------
+
+(1 row)
+
+COMMIT;
diff --git a/src/test/modules/buffile/expected/test_09.out b/src/test/modules/buffile/expected/test_09.out
new file mode 100644
index 0000000000..b9d325b676
--- /dev/null
+++ b/src/test/modules/buffile/expected/test_09.out
@@ -0,0 +1,39 @@
+BEGIN;
+SELECT buffile_create();
+ buffile_create
+----------------
+
+(1 row)
+
+-- Write data across buffer boundary and try to read it.
+SELECT buffile_seek(0, 8190);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_write('abcd');
+ buffile_write
+---------------
+ 4
+(1 row)
+
+SELECT buffile_seek(0, 8190);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_read(8);
+ buffile_read
+--------------
+ \x61626364
+(1 row)
+
+SELECT buffile_close();
+ buffile_close
+---------------
+
+(1 row)
+
+COMMIT;
diff --git a/src/test/modules/buffile/expected/test_10.out b/src/test/modules/buffile/expected/test_10.out
new file mode 100644
index 0000000000..8e457fdcda
--- /dev/null
+++ b/src/test/modules/buffile/expected/test_10.out
@@ -0,0 +1,76 @@
+BEGIN;
+SELECT buffile_create();
+ buffile_create
+----------------
+
+(1 row)
+
+-- Write some data at the end of the buffer.
+SELECT buffile_seek(0, 8188);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_write('abcd');
+ buffile_write
+---------------
+ 4
+(1 row)
+
+SELECT buffile_seek(0, 8189);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+-- Enforce flush with the write position not at the end of the buffer. This is
+-- special by not moving curOffset to the next buffer.
+SELECT buffile_read(1);
+ buffile_read
+--------------
+ \x62
+(1 row)
+
+-- Therefore the next writes should eventually affect the original data. (Here
+-- we also test going directly from read to write and vice versa.)
+SELECT buffile_write('x');
+ buffile_write
+---------------
+ 1
+(1 row)
+
+SELECT buffile_read(1);
+ buffile_read
+--------------
+ \x64
+(1 row)
+
+-- Start a new buffer, i.e. force flushing of the previous one.
+SELECT buffile_write('z');
+ buffile_write
+---------------
+ 1
+(1 row)
+
+-- Check that the 'x' and 'y' letters are in the first buffer, not in the
+-- 2nd. (We read enough data to find any non-zero bytes in the 2nd buffer.)
+SELECT buffile_seek(0, 8188);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_read(4 + 8192);
+ buffile_read
+--------------
+ \x616278647a
+(1 row)
+
+SELECT buffile_close();
+ buffile_close
+---------------
+
+(1 row)
+
+COMMIT;
diff --git a/src/test/modules/buffile/expected/test_11.out b/src/test/modules/buffile/expected/test_11.out
new file mode 100644
index 0000000000..c6804d08af
--- /dev/null
+++ b/src/test/modules/buffile/expected/test_11.out
@@ -0,0 +1,34 @@
+BEGIN;
+SELECT buffile_create();
+ buffile_create
+----------------
+
+(1 row)
+
+SELECT buffile_write('abcd');
+ buffile_write
+---------------
+ 4
+(1 row)
+
+-- Seek beyond EOF not followed by write.
+SELECT buffile_seek(0, 5);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+-- Nothing should be fetched.
+SELECT buffile_read(8);
+ buffile_read
+--------------
+ \x
+(1 row)
+
+SELECT buffile_close();
+ buffile_close
+---------------
+
+(1 row)
+
+COMMIT;
diff --git a/src/test/modules/buffile/expected/test_12.out b/src/test/modules/buffile/expected/test_12.out
new file mode 100644
index 0000000000..3dce7eaf0f
--- /dev/null
+++ b/src/test/modules/buffile/expected/test_12.out
@@ -0,0 +1,12 @@
+SELECT buffile_test_shared();
+ buffile_test_shared
+---------------------
+
+(1 row)
+
+SELECT buffile_test_shared_append();
+ buffile_test_shared_append
+----------------------------
+
+(1 row)
+
diff --git a/src/test/modules/buffile/sql/test_00.sql b/src/test/modules/buffile/sql/test_00.sql
new file mode 100644
index 0000000000..c907a5e195
--- /dev/null
+++ b/src/test/modules/buffile/sql/test_00.sql
@@ -0,0 +1,18 @@
+CREATE EXTENSION IF NOT EXISTS buffile;
+
+-- This test only verifies that PG is compiled with a component file size of
+-- 32 kB (i.e. 4 buffers of 8 kB) instead of 1 GB. That seems appropriate for
+-- testing. Some other tests may rely on it.
+BEGIN;
+SELECT buffile_create();
+-- Skip the first component file.
+SELECT buffile_seek(0, 32768);
+-- Write the first byte of the second component file. We can't simply
+-- buffile_seek() beyond the position 32768 as it would return EOF.
+SELECT buffile_write('a');
+-- Enforce BufFileFlush(), which actually adds the component file.
+SELECT buffile_read(1);
+-- Check that we're in the 2nd file, i.e. the file size is as expected.
+SELECT buffile_assert_fileno(1);
+SELECT buffile_close();
+COMMIT;
diff --git a/src/test/modules/buffile/sql/test_01.sql b/src/test/modules/buffile/sql/test_01.sql
new file mode 100644
index 0000000000..8cc87de841
--- /dev/null
+++ b/src/test/modules/buffile/sql/test_01.sql
@@ -0,0 +1,14 @@
+BEGIN;
+SELECT buffile_create();
+SELECT buffile_seek(0, 1);
+SELECT buffile_write('abc');
+SELECT buffile_seek(0, 0);
+-- Check that the trailing zeroes are not fetched.
+SELECT buffile_read(16);
+-- Adjust the number of useful bytes.
+SELECT buffile_write('abc');
+-- ... and check again.
+SELECT buffile_seek(0, 0);
+SELECT buffile_read(16);
+SELECT buffile_close();
+COMMIT;
diff --git a/src/test/modules/buffile/sql/test_02.sql b/src/test/modules/buffile/sql/test_02.sql
new file mode 100644
index 0000000000..35d34722d7
--- /dev/null
+++ b/src/test/modules/buffile/sql/test_02.sql
@@ -0,0 +1,13 @@
+BEGIN;
+SELECT buffile_create();
+SELECT buffile_seek(0, 8189);
+-- Initialize the last 3 positions of the first buffer and the initial 3
+-- positions of the 2nd buffer.
+SELECT buffile_write('abcdef');
+SELECT buffile_seek(0, 0);
+-- Read the first buffer.
+SELECT length(buffile_read(8192));
+-- Only 3 bytes of the 2nd buffer should be fetched.
+SELECT length(buffile_read(8192));
+SELECT buffile_close();
+COMMIT;
diff --git a/src/test/modules/buffile/sql/test_03.sql b/src/test/modules/buffile/sql/test_03.sql
new file mode 100644
index 0000000000..a95391f7c3
--- /dev/null
+++ b/src/test/modules/buffile/sql/test_03.sql
@@ -0,0 +1,7 @@
+BEGIN;
+SELECT buffile_create();
+-- Read from an empty file.
+SELECT buffile_seek(0, 8);
+SELECT buffile_read(16);
+SELECT buffile_close();
+COMMIT;
diff --git a/src/test/modules/buffile/sql/test_04.sql b/src/test/modules/buffile/sql/test_04.sql
new file mode 100644
index 0000000000..64e8d39f94
--- /dev/null
+++ b/src/test/modules/buffile/sql/test_04.sql
@@ -0,0 +1,25 @@
+BEGIN;
+SELECT buffile_create();
+-- Write something near the end of the first buffer, but leave some trailing
+-- space.
+SELECT buffile_seek(0, 8184);
+SELECT buffile_write('abcd');
+-- Leave the 2nd buffer empty, as well as a few leading bytes. Thus we should
+-- get a hole that spans the whole 2nd buffer as well as a few adjacent bytes
+-- on each side.
+SELECT buffile_seek(0, 2 * 8192 + 4);
+SELECT buffile_write('efgh');
+-- Check the initial part of the hole, which crosses the boundary of the 1st
+-- and the 2nd buffer.
+SELECT buffile_seek(0, 8184);
+SELECT buffile_read(16);
+-- Check the trailing part of the whole, which crosses the boundary of the 2nd
+-- and the 3rd buffer.
+SELECT buffile_seek(0, 2 * 8192 - 8);
+SELECT buffile_read(16);
+-- Check that the hole contains nothing but zeroes.
+SELECT buffile_seek(0, 8192 - 4);
+SELECT btrim(buffile_read(8192 + 8), '\x00');
+
+SELECT buffile_close();
+COMMIT;
diff --git a/src/test/modules/buffile/sql/test_05.sql b/src/test/modules/buffile/sql/test_05.sql
new file mode 100644
index 0000000000..5fd642e558
--- /dev/null
+++ b/src/test/modules/buffile/sql/test_05.sql
@@ -0,0 +1,8 @@
+BEGIN;
+SELECT buffile_create();
+-- Seek does not extend the file if it's not followed by write.
+SELECT buffile_seek(0, 1);
+SELECT buffile_seek(0, 0);
+SELECT buffile_read(2);
+SELECT buffile_close();
+COMMIT;
diff --git a/src/test/modules/buffile/sql/test_06.sql b/src/test/modules/buffile/sql/test_06.sql
new file mode 100644
index 0000000000..bbf506ec0e
--- /dev/null
+++ b/src/test/modules/buffile/sql/test_06.sql
@@ -0,0 +1,11 @@
+-- This test shows that the first component file (segment) stays empty, read
+-- stops prematurely even if it starts on that segment, even though it'd
+-- otherwise receive some data from the following one.
+BEGIN;
+SELECT buffile_create();
+SELECT buffile_seek(0, 32768);
+SELECT buffile_write('a');
+SELECT buffile_seek(0, 32767);
+SELECT buffile_read(2);
+SELECT buffile_close();
+COMMIT;
diff --git a/src/test/modules/buffile/sql/test_07.sql b/src/test/modules/buffile/sql/test_07.sql
new file mode 100644
index 0000000000..ca78506ecb
--- /dev/null
+++ b/src/test/modules/buffile/sql/test_07.sql
@@ -0,0 +1,9 @@
+BEGIN;
+SELECT buffile_create();
+-- Write data at component file boundary and try to read it.
+SELECT buffile_seek(0, 32768);
+SELECT buffile_write('abcd');
+SELECT buffile_seek(0, 32768);
+SELECT buffile_read(8);
+SELECT buffile_close();
+COMMIT;
diff --git a/src/test/modules/buffile/sql/test_08.sql b/src/test/modules/buffile/sql/test_08.sql
new file mode 100644
index 0000000000..0e5b1de383
--- /dev/null
+++ b/src/test/modules/buffile/sql/test_08.sql
@@ -0,0 +1,9 @@
+BEGIN;
+SELECT buffile_create();
+-- Write data across component file boundary and try to read it.
+SELECT buffile_seek(0, 32766);
+SELECT buffile_write('abcd');
+SELECT buffile_seek(0, 32766);
+SELECT buffile_read(8);
+SELECT buffile_close();
+COMMIT;
diff --git a/src/test/modules/buffile/sql/test_09.sql b/src/test/modules/buffile/sql/test_09.sql
new file mode 100644
index 0000000000..cc7060932e
--- /dev/null
+++ b/src/test/modules/buffile/sql/test_09.sql
@@ -0,0 +1,9 @@
+BEGIN;
+SELECT buffile_create();
+-- Write data across buffer boundary and try to read it.
+SELECT buffile_seek(0, 8190);
+SELECT buffile_write('abcd');
+SELECT buffile_seek(0, 8190);
+SELECT buffile_read(8);
+SELECT buffile_close();
+COMMIT;
diff --git a/src/test/modules/buffile/sql/test_10.sql b/src/test/modules/buffile/sql/test_10.sql
new file mode 100644
index 0000000000..63af760d9f
--- /dev/null
+++ b/src/test/modules/buffile/sql/test_10.sql
@@ -0,0 +1,25 @@
+BEGIN;
+SELECT buffile_create();
+-- Write some data at the end of the buffer.
+SELECT buffile_seek(0, 8188);
+SELECT buffile_write('abcd');
+SELECT buffile_seek(0, 8189);
+-- Enforce flush with the write position not at the end of the buffer. This is
+-- special by not moving curOffset to the next buffer.
+SELECT buffile_read(1);
+
+-- Therefore the next writes should eventually affect the original data. (Here
+-- we also test going directly from read to write and vice versa.)
+SELECT buffile_write('x');
+SELECT buffile_read(1);
+
+-- Start a new buffer, i.e. force flushing of the previous one.
+SELECT buffile_write('z');
+
+-- Check that the 'x' and 'y' letters are in the first buffer, not in the
+-- 2nd. (We read enough data to find any non-zero bytes in the 2nd buffer.)
+SELECT buffile_seek(0, 8188);
+SELECT buffile_read(4 + 8192);
+
+SELECT buffile_close();
+COMMIT;
diff --git a/src/test/modules/buffile/sql/test_11.sql b/src/test/modules/buffile/sql/test_11.sql
new file mode 100644
index 0000000000..94300d253f
--- /dev/null
+++ b/src/test/modules/buffile/sql/test_11.sql
@@ -0,0 +1,9 @@
+BEGIN;
+SELECT buffile_create();
+SELECT buffile_write('abcd');
+-- Seek beyond EOF not followed by write.
+SELECT buffile_seek(0, 5);
+-- Nothing should be fetched.
+SELECT buffile_read(8);
+SELECT buffile_close();
+COMMIT;
diff --git a/src/test/modules/buffile/sql/test_12.sql b/src/test/modules/buffile/sql/test_12.sql
new file mode 100644
index 0000000000..a28ae395ef
--- /dev/null
+++ b/src/test/modules/buffile/sql/test_12.sql
@@ -0,0 +1,2 @@
+SELECT buffile_test_shared();
+SELECT buffile_test_shared_append();
README.encryption (text/plain, 2.8K) download
PostgreSQL instance encryption
==============================
If the instance (cluster) encryption is enabled, data (including WAL) is
encrypted before it's stored to disk. Conversely, decryption is the first
thing to happen when the data is read from disk into memory.
Installation
------------
1. openssl library is used to perform the actual encryption. Therefore you
need to make sure that you have openssl installed.
2. Run the configure script and (besides other options) pass --with-openssl to
it.
3. make install
Create encrypted instance
-------------------------
In order to encrypt / decrypt data, the PostgreSQL processes need an
encryption key. You need to provide the server either with the actual key or
with a password from which the key will be derived.
The initdb utility accepts command line option -K, whose argument is a command
that returns the key or the password. Depending on your choice, the expected
output of the command is
encryption_key=<your key>
(The key length is 64 bytes and it's expected in hexadecimal format, hence 128
hexadecimal digits.)
or
encryption_password=<your password>
Whichever kind of credentials you prefer, simply run initdb with the -K
option, and that's it. For example:
initdb -K /data/my_pwd_command.sh -D data_encrypted
The command should now appear in your postgresql.conf file (see the variable
encryption_key_command) so you don't have to pass it again when starting the
instance.
If you passed encryption password (as opposed the encryption key), it's
recommended to generate the key as well. While the PostgreSQL server (backend)
accepts either kind of credentials, the front-end utilities (pg_waldump,
pg_resetwal, pg_rewind, etc.) only accept the encryption key.
To derive the key, run the pg_keysetup utility and pass your data directory to
it. The utility reads the password from standard input and writes the key to
the standard output. For example:
echo <encryption password> | pg_keysetup -D data_encrypted
Once you have a command that returns this key (with the "encryption_key="
prefix as explained above), you can set it as the value of the
encryption_key_command configuration variable or use it to run the front-end
commands. For example:
pg_waldump -K /data/my_key_command.sh -p data_encrypted/pg_wal/ 000000010000000000000001
CAUTION! Different key is derived if you run pg_keysetup on a different data
directory, even if the password is identical.
Startup
-------
Since the key command is in postgresql.conf, there's nothing special about the
instance startup. Once you have your instance running, the first thing you
should do is to connect and check the "data_encryption" variable:
SHOW data_encryption;
data_encryption
-----------------
on
(1 row)
The "on" value indicates that encryption of your new instance was successfully
enabled.
* Re: [HACKERS] WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2018-06-27 08:02 ` Re: [HACKERS] WIP: Data at rest encryption Antonin Houska <ah@cybertec.at>
@ 2018-07-13 07:56 ` Toshi Harada <harada.toshi@po.ntt-tx.co.jp>
2018-07-25 11:47 ` Re: [HACKERS] WIP: Data at rest encryption Antonin Houska <ah@cybertec.at>
0 siblings, 1 reply; 93+ messages in thread
From: Toshi Harada @ 2018-07-13 07:56 UTC (permalink / raw)
To: Antonin Houska <ah@cybertec.at>; +Cc: pgsql-hackers
Hi.
I am interested in a patch of "WIP: Data at rest encryption".
This patch("data-at-rest-encryption-wip-2018.06.27.patch") is applied to PostgreSQL 11-beta 2 and it is running.
In the explanation of this patch, since "data stored during logical decoding" is written,
we tried logical decoding by the test_decoding module, but the following error occurs when creating a slot.
pgbench_db=# SELECT * FROM pg_create_logical_replication_slot('my_slot', 'test_decoding');
ERROR: invalid magic number B419 in log segment 000000020000000000000010, offset 0
pgbench_db=#
(Also, if you run "CREATE SUNSCRIPTION" for logical replication from another server, a similar error will occur.)
Question.
In "data-at-rest-encryption-wip-2018.06.27.patch", is logical decoding still not implemented?
Or is there a need for another logical decoding plug-in for "WIP: Data at rest encryption"?
Regards.
Antonin Houska <ah@cybertec.at> wrote:
> Ants Aasma <ants.aasma@gmail.com> wrote:
>
> > Attached to this mail is a work in progress patch that adds an
> > extensible encryption mechanism. There are some loose ends left to tie
> > up, but the general concept and architecture is at a point where it's
> > ready for some feedback, fresh ideas and bikeshedding.
>
> Rebased patch is attached here, in case it helps to achieve (some of) the
> goals mentioned in the related thread [1].
>
> Besides encrypting table and WAL pages, it encrypts the temporary files
> (buffile.c), data stored during logical decoding (reorderbuffer.c) and
> statistics temporary files (pgstat.c). Unlike the previous version, SLRU files
> (e.g. CLOG) are not encrypted (it does not seem critical and the encryption
> makes torn page write quite difficult to handle).
>
> Another difference is that we use the OpenSSL of the (tweaked) AES XTS cipher
> now.
>
> Binary upgrade from unencrypted to encrypted cluster is not implemented yet.
>
>
> [1] https://www.postgresql.org/message-id/031401d3f41d$5c70ed90$1552c8b0$@lab.ntt.co.jp
>
> --
> Antonin Houska
> Cybertec Schonig & Schonig GmbH
> Grohrmuhlgasse 26, A-2700 Wiener Neustadt
> Web: https://www.cybertec-postgresql.com
>
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: [HACKERS] WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2018-06-27 08:02 ` Re: [HACKERS] WIP: Data at rest encryption Antonin Houska <ah@cybertec.at>
2018-07-13 07:56 ` Re: [HACKERS] WIP: Data at rest encryption Toshi Harada <harada.toshi@po.ntt-tx.co.jp>
@ 2018-07-25 11:47 ` Antonin Houska <ah@cybertec.at>
0 siblings, 0 replies; 93+ messages in thread
From: Antonin Houska @ 2018-07-25 11:47 UTC (permalink / raw)
To: Toshi Harada <harada.toshi@po.ntt-tx.co.jp>; +Cc: pgsql-hackers
Toshi Harada <harada.toshi@po.ntt-tx.co.jp> wrote:
> I am interested in a patch of "WIP: Data at rest encryption".
> This patch("data-at-rest-encryption-wip-2018.06.27.patch") is applied to PostgreSQL 11-beta 2 and it is running.
>
> In the explanation of this patch, since "data stored during logical decoding" is written,
> we tried logical decoding by the test_decoding module, but the following error occurs when creating a slot.
>
>
> pgbench_db=# SELECT * FROM pg_create_logical_replication_slot('my_slot', 'test_decoding');
> ERROR: invalid magic number B419 in log segment 000000020000000000000010, offset 0
> pgbench_db=#
I could not reproduce this error ...
> (Also, if you run "CREATE SUNSCRIPTION" for logical replication from another server, a similar error will occur.)
... but I succeeded for this. The problem was that a separate XLOG reader
callback is used for logical replication and that did not decrypt the XLOG
page. Fixed in the new version of the patch. Do you still see
pg_create_logical_replication_slot() failing?
> Question.
> In "data-at-rest-encryption-wip-2018.06.27.patch", is logical decoding still not implemented?
> Or is there a need for another logical decoding plug-in for "WIP: Data at rest encryption"?
No, there's nothing special about logical decoding.
--
Antonin Houska
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26, A-2700 Wiener Neustadt
Web: https://www.cybertec-postgresql.com
Attachments:
data-at-rest-encryption-wip-2018.07.25.patch (text/x-diff, 206.4K) download
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 493f1db7b9..0b05b761f3 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -54,6 +54,7 @@
#include "replication/walreceiver.h"
#include "replication/walsender.h"
#include "storage/bufmgr.h"
+#include "storage/encryption.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/large_object.h"
@@ -77,6 +78,7 @@
#include "pg_trace.h"
extern uint32 bootstrap_data_checksum_version;
+extern char *bootstrap_encryption_sample;
/* File path names (all relative to $PGDATA) */
#define RECOVERY_COMMAND_FILE "recovery.conf"
@@ -868,6 +870,7 @@ static void LocalSetXLogInsertAllowed(void);
static void CreateEndOfRecoveryRecord(void);
static void CheckPointGuts(XLogRecPtr checkPointRedo, int flags);
static void KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo);
+static void XLogWritePages(char *from, int npages);
static XLogRecPtr XLogGetReplicationSlotMinimumLSN(void);
static void AdvanceXLInsertBuffer(XLogRecPtr upto, bool opportunistic);
@@ -1240,7 +1243,7 @@ ReserveXLogInsertLocation(int size, XLogRecPtr *StartPos, XLogRecPtr *EndPos,
uint64 endbytepos;
uint64 prevbytepos;
- size = MAXALIGN(size);
+ size = XLOG_REC_ALIGN(size);
/* All (non xlog-switch) records should contain data. */
Assert(size > SizeOfXLogRecord);
@@ -1294,7 +1297,7 @@ ReserveXLogSwitch(XLogRecPtr *StartPos, XLogRecPtr *EndPos, XLogRecPtr *PrevPtr)
uint64 startbytepos;
uint64 endbytepos;
uint64 prevbytepos;
- uint32 size = MAXALIGN(SizeOfXLogRecord);
+ uint32 size = XLOG_REC_ALIGN(SizeOfXLogRecord);
XLogRecPtr ptr;
uint32 segleft;
@@ -1590,7 +1593,7 @@ CopyXLogRecordToWAL(int write_len, bool isLogSwitch, XLogRecData *rdata,
else
{
/* Align the end position, so that the next record starts aligned */
- CurrPos = MAXALIGN64(CurrPos);
+ CurrPos = XLOG_REC_ALIGN(CurrPos);
}
if (CurrPos != EndPos)
@@ -2474,8 +2477,6 @@ XLogWrite(XLogwrtRqst WriteRqst, bool flexible)
{
char *from;
Size nbytes;
- Size nleft;
- int written;
/* Need to seek in the file? */
if (openLogOff != startoffset)
@@ -2492,30 +2493,40 @@ XLogWrite(XLogwrtRqst WriteRqst, bool flexible)
/* OK to write the page(s) */
from = XLogCtl->pages + startidx * (Size) XLOG_BLCKSZ;
nbytes = npages * (Size) XLOG_BLCKSZ;
- nleft = nbytes;
- do
+ if (data_encrypted)
{
- errno = 0;
- pgstat_report_wait_start(WAIT_EVENT_WAL_WRITE);
- written = write(openLogFile, from, nleft);
- pgstat_report_wait_end();
- if (written <= 0)
+#ifdef USE_ENCRYPTION
+ int i;
+
+ /*
+ * XXX: use larger encryption buffer to enable larger writes
+ * and reduce number of syscalls?
+ */
+ for (i = 0; i < npages; i++)
{
- if (errno == EINTR)
- continue;
- ereport(PANIC,
- (errcode_for_file_access(),
- errmsg("could not write to log file %s "
- "at offset %u, length %zu: %m",
- XLogFileNameP(ThisTimeLineID, openLogSegNo),
- openLogOff, nbytes)));
- }
- nleft -= written;
- from += written;
- } while (nleft > 0);
+ char buf[XLOG_BLCKSZ];
+ char tweak[TWEAK_SIZE];
+
+ XLogEncryptionTweak(tweak, openLogSegNo, openLogOff);
+ encrypt_block(from, buf, XLOG_BLCKSZ, tweak);
+ pgstat_report_wait_start(WAIT_EVENT_WAL_WRITE);
+ XLogWritePages(buf, 1);
+ pgstat_report_wait_end();
+
+ from += XLOG_BLCKSZ;
+ openLogOff += XLOG_BLCKSZ;
+ }
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+ else
+ {
+ XLogWritePages(from, npages);
+ openLogOff += nbytes;
+ }
/* Update state for write */
- openLogOff += nbytes;
npages = 0;
/*
@@ -2631,6 +2642,32 @@ XLogWrite(XLogwrtRqst WriteRqst, bool flexible)
}
}
+static void
+XLogWritePages(char *from, int npages)
+{
+ Size nleft = npages * (Size) XLOG_BLCKSZ;
+ Size written;
+
+ do
+ {
+ errno = 0;
+ written = write(openLogFile, from, nleft);
+ if (written <= 0)
+ {
+ if (errno == EINTR)
+ continue;
+ ereport(PANIC,
+ (errcode_for_file_access(),
+ errmsg("could not write to log file %s "
+ "at offset %u, length %zu: %m",
+ XLogFileNameP(ThisTimeLineID, openLogSegNo),
+ openLogOff, npages * (Size) XLOG_BLCKSZ)));
+ }
+ nleft -= written;
+ from += written;
+ } while (nleft > 0);
+}
+
/*
* Record the LSN for an asynchronous transaction commit/abort
* and nudge the WALWriter if there is work for it to do.
@@ -4735,6 +4772,31 @@ ReadControlFile(void)
ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("\"max_wal_size\" must be at least twice \"wal_segment_size\".")));
+ /*
+ * Initialize encryption, but not if the current backend has already done
+ * that.
+ */
+ if (ControlFile->data_encrypted && !data_encrypted)
+ {
+ char sample[ENCRYPTION_SAMPLE_SIZE];
+
+ setup_encryption(false);
+
+ memset(sample, 0, ENCRYPTION_SAMPLE_SIZE);
+ sample_encryption(sample);
+ if (memcmp(ControlFile->encryption_verification, sample, ENCRYPTION_SAMPLE_SIZE))
+ ereport(FATAL,
+ (errmsg("invalid encryption key"),
+ errdetail("The passed encryption key does not match"
+ " database encryption key.")));
+ }
+ SetConfigOption("data_encryption", DataEncryptionEnabled() ? "yes" : "no",
+ PGC_INTERNAL, PGC_S_OVERRIDE);
+
+ /*
+ * This calculation relies on data_encryption (in particular the header
+ * sizes do), so we could not do it earlier.
+ */
UsableBytesInSegment =
(wal_segment_size / XLOG_BLCKSZ * UsableBytesInPage) -
(SizeOfXLogLongPHD - SizeOfXLogShortPHD);
@@ -4824,6 +4886,16 @@ DataChecksumsEnabled(void)
}
/*
+ * Is this cluster encrypted?
+ */
+bool
+DataEncryptionEnabled(void)
+{
+ Assert(ControlFile != NULL);
+ return (ControlFile->data_encrypted);
+}
+
+/*
* Returns a fake LSN for unlogged relations.
*
* Each call generates an LSN that is greater than any previous value
@@ -5202,6 +5274,18 @@ BootStrapXLOG(void)
use_existent = false;
openLogFile = XLogFileInit(1, &use_existent, false);
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ char tweak[TWEAK_SIZE];
+
+ XLogEncryptionTweak(tweak, 1, 0);
+ encrypt_block((char *) page, (char *) page, XLOG_BLCKSZ, tweak);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
/* Write the first page with the initial record */
errno = 0;
pgstat_report_wait_start(WAIT_EVENT_WAL_BOOTSTRAP_WRITE);
@@ -5251,6 +5335,22 @@ BootStrapXLOG(void)
ControlFile->wal_log_hints = wal_log_hints;
ControlFile->track_commit_timestamp = track_commit_timestamp;
ControlFile->data_checksum_version = bootstrap_data_checksum_version;
+ ControlFile->data_encrypted = data_encrypted;
+
+ if (data_encrypted)
+ {
+ char *sample;
+
+ sample = palloc0(ENCRYPTION_SAMPLE_SIZE);
+ sample_encryption(sample);
+
+ memcpy(ControlFile->encryption_verification, sample,
+ ENCRYPTION_SAMPLE_SIZE);
+ pfree(sample);
+ }
+ else
+ memset(ControlFile->encryption_verification, 0,
+ ENCRYPTION_SAMPLE_SIZE);
/* some additional ControlFile fields are set in WriteControlFile() */
@@ -11838,6 +11938,18 @@ retry:
Assert(targetPageOff == readOff);
Assert(reqLen <= readLen);
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ char tweak[TWEAK_SIZE];
+
+ XLogEncryptionTweak(tweak, readSegNo, readOff);
+ decrypt_block(readBuf, readBuf, XLOG_BLCKSZ, tweak);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
*readTLI = curFileTLI;
/*
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 5bea073a2b..148d436632 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -28,6 +28,7 @@
#include "miscadmin.h"
#include "replication/origin.h"
#include "storage/bufmgr.h"
+#include "storage/encryption.h"
#include "storage/proc.h"
#include "utils/memutils.h"
#include "pg_trace.h"
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 4c633c6c49..76b3bf7eab 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -24,6 +24,7 @@
#include "catalog/pg_control.h"
#include "common/pg_lzcompress.h"
#include "replication/origin.h"
+#include "storage/encryption.h"
#ifndef FRONTEND
#include "utils/memutils.h"
@@ -317,9 +318,9 @@ XLogReadRecord(XLogReaderState *state, XLogRecPtr RecPtr, char **errormsg)
*
* NB: Even though we use an XLogRecord pointer here, the whole record
* header might not fit on this page. xl_tot_len is the first field of the
- * struct, so it must be on this page (the records are MAXALIGNed), but we
- * cannot access any other fields until we've verified that we got the
- * whole header.
+ * struct, so it must be on this page (the records are XLOG_REC_ALIGNed)
+ * but we cannot access any other fields until we've verified that we got
+ * the whole header.
*/
record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ);
total_len = record->xl_tot_len;
@@ -461,8 +462,8 @@ XLogReadRecord(XLogReaderState *state, XLogRecPtr RecPtr, char **errormsg)
pageHeaderSize = XLogPageHeaderSize((XLogPageHeader) state->readBuf);
state->ReadRecPtr = RecPtr;
- state->EndRecPtr = targetPagePtr + pageHeaderSize
- + MAXALIGN(pageHeader->xlp_rem_len);
+ state->EndRecPtr = targetPagePtr + pageHeaderSize;
+ state->EndRecPtr += XLOG_REC_ALIGN(pageHeader->xlp_rem_len);
}
else
{
@@ -476,7 +477,7 @@ XLogReadRecord(XLogReaderState *state, XLogRecPtr RecPtr, char **errormsg)
if (!ValidXLogRecord(state, record, RecPtr))
goto err;
- state->EndRecPtr = RecPtr + MAXALIGN(total_len);
+ state->EndRecPtr = RecPtr + XLOG_REC_ALIGN(total_len);
state->ReadRecPtr = RecPtr;
memcpy(state->readRecordBuf, record, total_len);
@@ -532,6 +533,15 @@ ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen)
Assert((pageptr % XLOG_BLCKSZ) == 0);
+ /*
+ * Make sure that we only return data that can be decrypted in a sensible
+ * way. If the valid data ended in the middle of encryption block, then
+ * decryption of that last block would turn the contained data into
+ * garbage.
+ */
+ if (data_encrypted)
+ reqLen = XLOG_REC_ALIGN(reqLen);
+
XLByteToSeg(pageptr, targetSegNo, state->wal_segment_size);
targetPageOff = XLogSegmentOffset(pageptr, state->wal_segment_size);
@@ -954,9 +964,10 @@ XLogFindNextRecord(XLogReaderState *state, XLogRecPtr RecPtr)
* next page header will contain the remaining length of the
* continuation data
*
- * Note that record headers are MAXALIGN'ed
+ * Note that record headers are XLOG_REC_ALIGN'ed
*/
- if (MAXALIGN(header->xlp_rem_len) > (XLOG_BLCKSZ - pageHeaderSize))
+ if (XLOG_REC_ALIGN(header->xlp_rem_len) >
+ (XLOG_BLCKSZ - pageHeaderSize))
tmpRecPtr = targetPagePtr + XLOG_BLCKSZ;
else
{
@@ -965,7 +976,7 @@ XLogFindNextRecord(XLogReaderState *state, XLogRecPtr RecPtr)
* tmpRecPtr to point to the first valid record
*/
tmpRecPtr = targetPagePtr + pageHeaderSize
- + MAXALIGN(header->xlp_rem_len);
+ + XLOG_REC_ALIGN(header->xlp_rem_len);
break;
}
}
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index 4ecdc9220f..aad05de2ef 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -25,6 +25,7 @@
#include "access/xlogutils.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "storage/encryption.h"
#include "storage/smgr.h"
#include "utils/guc.h"
#include "utils/hsearch.h"
@@ -653,12 +654,15 @@ XLogTruncateRelation(RelFileNode rnode, ForkNumber forkNum,
* frontend). Probably these should be merged at some point.
*/
static void
-XLogRead(char *buf, int segsize, TimeLineID tli, XLogRecPtr startptr,
- Size count)
+XLogRead(char *buf, int segsize, TimeLineID tli, XLogRecPtr startptr, Size count)
{
- char *p;
XLogRecPtr recptr;
Size nbytes;
+#ifdef USE_ENCRYPTION
+ char *decrypt_p;
+ uint32 decryptOff;
+#endif
+ char *p;
/* state maintained across calls */
static int sendFile = -1;
@@ -669,6 +673,9 @@ XLogRead(char *buf, int segsize, TimeLineID tli, XLogRecPtr startptr,
Assert(segsize == wal_segment_size);
p = buf;
+#ifdef USE_ENCRYPTION
+ decrypt_p = p;
+#endif
recptr = startptr;
nbytes = count;
@@ -679,6 +686,9 @@ XLogRead(char *buf, int segsize, TimeLineID tli, XLogRecPtr startptr,
int readbytes;
startoff = XLogSegmentOffset(recptr, segsize);
+#ifdef USE_ENCRYPTION
+ decryptOff = startoff;
+#endif
/* Do we need to switch to a different xlog segment? */
if (sendFile < 0 || !XLByteInSeg(recptr, sendSegNo, segsize) ||
@@ -758,6 +768,25 @@ XLogRead(char *buf, int segsize, TimeLineID tli, XLogRecPtr startptr,
sendOff += readbytes;
nbytes -= readbytes;
p += readbytes;
+
+ /* Decrypt completed blocks */
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ while (decrypt_p + XLOG_BLCKSZ <= p)
+ {
+ char tweak[TWEAK_SIZE];
+
+ XLogEncryptionTweak(tweak, sendSegNo, decryptOff);
+ decrypt_block(decrypt_p, decrypt_p, XLOG_BLCKSZ, tweak);
+
+ decrypt_p += XLOG_BLCKSZ;
+ decryptOff += XLOG_BLCKSZ;
+ }
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
}
}
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index cdd71a9bc3..daf927ddc8 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -23,6 +23,7 @@
#include "bootstrap/bootstrap.h"
#include "catalog/index.h"
#include "catalog/pg_collation.h"
+#include "catalog/pg_control.h"
#include "catalog/pg_type.h"
#include "libpq/pqsignal.h"
#include "miscadmin.h"
@@ -36,6 +37,7 @@
#include "storage/bufmgr.h"
#include "storage/bufpage.h"
#include "storage/condition_variable.h"
+#include "storage/encryption.h"
#include "storage/ipc.h"
#include "storage/proc.h"
#include "tcop/tcopprot.h"
@@ -49,7 +51,6 @@
uint32 bootstrap_data_checksum_version = 0; /* No checksum */
-
#define ALLOC(t, c) \
((t *) MemoryContextAllocZero(TopMemoryContext, (unsigned)(c) * sizeof(t)))
@@ -223,7 +224,7 @@ AuxiliaryProcessMain(int argc, char *argv[])
/* If no -x argument, we are a CheckerProcess */
MyAuxProcType = CheckerProcess;
- while ((flag = getopt(argc, argv, "B:c:d:D:Fkr:x:X:-:")) != -1)
+ while ((flag = getopt(argc, argv, "B:c:d:D:FkK:r:x:X:-:")) != -1)
{
switch (flag)
{
@@ -249,6 +250,25 @@ AuxiliaryProcessMain(int argc, char *argv[])
case 'F':
SetConfigOption("fsync", "false", PGC_POSTMASTER, PGC_S_ARGV);
break;
+#ifdef USE_OPENSSL
+ case 'K':
+ encryption_key_command = strdup(optarg);
+
+ /*
+ * When auxiliary process (typically bootstrap) starts, the
+ * control file might not exist yet. In this case we also use
+ * encryption_key_command to indicate that the encryption is
+ * enabled.
+ *
+ * Postmaster should not set this variable. Instead, it just
+ * sets data_encrypted according to the control file and child
+ * processes inherit that.
+ */
+ Assert(!IsUnderPostmaster);
+ data_encrypted = true;
+
+ break;
+#endif /* USE_OPENSSL */
case 'k':
bootstrap_data_checksum_version = PG_DATA_CHECKSUM_VERSION;
break;
@@ -370,6 +390,20 @@ AuxiliaryProcessMain(int argc, char *argv[])
if (!IsUnderPostmaster)
InitializeMaxBackends();
+ /*
+ * If data_encryption is set because of command line argument, do the
+ * setup now. (If set by postmaster, postmaster should have performed the
+ * setup.)
+ *
+ * This should only be useful for the bootstrap process. Anyone else
+ * initializes the encryption via ReadControlFile().
+ */
+ if (data_encrypted && MyAuxProcType == BootstrapProcess)
+ {
+ Assert(!IsUnderPostmaster);
+ setup_encryption(true);
+ }
+
BaseInit();
/*
diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c
index 5342f217c0..0645ac00ba 100644
--- a/src/backend/commands/dbcommands.c
+++ b/src/backend/commands/dbcommands.c
@@ -625,7 +625,12 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
*
* We don't need to copy subdirectories
*/
- copydir(srcpath, dstpath, false);
+ {
+ RelFileNode fromNode = {srctablespace, src_dboid, InvalidOid};
+ RelFileNode toNode = {dsttablespace, dboid, InvalidOid};
+
+ copydir(srcpath, dstpath, &fromNode, &toNode);
+ }
/* Record the filesystem change in XLOG */
{
@@ -1254,7 +1259,12 @@ movedb(const char *dbname, const char *tblspcname)
/*
* Copy files from the old tablespace to the new one
*/
- copydir(src_dbpath, dst_dbpath, false);
+ {
+ RelFileNode fromNode = {src_tblspcoid, db_id, InvalidOid};
+ RelFileNode toNode = {dst_tblspcoid, db_id, InvalidOid};
+
+ copydir(src_dbpath, dst_dbpath, &fromNode, &toNode);
+ }
/*
* Record the filesystem change in XLOG
@@ -2116,7 +2126,12 @@ dbase_redo(XLogReaderState *record)
*
* We don't need to copy subdirectories
*/
- copydir(src_path, dst_path, false);
+ {
+ RelFileNode fromNode = {xlrec->src_tablespace_id, xlrec->src_db_id, InvalidOid};
+ RelFileNode toNode = {xlrec->tablespace_id, xlrec->db_id, InvalidOid};
+
+ copydir(src_path, dst_path, &fromNode, &toNode);
+ }
}
else if (info == XLOG_DBASE_DROP)
{
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index bbe73618c7..aceb3a3f1d 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -42,6 +42,7 @@
#include "catalog/pg_database.h"
#include "catalog/pg_proc.h"
#include "common/ip.h"
+#include "lib/stringinfo.h"
#include "libpq/libpq.h"
#include "libpq/pqsignal.h"
#include "mb/pg_wchar.h"
@@ -52,6 +53,7 @@
#include "postmaster/postmaster.h"
#include "replication/walsender.h"
#include "storage/backendid.h"
+#include "storage/encryption.h"
#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
@@ -105,6 +107,10 @@
#define PGSTAT_TAB_HASH_SIZE 512
#define PGSTAT_FUNCTION_HASH_SIZE 512
+/*
+ * Size of a buffer used to read stats files from file.
+ */
+#define PGSTAT_FILE_BUFFER_SIZE 1024
/* ----------
* Total number of backends including auxiliary
@@ -309,6 +315,9 @@ static HTAB *pgstat_collect_oids(Oid catalogid);
static PgStat_TableStatus *get_tabstat_entry(Oid rel_id, bool isshared);
static void pgstat_setup_memcxt(void);
+static StringInfo pgstat_setup_serial_buffer(MemoryContext *context);
+static bool pgstat_read_file(StringInfo buf, const char *statfile,
+ FILE *fpin);
static const char *pgstat_get_wait_activity(WaitEventActivity w);
static const char *pgstat_get_wait_client(WaitEventClient w);
@@ -337,6 +346,12 @@ static void pgstat_recv_recoveryconflict(PgStat_MsgRecoveryConflict *msg, int le
static void pgstat_recv_deadlock(PgStat_MsgDeadlock *msg, int len);
static void pgstat_recv_tempfile(PgStat_MsgTempFile *msg, int len);
+static bool readFromStringInfo(StringInfo str, void *data, Size size);
+
+#ifdef USE_ENCRYPTION
+static void pgstat_encryption_tweak(char *tweak, Oid dbid, bool permament);
+#endif
+
/* ------------------------------------------------------------
* Public functions called from postmaster follow
* ------------------------------------------------------------
@@ -3784,6 +3799,15 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_DSM_FILL_ZERO_WRITE:
event_name = "DSMFillZeroWrite";
break;
+ case WAIT_EVENT_KDF_FILE_READ:
+ event_name = "KDFFileRead";
+ break;
+ case WAIT_EVENT_KDF_FILE_SYNC:
+ event_name = "KDFFileSync";
+ break;
+ case WAIT_EVENT_KDF_FILE_WRITE:
+ event_name = "KDFFileWrite";
+ break;
case WAIT_EVENT_LOCK_FILE_ADDTODATADIR_READ:
event_name = "LockFileAddToDataDirRead";
break;
@@ -4683,6 +4707,8 @@ pgstat_write_statsfiles(bool permanent, bool allDbs)
const char *tmpfile = permanent ? PGSTAT_STAT_PERMANENT_TMPFILE : pgstat_stat_tmpname;
const char *statfile = permanent ? PGSTAT_STAT_PERMANENT_FILENAME : pgstat_stat_filename;
int rc;
+ StringInfo buf;
+ MemoryContext serialContext;
elog(DEBUG2, "writing stats file \"%s\"", statfile);
@@ -4705,23 +4731,25 @@ pgstat_write_statsfiles(bool permanent, bool allDbs)
globalStats.stats_timestamp = GetCurrentTimestamp();
/*
+ * Memory for data serialization.
+ */
+ buf = pgstat_setup_serial_buffer(&serialContext);
+
+ /*
* Write the file header --- currently just a format ID.
*/
format_id = PGSTAT_FILE_FORMAT_ID;
- rc = fwrite(&format_id, sizeof(format_id), 1, fpout);
- (void) rc; /* we'll check for error with ferror */
+ appendBinaryStringInfo(buf, (char *) &format_id, sizeof(format_id));
/*
* Write global stats struct
*/
- rc = fwrite(&globalStats, sizeof(globalStats), 1, fpout);
- (void) rc; /* we'll check for error with ferror */
+ appendBinaryStringInfo(buf, (char *) &globalStats, sizeof(globalStats));
/*
* Write archiver stats struct
*/
- rc = fwrite(&archiverStats, sizeof(archiverStats), 1, fpout);
- (void) rc; /* we'll check for error with ferror */
+ appendBinaryStringInfo(buf, (char *) &archiverStats, sizeof(archiverStats));
/*
* Walk through the database table.
@@ -4745,18 +4773,53 @@ pgstat_write_statsfiles(bool permanent, bool allDbs)
* Write out the DB entry. We don't write the tables or functions
* pointers, since they're of no use to any other process.
*/
- fputc('D', fpout);
- rc = fwrite(dbentry, offsetof(PgStat_StatDBEntry, tables), 1, fpout);
- (void) rc; /* we'll check for error with ferror */
+ appendBinaryStringInfo(buf, (char *) "D", 1);
+ appendBinaryStringInfo(buf, (char *) dbentry,
+ offsetof(PgStat_StatDBEntry, tables));
+ }
+
+ /*
+ * No more output to be done.
+ */
+ appendBinaryStringInfo(buf, (char *) "E", 1);
+
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ char tweak[TWEAK_SIZE];
+
+ /*
+ * Make sure the data is aligned to ENCRYPTION_BLOCK.
+ */
+ if (buf->len % ENCRYPTION_BLOCK > 0)
+ {
+ char zerobuf[ENCRYPTION_BLOCK];
+
+ memset(zerobuf, 0, ENCRYPTION_BLOCK);
+ appendBinaryStringInfo(buf,
+ zerobuf,
+ ENCRYPTION_BLOCK - buf->len % ENCRYPTION_BLOCK);
+ }
+
+ pgstat_encryption_tweak(tweak, InvalidOid, permanent);
+ encrypt_block(buf->data, buf->data, buf->len, tweak);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
}
/*
- * No more output to be done. Close the temp file and replace the old
- * pgstat.stat with it. The ferror() check replaces testing for error
- * after each individual fputc or fwrite above.
+ * Write the data to the file.
*/
- fputc('E', fpout);
+ rc = fwrite(buf->data, buf->len, 1, fpout);
+ (void) rc; /* we'll check for error with ferror */
+
+ /* Free the temporary storage. */
+ MemoryContextResetOnly(serialContext);
+ /*
+ * Close the temp file and replace the old pgstat.stat with it.
+ */
if (ferror(fpout))
{
ereport(LOG,
@@ -4837,6 +4900,8 @@ pgstat_write_db_statsfile(PgStat_StatDBEntry *dbentry, bool permanent)
int rc;
char tmpfile[MAXPGPATH];
char statfile[MAXPGPATH];
+ StringInfo buf;
+ MemoryContext serialContext;
get_dbstat_filename(permanent, true, dbid, tmpfile, MAXPGPATH);
get_dbstat_filename(permanent, false, dbid, statfile, MAXPGPATH);
@@ -4857,11 +4922,15 @@ pgstat_write_db_statsfile(PgStat_StatDBEntry *dbentry, bool permanent)
}
/*
+ * Memory for data serialization.
+ */
+ buf = pgstat_setup_serial_buffer(&serialContext);
+
+ /*
* Write the file header --- currently just a format ID.
*/
format_id = PGSTAT_FILE_FORMAT_ID;
- rc = fwrite(&format_id, sizeof(format_id), 1, fpout);
- (void) rc; /* we'll check for error with ferror */
+ appendBinaryStringInfo(buf, (char *) &format_id, sizeof(format_id));
/*
* Walk through the database's access stats per table.
@@ -4869,9 +4938,8 @@ pgstat_write_db_statsfile(PgStat_StatDBEntry *dbentry, bool permanent)
hash_seq_init(&tstat, dbentry->tables);
while ((tabentry = (PgStat_StatTabEntry *) hash_seq_search(&tstat)) != NULL)
{
- fputc('T', fpout);
- rc = fwrite(tabentry, sizeof(PgStat_StatTabEntry), 1, fpout);
- (void) rc; /* we'll check for error with ferror */
+ appendBinaryStringInfo(buf, (char *) "T", 1);
+ appendBinaryStringInfo(buf, (char *) tabentry, sizeof(PgStat_StatTabEntry));
}
/*
@@ -4880,18 +4948,52 @@ pgstat_write_db_statsfile(PgStat_StatDBEntry *dbentry, bool permanent)
hash_seq_init(&fstat, dbentry->functions);
while ((funcentry = (PgStat_StatFuncEntry *) hash_seq_search(&fstat)) != NULL)
{
- fputc('F', fpout);
- rc = fwrite(funcentry, sizeof(PgStat_StatFuncEntry), 1, fpout);
- (void) rc; /* we'll check for error with ferror */
+ appendBinaryStringInfo(buf, (char *) "F", 1);
+ appendBinaryStringInfo(buf, (char *) funcentry, sizeof(PgStat_StatFuncEntry));
}
/*
- * No more output to be done. Close the temp file and replace the old
- * pgstat.stat with it. The ferror() check replaces testing for error
- * after each individual fputc or fwrite above.
+ * No more output to be done.
*/
- fputc('E', fpout);
+ appendBinaryStringInfo(buf, (char *) "E", 1);
+
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ char tweak[TWEAK_SIZE];
+
+ /*
+ * Make sure the data is aligned to ENCRYPTION_BLOCK.
+ */
+ if (buf->len % ENCRYPTION_BLOCK > 0)
+ {
+ char zerobuf[ENCRYPTION_BLOCK];
+ memset(zerobuf, 0, ENCRYPTION_BLOCK);
+ appendBinaryStringInfo(buf,
+ zerobuf,
+ ENCRYPTION_BLOCK - buf->len % ENCRYPTION_BLOCK);
+ }
+
+ pgstat_encryption_tweak(tweak, dbid, permanent);
+ encrypt_block(buf->data, buf->data, buf->len, tweak);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
+ /*
+ * Write the data to the file.
+ */
+ rc = fwrite(buf->data, buf->len, 1, fpout);
+ (void) rc; /* we'll check for error with ferror */
+
+ /* Free the temporary storage. */
+ MemoryContextResetOnly(serialContext);
+
+ /*
+ * Close the temp file and replace the old pgstat.stat with it.
+ */
if (ferror(fpout))
{
ereport(LOG,
@@ -4958,6 +5060,8 @@ pgstat_read_statsfiles(Oid onlydb, bool permanent, bool deep)
int32 format_id;
bool found;
const char *statfile = permanent ? PGSTAT_STAT_PERMANENT_FILENAME : pgstat_stat_filename;
+ StringInfo buf;
+ MemoryContext serialContext;
/*
* The tables will live in pgStatLocalContext.
@@ -5008,9 +5112,38 @@ pgstat_read_statsfiles(Oid onlydb, bool permanent, bool deep)
}
/*
+ * Memory for data serialization.
+ */
+ buf = pgstat_setup_serial_buffer(&serialContext);
+
+ /*
+ * Read the data into memory.
+ */
+ if (!pgstat_read_file(buf, statfile, fpin))
+ goto done;
+
+ /*
+ * Decrypt the data if it's encrypted.
+ */
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ char tweak[TWEAK_SIZE];
+
+ pgstat_encryption_tweak(tweak, InvalidOid, permanent);
+ decrypt_block(buf->data, buf->data, buf->len, tweak);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
+ /* Prepare for reading. */
+ buf->cursor = 0;
+
+ /*
* Verify it's of the expected format.
*/
- if (fread(&format_id, 1, sizeof(format_id), fpin) != sizeof(format_id) ||
+ if (!readFromStringInfo(buf, &format_id, sizeof(format_id)) ||
format_id != PGSTAT_FILE_FORMAT_ID)
{
ereport(pgStatRunningInCollector ? LOG : WARNING,
@@ -5021,7 +5154,7 @@ pgstat_read_statsfiles(Oid onlydb, bool permanent, bool deep)
/*
* Read global stats struct
*/
- if (fread(&globalStats, 1, sizeof(globalStats), fpin) != sizeof(globalStats))
+ if (!readFromStringInfo(buf, &globalStats, sizeof(globalStats)))
{
ereport(pgStatRunningInCollector ? LOG : WARNING,
(errmsg("corrupted statistics file \"%s\"", statfile)));
@@ -5042,7 +5175,7 @@ pgstat_read_statsfiles(Oid onlydb, bool permanent, bool deep)
/*
* Read archiver stats struct
*/
- if (fread(&archiverStats, 1, sizeof(archiverStats), fpin) != sizeof(archiverStats))
+ if (!readFromStringInfo(buf, &archiverStats, sizeof(archiverStats)))
{
ereport(pgStatRunningInCollector ? LOG : WARNING,
(errmsg("corrupted statistics file \"%s\"", statfile)));
@@ -5056,15 +5189,24 @@ pgstat_read_statsfiles(Oid onlydb, bool permanent, bool deep)
*/
for (;;)
{
- switch (fgetc(fpin))
+ char kind;
+
+ if (!readFromStringInfo(buf, &kind, 1))
+ {
+ ereport(pgStatRunningInCollector ? LOG : WARNING,
+ (errmsg("corrupted statistics file \"%s\"", statfile)));
+ goto done;
+ }
+
+ switch (kind)
{
/*
* 'D' A PgStat_StatDBEntry struct describing a database
* follows.
*/
case 'D':
- if (fread(&dbbuf, 1, offsetof(PgStat_StatDBEntry, tables),
- fpin) != offsetof(PgStat_StatDBEntry, tables))
+ if (!readFromStringInfo(buf, &dbbuf,
+ offsetof(PgStat_StatDBEntry, tables)))
{
ereport(pgStatRunningInCollector ? LOG : WARNING,
(errmsg("corrupted statistics file \"%s\"",
@@ -5154,6 +5296,9 @@ pgstat_read_statsfiles(Oid onlydb, bool permanent, bool deep)
done:
FreeFile(fpin);
+ /* Free the temporary storage. */
+ MemoryContextResetOnly(serialContext);
+
/* If requested to read the permanent file, also get rid of it. */
if (permanent)
{
@@ -5191,6 +5336,8 @@ pgstat_read_db_statsfile(Oid databaseid, HTAB *tabhash, HTAB *funchash,
int32 format_id;
bool found;
char statfile[MAXPGPATH];
+ StringInfo buf;
+ MemoryContext serialContext;
get_dbstat_filename(permanent, false, databaseid, statfile, MAXPGPATH);
@@ -5214,9 +5361,39 @@ pgstat_read_db_statsfile(Oid databaseid, HTAB *tabhash, HTAB *funchash,
}
/*
+ * Memory for data serialization.
+ */
+ buf = pgstat_setup_serial_buffer(&serialContext);
+
+ /*
+ * Read the data into memory.
+ */
+
+ if (!pgstat_read_file(buf, statfile, fpin))
+ goto done;
+
+ /*
+ * Decrypt the data if it's encrypted.
+ */
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ char tweak[TWEAK_SIZE];
+
+ pgstat_encryption_tweak(tweak, databaseid, permanent);
+ decrypt_block(buf->data, buf->data, buf->len, tweak);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
+ /* Prepare for reading. */
+ buf->cursor = 0;
+
+ /*
* Verify it's of the expected format.
*/
- if (fread(&format_id, 1, sizeof(format_id), fpin) != sizeof(format_id) ||
+ if (!readFromStringInfo(buf, &format_id, sizeof(format_id)) ||
format_id != PGSTAT_FILE_FORMAT_ID)
{
ereport(pgStatRunningInCollector ? LOG : WARNING,
@@ -5230,14 +5407,22 @@ pgstat_read_db_statsfile(Oid databaseid, HTAB *tabhash, HTAB *funchash,
*/
for (;;)
{
- switch (fgetc(fpin))
+ char kind;
+
+ if (!readFromStringInfo(buf, &kind, 1))
+ {
+ ereport(pgStatRunningInCollector ? LOG : WARNING,
+ (errmsg("corrupted statistics file \"%s\"", statfile)));
+ goto done;
+ }
+
+ switch (kind)
{
/*
* 'T' A PgStat_StatTabEntry follows.
*/
case 'T':
- if (fread(&tabbuf, 1, sizeof(PgStat_StatTabEntry),
- fpin) != sizeof(PgStat_StatTabEntry))
+ if (!readFromStringInfo(buf, &tabbuf, sizeof(PgStat_StatTabEntry)))
{
ereport(pgStatRunningInCollector ? LOG : WARNING,
(errmsg("corrupted statistics file \"%s\"",
@@ -5270,8 +5455,8 @@ pgstat_read_db_statsfile(Oid databaseid, HTAB *tabhash, HTAB *funchash,
* 'F' A PgStat_StatFuncEntry follows.
*/
case 'F':
- if (fread(&funcbuf, 1, sizeof(PgStat_StatFuncEntry),
- fpin) != sizeof(PgStat_StatFuncEntry))
+ if (!readFromStringInfo(buf, &funcbuf,
+ sizeof(PgStat_StatFuncEntry)))
{
ereport(pgStatRunningInCollector ? LOG : WARNING,
(errmsg("corrupted statistics file \"%s\"",
@@ -5317,6 +5502,9 @@ pgstat_read_db_statsfile(Oid databaseid, HTAB *tabhash, HTAB *funchash,
done:
FreeFile(fpin);
+ /* Free the temporary storage. */
+ MemoryContextResetOnly(serialContext);
+
if (permanent)
{
elog(DEBUG2, "removing permanent stats file \"%s\"", statfile);
@@ -5351,6 +5539,8 @@ pgstat_read_db_statsfile_timestamp(Oid databaseid, bool permanent,
FILE *fpin;
int32 format_id;
const char *statfile = permanent ? PGSTAT_STAT_PERMANENT_FILENAME : pgstat_stat_filename;
+ StringInfo buf;
+ MemoryContext serialContext;
/*
* Try to open the stats file. As above, anything but ENOENT is worthy of
@@ -5367,38 +5557,68 @@ pgstat_read_db_statsfile_timestamp(Oid databaseid, bool permanent,
}
/*
+ * Memory for data serialization.
+ */
+ buf = pgstat_setup_serial_buffer(&serialContext);
+
+ /*
+ * Read the data into memory.
+ */
+ if (!pgstat_read_file(buf, statfile, fpin))
+ goto done;
+
+ /*
+ * Decrypt the data if it's encrypted.
+ */
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ char tweak[TWEAK_SIZE];
+
+ pgstat_encryption_tweak(tweak, InvalidOid, permanent);
+ decrypt_block(buf->data, buf->data, buf->len, tweak);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
+ /* Prepare for reading. */
+ buf->cursor = 0;
+
+ /*
* Verify it's of the expected format.
*/
- if (fread(&format_id, 1, sizeof(format_id), fpin) != sizeof(format_id) ||
+ if (!readFromStringInfo(buf, &format_id, sizeof(format_id)) ||
format_id != PGSTAT_FILE_FORMAT_ID)
{
ereport(pgStatRunningInCollector ? LOG : WARNING,
(errmsg("corrupted statistics file \"%s\"", statfile)));
FreeFile(fpin);
+ MemoryContextResetOnly(serialContext);
return false;
}
/*
* Read global stats struct
*/
- if (fread(&myGlobalStats, 1, sizeof(myGlobalStats),
- fpin) != sizeof(myGlobalStats))
+ if (!readFromStringInfo(buf, &myGlobalStats, sizeof(myGlobalStats)))
{
ereport(pgStatRunningInCollector ? LOG : WARNING,
(errmsg("corrupted statistics file \"%s\"", statfile)));
FreeFile(fpin);
+ MemoryContextResetOnly(serialContext);
return false;
}
/*
* Read archiver stats struct
*/
- if (fread(&myArchiverStats, 1, sizeof(myArchiverStats),
- fpin) != sizeof(myArchiverStats))
+ if (!readFromStringInfo(buf, &myArchiverStats, sizeof(myArchiverStats)))
{
ereport(pgStatRunningInCollector ? LOG : WARNING,
(errmsg("corrupted statistics file \"%s\"", statfile)));
FreeFile(fpin);
+ MemoryContextResetOnly(serialContext);
return false;
}
@@ -5411,15 +5631,24 @@ pgstat_read_db_statsfile_timestamp(Oid databaseid, bool permanent,
*/
for (;;)
{
- switch (fgetc(fpin))
+ char kind;
+
+ if (!readFromStringInfo(buf, &kind, 1))
+ {
+ ereport(pgStatRunningInCollector ? LOG : WARNING,
+ (errmsg("corrupted statistics file \"%s\"", statfile)));
+ goto done;
+ }
+
+ switch (kind)
{
/*
* 'D' A PgStat_StatDBEntry struct describing a database
* follows.
*/
case 'D':
- if (fread(&dbentry, 1, offsetof(PgStat_StatDBEntry, tables),
- fpin) != offsetof(PgStat_StatDBEntry, tables))
+ if (!readFromStringInfo(buf, &dbentry,
+ offsetof(PgStat_StatDBEntry, tables)))
{
ereport(pgStatRunningInCollector ? LOG : WARNING,
(errmsg("corrupted statistics file \"%s\"",
@@ -5452,6 +5681,7 @@ pgstat_read_db_statsfile_timestamp(Oid databaseid, bool permanent,
done:
FreeFile(fpin);
+ MemoryContextResetOnly(serialContext);
return true;
}
@@ -5603,6 +5833,61 @@ pgstat_setup_memcxt(void)
ALLOCSET_SMALL_SIZES);
}
+/* ----------
+ * pgstat_setup_serial_buffer() -
+ *
+ * Allocate memory for data serialization and return the containing memory
+ * context.
+ *
+ * ----------
+ */
+static StringInfo
+pgstat_setup_serial_buffer(MemoryContext *context)
+{
+ MemoryContext oldCxt;
+ StringInfo result;
+
+ *context = AllocSetContextCreate(TopMemoryContext,
+ "Statistics encryption",
+ ALLOCSET_DEFAULT_SIZES);
+ oldCxt = MemoryContextSwitchTo(*context);
+ result = makeStringInfo();
+ MemoryContextSwitchTo(oldCxt);
+
+ return result;
+}
+
+/*
+ * Read statistics data from file into memory.
+ */
+static bool
+pgstat_read_file(StringInfo buf, const char *statfile, FILE *fpin)
+{
+ for (;;)
+ {
+ char bufread[PGSTAT_FILE_BUFFER_SIZE];
+ Size bytesread;
+
+ bytesread = fread(bufread, 1, PGSTAT_FILE_BUFFER_SIZE, fpin);
+ appendBinaryStringInfo(buf, bufread, bytesread);
+
+ if (bytesread < PGSTAT_FILE_BUFFER_SIZE)
+ break;
+ }
+
+ /*
+ * Check if the amount of data makes sense.
+ */
+ if (buf->len == 0 || (data_encrypted && buf->len % ENCRYPTION_BLOCK > 0))
+ {
+ ereport(pgStatRunningInCollector ? LOG : WARNING,
+ (errmsg("corrupted statistics file \"%s\"", statfile)));
+ return false;
+ }
+
+ return true;
+}
+
/* ----------
* pgstat_clear_snapshot() -
@@ -6312,6 +6597,35 @@ pgstat_write_statsfile_needed(void)
return false;
}
+/*
+ * Read "size" bytes into "data" from StringInfo.
+ *
+ * Return true iff there was enough data available.
+ *
+ * XXX Consider moving this function to stringinfo.c.
+ */
+static bool
+readFromStringInfo(StringInfo str, void *data, Size size)
+{
+ if (str->cursor + size > str->len)
+ return false;
+
+ memcpy(data, str->data + str->cursor, size);
+ str->cursor += size;
+
+ return true;
+}
+
+#ifdef USE_ENCRYPTION
+static void
+pgstat_encryption_tweak(char *tweak, Oid dbid, bool permanent)
+{
+ memset(tweak, 0, TWEAK_SIZE);
+ memcpy(tweak, &dbid, sizeof(Oid));
+ memcpy(tweak + sizeof(Oid), &permanent, sizeof(bool));
+}
+#endif /* USE_ENCRYPTION */
+
/* ----------
* pgstat_db_requested() -
*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index a4b53b33cd..907a2f5cf0 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -116,6 +116,7 @@
#include "postmaster/syslogger.h"
#include "replication/logicallauncher.h"
#include "replication/walsender.h"
+#include "storage/encryption.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/pg_shmem.h"
@@ -674,7 +675,7 @@ PostmasterMain(int argc, char *argv[])
* tcop/postgres.c (the option sets should not conflict) and with the
* common help() function in main/main.c.
*/
- while ((opt = getopt(argc, argv, "B:bc:C:D:d:EeFf:h:ijk:lN:nOo:Pp:r:S:sTt:W:-:")) != -1)
+ while ((opt = getopt(argc, argv, "B:bc:C:D:d:EeFf:h:ijK:k:lN:nOo:Pp:r:S:sTt:W:-:")) != -1)
{
switch (opt)
{
@@ -732,6 +733,12 @@ PostmasterMain(int argc, char *argv[])
/* only used by interactive backend */
break;
+#ifdef USE_OPENSSL
+ case 'K':
+ encryption_key_command = strdup(optarg);
+ break;
+#endif /* USE_OPENSSL */
+
case 'k':
SetConfigOption("unix_socket_directories", optarg, PGC_POSTMASTER, PGC_S_ARGV);
break;
@@ -1175,6 +1182,11 @@ PostmasterMain(int argc, char *argv[])
/*
* Set up shared memory and semaphores.
+ *
+ * This includes call of setup_encryption() as soon as we realize that
+ * "data_encrypted" field of the control file is set. The encryption must
+ * be initialized at the point so that "encryption_verification" field of
+ * the control file can be checked.
*/
reset_shared(PostPortNumber);
diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c
index 3f1eae38a9..c4fcb1e33f 100644
--- a/src/backend/replication/basebackup.c
+++ b/src/backend/replication/basebackup.c
@@ -1102,7 +1102,7 @@ sendDir(const char *path, int basepathlen, bool sizeonly, List *tablespaces,
/* Exclude all forks for unlogged tables except the init fork */
if (isDbDir &&
parse_filename_for_nontemp_relation(de->d_name, &relOidChars,
- &relForkNum))
+ &relForkNum, NULL))
{
/* Never exclude init forks */
if (relForkNum != INIT_FORKNUM)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 9b55b94227..8692c94f27 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -70,6 +70,7 @@
#include "replication/slot.h"
#include "replication/snapbuild.h" /* just for SnapBuildSnapDecRefcount */
#include "storage/bufmgr.h"
+#include "storage/encryption.h"
#include "storage/fd.h"
#include "storage/sinval.h"
#include "utils/builtins.h"
@@ -192,7 +193,7 @@ static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTX
static void ReorderBufferCheckSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
- int fd, ReorderBufferChange *change);
+ int fd, XLogSegNo segno, ReorderBufferChange *change);
static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn,
int *fd, XLogSegNo *segno);
static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
@@ -218,6 +219,20 @@ static void ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *t
Relation relation, ReorderBufferChange *change);
+/* ----------------------------------------------
+ * encryption / decryption of serialized changes
+ * ----------------------------------------------
+ */
+
+/*
+ * Compute encryption tweak for buffer change.
+ */
+#define REORDER_BUFFER_CHANGE_TWEAK(tweak, segno) \
+ StaticAssertStmt(sizeof(XLogSegNo) <= TWEAK_SIZE, \
+ "XLogSegNo does not fit into encryption tweak"); \
+ memset((tweak), 0, TWEAK_SIZE); \
+ memcpy((tweak), (char *) &segno, sizeof(XLogSegNo))
+
/*
* Allocate a new ReorderBuffer and clean out any old serialized state from
* prior ReorderBuffer instances for the same slot.
@@ -2257,7 +2272,7 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
errmsg("could not open file \"%s\": %m", path)));
}
- ReorderBufferSerializeChange(rb, txn, fd, change);
+ ReorderBufferSerializeChange(rb, txn, fd, curOpenSegNo, change);
dlist_delete(&change->node);
ReorderBufferReturnChange(rb, change);
@@ -2278,12 +2293,26 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
*/
static void
ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
- int fd, ReorderBufferChange *change)
+ int fd, XLogSegNo segno,
+ ReorderBufferChange *change)
{
ReorderBufferDiskChange *ondisk;
- Size sz = sizeof(ReorderBufferDiskChange);
+ Size sz,
+ sz_hdr;
+ char *outbuf;
+
+ sz = sizeof(ReorderBufferDiskChange);
- ReorderBufferSerializeReserve(rb, sz);
+ /*
+ * As the on-disk change has variable length, the header will have to be
+ * de-serialized separate, see ReorderBufferRestoreChanges(). Therefore we
+ * also need to encrypt it separate. Make sure the size is appropriate.
+ */
+ if (data_encrypted)
+ sz = TYPEALIGN(ENCRYPTION_BLOCK, sz);
+
+ sz_hdr = sz;
+ ReorderBufferSerializeReserve(rb, sz_hdr);
ondisk = (ReorderBufferDiskChange *) rb->outbuf;
memcpy(&ondisk->change, change, sizeof(ReorderBufferChange));
@@ -2320,6 +2349,15 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
}
/* make sure we have enough space */
+ if (data_encrypted)
+ {
+ /*
+ * Encryption works with blocks. As the changes are stored
+ * and retrieved separately, we also have to decrypt them
+ * alone.
+ */
+ sz = TYPEALIGN(ENCRYPTION_BLOCK, sz);
+ }
ReorderBufferSerializeReserve(rb, sz);
data = ((char *) rb->outbuf) + sizeof(ReorderBufferDiskChange);
@@ -2350,12 +2388,15 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
char *data;
Size prefix_size = strlen(change->data.msg.prefix) + 1;
+
sz += prefix_size + change->data.msg.message_size +
sizeof(Size) + sizeof(Size);
+
+ if (data_encrypted)
+ sz = TYPEALIGN(ENCRYPTION_BLOCK, sz);
ReorderBufferSerializeReserve(rb, sz);
data = ((char *) rb->outbuf) + sizeof(ReorderBufferDiskChange);
-
/* might have been reallocated above */
ondisk = (ReorderBufferDiskChange *) rb->outbuf;
@@ -2388,11 +2429,13 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
;
/* make sure we have enough space */
+ if (data_encrypted)
+ sz = TYPEALIGN(ENCRYPTION_BLOCK, sz);
ReorderBufferSerializeReserve(rb, sz);
+
data = ((char *) rb->outbuf) + sizeof(ReorderBufferDiskChange);
/* might have been reallocated above */
ondisk = (ReorderBufferDiskChange *) rb->outbuf;
-
memcpy(data, snap, sizeof(SnapshotData));
data += sizeof(SnapshotData);
@@ -2421,8 +2464,50 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
ondisk->size = sz;
+ /*
+ * Encrypt the change if encryption is required.
+ */
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ char tweak[TWEAK_SIZE];
+
+ REORDER_BUFFER_CHANGE_TWEAK(tweak, segno);
+
+ enlarge_encryption_buffer(ondisk->size);
+
+ /*
+ * Encrypt the header and the payload separate as explained at the top
+ * of the function.
+ */
+ encrypt_block((char *) rb->outbuf, encryption_buffer, sz_hdr,
+ tweak);
+
+ if (ondisk->size > sz_hdr)
+ encrypt_block((char *) rb->outbuf + sz_hdr,
+ encryption_buffer + sz_hdr,
+ ondisk->size - sz_hdr, tweak);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
+ /*
+ * Make sure the correct buffer is written to disk.
+ */
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ outbuf = encryption_buffer;
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+ else
+ outbuf = rb->outbuf;
+
pgstat_report_wait_start(WAIT_EVENT_REORDER_BUFFER_WRITE);
- if (write(fd, rb->outbuf, ondisk->size) != ondisk->size)
+ if (write(fd, outbuf, ondisk->size) != ondisk->size)
{
int save_errno = errno;
@@ -2450,6 +2535,10 @@ ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn,
Size restored = 0;
XLogSegNo last_segno;
dlist_mutable_iter cleanup_iter;
+ Size sz_hdr;
+#ifdef USE_ENCRYPTION
+ char encryption_tweak[TWEAK_SIZE];
+#endif
Assert(txn->first_lsn != InvalidXLogRecPtr);
Assert(txn->final_lsn != InvalidXLogRecPtr);
@@ -2468,6 +2557,17 @@ ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn,
XLByteToSeg(txn->final_lsn, last_segno, wal_segment_size);
+ /*
+ * See ReorderBufferSerializeChange for explanation.
+ */
+ sz_hdr = sizeof(ReorderBufferDiskChange);
+ if (data_encrypted)
+#ifdef USE_ENCRYPTION
+ sz_hdr = TYPEALIGN(ENCRYPTION_BLOCK, sz_hdr);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+
while (restored < max_changes_in_memory && *segno <= last_segno)
{
int readBytes;
@@ -2509,9 +2609,9 @@ ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn,
* about the total size. If we couldn't read a record, we're at the
* end of this file.
*/
- ReorderBufferSerializeReserve(rb, sizeof(ReorderBufferDiskChange));
+ ReorderBufferSerializeReserve(rb, sz_hdr);
pgstat_report_wait_start(WAIT_EVENT_REORDER_BUFFER_READ);
- readBytes = read(*fd, rb->outbuf, sizeof(ReorderBufferDiskChange));
+ readBytes = read(*fd, rb->outbuf, sz_hdr);
pgstat_report_wait_end();
/* eof */
@@ -2526,38 +2626,64 @@ ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn,
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not read from reorderbuffer spill file: %m")));
- else if (readBytes != sizeof(ReorderBufferDiskChange))
+ else if (readBytes != sz_hdr)
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not read from reorderbuffer spill file: read %d instead of %u bytes",
readBytes,
- (uint32) sizeof(ReorderBufferDiskChange))));
+ (uint32) sz_hdr)));
+
+ /*
+ * Decrypt the header if the change is decrypted, so that we know the
+ * change size.
+ */
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ REORDER_BUFFER_CHANGE_TWEAK(encryption_tweak, *segno);
+ decrypt_block(rb->outbuf, rb->outbuf, sz_hdr, encryption_tweak);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
ondisk = (ReorderBufferDiskChange *) rb->outbuf;
- ReorderBufferSerializeReserve(rb,
- sizeof(ReorderBufferDiskChange) + ondisk->size);
+ ReorderBufferSerializeReserve(rb, ondisk->size);
ondisk = (ReorderBufferDiskChange *) rb->outbuf;
pgstat_report_wait_start(WAIT_EVENT_REORDER_BUFFER_READ);
- readBytes = read(*fd, rb->outbuf + sizeof(ReorderBufferDiskChange),
- ondisk->size - sizeof(ReorderBufferDiskChange));
+ readBytes = read(*fd, rb->outbuf + sz_hdr, ondisk->size - sz_hdr);
pgstat_report_wait_end();
if (readBytes < 0)
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not read from reorderbuffer spill file: %m")));
- else if (readBytes != ondisk->size - sizeof(ReorderBufferDiskChange))
+ else if (readBytes != ondisk->size - sz_hdr)
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not read from reorderbuffer spill file: read %d instead of %u bytes",
readBytes,
- (uint32) (ondisk->size - sizeof(ReorderBufferDiskChange)))));
+ (uint32) (ondisk->size - sz_hdr))));
+
+ /*
+ * ok, read a full change from disk, now decrypt it if it's encrypted
+ * and if there's some data beyond the header.
+ */
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ if (ondisk->size > sz_hdr)
+ decrypt_block(rb->outbuf + sz_hdr, rb->outbuf + sz_hdr,
+ ondisk->size - sz_hdr, encryption_tweak);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
/*
- * ok, read a full change from disk, now restore it into proper
- * in-memory format
+ * Restore the change into proper in-memory format
*/
ReorderBufferRestoreChange(rb, txn, rb->outbuf);
restored++;
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index d60026dfd1..edd87ab221 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -77,6 +77,7 @@
#include "replication/walsender.h"
#include "replication/walsender_private.h"
#include "storage/condition_variable.h"
+#include "storage/encryption.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/pmsignal.h"
@@ -2327,9 +2328,16 @@ XLogRead(char *buf, XLogRecPtr startptr, Size count)
XLogRecPtr recptr;
Size nbytes;
XLogSegNo segno;
+#ifdef USE_ENCRYPTION
+ char *decrypt_p;
+ uint32 decryptOff;
+#endif
retry:
p = buf;
+#ifdef USE_ENCRYPTION
+ decrypt_p = p;
+#endif
recptr = startptr;
nbytes = count;
@@ -2340,6 +2348,9 @@ retry:
int readbytes;
startoff = XLogSegmentOffset(recptr, wal_segment_size);
+#ifdef USE_ENCRYPTION
+ decryptOff = startoff;
+#endif
if (sendFile < 0 || !XLByteInSeg(recptr, sendSegNo, wal_segment_size))
{
@@ -2455,6 +2466,25 @@ retry:
sendOff += readbytes;
nbytes -= readbytes;
p += readbytes;
+
+ /* Decrypt completed blocks */
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ while (decrypt_p + XLOG_BLCKSZ <= p)
+ {
+ char tweak[TWEAK_SIZE];
+
+ XLogEncryptionTweak(tweak, sendSegNo, decryptOff);
+ decrypt_block(decrypt_p, decrypt_p, XLOG_BLCKSZ, tweak);
+
+ decrypt_p += XLOG_BLCKSZ;
+ decryptOff += XLOG_BLCKSZ;
+ }
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
}
/*
diff --git a/src/backend/storage/file/buffile.c b/src/backend/storage/file/buffile.c
index efbede7629..ca0848dd20 100644
--- a/src/backend/storage/file/buffile.c
+++ b/src/backend/storage/file/buffile.c
@@ -47,12 +47,17 @@
#include "storage/fd.h"
#include "storage/buffile.h"
#include "storage/buf_internals.h"
+#include "storage/encryption.h"
+#include "utils/datetime.h"
#include "utils/resowner.h"
/*
* We break BufFiles into gigabyte-sized segments, regardless of RELSEG_SIZE.
* The reason is that we'd like large BufFiles to be spread across multiple
* tablespaces when available.
+ *
+ * The number of useful bytes is appended to each segment of shared encrypted
+ * file, see BufFile.useful.
*/
#define MAX_PHYSICAL_FILESIZE 0x40000000
#define BUFFILE_SEG_SIZE (MAX_PHYSICAL_FILESIZE / BLCKSZ)
@@ -68,6 +73,35 @@ struct BufFile
/* all files except the last have length exactly MAX_PHYSICAL_FILESIZE */
File *files; /* palloc'd array with numFiles entries */
off_t *offsets; /* palloc'd array with numFiles entries */
+#ifdef USE_ENCRYPTION
+
+ /*
+ * If the file is encrypted, only the whole buffer can be loaded / dumped
+ * --- see BufFileLoadBuffer() for more info --- whether it's space is
+ * used up or not. Therefore we need to keep track of the actual on-disk
+ * size buffer of each component file, as it would be if there was no
+ * encryption.
+ *
+ * List would make coding simpler, however would not contribute to
+ * performance. Random access is important here.
+ */
+ off_t *useful;
+
+ /*
+ * The array may need to be expanded independent from extendBufFile()
+ * (i.e. earlier than the buffer gets dumped), so store the number of
+ * elements separate from numFiles.
+ */
+ int nuseful;
+
+ /*
+ * Segment number is used to compute encryption tweak so we must remember
+ * the original numbers of segments if the file is encrypted and if it was
+ * passed as target to BufFileAppend() at least once. If this field is
+ * NULL, ->curFile is used to compute the tweak.
+ */
+ off_t *segnos;
+#endif
/*
* offsets[i] is the current seek position of files[i]. We use this to
@@ -106,6 +140,10 @@ static void BufFileLoadBuffer(BufFile *file);
static void BufFileDumpBuffer(BufFile *file);
static int BufFileFlush(BufFile *file);
static File MakeNewSharedSegment(BufFile *file, int segment);
+#ifdef USE_ENCRYPTION
+static void BufFileTweak(char *tweak, BufFile *file);
+static void ensureUsefulArraySize(BufFile *file, int required);
+#endif
/*
* Create BufFile and perform the common initialization.
@@ -125,6 +163,28 @@ makeBufFileCommon(int nfiles)
file->pos = 0;
file->nbytes = 0;
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ file->useful = (off_t *) palloc0(sizeof(off_t) * nfiles);
+ file->nuseful = nfiles;
+ file->segnos = NULL;
+
+ /*
+ * The unused (trailing) part of the buffer should not contain
+ * undefined data: if we encrypt such a buffer and flush it to disk,
+ * the encrypted form of that "undefined part" can get zeroed due to
+ * seek and write beyond EOF. If such a buffer gets loaded and
+ * decrypted, the change of the undefined part to zeroes can affect
+ * the valid part if it does not end at block boundary. By setting the
+ * whole buffer to zeroes we ensure that the unused part always
+ * contains zeroes.
+ */
+ MemSet(file->buffer, 0, BLCKSZ);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
return file;
}
@@ -172,6 +232,22 @@ extendBufFile(BufFile *file)
(file->numFiles + 1) * sizeof(File));
file->offsets = (off_t *) repalloc(file->offsets,
(file->numFiles + 1) * sizeof(off_t));
+
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ ensureUsefulArraySize(file, file->numFiles + 1);
+
+ if (file->segnos)
+ {
+ file->segnos = (off_t *) repalloc(file->segnos,
+ (file->numFiles + 1) * sizeof(off_t));
+ file->segnos[file->numFiles] = file->numFiles;
+ }
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
file->files[file->numFiles] = pfile;
file->offsets[file->numFiles] = 0L;
file->numFiles++;
@@ -302,6 +378,7 @@ BufFileOpenShared(SharedFileSet *fileset, const char *name)
files[nfiles] = SharedFileSetOpen(fileset, segment_name);
if (files[nfiles] <= 0)
break;
+
++nfiles;
CHECK_FOR_INTERRUPTS();
@@ -317,7 +394,64 @@ BufFileOpenShared(SharedFileSet *fileset, const char *name)
errmsg("could not open BufFile \"%s\"", name)));
file = makeBufFileCommon(nfiles);
+
+ /*
+ * Shared encrypted segment should contain information on the number of
+ * useful bytes at the end.
+ */
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ off_t pos;
+ int i;
+
+ for (i = 0; i < nfiles; i++)
+ {
+ int nbytes;
+ File segment = files[i];
+
+ pos = FileSeek(segment, -sizeof(off_t), SEEK_END);
+
+ /*
+ * The word must immediately follow the last buffer of the
+ * segment.
+ */
+ if (pos <= 0 || pos % BLCKSZ != 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not find padding info in BufFile \"%s\"",
+ name)));
+
+ nbytes = FileRead(segment, (char *) &file->useful[i],
+ sizeof(off_t), WAIT_EVENT_BUFFILE_READ);
+ if (nbytes != sizeof(off_t))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read padding info from BufFile \"%s\"",
+ name)));
+ Assert(file->useful[i] > 0);
+
+ /*
+ * Keep ->offsets up-to-date.
+ */
+ file->offsets[i] = FileTell(segment);
+
+ CHECK_FOR_INTERRUPTS();
+ }
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
file->files = files;
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ file->nuseful = nfiles;
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
file->readOnly = true; /* Can't write to files opened this way */
file->fileset = fileset;
file->name = pstrdup(name);
@@ -397,6 +531,17 @@ BufFileClose(BufFile *file)
/* release the buffer space */
pfree(file->files);
pfree(file->offsets);
+
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ if (file->segnos)
+ pfree(file->segnos);
+ pfree(file->useful);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
pfree(file);
}
@@ -404,7 +549,7 @@ BufFileClose(BufFile *file)
* BufFileLoadBuffer
*
* Load some data into buffer, if possible, starting from curOffset.
- * At call, must have dirty = false, pos and nbytes = 0.
+ * At call, must have dirty = false, nbytes = 0.
* On exit, nbytes is number of bytes loaded.
*/
static void
@@ -413,6 +558,22 @@ BufFileLoadBuffer(BufFile *file)
File thisfile;
/*
+ * Only whole multiple of ENCRYPTION_BLOCK can be encrypted / decrypted,
+ * but we choose to use BLCKSZ (i.e. BufFile buffer) as the unit. The
+ * point is that curOffset is a component of the encryption tweak, and all
+ * data within particular call of encrypt_block() / decrypt_block() must
+ * have the same tweak. So whichever unit we choose we must stick on it
+ * and never encrypt / decrypt multiple units at a time.
+ *
+ * BLCKSZ also seems better choice than ENCRYPTION_BLOCK for performance
+ * purposes. We assume that alignment to BLCKSZ implies alignment to
+ * ENCRYPTION_BLOCK.
+ */
+ Assert((file->curOffset % BLCKSZ == 0 &&
+ file->curOffset % ENCRYPTION_BLOCK == 0) ||
+ !data_encrypted);
+
+ /*
* Advance to next component file if necessary and possible.
*/
if (file->curOffset >= MAX_PHYSICAL_FILESIZE &&
@@ -423,6 +584,20 @@ BufFileLoadBuffer(BufFile *file)
}
/*
+ * See makeBufFile().
+ *
+ * Actually here we only handle the case of FileRead() returning zero
+ * bytes below. In contrast, if the buffer contains any data but it's not
+ * full, it should already have the trailing zeroes (encrypted) on disk.
+ * And as the encrypted buffer is always loaded in its entirety (i.e. EOF
+ * should only appear at buffer boundary if the data is encrypted), all
+ * unused bytes of the buffer should eventually be zeroes after the
+ * decryption.
+ */
+ if (data_encrypted)
+ MemSet(file->buffer, 0, BLCKSZ);
+
+ /*
* May need to reposition physical file.
*/
thisfile = file->files[file->curFile];
@@ -445,6 +620,64 @@ BufFileLoadBuffer(BufFile *file)
file->offsets[file->curFile] += file->nbytes;
/* we choose not to advance curOffset here */
+ if (data_encrypted && file->nbytes > 0)
+ {
+#ifdef USE_ENCRYPTION
+ char tweak[TWEAK_SIZE];
+ int nbytes = file->nbytes;
+
+ /*
+ * The encrypted component file can only consist of whole number of
+ * our encryption units. (Only the whole buffers are dumped / loaded.)
+ * The only exception is that we're at the end of segment file and
+ * found the word indicating the number of useful bytes in the
+ * segment. This can only happen for shared file.
+ */
+ if (nbytes % BLCKSZ != 0)
+ {
+ Assert(nbytes == sizeof(off_t) && file->fileset != NULL);
+
+ /*
+ * This special word cannot start within the useful data.
+ */
+ Assert(file->offsets[file->curFile] - nbytes >=
+ file->useful[file->curFile]);
+
+ /*
+ * This metadata his hidden to caller, so all he needs to know
+ * that there's no real data at the end of the file.
+ */
+ file->nbytes = 0;
+ return;
+ }
+
+ BufFileTweak(tweak, file);
+
+ /*
+ * The whole block is encrypted / decrypted at once as explained
+ * above.
+ */
+ decrypt_block(file->buffer, file->buffer, BLCKSZ, tweak);
+
+#ifdef USE_ASSERT_CHECKING
+
+ /*
+ * The unused part of the buffer which we've read from disk and
+ * decrypted should only contain zeroes, as explained in front of the
+ * MemSet() call.
+ */
+ {
+ int i;
+
+ for (i = file->nbytes; i < BLCKSZ; i++)
+ Assert(file->buffer[i] == 0);
+ }
+#endif /* USE_ASSERT_CHECKING */
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
if (file->nbytes > 0)
pgBufferUsage.temp_blks_read++;
}
@@ -455,6 +688,9 @@ BufFileLoadBuffer(BufFile *file)
* Dump buffer contents starting at curOffset.
* At call, should have dirty = true, nbytes > 0.
* On exit, dirty is cleared if successful write, and curOffset is advanced.
+ *
+ * XXX Consider separate function for encrypted buffer. (No loop is needed to
+ * dump the encrypted buffer.)
*/
static void
BufFileDumpBuffer(BufFile *file)
@@ -462,6 +698,52 @@ BufFileDumpBuffer(BufFile *file)
int wpos = 0;
int bytestowrite;
File thisfile;
+ char *write_ptr;
+
+ /*
+ * See comments in BufFileLoadBuffer();
+ */
+ Assert((file->curOffset % BLCKSZ == 0 &&
+ file->curOffset % ENCRYPTION_BLOCK == 0) ||
+ !data_encrypted);
+
+ /*
+ * Caller's responsibility.
+ */
+ Assert(file->pos <= file->nbytes);
+
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ char tweak[TWEAK_SIZE];
+
+ BufFileTweak(tweak, file);
+
+ /*
+ * The amount of data encrypted must be a multiple of
+ * ENCRYPTION_BLOCK. We meet this condition simply by encrypting the
+ * whole buffer.
+ *
+ * XXX Alternatively we could get the encrypted chunk length by
+ * rounding file->nbytes up to the nearest multiple of
+ * ENCRYPTION_BLOCK, and for decryption use the
+ * file->useful[file->curFile] value to find out how many blocks
+ * should be decrypted. That would reduce I/O if the buffer is mostly
+ * empty, but (BLCKSZ / ENCRYPTION_BLOCK) calls of encrypt_block()
+ * would be needed for full buffers. See also BufFileLoadBuffer() for
+ * explanation why we must stick on the unit of data amount encrypted
+ * / decrypted.
+ */
+ if (encryption_buf_size < BLCKSZ)
+ enlarge_encryption_buffer(BLCKSZ);
+ encrypt_block(file->buffer, encryption_buffer, BLCKSZ, tweak);
+ write_ptr = encryption_buffer;
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+ else
+ write_ptr = file->buffer;
/*
* Unlike BufFileLoadBuffer, we must dump the whole buffer even if it
@@ -469,8 +751,6 @@ BufFileDumpBuffer(BufFile *file)
*/
while (wpos < file->nbytes)
{
- off_t availbytes;
-
/*
* Advance to next component file if necessary and possible.
*/
@@ -482,31 +762,56 @@ BufFileDumpBuffer(BufFile *file)
file->curOffset = 0L;
}
- /*
- * Determine how much we need to write into this file.
- */
- bytestowrite = file->nbytes - wpos;
- availbytes = MAX_PHYSICAL_FILESIZE - file->curOffset;
+ if (!data_encrypted)
+ {
+ off_t availbytes;
+
+ bytestowrite = file->nbytes - wpos;
+ availbytes = MAX_PHYSICAL_FILESIZE - file->curOffset;
- if ((off_t) bytestowrite > availbytes)
- bytestowrite = (int) availbytes;
+ if ((off_t) bytestowrite > availbytes)
+ bytestowrite = (int) availbytes;
+ }
+ else
+ {
+ /*
+ * This condition plus the alignment of curOffset to BLCKSZ
+ * (checked above) ensure that the encrypted buffer never crosses
+ * component file boundary.
+ */
+ StaticAssertStmt((MAX_PHYSICAL_FILESIZE % BLCKSZ) == 0,
+ "BLCKSZ is not whole multiple of MAX_PHYSICAL_FILESIZE");
+
+ /*
+ * Encrypted data is dumped all at once.
+ *
+ * Here we don't have to check availbytes because --- according to
+ * the assertions above --- currOffset should be lower than
+ * MAX_PHYSICAL_FILESIZE by non-zero multiple of BLCKSZ.
+ */
+ bytestowrite = BLCKSZ;
+ }
/*
* May need to reposition physical file.
*/
thisfile = file->files[file->curFile];
+
if (file->curOffset != file->offsets[file->curFile])
{
if (FileSeek(thisfile, file->curOffset, SEEK_SET) != file->curOffset)
return; /* seek failed, give up */
+
file->offsets[file->curFile] = file->curOffset;
}
bytestowrite = FileWrite(thisfile,
- file->buffer + wpos,
+ write_ptr + wpos,
bytestowrite,
WAIT_EVENT_BUFFILE_WRITE);
- if (bytestowrite <= 0)
+ if (bytestowrite <= 0 ||
+ (data_encrypted && bytestowrite != BLCKSZ))
return; /* failed to write */
+
file->offsets[file->curFile] += bytestowrite;
file->curOffset += bytestowrite;
wpos += bytestowrite;
@@ -515,25 +820,130 @@ BufFileDumpBuffer(BufFile *file)
}
file->dirty = false;
- /*
- * At this point, curOffset has been advanced to the end of the buffer,
- * ie, its original value + nbytes. We need to make it point to the
- * logical file position, ie, original value + pos, in case that is less
- * (as could happen due to a small backwards seek in a dirty buffer!)
- */
- file->curOffset -= (file->nbytes - file->pos);
- if (file->curOffset < 0) /* handle possible segment crossing */
+ if (!data_encrypted)
{
- file->curFile--;
- Assert(file->curFile >= 0);
- file->curOffset += MAX_PHYSICAL_FILESIZE;
+ /*
+ * At this point, curOffset has been advanced to the end of the
+ * buffer, ie, its original value + nbytes. We need to make it point
+ * to the logical file position, ie, original value + pos, in case
+ * that is less (as could happen due to a small backwards seek in a
+ * dirty buffer!)
+ */
+ file->curOffset -= (file->nbytes - file->pos);
+ if (file->curOffset < 0) /* handle possible segment crossing */
+ {
+ file->curFile--;
+ Assert(file->curFile >= 0);
+ file->curOffset += MAX_PHYSICAL_FILESIZE;
+ }
+
+ /*
+ * Now we can set the buffer empty without changing the logical
+ * position
+ */
+ file->pos = 0;
+ file->nbytes = 0;
}
+ else
+ {
+ /*
+ * curOffset should be at buffer boundary and buffer is the smallest
+ * I/O unit for encrypted data.
+ */
+ Assert(file->curOffset % BLCKSZ == 0);
- /*
- * Now we can set the buffer empty without changing the logical position
- */
- file->pos = 0;
- file->nbytes = 0;
+ /*
+ * The number of useful bytes needs to be written at the end of each
+ * encrypted segment of a shared file so that the other backends know
+ * how many bytes of the last buffer are useful.
+ */
+ if (file->fileset != NULL)
+ {
+ off_t useful;
+
+#ifdef USE_ENCRYPTION
+
+ /*
+ * nuseful may be increased earlier than numFiles but not later,
+ * so the corresponding entry should always exist in ->useful.
+ */
+ Assert(file->curFile < file->nuseful);
+
+ /*
+ * The number of useful bytes in the current segment file.
+ */
+ useful = file->useful[file->curFile];
+#endif
+
+ /*
+ * Have we dumped the last buffer of the segment, i.e. the one
+ * that can contain padding?
+ */
+ if (file->curOffset >= useful)
+ {
+ int bytes_extra;
+
+ /*
+ * Write the number of useful bytes in the segment.
+ */
+ bytes_extra = FileWrite(thisfile,
+ (char *) &useful,
+ sizeof(useful),
+ WAIT_EVENT_BUFFILE_WRITE);
+ if (bytes_extra != sizeof(useful))
+ return; /* failed to write */
+
+ /*
+ * Remember the file position at OS level, but do not increase
+ * curOffset. That ensures that the next buffer appended will
+ * overwrite the "useful" value just written, instead of being
+ * appended to it.
+ *
+ * XXX The additional seek backwards at OS level when the next
+ * buffer is gonna be written is hard to avoid by writing this
+ * info higher in the stack (e.g. in BufFileWrite() or
+ * BufFileFlush()). Once we've cleared the "dirty" flag, we
+ * cannot recognize easily later that the number of useful
+ * bytes might need to be updated on disk because of this
+ * buffer. Is it worth introducing another bool field and
+ * making the logic trickier?
+ */
+ file->offsets[file->curFile] += bytes_extra;
+ }
+ }
+
+ if (file->pos >= BLCKSZ)
+ {
+ Assert(file->pos == BLCKSZ);
+
+ /*
+ * curOffset points to the beginning of the next buffer, so just
+ * reset pos and nbytes.
+ */
+ file->pos = 0;
+ file->nbytes = 0;
+
+ /* See makeBufFile() */
+ if (data_encrypted)
+ MemSet(file->buffer, 0, BLCKSZ);
+ }
+ else
+ {
+ /*
+ * Move curOffset to the beginning of the just-written buffer and
+ * preserve pos.
+ */
+ file->curOffset -= BLCKSZ;
+
+ /*
+ * At least pos bytes should be written even if the first change
+ * since now appears at pos == nbytes, but in fact the whole
+ * buffer will be written regardless pos. This is the price we pay
+ * for the choosing BLCKSZ as the I/O unit for encrypted data.
+ */
+ file->nbytes = BLCKSZ;
+ }
+ }
}
/*
@@ -556,18 +966,107 @@ BufFileRead(BufFile *file, void *ptr, size_t size)
while (size > 0)
{
+ int avail;
+
if (file->pos >= file->nbytes)
{
+ /*
+ * Neither read nor write nor seek should leave pos greater than
+ * nbytes, regardless the data is encrypted or not. pos can only
+ * be greater if nbytes is zero --- this situation can be caused
+ * by BufFileSeek().
+ */
+ Assert(file->pos == file->nbytes || file->nbytes == 0);
+
+ /*
+ * The Assert() above implies that pos is a whole multiple of
+ * BLCKSZ, so curOffset has meet the same encryption-specific
+ * requirement too.
+ */
+ Assert(file->curOffset % BLCKSZ == 0 || !data_encrypted);
+
/* Try to load more data into buffer. */
- file->curOffset += file->pos;
- file->pos = 0;
- file->nbytes = 0;
- BufFileLoadBuffer(file);
- if (file->nbytes <= 0)
- break; /* no more data available */
+ if (!data_encrypted || file->pos % BLCKSZ == 0)
+ {
+ file->curOffset += file->pos;
+ file->pos = 0;
+ file->nbytes = 0;
+ BufFileLoadBuffer(file);
+ if (file->nbytes <= 0)
+ break; /* no more data available */
+ }
+ else
+ {
+ int nbytes_orig = file->nbytes;
+
+ /*
+ * Given that BLCKSZ is the I/O unit for encrypted data (see
+ * comments in BufFileLoadBuffer()), we cannot add pos to
+ * curOffset because that would make it point outside block
+ * boundary. The only thing we can do is to reload the whole
+ * buffer and see if more data is eventually there than the
+ * previous load has fetched.
+ */
+ BufFileLoadBuffer(file);
+ if (file->nbytes <= nbytes_orig)
+ break; /* no more data available */
+ }
+ }
+
+ avail = file->nbytes;
+ nthistime = avail - file->pos;
+
+ /*
+ * The buffer can contain trailing zeroes because BLCKSZ is the I/O
+ * unit for encrypted data. These are not available for reading.
+ */
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ off_t useful = file->useful[file->curFile];
+
+ /*
+ * The criterion is whether the useful data end within the
+ * currently loaded buffer.
+ */
+ if (useful < file->curOffset + BLCKSZ)
+ {
+ /*
+ * Compute the number of bytes available in the current
+ * buffer.
+ */
+ avail = useful - file->curOffset;
+ Assert(avail >= 0);
+
+ /*
+ * An empty buffer can exist, e.g. after a seek to the end of
+ * the last component file.
+ */
+ if (avail == 0)
+ break;
+
+ /*
+ * Seek beyond the current EOF, which was not followed by
+ * write, could have resulted in position outside the useful
+ * data
+ */
+ if (file->pos > avail)
+ break;
+
+ nthistime = avail - file->pos;
+ Assert(nthistime >= 0);
+
+ /*
+ * Have we reached the end of the valid data?
+ */
+ if (nthistime == 0)
+ break;
+ }
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
}
- nthistime = file->nbytes - file->pos;
if (nthistime > size)
nthistime = size;
Assert(nthistime > 0);
@@ -609,11 +1108,28 @@ BufFileWrite(BufFile *file, void *ptr, size_t size)
}
else
{
- /* Hmm, went directly from reading to writing? */
+ /*
+ * Hmm, went directly from reading to writing?
+ *
+ * As pos should be exactly BLCKSZ, there is nothing special
+ * to do about data_encrypted. Except for zeroing the buffer.
+ */
+ Assert(file->pos == BLCKSZ);
+
file->curOffset += file->pos;
file->pos = 0;
file->nbytes = 0;
+
+ /* See makeBufFile() */
+ if (data_encrypted)
+ MemSet(file->buffer, 0, BLCKSZ);
}
+
+ /*
+ * If curOffset changed above, it should still meet the assumption
+ * that buffer is the I/O unit for encrypted data.
+ */
+ Assert(file->curOffset % BLCKSZ == 0 || !data_encrypted);
}
nthistime = BLCKSZ - file->pos;
@@ -627,6 +1143,65 @@ BufFileWrite(BufFile *file, void *ptr, size_t size)
file->pos += nthistime;
if (file->nbytes < file->pos)
file->nbytes = file->pos;
+
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ off_t new_offset;
+ int fileno = file->curFile;
+
+ /*
+ * curFile does not necessarily correspond to the offset: it can
+ * still have the initial value if BufFileSeek() skipped the
+ * previous file w/o dumping anything of it. While curFile will be
+ * fixed during the next dump, we need valid fileno now.
+ */
+ if (file->curOffset >= MAX_PHYSICAL_FILESIZE)
+ {
+ /*
+ * Even BufFileSeek() should not allow curOffset to become
+ * more than MAX_PHYSICAL_FILESIZE (if caller passes higher
+ * offset, curFile gets increased instead).
+ */
+ Assert(file->curOffset == MAX_PHYSICAL_FILESIZE);
+
+ fileno++;
+ }
+
+ /*
+ * fileno can point to a segment that does not exist on disk yet.
+ */
+ ensureUsefulArraySize(file, fileno + 1);
+
+ /*
+ * Update the "useful offset" of the underlying component file if
+ * we've added any useful data.
+ */
+ new_offset = file->curOffset + file->pos;
+
+ /*
+ * Make sure the offset is relative to the correct component file.
+ * It should have been adjusted during sequential write, but if
+ * we've just used BufFileSeek() to jump to segment boundary w/o
+ * writing, the value is relative to the start of the *previous*
+ * segment.
+ */
+ if (file->curOffset % MAX_PHYSICAL_FILESIZE == 0)
+ new_offset %= MAX_PHYSICAL_FILESIZE;
+
+ /*
+ * Adjust the number of useful bytes in the file if needed. This
+ * has to happen immediately, independent from
+ * BufFileDumpBuffer(), so that BufFileRead() works correctly
+ * anytime.
+ */
+ if (new_offset > file->useful[fileno])
+ file->useful[fileno] = new_offset;
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
ptr = (void *) ((char *) ptr + nthistime);
size -= nthistime;
nwritten += nthistime;
@@ -741,9 +1316,43 @@ BufFileSeek(BufFile *file, int fileno, off_t offset, int whence)
return EOF;
/* Seek is OK! */
file->curFile = newFile;
- file->curOffset = newOffset;
- file->pos = 0;
- file->nbytes = 0;
+ if (!data_encrypted)
+ {
+ file->curOffset = newOffset;
+ file->pos = 0;
+ file->nbytes = 0;
+ }
+ else
+ {
+ /*
+ * Offset of an encrypted buffer must be a multiple of BLCKSZ.
+ */
+ file->pos = newOffset % BLCKSZ;
+ file->curOffset = newOffset - file->pos;
+
+ /*
+ * BufFileLoadBuffer() will set nbytes iff it can read something.
+ */
+ file->nbytes = 0;
+
+ /*
+ * Load and decrypt the existing part of the buffer.
+ */
+ BufFileLoadBuffer(file);
+ if (file->nbytes == 0)
+ {
+ /*
+ * The data requested is not in the file, but this is not an
+ * error.
+ */
+ return 0;
+ }
+
+ /*
+ * The whole buffer should have been loaded.
+ */
+ Assert(file->nbytes == BLCKSZ);
+ }
return 0;
}
@@ -774,6 +1383,55 @@ BufFileSeekBlock(BufFile *file, long blknum)
SEEK_SET);
}
+#ifdef USE_ENCRYPTION
+static void
+BufFileTweak(char *tweak, BufFile *file)
+{
+ off_t block;
+ int curFile = file->curFile;
+
+ /*
+ * If the file was produced by BufFileAppend(), we need the original
+ * curFile, as it was used originally for encryption.
+ */
+ if (file->segnos)
+ curFile = file->segnos[curFile];
+
+ block = curFile * BUFFILE_SEG_SIZE + file->curOffset / BLCKSZ;
+
+ /*
+ * The unused bytes should always be defined.
+ */
+ memset(tweak, 0, TWEAK_SIZE);
+ StaticAssertStmt(sizeof(block) <= TWEAK_SIZE,
+ "block number does not fit into encryption tweak");
+ memcpy(tweak, &block, sizeof(off_t));
+}
+
+/*
+ * Make sure that BufFile.useful array has the required size.
+ */
+static void
+ensureUsefulArraySize(BufFile *file, int required)
+{
+ /*
+ * Does the array already have enough space?
+ */
+ if (required <= file->nuseful)
+ return;
+
+ /*
+ * It shouldn't be possible to jump beyond the end of the last segment,
+ * i.e. skip more than 1 segment.
+ */
+ Assert(file->nuseful + 1 == required);
+
+ file->useful = (off_t *) repalloc(file->useful, required * sizeof(off_t));
+ file->useful[file->nuseful] = 0L;
+ file->nuseful++;
+}
+#endif
+
#ifdef NOT_USED
/*
* BufFileTellBlock --- block-oriented tell
@@ -803,11 +1461,33 @@ BufFileSize(BufFile *file)
{
off_t lastFileSize;
- /* Get the size of the last physical file by seeking to end. */
- lastFileSize = FileSeek(file->files[file->numFiles - 1], 0, SEEK_END);
- if (lastFileSize < 0)
- return -1;
- file->offsets[file->numFiles - 1] = lastFileSize;
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ /*
+ * "useful" should be initialized even for shared file, see
+ * BufFileOpenShared().
+ */
+ Assert(file->useful != NULL && file->nuseful >= file->numFiles);
+
+ /*
+ * The number of useful bytes in the segment is what caller is
+ * interested in.
+ */
+ lastFileSize = file->useful[file->nuseful - 1];
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+ else
+ {
+ /* Get the size of the last physical file by seeking to end. */
+ lastFileSize = FileSeek(file->files[file->numFiles - 1], 0, SEEK_END);
+ if (lastFileSize < 0)
+ return -1;
+
+ file->offsets[file->numFiles - 1] = lastFileSize;
+ }
return ((file->numFiles - 1) * (off_t) MAX_PHYSICAL_FILESIZE) +
lastFileSize;
@@ -837,6 +1517,9 @@ BufFileAppend(BufFile *target, BufFile *source)
{
long startBlock = target->numFiles * BUFFILE_SEG_SIZE;
int newNumFiles = target->numFiles + source->numFiles;
+#ifdef USE_ENCRYPTION
+ int newNUseful = target->nuseful + source->nuseful;
+#endif
int i;
Assert(target->fileset != NULL);
@@ -851,11 +1534,75 @@ BufFileAppend(BufFile *target, BufFile *source)
repalloc(target->files, sizeof(File) * newNumFiles);
target->offsets = (off_t *)
repalloc(target->offsets, sizeof(off_t) * newNumFiles);
+
for (i = target->numFiles; i < newNumFiles; i++)
{
- target->files[i] = source->files[i - target->numFiles];
- target->offsets[i] = source->offsets[i - target->numFiles];
+ int n = i - target->numFiles;
+
+ target->files[i] = source->files[n];
+ target->offsets[i] = source->offsets[n];
}
+
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ /*
+ * XXX As the typical use case is that parallel workers expose file to
+ * the leader, can we expect both target and source to have been
+ * exported, i.e. flushed? In such a case "nuseful" would have to be
+ * equal to "numFiles" for both input files and the code could get a
+ * bit simpler. It seems that at least source should be flushed, as
+ * source->readOnly is expected to be true above.
+ */
+ target->useful = (off_t *)
+ repalloc(target->useful, sizeof(off_t) * newNUseful);
+
+ for (i = target->nuseful; i < newNUseful; i++)
+ target->useful[i] = source->useful[i - target->nuseful];
+ target->nuseful = newNUseful;
+
+ /*
+ * File segments can appear at different position due to
+ * concatenation, so make sure we remember the original positions for
+ * the sake of encryption tweak.
+ */
+ if (target->segnos == NULL)
+ {
+ /*
+ * If the target does not have the array yet, allocate it for both
+ * target and source and initialize the target part.
+ */
+ target->segnos = (off_t *) palloc(newNumFiles * sizeof(off_t));
+ for (i = 0; i < target->numFiles; i++)
+ target->segnos[i] = i;
+ }
+ else
+ {
+ /*
+ * Use the existing target part and add space for the source part.
+ */
+ target->segnos = (off_t *) repalloc(target->segnos,
+ newNumFiles * sizeof(off_t));
+ }
+
+ /*
+ * The source segment number either equals to (0-based) index of the
+ * segment, or to an element of an already existing array.
+ */
+ for (i = target->numFiles; i < newNumFiles; i++)
+ {
+ off_t segno = i - target->numFiles;
+
+ if (source->segnos == NULL)
+ target->segnos[i] = segno;
+ else
+ target->segnos[i] = source->segnos[segno];
+ }
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
target->numFiles = newNumFiles;
return startBlock;
diff --git a/src/backend/storage/file/copydir.c b/src/backend/storage/file/copydir.c
index 4a0d23b11e..eb4ad205e1 100644
--- a/src/backend/storage/file/copydir.c
+++ b/src/backend/storage/file/copydir.c
@@ -23,24 +23,30 @@
#include <sys/stat.h>
#include "storage/copydir.h"
+#include "storage/encryption.h"
#include "storage/fd.h"
+#include "storage/reinit.h"
+#include "storage/smgr.h"
#include "miscadmin.h"
#include "pgstat.h"
/*
* copydir: copy a directory
*
- * If recurse is false, subdirectories are ignored. Anything that's not
- * a directory or a regular file is ignored.
+ * RelFileNode values must specify tablespace and database oids for source
+ * and target to support re-encryption if necessary. relNode value in provided
+ * structs will be clobbered.
*/
void
-copydir(char *fromdir, char *todir, bool recurse)
+copydir(char *fromdir, char *todir, RelFileNode *fromNode, RelFileNode *toNode)
{
DIR *xldir;
struct dirent *xlde;
char fromfile[MAXPGPATH * 2];
char tofile[MAXPGPATH * 2];
+ Assert(!data_encrypted || (fromNode != NULL && toNode != NULL));
+
if (MakePGDirectory(todir) != 0)
ereport(ERROR,
(errcode_for_file_access(),
@@ -67,14 +73,32 @@ copydir(char *fromdir, char *todir, bool recurse)
(errcode_for_file_access(),
errmsg("could not stat file \"%s\": %m", fromfile)));
- if (S_ISDIR(fst.st_mode))
+ if (S_ISREG(fst.st_mode))
{
- /* recurse to handle subdirectories */
- if (recurse)
- copydir(fromfile, tofile, true);
+ int oidchars;
+ ForkNumber forkNum;
+ int segment;
+
+ /*
+ * For encrypted databases we need to reencrypt files with new
+ * tweaks.
+ */
+ if (data_encrypted &&
+ parse_filename_for_nontemp_relation(xlde->d_name,
+ &oidchars, &forkNum, &segment))
+ {
+ char oidbuf[OIDCHARS + 1];
+
+ memcpy(oidbuf, xlde->d_name, oidchars);
+ oidbuf[oidchars] = '\0';
+
+ /* We scribble over the provided RelFileNodes here */
+ fromNode->relNode = toNode->relNode = atol(oidbuf);
+ copy_file(fromfile, tofile, fromNode, toNode, forkNum, forkNum, segment);
+ }
+ else
+ copy_file(fromfile, tofile, NULL, NULL, 0, 0, 0);
}
- else if (S_ISREG(fst.st_mode))
- copy_file(fromfile, tofile);
}
FreeDir(xldir);
@@ -121,17 +145,24 @@ copydir(char *fromdir, char *todir, bool recurse)
}
/*
- * copy one file
+ * copy one file. If decryption and reencryption may be needed specify
+ * relfilenodes for source and target.
*/
void
-copy_file(char *fromfile, char *tofile)
+copy_file(char *fromfile, char *tofile, RelFileNode *fromNode,
+ RelFileNode *toNode, ForkNumber fromForkNum, ForkNumber toForkNum,
+ int segment)
{
char *buffer;
int srcfd;
int dstfd;
int nbytes;
+ int bytesread;
off_t offset;
off_t flush_offset;
+#ifdef USE_ENCRYPTION
+ BlockNumber blockNum = segment * RELSEG_SIZE;
+#endif
/* Size of copy buffer (read and write requests) */
#define COPY_BUF_SIZE (8 * BLCKSZ)
@@ -186,16 +217,55 @@ copy_file(char *fromfile, char *tofile)
flush_offset = offset;
}
- pgstat_report_wait_start(WAIT_EVENT_COPY_FILE_READ);
- nbytes = read(srcfd, buffer, COPY_BUF_SIZE);
- pgstat_report_wait_end();
- if (nbytes < 0)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not read file \"%s\": %m", fromfile)));
+ /*
+ * Try to read as much as we fit in the buffer so we can deal with
+ * complete blocks if we need to reencrypt.
+ */
+ nbytes = 0;
+ while (nbytes < COPY_BUF_SIZE)
+ {
+ pgstat_report_wait_start(WAIT_EVENT_COPY_FILE_READ);
+ bytesread = read(srcfd, buffer + nbytes, COPY_BUF_SIZE - nbytes);
+ pgstat_report_wait_end();
+ if (bytesread < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m", fromfile)));
+ nbytes += bytesread;
+ if (bytesread == 0)
+ break;
+ }
if (nbytes == 0)
break;
+
+ /*
+ * If encryption is in place, only complete blocks should have been
+ * read. Thus we do not have to care whether any encryption block is
+ * affected by the previous buffer contents.
+ */
+ Assert(nbytes % BLCKSZ == 0 || (fromNode == NULL && toNode == NULL));
+
errno = 0;
+
+ /*
+ * If the database is encrypted we need to decrypt the data here and
+ * reencrypt it to adjust the tweak values of blocks.
+ */
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ if (fromNode != NULL)
+ {
+ Assert(toNode != NULL);
+ blockNum = ReencryptBlock(buffer, nbytes / BLCKSZ,
+ fromNode, toNode, fromForkNum,
+ toForkNum, blockNum);
+ }
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
pgstat_report_wait_start(WAIT_EVENT_COPY_FILE_WRITE);
if ((int) write(dstfd, buffer, nbytes) != nbytes)
{
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 8dd51f1767..b3e3a50d30 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -2163,10 +2163,6 @@ FileSeek(File file, off_t offset, int whence)
return vfdP->seekPos;
}
-/*
- * XXX not actually used but here for completeness
- */
-#ifdef NOT_USED
off_t
FileTell(File file)
{
@@ -2175,7 +2171,6 @@ FileTell(File file)
file, VfdCache[file].fileName));
return VfdCache[file].seekPos;
}
-#endif
int
FileTruncate(File file, off_t offset, uint32 wait_event_info)
diff --git a/src/backend/storage/file/reinit.c b/src/backend/storage/file/reinit.c
index 74ff6c359b..70e138503e 100644
--- a/src/backend/storage/file/reinit.c
+++ b/src/backend/storage/file/reinit.c
@@ -16,6 +16,7 @@
#include <unistd.h>
+#include "catalog/pg_tablespace.h"
#include "common/relpath.h"
#include "storage/copydir.h"
#include "storage/fd.h"
@@ -24,9 +25,9 @@
#include "utils/memutils.h"
static void ResetUnloggedRelationsInTablespaceDir(const char *tsdirname,
- int op);
+ int op, Oid spcOid);
static void ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname,
- int op);
+ int op, Oid spcOid, Oid dbOid);
typedef struct
{
@@ -68,7 +69,7 @@ ResetUnloggedRelations(int op)
/*
* First process unlogged files in pg_default ($PGDATA/base)
*/
- ResetUnloggedRelationsInTablespaceDir("base", op);
+ ResetUnloggedRelationsInTablespaceDir("base", op, DEFAULTTABLESPACE_OID);
/*
* Cycle through directories for all non-default tablespaces.
@@ -77,13 +78,16 @@ ResetUnloggedRelations(int op)
while ((spc_de = ReadDir(spc_dir, "pg_tblspc")) != NULL)
{
+ Oid spcOid;
+
if (strcmp(spc_de->d_name, ".") == 0 ||
strcmp(spc_de->d_name, "..") == 0)
continue;
snprintf(temp_path, sizeof(temp_path), "pg_tblspc/%s/%s",
spc_de->d_name, TABLESPACE_VERSION_DIRECTORY);
- ResetUnloggedRelationsInTablespaceDir(temp_path, op);
+ spcOid = atoi(spc_de->d_name);
+ ResetUnloggedRelationsInTablespaceDir(temp_path, op, spcOid);
}
FreeDir(spc_dir);
@@ -99,7 +103,8 @@ ResetUnloggedRelations(int op)
* Process one tablespace directory for ResetUnloggedRelations
*/
static void
-ResetUnloggedRelationsInTablespaceDir(const char *tsdirname, int op)
+ResetUnloggedRelationsInTablespaceDir(const char *tsdirname, int op,
+ Oid spcOid)
{
DIR *ts_dir;
struct dirent *de;
@@ -126,6 +131,8 @@ ResetUnloggedRelationsInTablespaceDir(const char *tsdirname, int op)
while ((de = ReadDir(ts_dir, tsdirname)) != NULL)
{
+ Oid dbOid;
+
/*
* We're only interested in the per-database directories, which have
* numeric names. Note that this code will also (properly) ignore "."
@@ -134,9 +141,10 @@ ResetUnloggedRelationsInTablespaceDir(const char *tsdirname, int op)
if (strspn(de->d_name, "0123456789") != strlen(de->d_name))
continue;
+ dbOid = atoi(de->d_name);
snprintf(dbspace_path, sizeof(dbspace_path), "%s/%s",
tsdirname, de->d_name);
- ResetUnloggedRelationsInDbspaceDir(dbspace_path, op);
+ ResetUnloggedRelationsInDbspaceDir(dbspace_path, op, spcOid, dbOid);
}
FreeDir(ts_dir);
@@ -146,7 +154,8 @@ ResetUnloggedRelationsInTablespaceDir(const char *tsdirname, int op)
* Process one per-dbspace directory for ResetUnloggedRelations
*/
static void
-ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op)
+ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op,
+ Oid spcOid, Oid dbOid)
{
DIR *dbspace_dir;
struct dirent *de;
@@ -187,7 +196,7 @@ ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op)
/* Skip anything that doesn't look like a relation data file. */
if (!parse_filename_for_nontemp_relation(de->d_name, &oidchars,
- &forkNum))
+ &forkNum, NULL))
continue;
/* Also skip it unless this is the init fork. */
@@ -229,7 +238,7 @@ ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op)
/* Skip anything that doesn't look like a relation data file. */
if (!parse_filename_for_nontemp_relation(de->d_name, &oidchars,
- &forkNum))
+ &forkNum, NULL))
continue;
/* We never remove the init fork. */
@@ -279,13 +288,14 @@ ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op)
{
ForkNumber forkNum;
int oidchars;
+ int segment;
char oidbuf[OIDCHARS + 1];
char srcpath[MAXPGPATH * 2];
char dstpath[MAXPGPATH];
/* Skip anything that doesn't look like a relation data file. */
if (!parse_filename_for_nontemp_relation(de->d_name, &oidchars,
- &forkNum))
+ &forkNum, &segment))
continue;
/* Also skip it unless this is the init fork. */
@@ -305,7 +315,13 @@ ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op)
/* OK, we're ready to perform the actual copy. */
elog(DEBUG2, "copying %s to %s", srcpath, dstpath);
- copy_file(srcpath, dstpath);
+ {
+ RelFileNode srcNode = {spcOid, dbOid, atol(oidbuf)};
+ RelFileNode dstNode = srcNode;
+
+ copy_file(srcpath, dstpath, &srcNode, &dstNode,
+ INIT_FORKNUM, MAIN_FORKNUM, segment);
+ }
}
FreeDir(dbspace_dir);
@@ -327,7 +343,7 @@ ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op)
/* Skip anything that doesn't look like a relation data file. */
if (!parse_filename_for_nontemp_relation(de->d_name, &oidchars,
- &forkNum))
+ &forkNum, NULL))
continue;
/* Also skip it unless this is the init fork. */
@@ -372,9 +388,10 @@ ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op)
*/
bool
parse_filename_for_nontemp_relation(const char *name, int *oidchars,
- ForkNumber *fork)
+ ForkNumber *fork, int *segment)
{
int pos;
+ int segstart = 0;
/* Look for a non-empty string of digits (that isn't too long). */
for (pos = 0; isdigit((unsigned char) name[pos]); ++pos)
@@ -401,6 +418,7 @@ parse_filename_for_nontemp_relation(const char *name, int *oidchars,
{
int segchar;
+ segstart = pos + 1;
for (segchar = 1; isdigit((unsigned char) name[pos + segchar]); ++segchar)
;
if (segchar <= 1)
@@ -411,5 +429,14 @@ parse_filename_for_nontemp_relation(const char *name, int *oidchars,
/* Now we should be at the end. */
if (name[pos] != '\0')
return false;
+
+ if (segment != NULL)
+ {
+ if (segstart == 0)
+ *segment = 0;
+ else
+ *segment = atoi(name + segstart);
+ }
+
return true;
}
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index dfbda5458f..1c3d9463d4 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -17,6 +17,7 @@
#include "access/htup_details.h"
#include "access/itup.h"
#include "access/xlog.h"
+#include "common/string.h"
#include "storage/checksum.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
@@ -81,11 +82,8 @@ bool
PageIsVerified(Page page, BlockNumber blkno)
{
PageHeader p = (PageHeader) page;
- size_t *pagebytes;
- int i;
bool checksum_failure = false;
bool header_sane = false;
- bool all_zeroes = false;
uint16 checksum = 0;
/*
@@ -118,26 +116,8 @@ PageIsVerified(Page page, BlockNumber blkno)
return true;
}
- /*
- * Check all-zeroes case. Luckily BLCKSZ is guaranteed to always be a
- * multiple of size_t - and it's much faster to compare memory using the
- * native word size.
- */
- StaticAssertStmt(BLCKSZ == (BLCKSZ / sizeof(size_t)) * sizeof(size_t),
- "BLCKSZ has to be a multiple of sizeof(size_t)");
-
- all_zeroes = true;
- pagebytes = (size_t *) page;
- for (i = 0; i < (BLCKSZ / sizeof(size_t)); i++)
- {
- if (pagebytes[i] != 0)
- {
- all_zeroes = false;
- break;
- }
- }
-
- if (all_zeroes)
+ /* Check all-zeroes case */
+ if (IsAllZero((char *) page, BLCKSZ))
return true;
/*
diff --git a/src/backend/storage/smgr/Makefile b/src/backend/storage/smgr/Makefile
index 2b95cb0df1..0601ea7894 100644
--- a/src/backend/storage/smgr/Makefile
+++ b/src/backend/storage/smgr/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/storage/smgr
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
-OBJS = md.o smgr.o smgrtype.o
+OBJS = encryption.o md.o smgr.o smgrtype.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/smgr/encryption.c b/src/backend/storage/smgr/encryption.c
new file mode 100644
index 0000000000..cef36b1f5a
--- /dev/null
+++ b/src/backend/storage/smgr/encryption.c
@@ -0,0 +1,814 @@
+/*-------------------------------------------------------------------------
+ *
+ * encryption.c
+ * This code handles encryption and decryption of data.
+ *
+ * Encryption is done by extension modules loaded by encryption_library GUC.
+ * The extension module must register itself and provide a cryptography
+ * implementation. Key setup is left to the extension module.
+ *
+ *
+ * Copyright (c) 2016, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/storage/smgr/encryption.c
+ *
+ * NOTES
+ * This file is compiled as both front-end and backend code, so it
+ * may not use ereport, server-defined static variables, etc.
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include <unistd.h>
+
+#include "common/fe_memutils.h"
+#include "common/sha2.h"
+#include "common/string.h"
+#include "catalog/pg_control.h"
+#ifndef FRONTEND
+#include "pgstat.h"
+#endif /* FRONTEND */
+#include "storage/bufpage.h"
+#include "storage/encryption.h"
+#include "storage/fd.h"
+#include "utils/memutils.h"
+#include "miscadmin.h"
+#include "fmgr.h"
+#include "port.h"
+
+#ifdef USE_ENCRYPTION
+#include <openssl/conf.h>
+#include <openssl/evp.h>
+#include <openssl/err.h>
+#endif
+
+#ifdef USE_ENCRYPTION
+unsigned char *encryption_key = NULL;
+const char *encryption_key_prefix = "encryption_key=";
+const char *encryption_pwd_prefix = "encryption_password=";
+#endif
+
+bool data_encrypted = false;
+
+#ifdef USE_ENCRYPTION
+char *encryption_key_command = NULL;
+char *encryption_buffer = NULL;
+Size encryption_buf_size = 0;
+
+static bool initialized = false;
+
+static void setup_encryption_internal(void);
+static char *run_encryption_key_command(bool *is_key_p, size_t *len_p);
+static void evp_error(void);
+
+/*
+ * Pointer to the KDF parameters.
+ *
+ * XXX Rename this and the write / read functions so they contain the
+ * 'keysetup' string?
+ */
+KDFParamsData *KDFParams = NULL;
+
+/*
+ * Initialize KDFParamsData and write it to a file.
+ *
+ * This is very similar to WriteControlFile().
+ */
+#ifndef FRONTEND
+extern void
+write_kdf_file(void)
+{
+ KDFParamsPBKDF2 *params;
+ int i,
+ fd;
+
+ StaticAssertStmt(sizeof(KDFParamsData) <= KDF_PARAMS_FILE_SIZE,
+ "kdf file is too large for atomic disk writes");
+
+ /*
+ * The initialization should not be repeated.
+ */
+ Assert(KDFParams == NULL);
+
+ KDFParams = MemoryContextAllocZero(TopMemoryContext,
+ KDF_PARAMS_FILE_SIZE);
+ KDFParams->function = KDF_OPENSSL_PKCS5_PBKDF2_HMAC_SHA;
+ params = &KDFParams->data.pbkdf2;
+ params->niter = ENCRYPTION_KDF_NITER;
+ for (i = 0; i < ENCRYPTION_KDF_SALT_LEN; i++)
+ params->salt[i] = (unsigned char) random();
+
+ /* Contents are protected with a CRC */
+ INIT_CRC32C(KDFParams->crc);
+ COMP_CRC32C(KDFParams->crc,
+ (char *) KDFParams,
+ offsetof(KDFParamsData, crc));
+ FIN_CRC32C(KDFParams->crc);
+
+ fd = BasicOpenFile(KDF_PARAMS_FILE,
+ O_RDWR | O_CREAT | O_EXCL | PG_BINARY);
+ if (fd < 0)
+ ereport(PANIC,
+ (errcode_for_file_access(),
+ errmsg("could not create key setup file \"%s\": %m",
+ KDF_PARAMS_FILE)));
+
+ pgstat_report_wait_start(WAIT_EVENT_KDF_FILE_WRITE);
+ if (write(fd, KDFParams, KDF_PARAMS_FILE_SIZE) != KDF_PARAMS_FILE_SIZE)
+ {
+ /* if write didn't set errno, assume problem is no disk space */
+ if (errno == 0)
+ errno = ENOSPC;
+ ereport(PANIC,
+ (errcode_for_file_access(),
+ errmsg("could not write to key setup file: %m")));
+ }
+ pgstat_report_wait_end();
+
+ pgstat_report_wait_start(WAIT_EVENT_KDF_FILE_SYNC);
+ if (pg_fsync(fd) != 0)
+ ereport(PANIC,
+ (errcode_for_file_access(),
+ errmsg("could not fsync key setup file: %m")));
+ pgstat_report_wait_end();
+
+ if (close(fd))
+ ereport(PANIC,
+ (errcode_for_file_access(),
+ errmsg("could not close key setup file: %m")));
+}
+
+/*
+ * Read KDFParamsData from file and store it in local memory.
+ *
+ * postmaster should call the function early enough for any other process to
+ * inherit valid pointer to the data.
+ */
+extern void
+read_kdf_file(void)
+{
+ pg_crc32c crc;
+ int fd;
+
+ KDFParams = MemoryContextAllocZero(TopMemoryContext,
+ KDF_PARAMS_FILE_SIZE);
+
+ fd = BasicOpenFile(KDF_PARAMS_FILE, O_RDONLY | PG_BINARY);
+
+ if (fd < 0)
+ {
+ ereport(PANIC,
+ (errcode_for_file_access(),
+ errmsg("could not open key setup file \"%s\": %m",
+ KDF_PARAMS_FILE)));
+ }
+
+ pgstat_report_wait_start(WAIT_EVENT_KDF_FILE_READ);
+
+ if (read(fd, KDFParams, sizeof(KDFParamsData)) != sizeof(KDFParamsData))
+ {
+ ereport(PANIC,
+ (errcode_for_file_access(),
+ errmsg("could not read from key setup file: %m")));
+ }
+
+ pgstat_report_wait_end();
+
+ close(fd);
+
+ /* Now check the CRC. */
+ INIT_CRC32C(crc);
+ COMP_CRC32C(crc,
+ (char *) KDFParams,
+ offsetof(KDFParamsData, crc));
+ FIN_CRC32C(crc);
+
+ if (!EQ_CRC32C(crc, KDFParams->crc))
+ ereport(FATAL,
+ (errmsg("incorrect checksum in key setup file")));
+
+ if (KDFParams->function != KDF_OPENSSL_PKCS5_PBKDF2_HMAC_SHA)
+ ereport(FATAL,
+ (errmsg("unsupported KDF function %d", KDFParams->function)));
+}
+#endif /* FRONTEND */
+#endif /* USE_ENCRYPTION */
+
+/*
+ * Encrypts a fixed value into *buf to verify that encryption key is correct.
+ * Caller provided buf needs to be able to hold at least ENCRYPTION_SAMPLE_SIZE
+ * bytes.
+ */
+void
+sample_encryption(char *buf)
+{
+#ifdef USE_ENCRYPTION
+ char tweak[TWEAK_SIZE];
+ int i;
+
+ for (i = 0; i < TWEAK_SIZE; i++)
+ tweak[i] = i;
+
+ encrypt_block("postgresqlcrypt", buf, ENCRYPTION_SAMPLE_SIZE, tweak);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+}
+
+/*
+ * Encrypts one block of data with a specified tweak value. May only be called
+ * when encryption_enabled is true.
+ *
+ * Input and output buffer may point to the same location.
+ *
+ * "size" must be a (non-zero) multiple of ENCRYPTION_BLOCK.
+ *
+ * "tweak" value must be TWEAK_SIZE bytes long.
+ *
+ * All-zero blocks are not encrypted to correctly handle relation extension,
+ * and also to simplify handling of holes created by seek past EOF and
+ * consequent write (see buffile.c).
+ */
+void
+encrypt_block(const char *input, char *output, Size size, const char *tweak)
+{
+#ifdef USE_ENCRYPTION
+ Assert(size >= ENCRYPTION_BLOCK && size % ENCRYPTION_BLOCK == 0);
+ Assert(initialized);
+
+ /*
+ * The EVP API does not seem to expect the output buffer to be equal to
+ * the input. Ensure that we pass separate pointers.
+ */
+ if (input == output)
+ {
+ if (size > encryption_buf_size)
+ enlarge_encryption_buffer(size);
+
+ memcpy(encryption_buffer, input, size);
+ input = encryption_buffer;
+ }
+
+ if (IsAllZero(input, size))
+ memset(output, 0, size);
+ else
+ {
+ int out_size;
+ EVP_CIPHER_CTX *ctx;
+
+ if ((ctx = EVP_CIPHER_CTX_new()) == NULL)
+ evp_error();
+
+ if (EVP_EncryptInit_ex(ctx, EVP_aes_256_xts(), NULL, encryption_key,
+ (unsigned char *) tweak) != 1)
+ evp_error();
+
+ /*
+ * No padding is needed, the input block size should already be a
+ * multiple of ENCRYPTION_BLOCK_OPENSSL.
+ */
+ EVP_CIPHER_CTX_set_padding(ctx, 0);
+
+ Assert(EVP_CIPHER_CTX_block_size(ctx) == ENCRYPTION_BLOCK_OPENSSL);
+ Assert(EVP_CIPHER_CTX_iv_length(ctx) == TWEAK_SIZE);
+ Assert(EVP_CIPHER_CTX_key_length(ctx) == ENCRYPTION_KEY_LENGTH);
+
+ /*
+ * Do the actual encryption. As the padding is disabled,
+ * EVP_EncryptFinal_ex() won't be needed.
+ */
+ if (EVP_EncryptUpdate(ctx, (unsigned char *) output, &out_size,
+ (unsigned char *) input, size) != 1)
+ evp_error();
+
+ /*
+ * The input size is a multiple of ENCRYPTION_BLOCK_OPENSSL, so the
+ * output of AES-XTS should meet this condition.
+ */
+ Assert(out_size == size);
+
+ if (EVP_CIPHER_CTX_cleanup(ctx) != 1)
+ evp_error();
+ }
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+}
+
+/*
+ * Decrypts one block of data with a specified tweak value. May only be called
+ * when encryption_enabled is true.
+ *
+ * Input and output buffer may point to the same location.
+ *
+ * "size" must be a (non-zero) multiple of ENCRYPTION_BLOCK.
+ *
+ * "tweak" value must be TWEAK_SIZE bytes long.
+ *
+ * All-zero blocks are not decrypted to correctly handle relation extension,
+ * and also to simplify handling of holes created by seek past EOF and
+ * consequent write (see buffile.c).
+ */
+void
+decrypt_block(const char *input, char *output, Size size, const char *tweak)
+{
+#ifdef USE_ENCRYPTION
+ Assert(size >= ENCRYPTION_BLOCK && size % ENCRYPTION_BLOCK == 0);
+ Assert(initialized);
+
+ /*
+ * The EVP API does not seem to expect the output buffer to be equal to
+ * the input. Ensure that we pass separate pointers.
+ */
+ if (input == output)
+ {
+ if (size > encryption_buf_size)
+ enlarge_encryption_buffer(size);
+
+ memcpy(encryption_buffer, input, size);
+ input = encryption_buffer;
+ }
+
+ if (IsAllZero(input, size))
+ memset(output, 0, size);
+ else
+ {
+ int out_size;
+ EVP_CIPHER_CTX *ctx;
+
+ if ((ctx = EVP_CIPHER_CTX_new()) == NULL)
+ evp_error();
+
+ if (EVP_DecryptInit_ex(ctx, EVP_aes_256_xts(), NULL, encryption_key,
+ (unsigned char *) tweak) != 1)
+ evp_error();
+
+ /* The same considerations apply below as those in encrypt_block(). */
+ EVP_CIPHER_CTX_set_padding(ctx, 0);
+ Assert(EVP_CIPHER_CTX_block_size(ctx) == ENCRYPTION_BLOCK_OPENSSL);
+ Assert(EVP_CIPHER_CTX_iv_length(ctx) == TWEAK_SIZE);
+ Assert(EVP_CIPHER_CTX_key_length(ctx) == ENCRYPTION_KEY_LENGTH);
+
+ if (EVP_DecryptUpdate(ctx, (unsigned char *) output, &out_size,
+ (unsigned char *) input, size) != 1)
+ evp_error();
+
+ Assert(out_size == size);
+
+ if (EVP_CIPHER_CTX_cleanup(ctx) != 1)
+ evp_error();
+ }
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+}
+
+/*
+ * Report an error in an universal way so that caller does not have to care
+ * whether it executes in backend or front-end.
+ */
+void
+encryption_error(bool fatal, char *message)
+{
+#ifndef FRONTEND
+ elog(fatal ? FATAL : INFO, "%s", message);
+#else
+ fprintf(stderr, "%s\n", message);
+ if (fatal)
+ exit(EXIT_FAILURE);
+#endif
+}
+
+/*
+ * Initialize encryption subsystem for use. Must be called before any
+ * encryptable data is read from or written to data directory.
+ */
+void
+setup_encryption(bool bootstrap)
+{
+#ifdef USE_ENCRYPTION
+ char *credentials;
+ bool is_key;
+ size_t len;
+
+ credentials = run_encryption_key_command(&is_key, &len);
+
+ /*
+ * Setup KDF if we need to derive the key from a password.
+ *
+ * Front-ends always need the key because some of them are not aware of
+ * the data directory and thus they'd need one more command line option to
+ * find the key setup file.
+ */
+#ifndef FRONTEND
+ if (!is_key)
+ {
+ if (bootstrap)
+ {
+ write_kdf_file();
+ }
+ else
+ read_kdf_file();
+ }
+#endif /* FRONTEND */
+
+ setup_encryption_key(credentials, is_key, len);
+ pfree(credentials);
+ setup_encryption_internal();
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+}
+
+#ifdef USE_ENCRYPTION
+static void
+setup_encryption_internal(void)
+{
+ /*
+ * Setup OpenSSL.
+ *
+ * None of these functions should return a value or raise error.
+ */
+ ERR_load_crypto_strings();
+ OpenSSL_add_all_algorithms();
+ OPENSSL_config(NULL);
+
+ /*
+ * It makes no sense to initialize the encryption multiple times.
+ */
+ Assert(!initialized);
+
+ initialized = true;
+}
+#endif /* USE_ENCRYPTION */
+
+/*
+ * If credentials is a key, just copy it to encryption_key. If it's a
+ * password, derive the key from it.
+ */
+void
+setup_encryption_key(char *credentials, bool is_key, size_t len)
+{
+#ifdef USE_ENCRYPTION
+ Assert(credentials != NULL);
+ Assert(encryption_key == NULL);
+
+ /*
+ * Although the function should be called in the TopMemoryContext, we
+ * should be pretty sure that the key does not become garbage due to
+ * pfree(). The cluster should crash in such a case, but if it did not
+ * happen immediately, some data could be encrypted using an invalid key
+ * and therefore become lost.
+ */
+#ifndef FRONTEND
+ encryption_key = (unsigned char *) MemoryContextAllocZero(TopMemoryContext,
+ ENCRYPTION_KEY_LENGTH);
+#else
+ encryption_key = (unsigned char *) palloc(ENCRYPTION_KEY_LENGTH);
+#endif /* FRONTEND */
+ if (is_key)
+ {
+ Assert(len = ENCRYPTION_KEY_LENGTH);
+ memcpy(encryption_key, credentials, len);
+ }
+ else
+ {
+ KDFParamsPBKDF2 *params;
+ int rc;
+
+ /*
+ * The file contains password so we need the KDF parameters to turn it
+ * to key.
+ */
+ if (KDFParams == NULL)
+ {
+#ifndef FRONTEND
+ ereport(FATAL,
+ (errmsg("this instance does not accept encryption password"),
+ errdetail("Encryption key was probably used to initialize the instance.")));
+#else
+ encryption_error(true,
+ "this instance does not accept encryption password.\n"
+ "Encryption key was probably used to initialize the instance.\n");
+#endif /* FRONTEND */
+ }
+
+ /*
+ * Turn the password into the encryption key.
+ */
+ params = &KDFParams->data.pbkdf2;
+ rc = PKCS5_PBKDF2_HMAC(credentials,
+ len,
+ params->salt,
+ ENCRYPTION_KDF_SALT_LEN,
+ params->niter,
+ EVP_sha(),
+ ENCRYPTION_KEY_LENGTH,
+ encryption_key);
+
+ if (rc != 1)
+ {
+#ifndef FRONTEND
+ ereport(FATAL,
+ (errmsg("failed to derive key from password")));
+#else
+ encryption_error(true, "failed to derive key from password");
+#endif /* FRONTEND */
+ }
+ }
+
+#else /* USE_ENCRYPTION */
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+}
+
+#ifdef USE_ENCRYPTION
+void
+enlarge_encryption_buffer(Size new_size)
+{
+ Assert(new_size > 0);
+
+ /*
+ * Shrinkage is not the use case for this routine.
+ */
+ if (new_size <= encryption_buf_size)
+ return;
+
+ /*
+ * Allocate a new chunk if nothing is there yet, else reallocate the
+ * existing one.
+ */
+ if (encryption_buf_size == 0)
+#ifndef FRONTEND
+ encryption_buffer = (char *) MemoryContextAlloc(TopMemoryContext,
+ new_size);
+#else
+ encryption_buffer = (char *) palloc(new_size);
+#endif /* FRONTEND */
+ else
+ encryption_buffer = (char *) repalloc(encryption_buffer, new_size);
+ encryption_buf_size = new_size;
+}
+
+/*
+ * Run the command stored in encryption_key_command and return the key or
+ * password.
+ *
+ * *is_key_p receives true if the command returns key, false if it's
+ * password. *len_p receives length of the data.
+ */
+static char *
+run_encryption_key_command(bool *is_key_p, size_t *len_p)
+{
+ FILE *fp;
+ char *buf,
+ *result;
+ bool is_key;
+ size_t key_pref_len,
+ pwd_pref_len,
+ read_len,
+ bytes_read;
+ size_t buf_size,
+ result_size;
+ int i;
+
+ if (encryption_key_command == NULL ||
+ !strlen(encryption_key_command))
+ {
+ /*
+ * encryption_key_command should have been set by initdb. It's weird
+ * if it was not, but there's no better recommendation we can give the
+ * user.
+ */
+#ifndef FRONTEND
+ ereport(FATAL,
+ (errmsg("encryption key not provided"),
+ errdetail("The database cluster was initialized with encryption"
+ " but the server was started without an encryption key."),
+ errhint("Set the encryption_key_command configuration variable.")));
+#else /* FRONTEND */
+ encryption_error(true,
+ "The database cluster was initialized with encryption"
+ " but the server was started without an encryption key. "
+ "Set the encryption_key_command configuration variable.\n");
+#endif /* FRONTEND */
+ }
+
+ encryption_error(false,
+ psprintf("Executing \"%s\" to set up encryption key",
+ encryption_key_command));
+
+ fp = popen(encryption_key_command, "r");
+ if (fp == NULL)
+ encryption_error(true,
+ psprintf("Failed to execute encryption_key_command \"%s\"",
+ encryption_key_command));
+
+ /*
+ * Check which prefix the file starts with.
+ *
+ * The prefixes probably won't change after the release but they might
+ * change during development. The reading logic should be generic so that
+ * change of prefix length requires no additional coding.
+ */
+ key_pref_len = strlen(encryption_key_prefix);
+ pwd_pref_len = strlen(encryption_pwd_prefix);
+
+ /*
+ * The buffer must accommodate either prefix.
+ */
+ buf_size = Max(key_pref_len, pwd_pref_len);
+ buf = (char *) palloc(buf_size);
+
+ /*
+ * Read as few bytes as necessary so that we don't have to move back in
+ * the buffer if the first comparison does not match.
+ */
+ read_len = Min(key_pref_len, pwd_pref_len);
+
+ if (fread(buf, 1, read_len, fp) != read_len)
+ encryption_error(true, "Not enough data received from encryption_key_command");
+
+ if (read_len == key_pref_len &&
+ strncmp(buf, encryption_key_prefix, key_pref_len) == 0)
+ is_key = true;
+ else if (read_len == pwd_pref_len &&
+ strncmp(buf, encryption_pwd_prefix, pwd_pref_len) == 0)
+ is_key = false;
+ else if (buf_size > read_len)
+ {
+ size_t len_diff;
+
+ /*
+ * Read enough data so that one of the prefixes must match.
+ */
+ len_diff = buf_size - read_len;
+ if (fread(buf + read_len, 1, len_diff, fp) != len_diff)
+ encryption_error(true,
+ "Not enough data received from encryption_key_command");
+ read_len += len_diff;
+
+ /*
+ * Try to match the prefixes again.
+ */
+ if (read_len == key_pref_len &&
+ strncmp(buf, encryption_key_prefix, key_pref_len) == 0)
+ is_key = true;
+ else if (read_len == pwd_pref_len &&
+ strncmp(buf, encryption_pwd_prefix, pwd_pref_len) == 0)
+ is_key = false;
+ else
+ encryption_error(true,
+ "Unknown data received from encryption_key_command");
+ }
+
+ *is_key_p = is_key;
+
+ /*
+ * Read the actual credentials.
+ */
+ read_len = is_key ? ENCRYPTION_KEY_LENGTH * 2 : ENCRYPTION_PWD_MAX_LENGTH;
+
+ /*
+ * One extra character can be read and this should be nothing but line
+ * delimiter, see below.
+ */
+ read_len++;
+
+ if (read_len > buf_size)
+ {
+ buf = (char *) repalloc(buf, read_len);
+ buf_size = read_len;
+ }
+
+ bytes_read = fread(buf, 1, read_len, fp);
+
+ if ((is_key && bytes_read < read_len) ||
+ (!is_key && bytes_read < (ENCRYPTION_PWD_MIN_LENGTH + 1)))
+ {
+ if (feof(fp))
+ encryption_error(true,
+ "Not enough data provided by encryption_key_command");
+ else
+ encryption_error(true,
+ psprintf("encryption_key_command returned error code %d",
+ ferror(fp)));
+ }
+ Assert(bytes_read > 0);
+
+ if (buf[bytes_read - 1] != '\n')
+ encryption_error(true, "Encryption key is too long.");
+
+ /*
+ * The credentials must not contain line delimiter.
+ *
+ * If that was allowed, the line delimiter would have to terminate
+ * password even if more characters followed, however those extra
+ * characters would be useless. Since distinguishing of the special (still
+ * legal) case "password\n<EOF>" would make the checks less simple and
+ * since the delimiter itself is useless in the case of a key (because the
+ * key has constant length), let's prohibit the line delimiters anywhere
+ * except for the last position.
+ */
+ for (i = 0; i < bytes_read; i++)
+ if (buf[i] == '\n' && i < (bytes_read - 1))
+ encryption_error(true,
+ "Neither password nor key may contain line break.");
+
+ /*
+ * For a key the result size is different from the amount of data read.
+ */
+ result_size = is_key ? ENCRYPTION_KEY_LENGTH : ENCRYPTION_PWD_MAX_LENGTH;
+
+ result = (char *) palloc(result_size);
+
+ if (is_key)
+ {
+ int i;
+
+ for (i = 0; i < ENCRYPTION_KEY_LENGTH; i++)
+ {
+ if (sscanf(buf + 2 * i, "%2hhx", result + i) == 0)
+ encryption_error(true,
+ psprintf("Invalid character in encryption key at position %d",
+ 2 * i));
+ }
+ *len_p = ENCRYPTION_KEY_LENGTH;
+ }
+ else
+ {
+ /*
+ * Ignore the line delimiter.
+ */
+ *len_p = bytes_read - 1;
+ memcpy(result, buf, *len_p);
+
+ }
+
+ /*
+ * No extra data is allowed.
+ */
+ if (fread(buf, 1, 1, fp) > 0)
+ encryption_error(true,
+ "Credentials are followed by useless data");
+
+
+ pfree(buf);
+ pclose(fp);
+
+ return result;
+}
+#endif /* USE_ENCRYPTION */
+
+/*
+ * Error callback for openssl.
+ */
+#ifdef USE_ENCRYPTION
+static void
+evp_error(void)
+{
+ ERR_print_errors_fp(stderr);
+#ifndef FRONTEND
+
+ /*
+ * FATAL is the appropriate level because backend can hardly fix anything
+ * if encryption / decryption has failed.
+ *
+ * XXX Do we yet need EVP_CIPHER_CTX_cleanup() here?
+ */
+ elog(FATAL, "OpenSSL encountered error during encryption or decryption.");
+#else
+ fprintf(stderr,
+ "OpenSSL encountered error during encryption or decryption.");
+ exit(EXIT_FAILURE);
+#endif /* FRONTEND */
+}
+#endif /* USE_ENCRYPTION */
+
+/*
+ * Xlog is encrypted page at a time. Each xlog page gets a unique tweak via
+ * segment and offset. Unfortunately we can't include timeline because
+ * exitArchiveRecovery() can copy part of the last segment of the old timeline
+ * into the first segment of the new timeline.
+ *
+ * TODO Consider teaching exitArchiveRecovery() to decrypt the copied pages
+ * and encrypt them using a tweak that mentions the new timeline.
+ *
+ * The function is located here rather than some of the xlog*.c modules so
+ * that front-end applications can easily use it too.
+ */
+void
+XLogEncryptionTweak(char *tweak, XLogSegNo segment, uint32 offset)
+{
+#ifdef USE_ENCRYPTION
+ memset(tweak, 0, TWEAK_SIZE);
+ memcpy(tweak, &segment, sizeof(XLogSegNo));
+ memcpy(tweak + sizeof(XLogSegNo), &offset, sizeof(offset));
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+}
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index 70fdebbc75..a18f243d92 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -33,6 +33,7 @@
#include "postmaster/bgwriter.h"
#include "storage/fd.h"
#include "storage/bufmgr.h"
+#include "storage/encryption.h"
#include "storage/relfilenode.h"
#include "storage/smgr.h"
#include "utils/hsearch.h"
@@ -113,7 +114,14 @@ typedef struct _MdfdVec
} MdfdVec;
static MemoryContext MdCxt; /* context for all MdfdVec objects */
-
+#ifdef USE_ENCRYPTION
+/*
+ * encryption_buffer from encryption.h is not used here because of the special
+ * memory context.
+ */
+static char *md_encryption_buffer;
+static char *md_encryption_tweak;
+#endif
/*
* In some contexts (currently, standalone backends and the checkpointer)
@@ -198,6 +206,11 @@ static MdfdVec *_mdfd_getseg(SMgrRelation reln, ForkNumber forkno,
static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum,
MdfdVec *seg);
+#ifdef USE_ENCRYPTION
+static void mdtweak(char *tweak, RelFileNode *relnode, ForkNumber forknum, BlockNumber blocknum);
+static void mdencrypt(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, char *buffer);
+static void mddecrypt(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, char *buffer);
+#endif
/*
* mdinit() -- Initialize private state for magnetic disk storage manager.
@@ -242,6 +255,11 @@ mdinit(void)
HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
pendingUnlinks = NIL;
}
+
+#ifdef USE_ENCRYPTION
+ md_encryption_buffer = MemoryContextAllocZero(MdCxt, BLCKSZ);
+ md_encryption_tweak = MemoryContextAllocZero(MdCxt, TWEAK_SIZE);
+#endif
}
/*
@@ -537,6 +555,16 @@ mdextend(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
errmsg("could not seek to block %u in file \"%s\": %m",
blocknum, FilePathName(v->mdfd_vfd))));
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ mdencrypt(reln, forknum, blocknum, buffer);
+ buffer = md_encryption_buffer;
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
if ((nbytes = FileWrite(v->mdfd_vfd, buffer, BLCKSZ, WAIT_EVENT_DATA_FILE_EXTEND)) != BLCKSZ)
{
if (nbytes < 0)
@@ -734,6 +762,7 @@ mdread(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
off_t seekpos;
int nbytes;
MdfdVec *v;
+ char *buffer_read = buffer;
TRACE_POSTGRESQL_SMGR_MD_READ_START(forknum, blocknum,
reln->smgr_rnode.node.spcNode,
@@ -754,7 +783,15 @@ mdread(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
errmsg("could not seek to block %u in file \"%s\": %m",
blocknum, FilePathName(v->mdfd_vfd))));
- nbytes = FileRead(v->mdfd_vfd, buffer, BLCKSZ, WAIT_EVENT_DATA_FILE_READ);
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ buffer_read = md_encryption_buffer;
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+ nbytes = FileRead(v->mdfd_vfd, buffer_read, BLCKSZ, WAIT_EVENT_DATA_FILE_READ);
TRACE_POSTGRESQL_SMGR_MD_READ_DONE(forknum, blocknum,
reln->smgr_rnode.node.spcNode,
@@ -789,6 +826,14 @@ mdread(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
blocknum, FilePathName(v->mdfd_vfd),
nbytes, BLCKSZ)));
}
+ else if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ mddecrypt(reln, forknum, blocknum, buffer);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
}
/*
@@ -830,6 +875,15 @@ mdwrite(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
errmsg("could not seek to block %u in file \"%s\": %m",
blocknum, FilePathName(v->mdfd_vfd))));
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ mdencrypt(reln, forknum, blocknum, buffer);
+ buffer = md_encryption_buffer;
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
nbytes = FileWrite(v->mdfd_vfd, buffer, BLCKSZ, WAIT_EVENT_DATA_FILE_WRITE);
TRACE_POSTGRESQL_SMGR_MD_WRITE_DONE(forknum, blocknum,
@@ -1987,3 +2041,58 @@ _mdnblocks(SMgrRelation reln, ForkNumber forknum, MdfdVec *seg)
/* note that this calculation will ignore any partial block at EOF */
return (BlockNumber) (len / BLCKSZ);
}
+
+#ifdef USE_ENCRYPTION
+/*
+ * md files are encrypted block at a time. Tweak will alias higher numbered
+ * forks for huge tables.
+ */
+static void
+mdtweak(char *tweak, RelFileNode *relnode, ForkNumber forknum, BlockNumber blocknum)
+{
+ uint32 fork_and_block = (forknum << 24) ^ blocknum;
+
+ memcpy(tweak, relnode, sizeof(RelFileNode));
+ memcpy(tweak + sizeof(RelFileNode), &fork_and_block, 4);
+}
+
+static void
+mdencrypt(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, char *buffer)
+{
+ mdtweak(md_encryption_tweak, &(reln->smgr_rnode.node), forknum, blocknum);
+ encrypt_block(buffer, md_encryption_buffer, BLCKSZ, md_encryption_tweak);
+}
+
+static void
+mddecrypt(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, char *dest)
+{
+ mdtweak(md_encryption_tweak, &(reln->smgr_rnode.node), forknum, blocknum);
+ decrypt_block(md_encryption_buffer, dest, BLCKSZ, md_encryption_tweak);
+}
+
+/*
+ * Copying relations between tablespaces/databases means that the tweak values
+ * of each block will change. This function transcodes a series of blocks with
+ * new tweak values. Returns the new block number for convenience.
+ */
+BlockNumber
+ReencryptBlock(char *buffer, int blocks,
+ RelFileNode *srcNode, RelFileNode *dstNode,
+ ForkNumber srcForkNum, ForkNumber dstForkNum,
+ BlockNumber blockNum)
+{
+ char *cur;
+ char srcTweak[TWEAK_SIZE];
+ char dstTweak[TWEAK_SIZE];
+
+ for (cur = buffer; cur < buffer + blocks * BLCKSZ; cur += BLCKSZ)
+ {
+ mdtweak(srcTweak, srcNode, srcForkNum, blockNum);
+ mdtweak(dstTweak, dstNode, dstForkNum, blockNum);
+ decrypt_block(cur, cur, BLCKSZ, srcTweak);
+ encrypt_block(cur, cur, BLCKSZ, dstTweak);
+ blockNum++;
+ }
+ return blockNum;
+}
+#endif /* USE_ENCRYPTION */
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index f4133953be..d946729d17 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -63,6 +63,7 @@
#include "replication/walsender.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/encryption.h"
#include "storage/ipc.h"
#include "storage/proc.h"
#include "storage/procsignal.h"
@@ -3401,7 +3402,7 @@ process_postgres_switches(int argc, char *argv[], GucContext ctx,
* postmaster/postmaster.c (the option sets should not conflict) and with
* the common help() function in main/main.c.
*/
- while ((flag = getopt(argc, argv, "B:bc:C:D:d:EeFf:h:ijk:lN:nOo:Pp:r:S:sTt:v:W:-:")) != -1)
+ while ((flag = getopt(argc, argv, "B:bc:C:D:d:EeFf:h:ijK:k:lN:nOo:Pp:r:S:sTt:v:W:-:")) != -1)
{
switch (flag)
{
@@ -3459,6 +3460,12 @@ process_postgres_switches(int argc, char *argv[], GucContext ctx,
UseSemiNewlineNewline = true;
break;
+#ifdef USE_OPENSSL
+ case 'K':
+ encryption_key_command = strdup(optarg);
+ break;
+#endif /* USE_OPENSSL */
+
case 'k':
SetConfigOption("unix_socket_directories", optarg, ctx, gucsource);
break;
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 7a6994ba24..80cfe1d719 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -1522,6 +1522,7 @@ ValidatePgVersion(const char *path)
* GUC variables: lists of library names to be preloaded at postmaster
* start and at backend start
*/
+char *encryption_library_string = NULL;
char *session_preload_libraries_string = NULL;
char *shared_preload_libraries_string = NULL;
char *local_preload_libraries_string = NULL;
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index a88ea6cfc9..98f05f0bde 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -69,6 +69,7 @@
#include "replication/walsender.h"
#include "storage/bufmgr.h"
#include "storage/dsm_impl.h"
+#include "storage/encryption.h"
#include "storage/standby.h"
#include "storage/fd.h"
#include "storage/large_object.h"
@@ -193,6 +194,9 @@ static bool check_cluster_name(char **newval, void **extra, GucSource source);
static const char *show_unix_socket_permissions(void);
static const char *show_log_file_mode(void);
static const char *show_data_directory_mode(void);
+#ifdef USE_OPENSSL
+static const char *show_encryption_key_command(void);
+#endif /* USE_OPENSSL */
/* Private functions in guc-file.l that need to be called from guc.c */
static ConfigVariable *ProcessConfigFileInternal(GucContext context,
@@ -1724,6 +1728,17 @@ static struct config_bool ConfigureNamesBool[] =
},
{
+ {"data_encryption", PGC_INTERNAL, PRESET_OPTIONS,
+ gettext_noop("Shows whether data encryption is turned on for this cluster."),
+ NULL,
+ GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE
+ },
+ &data_encrypted,
+ false,
+ NULL, NULL, NULL
+ },
+
+ {
{"syslog_sequence_numbers", PGC_SIGHUP, LOGGING_WHERE,
gettext_noop("Add sequence number to syslog messages to avoid duplicate suppression."),
NULL
@@ -3889,6 +3904,19 @@ static struct config_string ConfigureNamesString[] =
NULL, NULL, NULL
},
+#ifdef USE_OPENSSL
+ {
+ {"encryption_key_command", PGC_POSTMASTER, 0,
+ gettext_noop("Sets the shell command that will be called to fetch database encryption key."),
+ NULL,
+ GUC_IS_NAME | GUC_NOT_IN_SAMPLE
+ },
+ &encryption_key_command,
+ NULL,
+ NULL, NULL, show_encryption_key_command
+ },
+#endif /* USE_OPENSSL */
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, NULL, NULL, NULL, NULL
@@ -10803,4 +10831,15 @@ show_data_directory_mode(void)
return buf;
}
+#ifdef USE_OPENSSL
+static const char *
+show_encryption_key_command(void)
+{
+ if (encryption_key_command)
+ return encryption_key_command;
+ else
+ return "(disabled)";
+}
+#endif /* USE_OPENSSL */
+
#include "guc-file.c"
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index c0d3fb8491..0fcfb6107a 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -88,6 +88,7 @@
#authentication_timeout = 1min # 1s-600s
#password_encryption = md5 # md5 or scram-sha-256
#db_user_namespace = off
+#encryption_key_command = ''
# GSSAPI using Kerberos
#krb_server_keyfile = ''
diff --git a/src/bin/Makefile b/src/bin/Makefile
index 8c11060a2f..431c406a65 100644
--- a/src/bin/Makefile
+++ b/src/bin/Makefile
@@ -21,6 +21,7 @@ SUBDIRS = \
pg_controldata \
pg_ctl \
pg_dump \
+ pg_keysetup \
pg_resetwal \
pg_rewind \
pg_test_fsync \
diff --git a/src/bin/initdb/Makefile b/src/bin/initdb/Makefile
index 8c23941930..ea7af9144f 100644
--- a/src/bin/initdb/Makefile
+++ b/src/bin/initdb/Makefile
@@ -26,7 +26,7 @@ ifneq (,$(with_system_tzdata))
override CPPFLAGS += '-DSYSTEMTZDIR="$(with_system_tzdata)"'
endif
-OBJS= initdb.o findtimezone.o localtime.o encnames.o $(WIN32RES)
+OBJS= initdb.o findtimezone.o localtime.o encnames.o encryption.o $(WIN32RES)
all: initdb
@@ -45,6 +45,9 @@ encnames.c: % : $(top_srcdir)/src/backend/utils/mb/%
localtime.c: % : $(top_srcdir)/src/timezone/%
rm -f $@ && $(LN_S) $< .
+encryption.c: % : $(top_srcdir)/src/backend/storage/smgr/%
+ rm -f $@ && $(LN_S) $< .
+
install: all installdirs
$(INSTALL_PROGRAM) initdb$(X) '$(DESTDIR)$(bindir)/initdb$(X)'
@@ -55,7 +58,7 @@ uninstall:
rm -f '$(DESTDIR)$(bindir)/initdb$(X)'
clean distclean maintainer-clean:
- rm -f initdb$(X) $(OBJS) encnames.c localtime.c
+ rm -f initdb$(X) $(OBJS) encnames.c localtime.c encryption.c
rm -rf tmp_check
# ensure that changes in datadir propagate into object file
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index 3f203c6ca6..9d58aff013 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -68,6 +68,9 @@
#include "common/restricted_token.h"
#include "common/username.h"
#include "fe_utils/string_utils.h"
+#include "storage/encryption.h"
+#include "lib/ilist.h"
+#include "mb/pg_wchar.h"
#include "getaddrinfo.h"
#include "getopt_long.h"
#include "mb/pg_wchar.h"
@@ -77,6 +80,12 @@
/* Ideally this would be in a .h file, but it hardly seems worth the trouble */
extern const char *select_default_timezone(const char *share_path);
+typedef struct
+{
+ dlist_node list_node;
+ char *value;
+} extra_option;
+
static const char *const auth_methods_host[] = {
"trust", "reject", "scram-sha-256", "md5", "password", "ident", "radius",
#ifdef ENABLE_GSS
@@ -141,9 +150,11 @@ static bool do_sync = true;
static bool sync_only = false;
static bool show_setting = false;
static bool data_checksums = false;
+static char *encr_key_cmd_str = NULL;
static char *xlog_dir = NULL;
static char *str_wal_segment_size_mb = NULL;
static int wal_segment_size_mb;
+static dlist_head extra_options = DLIST_STATIC_INIT(extra_options);
/* internal vars */
@@ -529,6 +540,7 @@ readfile(const char *path)
fclose(infile);
free(buffer);
+
result[n] = NULL;
return result;
@@ -1076,6 +1088,39 @@ pretty_wal_size(int segment_count)
return result;
}
+static void
+append_extra_options(char ***conflines)
+{
+ dlist_iter iter;
+ int n_extra = 0;
+ int n_current = 0;
+ int i = 0;
+ char **new_conflines;
+
+ dlist_foreach(iter, &extra_options)
+ {
+ n_extra++;
+ }
+ while ((*conflines)[i++] != NULL)
+ n_current++;
+
+ new_conflines = (char **) pg_malloc((n_current + n_extra + 1) * sizeof(char *));
+ for (i = 0; i < n_current; i++)
+ new_conflines[i] = (*conflines)[i];
+
+ dlist_foreach(iter, &extra_options)
+ {
+ extra_option *opt = dlist_container(extra_option, list_node, iter.cur);
+
+ new_conflines[i++] = opt->value;
+ }
+
+ new_conflines[i] = NULL;
+ pg_free(*conflines);
+ *conflines = new_conflines;
+}
+
+
/*
* set up all the config files
*/
@@ -1095,6 +1140,8 @@ setup_config(void)
conflines = readfile(conf_file);
+ append_extra_options(&conflines);
+
snprintf(repltok, sizeof(repltok), "max_connections = %d", n_connections);
conflines = replace_token(conflines, "#max_connections = 100", repltok);
@@ -1237,6 +1284,17 @@ setup_config(void)
"log_file_mode = 0640");
}
+#ifdef USE_OPENSSL
+ if (encryption_key_command)
+ {
+ snprintf(repltok, sizeof(repltok), "encryption_key_command = '%s'",
+ encryption_key_command);
+ conflines = replace_token(conflines,
+ "#encryption_key_command = ''", repltok);
+ }
+#endif /* USE_OPENSSL */
+
+
snprintf(path, sizeof(path), "%s/postgresql.conf", pg_data);
writefile(path, conflines);
@@ -1454,11 +1512,29 @@ bootstrap_template1(void)
/* Also ensure backend isn't confused by this environment var: */
unsetenv("PGCLIENTENCODING");
+#ifdef USE_OPENSSL
+ /* Prepare the -K option for the backend. */
+ if (encryption_key_command)
+ {
+ size_t len;
+
+ len = 3 + strlen(encryption_key_command) + 1;
+ encr_key_cmd_str = (char *) pg_malloc(len);
+ snprintf(encr_key_cmd_str, len, "-K %s", encryption_key_command);
+ }
+ else
+ {
+ encr_key_cmd_str = (char *) pg_malloc(1);
+ encr_key_cmd_str[0] = '\0';
+ }
+#endif /* USE_OPENSSL */
+
snprintf(cmd, sizeof(cmd),
- "\"%s\" --boot -x1 -X %u %s %s %s",
+ "\"%s\" --boot -x1 -X %u %s %s %s %s",
backend_exec,
wal_segment_size_mb * (1024 * 1024),
data_checksums ? "-k" : "",
+ encr_key_cmd_str,
boot_options,
debug ? "-d 5" : "");
@@ -2409,6 +2485,10 @@ usage(const char *progname)
printf(_("\nLess commonly used options:\n"));
printf(_(" -d, --debug generate lots of debugging output\n"));
printf(_(" -k, --data-checksums use data page checksums\n"));
+#ifdef USE_OPENSSL
+ printf(_(" -K, --encryption-key-command\n"
+ " command that returns encryption key\n"));
+#endif /* USE_OPENSSL */
printf(_(" -L DIRECTORY where to find the input files\n"));
printf(_(" -n, --no-clean do not clean up after errors\n"));
printf(_(" -N, --no-sync do not wait for changes to be written safely to disk\n"));
@@ -2515,7 +2595,6 @@ setup_pgdata(void)
putenv(pgdata_set_env);
}
-
void
setup_bin_paths(const char *argv0)
{
@@ -3028,8 +3107,8 @@ initialize_data_directory(void)
fflush(stdout);
snprintf(cmd, sizeof(cmd),
- "\"%s\" %s template1 >%s",
- backend_exec, backend_options,
+ "\"%s\" %s %s template1 >%s",
+ backend_exec, backend_options, encr_key_cmd_str,
DEVNULL);
PG_CMD_OPEN;
@@ -3070,6 +3149,23 @@ initialize_data_directory(void)
check_ok();
}
+static void
+parse_extra_option_arg(char *optarg)
+{
+ extra_option *opt;
+
+ if (!strchr(optarg, '='))
+ {
+ fprintf(stderr, _("Option value is not in key=value format"));
+ exit(1);
+ }
+
+ opt = malloc(sizeof(extra_option));
+ if (asprintf(&opt->value, "%s\n", optarg) < 0)
+ exit(1);
+
+ dlist_push_tail(&extra_options, &opt->list_node);
+}
int
main(int argc, char *argv[])
@@ -3104,6 +3200,9 @@ main(int argc, char *argv[])
{"waldir", required_argument, NULL, 'X'},
{"wal-segsize", required_argument, NULL, 12},
{"data-checksums", no_argument, NULL, 'k'},
+#ifdef USE_OPENSSL
+ {"encryption-key-command", required_argument, NULL, 'K'},
+#endif /* USE_OPENSSL */
{"allow-group-access", no_argument, NULL, 'g'},
{NULL, 0, NULL, 0}
};
@@ -3146,7 +3245,7 @@ main(int argc, char *argv[])
/* process command-line options */
- while ((c = getopt_long(argc, argv, "dD:E:kL:nNU:WA:sST:X:g", long_options, &option_index)) != -1)
+ while ((c = getopt_long(argc, argv, "dD:E:kK:L:nNU:WA:sST:X:g", long_options, &option_index)) != -1)
{
switch (c)
{
@@ -3198,6 +3297,14 @@ main(int argc, char *argv[])
case 'k':
data_checksums = true;
break;
+#ifdef USE_OPENSSL
+ case 'K':
+ encryption_key_command = pg_strdup(optarg);
+ break;
+#endif /* USE_OPENSSL */
+ case 'c':
+ parse_extra_option_arg(optarg);
+ break;
case 'L':
share_path = pg_strdup(optarg);
break;
@@ -3371,7 +3478,12 @@ main(int argc, char *argv[])
if (pwprompt || pwfilename)
get_su_pwd();
- printf("\n");
+#ifdef USE_OPENSSL
+ if (encryption_key_command)
+ printf(_("Data encryption is enabled.\n"));
+ else
+ printf(_("Data encryption is disabled.\n"));
+#endif /* USE_OPENSSL */
initialize_data_directory();
diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c
index 895a51f89d..32c702a978 100644
--- a/src/bin/pg_controldata/pg_controldata.c
+++ b/src/bin/pg_controldata/pg_controldata.c
@@ -340,5 +340,14 @@ main(int argc, char *argv[])
ControlFile->data_checksum_version);
printf(_("Mock authentication nonce: %s\n"),
mock_auth_nonce_str);
+ printf(_("Data encryption: %s\n"),
+ ControlFile->data_encrypted ? _("on") : _("off"));
+ if (ControlFile->data_encrypted)
+ printf(_("Data encryption fingerprint: %08X%08X%08X%08X\n"),
+ htonl(((uint32 *) ControlFile->encryption_verification)[0]),
+ htonl(((uint32 *) ControlFile->encryption_verification)[1]),
+ htonl(((uint32 *) ControlFile->encryption_verification)[2]),
+ htonl(((uint32 *) ControlFile->encryption_verification)[3])
+ );
return 0;
}
diff --git a/src/bin/pg_keysetup/Makefile b/src/bin/pg_keysetup/Makefile
new file mode 100644
index 0000000000..c9fe1064d0
--- /dev/null
+++ b/src/bin/pg_keysetup/Makefile
@@ -0,0 +1,32 @@
+# src/bin/pg_keysetup/Makefile
+
+PGFILEDESC = "pg_keysetup - generate encryption key from a password"
+PGAPPICON=win32
+
+subdir = src/bin/pg_keysetup
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+OBJS = pg_keysetup.o $(RMGRDESCOBJS) encryption.o $(WIN32RES)
+
+override CPPFLAGS := -DFRONTEND $(CPPFLAGS)
+
+all: pg_keysetup
+
+pg_keysetup: $(OBJS) | submake-libpgport
+ $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+
+encryption.c: % : $(top_srcdir)/src/backend/storage/smgr/%
+ rm -f $@ && $(LN_S) $< .
+
+install: all installdirs
+ $(INSTALL_PROGRAM) pg_keysetup$(X) '$(DESTDIR)$(bindir)/pg_keysetup$(X)'
+
+installdirs:
+ $(MKDIR_P) '$(DESTDIR)$(bindir)'
+
+uninstall:
+ rm -f '$(DESTDIR)$(bindir)/pg_keysetup$(X)'
+
+clean distclean maintainer-clean:
+ rm -f pg_keysetup$(X) $(OBJS) encryption.c
diff --git a/src/bin/pg_keysetup/pg_keysetup.c b/src/bin/pg_keysetup/pg_keysetup.c
new file mode 100644
index 0000000000..68c33b26f7
--- /dev/null
+++ b/src/bin/pg_keysetup/pg_keysetup.c
@@ -0,0 +1,208 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_keysetup.c - Turn password into encryption key.
+ *
+ * Copyright (c) 2013-2017, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/bin/pg_keysetup/pg_keysetup.c
+ *-------------------------------------------------------------------------
+ */
+
+#define FRONTEND 1
+#include "postgres.h"
+
+#include <dirent.h>
+#include <unistd.h>
+
+#include "port/pg_crc32c.h"
+#include "storage/encryption.h"
+#include "getopt_long.h"
+
+#ifdef USE_ENCRYPTION
+static const char *progname;
+
+static void
+usage(const char *progname)
+{
+ printf(_("%s derives encryption key from a password.\n\n"),
+ progname);
+ printf(_("Usage:\n"));
+ printf(_(" %s [OPTION]...\n"), progname);
+ printf(_("\nOptions:\n"));
+ printf(_(" [-D] DATADIR data directory\n"));
+ printf(_(" -?, --help show this help, then exit\n\n"));
+ printf(_("Password is read from stdin and the key is sent to stdout\n"));
+}
+
+static void fatal_error(const char *fmt,...) pg_attribute_printf(1, 2);
+
+/*
+ * Big red button to push when things go horribly wrong.
+ */
+static void
+fatal_error(const char *fmt,...)
+{
+ va_list args;
+
+ fflush(stdout);
+
+ fprintf(stderr, _("%s: FATAL: "), progname);
+ va_start(args, fmt);
+ vfprintf(stderr, _(fmt), args);
+ va_end(args);
+ fputc('\n', stderr);
+
+ exit(EXIT_FAILURE);
+}
+
+/*
+ * This function does the same as read_kdf_file() in the backend. However most
+ * of the calls look different on frontend side, it seems better to write it
+ * here from scratch than to use too many #ifdef-#else-#endif constructs.
+ */
+static void
+read_kdf_file(char *path)
+{
+ pg_crc32c crc;
+ int fd;
+
+ KDFParams = palloc(KDF_PARAMS_FILE_SIZE);
+
+ fd = open(path, O_RDONLY | PG_BINARY, S_IRUSR);
+
+ if (fd < 0)
+ fatal_error("could not open key setup file \"%s\": %m",
+ KDF_PARAMS_FILE);
+
+ if (read(fd, KDFParams, sizeof(KDFParamsData)) != sizeof(KDFParamsData))
+ fatal_error("could not read from key setup file: %m");
+
+ close(fd);
+
+ /* Now check the CRC. */
+ INIT_CRC32C(crc);
+ COMP_CRC32C(crc,
+ (char *) KDFParams,
+ offsetof(KDFParamsData, crc));
+ FIN_CRC32C(crc);
+
+ if (!EQ_CRC32C(crc, KDFParams->crc))
+ fatal_error("incorrect checksum in key setup file");
+
+ if (KDFParams->function != KDF_OPENSSL_PKCS5_PBKDF2_HMAC_SHA)
+ fatal_error("unsupported KDF function %d", KDFParams->function);
+}
+#endif /* USE_ENCRYPTION */
+
+int
+main(int argc, char **argv)
+{
+#ifdef USE_ENCRYPTION
+ int c;
+ char *DataDir = NULL;
+ char fpath[MAXPGPATH];
+ int fd;
+ char password[ENCRYPTION_PWD_MAX_LENGTH];
+ size_t pwd_len,
+ i;
+
+ progname = get_progname(argv[0]);
+
+ if (argc > 1)
+ {
+ if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0)
+ {
+ usage(progname);
+ exit(0);
+ }
+ if (strcmp(argv[1], "--version") == 0 || strcmp(argv[1], "-V") == 0)
+ {
+ puts("pg_controldata (PostgreSQL) " PG_VERSION);
+ exit(0);
+ }
+ }
+
+ while ((c = getopt(argc, argv, "D:")) != -1)
+ {
+ switch (c)
+ {
+ case 'D':
+ DataDir = optarg;
+ break;
+
+ default:
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+ exit(1);
+ }
+ }
+
+ if (DataDir == NULL)
+ {
+ if (optind < argc)
+ DataDir = argv[optind++];
+ else
+ DataDir = getenv("PGDATA");
+ }
+
+ /* Complain if any arguments remain */
+ if (optind < argc)
+ {
+ fprintf(stderr, _("%s: too many command-line arguments (first is \"%s\")\n"),
+ progname, argv[optind]);
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+ progname);
+ exit(1);
+ }
+
+ if (DataDir == NULL)
+ {
+ fprintf(stderr, _("%s: no data directory specified\n"), progname);
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+ exit(1);
+ }
+
+ snprintf(fpath, MAXPGPATH, "%s/%s", DataDir, KDF_PARAMS_FILE);
+ fd = open(fpath, O_RDONLY | PG_BINARY, 0);
+ if (fd < 0)
+ fatal_error("could not open file \"%s\"", fpath);
+
+ read_kdf_file(fpath);
+ close(fd);
+
+ /*
+ * Read the password.
+ */
+ pwd_len = 0;
+ while (true)
+ {
+ int c = getchar();
+
+ if (c == EOF || c == '\n')
+ break;
+
+ if (pwd_len >= ENCRYPTION_PWD_MAX_LENGTH)
+ fatal_error("The password is too long");
+
+ password[pwd_len++] = c;
+ }
+
+ if (pwd_len < ENCRYPTION_PWD_MIN_LENGTH)
+ fatal_error("The password is too short");
+
+ /*
+ * Run the key derivation function.
+ */
+ setup_encryption_key(password, false, pwd_len);
+
+ /*
+ * Finally print the encryption key.
+ */
+ for (i = 0; i < ENCRYPTION_KEY_LENGTH; i++)
+ printf("%.2x", encryption_key[i]);
+ printf("\n");
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ return 0;
+}
diff --git a/src/bin/pg_resetwal/Makefile b/src/bin/pg_resetwal/Makefile
index e49d52b928..6ea9c515f7 100644
--- a/src/bin/pg_resetwal/Makefile
+++ b/src/bin/pg_resetwal/Makefile
@@ -15,13 +15,18 @@ subdir = src/bin/pg_resetwal
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-OBJS= pg_resetwal.o $(WIN32RES)
+override CPPFLAGS := -DFRONTEND $(CPPFLAGS)
+
+OBJS= pg_resetwal.o encryption.o $(WIN32RES)
all: pg_resetwal
pg_resetwal: $(OBJS) | submake-libpgport
$(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+encryption.c: % : $(top_srcdir)/src/backend/storage/smgr/%
+ rm -f $@ && $(LN_S) $< .
+
install: all installdirs
$(INSTALL_PROGRAM) pg_resetwal$(X) '$(DESTDIR)$(bindir)/pg_resetwal$(X)'
@@ -32,7 +37,7 @@ uninstall:
rm -f '$(DESTDIR)$(bindir)/pg_resetwal$(X)'
clean distclean maintainer-clean:
- rm -f pg_resetwal$(X) $(OBJS)
+ rm -f pg_resetwal$(X) $(OBJS) encryption.c
rm -rf tmp_check
check:
diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c
index d63a3a27f6..719ca91288 100644
--- a/src/bin/pg_resetwal/pg_resetwal.c
+++ b/src/bin/pg_resetwal/pg_resetwal.c
@@ -54,11 +54,11 @@
#include "common/fe_memutils.h"
#include "common/file_perm.h"
#include "common/restricted_token.h"
+#include "storage/encryption.h"
#include "storage/large_object.h"
#include "pg_getopt.h"
#include "getopt_long.h"
-
static ControlFileData ControlFile; /* pg_control values */
static XLogSegNo newXlogSegNo; /* new XLOG segment # */
static bool guessed = false; /* T if we had to guess at any values */
@@ -135,7 +135,7 @@ main(int argc, char *argv[])
}
- while ((c = getopt_long(argc, argv, "c:D:e:fl:m:no:O:x:", long_options, NULL)) != -1)
+ while ((c = getopt_long(argc, argv, "c:D:e:fK:l:m:no:O:x:", long_options, NULL)) != -1)
{
switch (c)
{
@@ -278,6 +278,12 @@ main(int argc, char *argv[])
}
break;
+#ifdef USE_ENCRYPTION
+ case 'K':
+ encryption_key_command = strdup(optarg);
+ break;
+#endif /* USE_ENCRYPTION */
+
case 'l':
if (strspn(optarg, "01234567890ABCDEFabcdef") != XLOG_FNAME_LEN)
{
@@ -415,6 +421,16 @@ main(int argc, char *argv[])
XLogFromFileName(log_fname, &minXlogTli, &minXlogSegNo, WalSegSz);
/*
+ * If the data is encrypted, we'll also have to encrypt the XLOG record
+ * below.
+ */
+ if (ControlFile.data_encrypted)
+ {
+ setup_encryption(false);
+ data_encrypted = true;
+ }
+
+ /*
* Also look at existing segment files to set up newXlogSegNo
*/
FindEndOfXLOG();
@@ -1255,6 +1271,18 @@ WriteEmptyXLOG(void)
FIN_CRC32C(crc);
record->xl_crc = crc;
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ char tweak[TWEAK_SIZE];
+
+ XLogEncryptionTweak(tweak, newXlogSegNo, 0);
+ encrypt_block(buffer, buffer, XLOG_BLCKSZ, tweak);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
/* Write the first page */
XLogFilePath(path, ControlFile.checkPointCopy.ThisTimeLineID,
newXlogSegNo, WalSegSz);
@@ -1318,6 +1346,10 @@ usage(void)
printf(_(" [-D, --pgdata=]DATADIR data directory\n"));
printf(_(" -e, --epoch=XIDEPOCH set next transaction ID epoch\n"));
printf(_(" -f, --force force update to be done\n"));
+#ifdef USE_ENCRYPTION
+ printf(_(" -K, --encryption-key-command\n"
+ " command that returns encryption key\n"));
+#endif /* USE_ENCRYPTION */
printf(_(" -l, --next-wal-file=WALFILE set minimum starting location for new WAL\n"));
printf(_(" -m, --multixact-ids=MXID,MXID set next and oldest multitransaction ID\n"));
printf(_(" -n, --dry-run no update, just show what would be done\n"));
diff --git a/src/bin/pg_rewind/Makefile b/src/bin/pg_rewind/Makefile
index 2bcfcc61af..50263c0438 100644
--- a/src/bin/pg_rewind/Makefile
+++ b/src/bin/pg_rewind/Makefile
@@ -20,7 +20,7 @@ LDFLAGS_INTERNAL += $(libpq_pgport)
OBJS = pg_rewind.o parsexlog.o xlogreader.o datapagemap.o timeline.o \
fetch.o file_ops.o copy_fetch.o libpq_fetch.o filemap.o logging.o \
- $(WIN32RES)
+ encryption.o $(WIN32RES)
EXTRA_CLEAN = xlogreader.c
@@ -32,6 +32,9 @@ pg_rewind: $(OBJS) | submake-libpq submake-libpgport
xlogreader.c: % : $(top_srcdir)/src/backend/access/transam/%
rm -f $@ && $(LN_S) $< .
+encryption.c: % : $(top_srcdir)/src/backend/storage/smgr/%
+ rm -f $@ && $(LN_S) $< .
+
install: all installdirs
$(INSTALL_PROGRAM) pg_rewind$(X) '$(DESTDIR)$(bindir)/pg_rewind$(X)'
@@ -42,7 +45,7 @@ uninstall:
rm -f '$(DESTDIR)$(bindir)/pg_rewind$(X)'
clean distclean maintainer-clean:
- rm -f pg_rewind$(X) $(OBJS) xlogreader.c
+ rm -f pg_rewind$(X) $(OBJS) xlogreader.c encryption.c
rm -rf tmp_check
check:
diff --git a/src/bin/pg_rewind/parsexlog.c b/src/bin/pg_rewind/parsexlog.c
index 40028471bf..72a0d07f84 100644
--- a/src/bin/pg_rewind/parsexlog.c
+++ b/src/bin/pg_rewind/parsexlog.c
@@ -23,7 +23,7 @@
#include "catalog/pg_control.h"
#include "catalog/storage_xlog.h"
#include "commands/dbcommands_xlog.h"
-
+#include "storage/encryption.h"
/*
* RmgrNames is an array of resource manager names, to make error messages
@@ -324,6 +324,18 @@ SimpleXLogPageRead(XLogReaderState *xlogreader, XLogRecPtr targetPagePtr,
return -1;
}
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ char tweak[TWEAK_SIZE];
+
+ XLogEncryptionTweak(tweak, xlogreadsegno, targetPageOff);
+ decrypt_block(readBuf, readBuf, XLOG_BLCKSZ, tweak);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
Assert(targetSegNo == xlogreadsegno);
*pageTLI = targetHistory[private->tliIndex].tli;
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index 9653106386..5f94559c64 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -29,6 +29,7 @@
#include "common/restricted_token.h"
#include "getopt_long.h"
#include "storage/bufpage.h"
+#include "storage/encryption.h"
static void usage(const char *progname);
@@ -71,6 +72,10 @@ usage(const char *progname)
printf(_(" -D, --target-pgdata=DIRECTORY existing data directory to modify\n"));
printf(_(" --source-pgdata=DIRECTORY source data directory to synchronize with\n"));
printf(_(" --source-server=CONNSTR source server to synchronize with\n"));
+#ifdef USE_OPENSSL
+ printf(_(" -K, --encryption-key-command=COMMAND\n"
+ " command that returns encryption key\n"));
+#endif /* USE_OPENSSL */
printf(_(" -n, --dry-run stop before modifying anything\n"));
printf(_(" -N, --no-sync do not wait for changes to be written\n"));
printf(_(" safely to disk\n"));
@@ -95,6 +100,9 @@ main(int argc, char **argv)
{"no-sync", no_argument, NULL, 'N'},
{"progress", no_argument, NULL, 'P'},
{"debug", no_argument, NULL, 3},
+#ifdef USE_OPENSSL
+ {"encryption-key-command", required_argument, NULL, 'K'},
+#endif /* USE_OPENSSL */
{NULL, 0, NULL, 0}
};
int option_index;
@@ -129,7 +137,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "D:nNP", long_options, &option_index)) != -1)
+ while ((c = getopt_long(argc, argv, "D:K:nNP", long_options, &option_index)) != -1)
{
switch (c)
{
@@ -163,6 +171,12 @@ main(int argc, char **argv)
case 2: /* --source-server */
connstr_source = pg_strdup(optarg);
break;
+#ifdef USE_OPENSSL
+ case 4: /* --encryption-key-command */
+ case 'K':
+ encryption_key_command = strdup(optarg);
+ break;
+#endif /* USE_OPENSSL */
}
}
@@ -242,6 +256,16 @@ main(int argc, char **argv)
sanityChecks();
/*
+ * Setup encryption if it's obvious that we'll have to deal with encrypted
+ * XLOG.
+ */
+ if (ControlFile_target.data_encrypted)
+ {
+ setup_encryption(false);
+ data_encrypted = true;
+ }
+
+ /*
* If both clusters are already on the same timeline, there's nothing to
* do.
*/
@@ -434,6 +458,24 @@ sanityChecks(void)
ControlFile_source.state != DB_SHUTDOWNED &&
ControlFile_source.state != DB_SHUTDOWNED_IN_RECOVERY)
pg_fatal("source data directory must be shut down cleanly\n");
+
+ /*
+ * Since slave receives XLOG stream encrypted by master, handling
+ * differently encrypted clusters is not the typical use case for
+ * pg_rewind. Yet we should check the encryption.
+ */
+ if (ControlFile_source.data_encrypted ||
+ ControlFile_target.data_encrypted)
+ {
+ if (ControlFile_source.data_encrypted !=
+ ControlFile_target.data_encrypted)
+ pg_fatal("source and target server must be both unencrypted or both encrypted\n");
+
+ if (memcmp(ControlFile_source.encryption_verification,
+ ControlFile_target.encryption_verification,
+ ENCRYPTION_SAMPLE_SIZE))
+ pg_fatal("both source and target server must use the same encryption key");
+ }
}
/*
diff --git a/src/bin/pg_upgrade/check.c b/src/bin/pg_upgrade/check.c
index 5a78d603dc..c37ad0bb65 100644
--- a/src/bin/pg_upgrade/check.c
+++ b/src/bin/pg_upgrade/check.c
@@ -267,6 +267,23 @@ check_cluster_versions(void)
GET_MAJOR_VERSION(new_cluster.bin_version))
pg_fatal("New cluster data and binary directories are from different major versions.\n");
+ /*
+ * As this is the first version that supports encryption, we actually do
+ * not expect the old cluster to be encrypted.
+ *
+ * TODO Change this logic as soon as we merge the encryption into the
+ * next, even minor version (pg_upgrade seems to accept clusters that
+ * differ only in the minor version), or for the PG core patch. Eventually
+ * we'll support the upgrade if either both clusters are unencrypted, or
+ * if both are encrypted using the same key. In the latter case, we need
+ * to ensure that the PGENCRYPTIONKEY environment variable is passed to
+ * both clusters.
+ */
+ if (old_cluster.controldata.data_encrypted)
+ pg_fatal("Old cluster is encrypted.\n");
+ if (new_cluster.controldata.data_encrypted)
+ pg_fatal("New cluster is encrypted.\n");
+
check_ok();
}
diff --git a/src/bin/pg_upgrade/controldata.c b/src/bin/pg_upgrade/controldata.c
index 0fe98a550e..d9fa8c63d1 100644
--- a/src/bin/pg_upgrade/controldata.c
+++ b/src/bin/pg_upgrade/controldata.c
@@ -58,6 +58,7 @@ get_control_data(ClusterInfo *cluster, bool live_check)
bool got_large_object = false;
bool got_date_is_int = false;
bool got_data_checksum_version = false;
+ bool got_data_encrypted = false;
char *lc_collate = NULL;
char *lc_ctype = NULL;
char *lc_monetary = NULL;
@@ -135,6 +136,13 @@ get_control_data(ClusterInfo *cluster, bool live_check)
got_data_checksum_version = true;
}
+ /* Only in <= 9.6 */
+ if (GET_MAJOR_VERSION(cluster->major_version) <= 906)
+ {
+ cluster->controldata.data_encrypted = false;
+ got_data_encrypted = true;
+ }
+
/* we have the result of cmd in "output". so parse it line by line now */
while (fgets(bufin, sizeof(bufin), output))
{
@@ -418,6 +426,34 @@ get_control_data(ClusterInfo *cluster, bool live_check)
cluster->controldata.data_checksum_version = str2uint(p);
got_data_checksum_version = true;
}
+ else if ((p = strstr(bufin, "encryption fingerprint")) != NULL)
+ {
+ int i;
+
+ p = strchr(p, ':');
+
+ if (p == NULL || strlen(p) <= 1)
+ pg_fatal("%d: controldata retrieval problem\n", __LINE__);
+
+ cluster->controldata.data_encrypted = true;
+
+ /* Skip the colon and any whitespace after it */
+ p = strchr(p, ':');
+ if (p == NULL || strlen(p) <= 1)
+ pg_fatal("%d: controldata retrieval problem\n", __LINE__);
+ p = strpbrk(p, "01234567890ABCDEF");
+ if (p == NULL || strlen(p) <= 1)
+ pg_fatal("%d: controldata retrieval problem\n", __LINE__);
+
+ /* Make sure it looks like a valid finerprint */
+ if (strspn(p, "0123456789ABCDEF") != 32)
+ pg_fatal("%d: controldata retrieval problem\n", __LINE__);
+
+ for (i = 0; i < 16; i++)
+ sscanf(p + 2 * i, "%2hhx",
+ cluster->controldata.encryption_verification + i);
+ got_data_encrypted = true;
+ }
}
pclose(output);
@@ -472,7 +508,7 @@ get_control_data(ClusterInfo *cluster, bool live_check)
!got_index || !got_toast ||
(!got_large_object &&
cluster->controldata.ctrl_ver >= LARGE_OBJECT_SIZE_PG_CONTROL_VER) ||
- !got_date_is_int || !got_data_checksum_version)
+ !got_date_is_int || !got_data_checksum_version || !got_data_encrypted)
{
if (cluster == &old_cluster)
pg_log(PG_REPORT,
@@ -538,6 +574,10 @@ get_control_data(ClusterInfo *cluster, bool live_check)
if (!got_data_checksum_version)
pg_log(PG_REPORT, " data checksum version\n");
+ /* value added in Postgres 10 */
+ if (!got_data_encrypted)
+ pg_log(PG_REPORT, " data encryption status\n");
+
pg_fatal("Cannot continue without required control information, terminating\n");
}
}
@@ -602,6 +642,18 @@ check_control_data(ControlData *oldctrl,
pg_fatal("old cluster uses data checksums but the new one does not\n");
else if (oldctrl->data_checksum_version != newctrl->data_checksum_version)
pg_fatal("old and new cluster pg_controldata checksum versions do not match\n");
+
+ if (oldctrl->data_encrypted && !newctrl->data_encrypted)
+ pg_fatal("old cluster is encrypted, but the new one is not\n");
+ else if (!oldctrl->data_encrypted && newctrl->data_encrypted)
+ pg_fatal("old cluster is not encrypted, but the new one is\n");
+ else if (oldctrl->data_encrypted && newctrl->data_encrypted)
+ {
+ if (oldctrl->encryption_verification != newctrl->encryption_verification)
+ pg_fatal("encryption keys do not match between old and new cluster\n");
+ else
+ pg_fatal("upgrading encrypted databases is not implemented yet\n"); /* TODO */
+ }
}
diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h
index 7e5e971294..66291cbf47 100644
--- a/src/bin/pg_upgrade/pg_upgrade.h
+++ b/src/bin/pg_upgrade/pg_upgrade.h
@@ -227,6 +227,8 @@ typedef struct
bool date_is_int;
bool float8_pass_by_value;
bool data_checksum_version;
+ bool data_encrypted;
+ uint8 encryption_verification[16];
} ControlData;
/*
diff --git a/src/bin/pg_waldump/Makefile b/src/bin/pg_waldump/Makefile
index f5957bd75a..5a8bd1fba9 100644
--- a/src/bin/pg_waldump/Makefile
+++ b/src/bin/pg_waldump/Makefile
@@ -7,8 +7,8 @@ subdir = src/bin/pg_waldump
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-OBJS = pg_waldump.o compat.o xlogreader.o rmgrdesc.o \
- $(RMGRDESCOBJS) $(WIN32RES)
+OBJS = pg_waldump.o compat.o xlogreader.o rmgrdesc.o encryption.o \
+ $(RMGRDESCOBJS) encryption.o $(WIN32RES)
override CPPFLAGS := -DFRONTEND $(CPPFLAGS)
@@ -27,6 +27,9 @@ xlogreader.c: % : $(top_srcdir)/src/backend/access/transam/%
$(RMGRDESCSOURCES): % : $(top_srcdir)/src/backend/access/rmgrdesc/%
rm -f $@ && $(LN_S) $< .
+encryption.c: % : $(top_srcdir)/src/backend/storage/smgr/%
+ rm -f $@ && $(LN_S) $< .
+
install: all installdirs
$(INSTALL_PROGRAM) pg_waldump$(X) '$(DESTDIR)$(bindir)/pg_waldump$(X)'
@@ -37,4 +40,5 @@ uninstall:
rm -f '$(DESTDIR)$(bindir)/pg_waldump$(X)'
clean distclean maintainer-clean:
- rm -f pg_waldump$(X) $(OBJS) $(RMGRDESCSOURCES) xlogreader.c
+ rm -f pg_waldump$(X) $(OBJS) $(RMGRDESCSOURCES) xlogreader.c \
+encryption.c
diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c
index 2a557b37e5..939b7392ee 100644
--- a/src/bin/pg_waldump/pg_waldump.c
+++ b/src/bin/pg_waldump/pg_waldump.c
@@ -21,12 +21,15 @@
#include "access/xlog_internal.h"
#include "access/transam.h"
#include "common/fe_memutils.h"
+#include "storage/encryption.h"
#include "getopt_long.h"
#include "rmgrdesc.h"
static const char *progname;
+static char *str_wal_segment_size_mb = NULL;
+static bool encr_inited = false;
static int WalSegSz;
typedef struct XLogDumpPrivate
@@ -215,8 +218,85 @@ search_directory(const char *directory, const char *fname)
r = read(fd, buf, XLOG_BLCKSZ);
if (r == XLOG_BLCKSZ)
{
- XLogLongPageHeader longhdr = (XLogLongPageHeader) buf;
+ XLogLongPageHeader longhdr;
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ uint32 tli;
+ char tweak[TWEAK_SIZE];
+ XLogSegNo segNo;
+ int wal_segment_size;
+ XLogPageHeader hdr;
+
+ /*
+ * Segment size is contained in the encryption tweak so we
+ * cannot get the size from the header until the page is
+ * decrypted. User needs to pass the size if he uses
+ * non-default value.
+ *
+ * TODO The processing of wal-segsize is almost copy & pasted
+ * from initdb.c. Reuse in a better way.
+ */
+ /* set wal segment size */
+ if (str_wal_segment_size_mb == NULL)
+ wal_segment_size = DEFAULT_XLOG_SEG_SIZE;
+ else
+ {
+ char *endptr;
+
+ /* check that the argument is a number */
+ wal_segment_size = strtol(str_wal_segment_size_mb, &endptr, 10);
+
+ /* verify that wal segment size is valid */
+ if (endptr == str_wal_segment_size_mb || *endptr != '\0')
+ {
+ fprintf(stderr,
+ _("%s: argument of --wal-segsize must be a number\n"),
+ progname);
+ exit(1);
+ }
+
+ wal_segment_size *= 1024 * 1024;
+ if (!IsValidWalSegSize(wal_segment_size))
+ {
+ fprintf(stderr,
+ _("%s: argument of --wal-segsize must be a power of 2 between 1 and 1024\n"),
+ progname);
+ exit(1);
+ }
+ }
+
+ XLogFromFileName(fname, &tli, &segNo, wal_segment_size);
+ XLogEncryptionTweak(tweak, segNo, 0);
+ setup_encryption(false);
+ encr_inited = true;
+ decrypt_block(buf, buf, XLOG_BLCKSZ, tweak);
+
+ hdr = (XLogPageHeader) buf;
+ if (hdr->xlp_magic != XLOG_PAGE_MAGIC)
+ {
+ fprintf(stderr,
+ _("invalid magic number %04X. Make sure you've passed correct argument for --wal-segsize"),
+ hdr->xlp_magic);
+ exit(1);
+ }
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+ else
+ {
+ if (str_wal_segment_size_mb != NULL)
+ {
+ fprintf(stderr,
+ _("%s: --wal-segsize is only accepted for encrypted WAL\n"),
+ progname);
+ exit(1);
+ }
+ }
+
+ longhdr = (XLogLongPageHeader) buf;
WalSegSz = longhdr->xlp_seg_size;
if (!IsValidWalSegSize(WalSegSz))
@@ -462,6 +542,23 @@ XLogDumpReadPage(XLogReaderState *state, XLogRecPtr targetPagePtr, int reqLen,
XLogDumpXLogRead(private->inpath, private->timeline, targetPagePtr,
readBuff, count);
+ if (data_encrypted)
+ {
+#ifdef USE_ENCRYPTION
+ char tweak[TWEAK_SIZE];
+ XLogSegNo readSegNo;
+ uint32 readSegOff;
+
+ XLByteToSeg(targetPagePtr, readSegNo, WalSegSz);
+ readSegOff = targetPagePtr % WalSegSz;
+
+ XLogEncryptionTweak(tweak, readSegNo, readSegOff);
+ decrypt_block(readBuff, readBuff, count, tweak);
+#else
+ ENCRYPTION_NOT_SUPPORTED_MSG;
+#endif /* USE_ENCRYPTION */
+ }
+
return count;
}
@@ -810,6 +907,12 @@ usage(void)
printf(_(" -b, --bkp-details output detailed information about backup blocks\n"));
printf(_(" -e, --end=RECPTR stop reading at WAL location RECPTR\n"));
printf(_(" -f, --follow keep retrying after reaching end of WAL\n"));
+#ifdef USE_ENCRYPTION
+ printf(_(" -K, --encryption-key-command=COMMAND\n"
+ " command that returns encryption key\n"));
+ printf(_(" -w, --wal-segsize=SIZE\n"
+ " size of WAL segments, in megabytes\n"));
+#endif /* USE_ENCRYPTION */
printf(_(" -n, --limit=N number of records to display\n"));
printf(_(" -p, --path=PATH directory in which to find log segment files or a\n"
" directory with a ./pg_wal that contains such files\n"
@@ -844,6 +947,7 @@ main(int argc, char **argv)
{"end", required_argument, NULL, 'e'},
{"follow", no_argument, NULL, 'f'},
{"help", no_argument, NULL, '?'},
+ {"encryption-key-command", required_argument, NULL, 'K'},
{"limit", required_argument, NULL, 'n'},
{"path", required_argument, NULL, 'p'},
{"rmgr", required_argument, NULL, 'r'},
@@ -851,6 +955,7 @@ main(int argc, char **argv)
{"timeline", required_argument, NULL, 't'},
{"xid", required_argument, NULL, 'x'},
{"version", no_argument, NULL, 'V'},
+ {"wal-segsize", required_argument, NULL, 'w'},
{"stats", optional_argument, NULL, 'z'},
{NULL, 0, NULL, 0}
};
@@ -886,7 +991,7 @@ main(int argc, char **argv)
goto bad_argument;
}
- while ((option = getopt_long(argc, argv, "be:?fn:p:r:s:t:Vx:z",
+ while ((option = getopt_long(argc, argv, "be:?fK:n:p:r:s:t:Vx:z",
long_options, &optindex)) != -1)
{
switch (option)
@@ -910,6 +1015,12 @@ main(int argc, char **argv)
usage();
exit(EXIT_SUCCESS);
break;
+#ifdef USE_ENCRYPTION
+ case 'K':
+ encryption_key_command = pg_strdup(optarg);
+ data_encrypted = true;
+ break;
+#endif /* USE_ENCRYPTION */
case 'n':
if (sscanf(optarg, "%d", &config.stop_after_records) != 1)
{
@@ -970,6 +1081,9 @@ main(int argc, char **argv)
puts("pg_waldump (PostgreSQL) " PG_VERSION);
exit(EXIT_SUCCESS);
break;
+ case 'w':
+ str_wal_segment_size_mb = pg_strdup(optarg);
+ break;
case 'x':
if (sscanf(optarg, "%u", &config.filter_by_xid) != 1)
{
@@ -1117,6 +1231,12 @@ main(int argc, char **argv)
/* done with argument parsing, do the actual work */
+ if (data_encrypted && !encr_inited)
+ {
+ setup_encryption(false);
+ encr_inited = false;
+ }
+
/* we have everything we need, start reading */
xlogreader_state = XLogReaderAllocate(WalSegSz, XLogDumpReadPage,
&private);
diff --git a/src/common/string.c b/src/common/string.c
index 3260d37a84..57ede1dd14 100644
--- a/src/common/string.c
+++ b/src/common/string.c
@@ -42,6 +42,45 @@ pg_str_endswith(const char *str, const char *end)
return strcmp(str, end) == 0;
}
+/*
+ * Helper function to check if a page is completely empty.
+ *
+ * TODO Invent name that is more consistent with that of the other function(s)
+ * in this module.
+ */
+bool
+IsAllZero(const char *input, Size size)
+{
+ const char *pos = input;
+ const char *aligned_start = (char *) MAXALIGN64(input);
+ const char *end = input + size;
+
+ /* Check 1 byte at a time until pos is 8 byte aligned */
+ while (pos < aligned_start)
+ if (*pos++ != 0)
+ return false;
+
+ /*
+ * Run 8 parallel 8 byte checks in one iteration. On 2016 hardware
+ * slightly faster than 4 parallel checks.
+ */
+ while (pos + 8 * sizeof(uint64) <= end)
+ {
+ uint64 *p = (uint64 *) pos;
+
+ if ((p[0] | p[1] | p[2] | p[3] | p[4] | p[5] | p[6] | p[7]) != 0)
+ return false;
+ pos += 8 * sizeof(uint64);
+ }
+
+ /* Handle unaligned tail. */
+ while (pos < end)
+ if (*pos++ != 0)
+ return false;
+
+ return true;
+}
+
/*
* strtoint --- just like strtol, but returns int not long
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 421ba6d775..c7fea0f6cb 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -258,6 +258,7 @@ extern void UpdateControlFile(void);
extern uint64 GetSystemIdentifier(void);
extern char *GetMockAuthenticationNonce(void);
extern bool DataChecksumsEnabled(void);
+extern bool DataEncryptionEnabled(void);
extern XLogRecPtr GetFakeLSNForUnloggedRel(void);
extern Size XLOGShmemSize(void);
extern void XLOGShmemInit(void);
diff --git a/src/include/access/xlog_internal.h b/src/include/access/xlog_internal.h
index 30610b3ea9..4990c0bc97 100644
--- a/src/include/access/xlog_internal.h
+++ b/src/include/access/xlog_internal.h
@@ -52,7 +52,7 @@ typedef struct XLogPageHeaderData
uint32 xlp_rem_len; /* total len of remaining data for record */
} XLogPageHeaderData;
-#define SizeOfXLogShortPHD MAXALIGN(sizeof(XLogPageHeaderData))
+#define SizeOfXLogShortPHD XLOG_REC_ALIGN(sizeof(XLogPageHeaderData))
typedef XLogPageHeaderData *XLogPageHeader;
@@ -69,7 +69,7 @@ typedef struct XLogLongPageHeaderData
uint32 xlp_xlog_blcksz; /* just as a cross-check */
} XLogLongPageHeaderData;
-#define SizeOfXLogLongPHD MAXALIGN(sizeof(XLogLongPageHeaderData))
+#define SizeOfXLogLongPHD XLOG_REC_ALIGN(sizeof(XLogLongPageHeaderData))
typedef XLogLongPageHeaderData *XLogLongPageHeader;
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index 773d9e6eba..ce4386d592 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -94,6 +94,11 @@ typedef enum DBState
} DBState;
/*
+ * Number of bytes reserved to store encryption sample in ControlFileData.
+ */
+#define ENCRYPTION_SAMPLE_SIZE 16
+
+/*
* Contents of pg_control.
*/
@@ -227,6 +232,11 @@ typedef struct ControlFileData
*/
char mock_authentication_nonce[MOCK_AUTH_NONCE_LEN];
+ /* Is data directory encrypted? */
+ bool data_encrypted;
+ /* Sample value for encryption key verification */
+ uint8 encryption_verification[ENCRYPTION_SAMPLE_SIZE];
+
/* CRC of all above ... MUST BE LAST! */
pg_crc32c crc;
} ControlFileData;
diff --git a/src/include/common/string.h b/src/include/common/string.h
index 78a450192e..26fbfdc51e 100644
--- a/src/include/common/string.h
+++ b/src/include/common/string.h
@@ -13,5 +13,5 @@
extern bool pg_str_endswith(const char *str, const char *end);
extern int strtoint(const char *pg_restrict str, char **pg_restrict endptr,
int base);
-
+extern bool IsAllZero(const char *input, Size size);
#endif /* COMMON_STRING_H */
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index e167ee8fcb..bc01ace87a 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -429,6 +429,7 @@ extern void BaseInit(void);
/* in utils/init/miscinit.c */
extern bool IgnoreSystemIndexes;
extern PGDLLIMPORT bool process_shared_preload_libraries_in_progress;
+extern char *encryption_library_string;
extern char *session_preload_libraries_string;
extern char *shared_preload_libraries_string;
extern char *local_preload_libraries_string;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index d59c24ae23..10d4f3423f 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -874,6 +874,9 @@ typedef enum
WAIT_EVENT_DATA_FILE_TRUNCATE,
WAIT_EVENT_DATA_FILE_WRITE,
WAIT_EVENT_DSM_FILL_ZERO_WRITE,
+ WAIT_EVENT_KDF_FILE_READ,
+ WAIT_EVENT_KDF_FILE_SYNC,
+ WAIT_EVENT_KDF_FILE_WRITE,
WAIT_EVENT_LOCK_FILE_ADDTODATADIR_READ,
WAIT_EVENT_LOCK_FILE_ADDTODATADIR_SYNC,
WAIT_EVENT_LOCK_FILE_ADDTODATADIR_WRITE,
diff --git a/src/include/storage/copydir.h b/src/include/storage/copydir.h
index 4fef3e2107..8aea3eb2ed 100644
--- a/src/include/storage/copydir.h
+++ b/src/include/storage/copydir.h
@@ -13,7 +13,9 @@
#ifndef COPYDIR_H
#define COPYDIR_H
-extern void copydir(char *fromdir, char *todir, bool recurse);
-extern void copy_file(char *fromfile, char *tofile);
+#include "storage/relfilenode.h"
+
+extern void copydir(char *fromdir, char *todir, RelFileNode *fromNode, RelFileNode *toNode);
+extern void copy_file(char *fromfile, char *tofile, RelFileNode *fromNode, RelFileNode *toNode, ForkNumber fromForkNum, ForkNumber toForkNum, int segment);
#endif /* COPYDIR_H */
diff --git a/src/include/storage/encryption.h b/src/include/storage/encryption.h
new file mode 100644
index 0000000000..0a49f5a949
--- /dev/null
+++ b/src/include/storage/encryption.h
@@ -0,0 +1,181 @@
+/*-------------------------------------------------------------------------
+ *
+ * encryption.h
+ * Full database encryption support
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/encryption.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef ENCRYPTION_H
+#define ENCRYPTION_H
+
+#include "access/xlogdefs.h"
+#include "port/pg_crc32c.h"
+
+/*
+ * OpenSSL is currently the only implementation of encryption we use.
+ */
+#ifdef USE_OPENSSL
+#define USE_ENCRYPTION
+#endif
+
+/*
+ * Common error message issued when particular code path cannot be executed
+ * due to absence of the OpenSSL library.
+ */
+#define ENCRYPTION_NOT_SUPPORTED_MSG \
+ encryption_error(true, \
+ "data encryption cannot be used because SSL is not supported by this build\n" \
+ "Compile with --with-openssl to use SSL connections.")
+#ifdef USE_ENCRYPTION
+/*
+ * Full database encryption key.
+ *
+ * EVP_aes_256_xts() needs the key twice as long as AES would do in general.
+ */
+#define ENCRYPTION_KEY_LENGTH 64
+
+/*
+ * TODO Tune these values.
+ */
+#define ENCRYPTION_PWD_MIN_LENGTH 8
+#define ENCRYPTION_PWD_MAX_LENGTH 16
+#define ENCRYPTION_KDF_NITER 1048576
+#define ENCRYPTION_KDF_SALT_LEN 8
+
+extern unsigned char *encryption_key;
+#endif /* USE_ENCRYPTION */
+
+/*
+ * The encrypted data is a series of blocks of size
+ * ENCRYPTION_BLOCK. Currently we use the EVP_aes_256_xts implementation. Make
+ * sure the following constants match if adopting another algorithm.
+ */
+#define ENCRYPTION_BLOCK 16
+
+#ifdef USE_ENCRYPTION
+/*
+ * The openssl EVP API refers to a block in terms of padding of the output
+ * chunk. That's the purpose of this constant. However the openssl
+ * implementation of AES XTS still uses the 16-byte block internally, as
+ * defined by ENCRYPTION_BLOCK.
+ */
+#define ENCRYPTION_BLOCK_OPENSSL 1
+
+#define TWEAK_SIZE 16
+
+#define KDF_PARAMS_FILE "global/pg_keysetup"
+#define KDF_PARAMS_FILE_SIZE 512
+
+/*
+ * Key derivation function.
+ */
+typedef enum KDFKind
+{
+ KDF_OPENSSL_PKCS5_PBKDF2_HMAC_SHA = 0
+} KFDKind;
+
+typedef struct KDFParamsPBKDF2
+{
+ unsigned long int niter;
+ unsigned char salt[ENCRYPTION_KDF_SALT_LEN];
+} KDFParamsPBKDF2;
+
+/*
+ * Parameters of the key derivation function.
+ *
+ * The parameters are generated by initdb and stored into a file, which is
+ * then read during PG startup. This is similar to storing various settings in
+ * pg_control. However an existing KDF file is read only, so it does not have
+ * to be stored in shared memory.
+ */
+typedef struct KDFParamsData
+{
+ KFDKind function;
+
+ /*
+ * Function-specific parameters.
+ */
+ union
+ {
+ KDFParamsPBKDF2 pbkdf2;
+ } data;
+
+ /* CRC of all above ... MUST BE LAST! */
+ pg_crc32c crc;
+} KDFParamsData;
+
+extern KDFParamsData *KDFParams;
+#endif /* USE_ENCRYPTION */
+
+extern PGDLLIMPORT bool data_encrypted;
+
+#ifdef USE_ENCRYPTION
+#ifndef FRONTEND
+extern void write_kdf_file(void);
+extern void read_kdf_file(void);
+#endif /* FRONTEND */
+
+extern PGDLLIMPORT char *encryption_key_command;
+
+extern char *encryption_buffer;
+extern Size encryption_buf_size;
+
+extern void enlarge_encryption_buffer(Size new_size);
+#endif /* USE_ENCRYPTION */
+
+/*
+ * If one XLOG record ended and the following one started in the same block,
+ * we'd have to either encrypt and decrypt both records together, or encrypt
+ * (after having zeroed the part of the block occupied by the other record)
+ * and decrypt them separate. Neither approach is compatible with streaming
+ * replication. In the first case we can't ask standby not to decrypt the
+ * first record until the second has been streamed. The second approach would
+ * imply streaming of two different versions of the same block two times.
+ *
+ * We avoid this problem by aligning XLOG records to the encryption block
+ * size. This way no adjacent XLOG records should appear in the same block.
+ *
+ * For similar reasons, the alignment to ENCRYPTION_BLOCK also has to be
+ * applied when storing changes to disk in reorderbuffer.c. Another module
+ * that takes the block into account is buffile.c.
+ *
+ * TODO If the configuration allows walsender to decrypt the XLOG stream
+ * before sending it, adjust this expression so that the additional padding is
+ * not added to XLOG records. (Since the XLOG alignment cannot change without
+ * initdb, the same would apply to the configuration variable that makes
+ * walsender perform the decryption. Does such a variable make sense?)
+ */
+#define DO_ENCRYPTION_BLOCK_ALIGN data_encrypted
+
+/*
+ * Use TYPEALIGN64 since besides record size we also need to align XLogRecPtr.
+ */
+#define ENCRYPTION_BLOCK_ALIGN(LEN) TYPEALIGN64(ENCRYPTION_BLOCK, (LEN))
+
+/*
+ * Universal computation of XLOG record alignment.
+ */
+#define XLOG_REC_ALIGN(LEN) ((DO_ENCRYPTION_BLOCK_ALIGN) ?\
+ ENCRYPTION_BLOCK_ALIGN(LEN) : MAXALIGN64(LEN))
+
+extern void setup_encryption(bool bootstrap);
+extern void setup_encryption_key(char *credentials, bool is_key, size_t len);
+extern void sample_encryption(char *buf);
+extern void encrypt_block(const char *input, char *output, Size size,
+ const char *tweak);
+extern void decrypt_block(const char *input, char *output, Size size,
+ const char *tweak);
+#ifdef USE_ENCRYPTION
+extern void encryption_error(bool fatal, char *message);
+#endif
+
+extern void XLogEncryptionTweak(char *tweak, XLogSegNo segment,
+ uint32 offset);
+
+#endif /* ENCRYPTION_H */
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index 8e7c9728f4..0567c68c0c 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -72,6 +72,7 @@ extern int FileRead(File file, char *buffer, int amount, uint32 wait_event_info)
extern int FileWrite(File file, char *buffer, int amount, uint32 wait_event_info);
extern int FileSync(File file, uint32 wait_event_info);
extern off_t FileSeek(File file, off_t offset, int whence);
+extern off_t FileTell(File file);
extern int FileTruncate(File file, off_t offset, uint32 wait_event_info);
extern void FileWriteback(File file, off_t offset, off_t nbytes, uint32 wait_event_info);
extern char *FilePathName(File file);
diff --git a/src/include/storage/reinit.h b/src/include/storage/reinit.h
index a62703c647..5efd101d03 100644
--- a/src/include/storage/reinit.h
+++ b/src/include/storage/reinit.h
@@ -20,7 +20,8 @@
extern void ResetUnloggedRelations(int op);
extern bool parse_filename_for_nontemp_relation(const char *name,
- int *oidchars, ForkNumber *fork);
+ int *oidchars, ForkNumber *fork,
+ int *segment);
#define UNLOGGED_RELATION_CLEANUP 0x0001
#define UNLOGGED_RELATION_INIT 0x0002
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index c843bbc969..d69611934c 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -145,4 +145,10 @@ extern void ForgetRelationFsyncRequests(RelFileNode rnode, ForkNumber forknum);
extern void ForgetDatabaseFsyncRequests(Oid dbid);
extern void DropRelationFiles(RelFileNode *delrels, int ndelrels, bool isRedo);
+#ifdef USE_ENCRYPTION
+extern BlockNumber ReencryptBlock(char *buffer, int blocks,
+ RelFileNode *srcNode, RelFileNode *dstNode,
+ ForkNumber srcForkNum, ForkNumber dstForkNum,
+ BlockNumber blockNum);
+#endif
#endif /* SMGR_H */
diff --git a/src/test/modules/buffile/Makefile b/src/test/modules/buffile/Makefile
new file mode 100644
index 0000000000..b658c0dd8d
--- /dev/null
+++ b/src/test/modules/buffile/Makefile
@@ -0,0 +1,21 @@
+PG_CONFIG ?= pg_config
+MODULE_big = buffile_test
+OBJS = buffile.o $(WIN32RES)
+PGFILEDESC = "buffile_test"
+
+EXTENSION = buffile
+DATA = buffile--1.0.sql
+
+REGRESS = test_00 test_01 test_02 test_03 test_04 test_05 test_06 test_07 \
+test_08 test_09 test_10 test_11 test_12
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/buffile
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/buffile/README b/src/test/modules/buffile/README
new file mode 100644
index 0000000000..9c80787797
--- /dev/null
+++ b/src/test/modules/buffile/README
@@ -0,0 +1,11 @@
+This extension was written to check if changes introduced due to cluster
+encryption do not break buffile.c.
+
+Caution: To make the test cheaper, it was decided to adjust the segment sizeq,
+see
+
+#define MAX_PHYSICAL_FILESIZE (4 * BLCKSZ)
+
+in buffile.c. BLCKSZ is 8192 (the default). All the tests rely on this
+value. So if you haven't compiled Postgres with this value, the tests will
+create 1 GB files and they will fail.
diff --git a/src/test/modules/buffile/buffile--1.0.sql b/src/test/modules/buffile/buffile--1.0.sql
new file mode 100644
index 0000000000..82baa043f7
--- /dev/null
+++ b/src/test/modules/buffile/buffile--1.0.sql
@@ -0,0 +1,39 @@
+CREATE FUNCTION buffile_create()
+RETURNS void
+AS 'MODULE_PATHNAME', 'buffile_create'
+LANGUAGE C;
+
+CREATE FUNCTION buffile_close()
+RETURNS void
+AS 'MODULE_PATHNAME', 'buffile_close'
+LANGUAGE C;
+
+CREATE FUNCTION buffile_write(text)
+RETURNS bigint
+AS 'MODULE_PATHNAME', 'buffile_write'
+LANGUAGE C;
+
+CREATE FUNCTION buffile_read(bigint)
+RETURNS bytea
+AS 'MODULE_PATHNAME', 'buffile_read'
+LANGUAGE C;
+
+CREATE FUNCTION buffile_seek(int, bigint)
+RETURNS int
+AS 'MODULE_PATHNAME', 'buffile_seek'
+LANGUAGE C;
+
+CREATE FUNCTION buffile_assert_fileno(int)
+RETURNS void
+AS 'MODULE_PATHNAME', 'buffile_assert_fileno'
+LANGUAGE C;
+
+CREATE FUNCTION buffile_test_shared()
+RETURNS void
+AS 'MODULE_PATHNAME', 'buffile_test_shared'
+LANGUAGE C;
+
+CREATE FUNCTION buffile_test_shared_append()
+RETURNS void
+AS 'MODULE_PATHNAME', 'buffile_test_shared_append'
+LANGUAGE C;
diff --git a/src/test/modules/buffile/buffile.c b/src/test/modules/buffile/buffile.c
new file mode 100644
index 0000000000..45ad6778ab
--- /dev/null
+++ b/src/test/modules/buffile/buffile.c
@@ -0,0 +1,353 @@
+#include "postgres.h"
+#include "fmgr.h"
+#include "lib/stringinfo.h"
+#include "storage/buffile.h"
+#include "utils/array.h"
+#include "utils/builtins.h"
+#include "utils/lsyscache.h"
+#include "utils/memutils.h"
+#include "utils/resowner.h"
+
+PG_MODULE_MAGIC;
+
+/*
+ * To cover various corner cases, the tests assume MAX_PHYSICAL_FILESIZE to be
+ * exactly MAX_PHYSICAL_FILESIZE_TEST.
+ */
+#define MAX_PHYSICAL_FILESIZE_TEST (4 * BLCKSZ)
+
+static BufFile *bf = NULL;
+
+static void check_file(void);
+
+extern Datum buffile_create(PG_FUNCTION_ARGS);
+
+PG_FUNCTION_INFO_V1(buffile_create);
+Datum
+buffile_create(PG_FUNCTION_ARGS)
+{
+ MemoryContext old_cxt;
+ ResourceOwner old_ro;
+
+ if (bf != NULL)
+ elog(ERROR, "file already exists");
+
+ old_cxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ /*
+ * Make sure the file is not deleted across function calls.
+ */
+ old_ro = CurrentResourceOwner;
+ CurrentResourceOwner = TopTransactionResourceOwner;
+
+ bf = BufFileCreateTemp(false);
+
+ CurrentResourceOwner = old_ro;
+ MemoryContextSwitchTo(old_cxt);
+
+ PG_RETURN_VOID();
+}
+
+extern Datum buffile_close(PG_FUNCTION_ARGS);
+
+PG_FUNCTION_INFO_V1(buffile_close);
+Datum
+buffile_close(PG_FUNCTION_ARGS)
+{
+ if (bf == NULL)
+ elog(ERROR, "there's no file to close");
+
+ BufFileClose(bf);
+ bf = NULL;
+
+ PG_RETURN_VOID();
+}
+
+extern Datum buffile_write(PG_FUNCTION_ARGS);
+
+PG_FUNCTION_INFO_V1(buffile_write);
+Datum
+buffile_write(PG_FUNCTION_ARGS)
+{
+ Datum d = PG_GETARG_DATUM(0);
+ char *s = TextDatumGetCString(d);
+ size_t res;
+
+ check_file();
+ res = BufFileWrite(bf, s, strlen(s));
+
+ PG_RETURN_INT64(res);
+}
+
+extern Datum buffile_read(PG_FUNCTION_ARGS);
+
+PG_FUNCTION_INFO_V1(buffile_read);
+Datum
+buffile_read(PG_FUNCTION_ARGS)
+{
+ int64 size = PG_GETARG_INT64(0);
+ StringInfo buf = makeStringInfo();
+ size_t res_size;
+ bytea *result;
+
+ check_file();
+
+ enlargeStringInfo(buf, size);
+ res_size = BufFileRead(bf, buf->data, size);
+ buf->len = res_size;
+
+ result = DatumGetByteaPP(DirectFunctionCall1(bytearecv,
+ PointerGetDatum(buf)));
+ PG_RETURN_BYTEA_P(result);
+}
+
+extern Datum buffile_seek(PG_FUNCTION_ARGS);
+
+PG_FUNCTION_INFO_V1(buffile_seek);
+Datum
+buffile_seek(PG_FUNCTION_ARGS)
+{
+ int32 fileno = PG_GETARG_INT32(0);
+ int64 offset = PG_GETARG_INT64(1);
+ int32 res;
+
+ check_file();
+ res = BufFileSeek(bf, fileno, offset, SEEK_SET);
+
+ PG_RETURN_INT32(res);
+}
+
+extern Datum buffile_assert_fileno(PG_FUNCTION_ARGS);
+
+PG_FUNCTION_INFO_V1(buffile_assert_fileno);
+Datum
+buffile_assert_fileno(PG_FUNCTION_ARGS)
+{
+ int32 fileno_expected = PG_GETARG_INT32(0);
+ int32 fileno;
+ off_t offset;
+
+ check_file();
+ BufFileTell(bf, &fileno, &offset);
+
+ if (fileno != fileno_expected)
+ {
+ /*
+ * Bring the backend down so that the following tests have no chance
+ * to create the 1GB files.
+ */
+ elog(FATAL, "file number does not match");
+ }
+
+ PG_RETURN_VOID();
+}
+
+static void
+check_file(void)
+{
+ if (bf == NULL)
+ elog(ERROR, "the file is not opened");
+}
+
+/*
+ * This test is especially important for shared encrypted files, see the
+ * comments below.
+ */
+extern Datum buffile_test_shared(PG_FUNCTION_ARGS);
+
+PG_FUNCTION_INFO_V1(buffile_test_shared);
+Datum
+buffile_test_shared(PG_FUNCTION_ARGS)
+{
+ dsm_segment *seg;
+ SharedFileSet *fileset;
+ BufFile *bf_1,
+ *bf_2;
+ char *data_1,
+ *data_2,
+ *data;
+ Size chunk_size_1,
+ chunk_size_2;
+ int fileno,
+ i;
+ off_t offset,
+ res,
+ total_size;
+
+ /*
+ * The size is not important, we actually do not need the shared memory.
+ * The segment is only needed to initialize the fileset.
+ */
+ seg = dsm_create(1024, 0);
+
+ /*
+ * The fileset must survive error handling, so that dsm_detach works fine.
+ * (The typical use case is that the fileset is in shared memory.)
+ */
+ fileset = (SharedFileSet *) MemoryContextAlloc(TopTransactionContext,
+ sizeof(SharedFileSet));
+ SharedFileSetInit(fileset, seg);
+
+ bf_1 = BufFileCreateShared(fileset, "file_1");
+
+ /*
+ * Write more data than the buffer size, so that we can check that the
+ * number of "useful bytes" word is only appended at the end of the
+ * segment, not after each buffer.
+ */
+ chunk_size_1 = BLCKSZ + 256;
+ data_1 = (char *) palloc(chunk_size_1);
+ memset(data_1, 1, chunk_size_1);
+ if (BufFileWrite(bf_1, data_1, chunk_size_1) != chunk_size_1)
+ elog(ERROR, "Failed to write data");
+ pfree(data_1);
+
+ /*
+ * Enforce buffer flush (The BufFileFlush() function is not exported).
+ * Thus the "useful bytes" metadata should appear at the current end the
+ * first file segment. The next write will have to seek back to overwrite
+ * the metadata.
+ */
+ BufFileTell(bf_1, &fileno, &offset);
+ if (BufFileSeek(bf_1, 0, 0, SEEK_SET) != 0)
+ elog(ERROR, "seek failed");
+ if (BufFileSeek(bf_1, fileno, offset, SEEK_SET) != 0)
+ elog(ERROR, "seek failed");
+
+ /*
+ * Write another chunk that does not fit into the first segment file. Thus
+ * the "useful bytes" metadata should appear at the end of both segments.
+ */
+ chunk_size_2 = 3 * BLCKSZ;
+ data_2 = (char *) palloc(chunk_size_2);
+ memset(data_2, 1, chunk_size_2);
+ if (BufFileWrite(bf_1, data_2, chunk_size_2) != chunk_size_2)
+ elog(ERROR, "Failed to write data");
+ pfree(data_2);
+ BufFileClose(bf_1);
+
+ /*
+ * The word indicating the number of "useful bytes" (i.e. the actual data
+ * w/o padding to buffer size) is stored at the end of each segment file.
+ * Check that this metadata is read correctly.
+ */
+ bf_2 = BufFileOpenShared(fileset, "file_1");
+ total_size = BufFileSize(bf_2);
+ if (total_size != (chunk_size_1 + chunk_size_2))
+ elog(ERROR, "Incorrect file size: %zu", total_size);
+
+ data = (char *) palloc(total_size);
+ res = BufFileRead(bf_2, data, total_size);
+ if (res != total_size)
+ elog(ERROR, "Incorrect chunk size read: %zu", res);
+ for (i = 0; i < total_size; i++)
+ if (data[i] != 1)
+ elog(ERROR, "Unexpected data read from the file");
+ pfree(data);
+ BufFileClose(bf_2);
+
+ dsm_detach(seg);
+
+ PG_RETURN_VOID();
+}
+
+
+/*
+ * Test BufFileAppend().
+ */
+extern Datum buffile_test_shared_append(PG_FUNCTION_ARGS);
+
+PG_FUNCTION_INFO_V1(buffile_test_shared_append);
+Datum
+buffile_test_shared_append(PG_FUNCTION_ARGS)
+{
+ dsm_segment *seg;
+ SharedFileSet *fileset;
+ BufFile *bf_1,
+ *bf_2,
+ *bf_3;
+ char *data;
+ Size chunk_size;
+ int fileno,
+ i;
+ off_t offset,
+ res,
+ total_size;
+
+ seg = dsm_create(1024, 0);
+
+ fileset = (SharedFileSet *) MemoryContextAlloc(TopTransactionContext,
+ sizeof(SharedFileSet));
+ SharedFileSetInit(fileset, seg);
+
+ /*
+ * XXX Does the chunk size matter much?
+ */
+ chunk_size = 8;
+ data = (char *) palloc(chunk_size);
+ memset(data, 1, chunk_size);
+
+ bf_1 = BufFileCreateShared(fileset, "file_1");
+ if (BufFileWrite(bf_1, data, chunk_size) != chunk_size)
+ elog(ERROR, "Failed to write data");
+
+ bf_2 = BufFileCreateShared(fileset, "file_2");
+ if (BufFileWrite(bf_2, data, chunk_size) != chunk_size)
+ elog(ERROR, "Failed to write data");
+
+ /*
+ * Make sure it's read-only so that BufFileAppend() can accept it as
+ * source.
+ */
+ BufFileClose(bf_2);
+ bf_2 = BufFileOpenShared(fileset, "file_2");
+
+ bf_3 = BufFileCreateShared(fileset, "file_3");
+ if (BufFileWrite(bf_3, data, chunk_size) != chunk_size)
+ elog(ERROR, "Failed to write data");
+ BufFileClose(bf_3);
+ bf_3 = BufFileOpenShared(fileset, "file_3");
+
+ BufFileAppend(bf_1, bf_2);
+ BufFileAppend(bf_1, bf_3);
+
+ total_size = BufFileSize(bf_1);
+
+ /*
+ * The result should contain complete segments of bf_1 and bf_2 and the
+ * valid part of bf_3.
+ */
+ if (total_size != (2 * MAX_PHYSICAL_FILESIZE_TEST + chunk_size))
+ elog(ERROR, "Incorrect total size of the appended data: %zu",
+ total_size);
+
+ /*
+ * Check that data of the 2nd segment was decrypted correctly.
+ */
+ if (BufFileSeek(bf_1, 1, 0, SEEK_SET) != 0)
+ elog(ERROR, "seek failed");
+ res = BufFileRead(bf_1, data, chunk_size);
+ if (res != chunk_size)
+ elog(ERROR, "Incorrect chunk size read: %zu", res);
+ for (i = 0; i < chunk_size; i++)
+ if (data[i] != 1)
+ elog(ERROR, "Unexpected data read from the file");
+
+ /*
+ * And the same for the 3rd segment.
+ *
+ * TODO Reuse the code above by putting it into a function.
+ */
+ if (BufFileSeek(bf_1, 2, 0, SEEK_SET) != 0)
+ elog(ERROR, "seek failed");
+ res = BufFileRead(bf_1, data, chunk_size);
+ if (res != chunk_size)
+ elog(ERROR, "Incorrect chunk size read: %zu", res);
+ for (i = 0; i < chunk_size; i++)
+ if (data[i] != 1)
+ elog(ERROR, "Unexpected data read from the file");
+
+ BufFileClose(bf_1);
+ dsm_detach(seg);
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/buffile/buffile.control b/src/test/modules/buffile/buffile.control
new file mode 100644
index 0000000000..8472c5a348
--- /dev/null
+++ b/src/test/modules/buffile/buffile.control
@@ -0,0 +1,5 @@
+# buffile_test extension
+comment = 'buffile_test'
+default_version = '1.0'
+module_pathname = '$libdir/buffile_test'
+relocatable = true
diff --git a/src/test/modules/buffile/expected/test_00.out b/src/test/modules/buffile/expected/test_00.out
new file mode 100644
index 0000000000..637c33689f
--- /dev/null
+++ b/src/test/modules/buffile/expected/test_00.out
@@ -0,0 +1,47 @@
+CREATE EXTENSION IF NOT EXISTS buffile;
+-- This test only verifies that PG is compiled with a component file size of
+-- 32 kB (i.e. 4 buffers of 8 kB) instead of 1 GB. That seems appropriate for
+-- testing. Some other tests may rely on it.
+BEGIN;
+SELECT buffile_create();
+ buffile_create
+----------------
+
+(1 row)
+
+-- Skip the first component file.
+SELECT buffile_seek(0, 32768);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+-- Write the first byte of the second component file. We can't simply
+-- buffile_seek() beyond the position 32768 as it would return EOF.
+SELECT buffile_write('a');
+ buffile_write
+---------------
+ 1
+(1 row)
+
+-- Enforce BufFileFlush(), which actually adds the component file.
+SELECT buffile_read(1);
+ buffile_read
+--------------
+ \x
+(1 row)
+
+-- Check that we're in the 2nd file, i.e. the file size is as expected.
+SELECT buffile_assert_fileno(1);
+ buffile_assert_fileno
+-----------------------
+
+(1 row)
+
+SELECT buffile_close();
+ buffile_close
+---------------
+
+(1 row)
+
+COMMIT;
diff --git a/src/test/modules/buffile/expected/test_01.out b/src/test/modules/buffile/expected/test_01.out
new file mode 100644
index 0000000000..448bac171c
--- /dev/null
+++ b/src/test/modules/buffile/expected/test_01.out
@@ -0,0 +1,59 @@
+BEGIN;
+SELECT buffile_create();
+ buffile_create
+----------------
+
+(1 row)
+
+SELECT buffile_seek(0, 1);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_write('abc');
+ buffile_write
+---------------
+ 3
+(1 row)
+
+SELECT buffile_seek(0, 0);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+-- Check that the trailing zeroes are not fetched.
+SELECT buffile_read(16);
+ buffile_read
+--------------
+ \x00616263
+(1 row)
+
+-- Adjust the number of useful bytes.
+SELECT buffile_write('abc');
+ buffile_write
+---------------
+ 3
+(1 row)
+
+-- ... and check again.
+SELECT buffile_seek(0, 0);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_read(16);
+ buffile_read
+------------------
+ \x00616263616263
+(1 row)
+
+SELECT buffile_close();
+ buffile_close
+---------------
+
+(1 row)
+
+COMMIT;
diff --git a/src/test/modules/buffile/expected/test_02.out b/src/test/modules/buffile/expected/test_02.out
new file mode 100644
index 0000000000..f783d0cb24
--- /dev/null
+++ b/src/test/modules/buffile/expected/test_02.out
@@ -0,0 +1,48 @@
+BEGIN;
+SELECT buffile_create();
+ buffile_create
+----------------
+
+(1 row)
+
+SELECT buffile_seek(0, 8189);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+-- Initialize the last 3 positions of the first buffer and the initial 3
+-- positions of the 2nd buffer.
+SELECT buffile_write('abcdef');
+ buffile_write
+---------------
+ 6
+(1 row)
+
+SELECT buffile_seek(0, 0);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+-- Read the first buffer.
+SELECT length(buffile_read(8192));
+ length
+--------
+ 8192
+(1 row)
+
+-- Only 3 bytes of the 2nd buffer should be fetched.
+SELECT length(buffile_read(8192));
+ length
+--------
+ 3
+(1 row)
+
+SELECT buffile_close();
+ buffile_close
+---------------
+
+(1 row)
+
+COMMIT;
diff --git a/src/test/modules/buffile/expected/test_03.out b/src/test/modules/buffile/expected/test_03.out
new file mode 100644
index 0000000000..e899fa3b38
--- /dev/null
+++ b/src/test/modules/buffile/expected/test_03.out
@@ -0,0 +1,27 @@
+BEGIN;
+SELECT buffile_create();
+ buffile_create
+----------------
+
+(1 row)
+
+-- Read from an empty file.
+SELECT buffile_seek(0, 8);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_read(16);
+ buffile_read
+--------------
+ \x
+(1 row)
+
+SELECT buffile_close();
+ buffile_close
+---------------
+
+(1 row)
+
+COMMIT;
diff --git a/src/test/modules/buffile/expected/test_04.out b/src/test/modules/buffile/expected/test_04.out
new file mode 100644
index 0000000000..1f8eeabe48
--- /dev/null
+++ b/src/test/modules/buffile/expected/test_04.out
@@ -0,0 +1,84 @@
+BEGIN;
+SELECT buffile_create();
+ buffile_create
+----------------
+
+(1 row)
+
+-- Write something near the end of the first buffer, but leave some trailing
+-- space.
+SELECT buffile_seek(0, 8184);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_write('abcd');
+ buffile_write
+---------------
+ 4
+(1 row)
+
+-- Leave the 2nd buffer empty, as well as a few leading bytes. Thus we should
+-- get a hole that spans the whole 2nd buffer as well as a few adjacent bytes
+-- on each side.
+SELECT buffile_seek(0, 2 * 8192 + 4);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_write('efgh');
+ buffile_write
+---------------
+ 4
+(1 row)
+
+-- Check the initial part of the hole, which crosses the boundary of the 1st
+-- and the 2nd buffer.
+SELECT buffile_seek(0, 8184);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_read(16);
+ buffile_read
+------------------------------------
+ \x61626364000000000000000000000000
+(1 row)
+
+-- Check the trailing part of the whole, which crosses the boundary of the 2nd
+-- and the 3rd buffer.
+SELECT buffile_seek(0, 2 * 8192 - 8);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_read(16);
+ buffile_read
+------------------------------------
+ \x00000000000000000000000065666768
+(1 row)
+
+-- Check that the hole contains nothing but zeroes.
+SELECT buffile_seek(0, 8192 - 4);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT btrim(buffile_read(8192 + 8), '\x00');
+ btrim
+-------
+ \x
+(1 row)
+
+SELECT buffile_close();
+ buffile_close
+---------------
+
+(1 row)
+
+COMMIT;
diff --git a/src/test/modules/buffile/expected/test_05.out b/src/test/modules/buffile/expected/test_05.out
new file mode 100644
index 0000000000..6a73147711
--- /dev/null
+++ b/src/test/modules/buffile/expected/test_05.out
@@ -0,0 +1,33 @@
+BEGIN;
+SELECT buffile_create();
+ buffile_create
+----------------
+
+(1 row)
+
+-- Seek does not extend the file if it's not followed by write.
+SELECT buffile_seek(0, 1);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_seek(0, 0);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_read(2);
+ buffile_read
+--------------
+ \x
+(1 row)
+
+SELECT buffile_close();
+ buffile_close
+---------------
+
+(1 row)
+
+COMMIT;
diff --git a/src/test/modules/buffile/expected/test_06.out b/src/test/modules/buffile/expected/test_06.out
new file mode 100644
index 0000000000..def9af5112
--- /dev/null
+++ b/src/test/modules/buffile/expected/test_06.out
@@ -0,0 +1,41 @@
+-- This test shows that the first component file (segment) stays empty, read
+-- stops prematurely even if it starts on that segment, even though it'd
+-- otherwise receive some data from the following one.
+BEGIN;
+SELECT buffile_create();
+ buffile_create
+----------------
+
+(1 row)
+
+SELECT buffile_seek(0, 32768);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_write('a');
+ buffile_write
+---------------
+ 1
+(1 row)
+
+SELECT buffile_seek(0, 32767);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_read(2);
+ buffile_read
+--------------
+ \x
+(1 row)
+
+SELECT buffile_close();
+ buffile_close
+---------------
+
+(1 row)
+
+COMMIT;
diff --git a/src/test/modules/buffile/expected/test_07.out b/src/test/modules/buffile/expected/test_07.out
new file mode 100644
index 0000000000..215280f0b0
--- /dev/null
+++ b/src/test/modules/buffile/expected/test_07.out
@@ -0,0 +1,39 @@
+BEGIN;
+SELECT buffile_create();
+ buffile_create
+----------------
+
+(1 row)
+
+-- Write data at component file boundary and try to read it.
+SELECT buffile_seek(0, 32768);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_write('abcd');
+ buffile_write
+---------------
+ 4
+(1 row)
+
+SELECT buffile_seek(0, 32768);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_read(8);
+ buffile_read
+--------------
+ \x61626364
+(1 row)
+
+SELECT buffile_close();
+ buffile_close
+---------------
+
+(1 row)
+
+COMMIT;
diff --git a/src/test/modules/buffile/expected/test_08.out b/src/test/modules/buffile/expected/test_08.out
new file mode 100644
index 0000000000..7c1069930c
--- /dev/null
+++ b/src/test/modules/buffile/expected/test_08.out
@@ -0,0 +1,39 @@
+BEGIN;
+SELECT buffile_create();
+ buffile_create
+----------------
+
+(1 row)
+
+-- Write data across component file boundary and try to read it.
+SELECT buffile_seek(0, 32766);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_write('abcd');
+ buffile_write
+---------------
+ 4
+(1 row)
+
+SELECT buffile_seek(0, 32766);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_read(8);
+ buffile_read
+--------------
+ \x61626364
+(1 row)
+
+SELECT buffile_close();
+ buffile_close
+---------------
+
+(1 row)
+
+COMMIT;
diff --git a/src/test/modules/buffile/expected/test_09.out b/src/test/modules/buffile/expected/test_09.out
new file mode 100644
index 0000000000..b9d325b676
--- /dev/null
+++ b/src/test/modules/buffile/expected/test_09.out
@@ -0,0 +1,39 @@
+BEGIN;
+SELECT buffile_create();
+ buffile_create
+----------------
+
+(1 row)
+
+-- Write data across buffer boundary and try to read it.
+SELECT buffile_seek(0, 8190);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_write('abcd');
+ buffile_write
+---------------
+ 4
+(1 row)
+
+SELECT buffile_seek(0, 8190);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_read(8);
+ buffile_read
+--------------
+ \x61626364
+(1 row)
+
+SELECT buffile_close();
+ buffile_close
+---------------
+
+(1 row)
+
+COMMIT;
diff --git a/src/test/modules/buffile/expected/test_10.out b/src/test/modules/buffile/expected/test_10.out
new file mode 100644
index 0000000000..8e457fdcda
--- /dev/null
+++ b/src/test/modules/buffile/expected/test_10.out
@@ -0,0 +1,76 @@
+BEGIN;
+SELECT buffile_create();
+ buffile_create
+----------------
+
+(1 row)
+
+-- Write some data at the end of the buffer.
+SELECT buffile_seek(0, 8188);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_write('abcd');
+ buffile_write
+---------------
+ 4
+(1 row)
+
+SELECT buffile_seek(0, 8189);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+-- Enforce flush with the write position not at the end of the buffer. This is
+-- special by not moving curOffset to the next buffer.
+SELECT buffile_read(1);
+ buffile_read
+--------------
+ \x62
+(1 row)
+
+-- Therefore the next writes should eventually affect the original data. (Here
+-- we also test going directly from read to write and vice versa.)
+SELECT buffile_write('x');
+ buffile_write
+---------------
+ 1
+(1 row)
+
+SELECT buffile_read(1);
+ buffile_read
+--------------
+ \x64
+(1 row)
+
+-- Start a new buffer, i.e. force flushing of the previous one.
+SELECT buffile_write('z');
+ buffile_write
+---------------
+ 1
+(1 row)
+
+-- Check that the 'x' and 'y' letters are in the first buffer, not in the
+-- 2nd. (We read enough data to find any non-zero bytes in the 2nd buffer.)
+SELECT buffile_seek(0, 8188);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+SELECT buffile_read(4 + 8192);
+ buffile_read
+--------------
+ \x616278647a
+(1 row)
+
+SELECT buffile_close();
+ buffile_close
+---------------
+
+(1 row)
+
+COMMIT;
diff --git a/src/test/modules/buffile/expected/test_11.out b/src/test/modules/buffile/expected/test_11.out
new file mode 100644
index 0000000000..c6804d08af
--- /dev/null
+++ b/src/test/modules/buffile/expected/test_11.out
@@ -0,0 +1,34 @@
+BEGIN;
+SELECT buffile_create();
+ buffile_create
+----------------
+
+(1 row)
+
+SELECT buffile_write('abcd');
+ buffile_write
+---------------
+ 4
+(1 row)
+
+-- Seek beyond EOF not followed by write.
+SELECT buffile_seek(0, 5);
+ buffile_seek
+--------------
+ 0
+(1 row)
+
+-- Nothing should be fetched.
+SELECT buffile_read(8);
+ buffile_read
+--------------
+ \x
+(1 row)
+
+SELECT buffile_close();
+ buffile_close
+---------------
+
+(1 row)
+
+COMMIT;
diff --git a/src/test/modules/buffile/expected/test_12.out b/src/test/modules/buffile/expected/test_12.out
new file mode 100644
index 0000000000..3dce7eaf0f
--- /dev/null
+++ b/src/test/modules/buffile/expected/test_12.out
@@ -0,0 +1,12 @@
+SELECT buffile_test_shared();
+ buffile_test_shared
+---------------------
+
+(1 row)
+
+SELECT buffile_test_shared_append();
+ buffile_test_shared_append
+----------------------------
+
+(1 row)
+
diff --git a/src/test/modules/buffile/sql/test_00.sql b/src/test/modules/buffile/sql/test_00.sql
new file mode 100644
index 0000000000..c907a5e195
--- /dev/null
+++ b/src/test/modules/buffile/sql/test_00.sql
@@ -0,0 +1,18 @@
+CREATE EXTENSION IF NOT EXISTS buffile;
+
+-- This test only verifies that PG is compiled with a component file size of
+-- 32 kB (i.e. 4 buffers of 8 kB) instead of 1 GB. That seems appropriate for
+-- testing. Some other tests may rely on it.
+BEGIN;
+SELECT buffile_create();
+-- Skip the first component file.
+SELECT buffile_seek(0, 32768);
+-- Write the first byte of the second component file. We can't simply
+-- buffile_seek() beyond the position 32768 as it would return EOF.
+SELECT buffile_write('a');
+-- Enforce BufFileFlush(), which actually adds the component file.
+SELECT buffile_read(1);
+-- Check that we're in the 2nd file, i.e. the file size is as expected.
+SELECT buffile_assert_fileno(1);
+SELECT buffile_close();
+COMMIT;
diff --git a/src/test/modules/buffile/sql/test_01.sql b/src/test/modules/buffile/sql/test_01.sql
new file mode 100644
index 0000000000..8cc87de841
--- /dev/null
+++ b/src/test/modules/buffile/sql/test_01.sql
@@ -0,0 +1,14 @@
+BEGIN;
+SELECT buffile_create();
+SELECT buffile_seek(0, 1);
+SELECT buffile_write('abc');
+SELECT buffile_seek(0, 0);
+-- Check that the trailing zeroes are not fetched.
+SELECT buffile_read(16);
+-- Adjust the number of useful bytes.
+SELECT buffile_write('abc');
+-- ... and check again.
+SELECT buffile_seek(0, 0);
+SELECT buffile_read(16);
+SELECT buffile_close();
+COMMIT;
diff --git a/src/test/modules/buffile/sql/test_02.sql b/src/test/modules/buffile/sql/test_02.sql
new file mode 100644
index 0000000000..35d34722d7
--- /dev/null
+++ b/src/test/modules/buffile/sql/test_02.sql
@@ -0,0 +1,13 @@
+BEGIN;
+SELECT buffile_create();
+SELECT buffile_seek(0, 8189);
+-- Initialize the last 3 positions of the first buffer and the initial 3
+-- positions of the 2nd buffer.
+SELECT buffile_write('abcdef');
+SELECT buffile_seek(0, 0);
+-- Read the first buffer.
+SELECT length(buffile_read(8192));
+-- Only 3 bytes of the 2nd buffer should be fetched.
+SELECT length(buffile_read(8192));
+SELECT buffile_close();
+COMMIT;
diff --git a/src/test/modules/buffile/sql/test_03.sql b/src/test/modules/buffile/sql/test_03.sql
new file mode 100644
index 0000000000..a95391f7c3
--- /dev/null
+++ b/src/test/modules/buffile/sql/test_03.sql
@@ -0,0 +1,7 @@
+BEGIN;
+SELECT buffile_create();
+-- Read from an empty file.
+SELECT buffile_seek(0, 8);
+SELECT buffile_read(16);
+SELECT buffile_close();
+COMMIT;
diff --git a/src/test/modules/buffile/sql/test_04.sql b/src/test/modules/buffile/sql/test_04.sql
new file mode 100644
index 0000000000..64e8d39f94
--- /dev/null
+++ b/src/test/modules/buffile/sql/test_04.sql
@@ -0,0 +1,25 @@
+BEGIN;
+SELECT buffile_create();
+-- Write something near the end of the first buffer, but leave some trailing
+-- space.
+SELECT buffile_seek(0, 8184);
+SELECT buffile_write('abcd');
+-- Leave the 2nd buffer empty, as well as a few leading bytes. Thus we should
+-- get a hole that spans the whole 2nd buffer as well as a few adjacent bytes
+-- on each side.
+SELECT buffile_seek(0, 2 * 8192 + 4);
+SELECT buffile_write('efgh');
+-- Check the initial part of the hole, which crosses the boundary of the 1st
+-- and the 2nd buffer.
+SELECT buffile_seek(0, 8184);
+SELECT buffile_read(16);
+-- Check the trailing part of the whole, which crosses the boundary of the 2nd
+-- and the 3rd buffer.
+SELECT buffile_seek(0, 2 * 8192 - 8);
+SELECT buffile_read(16);
+-- Check that the hole contains nothing but zeroes.
+SELECT buffile_seek(0, 8192 - 4);
+SELECT btrim(buffile_read(8192 + 8), '\x00');
+
+SELECT buffile_close();
+COMMIT;
diff --git a/src/test/modules/buffile/sql/test_05.sql b/src/test/modules/buffile/sql/test_05.sql
new file mode 100644
index 0000000000..5fd642e558
--- /dev/null
+++ b/src/test/modules/buffile/sql/test_05.sql
@@ -0,0 +1,8 @@
+BEGIN;
+SELECT buffile_create();
+-- Seek does not extend the file if it's not followed by write.
+SELECT buffile_seek(0, 1);
+SELECT buffile_seek(0, 0);
+SELECT buffile_read(2);
+SELECT buffile_close();
+COMMIT;
diff --git a/src/test/modules/buffile/sql/test_06.sql b/src/test/modules/buffile/sql/test_06.sql
new file mode 100644
index 0000000000..bbf506ec0e
--- /dev/null
+++ b/src/test/modules/buffile/sql/test_06.sql
@@ -0,0 +1,11 @@
+-- This test shows that the first component file (segment) stays empty, read
+-- stops prematurely even if it starts on that segment, even though it'd
+-- otherwise receive some data from the following one.
+BEGIN;
+SELECT buffile_create();
+SELECT buffile_seek(0, 32768);
+SELECT buffile_write('a');
+SELECT buffile_seek(0, 32767);
+SELECT buffile_read(2);
+SELECT buffile_close();
+COMMIT;
diff --git a/src/test/modules/buffile/sql/test_07.sql b/src/test/modules/buffile/sql/test_07.sql
new file mode 100644
index 0000000000..ca78506ecb
--- /dev/null
+++ b/src/test/modules/buffile/sql/test_07.sql
@@ -0,0 +1,9 @@
+BEGIN;
+SELECT buffile_create();
+-- Write data at component file boundary and try to read it.
+SELECT buffile_seek(0, 32768);
+SELECT buffile_write('abcd');
+SELECT buffile_seek(0, 32768);
+SELECT buffile_read(8);
+SELECT buffile_close();
+COMMIT;
diff --git a/src/test/modules/buffile/sql/test_08.sql b/src/test/modules/buffile/sql/test_08.sql
new file mode 100644
index 0000000000..0e5b1de383
--- /dev/null
+++ b/src/test/modules/buffile/sql/test_08.sql
@@ -0,0 +1,9 @@
+BEGIN;
+SELECT buffile_create();
+-- Write data across component file boundary and try to read it.
+SELECT buffile_seek(0, 32766);
+SELECT buffile_write('abcd');
+SELECT buffile_seek(0, 32766);
+SELECT buffile_read(8);
+SELECT buffile_close();
+COMMIT;
diff --git a/src/test/modules/buffile/sql/test_09.sql b/src/test/modules/buffile/sql/test_09.sql
new file mode 100644
index 0000000000..cc7060932e
--- /dev/null
+++ b/src/test/modules/buffile/sql/test_09.sql
@@ -0,0 +1,9 @@
+BEGIN;
+SELECT buffile_create();
+-- Write data across buffer boundary and try to read it.
+SELECT buffile_seek(0, 8190);
+SELECT buffile_write('abcd');
+SELECT buffile_seek(0, 8190);
+SELECT buffile_read(8);
+SELECT buffile_close();
+COMMIT;
diff --git a/src/test/modules/buffile/sql/test_10.sql b/src/test/modules/buffile/sql/test_10.sql
new file mode 100644
index 0000000000..63af760d9f
--- /dev/null
+++ b/src/test/modules/buffile/sql/test_10.sql
@@ -0,0 +1,25 @@
+BEGIN;
+SELECT buffile_create();
+-- Write some data at the end of the buffer.
+SELECT buffile_seek(0, 8188);
+SELECT buffile_write('abcd');
+SELECT buffile_seek(0, 8189);
+-- Enforce flush with the write position not at the end of the buffer. This is
+-- special by not moving curOffset to the next buffer.
+SELECT buffile_read(1);
+
+-- Therefore the next writes should eventually affect the original data. (Here
+-- we also test going directly from read to write and vice versa.)
+SELECT buffile_write('x');
+SELECT buffile_read(1);
+
+-- Start a new buffer, i.e. force flushing of the previous one.
+SELECT buffile_write('z');
+
+-- Check that the 'x' and 'y' letters are in the first buffer, not in the
+-- 2nd. (We read enough data to find any non-zero bytes in the 2nd buffer.)
+SELECT buffile_seek(0, 8188);
+SELECT buffile_read(4 + 8192);
+
+SELECT buffile_close();
+COMMIT;
diff --git a/src/test/modules/buffile/sql/test_11.sql b/src/test/modules/buffile/sql/test_11.sql
new file mode 100644
index 0000000000..94300d253f
--- /dev/null
+++ b/src/test/modules/buffile/sql/test_11.sql
@@ -0,0 +1,9 @@
+BEGIN;
+SELECT buffile_create();
+SELECT buffile_write('abcd');
+-- Seek beyond EOF not followed by write.
+SELECT buffile_seek(0, 5);
+-- Nothing should be fetched.
+SELECT buffile_read(8);
+SELECT buffile_close();
+COMMIT;
diff --git a/src/test/modules/buffile/sql/test_12.sql b/src/test/modules/buffile/sql/test_12.sql
new file mode 100644
index 0000000000..a28ae395ef
--- /dev/null
+++ b/src/test/modules/buffile/sql/test_12.sql
@@ -0,0 +1,2 @@
+SELECT buffile_test_shared();
+SELECT buffile_test_shared_append();
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
@ 2019-08-02 06:40 ` Shawn Wang <shawn.wang@highgo.ca>
2019-09-03 16:56 ` Re: WIP: Data at rest encryption Alvaro Herrera <alvherre@2ndquadrant.com>
3 siblings, 1 reply; 93+ messages in thread
From: Shawn Wang @ 2019-08-02 06:40 UTC (permalink / raw)
To: pgsql-hackers@lists.postgresql.org; +Cc: Antonin Houska <ah@cybertec.at>; Ants Aasma <ants.aasma@eesti.ee>
The following review has been posted through the commitfest application:
make installcheck-world: tested, passed
Implements feature: tested, failed
Spec compliant: tested, failed
Documentation: not tested
Hi Antonin,
It is very glad to see the new patch. I used the public patches a long time ago.
I did some tests like the stream replication, much data running, temporary files encryption.
I found that there is an issue in the src/backend/storage/file/encryption.c. You should put block_size = EVP_CIPHER_CTX_block_size(ctx); under the #ifdef USE_ASSERT_CHECKING.
There is some problem to merge your patches to the latest kernel in the pg_ctl.c.
Regards,
--
Shawn Wang
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2019-08-02 06:40 ` Re: WIP: Data at rest encryption Shawn Wang <shawn.wang@highgo.ca>
@ 2019-09-03 16:56 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-03 17:30 ` Re: WIP: Data at rest encryption Ibrar Ahmed <ibrar.ahmad@gmail.com>
2019-09-04 04:56 ` Re: WIP: Data at rest encryption Antonin Houska <ah@cybertec.at>
2019-09-04 09:23 ` Re: WIP: Data at rest encryption Shawn Wang <shawn.wang@highgo.ca>
0 siblings, 3 replies; 93+ messages in thread
From: Alvaro Herrera @ 2019-09-03 16:56 UTC (permalink / raw)
To: Shawn Wang <shawn.wang@highgo.ca>; +Cc: pgsql-hackers@lists.postgresql.org, Antonin Houska <ah@cybertec.at>; Ants Aasma <ants.aasma@eesti.ee>
On 2019-Aug-02, Shawn Wang wrote:
> Hi Antonin,
> It is very glad to see the new patch. I used the public patches a long time ago.
> I did some tests like the stream replication, much data running, temporary files encryption.
> I found that there is an issue in the src/backend/storage/file/encryption.c. You should put block_size = EVP_CIPHER_CTX_block_size(ctx); under the #ifdef USE_ASSERT_CHECKING.
> There is some problem to merge your patches to the latest kernel in the pg_ctl.c.
Is a new, fixed version going to be posted soon? It's been a while.
Also, apologies if this has been asked before, but: how does this patch
relate to the stuff being discussed in
https://postgr.es/m/031401d3f41d$5c70ed90$1552c8b0$@lab.ntt.co.jp ?
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2019-08-02 06:40 ` Re: WIP: Data at rest encryption Shawn Wang <shawn.wang@highgo.ca>
2019-09-03 16:56 ` Re: WIP: Data at rest encryption Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2019-09-03 17:30 ` Ibrar Ahmed <ibrar.ahmad@gmail.com>
2 siblings, 0 replies; 93+ messages in thread
From: Ibrar Ahmed @ 2019-09-03 17:30 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Shawn Wang <shawn.wang@highgo.ca>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>; Antonin Houska <ah@cybertec.at>; Ants Aasma <ants.aasma@eesti.ee>
On Tue, Sep 3, 2019 at 10:21 PM Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:
> On 2019-Aug-02, Shawn Wang wrote:
>
> > Hi Antonin,
> > It is very glad to see the new patch. I used the public patches a long
> time ago.
> > I did some tests like the stream replication, much data running,
> temporary files encryption.
> > I found that there is an issue in the
> src/backend/storage/file/encryption.c. You should put block_size =
> EVP_CIPHER_CTX_block_size(ctx); under the #ifdef USE_ASSERT_CHECKING.
> > There is some problem to merge your patches to the latest kernel in the
> pg_ctl.c.
>
> Is a new, fixed version going to be posted soon? It's been a while.
>
> Also, apologies if this has been asked before, but: how does this patch
> relate to the stuff being discussed in
> https://postgr.es/m/031401d3f41d$5c70ed90$1552c8b0$@lab.ntt.co.jp ?
>
> Yes, it is related to that, but we don't have that on our agenda in a
weekly meeting. It has
many conflicting points about what we are doing. Swada / Bruce can comment
on that.
> --
> Álvaro Herrera https://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>
>
>
--
Ibrar Ahmed
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2019-08-02 06:40 ` Re: WIP: Data at rest encryption Shawn Wang <shawn.wang@highgo.ca>
2019-09-03 16:56 ` Re: WIP: Data at rest encryption Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2019-09-04 04:56 ` Antonin Houska <ah@cybertec.at>
2019-11-25 08:13 ` Re: WIP: Data at rest encryption Michael Paquier <michael@paquier.xyz>
2 siblings, 1 reply; 93+ messages in thread
From: Antonin Houska @ 2019-09-04 04:56 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Shawn Wang <shawn.wang@highgo.ca>; pgsql-hackers@lists.postgresql.org, Ants Aasma <ants.aasma@eesti.ee>
Alvaro Herrera <alvherre@2ndquadrant.com> wrote:
> On 2019-Aug-02, Shawn Wang wrote:
>
> > Hi Antonin,
> > It is very glad to see the new patch. I used the public patches a long time ago.
> > I did some tests like the stream replication, much data running, temporary files encryption.
> > I found that there is an issue in the src/backend/storage/file/encryption.c. You should put block_size = EVP_CIPHER_CTX_block_size(ctx); under the #ifdef USE_ASSERT_CHECKING.
> > There is some problem to merge your patches to the latest kernel in the pg_ctl.c.
>
> Is a new, fixed version going to be posted soon? It's been a while.
>
> Also, apologies if this has been asked before, but: how does this patch
> relate to the stuff being discussed in
> https://postgr.es/m/031401d3f41d$5c70ed90$1552c8b0$@lab.ntt.co.jp ?
This thread started later than our effort but important design questions are
being discussed there. So far there seems to be no consensus whether
full-instance encryption should be implemented first, so any effort spent on
this patch might get wasted. When/if there will be an agreement on the design,
we'll see how much of this patch can be used.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2019-08-02 06:40 ` Re: WIP: Data at rest encryption Shawn Wang <shawn.wang@highgo.ca>
2019-09-03 16:56 ` Re: WIP: Data at rest encryption Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-04 04:56 ` Re: WIP: Data at rest encryption Antonin Houska <ah@cybertec.at>
@ 2019-11-25 08:13 ` Michael Paquier <michael@paquier.xyz>
0 siblings, 0 replies; 93+ messages in thread
From: Michael Paquier @ 2019-11-25 08:13 UTC (permalink / raw)
To: Antonin Houska <ah@cybertec.at>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; Shawn Wang <shawn.wang@highgo.ca>; pgsql-hackers@lists.postgresql.org, Ants Aasma <ants.aasma@eesti.ee>
On Wed, Sep 04, 2019 at 06:56:18AM +0200, Antonin Houska wrote:
> This thread started later than our effort but important design questions are
> being discussed there. So far there seems to be no consensus whether
> full-instance encryption should be implemented first, so any effort spent on
> this patch might get wasted. When/if there will be an agreement on the design,
> we'll see how much of this patch can be used.
I see. I have marked the patch as returned with feedback in CF
2019-11. Let's see how the other one finishes, before perhaps coming
back to this one.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: WIP: Data at rest encryption
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2019-08-02 06:40 ` Re: WIP: Data at rest encryption Shawn Wang <shawn.wang@highgo.ca>
2019-09-03 16:56 ` Re: WIP: Data at rest encryption Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2019-09-04 09:23 ` Shawn Wang <shawn.wang@highgo.ca>
2 siblings, 0 replies; 93+ messages in thread
From: Shawn Wang @ 2019-09-04 09:23 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: pgsql-hackers <pgsql-hackers@lists.postgresql.org>; Antonin Houska <ah@cybertec.at>; Ants Aasma <ants.aasma@eesti.ee>
---- On Wed, 04 Sep 2019 00:56:15 +0800 Alvaro Herrera <mailto:alvherre@2ndquadrant.com> wrote ----
On 2019-Aug-02, Shawn Wang wrote:
> Hi Antonin,
> It is very glad to see the new patch. I used the public patches a long time ago.
> I did some tests like the stream replication, much data running, temporary files encryption.
> I found that there is an issue in the src/backend/storage/file/encryption.c. You should put block_size = EVP_CIPHER_CTX_block_size(ctx); under the #ifdef USE_ASSERT_CHECKING.
> There is some problem to merge your patches to the latest kernel in the pg_ctl.c.
Is a new, fixed version going to be posted soon? It's been a while.
Also, apologies if this has been asked before, but: how does this patch
relate to the stuff being discussed in
https://postgr.es/m/031401d3f41d$5c70ed90$1552c8b0$@lab.ntt.co.jp ?
Hi Álvaro,
Thank you for a reply.
I mainly said that the issues in the src/backend/storage/file/encryption.c. If somebody want to use these patches, I think Antonin need to fix it.
It does not relate to the stuff being discussed in TDE. As I know, some company use these patches to encrypt data, even if these issues don't matter.
Regards,
--
Shawn Wang
^ permalink raw reply [nested|flat] 93+ messages in thread
end of thread, other threads:[~2019-11-25 08:13 UTC | newest]
Thread overview: 93+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2016-06-07 13:56 WIP: Data at rest encryption Ants Aasma <ants.aasma@gmail.com>
2016-06-10 02:23 ` Haribabu Kommi <kommi.haribabu@gmail.com>
2016-06-12 07:13 ` Ants Aasma <ants.aasma@gmail.com>
2016-06-13 05:17 ` Michael Paquier <michael.paquier@gmail.com>
2016-06-13 13:10 ` Ants Aasma <ants.aasma@eesti.ee>
2016-06-13 15:18 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2016-06-14 02:20 ` Haribabu Kommi <kommi.haribabu@gmail.com>
2016-06-14 19:59 ` Jim Nasby <Jim.Nasby@BlueTreble.com>
2016-06-15 20:05 ` PostgreSQL - Hans-Jürgen Schönig <postgres@cybertec.at>
2016-06-13 15:07 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-12 19:38 ` Robert Haas <robertmhaas@gmail.com>
2017-06-12 21:11 ` Ants Aasma <ants.aasma@eesti.ee>
2017-06-13 13:07 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 09:04 ` Aleksander Alekseev <a.alekseev@postgrespro.ru>
2017-06-14 12:46 ` Kenneth Marshall <ktm@rice.edu>
2017-06-14 13:05 ` Aleksander Alekseev <a.alekseev@postgrespro.ru>
2017-06-14 13:13 ` Aleksander Alekseev <a.alekseev@postgrespro.ru>
2017-06-14 14:31 ` Bruce Momjian <bruce@momjian.us>
2017-06-14 21:32 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 13:24 ` Stephen Frost <sfrost@snowman.net>
2017-06-13 14:28 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 15:04 ` Stephen Frost <sfrost@snowman.net>
2017-06-13 16:04 ` Bruce Momjian <bruce@momjian.us>
2017-06-13 16:23 ` Stephen Frost <sfrost@snowman.net>
2017-06-13 16:28 ` Robert Haas <robertmhaas@gmail.com>
2017-06-13 16:55 ` Joe Conway <mail@joeconway.com>
2017-06-13 16:58 ` Bruce Momjian <bruce@momjian.us>
2017-06-13 17:05 ` Stephen Frost <sfrost@snowman.net>
2017-06-13 17:09 ` Joe Conway <mail@joeconway.com>
2017-06-13 17:20 ` Stephen Frost <sfrost@snowman.net>
2017-06-13 17:26 ` Joe Conway <mail@joeconway.com>
2017-06-13 17:44 ` Stephen Frost <sfrost@snowman.net>
2017-06-13 17:48 ` Bruce Momjian <bruce@momjian.us>
2017-06-13 18:06 ` Stephen Frost <sfrost@snowman.net>
2017-06-13 16:44 ` Bruce Momjian <bruce@momjian.us>
2017-06-13 17:01 ` Stephen Frost <sfrost@snowman.net>
2017-06-13 17:13 ` Bruce Momjian <bruce@momjian.us>
2017-06-13 17:25 ` Stephen Frost <sfrost@snowman.net>
2017-06-13 17:43 ` Bruce Momjian <bruce@momjian.us>
2017-06-13 18:23 ` Stephen Frost <sfrost@snowman.net>
2017-06-13 18:29 ` Bruce Momjian <bruce@momjian.us>
2017-06-13 18:38 ` Stephen Frost <sfrost@snowman.net>
2017-06-13 18:56 ` Bruce Momjian <bruce@momjian.us>
2017-06-13 19:20 ` Stephen Frost <sfrost@snowman.net>
2017-06-13 19:43 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 22:11 ` Stephen Frost <sfrost@snowman.net>
2017-06-14 21:36 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-14 21:41 ` Stephen Frost <sfrost@snowman.net>
2017-06-15 15:53 ` Robert Haas <robertmhaas@gmail.com>
2017-06-15 16:06 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-15 19:09 ` Robert Haas <robertmhaas@gmail.com>
2017-06-15 20:29 ` Bruce Momjian <bruce@momjian.us>
2017-06-15 21:04 ` Robert Haas <robertmhaas@gmail.com>
2017-06-15 21:12 ` Bruce Momjian <bruce@momjian.us>
2017-06-15 22:41 ` Stephen Frost <sfrost@snowman.net>
2017-06-15 23:13 ` Bruce Momjian <bruce@momjian.us>
2017-06-15 23:27 ` Stephen Frost <sfrost@snowman.net>
2017-06-15 23:44 ` Bruce Momjian <bruce@momjian.us>
2017-06-15 23:51 ` Stephen Frost <sfrost@snowman.net>
2017-06-15 23:51 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2017-06-15 23:55 ` Bruce Momjian <bruce@momjian.us>
2017-06-19 16:30 ` Robert Haas <robertmhaas@gmail.com>
2017-06-19 16:30 ` Robert Haas <robertmhaas@gmail.com>
2017-06-15 23:56 ` Andres Freund <andres@anarazel.de>
2017-06-16 00:08 ` Bruce Momjian <bruce@momjian.us>
2017-06-16 02:17 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2017-06-16 08:06 ` Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
2017-06-16 16:04 ` Bruce Momjian <bruce@momjian.us>
2017-06-16 15:41 ` Bruce Momjian <bruce@momjian.us>
2017-06-13 19:45 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 22:13 ` Stephen Frost <sfrost@snowman.net>
2017-06-13 19:51 ` Bruce Momjian <bruce@momjian.us>
2017-06-13 20:08 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2017-06-13 20:10 ` Bruce Momjian <bruce@momjian.us>
2017-06-13 22:29 ` Stephen Frost <sfrost@snowman.net>
2017-06-14 14:53 ` Bruce Momjian <bruce@momjian.us>
2017-06-13 20:52 ` David Fetter <david@fetter.org>
2017-06-13 15:35 ` Robert Haas <robertmhaas@gmail.com>
2017-06-13 15:51 ` Bruce Momjian <bruce@momjian.us>
2017-06-14 15:10 ` Ants Aasma <ants.aasma@eesti.ee>
2017-06-14 15:26 ` Bruce Momjian <bruce@momjian.us>
2017-06-14 15:41 ` Ants Aasma <ants.aasma@eesti.ee>
2017-06-14 15:57 ` Bruce Momjian <bruce@momjian.us>
2017-06-18 22:11 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-06-27 08:02 ` Antonin Houska <ah@cybertec.at>
2018-07-13 07:56 ` Toshi Harada <harada.toshi@po.ntt-tx.co.jp>
2018-07-25 11:47 ` Antonin Houska <ah@cybertec.at>
2019-08-02 06:40 ` Shawn Wang <shawn.wang@highgo.ca>
2019-09-03 16:56 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-03 17:30 ` Ibrar Ahmed <ibrar.ahmad@gmail.com>
2019-09-04 04:56 ` Antonin Houska <ah@cybertec.at>
2019-11-25 08:13 ` Michael Paquier <michael@paquier.xyz>
2019-09-04 09:23 ` Shawn Wang <shawn.wang@highgo.ca>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox