pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feed"WIP: Data at rest encryption" patch and, 2 phase commit.
3+ messages / 2 participants
[nested] [flat]
* "WIP: Data at rest encryption" patch and, 2 phase commit.
@ 2018-07-24 22:51 Toshi Harada <harada.toshi@po.ntt-tx.co.jp>
1 sibling, 0 replies; 3+ messages in thread
From: Toshi Harada @ 2018-07-24 22:51 UTC (permalink / raw)
To: pgsql-hackers
Hi.
I applied the patch "WIP: Data at rest encryption" to PostgreSQL 11 - beta 2 and I'm working on it.
When this patch is applied, the following problem occurs.
* An error occurs when CHECKPOINT is executed during two-phase commit.
* After an error occurs, if you stop PostgreSQL, it will never start again.
(1) First, execute PREPARE TRANSACTION.
postgres=# BEGIN;
BEGIN
postgres=# PREPARE TRANSACTION 'foo';
PREPARE TRANSACTION
postgres=#
(2) Execute the CHECKPOINT command from another terminal.
CHEKPOINT command fails.
postgres=# CHECKPOINT;
ERROR: checkpoint request failed
HINT: Consult recent messages in the server log for details.
postgres=#
(3) ROLLBACK PREPARED command also fails.
postgres=# ROLLBACK PREPARED 'foo';
ERROR: could not read two-phase state from WAL at 0/167EBA0
postgres=#
(4) Shut down the PostgreSQL server.
During shutdown, a "could not read two-phase state from WAL" error occurs.
2018-07-23 14:49:08.924 JST [15821] LOG: received fast shutdown request
2018-07-23 14:49:08.925 JST [15821] LOG: aborting any active transactions
2018-07-23 14:49:08.925 JST [15831] FATAL: terminating connection due to administrator command
2018-07-23 14:49:08.928 JST [15821] LOG: background worker "logical replication launcher" (PID 15829) exited with exit code 1
2018-07-23 14:49:08.928 JST [15824] LOG: shutting down
2018-07-23 14:49:08.935 JST [15824] FATAL: could not read two-phase state from WAL at 0/167EBA0
2018-07-23 14:49:08.936 JST [15821] LOG: checkpointer process (PID 15824) exited with exit code 1
2018-07-23 14:49:08.936 JST [15821] LOG: terminating any other active server processes
2018-07-23 14:49:08.937 JST [15821] LOG: abnormal database system shutdown
2018-07-23 14:49:08.945 JST [15821] LOG: database system is shut down
(5) When restarting the PostgreSQL server, an error(could not read two-phase state from WAL) occurs
and the PostgreSQL server can not be started.
2018-07-23 14:49:42.489 JST [15864] LOG: listening on IPv6 address "::1", port 5432
2018-07-23 14:49:42.489 JST [15864] LOG: listening on IPv4 address "127.0.0.1", port 5432
2018-07-23 14:49:42.492 JST [15864] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
2018-07-23 14:49:42.521 JST [15866] LOG: database system shutdown was interrupted; last known up at 2018-07-23 14:49:08 JST
2018-07-23 14:49:42.674 JST [15866] LOG: database system was not properly shut down; automatic recovery in progress
2018-07-23 14:49:42.676 JST [15866] LOG: redo starts at 0/167EB60
2018-07-23 14:49:42.676 JST [15866] LOG: invalid record length at 0/167EC70: wanted 24, got 0
2018-07-23 14:49:42.676 JST [15866] LOG: redo done at 0/167EC30
2018-07-23 14:49:42.677 JST [15866] FATAL: could not read two-phase state from WAL at 0/167EBA0
2018-07-23 14:49:42.678 JST [15864] LOG: startup process (PID 15866) exited with exit code 1
2018-07-23 14:49:42.678 JST [15864] LOG: aborting startup due to startup process failure
2018-07-23 14:49:42.682 JST [15864] LOG: database system is shut down
Regards.
----
Harada Toshi.
NTT TechnoCross Corporation
Antonin Houska <ah@cybertec.at> wrote:
> Ants Aasma <ants.aasma@gmail.com> wrote:
>
> > Attached to this mail is a work in progress patch that adds an
> > extensible encryption mechanism. There are some loose ends left to tie
> > up, but the general concept and architecture is at a point where it's
> > ready for some feedback, fresh ideas and bikeshedding.
>
> Rebased patch is attached here, in case it helps to achieve (some of) the
> goals mentioned in the related thread [1].
>
> Besides encrypting table and WAL pages, it encrypts the temporary files
> (buffile.c), data stored during logical decoding (reorderbuffer.c) and
> statistics temporary files (pgstat.c). Unlike the previous version, SLRU files
> (e.g. CLOG) are not encrypted (it does not seem critical and the encryption
> makes torn page write quite difficult to handle).
>
> Another difference is that we use the OpenSSL of the (tweaked) AES XTS cipher
> now.
>
> Binary upgrade from unencrypted to encrypted cluster is not implemented yet.
>
>
> [1] https://www.postgresql.org/message-id/031401d3f41d$5c70ed90$1552c8b0$@lab.ntt.co.jp
>
> --
> Antonin Houska
> Cybertec Schonig & Schonig GmbH
> Grohrmuhlgasse 26, A-2700 Wiener Neustadt
> Web: https://www.cybertec-postgresql.com
>
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: "WIP: Data at rest encryption" patch and, 2 phase commit.
@ 2018-07-25 11:54 Antonin Houska <ah@cybertec.at>
1 sibling, 1 reply; 3+ messages in thread
From: Antonin Houska @ 2018-07-25 11:54 UTC (permalink / raw)
To: Toshi Harada <harada.toshi@po.ntt-tx.co.jp>; +Cc: pgsql-hackers
Toshi Harada <harada.toshi@po.ntt-tx.co.jp> wrote:
> Hi.
>
> I applied the patch "WIP: Data at rest encryption" to PostgreSQL 11 - beta 2 and I'm working on it.
>
> When this patch is applied, the following problem occurs.
>
> * An error occurs when CHECKPOINT is executed during two-phase commit.
> * After an error occurs, if you stop PostgreSQL, it will never start again.
>
> (1) First, execute PREPARE TRANSACTION.
>
> postgres=# BEGIN;
> BEGIN
> postgres=# PREPARE TRANSACTION 'foo';
> PREPARE TRANSACTION
> postgres=#
>
> (2) Execute the CHECKPOINT command from another terminal.
> CHEKPOINT command fails.
>
> postgres=# CHECKPOINT;
> ERROR: checkpoint request failed
> HINT: Consult recent messages in the server log for details.
> postgres=#
The patch version I posted in
https://www.postgresql.org/message-id/11678.1532519255%40localhost
fixes an issue (unitialized pointer) that caused failure here, but it was
SEGFAULT rather than ERROR. And the scope of the bug was broader than just
CHECKPOINT.
Can you please test it again with the new version of the patch?
Anyway, thanks for your reports!
--
Antonin Houska
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26, A-2700 Wiener Neustadt
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: "WIP: Data at rest encryption" patch and, 2 phase commit.
@ 2018-07-27 07:08 Toshi Harada <harada.toshi@po.ntt-tx.co.jp>
parent: Antonin Houska <ah@cybertec.at>
0 siblings, 0 replies; 3+ messages in thread
From: Toshi Harada @ 2018-07-27 07:08 UTC (permalink / raw)
To: Antonin Houska <ah@cybertec.at>; +Cc: pgsql-hackers
Hi.
Applying https://www.postgresql.org/message-id/11678.1532519255%40localhost patch,
the problem of pg_create_logical_replication_slot () and the 2PC problem were solved.
Thanks.
Antonin Houska <ah@cybertec.at> wrote:
> Toshi Harada <harada.toshi@po.ntt-tx.co.jp> wrote:
>
> > Hi.
> >
> > I applied the patch "WIP: Data at rest encryption" to PostgreSQL 11 - beta 2 and I'm working on it.
> >
> > When this patch is applied, the following problem occurs.
> >
> > * An error occurs when CHECKPOINT is executed during two-phase commit.
> > * After an error occurs, if you stop PostgreSQL, it will never start again.
> >
> > (1) First, execute PREPARE TRANSACTION.
> >
> > postgres=# BEGIN;
> > BEGIN
> > postgres=# PREPARE TRANSACTION 'foo';
> > PREPARE TRANSACTION
> > postgres=#
> >
> > (2) Execute the CHECKPOINT command from another terminal.
> > CHEKPOINT command fails.
> >
> > postgres=# CHECKPOINT;
> > ERROR: checkpoint request failed
> > HINT: Consult recent messages in the server log for details.
> > postgres=#
>
> The patch version I posted in
>
> https://www.postgresql.org/message-id/11678.1532519255%40localhost
>
> fixes an issue (unitialized pointer) that caused failure here, but it was
> SEGFAULT rather than ERROR. And the scope of the bug was broader than just
> CHECKPOINT.
>
> Can you please test it again with the new version of the patch?
>
> Anyway, thanks for your reports!
>
>
> --
> Antonin Houska
> Cybertec Schonig & Schonig GmbH
> Grohrmuhlgasse 26, A-2700 Wiener Neustadt
> Web: https://www.cybertec-postgresql.com
>
^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2018-07-27 07:08 UTC | newest]
Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2018-07-24 22:51 ` Toshi Harada <harada.toshi@po.ntt-tx.co.jp>
2018-07-25 11:54 ` Antonin Houska <ah@cybertec.at>
2018-07-27 07:08 ` Toshi Harada <harada.toshi@po.ntt-tx.co.jp>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox