pg.ddx.io pgsql-general@postgresql.org mailing list archive
help / color / mirror / Atom feedinvisible commit question for sync replication
6+ messages / 4 participants
[nested] [flat]
* invisible commit question for sync replication
@ 2023-02-01 06:52 qihua wu <staywithpin@gmail.com>
0 siblings, 2 replies; 6+ messages in thread
From: qihua wu @ 2023-02-01 06:52 UTC (permalink / raw)
To: pgsql-general@lists.postgresql.org
When run a cluster with sync replication, if DML is done on primary, but
primary is isolated from all slave, then the DML will hang, if cancel it
DML, it will say:
WARNING: canceling wait for synchronous replication due to user request
DETAIL: The transaction has already committed locally, but might not have
been replicated to the standby
So the workflow is
1: commit to local.
2: waiting for ACK from remote sync.
When cancel the DML at step 2. the data are arealy on local, that's why
it's warning.
But when runs an insert which is waiting for remote ACK, and then query
from another session, I didn't find that row. Why this happen? If the
insert is already one locally, whey another session can't read it?
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: invisible commit question for sync replication
@ 2023-02-01 07:11 Julien Rouhaud <rjuju123@gmail.com>
parent: qihua wu <staywithpin@gmail.com>
1 sibling, 1 reply; 6+ messages in thread
From: Julien Rouhaud @ 2023-02-01 07:11 UTC (permalink / raw)
To: qihua wu <staywithpin@gmail.com>; +Cc: pgsql-general@lists.postgresql.org
Hi,
On Wed, Feb 01, 2023 at 02:52:49PM +0800, qihua wu wrote:
> When run a cluster with sync replication, if DML is done on primary, but
> primary is isolated from all slave, then the DML will hang, if cancel it
> DML, it will say:
> WARNING: canceling wait for synchronous replication due to user request
> DETAIL: The transaction has already committed locally, but might not have
> been replicated to the standby
>
> So the workflow is
> 1: commit to local.
> 2: waiting for ACK from remote sync.
>
> When cancel the DML at step 2. the data are arealy on local, that's why
> it's warning.
>
> But when runs an insert which is waiting for remote ACK, and then query
> from another session, I didn't find that row. Why this happen? If the
> insert is already one locally, whey another session can't read it?
It works as expected for me, are you sure both sessions are actually connected
to the same server and/or querying the same table?
[1456]rjuju@127.0.0.1:14295) rjuju=# select * from tt;
id | val
----+-----
(0 rows)
[1456]rjuju@127.0.0.1:14295) rjuju=# insert into tt select 1;
^CCancel request sent
WARNING: 01000: canceling wait for synchronous replication due to user request
DETAIL: The transaction has already committed locally, but might not have been replicated to the standby.
LOCATION: SyncRepWaitForLSN, syncrep.c:287
INSERT 0 1
[1456]rjuju@127.0.0.1:14295) rjuju=# select pg_backend_pid(), * from tt;
pg_backend_pid | id | val
----------------+----+--------
1456 | 1 | <NULL>
(1 row)
and another session:
[3327]rjuju@127.0.0.1:14295) rjuju=# select pg_backend_pid(), * from tt;
pg_backend_pid | id | val
----------------+----+--------
3327 | 1 | <NULL>
(1 row)
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: invisible commit question for sync replication
@ 2023-02-01 07:21 David G. Johnston <david.g.johnston@gmail.com>
parent: Julien Rouhaud <rjuju123@gmail.com>
0 siblings, 1 reply; 6+ messages in thread
From: David G. Johnston @ 2023-02-01 07:21 UTC (permalink / raw)
To: Julien Rouhaud <rjuju123@gmail.com>; +Cc: qihua wu <staywithpin@gmail.com>; pgsql-general@lists.postgresql.org <pgsql-general@lists.postgresql.org>
On Wednesday, February 1, 2023, Julien Rouhaud <rjuju123@gmail.com> wrote:
> Hi,
>
> On Wed, Feb 01, 2023 at 02:52:49PM +0800, qihua wu wrote:
> > When run a cluster with sync replication, if DML is done on primary, but
> > primary is isolated from all slave, then the DML will hang, if cancel it
> > DML, it will say:
> > WARNING: canceling wait for synchronous replication due to user request
> > DETAIL: The transaction has already committed locally, but might not
> have
> > been replicated to the standby
> >
> > So the workflow is
> > 1: commit to local.
> > 2: waiting for ACK from remote sync.
> >
> > When cancel the DML at step 2. the data are arealy on local, that's why
> > it's warning.
> >
> > But when runs an insert which is waiting for remote ACK, and then query
> > from another session, I didn't find that row. Why this happen? If the
> > insert is already one locally, whey another session can't read it?
>
> It works as expected for me, are you sure both sessions are actually
> connected
> to the same server and/or querying the same table?
>
> [1456]rjuju@127.0.0.1:14295) rjuju=# select * from tt;
> id | val
> ----+-----
> (0 rows)
>
> [1456]rjuju@127.0.0.1:14295) rjuju=# insert into tt select 1;
> ^CCancel request sent
> WARNING: 01000: canceling wait for synchronous replication due to user
> request
> DETAIL: The transaction has already committed locally, but might not have
> been replicated to the standby.
> LOCATION: SyncRepWaitForLSN, syncrep.c:287
> INSERT 0 1
>
> [1456]rjuju@127.0.0.1:14295) rjuju=# select pg_backend_pid(), * from tt;
> pg_backend_pid | id | val
> ----------------+----+--------
> 1456 | 1 | <NULL>
> (1 row)
>
>
> and another session:
>
> [3327]rjuju@127.0.0.1:14295) rjuju=# select pg_backend_pid(), * from tt;
> pg_backend_pid | id | val
> ----------------+----+--------
> 3327 | 1 | <NULL>
> (1 row)
>
>
>
This wasn’t the question though. Can the second session see the inserted
row before you cancel the insert that is waiting for sync ack?
Supposedly it can (not able to test myself). Basically, the primary waits
to make the local transaction visible until either sync ack or until the
wait for sync ack is cancelled. It doesn’t make sense to make it visible
while waiting for sync ack since that would defeat the very behavior sync
ack provides for.
David J.
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: invisible commit question for sync replication
@ 2023-02-01 08:05 qihua wu <staywithpin@gmail.com>
parent: David G. Johnston <david.g.johnston@gmail.com>
0 siblings, 0 replies; 6+ messages in thread
From: qihua wu @ 2023-02-01 08:05 UTC (permalink / raw)
To: David G. Johnston <david.g.johnston@gmail.com>; +Cc: Julien Rouhaud <rjuju123@gmail.com>; pgsql-general@lists.postgresql.org <pgsql-general@lists.postgresql.org>
==》Can the second session see the inserted row before you cancel the insert
that is waiting for sync ack?
The second session can NOT see the inserted row if the first session is
still waiting for sync ACK.
I checked the source code, it makes sense to me now:
The waiting for sync ACK is called in
EndPrepare(gxact);
and after that it will call
ProcArrayClearTransaction(MyProc);
and only after ProcArrayClearTransaction(MyProc) is called, will the row be
visible to others.
On Wed, Feb 1, 2023 at 3:21 PM David G. Johnston <david.g.johnston@gmail.com>
wrote:
> On Wednesday, February 1, 2023, Julien Rouhaud <rjuju123@gmail.com> wrote:
>
>> Hi,
>>
>> On Wed, Feb 01, 2023 at 02:52:49PM +0800, qihua wu wrote:
>> > When run a cluster with sync replication, if DML is done on primary, but
>> > primary is isolated from all slave, then the DML will hang, if cancel it
>> > DML, it will say:
>> > WARNING: canceling wait for synchronous replication due to user request
>> > DETAIL: The transaction has already committed locally, but might not
>> have
>> > been replicated to the standby
>> >
>> > So the workflow is
>> > 1: commit to local.
>> > 2: waiting for ACK from remote sync.
>> >
>> > When cancel the DML at step 2. the data are arealy on local, that's why
>> > it's warning.
>> >
>> > But when runs an insert which is waiting for remote ACK, and then query
>> > from another session, I didn't find that row. Why this happen? If the
>> > insert is already one locally, whey another session can't read it?
>>
>> It works as expected for me, are you sure both sessions are actually
>> connected
>> to the same server and/or querying the same table?
>>
>> [1456]rjuju@127.0.0.1:14295) rjuju=# select * from tt;
>> id | val
>> ----+-----
>> (0 rows)
>>
>> [1456]rjuju@127.0.0.1:14295) rjuju=# insert into tt select 1;
>> ^CCancel request sent
>> WARNING: 01000: canceling wait for synchronous replication due to user
>> request
>> DETAIL: The transaction has already committed locally, but might not
>> have been replicated to the standby.
>> LOCATION: SyncRepWaitForLSN, syncrep.c:287
>> INSERT 0 1
>>
>> [1456]rjuju@127.0.0.1:14295) rjuju=# select pg_backend_pid(), * from tt;
>> pg_backend_pid | id | val
>> ----------------+----+--------
>> 1456 | 1 | <NULL>
>> (1 row)
>>
>>
>> and another session:
>>
>> [3327]rjuju@127.0.0.1:14295) rjuju=# select pg_backend_pid(), * from tt;
>> pg_backend_pid | id | val
>> ----------------+----+--------
>> 3327 | 1 | <NULL>
>> (1 row)
>>
>>
>>
> This wasn’t the question though. Can the second session see the inserted
> row before you cancel the insert that is waiting for sync ack?
>
> Supposedly it can (not able to test myself). Basically, the primary waits
> to make the local transaction visible until either sync ack or until the
> wait for sync ack is cancelled. It doesn’t make sense to make it visible
> while waiting for sync ack since that would defeat the very behavior sync
> ack provides for.
>
> David J.
>
>
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: invisible commit question for sync replication
@ 2023-02-01 08:38 Laurenz Albe <laurenz.albe@cybertec.at>
parent: qihua wu <staywithpin@gmail.com>
1 sibling, 1 reply; 6+ messages in thread
From: Laurenz Albe @ 2023-02-01 08:38 UTC (permalink / raw)
To: qihua wu <staywithpin@gmail.com>; pgsql-general@lists.postgresql.org
On Wed, 2023-02-01 at 14:52 +0800, qihua wu wrote:
> When run a cluster with sync replication, if DML is done on primary, but primary is
> isolated from all slave, then the DML will hang, if cancel it DML, it will say:
> WARNING: canceling wait for synchronous replication due to user request
> DETAIL: The transaction has already committed locally, but might not have been replicated to the standby
>
> So the workflow is
> 1: commit to local.
> 2: waiting for ACK from remote sync.
>
> When cancel the DML at step 2. the data are arealy on local, that's why it's warning.
>
> But when runs an insert which is waiting for remote ACK, and then query from another
> session, I didn't find that row. Why this happen? If the insert is already one locally,
> whey another session can't read it?
COMMIT is not as atomic as it appears. When the backend is waiting for the standby,
it has already committed the transaction on disk, but that fact is not advertised to
the other backends yet.
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: invisible commit question for sync replication
@ 2023-02-01 09:25 qihua wu <staywithpin@gmail.com>
parent: Laurenz Albe <laurenz.albe@cybertec.at>
0 siblings, 0 replies; 6+ messages in thread
From: qihua wu @ 2023-02-01 09:25 UTC (permalink / raw)
To: Laurenz Albe <laurenz.albe@cybertec.at>; +Cc: pgsql-general@lists.postgresql.org
In the code it will write a warning to postgresql log. Why not also write
the detailed sql? with the exact sql, DBA might do something to fix the
issue.
if (ProcDiePending)
{
ereport(WARNING,
(errcode(ERRCODE_ADMIN_SHUTDOWN),
errmsg("canceling the wait for synchronous replication and terminating
connection due to administrator command"),
errdetail("The transaction has already committed locally, but might not
have been replicated to the standby.")));
whereToSendOutput = DestNone;
SyncRepCancelWait();
break;
}
On Wed, Feb 1, 2023 at 4:38 PM Laurenz Albe <laurenz.albe@cybertec.at>
wrote:
> On Wed, 2023-02-01 at 14:52 +0800, qihua wu wrote:
> > When run a cluster with sync replication, if DML is done on primary, but
> primary is
> > isolated from all slave, then the DML will hang, if cancel it DML, it
> will say:
> > WARNING: canceling wait for synchronous replication due to user request
> > DETAIL: The transaction has already committed locally, but might not
> have been replicated to the standby
> >
> > So the workflow is
> > 1: commit to local.
> > 2: waiting for ACK from remote sync.
> >
> > When cancel the DML at step 2. the data are arealy on local, that's why
> it's warning.
> >
> > But when runs an insert which is waiting for remote ACK, and then query
> from another
> > session, I didn't find that row. Why this happen? If the insert is
> already one locally,
> > whey another session can't read it?
>
> COMMIT is not as atomic as it appears. When the backend is waiting for
> the standby,
> it has already committed the transaction on disk, but that fact is not
> advertised to
> the other backends yet.
>
> Yours,
> Laurenz Albe
>
^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2023-02-01 09:25 UTC | newest]
Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2023-02-01 06:52 invisible commit question for sync replication qihua wu <staywithpin@gmail.com>
2023-02-01 07:11 ` Julien Rouhaud <rjuju123@gmail.com>
2023-02-01 07:21 ` David G. Johnston <david.g.johnston@gmail.com>
2023-02-01 08:05 ` qihua wu <staywithpin@gmail.com>
2023-02-01 08:38 ` Laurenz Albe <laurenz.albe@cybertec.at>
2023-02-01 09:25 ` qihua wu <staywithpin@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox