pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedBUG #17695: Failed Assert in logical replication snapbuild.
17+ messages / 8 participants
[nested] [flat]
* BUG #17695: Failed Assert in logical replication snapbuild.
@ 2022-11-24 10:15 PG Bug reporting form <noreply@postgresql.org>
0 siblings, 3 replies; 17+ messages in thread
From: PG Bug reporting form @ 2022-11-24 10:15 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org; +Cc: zxwsbg@qq.com
The following bug has been logged on the website:
Bug reference: 17695
Logged by: 施博文
Email address: zxwsbg@qq.com
PostgreSQL version: 14.6
Operating system: centos
Description:
In PG14 or higher version, I notice that SnapBuildRestore don't set
builder->next_phase_at=InvalidTransactionId .
But in SnapBuildSerialize function, the assert check this condition.
Assert(builder->next_phase_at == InvalidTransactionId);
This would cause some problems, and I have repeat it with the perl test case
which I would update later, the problem is following:
TRAP: FailedAssertion("builder->next_phase_at == InvalidTransactionId",
File: "snapbuild.c", Line: 1604, PID: 29974)
postgres: master: walsender postgres [local]
START_REPLICATION(ExceptionalCondition+0xb9)[0xb1c9bd]
postgres: master: walsender postgres [local] START_REPLICATION[0x8f548d]
postgres: master: walsender postgres [local]
START_REPLICATION(SnapBuildProcessRunningXacts+0x55)[0x8f4c5c]
postgres: master: walsender postgres [local] START_REPLICATION[0x8dd8be]
postgres: master: walsender postgres [local]
START_REPLICATION(LogicalDecodingProcessRecord+0xd1)[0x8dd243]
postgres: master: walsender postgres [local] START_REPLICATION[0x915eb1]
postgres: master: walsender postgres [local] START_REPLICATION[0x91520c]
postgres: master: walsender postgres [local] START_REPLICATION[0x913bdd]
postgres: master: walsender postgres [local]
START_REPLICATION(exec_replication_command+0x42c)[0x914593]
postgres: master: walsender postgres [local]
START_REPLICATION(PostgresMain+0x7be)[0x984ddf]
postgres: master: walsender postgres [local] START_REPLICATION[0x8c0d41]
postgres: master: walsender postgres [local] START_REPLICATION[0x8c06b3]
postgres: master: walsender postgres [local] START_REPLICATION[0x8bc9c4]
postgres: master: walsender postgres [local]
START_REPLICATION(PostmasterMain+0x117a)[0x8bc29b]
postgres: master: walsender postgres [local] START_REPLICATION[0x7bdaf9]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f815dd92555]
postgres: master: walsender postgres [local] START_REPLICATION[0x488d09]
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: BUG #17695: Failed Assert in logical replication snapbuild.
@ 2022-11-24 12:39 vignesh C <vignesh21@gmail.com>
parent: PG Bug reporting form <noreply@postgresql.org>
2 siblings, 0 replies; 17+ messages in thread
From: vignesh C @ 2022-11-24 12:39 UTC (permalink / raw)
To: 施博文 <zxwsbg@qq.com>; +Cc: pgsql-bugs <pgsql-bugs@lists.postgresql.org>
On Thu, 24 Nov 2022 at 15:58, 施博文 <zxwsbg@qq.com> wrote:
>
> I write this perl test case to reproduce problem, it needs both superuser and database name 'postgres'.
>
> Also, I write the patch to fix that problem.
Thanks for sharing the script. I was able to reproduce the problem in
PG 14.6 with your scripts. I was not able to reproduce the problem in
HEAD after making the required changes in the script for HEAD. I was
not sure if this issue is only in 14.6 or if the issue is present in
HEAD too. Since the Assert is there and code changes suggested by you
is not in HEAD, I thought this issue might be present in HEAD too.
Does it work for you on HEAD?
Regards,
Vignesh
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: BUG #17695: Failed Assert in logical replication snapbuild.
@ 2022-11-25 05:58 Masahiko Sawada <sawada.mshk@gmail.com>
parent: PG Bug reporting form <noreply@postgresql.org>
2 siblings, 0 replies; 17+ messages in thread
From: Masahiko Sawada @ 2022-11-25 05:58 UTC (permalink / raw)
To: zxwsbg@qq.com; pgsql-bugs@lists.postgresql.org
On Thu, Nov 24, 2022 at 7:28 PM PG Bug reporting form
<noreply@postgresql.org> wrote:
>
> The following bug has been logged on the website:
>
> Bug reference: 17695
> Logged by: 施博文
> Email address: zxwsbg@qq.com
> PostgreSQL version: 14.6
> Operating system: centos
> Description:
>
> In PG14 or higher version, I notice that SnapBuildRestore don't set
> builder->next_phase_at=InvalidTransactionId .
>
> But in SnapBuildSerialize function, the assert check this condition.
>
> Assert(builder->next_phase_at == InvalidTransactionId);
>
> This would cause some problems, and I have repeat it with the perl test case
> which I would update later, the problem is following:
>
> TRAP: FailedAssertion("builder->next_phase_at == InvalidTransactionId",
> File: "snapbuild.c", Line: 1604, PID: 29974)
> postgres: master: walsender postgres [local]
> START_REPLICATION(ExceptionalCondition+0xb9)[0xb1c9bd]
> postgres: master: walsender postgres [local] START_REPLICATION[0x8f548d]
> postgres: master: walsender postgres [local]
> START_REPLICATION(SnapBuildProcessRunningXacts+0x55)[0x8f4c5c]
> postgres: master: walsender postgres [local] START_REPLICATION[0x8dd8be]
> postgres: master: walsender postgres [local]
> START_REPLICATION(LogicalDecodingProcessRecord+0xd1)[0x8dd243]
> postgres: master: walsender postgres [local] START_REPLICATION[0x915eb1]
> postgres: master: walsender postgres [local] START_REPLICATION[0x91520c]
> postgres: master: walsender postgres [local] START_REPLICATION[0x913bdd]
> postgres: master: walsender postgres [local]
> START_REPLICATION(exec_replication_command+0x42c)[0x914593]
> postgres: master: walsender postgres [local]
> START_REPLICATION(PostgresMain+0x7be)[0x984ddf]
> postgres: master: walsender postgres [local] START_REPLICATION[0x8c0d41]
> postgres: master: walsender postgres [local] START_REPLICATION[0x8c06b3]
> postgres: master: walsender postgres [local] START_REPLICATION[0x8bc9c4]
> postgres: master: walsender postgres [local]
> START_REPLICATION(PostmasterMain+0x117a)[0x8bc29b]
> postgres: master: walsender postgres [local] START_REPLICATION[0x7bdaf9]
> /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f815dd92555]
> postgres: master: walsender postgres [local] START_REPLICATION[0x488d09]
>
Thank you for reporting the issue!
I could reproduce this issue with a small change and the following steps:
1. Add sleep after setting SNAPBUILD_BUILDING_SNAPSHOT.
@@ -1409,6 +1409,9 @@ SnapBuildFindSnapshot(SnapBuild *builder,
XLogRecPtr lsn, xl_running_xacts *runn
running->xcnt, running->nextXid)));
SnapBuildWaitSnapshot(running, running->nextXid);
+
+ elog(LOG, "sleep 10s");
+ pg_usleep(10 * 1000000);
}
2. Create and start database instance and create a table and the
replication slot 's1'.
create table test (i int);
select pg_logical_replication_slot('s1', 'test_decoding');
3. Create another replication slot "s2" while a transaction is running.
tx-1:
begin;
insert into test values (1);
tx-2:
select pg_logical_replication_slot('s2', 'test_decoding');
checkpoint;
commit;
Note that replication slot creation will wait for 10 sec after
commiting the tx-1, please go to the next step once the slot has been
created.
4. Start decoding on slot 's2'.
select pg_logical_slot_get_changes('s2', null, null);
The logical decoding will sleep. Please go to the next step after
seeing the log "sleep 10s".
5. While the logical decoding started at step 4 is sleeping, start
decoding on slot 's1'.
select pg_logical_slot_get_changes('s1', null, null);
6. After the decoding on slot 's1' wakes up, it fails due to the
assertion check.
This scenario simulates the case where the logical decoding (re)starts
where RUNNING_XACTS record having a running transactions and it
restores the serialized snapshot when decoding the next RUNNING_XACTS
record. Since we don't reset builder->next_phase_at when restoring a
serialized snapshot, the assertion check in SnapBuildSerialize fails.
Regarding the proposed patch, I've confirmed it fixes this issue. But
I think it's better to reset builder->next_phase_at right after the
following assertion check:
/* consistent snapshots have no next phase */
Assert(ondisk.builder.next_phase_at == InvalidTransactionId);
I could not reproduce this issue with your script in my environment. I
think it's better to include the reproducible test case in the patch
but I'm not sure how to do that without adding sleep/gdb attach.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: BUG #17695: Failed Assert in logical replication snapbuild.
@ 2022-11-25 07:44 bowenshi <zxwsbg@qq.com>
parent: PG Bug reporting form <noreply@postgresql.org>
2 siblings, 3 replies; 17+ messages in thread
From: bowenshi @ 2022-11-25 07:44 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
--=_mailmaster-63807243_4486b095_1174d_=
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: base64
PGh0bWw+DQo8aGVhZD4NCiAgICA8bWV0YSBodHRwLWVxdWl2PSdDb250ZW50LVR5cGUnIGNvbnRl
bnQ9J3RleHQvaHRtbDsgY2hhcnNldD1VVEYtOCc+DQo8L2hlYWQ+DQo8Ym9keT4NCjxzdHlsZT4N
CiAgICBmb250ew0KICAgICAgICBsaW5lLWhlaWdodDogMS42Ow0KICAgIH0NCiAgICB1bCxvbHsN
CiAgICAgICAgcGFkZGluZy1sZWZ0OiAyMHB4Ow0KICAgICAgICBsaXN0LXN0eWxlLXBvc2l0aW9u
OiBpbnNpZGU7DQogICAgfQ0KPC9zdHlsZT4NCjxkaXYgc3R5bGUgPSAnZm9udC1mYW1pbHk6SGVs
dmV0aWNhLEhlbHZldGljYSzlvq7ova/pm4Xpu5EsIOWui+S9kzsgbGluZS1oZWlnaHQ6MS42Oyc+
DQogICAgPGRpdiA+PC9kaXY+PGRpdj4KICAgIDxkaXY+VGhhbmtzIGZvciB5b3VyIGFkdmljZSEg
SSB1cGRhdGUgbXkgdGVzdCBzY3JpcHQsIG5vdyBpdCBjYW4gMTAwJSByZXByb2R1Y2UgcHJvYmxl
bSBpbiBteSBjb21wdXRlci4mbmJzcDs8YnI+PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5UaGUg
bmV3IHNjcmlwdCBhbmQgZml4IGNvZGUgYXJlIGJvdGggaW4gdGhlIHBhdGNoLiBIb3dldmVyLCBJ
IG1ldCBuZXcgcHJvYmxlbSBhZnRlciBhZGRpbmcgZml4IGNvZGUuIEl0IGZhaWxzIGludG8gYSBu
ZXcgdHJhcCBmb2xsb3dpbmc6PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5UUkFQOiBGYWlsZWRB
c3NlcnRpb24oIlRyYW5zYWN0aW9uSWRQcmVjZWRlc09yRXF1YWxzKHNhZmVYaWQsIHNuYXAtJmd0
O3htaW4pIiwgRmlsZTogInNuYXBidWlsZC5jIiwgTGluZTogNTkzLCBQSUQ6IDE1NzYpPGJyPjwv
ZGl2PnBvc3RncmVzOiBtYXN0ZXI6IHdhbHNlbmRlciBwb3N0Z3JlcyBbbG9jYWxdIENSRUFURV9S
RVBMSUNBVElPTl9TTE9UKEV4Y2VwdGlvbmFsQ29uZGl0aW9uKzB4YjkpWzB4YjFjYTI4XTxicj5w
b3N0Z3JlczogbWFzdGVyOiB3YWxzZW5kZXIgcG9zdGdyZXMgW2xvY2FsXSBDUkVBVEVfUkVQTElD
QVRJT05fU0xPVChTbmFwQnVpbGRJbml0aWFsU25hcHNob3QrMHgxYjMpWzB4OGYzYzc5XTxicj5w
b3N0Z3JlczogbWFzdGVyOiB3YWxzZW5kZXIgcG9zdGdyZXMgW2xvY2FsXSBDUkVBVEVfUkVQTElD
QVRJT05fU0xPVFsweDkxMzZiOV08YnI+cG9zdGdyZXM6IG1hc3Rlcjogd2Fsc2VuZGVyIHBvc3Rn
cmVzIFtsb2NhbF0gQ1JFQVRFX1JFUExJQ0FUSU9OX1NMT1QoZXhlY19yZXBsaWNhdGlvbl9jb21t
YW5kKzB4Mzk4KVsweDkxNDU2YV08YnI+cG9zdGdyZXM6IG1hc3Rlcjogd2Fsc2VuZGVyIHBvc3Rn
cmVzIFtsb2NhbF0gQ1JFQVRFX1JFUExJQ0FUSU9OX1NMT1QoUG9zdGdyZXNNYWluKzB4N2JlKVsw
eDk4NGU0YV08YnI+cG9zdGdyZXM6IG1hc3Rlcjogd2Fsc2VuZGVyIHBvc3RncmVzIFtsb2NhbF0g
Q1JFQVRFX1JFUExJQ0FUSU9OX1NMT1RbMHg4YzBkNWJdPGJyPnBvc3RncmVzOiBtYXN0ZXI6IHdh
bHNlbmRlciBwb3N0Z3JlcyBbbG9jYWxdIENSRUFURV9SRVBMSUNBVElPTl9TTE9UWzB4OGMwNmNk
XTxicj5wb3N0Z3JlczogbWFzdGVyOiB3YWxzZW5kZXIgcG9zdGdyZXMgW2xvY2FsXSBDUkVBVEVf
UkVQTElDQVRJT05fU0xPVFsweDhiYzlkZV08YnI+cG9zdGdyZXM6IG1hc3Rlcjogd2Fsc2VuZGVy
IHBvc3RncmVzIFtsb2NhbF0gQ1JFQVRFX1JFUExJQ0FUSU9OX1NMT1QoUG9zdG1hc3Rlck1haW4r
MHgxMTdhKVsweDhiYzJiNV08YnI+cG9zdGdyZXM6IG1hc3Rlcjogd2Fsc2VuZGVyIHBvc3RncmVz
IFtsb2NhbF0gQ1JFQVRFX1JFUExJQ0FUSU9OX1NMT1RbMHg3YmRiMTNdPGRpdj48YnI+PC9kaXY+
PGRpdj4KICAgIDwvZGl2PgogICAgPGRpdiBpZD0ibnRlcy1wY21hYy1zaWduYXR1cmUiIHN0eWxl
PSJmb250LWZhbWlseToiPgogICAgCiAgICA8ZGl2IHN0eWxlPSJmb250LXNpemU6MTRweDsgcGFk
ZGluZzogMDsgIG1hcmdpbjowOyI+CgogICAgPC9kaXY+CjwvZGl2PgogICAgVGhlIHByb2JsZW0g
aXMgbm90IHZlcnkgaGFyZCB0byByZXByb2R1Y2UsIGFuZCBJ4oCZbSBmaXhpbmcgaXQgbm93Ljwv
ZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+UmVnYXJkcyw8L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2
Pi0tPC9kaXY+PGRpdj48ZGl2PjxzcGFuIHN0eWxlPSJjb2xvcjogcmdiKDQ5LCA1MywgNTkpOyBm
b250LXNpemU6IDE0cHg7Ij5ib3dlbnNoaTwvc3Bhbj48L2Rpdj48L2Rpdj48ZGl2PjxzcGFuIHN0
eWxlPSJjb2xvcjogcmdiKDEyMCwgMTIwLCAxMjIpOyI+PGJyPjwvc3Bhbj48L2Rpdj48ZGl2Pjxz
cGFuIHN0eWxlPSJjb2xvcjogcmdiKDEyMCwgMTIwLCAxMjIpOyI+LS0tLSBSZXBsaWVkIE1lc3Nh
Z2UgLS0tLTwvc3Bhbj48L2Rpdj48ZGl2IGNsYXNzPSJKLXJlcGx5IG50ZXMtbWFpbG1hc3Rlci1x
dW90ZSIgc3R5bGU9InBhZGRpbmctdG9wOiAxcHg7IHBhZGRpbmctYm90dG9tOiAxcHgiPgogIDxk
aXYgc3R5bGU9IgogICAgICBtYXJnaW4tYm90dG9tOiAxZW07CiAgICAgIGZvbnQtc2l6ZTogMTJw
eDsKICAgICAgbGluZS1oZWlnaHQ6IDEuMjU7CiAgICAgIGNvbG9yOiAjMjMyMzI0OwogICAgICBw
YWRkaW5nOiAwLjVlbSAwLjI1ZW07CiAgICAgIGJvcmRlci1yYWRpdXM6IDAuNjdlbTsKICAgICAg
YmFja2dyb3VuZC1jb2xvcjogI2YwZjBmMDsKICAgICI+CiAgICA8dGFibGUgd2lkdGg9IjEwMCUi
IGNlbGxwYWRkaW5nPSIwIiBjZWxsc3BhY2luZz0iOSIgYm9yZGVyPSIwIj4KICAgICAgPHRib2R5
Pjx0ciBjbGFzcz0iZmlyc3RSb3ciPgogICAgICAgIDx0ZCB2YWxpZ249InRvcCIgc3R5bGU9Igog
ICAgICAgICAgICB3aWR0aDogNC4yNWVtOwogICAgICAgICAgICBmb250LXNpemU6IDEycHg7CiAg
ICAgICAgICAgIGxpbmUtaGVpZ2h0OiAxLjI1OwogICAgICAgICAgICBjb2xvcjogIzc4Nzg3YTsK
ICAgICAgICAgICI+CiAgICAgICAgIEZyb20gCiAgICAgICAgPC90ZD4KICAgICAgICA8dGQgdmFs
aWduPSJ0b3AiIHN0eWxlPSIKICAgICAgICAgICAgZm9udC1zaXplOiAxMnB4OwogICAgICAgICAg
ICBsaW5lLWhlaWdodDogMS4yNTsKICAgICAgICAgICAgY29sb3I6ICMyMzIzMjQ7CiAgICAgICAg
ICAgIHdvcmQtYnJlYWs6IGJyZWFrLWFsbDsKICAgICAgICAgICI+CiAgICAgICAgICAgIDxhIGNs
YXNzPSJtYWlsLWZyb20iIHN0eWxlPSJjb2xvcjogIzMwNjFmMjsgdGV4dC1kZWNvcmF0aW9uOiBu
b25lIiBocmVmPSJtYWlsdG86c2F3YWRhLm1zaGtAZ21haWwuY29tIj5NYXNhaGlrbyBTYXdhZGEm
bHQ7c2F3YWRhLm1zaGtAZ21haWwuY29tJmd0OwogICAgICAgICAgICA8L2E+CiAgICAgICAgPC90
ZD4KICAgICAgPC90cj4KICAgICAgPHRyPgogICAgICAgIDx0ZCB2YWxpZ249InRvcCIgc3R5bGU9
IgogICAgICAgICAgICB3aWR0aDogNC4yNWVtOwogICAgICAgICAgICBmb250LXNpemU6IDEycHg7
CiAgICAgICAgICAgIGxpbmUtaGVpZ2h0OiAxLjI1OwogICAgICAgICAgICBjb2xvcjogIzc4Nzg3
YTsKICAgICAgICAgICI+CiAgICAgICAgIERhdGUgCiAgICAgICAgPC90ZD4KICAgICAgICA8dGQg
Y2xhc3M9Im1haWwtZGF0ZSIgdmFsaWduPSJ0b3AiIHN0eWxlPSIKICAgICAgICAgICAgZm9udC1z
aXplOiAxMnB4OwogICAgICAgICAgICBsaW5lLWhlaWdodDogMS4yNTsKICAgICAgICAgICAgY29s
b3I6ICMyMzIzMjQ7CiAgICAgICAgICAgIHdvcmQtYnJlYWs6IGJyZWFrLWFsbDsKICAgICAgICAg
ICI+CiAgICAgICAgMTEvMjUvMjAyMiAxMzo1OAogICAgICAgIDwvdGQ+CiAgICAgIDwvdHI+CiAg
ICAgIDx0cj4KICAgICAgICA8dGQgdmFsaWduPSJ0b3AiIHN0eWxlPSIKICAgICAgICAgICAgd2lk
dGg6IDQuMjVlbTsKICAgICAgICAgICAgZm9udC1zaXplOiAxMnB4OwogICAgICAgICAgICBsaW5l
LWhlaWdodDogMS4yNTsKICAgICAgICAgICAgY29sb3I6ICM3ODc4N2E7CiAgICAgICAgICAiPgog
ICAgICAgICBUbyAKICAgICAgICA8L3RkPgogICAgICAgIDx0ZCB2YWxpZ249InRvcCIgc3R5bGU9
IgogICAgICAgICAgICBmb250LXNpemU6IDEycHg7CiAgICAgICAgICAgIGxpbmUtaGVpZ2h0OiAx
LjI1OwogICAgICAgICAgICBjb2xvcjogIzIzMjMyNDsKICAgICAgICAgICAgd29yZC1icmVhazog
YnJlYWstYWxsOwogICAgICAgICAgIj4KICAgICAgICAgCiAgICAgICAgICA8YSBjbGFzcz0ibWFp
bC10byIgc3R5bGU9ImNvbG9yOiAjMzA2MWYyOyB0ZXh0LWRlY29yYXRpb246IG5vbmUiIGhyZWY9
Im1haWx0bzp6eHdzYmdAcXEuY29tIj4KICAgICAgICAgICAgJmx0O3p4d3NiZ0BxcS5jb20mZ3Q7
CiAgICAgICAgICAgICw8YnI+CiAgICAgICAgICA8L2E+CiAgICAgICAgICA8YSBjbGFzcz0ibWFp
bC10byIgc3R5bGU9ImNvbG9yOiAjMzA2MWYyOyB0ZXh0LWRlY29yYXRpb246IG5vbmUiIGhyZWY9
Im1haWx0bzpwZ3NxbC1idWdzQGxpc3RzLnBvc3RncmVzcWwub3JnIj4KICAgICAgICAgICAgJmx0
O3Bnc3FsLWJ1Z3NAbGlzdHMucG9zdGdyZXNxbC5vcmcmZ3Q7CiAgICAgICAgICAgIAogICAgICAg
ICAgPC9hPgogICAgICAgIDwvdGQ+CiAgICAgIDwvdHI+CiAgICAgIDx0cj4KICAgICAgICA8dGQg
dmFsaWduPSJ0b3AiIHN0eWxlPSIKICAgICAgICAgICAgd2lkdGg6IDQuMjVlbTsKICAgICAgICAg
ICAgZm9udC1zaXplOiAxMnB4OwogICAgICAgICAgICBsaW5lLWhlaWdodDogMS4yNTsKICAgICAg
ICAgICAgY29sb3I6ICM3ODc4N2E7CiAgICAgICAgICAiPgogICAgICAgICBTdWJqZWN0IAogICAg
ICAgIDwvdGQ+CiAgICAgICAgPHRkIGNsYXNzPSJtYWlsLXN1YmplY3QiIHZhbGlnbj0idG9wIiBz
dHlsZT0iCiAgICAgICAgICAgIGZvbnQtc2l6ZTogMTJweDsKICAgICAgICAgICAgbGluZS1oZWln
aHQ6IDEuMjU7CiAgICAgICAgICAgIGNvbG9yOiAjMjMyMzI0OwogICAgICAgICAgICB3b3JkLWJy
ZWFrOiBicmVhay1hbGw7CiAgICAgICAgICAiPgogICAgICAgICAgICAgIFJlOiBCVUcgIzE3Njk1
OiBGYWlsZWQgQXNzZXJ0IGluIGxvZ2ljYWwgcmVwbGljYXRpb24gc25hcGJ1aWxkLgogICAgICAg
IDwvdGQ+CiAgICAgIDwvdHI+CiAgICA8L3Rib2R5PjwvdGFibGU+CiAgPC9kaXY+CiAgPGRpdiBp
ZD0ibnRlcy1wY21haWwtcXVvdGUiIHN0eWxlPSJtYXJnaW46IDA7IHBhZGRpbmc6IDA7IGZvbnQt
c2l6ZTogMTRweDsgZm9udC1mYW1pbHk6ICcnSGVsdmV0aWNhJywnTWljcm9zb2Z0IFlhaGVpJywg
J+W+rui9r+mbhem7kScnOyI+T24gVGh1LCBOb3YgMjQsIDIwMjIgYXQgNzoyOCBQTSBQRyBCdWcg
cmVwb3J0aW5nIGZvcm08YnI+Jmx0O25vcmVwbHlAcG9zdGdyZXNxbC5vcmcmZ3Q7IHdyb3RlOjxi
cj48YmxvY2txdW90ZSBjbGFzcz0ibW1icWMxIj48YnI+IFRoZSBmb2xsb3dpbmcgYnVnIGhhcyBi
ZWVuIGxvZ2dlZCBvbiB0aGUgd2Vic2l0ZTo8YnI+PGJyPiBCdWcgcmVmZXJlbmNlOiAmbmJzcDsm
bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsxNzY5NTxicj4gTG9nZ2VkIGJ5OiAmbmJzcDsmbmJzcDsm
bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDvmlr3ljZrmloc8YnI+IEVt
YWlsIGFkZHJlc3M6ICZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwO3p4d3NiZ0BxcS5jb208
YnI+IFBvc3RncmVTUUwgdmVyc2lvbjogMTQuNjxicj4gT3BlcmF0aW5nIHN5c3RlbTogJm5ic3A7
Jm5ic3A7Y2VudG9zPGJyPiBEZXNjcmlwdGlvbjo8YnI+PGJyPiBJbiBQRzE0IG9yIGhpZ2hlciB2
ZXJzaW9uLCBJIG5vdGljZSB0aGF0IFNuYXBCdWlsZFJlc3RvcmUgZG9uJ3Qgc2V0PGJyPiBidWls
ZGVyLSZndDtuZXh0X3BoYXNlX2F0PUludmFsaWRUcmFuc2FjdGlvbklkIC48YnI+PGJyPiBCdXQg
aW4gU25hcEJ1aWxkU2VyaWFsaXplIGZ1bmN0aW9uLCB0aGUgYXNzZXJ0IGNoZWNrIHRoaXMgY29u
ZGl0aW9uLjxicj48YnI+IEFzc2VydChidWlsZGVyLSZndDtuZXh0X3BoYXNlX2F0ID09IEludmFs
aWRUcmFuc2FjdGlvbklkKTs8YnI+PGJyPiBUaGlzIHdvdWxkIGNhdXNlIHNvbWUgcHJvYmxlbXMs
IGFuZCBJIGhhdmUgcmVwZWF0IGl0IHdpdGggdGhlIHBlcmwgdGVzdCBjYXNlPGJyPiB3aGljaCBJ
IHdvdWxkIHVwZGF0ZSBsYXRlciwgdGhlIHByb2JsZW0gaXMgZm9sbG93aW5nOjxicj48YnI+IFRS
QVA6IEZhaWxlZEFzc2VydGlvbigiYnVpbGRlci0mZ3Q7bmV4dF9waGFzZV9hdCA9PSBJbnZhbGlk
VHJhbnNhY3Rpb25JZCIsPGJyPiBGaWxlOiAic25hcGJ1aWxkLmMiLCBMaW5lOiAxNjA0LCBQSUQ6
IDI5OTc0KTxicj4gcG9zdGdyZXM6IG1hc3Rlcjogd2Fsc2VuZGVyIHBvc3RncmVzIFtsb2NhbF08
YnI+IFNUQVJUX1JFUExJQ0FUSU9OKEV4Y2VwdGlvbmFsQ29uZGl0aW9uKzB4YjkpWzB4YjFjOWJk
XTxicj4gcG9zdGdyZXM6IG1hc3Rlcjogd2Fsc2VuZGVyIHBvc3RncmVzIFtsb2NhbF0gU1RBUlRf
UkVQTElDQVRJT05bMHg4ZjU0OGRdPGJyPiBwb3N0Z3JlczogbWFzdGVyOiB3YWxzZW5kZXIgcG9z
dGdyZXMgW2xvY2FsXTxicj4gU1RBUlRfUkVQTElDQVRJT04oU25hcEJ1aWxkUHJvY2Vzc1J1bm5p
bmdYYWN0cysweDU1KVsweDhmNGM1Y108YnI+IHBvc3RncmVzOiBtYXN0ZXI6IHdhbHNlbmRlciBw
b3N0Z3JlcyBbbG9jYWxdIFNUQVJUX1JFUExJQ0FUSU9OWzB4OGRkOGJlXTxicj4gcG9zdGdyZXM6
IG1hc3Rlcjogd2Fsc2VuZGVyIHBvc3RncmVzIFtsb2NhbF08YnI+IFNUQVJUX1JFUExJQ0FUSU9O
KExvZ2ljYWxEZWNvZGluZ1Byb2Nlc3NSZWNvcmQrMHhkMSlbMHg4ZGQyNDNdPGJyPiBwb3N0Z3Jl
czogbWFzdGVyOiB3YWxzZW5kZXIgcG9zdGdyZXMgW2xvY2FsXSBTVEFSVF9SRVBMSUNBVElPTlsw
eDkxNWViMV08YnI+IHBvc3RncmVzOiBtYXN0ZXI6IHdhbHNlbmRlciBwb3N0Z3JlcyBbbG9jYWxd
IFNUQVJUX1JFUExJQ0FUSU9OWzB4OTE1MjBjXTxicj4gcG9zdGdyZXM6IG1hc3Rlcjogd2Fsc2Vu
ZGVyIHBvc3RncmVzIFtsb2NhbF0gU1RBUlRfUkVQTElDQVRJT05bMHg5MTNiZGRdPGJyPiBwb3N0
Z3JlczogbWFzdGVyOiB3YWxzZW5kZXIgcG9zdGdyZXMgW2xvY2FsXTxicj4gU1RBUlRfUkVQTElD
QVRJT04oZXhlY19yZXBsaWNhdGlvbl9jb21tYW5kKzB4NDJjKVsweDkxNDU5M108YnI+IHBvc3Rn
cmVzOiBtYXN0ZXI6IHdhbHNlbmRlciBwb3N0Z3JlcyBbbG9jYWxdPGJyPiBTVEFSVF9SRVBMSUNB
VElPTihQb3N0Z3Jlc01haW4rMHg3YmUpWzB4OTg0ZGRmXTxicj4gcG9zdGdyZXM6IG1hc3Rlcjog
d2Fsc2VuZGVyIHBvc3RncmVzIFtsb2NhbF0gU1RBUlRfUkVQTElDQVRJT05bMHg4YzBkNDFdPGJy
PiBwb3N0Z3JlczogbWFzdGVyOiB3YWxzZW5kZXIgcG9zdGdyZXMgW2xvY2FsXSBTVEFSVF9SRVBM
SUNBVElPTlsweDhjMDZiM108YnI+IHBvc3RncmVzOiBtYXN0ZXI6IHdhbHNlbmRlciBwb3N0Z3Jl
cyBbbG9jYWxdIFNUQVJUX1JFUExJQ0FUSU9OWzB4OGJjOWM0XTxicj4gcG9zdGdyZXM6IG1hc3Rl
cjogd2Fsc2VuZGVyIHBvc3RncmVzIFtsb2NhbF08YnI+IFNUQVJUX1JFUExJQ0FUSU9OKFBvc3Rt
YXN0ZXJNYWluKzB4MTE3YSlbMHg4YmMyOWJdPGJyPiBwb3N0Z3JlczogbWFzdGVyOiB3YWxzZW5k
ZXIgcG9zdGdyZXMgW2xvY2FsXSBTVEFSVF9SRVBMSUNBVElPTlsweDdiZGFmOV08YnI+IC9saWI2
NC9saWJjLnNvLjYoX19saWJjX3N0YXJ0X21haW4rMHhmNSlbMHg3ZjgxNWRkOTI1NTVdPGJyPiBw
b3N0Z3JlczogbWFzdGVyOiB3YWxzZW5kZXIgcG9zdGdyZXMgW2xvY2FsXSBTVEFSVF9SRVBMSUNB
VElPTlsweDQ4OGQwOV08YnI+PGJyPjwvYmxvY2txdW90ZT48YnI+VGhhbmsgeW91IGZvciByZXBv
cnRpbmcgdGhlIGlzc3VlITxicj48YnI+SSBjb3VsZCByZXByb2R1Y2UgdGhpcyBpc3N1ZSB3aXRo
IGEgc21hbGwgY2hhbmdlIGFuZCB0aGUgZm9sbG93aW5nIHN0ZXBzOjxicj48YnI+MS4gQWRkIHNs
ZWVwIGFmdGVyIHNldHRpbmcgU05BUEJVSUxEX0JVSUxESU5HX1NOQVBTSE9ULjxicj48YnI+QEAg
LTE0MDksNiArMTQwOSw5IEBAIFNuYXBCdWlsZEZpbmRTbmFwc2hvdChTbmFwQnVpbGQgKmJ1aWxk
ZXIsPGJyPlhMb2dSZWNQdHIgbHNuLCB4bF9ydW5uaW5nX3hhY3RzICpydW5uPGJyPiAgICAgICAg
ICAgICAgICAgICAgICAgICAgIHJ1bm5pbmctJmd0O3hjbnQsIHJ1bm5pbmctJmd0O25leHRYaWQp
KSk7PGJyPjxicj4gICAgICAgIFNuYXBCdWlsZFdhaXRTbmFwc2hvdChydW5uaW5nLCBydW5uaW5n
LSZndDtuZXh0WGlkKTs8YnI+Kzxicj4rICZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZu
YnNwO2Vsb2coTE9HLCAic2xlZXAgMTBzIik7PGJyPisgJm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7
Jm5ic3A7Jm5ic3A7cGdfdXNsZWVwKDEwICogMTAwMDAwMCk7PGJyPiAgICB9PGJyPjxicj4yLiBD
cmVhdGUgYW5kIHN0YXJ0IGRhdGFiYXNlIGluc3RhbmNlIGFuZCBjcmVhdGUgYSB0YWJsZSBhbmQg
dGhlPGJyPnJlcGxpY2F0aW9uIHNsb3QgJ3MxJy48YnI+Y3JlYXRlIHRhYmxlIHRlc3QgKGkgaW50
KTs8YnI+c2VsZWN0IHBnX2xvZ2ljYWxfcmVwbGljYXRpb25fc2xvdCgnczEnLCAndGVzdF9kZWNv
ZGluZycpOzxicj48YnI+My4gQ3JlYXRlIGFub3RoZXIgcmVwbGljYXRpb24gc2xvdCAiczIiIHdo
aWxlIGEgdHJhbnNhY3Rpb24gaXMgcnVubmluZy48YnI+PGJyPnR4LTE6PGJyPmJlZ2luOzxicj5p
bnNlcnQgaW50byB0ZXN0IHZhbHVlcyAoMSk7PGJyPjxicj4gICAgdHgtMjo8YnI+ICAgIHNlbGVj
dCBwZ19sb2dpY2FsX3JlcGxpY2F0aW9uX3Nsb3QoJ3MyJywgJ3Rlc3RfZGVjb2RpbmcnKTs8YnI+
PGJyPmNoZWNrcG9pbnQ7PGJyPmNvbW1pdDs8YnI+PGJyPk5vdGUgdGhhdCByZXBsaWNhdGlvbiBz
bG90IGNyZWF0aW9uIHdpbGwgd2FpdCBmb3IgMTAgc2VjIGFmdGVyPGJyPmNvbW1pdGluZyB0aGUg
dHgtMSwgcGxlYXNlIGdvIHRvIHRoZSBuZXh0IHN0ZXAgb25jZSB0aGUgc2xvdCBoYXMgYmVlbjxi
cj5jcmVhdGVkLjxicj48YnI+NC4gU3RhcnQgZGVjb2Rpbmcgb24gc2xvdCAnczInLjxicj5zZWxl
Y3QgcGdfbG9naWNhbF9zbG90X2dldF9jaGFuZ2VzKCdzMicsIG51bGwsIG51bGwpOzxicj48YnI+
VGhlIGxvZ2ljYWwgZGVjb2Rpbmcgd2lsbCBzbGVlcC4gUGxlYXNlIGdvIHRvIHRoZSBuZXh0IHN0
ZXAgYWZ0ZXI8YnI+c2VlaW5nIHRoZSBsb2cgInNsZWVwIDEwcyIuPGJyPjxicj41LiBXaGlsZSB0
aGUgbG9naWNhbCBkZWNvZGluZyBzdGFydGVkIGF0IHN0ZXAgNCBpcyBzbGVlcGluZywgc3RhcnQ8
YnI+ZGVjb2Rpbmcgb24gc2xvdCAnczEnLjxicj5zZWxlY3QgcGdfbG9naWNhbF9zbG90X2dldF9j
aGFuZ2VzKCdzMScsIG51bGwsIG51bGwpOzxicj48YnI+Ni4gQWZ0ZXIgdGhlIGRlY29kaW5nIG9u
IHNsb3QgJ3MxJyB3YWtlcyB1cCwgaXQgZmFpbHMgZHVlIHRvIHRoZTxicj5hc3NlcnRpb24gY2hl
Y2suPGJyPjxicj5UaGlzIHNjZW5hcmlvIHNpbXVsYXRlcyB0aGUgY2FzZSB3aGVyZSB0aGUgbG9n
aWNhbCBkZWNvZGluZyAocmUpc3RhcnRzPGJyPndoZXJlIFJVTk5JTkdfWEFDVFMgcmVjb3JkIGhh
dmluZyBhIHJ1bm5pbmcgdHJhbnNhY3Rpb25zIGFuZCBpdDxicj5yZXN0b3JlcyB0aGUgc2VyaWFs
aXplZCBzbmFwc2hvdCB3aGVuIGRlY29kaW5nIHRoZSBuZXh0IFJVTk5JTkdfWEFDVFM8YnI+cmVj
b3JkLiBTaW5jZSB3ZSBkb24ndCByZXNldCBidWlsZGVyLSZndDtuZXh0X3BoYXNlX2F0IHdoZW4g
cmVzdG9yaW5nIGE8YnI+c2VyaWFsaXplZCBzbmFwc2hvdCwgdGhlIGFzc2VydGlvbiBjaGVjayBp
biBTbmFwQnVpbGRTZXJpYWxpemUgZmFpbHMuPGJyPjxicj5SZWdhcmRpbmcgdGhlIHByb3Bvc2Vk
IHBhdGNoLCBJJ3ZlIGNvbmZpcm1lZCBpdCBmaXhlcyB0aGlzIGlzc3VlLiBCdXQ8YnI+SSB0aGlu
ayBpdCdzIGJldHRlciB0byByZXNldCBidWlsZGVyLSZndDtuZXh0X3BoYXNlX2F0IHJpZ2h0IGFm
dGVyIHRoZTxicj5mb2xsb3dpbmcgYXNzZXJ0aW9uIGNoZWNrOjxicj48YnI+ICAgLyogY29uc2lz
dGVudCBzbmFwc2hvdHMgaGF2ZSBubyBuZXh0IHBoYXNlICovPGJyPiAgIEFzc2VydChvbmRpc2su
YnVpbGRlci5uZXh0X3BoYXNlX2F0ID09IEludmFsaWRUcmFuc2FjdGlvbklkKTs8YnI+PGJyPkkg
Y291bGQgbm90IHJlcHJvZHVjZSB0aGlzIGlzc3VlIHdpdGggeW91ciBzY3JpcHQgaW4gbXkgZW52
aXJvbm1lbnQuIEk8YnI+dGhpbmsgaXQncyBiZXR0ZXIgdG8gaW5jbHVkZSB0aGUgcmVwcm9kdWNp
YmxlIHRlc3QgY2FzZSBpbiB0aGUgcGF0Y2g8YnI+YnV0IEknbSBub3Qgc3VyZSBob3cgdG8gZG8g
dGhhdCB3aXRob3V0IGFkZGluZyBzbGVlcC9nZGIgYXR0YWNoLjxicj48YnI+UmVnYXJkcyw8YnI+
PGJyPi0tIDxicj5NYXNhaGlrbyBTYXdhZGE8YnI+QW1hem9uIFdlYiBTZXJ2aWNlczogaHR0cHM6
Ly9hd3MuYW1hem9uLmNvbTwvZGl2Pgo8L2Rpdj4KPCEtLe+/vS0tPg0KPC9kaXY+DQo8L2JvZHk+
DQo8L2h0bWw+
--=_mailmaster-63807243_4486b095_1174d_=
Content-Type: application/octet-stream;
name="0001-Fix-snapshot-builder-s-state-after-restoring-snapsho.patch"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
filename="0001-Fix-snapshot-builder-s-state-after-restoring-snapsho.patch"
RnJvbSBlYTdlNTA0ZDlhYzkwYmEwZGZhODczMWY3YjM0NjVlMDdkMTliYWY4IE1vbiBTZXAgMTcg
MDA6MDA6MDAgMjAwMQpGcm9tOiBib3ZlbnNoaSA8Ym92ZW5zaGlAdGVuY2VudC5jb20+CkRhdGU6
IEZyaSwgMjUgTm92IDIwMjIgMTU6NDI6MDEgKzA4MDAKU3ViamVjdDogW1BBVENIXSBGaXggc25h
cHNob3QgYnVpbGRlcidzIHN0YXRlIGFmdGVyIHJlc3RvcmluZyBzbmFwc2hvdCBmcm9tCiBkaXNr
LgoKU2V0IHNuYXBzaG90IGJ1aWxkZXIncyBzdGF0ZSBhZnRlciByZXN0b3Jpbmcgc25hcHNob3Qg
ZnJvbSBkaXNrLCBzaW5jZSBjb25zaXN0ZW50IHNuYXBzaG90cyBoYXZlIG5vIG5leHQgcGhhc2Uu
CgpQZXIgcmVwb3J0cyBmcm9tIGJvd2Vuc2hpLgoKRGlzY3Vzc2lvbjogaHR0cHM6Ly93d3cucG9z
dGdyZXNxbC5vcmcvbWVzc2FnZS1pZC8xNzY5NS02YmU5Mjc3YzkyOTU5ODVmJTQwcG9zdGdyZXNx
bC5vcmcKLS0tCiBzcmMvYmFja2VuZC9yZXBsaWNhdGlvbi9sb2dpY2FsL3NuYXBidWlsZC5jICAg
ICAgfCAgIDYgKysKIHNyYy90ZXN0L3JlY292ZXJ5L3QvMDM0X3JlcGxpY2F0aW9uX3NuYXBidWls
ZC5wbCB8IDEwMCArKysrKysrKysrKysrKysrKysrKysrKwogMiBmaWxlcyBjaGFuZ2VkLCAxMDYg
aW5zZXJ0aW9ucygrKQogY3JlYXRlIG1vZGUgMTAwNjQ0IHNyYy90ZXN0L3JlY292ZXJ5L3QvMDM0
X3JlcGxpY2F0aW9uX3NuYXBidWlsZC5wbAoKZGlmZiAtLWdpdCBhL3NyYy9iYWNrZW5kL3JlcGxp
Y2F0aW9uL2xvZ2ljYWwvc25hcGJ1aWxkLmMgYi9zcmMvYmFja2VuZC9yZXBsaWNhdGlvbi9sb2dp
Y2FsL3NuYXBidWlsZC5jCmluZGV4IGI1YjAxOGUuLjAzZTc0NTggMTAwNjQ0Ci0tLSBhL3NyYy9i
YWNrZW5kL3JlcGxpY2F0aW9uL2xvZ2ljYWwvc25hcGJ1aWxkLmMKKysrIGIvc3JjL2JhY2tlbmQv
cmVwbGljYXRpb24vbG9naWNhbC9zbmFwYnVpbGQuYwpAQCAtMTk2NCw2ICsxOTY0LDEyIEBAIFNu
YXBCdWlsZFJlc3RvcmUoU25hcEJ1aWxkICpidWlsZGVyLCBYTG9nUmVjUHRyIGxzbikKIAkvKiBj
b25zaXN0ZW50IHNuYXBzaG90cyBoYXZlIG5vIG5leHQgcGhhc2UgKi8KIAlBc3NlcnQob25kaXNr
LmJ1aWxkZXIubmV4dF9waGFzZV9hdCA9PSBJbnZhbGlkVHJhbnNhY3Rpb25JZCk7CiAKKwkvKiAK
KwkgKiBXZSBhcmUgbm93IGluIGNvbnNpc3RlbnQgcG9pbnQgYW5kIGhhdmUgbm8gbmV4dCBwaGFz
ZSwgY29weSB0aGUgc3RhdGUKKwkgKiBmcm9tIHRoZSBzbmFwc2hvdCBpbiBkaXNrLgorCSAqLwor
CWJ1aWxkZXItPm5leHRfcGhhc2VfYXQgPSBvbmRpc2suYnVpbGRlci5uZXh0X3BoYXNlX2F0Owor
CiAJLyogb2ssIHdlIHRoaW5rIHRoZSBzbmFwc2hvdCBpcyBzZW5zaWJsZSwgY29weSBvdmVyIGV2
ZXJ5dGhpbmcgaW1wb3J0YW50ICovCiAJYnVpbGRlci0+eG1pbiA9IG9uZGlzay5idWlsZGVyLnht
aW47CiAJYnVpbGRlci0+eG1heCA9IG9uZGlzay5idWlsZGVyLnhtYXg7CmRpZmYgLS1naXQgYS9z
cmMvdGVzdC9yZWNvdmVyeS90LzAzNF9yZXBsaWNhdGlvbl9zbmFwYnVpbGQucGwgYi9zcmMvdGVz
dC9yZWNvdmVyeS90LzAzNF9yZXBsaWNhdGlvbl9zbmFwYnVpbGQucGwKbmV3IGZpbGUgbW9kZSAx
MDA2NDQKaW5kZXggMDAwMDAwMC4uOWE0MGYyNQotLS0gL2Rldi9udWxsCisrKyBiL3NyYy90ZXN0
L3JlY292ZXJ5L3QvMDM0X3JlcGxpY2F0aW9uX3NuYXBidWlsZC5wbApAQCAtMCwwICsxLDEwMCBA
QAorIyBDb3B5cmlnaHQgKGMpIDIwMjEtMjAyMiwgUG9zdGdyZVNRTCBHbG9iYWwgRGV2ZWxvcG1l
bnQgR3JvdXAKKworIyBUZXN0IHJlcGxpY2F0aW9uIHNuYXBzaG90IHJlYnVpbGQuCit1c2Ugc3Ry
aWN0OwordXNlIHdhcm5pbmdzOwordXNlIFRlc3RMaWI7Cit1c2UgUG9zdGdyZXNOb2RlOwordXNl
IFRlc3Q6Ok1vcmU7Cit1c2UgREJJOwordXNlIFRlc3Q6Ok1vcmUgdGVzdHMgPT4gMTsKKworIyBz
ZXQgc2xvdCBudW1iZXIKK215ICRzbG90X251bSA9IDEwMDsKKworIyBpbml0aWFsaXplIG1hc3Rl
ciBub2RlCitteSAkbWFzdGVyID0gZ2V0X25ld19ub2RlKCdtYXN0ZXInKTsKKyRtYXN0ZXItPmlu
aXQ7CiskbWFzdGVyLT5hcHBlbmRfY29uZigKKwkncG9zdGdyZXNxbC5jb25mJywgcXEoCit3YWxf
bGV2ZWw9bG9naWNhbAorbWF4X2Nvbm5lY3Rpb25zPTEwMDAKK21heF93b3JrZXJfcHJvY2Vzc2Vz
PTUwMAorbWF4X3dhbF9zZW5kZXJzPTUwMAorbWF4X3JlcGxpY2F0aW9uX3Nsb3RzPTUwMAorbWF4
X2xvZ2ljYWxfcmVwbGljYXRpb25fd29ya2Vycz01MDAKK2xvZ19lcnJvcl92ZXJib3NpdHkgPSB2
ZXJib3NlCitsb2dfc3RhdGVtZW50ID0gYWxsCispKTsKKyRtYXN0ZXItPnN0YXJ0OworbXkgJG1h
c3Rlcl9wb3J0ID0gJG1hc3Rlci0+cG9ydDsKKworIyBtYXN0ZXIgaW5pdAorZm9yIChteSAkaSA9
IDA7ICRpPCRzbG90X251bTsgJGkgPSAkaSsxKQoreworCW15ICR0YWJsZV9uYW1lPXFxKHRfJGkp
OworCW15ICRwdWJfbmFtZT1xcShwdWJfJGkpOworCSRtYXN0ZXItPnNhZmVfcHNxbCgncG9zdGdy
ZXMnLCAiY3JlYXRlIHRhYmxlICR0YWJsZV9uYW1lKGEgaW50KTsiKTsKKwkkbWFzdGVyLT5zYWZl
X3BzcWwoJ3Bvc3RncmVzJywgImNyZWF0ZSBwdWJsaWNhdGlvbiAkcHViX25hbWUgZm9yIHRhYmxl
ICR0YWJsZV9uYW1lOyIpOworfQorCisjIGNyZWF0ZSBzdWJzY3JpcHRpb24KK215ICRzdWJzY3Jp
YmVyID0gZ2V0X25ld19ub2RlKCdzdWJzY3JpYmVyJyk7Ciskc3Vic2NyaWJlci0+aW5pdDsKKyRz
dWJzY3JpYmVyLT5hcHBlbmRfY29uZigKKwkncG9zdGdyZXNxbC5jb25mJywgcXEoCit3YWxfbGV2
ZWw9bG9naWNhbAorbWF4X2Nvbm5lY3Rpb25zPTEwMDAKK21heF93b3JrZXJfcHJvY2Vzc2VzPTUw
MAorbWF4X3JlcGxpY2F0aW9uX3Nsb3RzPTUwMAorbG9nX2Vycm9yX3ZlcmJvc2l0eSA9IHZlcmJv
c2UKK21heF9sb2dpY2FsX3JlcGxpY2F0aW9uX3dvcmtlcnM9NTAwCitsb2dfc3RhdGVtZW50ID0g
YWxsCispKTsKKyRzdWJzY3JpYmVyLT5zdGFydDsKKworIyBzdWJzY3JpYmVyIGluaXQKK2ZvciAo
bXkgJGkgPSAwOyAkaTwkc2xvdF9udW07ICRpID0gJGkrMSkKK3sKKwlteSAkdGFibGVfbmFtZT1x
cSh0XyRpKTsKKwlteSAkcHViX25hbWU9cXEocHViXyRpKTsKKwlteSAkc3ViX25hbWU9cXEoc3Vi
XyRpKTsKKworCSRzdWJzY3JpYmVyLT5zYWZlX3BzcWwoJ3Bvc3RncmVzJywKKwkiY3JlYXRlIHRh
YmxlICR0YWJsZV9uYW1lKGEgaW50KTsiKTsKK30KKworbXkgJHNjcmlwdCA9ICRtYXN0ZXItPmJh
c2VkaXIgLiAnL3BnYmVuY2hfc2NyaXB0JzsKK2ZvciAobXkgJGkgPSAwOyAkaTwkc2xvdF9udW07
ICRpID0gJGkrMSkKK3sKKwlteSAkdGFibGVfbmFtZT1xcSh0XyRpKTsKKworCWFwcGVuZF90b19m
aWxlKCRzY3JpcHQsCisJIklOU0VSVCBJTlRPICR0YWJsZV9uYW1lIFNFTEVDVCBGUk9NIGdlbmVy
YXRlX3NlcmllcygxLDEwMCk7Iik7Cit9CisKK215ICRwaWQgPSBmb3JrKCk7CitpZiAoJHBpZCA9
PSAwKQoreworCXNsZWVwKDEpOworCWZvciAobXkgJGkgPSAwOyAkaTwkc2xvdF9udW07ICRpID0g
JGkrMSkKKwl7CisJCW15ICRwdWJfbmFtZT1xcShwdWJfJGkpOworCQlteSAkc3ViX25hbWU9cXEo
c3ViXyRpKTsKKworCQkkc3Vic2NyaWJlci0+c2FmZV9wc3FsKCdwb3N0Z3JlcycsCisJCSJjcmVh
dGUgc3Vic2NyaXB0aW9uICRzdWJfbmFtZSBjb25uZWN0aW9uICdwb3J0PSRtYXN0ZXJfcG9ydCBk
Ym5hbWU9cG9zdGdyZXMgdXNlcj1wb3N0Z3JlcycgcHVibGljYXRpb24gJHB1Yl9uYW1lOyIpOwor
CX0KKwlleGVjICJlY2hvIiwgIlxuYnllXG4iOworfQorZWxzaWYgKCRwaWQgPCAwKQoreworCWRp
ZSAiZmFpbGVkIHRvIGZvcmsgbmV3IHByb2Nlc3MiOworfQorCiskbWFzdGVyLT5ydW5fbG9nKAor
CVtxdyhwZ2JlbmNoIC0tbm8tdmFjdXVtIC0tY2xpZW50PTEwIC1VIHBvc3RncmVzCisJLVQgMjUg
LS1maWxlKSwgJHNjcmlwdCBdKTsKKworbXkgJHJlc3VsdCA9ICRtYXN0ZXItPnNhZmVfcHNxbCgn
cG9zdGdyZXMnLCAic2VsZWN0IGNvdW50KCopIGZyb20gcGdfcmVwbGljYXRpb25fc2xvdHM7Iik7
CitpcygkcmVzdWx0LCBxcSgkc2xvdF9udW0pLCAiQ3JlYXRlIHNsb3RzIHN1Y2Nlc3NmdWxseS4i
KQpcIE5vIG5ld2xpbmUgYXQgZW5kIG9mIGZpbGUKLS0gCjIuOS4zCgo=
--=_mailmaster-63807243_4486b095_1174d_=--
Attachments:
[application/octet-stream] 0001-Fix-snapshot-builder-s-state-after-restoring-snapsho.patch (4.0K, ../../tencent_7EB71DA5D7BA00EB0B429DCE45D0452B6406@qq.com/2-0001-Fix-snapshot-builder-s-state-after-restoring-snapsho.patch)
download | inline diff:
From ea7e504d9ac90ba0dfa8731f7b3465e07d19baf8 Mon Sep 17 00:00:00 2001
From: bovenshi <bovenshi@tencent.com>
Date: Fri, 25 Nov 2022 15:42:01 +0800
Subject: [PATCH] Fix snapshot builder's state after restoring snapshot from
disk.
Set snapshot builder's state after restoring snapshot from disk, since consistent snapshots have no next phase.
Per reports from bowenshi.
Discussion: https://www.postgresql.org/message-id/17695-6be9277c9295985f%40postgresql.org
---
src/backend/replication/logical/snapbuild.c | 6 ++
src/test/recovery/t/034_replication_snapbuild.pl | 100 +++++++++++++++++++++++
2 files changed, 106 insertions(+)
create mode 100644 src/test/recovery/t/034_replication_snapbuild.pl
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index b5b018e..03e7458 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -1964,6 +1964,12 @@ SnapBuildRestore(SnapBuild *builder, XLogRecPtr lsn)
/* consistent snapshots have no next phase */
Assert(ondisk.builder.next_phase_at == InvalidTransactionId);
+ /*
+ * We are now in consistent point and have no next phase, copy the state
+ * from the snapshot in disk.
+ */
+ builder->next_phase_at = ondisk.builder.next_phase_at;
+
/* ok, we think the snapshot is sensible, copy over everything important */
builder->xmin = ondisk.builder.xmin;
builder->xmax = ondisk.builder.xmax;
diff --git a/src/test/recovery/t/034_replication_snapbuild.pl b/src/test/recovery/t/034_replication_snapbuild.pl
new file mode 100644
index 0000000..9a40f25
--- /dev/null
+++ b/src/test/recovery/t/034_replication_snapbuild.pl
@@ -0,0 +1,100 @@
+# Copyright (c) 2021-2022, PostgreSQL Global Development Group
+
+# Test replication snapshot rebuild.
+use strict;
+use warnings;
+use TestLib;
+use PostgresNode;
+use Test::More;
+use DBI;
+use Test::More tests => 1;
+
+# set slot number
+my $slot_num = 100;
+
+# initialize master node
+my $master = get_new_node('master');
+$master->init;
+$master->append_conf(
+ 'postgresql.conf', qq(
+wal_level=logical
+max_connections=1000
+max_worker_processes=500
+max_wal_senders=500
+max_replication_slots=500
+max_logical_replication_workers=500
+log_error_verbosity = verbose
+log_statement = all
+));
+$master->start;
+my $master_port = $master->port;
+
+# master init
+for (my $i = 0; $i<$slot_num; $i = $i+1)
+{
+ my $table_name=qq(t_$i);
+ my $pub_name=qq(pub_$i);
+ $master->safe_psql('postgres', "create table $table_name(a int);");
+ $master->safe_psql('postgres', "create publication $pub_name for table $table_name;");
+}
+
+# create subscription
+my $subscriber = get_new_node('subscriber');
+$subscriber->init;
+$subscriber->append_conf(
+ 'postgresql.conf', qq(
+wal_level=logical
+max_connections=1000
+max_worker_processes=500
+max_replication_slots=500
+log_error_verbosity = verbose
+max_logical_replication_workers=500
+log_statement = all
+));
+$subscriber->start;
+
+# subscriber init
+for (my $i = 0; $i<$slot_num; $i = $i+1)
+{
+ my $table_name=qq(t_$i);
+ my $pub_name=qq(pub_$i);
+ my $sub_name=qq(sub_$i);
+
+ $subscriber->safe_psql('postgres',
+ "create table $table_name(a int);");
+}
+
+my $script = $master->basedir . '/pgbench_script';
+for (my $i = 0; $i<$slot_num; $i = $i+1)
+{
+ my $table_name=qq(t_$i);
+
+ append_to_file($script,
+ "INSERT INTO $table_name SELECT FROM generate_series(1,100);");
+}
+
+my $pid = fork();
+if ($pid == 0)
+{
+ sleep(1);
+ for (my $i = 0; $i<$slot_num; $i = $i+1)
+ {
+ my $pub_name=qq(pub_$i);
+ my $sub_name=qq(sub_$i);
+
+ $subscriber->safe_psql('postgres',
+ "create subscription $sub_name connection 'port=$master_port dbname=postgres user=postgres' publication $pub_name;");
+ }
+ exec "echo", "\nbye\n";
+}
+elsif ($pid < 0)
+{
+ die "failed to fork new process";
+}
+
+$master->run_log(
+ [qw(pgbench --no-vacuum --client=10 -U postgres
+ -T 25 --file), $script ]);
+
+my $result = $master->safe_psql('postgres', "select count(*) from pg_replication_slots;");
+is($result, qq($slot_num), "Create slots successfully.")
\ No newline at end of file
--
2.9.3
^ permalink raw reply [nested|flat] 17+ messages in thread
* RE: BUG #17695: Failed Assert in logical replication snapbuild.
@ 2022-11-25 09:07 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: bowenshi <zxwsbg@qq.com>
2 siblings, 0 replies; 17+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2022-11-25 09:07 UTC (permalink / raw)
To: bowenshi <zxwsbg@qq.com>; Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Dear Bowenshi,
> Thanks for your advice! I update my test script, now it can 100% reproduce problem in my computer.
Thanks for updates. Note that even if I used the updated script in my environment (HEAD, 2cf41cd30),
I could not perfectly reproduce the issue. The publisher rarely crashed and output the core-file.
The failed assertion was same as you reported firstly[1], not newer one.
[1]: https://www.postgresql.org/message-id/17695-6be9277c9295985f%40postgresql.org
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: BUG #17695: Failed Assert in logical replication snapbuild.
@ 2022-11-25 09:56 bowenshi <zxwsbg@qq.com>
parent: bowenshi <zxwsbg@qq.com>
2 siblings, 0 replies; 17+ messages in thread
From: bowenshi @ 2022-11-25 09:56 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; +Cc: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; Masahiko Sawada <sawada.mshk@gmail.com>
<html>
<head>
<meta http-equiv='Content-Type' content='text/html; charset=UTF-8'>
</head>
<body>
<style>
font{
line-height: 1.6;
}
ul,ol{
padding-left: 20px;
list-style-position: inside;
}
</style>
<div style = 'font-family:Helvetica,Helvetica,微软雅黑, 宋体; line-height:1.6;'>
<div ></div><div>
<div><div>I add my fix code which in the patch. The code fix the origin problem, but it failed for the following new reason.</div><div><div style="font-family: Helvetica, Helvetica, 微软雅黑, 宋体;">TRAP: FailedAssertion("TransactionIdPrecedesOrEquals(safeXid, snap->xmin)", File: "snapbuild.c", Line: 593, PID: 1576)<br></div><span style="font-family: Helvetica, Helvetica, 微软雅黑, 宋体;">postgres: master: walsender postgres [local] CREATE_REPLICATION_SLOT(ExceptionalCondition+0xb9)[0xb1ca28]</span><br style="font-family: Helvetica, Helvetica, 微软雅黑, 宋体;"><span style="font-family: Helvetica, Helvetica, 微软雅黑, 宋体;">postgres: master: walsender postgres [local] CREATE_REPLICATION_SLOT(SnapBuildInitialSnapshot+0x1b3)[0x8f3c79]</span><br style="font-family: Helvetica, Helvetica, 微软雅黑, 宋体;"><span style="font-family: Helvetica, Helvetica, 微软雅黑, 宋体;">postgres: master: walsender postgres [local] CREATE_REPLICATION_SLOT[0x9136b9]</span><br style="font-family: Helvetica, Helvetica, 微软雅黑, 宋体;"><span style="font-family: Helvetica, Helvetica, 微软雅黑, 宋体;">postgres: master: walsender postgres [local] CREATE_REPLICATION_SLOT(exec_replication_command+0x398)[0x91456a]</span><br style="font-family: Helvetica, Helvetica, 微软雅黑, 宋体;"><span style="font-family: Helvetica, Helvetica, 微软雅黑, 宋体;">postgres: master: walsender postgres [local] CREATE_REPLICATION_SLOT(PostgresMain+0x7be)[0x984e4a]</span><br style="font-family: Helvetica, Helvetica, 微软雅黑, 宋体;"><span style="font-family: Helvetica, Helvetica, 微软雅黑, 宋体;">postgres: master: walsender postgres [local] CREATE_REPLICATION_SLOT[0x8c0d5b]</span><br style="font-family: Helvetica, Helvetica, 微软雅黑, 宋体;"><span style="font-family: Helvetica, Helvetica, 微软雅黑, 宋体;">postgres: master: walsender postgres [local] CREATE_REPLICATION_SLOT[0x8c06cd]</span><br style="font-family: Helvetica, Helvetica, 微软雅黑, 宋体;"><span style="font-family: Helvetica, Helvetica, 微软雅黑, 宋体;">postgres: master: walsender postgres [local] CREATE_REPLICATION_SLOT[0x8bc9de]</span><br style="font-family: Helvetica, Helvetica, 微软雅黑, 宋体;"><span style="font-family: Helvetica, Helvetica, 微软雅黑, 宋体;">postgres: master: walsender postgres [local] CREATE_REPLICATION_SLOT(PostmasterMain+0x117a)[0x8bc2b5]</span><br style="font-family: Helvetica, Helvetica, 微软雅黑, 宋体;"><span style="font-family: Helvetica, Helvetica, 微软雅黑, 宋体;">postgres: master: walsender postgres [local] CREATE_REPLICATION_SLOT[0x7bdb13]</span><br></div></div></div><div><span style="font-family: Helvetica, Helvetica, 微软雅黑, 宋体;"><br></span></div>
<!--�-->
</div>
</body>
</html>
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: BUG #17695: Failed Assert in logical replication snapbuild.
@ 2022-11-28 05:07 Masahiko Sawada <sawada.mshk@gmail.com>
parent: bowenshi <zxwsbg@qq.com>
2 siblings, 1 reply; 17+ messages in thread
From: Masahiko Sawada @ 2022-11-28 05:07 UTC (permalink / raw)
To: bowenshi <zxwsbg@qq.com>; +Cc: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Fri, Nov 25, 2022 at 4:44 PM bowenshi <zxwsbg@qq.com> wrote:
>
> Thanks for your advice! I update my test script, now it can 100% reproduce problem in my computer.
Hmm, I could not reproduce the issue even with the updated patch.
> The new script and fix code are both in the patch. However, I met new problem after adding fix code. It fails into a new trap following:
>
> TRAP: FailedAssertion("TransactionIdPrecedesOrEquals(safeXid, snap->xmin)", File: "snapbuild.c", Line: 593, PID: 1576)
> postgres: master: walsender postgres [local] CREATE_REPLICATION_SLOT(ExceptionalCondition+0xb9)[0xb1ca28]
> postgres: master: walsender postgres [local] CREATE_REPLICATION_SLOT(SnapBuildInitialSnapshot+0x1b3)[0x8f3c79]
> postgres: master: walsender postgres [local] CREATE_REPLICATION_SLOT[0x9136b9]
> postgres: master: walsender postgres [local] CREATE_REPLICATION_SLOT(exec_replication_command+0x398)[0x91456a]
> postgres: master: walsender postgres [local] CREATE_REPLICATION_SLOT(PostgresMain+0x7be)[0x984e4a]
> postgres: master: walsender postgres [local] CREATE_REPLICATION_SLOT[0x8c0d5b]
> postgres: master: walsender postgres [local] CREATE_REPLICATION_SLOT[0x8c06cd]
> postgres: master: walsender postgres [local] CREATE_REPLICATION_SLOT[0x8bc9de]
> postgres: master: walsender postgres [local] CREATE_REPLICATION_SLOT(PostmasterMain+0x117a)[0x8bc2b5]
> postgres: master: walsender postgres [local] CREATE_REPLICATION_SLOT[0x7bdb13]
>
Testing this script in some environments, I sometimes get this
assertion failure. I think this is the same issue that has been
reported in this thread[1]. I'll investigate this issue as well and
share my findings.
[1] https://www.postgresql.org/message-id/CAA4eK1L8wYcyTPxNzPGkhuO52WBGoOZbT0A73Le%3DZUWYAYmdfw%40mail.g...
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: BUG #17695: Failed Assert in logical replication snapbuild.
@ 2023-03-27 11:49 Daniel Gustafsson <daniel@yesql.se>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 17+ messages in thread
From: Daniel Gustafsson @ 2023-03-27 11:49 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: bowenshi <zxwsbg@qq.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
This thread has been idle for quite some time, and the test isn't running at
all, whereas there is more activity on the linked thread. AFAICT from the
thread this issue is reproducible in HEAD so the test should be for master
right? Should this be closed in favor of the other patch?
A quick glance at the failing test, assuming we want it for master:
+use TestLib;
+use PostgresNode;
This needs updating after b3b4d8e68ae which moved the test modules to a proper namespace.
+use DBI;
Not used in the test, and I can't see why it should be?
+use Test::More tests => 1;
We've moved away from explicit plans in favor of done_testing().
--
Daniel Gustafsson
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: BUG #17695: Failed Assert in logical replication snapbuild.
@ 2023-05-17 05:34 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Daniel Gustafsson <daniel@yesql.se>
0 siblings, 1 reply; 17+ messages in thread
From: Masahiko Sawada @ 2023-05-17 05:34 UTC (permalink / raw)
To: Daniel Gustafsson <daniel@yesql.se>; +Cc: bowenshi <zxwsbg@qq.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Mon, Mar 27, 2023 at 8:49 PM Daniel Gustafsson <daniel@yesql.se> wrote:
>
> This thread has been idle for quite some time, and the test isn't running at
> all, whereas there is more activity on the linked thread. AFAICT from the
> thread this issue is reproducible in HEAD so the test should be for master
> right? Should this be closed in favor of the other patch?
>
Reviewing the thread, one assertion failure was reported[1] and then
another assertion failure was reported[2]. But the later one was
already under discussion on another thread[3]. So I think we should
tackle the former one (i.e. originally reported issue) on this thread.
When it comes to the original issue, I already shared the reproducible
steps[4] and I've confirmed again with the steps that the issue still
happens on 14 or later and the patch . However I don't find a way to
reproduce it without sleep/gdb attach.
The fix is straightforward and it seems OK to me but the regression
test part might need more discussion.
Regards,
[1] https://www.postgresql.org/message-id/17695-6be9277c9295985f%40postgresql.org
[2] https://www.postgresql.org/message-id/tencent_7EB71DA5D7BA00EB0B429DCE45D0452B6406%40qq.com
[3] https://www.postgresql.org/message-id/CAA4eK1L8wYcyTPxNzPGkhuO52WBGoOZbT0A73Le=ZUWYAYmdfw@mail.gmail...
[4] https://www.postgresql.org/message-id/CAD21AoCsxvV3Mpzv8Q3hG7CdQZ7vSBWyEFLgY3QZARRDcgzVwA%40mail.gma...
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] reset_next_phase_at.patch (842B, ../../CAD21AoDNv09ZMr-E+fNzhduvkE6eK2fjCRA7wJHOhF8APH5JdQ@mail.gmail.com/2-reset_next_phase_at.patch)
download | inline diff:
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 62542827e4..b624893051 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -1950,8 +1950,12 @@ SnapBuildRestore(SnapBuild *builder, XLogRecPtr lsn)
if (TransactionIdPrecedes(ondisk.builder.xmin, builder->initial_xmin_horizon))
goto snapshot_not_interesting;
- /* consistent snapshots have no next phase */
+ /*
+ * Consistent snapshots have no next phase. Reset next_phase_at as it is
+ * possible that an old value may remain.
+ */
Assert(ondisk.builder.next_phase_at == InvalidTransactionId);
+ builder->next_phase_at = InvalidTransactionId;
/* ok, we think the snapshot is sensible, copy over everything important */
builder->xmin = ondisk.builder.xmin;
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: BUG #17695: Failed Assert in logical replication snapbuild.
@ 2023-05-18 14:00 Alexander Lakhin <exclusion@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 17+ messages in thread
From: Alexander Lakhin @ 2023-05-18 14:00 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; Daniel Gustafsson <daniel@yesql.se>; +Cc: bowenshi <zxwsbg@qq.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Hello Sawada-san,
17.05.2023 08:34, Masahiko Sawada wrote:
>
> When it comes to the original issue, I already shared the reproducible
> steps[4] and I've confirmed again with the steps that the issue still
> happens on 14 or later and the patch . However I don't find a way to
> reproduce it without sleep/gdb attach.
I can easily (without gdb and sleep()) reproduce the issue on master with
the following script:
numclients=10
rm -rf contrib/test_decoding_*
for ((c=1;c<=numclients;c++)); do
cp -r contrib/test_decoding contrib/test_decoding_$c
done
for ((c=1;c<=numclients;c++)); do
EXTRA_REGRESS_OPTS="--dbname=regress_$c" make -s installcheck-force -C contrib/test_decoding_$c USE_MODULE_DB=1
>"installcheck-$c.log" 2>&1 &
done
wait
It leads to:
TRAP: failed Assert("builder->next_phase_at == InvalidTransactionId"), File: "snapbuild.c", Line: 1628, PID: 907918
...
2023-05-18 16:23:33.290 MSK [907502] LOG: server process (PID 907918) was terminated by signal 6: Aborted
2023-05-18 16:23:33.290 MSK [907502] DETAIL: Failed process was running: SELECT count(*) FROM
pg_logical_slot_get_changes('regression_slot_stats1', NULL, NULL, 'skip-empty-xacts', '1');
...
Core was generated by `postgres: postgres regress_10 [local] SELECT '.
Program terminated with signal SIGABRT, Aborted.
warning: Section `.reg-xstate/907918' in core file too small.
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=140405033059264) at ./nptl/pthread_kill.c:44
44 ./nptl/pthread_kill.c: No such file or directory.
(gdb) bt
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=140405033059264) at ./nptl/pthread_kill.c:44
#1 __pthread_kill_internal (signo=6, threadid=140405033059264) at ./nptl/pthread_kill.c:78
#2 __GI___pthread_kill (threadid=140405033059264, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3 0x00007fb29a0cc476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4 0x00007fb29a0b27f3 in __GI_abort () at ./stdlib/abort.c:79
#5 0x0000557371bd57bb in ExceptionalCondition (
conditionName=conditionName@entry=0x557371d56860 "builder->next_phase_at == InvalidTransactionId",
fileName=fileName@entry=0x557371d572e7 "snapbuild.c", lineNumber=lineNumber@entry=1628) at assert.c:66
#6 0x0000557371a28a29 in SnapBuildSerialize (builder=builder@entry=0x557372879158, lsn=lsn@entry=312723008)
at snapbuild.c:1628
#7 0x0000557371a2a657 in SnapBuildProcessRunningXacts (builder=builder@entry=0x557372879158, lsn=312723008,
running=running@entry=0x557373095190) at snapbuild.c:1230
...
If it would be helpful, I can reduce it to concrete sql queries.
Best regards,
Alexander
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: BUG #17695: Failed Assert in logical replication snapbuild.
@ 2023-05-22 00:56 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Alexander Lakhin <exclusion@gmail.com>
0 siblings, 1 reply; 17+ messages in thread
From: Masahiko Sawada @ 2023-05-22 00:56 UTC (permalink / raw)
To: Alexander Lakhin <exclusion@gmail.com>; +Cc: Daniel Gustafsson <daniel@yesql.se>; bowenshi <zxwsbg@qq.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Thu, May 18, 2023 at 11:00 PM Alexander Lakhin <exclusion@gmail.com> wrote:
>
> Hello Sawada-san,
>
> 17.05.2023 08:34, Masahiko Sawada wrote:
> >
> > When it comes to the original issue, I already shared the reproducible
> > steps[4] and I've confirmed again with the steps that the issue still
> > happens on 14 or later and the patch . However I don't find a way to
> > reproduce it without sleep/gdb attach.
>
> I can easily (without gdb and sleep()) reproduce the issue on master with
> the following script:
> numclients=10
> rm -rf contrib/test_decoding_*
> for ((c=1;c<=numclients;c++)); do
> cp -r contrib/test_decoding contrib/test_decoding_$c
> done
>
> for ((c=1;c<=numclients;c++)); do
> EXTRA_REGRESS_OPTS="--dbname=regress_$c" make -s installcheck-force -C contrib/test_decoding_$c USE_MODULE_DB=1
> >"installcheck-$c.log" 2>&1 &
> done
> wait
Thank you for sharing the script. But it seems not stable as I could
not reproduce the issue in my environment. I think we need a stable
reproducer so that we can include it in core regression tests. Or it
may be okay not to include it if we could not find a convenient way
and the fix is trivial.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: BUG #17695: Failed Assert in logical replication snapbuild.
@ 2023-05-23 11:00 Alexander Lakhin <exclusion@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 17+ messages in thread
From: Alexander Lakhin @ 2023-05-23 11:00 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Daniel Gustafsson <daniel@yesql.se>; bowenshi <zxwsbg@qq.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
22.05.2023 03:56, Masahiko Sawada wrote:
> On Thu, May 18, 2023 at 11:00 PM Alexander Lakhin <exclusion@gmail.com> wrote:
>
>> I can easily (without gdb and sleep()) reproduce the issue on master with
>> the following script:
>> ...
> Thank you for sharing the script. But it seems not stable as I could
> not reproduce the issue in my environment. I think we need a stable
> reproducer so that we can include it in core regression tests. Or it
> may be okay not to include it if we could not find a convenient way
> and the fix is trivial.
I've came to the minimal reproducer:
numclients=40
for ((c=1;c<=numclients;c++)); do
createdb regress_$c
done
for ((c=1;c<=numclients;c++)); do
(
echo "
CREATE TABLE replication_example(id SERIAL PRIMARY KEY, somedata int, text varchar(120));
SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot_$c', 'test_decoding');
SELECT data FROM pg_logical_slot_get_changes('regression_slot_$c', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts',
'1');
" | psql -d regress_$c >psql-$c.log
) &
done
wait
grep TRAP server.log
(I've set
fsync = off
wal_level = logical
in postgresql.conf)
When using a build made with ASAN (and gcc-12), I get several asserts at once:
grep TRAP server.log | wc -l
12
Without ASAN, I get no failures with numclients = 40, but still get series of
those with numclients=80...
It's hardly suitable for the regression test, but it clearly demonstrates the
issue without using gdb. With the fix from [1] applied, I've got no failures,
even with numclients=100, for 10 runs.
I also think, that the fix is simple enough to be committed without a
complicated/resource-intensive regression test.
[1] https://www.postgresql.org/message-id/CAD21AoDNv09ZMr-E%2BfNzhduvkE6eK2fjCRA7wJHOhF8APH5JdQ%40mail.g...
Best regards,
Alexander
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: BUG #17695: Failed Assert in logical replication snapbuild.
@ 2023-07-04 15:43 Daniel Gustafsson <daniel@yesql.se>
parent: Alexander Lakhin <exclusion@gmail.com>
0 siblings, 2 replies; 17+ messages in thread
From: Daniel Gustafsson @ 2023-07-04 15:43 UTC (permalink / raw)
To: Alexander Lakhin <exclusion@gmail.com>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; bowenshi <zxwsbg@qq.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
> On 23 May 2023, at 13:00, Alexander Lakhin <exclusion@gmail.com> wrote:
>
> 22.05.2023 03:56, Masahiko Sawada wrote:
>> On Thu, May 18, 2023 at 11:00 PM Alexander Lakhin <exclusion@gmail.com> wrote:
>>
>>> I can easily (without gdb and sleep()) reproduce the issue on master with
>>> the following script:
>>> ...
>> Thank you for sharing the script. But it seems not stable as I could
>> not reproduce the issue in my environment. I think we need a stable
>> reproducer so that we can include it in core regression tests. Or it
>> may be okay not to include it if we could not find a convenient way
>> and the fix is trivial.
>
> I've came to the minimal reproducer:
Thanks for the reproducer, I was able to reproduce this in HEAD and v16.
> It's hardly suitable for the regression test, but it clearly demonstrates the
> issue without using gdb. With the fix from [1] applied, I've got no failures,
> even with numclients=100, for 10 runs.
>
> I also think, that the fix is simple enough to be committed without a
> complicated/resource-intensive regression test.
I'm not convinced we need a regression test for this as it would be very
expensive and potentially brittle for older/slower buildfarm members while
giving few gains.
I've applied this to HEAD and backpatched it to v16.
--
Daniel Gustafsson
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: BUG #17695: Failed Assert in logical replication snapbuild.
@ 2023-07-05 01:23 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Daniel Gustafsson <daniel@yesql.se>
1 sibling, 0 replies; 17+ messages in thread
From: Masahiko Sawada @ 2023-07-05 01:23 UTC (permalink / raw)
To: Daniel Gustafsson <daniel@yesql.se>; +Cc: Alexander Lakhin <exclusion@gmail.com>; bowenshi <zxwsbg@qq.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Wed, Jul 5, 2023 at 12:43 AM Daniel Gustafsson <daniel@yesql.se> wrote:
>
> > On 23 May 2023, at 13:00, Alexander Lakhin <exclusion@gmail.com> wrote:
> >
> > 22.05.2023 03:56, Masahiko Sawada wrote:
> >> On Thu, May 18, 2023 at 11:00 PM Alexander Lakhin <exclusion@gmail.com> wrote:
> >>
> >>> I can easily (without gdb and sleep()) reproduce the issue on master with
> >>> the following script:
> >>> ...
> >> Thank you for sharing the script. But it seems not stable as I could
> >> not reproduce the issue in my environment. I think we need a stable
> >> reproducer so that we can include it in core regression tests. Or it
> >> may be okay not to include it if we could not find a convenient way
> >> and the fix is trivial.
> >
> > I've came to the minimal reproducer:
>
> Thanks for the reproducer, I was able to reproduce this in HEAD and v16.
>
> > It's hardly suitable for the regression test, but it clearly demonstrates the
> > issue without using gdb. With the fix from [1] applied, I've got no failures,
> > even with numclients=100, for 10 runs.
> >
> > I also think, that the fix is simple enough to be committed without a
> > complicated/resource-intensive regression test.
>
> I'm not convinced we need a regression test for this as it would be very
> expensive and potentially brittle for older/slower buildfarm members while
> giving few gains.
>
> I've applied this to HEAD and backpatched it to v16.
Thanks!
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: BUG #17695: Failed Assert in logical replication snapbuild.
@ 2023-09-19 08:15 Alexander Pyhalov <a.pyhalov@postgrespro.ru>
parent: Daniel Gustafsson <daniel@yesql.se>
1 sibling, 1 reply; 17+ messages in thread
From: Alexander Pyhalov @ 2023-09-19 08:15 UTC (permalink / raw)
To: Daniel Gustafsson <daniel@yesql.se>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Masahiko Sawada <sawada.mshk@gmail.com>; bowenshi <zxwsbg@qq.com>; pgsql-bugs@lists.postgresql.org
Daniel Gustafsson писал 2023-07-04 18:43:
>> On 23 May 2023, at 13:00, Alexander Lakhin <exclusion@gmail.com>
>> wrote:
>>
>> 22.05.2023 03:56, Masahiko Sawada wrote:
>>> On Thu, May 18, 2023 at 11:00 PM Alexander Lakhin
>>> <exclusion@gmail.com> wrote:
>>>
>>>> I can easily (without gdb and sleep()) reproduce the issue on master
>>>> with
>>>> the following script:
>>>> ...
>>> Thank you for sharing the script. But it seems not stable as I could
>>> not reproduce the issue in my environment. I think we need a stable
>>> reproducer so that we can include it in core regression tests. Or it
>>> may be okay not to include it if we could not find a convenient way
>>> and the fix is trivial.
>>
>> I've came to the minimal reproducer:
>
> Thanks for the reproducer, I was able to reproduce this in HEAD and
> v16.
>
>> It's hardly suitable for the regression test, but it clearly
>> demonstrates the
>> issue without using gdb. With the fix from [1] applied, I've got no
>> failures,
>> even with numclients=100, for 10 runs.
>>
>> I also think, that the fix is simple enough to be committed without a
>> complicated/resource-intensive regression test.
>
> I'm not convinced we need a regression test for this as it would be
> very
> expensive and potentially brittle for older/slower buildfarm members
> while
> giving few gains.
>
> I've applied this to HEAD and backpatched it to v16.
>
Hi.
It seems we've managed to get this issue on 14.8. Is there any reason
why it wasn't applied to earlier versions?
--
Best regards,
Alexander Pyhalov,
Postgres Professional
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: BUG #17695: Failed Assert in logical replication snapbuild.
@ 2025-04-30 18:19 Daniel Gustafsson <daniel@yesql.se>
parent: Alexander Pyhalov <a.pyhalov@postgrespro.ru>
0 siblings, 1 reply; 17+ messages in thread
From: Daniel Gustafsson @ 2025-04-30 18:19 UTC (permalink / raw)
To: Alexander Pyhalov <a.pyhalov@postgrespro.ru>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Masahiko Sawada <sawada.mshk@gmail.com>; bowenshi <zxwsbg@qq.com>; pgsql-bugs@lists.postgresql.org
> On 19 Sep 2023, at 10:15, Alexander Pyhalov <a.pyhalov@postgrespro.ru> wrote:
> Daniel Gustafsson писал 2023-07-04 18:43:
>> I've applied this to HEAD and backpatched it to v16.
>
> It seems we've managed to get this issue on 14.8. Is there any reason why it wasn't applied to earlier versions?
(Revisiting a very old thread) This report was missed at the time, and it has
been off-list reported to me to have been identified in 14 and 15 as well, so I
plan on backpatching this fix to 14 and 15.
--
Daniel Gustafsson
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: BUG #17695: Failed Assert in logical replication snapbuild.
@ 2025-04-30 19:55 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Daniel Gustafsson <daniel@yesql.se>
0 siblings, 0 replies; 17+ messages in thread
From: Masahiko Sawada @ 2025-04-30 19:55 UTC (permalink / raw)
To: Daniel Gustafsson <daniel@yesql.se>; +Cc: Alexander Pyhalov <a.pyhalov@postgrespro.ru>; Alexander Lakhin <exclusion@gmail.com>; bowenshi <zxwsbg@qq.com>; pgsql-bugs@lists.postgresql.org
On Wed, Apr 30, 2025 at 11:19 AM Daniel Gustafsson <daniel@yesql.se> wrote:
>
> > On 19 Sep 2023, at 10:15, Alexander Pyhalov <a.pyhalov@postgrespro.ru> wrote:
> > Daniel Gustafsson писал 2023-07-04 18:43:
>
> >> I've applied this to HEAD and backpatched it to v16.
> >
> > It seems we've managed to get this issue on 14.8. Is there any reason why it wasn't applied to earlier versions?
>
> (Revisiting a very old thread) This report was missed at the time,
Yeah, that's bad.
> and it has
> been off-list reported to me to have been identified in 14 and 15 as well, so I
> plan on backpatching this fix to 14 and 15.
+1. I'd confirmed this issue existed on v14 or later[1].
Regards,
[1] https://www.postgresql.org/message-id/CAD21AoDNv09ZMr-E%2BfNzhduvkE6eK2fjCRA7wJHOhF8APH5JdQ%40mail.g...
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 17+ messages in thread
end of thread, other threads:[~2025-04-30 19:55 UTC | newest]
Thread overview: 17+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-11-24 10:15 BUG #17695: Failed Assert in logical replication snapbuild. PG Bug reporting form <noreply@postgresql.org>
2022-11-24 12:39 ` vignesh C <vignesh21@gmail.com>
2022-11-25 05:58 ` Masahiko Sawada <sawada.mshk@gmail.com>
2022-11-25 07:44 ` bowenshi <zxwsbg@qq.com>
2022-11-25 09:07 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2022-11-25 09:56 ` bowenshi <zxwsbg@qq.com>
2022-11-28 05:07 ` Masahiko Sawada <sawada.mshk@gmail.com>
2023-03-27 11:49 ` Daniel Gustafsson <daniel@yesql.se>
2023-05-17 05:34 ` Masahiko Sawada <sawada.mshk@gmail.com>
2023-05-18 14:00 ` Alexander Lakhin <exclusion@gmail.com>
2023-05-22 00:56 ` Masahiko Sawada <sawada.mshk@gmail.com>
2023-05-23 11:00 ` Alexander Lakhin <exclusion@gmail.com>
2023-07-04 15:43 ` Daniel Gustafsson <daniel@yesql.se>
2023-07-05 01:23 ` Masahiko Sawada <sawada.mshk@gmail.com>
2023-09-19 08:15 ` Alexander Pyhalov <a.pyhalov@postgrespro.ru>
2025-04-30 18:19 ` Daniel Gustafsson <daniel@yesql.se>
2025-04-30 19:55 ` Masahiko Sawada <sawada.mshk@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox