From: Улаев Александр Сергеевич <alexander.ulaev@rtlabs.ru>
To: Sergei Kornilov <sk@zsrv.org>
Cc: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Cc: PG Bug reporting form <noreply@postgresql.org>
Subject: RE: Re:BUG #17392: archiver process exited with exit code 2 was unexpectedly cause for immediate shutdown request
Date: Thu, 3 Feb 2022 10:29:18 +0000
Message-ID: <ca8f25eb70214fc5b320540375877c2a@rtlabs.ru> (raw)
In-Reply-To: <17309001643882771@iva1-b50b8ed859e3.qloud-c.yandex.net>
References: <17392-ae1e272049dfec87@postgresql.org>
<22132351643878322@sas1-941a8864505d.qloud-c.yandex.net>
<779598bff1e7430f9e032d1b62b8006b@rtlabs.ru>
<17309001643882771@iva1-b50b8ed859e3.qloud-c.yandex.net>
We found on ETCD1 such errors in the syslog:
Feb 1 16:12:01 etcd1 etcd: got unexpected response error (etcdserver: request timed out)
Feb 1 16:12:02 etcd1 etcd: got unexpected response error (etcdserver: request timed out) [merged 1 repeated lines in 1.21s]
Feb 1 16:12:03 etcd1 etcd: got unexpected response error (etcdserver: request timed out) [merged 1 repeated lines in 1s]
Feb 1 16:12:20 etcd1 etcd: sync duration of 29.69857369s, expected less than 1s
Feb 1 16:26:55 etcd1 etcd: got unexpected response error (etcdserver: request timed out)
Feb 1 16:27:03 etcd1 etcd: got unexpected response error (etcdserver: request timed out)
Feb 1 16:27:17 etcd1 etcd: sync duration of 1m0.745329542s, expected less than 1s
So, this problem related to SAN => i\o freeze many VMs including DB's, ETCD nodes => etcd i\o long delay affect => patroni reaction to demoted self postgresql instance
Sergey, Thank you much for your support!
Best regards,
Ulaev Alexander
-----Original Message-----
From: Sergei Kornilov [mailto:sk@zsrv.org]
Sent: Thursday, February 3, 2022 1:06 PM
To: Улаев Александр Сергеевич <alexander.ulaev@rtlabs.ru>
Cc: pgsql-bugs@lists.postgresql.org; PG Bug reporting form <noreply@postgresql.org>
Subject: Re:BUG #17392: archiver process exited with exit code 2 was unexpectedly cause for immediate shutdown request
Hello
> 2022-02-01 16:12:24,928 ERROR: failed to update leader lock
> 2022-02-01 16:12:27,063 INFO: demoted self because failed to update leader lock in DCS
Between these two messages, an immediate shutdown is called: https://github.com/zalando/patroni/blob/v2.1.2/patroni/ha.py#L1045
Regards, Sergei
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-bugs@postgresql.org
Cc: alexander.ulaev@rtlabs.ru, sk@zsrv.org, pgsql-bugs@lists.postgresql.org, noreply@postgresql.org
Subject: RE: Re:BUG #17392: archiver process exited with exit code 2 was unexpectedly cause for immediate shutdown request
In-Reply-To: <ca8f25eb70214fc5b320540375877c2a@rtlabs.ru>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox