Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kIuHh-0007km-AO for pgsql-hackers@arkaria.postgresql.org; Thu, 17 Sep 2020 13:49:13 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1kIuHf-0000Zm-TD for pgsql-hackers@arkaria.postgresql.org; Thu, 17 Sep 2020 13:49:11 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kIuHf-0000Ze-M9 for pgsql-hackers@lists.postgresql.org; Thu, 17 Sep 2020 13:49:11 +0000 Received: from oss.nttdata.com ([49.212.34.109]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kIuHc-0003IF-P8 for pgsql-hackers@postgresql.org; Thu, 17 Sep 2020 13:49:11 +0000 Received: from hnk.local (p2368131-ipbf2226funabasi.chiba.ocn.ne.jp [114.177.250.131]) by oss.nttdata.com (Postfix) with ESMTPSA id 00B7460560 for ; Thu, 17 Sep 2020 22:49:03 +0900 (JST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.102.3 at oss.nttdata.com To: pgsql-hackers@postgresql.org From: Fujii Masao Subject: Delay of standby shutdown Message-ID: <9d7e6ab0-8a53-ddb9-63cd-289bcb25fe0e@oss.nttdata.com> Date: Thu, 17 Sep 2020 22:49:03 +0900 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Thunderbird/68.12.0 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="------------5FE95CAEA85B10F6A23DA8D9" Content-Language: en-US List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk This is a multi-part message in MIME format. --------------5FE95CAEA85B10F6A23DA8D9 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Hi, When I shut down the standby server just after setting up replication environment and shutting down the primary, I found that it took five seconds to complete the shutdown of the standby server. Note that no WAL was generated between the setup of replication and the server shutdown, in this case. That is, basically neither WAL replay nor shutdown restart point didn't cause this issue, I think. Here are the log messages in that standby. I have observed this issue some times so far. 2020-09-17 21:29:54 JST [postmaster] sby1 LOG: received fast shutdown request 2020-09-17 21:29:54 JST [postmaster] sby1 LOG: aborting any active transactions 2020-09-17 21:29:59 JST [checkpointer] sby1 LOG: shutting down 2020-09-17 21:29:59 JST [checkpointer] sby1 LOG: restartpoint starting: shutdown immediate 2020-09-17 21:29:59 JST [checkpointer] sby1 LOG: restartpoint complete: wrote 0 buffers (0.0%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.014 s, sync=0.000 s, total=0.022 s; sync files=0, longest=0.000 s, average=0.000 s; distance=16384 kB, estimate=16384 kB 2020-09-17 21:29:59 JST [checkpointer] sby1 LOG: recovery restart point at 0/5000060 2020-09-17 21:29:59 JST [postmaster] sby1 LOG: database system is shut down Per my analysis, this issue could happen in the following way. 1. In WaitForWALToBecomeAvailable(), the startup proesss receives the shutdown request during sleeping wal_retrieve_retry_interval. 2. In ReadRecord(), the startup process thinks no valid record available and loops back to retry reading WAL from stream. 3. In WaitForWALToBecomeAvailable(), the startup proesss tries to start up walreceiver and then waits for recoveryWakeupLatch. 4. After the timeout (5 seconds is used for that wait for recoveryWakeupLatch) passes, HandleStartupProcInterrupts() is called and the shutdown request is processed. So waiting at #3 for recoveryWakeupLatch caused the delay of the standby shutdown. To avoid this issue, I think that ReadRecord() should call HandleStartupProcInterrupts() whenever looping back to retry. Patch attached. Thought? Regards, -- Fujii Masao Advanced Computing Technology Center Research and Development Headquarters NTT DATA CORPORATION --------------5FE95CAEA85B10F6A23DA8D9 Content-Type: text/plain; charset=UTF-8; x-mac-type="0"; x-mac-creator="0"; name="shutdown_in_standby_v1.patch" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="shutdown_in_standby_v1.patch" ZGlmZiAtLWdpdCBhL3NyYy9iYWNrZW5kL2FjY2Vzcy90cmFuc2FtL3hsb2cuYyBiL3NyYy9i YWNrZW5kL2FjY2Vzcy90cmFuc2FtL3hsb2cuYwppbmRleCA2MTc1NDMxMmUyLi5mYWY4ZDlj NGYwIDEwMDY0NAotLS0gYS9zcmMvYmFja2VuZC9hY2Nlc3MvdHJhbnNhbS94bG9nLmMKKysr IGIvc3JjL2JhY2tlbmQvYWNjZXNzL3RyYW5zYW0veGxvZy5jCkBAIC00NDU2LDE1ICs0NDU2 LDE2IEBAIFJlYWRSZWNvcmQoWExvZ1JlYWRlclN0YXRlICp4bG9ncmVhZGVyLCBpbnQgZW1v ZGUsCiAJCQkJICovCiAJCQkJbGFzdFNvdXJjZUZhaWxlZCA9IGZhbHNlOwogCQkJCWN1cnJl bnRTb3VyY2UgPSBYTE9HX0ZST01fQU5ZOwotCi0JCQkJY29udGludWU7CiAJCQl9Ci0KIAkJ CS8qIEluIHN0YW5kYnkgbW9kZSwgbG9vcCBiYWNrIHRvIHJldHJ5LiBPdGhlcndpc2UsIGdp dmUgdXAuICovCi0JCQlpZiAoU3RhbmRieU1vZGUgJiYgIUNoZWNrRm9yU3RhbmRieVRyaWdn ZXIoKSkKLQkJCQljb250aW51ZTsKLQkJCWVsc2UKKwkJCWVsc2UgaWYgKCFTdGFuZGJ5TW9k ZSB8fCBDaGVja0ZvclN0YW5kYnlUcmlnZ2VyKCkpCiAJCQkJcmV0dXJuIE5VTEw7CisKKwkJ CS8qCisJCQkgKiBUaGlzIHBvc3NpYmx5LWxvbmcgbG9vcCBuZWVkcyB0byBoYW5kbGUgaW50 ZXJydXB0cyBvZiBzdGFydHVwCisJCQkgKiBwcm9jZXNzLgorCQkJICovCisJCQlIYW5kbGVT dGFydHVwUHJvY0ludGVycnVwdHMoKTsKIAkJfQogCX0KIH0K --------------5FE95CAEA85B10F6A23DA8D9--