Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1ibpp0-0001dZ-FB for pgsql-bugs@arkaria.postgresql.org; Mon, 02 Dec 2019 17:49:19 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1ibpoy-0007bg-RS for pgsql-bugs@arkaria.postgresql.org; Mon, 02 Dec 2019 17:49:16 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1ibpoy-0007bZ-Em for pgsql-bugs@lists.postgresql.org; Mon, 02 Dec 2019 17:49:16 +0000 Received: from cyclops.postgrespro.ru ([93.174.131.138] helo=mail.postgrespro.ru) by magus.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1ibpou-0002vU-J9 for pgsql-bugs@postgresql.org; Mon, 02 Dec 2019 17:49:15 +0000 Received: from localhost (localhost [127.0.0.1]) by mail.postgrespro.ru (Postfix) with ESMTP id 67EFE21C4746; Mon, 2 Dec 2019 20:49:10 +0300 (MSK) X-Virus-Scanned: Debian amavisd-new at postgrespro.ru X-Spam-Flag: NO X-Spam-Score: 0 X-Spam-Level: X-Spam-Status: No, score=x tagged_above=-99 required=4 WHITELISTED tests=[] autolearn=unavailable Received: from [192.168.27.174] (gw.postgrespro.ru [93.174.131.141]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mail.postgrespro.ru (Postfix) with ESMTPSA id 1669E21C4724; Mon, 2 Dec 2019 20:49:09 +0300 (MSK) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=postgrespro.ru; s=mail; t=1575308950; bh=Tn9pRctwvBEovv3QvZdYHDMK6oludbOuWzdcB271E/I=; h=Subject:To:Cc:References:From:Date:In-Reply-To; b=N/UN76RWwXcM71E93Me76J1uewqgTN/jROXS/4jFXHDYRJicXpXwLizT/AsWmQK0T ZCYZ8dpVGY9sg8d1efCd1UrASWxpOzgYKPFSU3osUk6p9Y7j+le5zHHba1BywBNUNv JXzEl8E13NBVhM8BwV0wq7/wwzskHRKn78/79yqc= Subject: Re: logical replication: could not create file "state.tmp": File exists To: Andres Freund Cc: Pg Bugs References: <08bbfab1-a61d-3750-fc18-4ab2c1aa7f09@postgrespro.ru> <20191202161222.sazl2omhhk5pl3nl@alap3.anarazel.de> From: Grigory Smolkin Message-ID: <14b3454f-2d68-c637-68e4-2b42ff976168@postgrespro.ru> Date: Mon, 2 Dec 2019 20:49:04 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: <20191202161222.sazl2omhhk5pl3nl@alap3.anarazel.de> Content-Type: multipart/mixed; boundary="------------5A1AF2E8908CDB5C79C7C1AC" Content-Language: en-US List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk This is a multi-part message in MIME format. --------------5A1AF2E8908CDB5C79C7C1AC Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit On 12/2/19 7:12 PM, Andres Freund wrote: > Hi, > > On 2019-11-30 15:09:39 +0300, Grigory Smolkin wrote: >> One of my colleagues encountered an out of space condition, which broke his >> logical replication setup. >> It`s manifested with the following errors: >> >> ERROR:  could not receive data from WAL stream: ERROR:  could not create >> file "pg_replslot/some_sub/state.tmp": File exists > Hm. What was the log output leading to this state? Some cases of this > would end up in a PANIC, which'd remove the .tmp file during > recovery. But there's some where we won't - it seems the right fix for > this would be to unlink the tmp file in that case? > > >> I`ve digged a bit into this problem, and it`s turned out that in >> SaveSlotToPath() temp file for replication slot is opened with 'O_CREAT | >> O_EXCL' flags, which makes this routine as not very reentrant. >> >> Since an exclusive lock is taken before temp file creation, I think it >> should be safe to replace O_EXCL with O_TRUNC. > I'm very doubtful about this. I think it's a good safety measure to > ensure that there's no previous state file that we're somehow > overwriting. > > >> Script to reproduce and patch are attached. > Well: > >> # Imitate out_of_space/write_operation_error >> touch ${PGDATA_PUB}/pg_replslot/mysub/state.tmp > Doesn't really replicate how we got into this state... I`ve managed to reproduce the issue using the attached script: 2019-12-02 20:32:12.547 MSK [4180] ERROR:  could not write to file "pg_logical/snapshots/0-160E9A28.snap.4180.tmp": No space left on device 2019-12-02 20:32:12.548 MSK [4178] ERROR:  could not receive data from WAL stream: ERROR:  could not write to file "pg_logical/snapshots/0-160E9A28.snap.4180.tmp": No space left on device 2019-12-02 20:32:12.557 MSK [4645] LOG:  logical replication apply worker for subscription "mysub" has started 2019-12-02 20:32:12.562 MSK [4091] LOG:  background worker "logical replication worker" (PID 4178) exited with exit code 1 2019-12-02 20:32:12.569 MSK [4648] LOG:  starting logical decoding for slot "mysub" 2019-12-02 20:32:12.569 MSK [4648] DETAIL:  Streaming transactions committing after 0/160E9A28, reading WAL from 0/1566A2D8. 2019-12-02 20:32:12.569 MSK [4648] LOG:  logical decoding found consistent point at 0/1566A2D8 2019-12-02 20:32:12.569 MSK [4648] DETAIL:  Logical decoding will begin using saved snapshot. 2019-12-02 20:32:12.672 MSK [4648] ERROR:  could not write to file "pg_replslot/mysub/state.tmp": No space left on device 2019-12-02 20:32:12.672 MSK [4645] ERROR:  could not receive data from WAL stream: ERROR:  could not write to file "pg_replslot/mysub/state.tmp": No space left on device 2019-12-02 20:32:12.672 MSK [4091] LOG:  background worker "logical replication worker" (PID 4645) exited with exit code 1 2019-12-02 20:32:17.684 MSK [4653] LOG:  logical replication apply worker for subscription "mysub" has started 2019-12-02 20:32:17.693 MSK [4654] LOG:  starting logical decoding for slot "mysub" 2019-12-02 20:32:17.693 MSK [4654] DETAIL:  Streaming transactions committing after 0/160E9A28, reading WAL from 0/1566A2D8. 2019-12-02 20:32:17.693 MSK [4654] LOG:  logical decoding found consistent point at 0/1566A2D8 2019-12-02 20:32:17.693 MSK [4654] DETAIL:  Logical decoding will begin using saved snapshot. 2019-12-02 20:32:17.847 MSK [4654] ERROR:  could not write to file "pg_logical/snapshots/0-160E9A28.snap.4654.tmp": No space left on device 2019-12-02 20:32:17.847 MSK [4653] ERROR:  could not receive data from WAL stream: ERROR:  could not write to file "pg_logical/snapshots/0-160E9A28.snap.4654.tmp": No space left on device 2019-12-02 20:32:17.848 MSK [4091] LOG:  background worker "logical replication worker" (PID 4653) exited with exit code 1 2019-12-02 20:32:22.859 MSK [4658] LOG:  logical replication apply worker for subscription "mysub" has started 2019-12-02 20:32:22.868 MSK [4659] LOG:  starting logical decoding for slot "mysub" 2019-12-02 20:32:22.868 MSK [4659] DETAIL:  Streaming transactions committing after 0/160E9A28, reading WAL from 0/1566A2D8. 2019-12-02 20:32:22.868 MSK [4659] LOG:  logical decoding found consistent point at 0/1566A2D8 2019-12-02 20:32:22.868 MSK [4659] DETAIL:  Logical decoding will begin using saved snapshot. 2019-12-02 20:32:22.868 MSK [4659] ERROR:  could not create file "pg_replslot/mysub/state.tmp": File exists 2019-12-02 20:32:22.869 MSK [4658] ERROR:  could not receive data from WAL stream: ERROR:  could not create file "pg_replslot/mysub/state.tmp": File exists > > Greetings, > > Andres Freund -- Grigory Smolkin Postgres Professional: http://www.postgrespro.com The Russian Postgres Company --------------5A1AF2E8908CDB5C79C7C1AC Content-Type: application/x-shellscript; name="replication_bug.sh" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="replication_bug.sh" IyEvYmluL2Jhc2gKCnNldCAtZXh1CnNldCAtbyBlcnJleGl0CgpQUkVGSVg9L2hvbWUvZ3Nt b2wvdGFzay8xM19kZXZlbApQR0RBVEFfUFVCPS90bXAvMjAwTUIvZGF0YV9wdWIgIyBjcmVh dGVkIGFzOiBta2RpciAtcCAvdG1wLzIwME1CLyAmJiBzdWRvIG1vdW50IC10IHRtcGZzIC1v IHNpemU9MjAwbSB0bXBmcyAvdG1wLzIwME1CClBVQl9XQUxfRElSPS90bXAvd2FsClBHREFU QV9TVUI9JHtQUkVGSVh9L2RhdGFfc3ViClBPUlRfUFVCPTE1NDMxClBPUlRfU1VCPTE1NDMw Cgoke1BSRUZJWH0vYmluL3BnX2N0bCBzdG9wIC1tIGltbWVkaWF0ZSAtRCAke1BHREFUQV9Q VUJ9IHx8IGVjaG8gImFsbCBpcyB3ZWxsIgoke1BSRUZJWH0vYmluL3BnX2N0bCBzdG9wIC1t IGltbWVkaWF0ZSAtRCAke1BHREFUQV9TVUJ9IHx8IGVjaG8gImFsbCBpcyB3ZWxsIgpybSAt cmYgJHtQR0RBVEFfUFVCfSAke1BHREFUQV9TVUJ9ICR7UFVCX1dBTF9ESVJ9Cgoke1BSRUZJ WH0vYmluL2luaXRkYiAtayAtRCAke1BHREFUQV9QVUJ9IC1YICR7UFVCX1dBTF9ESVJ9CmVj aG8gInBvcnQgPSAke1BPUlRfUFVCfSIgPj4gJHtQR0RBVEFfUFVCfS9wb3N0Z3Jlc3FsLmF1 dG8uY29uZgplY2hvICJ3YWxfbGV2ZWwgPSBsb2dpY2FsIiA+PiAke1BHREFUQV9QVUJ9L3Bv c3RncmVzcWwuYXV0by5jb25mCmVjaG8gImF1dG92YWN1dW0gPSBvZmYiID4+ICR7UEdEQVRB X1BVQn0vcG9zdGdyZXNxbC5hdXRvLmNvbmYKCiR7UFJFRklYfS9iaW4vaW5pdGRiIC1rIC1E ICR7UEdEQVRBX1NVQn0KZWNobyAicG9ydCA9ICR7UE9SVF9TVUJ9IiA+PiAke1BHREFUQV9T VUJ9L3Bvc3RncmVzcWwuYXV0by5jb25mCgojIHN0YXJ0IHB1Ymxpc2hlciBjbHVzdGVyCiR7 UFJFRklYfS9iaW4vcGdfY3RsIHN0YXJ0IC1EICR7UEdEQVRBX1BVQn0KCiMgc3RhcnQgc3Vi c2NyaWJlciBjbHVzdGVyCiR7UFJFRklYfS9iaW4vcGdfY3RsIHN0YXJ0IC1EICR7UEdEQVRB X1NVQn0KCiMgZ2VuZXJhdGUgc29tZSBkYXRhCiR7UFJFRklYfS9iaW4vcGdiZW5jaCAtaSAt cyA5IC1wICR7UE9SVF9QVUJ9IHBvc3RncmVzCgojIFBVQkxJQ0FUSU9OCiR7UFJFRklYfS9i aW4vcHNxbCAtcCAke1BPUlRfUFVCfSAtZCBwb3N0Z3JlcyAtYyAiQ1JFQVRFIFBVQkxJQ0FU SU9OIG15cHViIEZPUiB0YWJsZSBwZ2JlbmNoX2FjY291bnRzIgoKIyBTVUJTQ1JJUFRJT04K JHtQUkVGSVh9L2Jpbi9wc3FsIC1wICR7UE9SVF9TVUJ9IC1kIHBvc3RncmVzIC1jICJDUkVB VEUgVEFCTEUgcGdiZW5jaF9hY2NvdW50cyhhaWQgaW50IG5vdCBudWxsIFBSSU1BUlkgS0VZ LCBiaWQgaW50LGFiYWxhbmNlIGludCxmaWxsZXIgY2hhcig4NCkpIgoke1BSRUZJWH0vYmlu L3BzcWwgLXAgJHtQT1JUX1NVQn0gLWQgcG9zdGdyZXMgLWMgIkNSRUFURSBTVUJTQ1JJUFRJ T04gbXlzdWIgQ09OTkVDVElPTiAnaG9zdD1sb2NhbGhvc3QgcG9ydD0ke1BPUlRfUFVCfSBk Ym5hbWU9cG9zdGdyZXMnIFBVQkxJQ0FUSU9OIG15cHViIgoKIyBlbXVsYXRlIG91dCBvZiBz cGFjZQpta2RpciAke1BHREFUQV9QVUJ9L3NvbWVkaXIKY3AgLWF2ICR7UFVCX1dBTF9ESVJ9 ICR7UEdEQVRBX1BVQn0vc29tZWRpciB8fCBlY2hvICJ3ZSBzaG91bGQgZ2V0IGVtcHR5IHN0 YXRlLnRtcCBcCmF0IHRoZSBuZXh0IGxhdW5jaCBvZiBzeW5jaHJvbml6YXRpb24gd29ya2Vy IGZyb20gcmVwbGljYSIKCg== --------------5A1AF2E8908CDB5C79C7C1AC--