From: "Drouvot, Bertrand" <bdrouvot@amazon.com>
Subject: [BUG] Failed Assertion in ReorderBufferChangeMemoryUpdate()
CC: Amit Kapila <amit.kapila16@gmail.com>, <dilipbalaut@gmail.com>
To: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Message-ID: <92b0ee65-b8bd-e42d-c082-4f3f4bf12d34@amazon.com>
Date: Fri, 13 Aug 2021 11:45:37 +0200
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0)
 Gecko/20100101 Thunderbird/78.13.0
MIME-Version: 1.0
Content-Type: multipart/mixed;
	boundary="------------1AF2C091E77C70C47BADFA94"
Content-Language: en-US
Archived-At: 
 <https://www.postgresql.org/message-id/92b0ee65-b8bd-e42d-c082-4f3f4bf12d34%40amazon.com>
Precedence: bulk

--------------1AF2C091E77C70C47BADFA94
Content-Type: multipart/alternative;
	boundary="------------CDC46AE52844C334A3A6E420"

--------------CDC46AE52844C334A3A6E420
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Transfer-Encoding: 8bit

Hi hackers,

While working on [1], it appears that (on master) the issue reproduction 
(of toast rewrite not resetting the toast_hash) is triggering a failed 
assertion:

#2  0x0000000000b29fab in ExceptionalCondition (conditionName=0xce6850
"(rb->size >= sz) && (txn->size >= sz)", errorType=0xce5f84
"FailedAssertion", fileName=0xce5fd0 "reorderbuffer.c", lineNumber=3141)
at assert.c:69
#3  0x00000000008ff1fb in ReorderBufferChangeMemoryUpdate (rb=0x11a7a40,
change=0x11c94b8, addition=false) at reorderbuffer.c:3141
#4  0x00000000008fab27 in ReorderBufferReturnChange (rb=0x11a7a40,
change=0x11c94b8, upd_mem=true) at reorderbuffer.c:477
#5  0x0000000000902ec1 in ReorderBufferToastReset (rb=0x11a7a40,
txn=0x11b1998) at reorderbuffer.c:4799
#6  0x00000000008faaa2 in ReorderBufferReturnTXN (rb=0x11a7a40,
txn=0x11b1998) at reorderbuffer.c:448
#7  0x00000000008fc95b in ReorderBufferCleanupTXN (rb=0x11a7a40,
txn=0x11b1998) at reorderbuffer.c:1540

while on 12.5 for example, we would get (with the exact same repro):

ERROR:  could not open relation with OID 0

The failed assertion is happening in the PG_CATCH() section of 
ReorderBufferProcessTXN().

We entered PG_CATCH() because elog(ERROR, "could not open relation with 
OID %u",...) has been triggered in ReorderBufferToastReplace().

But this elog(ERROR,) is being called after 
ReorderBufferChangeMemoryUpdate() being triggered with "addition" set to 
false.

As a consequence of elog(ERROR,) then ReorderBufferChangeMemoryUpdate() 
with "addition" set to true is not called at the end of 
ReorderBufferToastReplace().

That leads to a subsequent call to ReorderBufferChangeMemoryUpdate() 
(being triggered by 4daa140a2f adding ReorderBufferToastReset() calls to 
ReorderBufferReturnTXN()) triggering the failed assertion.


Please find attached a patch proposal to avoid the failed assertion (by 
ensuring that ReorderBufferChangeMemoryUpdate() being triggered with 
"addition" set to false in ReorderBufferToastReplace() is done after the 
elog(ERROR,)).

Adding Amit and Dilip as they are also aware of [1] and have worked on 
4daa140a2f.

Also adding this patch in the commitfest.

Thanks

Bertrand

[1]: 
https://www.postgresql.org/message-id/b5146fb1-ad9e-7d6e-f980-98ed68744a7c@amazon.com


--------------CDC46AE52844C334A3A6E420
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: 8bit

<html>
  <head>

    <meta http-equiv="content-type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <p>Hi hackers,</p>
    <p>While working on [1], it appears that (on master) the issue
      reproduction (of toast rewrite not resetting the toast_hash) is
      triggering a failed assertion:</p>
    <p><font face="Courier New">#2  0x0000000000b29fab in
        ExceptionalCondition (conditionName=0xce6850 <br>
        "(rb-&gt;size &gt;= sz) &amp;&amp; (txn-&gt;size &gt;= sz)",
        errorType=0xce5f84 <br>
        "FailedAssertion", fileName=0xce5fd0 "reorderbuffer.c",
        lineNumber=3141) <br>
        at assert.c:69<br>
        #3  0x00000000008ff1fb in ReorderBufferChangeMemoryUpdate
        (rb=0x11a7a40, <br>
        change=0x11c94b8, addition=false) at reorderbuffer.c:3141<br>
        #4  0x00000000008fab27 in ReorderBufferReturnChange
        (rb=0x11a7a40, <br>
        change=0x11c94b8, upd_mem=true) at reorderbuffer.c:477<br>
        #5  0x0000000000902ec1 in ReorderBufferToastReset (rb=0x11a7a40,
        <br>
        txn=0x11b1998) at reorderbuffer.c:4799<br>
        #6  0x00000000008faaa2 in ReorderBufferReturnTXN (rb=0x11a7a40,
        <br>
        txn=0x11b1998) at reorderbuffer.c:448<br>
        #7  0x00000000008fc95b in ReorderBufferCleanupTXN (rb=0x11a7a40,
        <br>
        txn=0x11b1998) at reorderbuffer.c:1540</font></p>
    <p>while on 12.5 for example, we would get (with the exact same
      repro):</p>
    <p><font face="Courier New">ERROR:  could not open relation with OID
        0</font></p>
    <p>The failed assertion is happening in the PG_CATCH() section of
      ReorderBufferProcessTXN().</p>
    <p>We entered PG_CATCH() because elog(ERROR, "could not open
      relation with OID %u",...) has been triggered in
      ReorderBufferToastReplace().</p>
    <p>But this elog(ERROR,) is being called after
      ReorderBufferChangeMemoryUpdate() being triggered with "addition"
      set to false. <br>
    </p>
    <p>As a consequence of elog(ERROR,) then
      ReorderBufferChangeMemoryUpdate() with "addition" set to true is
      not called at the end of ReorderBufferToastReplace().</p>
    <p>That leads to a subsequent call to
      ReorderBufferChangeMemoryUpdate() (being triggered by 4daa140a2f
      adding ReorderBufferToastReset() calls to
      ReorderBufferReturnTXN()) triggering the failed assertion.</p>
    <p><br>
    </p>
    <p>Please find attached a patch proposal to avoid the failed
      assertion (by ensuring that ReorderBufferChangeMemoryUpdate()
      being triggered with "addition" set to false in
      ReorderBufferToastReplace() is done after the elog(ERROR,)).</p>
    <p>Adding Amit and Dilip as they are also aware of [1] and have
      worked on 4daa140a2f.</p>
    <p>Also adding this patch in the commitfest. <br>
    </p>
    <p>Thanks</p>
    <p>Bertrand<br>
    </p>
    <p>[1]:
<a class="moz-txt-link-freetext" href="https://www.postgresql.org/message-id/b5146fb1-ad9e-7d6e-f980-98ed68744a7c@amazon.com">https://www.postgresql.org/message-id/b5146fb1-ad9e-7d6e-f980-98ed68744a7c@amazon.com</a><br>
    </p>
  </body>
</html>

--------------CDC46AE52844C334A3A6E420--

--------------1AF2C091E77C70C47BADFA94
Content-Type: text/plain; charset="UTF-8"; x-mac-type=0; x-mac-creator=0;
	name="v1-0001-ReorderBufferChangeMemoryUpdate-failed-assertion.patch"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
	filename*0="v1-0001-ReorderBufferChangeMemoryUpdate-failed-assertion.pat";
	filename*1="ch"

ZGlmZiAtLWdpdCBhL3NyYy9iYWNrZW5kL3JlcGxpY2F0aW9uL2xvZ2ljYWwvcmVvcmRlcmJ1
ZmZlci5jIGIvc3JjL2JhY2tlbmQvcmVwbGljYXRpb24vbG9naWNhbC9yZW9yZGVyYnVmZmVy
LmMKaW5kZXggNzM3OGJlYjY4NC4uMjczMzJhNGNmZCAxMDA2NDQKLS0tIGEvc3JjL2JhY2tl
bmQvcmVwbGljYXRpb24vbG9naWNhbC9yZW9yZGVyYnVmZmVyLmMKKysrIGIvc3JjL2JhY2tl
bmQvcmVwbGljYXRpb24vbG9naWNhbC9yZW9yZGVyYnVmZmVyLmMKQEAgLTQ2MTAsNiArNDYx
MCwxMSBAQCBSZW9yZGVyQnVmZmVyVG9hc3RSZXBsYWNlKFJlb3JkZXJCdWZmZXIgKnJiLCBS
ZW9yZGVyQnVmZmVyVFhOICp0eG4sCiAJaWYgKHR4bi0+dG9hc3RfaGFzaCA9PSBOVUxMKQog
CQlyZXR1cm47CiAKKwl0b2FzdF9yZWwgPSBSZWxhdGlvbklkR2V0UmVsYXRpb24ocmVsYXRp
b24tPnJkX3JlbC0+cmVsdG9hc3RyZWxpZCk7CisJaWYgKCFSZWxhdGlvbklzVmFsaWQodG9h
c3RfcmVsKSkKKwkJZWxvZyhFUlJPUiwgImNvdWxkIG5vdCBvcGVuIHJlbGF0aW9uIHdpdGgg
T0lEICV1IiwKKwkJCSByZWxhdGlvbi0+cmRfcmVsLT5yZWx0b2FzdHJlbGlkKTsKKwogCS8q
CiAJICogV2UncmUgZ29pbmcgdG8gbW9kaWZ5IHRoZSBzaXplIG9mIHRoZSBjaGFuZ2UsIHNv
IHRvIG1ha2Ugc3VyZSB0aGUKIAkgKiBhY2NvdW50aW5nIGlzIGNvcnJlY3Qgd2UnbGwgbWFr
ZSBpdCBsb29rIGxpa2Ugd2UncmUgcmVtb3ZpbmcgdGhlIGNoYW5nZQpAQCAtNDYyNCwxMSAr
NDYyOSw2IEBAIFJlb3JkZXJCdWZmZXJUb2FzdFJlcGxhY2UoUmVvcmRlckJ1ZmZlciAqcmIs
IFJlb3JkZXJCdWZmZXJUWE4gKnR4biwKIAogCWRlc2MgPSBSZWxhdGlvbkdldERlc2NyKHJl
bGF0aW9uKTsKIAotCXRvYXN0X3JlbCA9IFJlbGF0aW9uSWRHZXRSZWxhdGlvbihyZWxhdGlv
bi0+cmRfcmVsLT5yZWx0b2FzdHJlbGlkKTsKLQlpZiAoIVJlbGF0aW9uSXNWYWxpZCh0b2Fz
dF9yZWwpKQotCQllbG9nKEVSUk9SLCAiY291bGQgbm90IG9wZW4gcmVsYXRpb24gd2l0aCBP
SUQgJXUiLAotCQkJIHJlbGF0aW9uLT5yZF9yZWwtPnJlbHRvYXN0cmVsaWQpOwotCiAJdG9h
c3RfZGVzYyA9IFJlbGF0aW9uR2V0RGVzY3IodG9hc3RfcmVsKTsKIAogCS8qIHNob3VsZCB3
ZSBhbGxvY2F0ZSBmcm9tIHN0YWNrIGluc3RlYWQ/ICovCg==
--------------1AF2C091E77C70C47BADFA94--