Date: Fri, 25 Oct 2019 17:59:29 +0200
From: Jehan-Guillaume de Rorthais <jgdr@dalibo.com>
To: pgsql-hackers@lists.postgresql.org
Subject: segmentation fault when cassert enabled
Message-ID: <20191025175929.7e90dbf5@firost>
Organization: Dalibo
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="MP_/9N6Uf8HyarwYF6jQy00n++P"
Precedence: bulk

--MP_/9N6Uf8HyarwYF6jQy00n++P
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Hi list,

When investigating for the bug reported in thread "logical replication -
negative bitmapset member not allowed", I found a way to seg fault postgresql
only when cassert is enabled.

See the scenario in attachment.

When executed against binaries compiled with --enable-cassert, I have the
following error in logs:

  LOG:  00000: background worker "logical replication worker" (PID 761) was
  terminated by signal 11: Segmentation fault

Here is the stack trace:

#0  in slot_store_cstrings (slot=0x55a3c6973b48, rel=0x55a3c6989468,
    values=0x7ffe08ae67b0) at worker.c:330
#1  in apply_handle_update (s=0x7ffe08aeddb0) at worker.c:712
#2  in apply_dispatch (s=0x7ffe08aeddb0) at worker.c:968
#3  in LogicalRepApplyLoop (last_received=87957952) at worker.c:1175
#4  in ApplyWorkerMain (main_arg=0) at worker.c:1733
#5  in StartBackgroundWorker () at bgworker.c:834
#6  in do_start_bgworker (rw=0x55a3c68c16d0) at postmaster.c:5763
#7  in maybe_start_bgworkers () at postmaster.c:5976
#8  in sigusr1_handler (postgres_signal_arg=10) at postmaster.c:5161
#9  <signal handler called>
#10 in __GI___select (nfds=6, readfds=0x7ffe08aee680, writefds=0x0,
    exceptfds=0x0, timeout=0x7ffe08aee700)
    at ../sysdeps/unix/sysv/linux/select.c:41
#11 in ServerLoop () at postmaster.c:1668
#12 in PostmasterMain (argc=3, argv=0x55a3c6899820) at postmaster.c:1377
#13 in main (argc=3, argv=0x55a3c6899820) at main.c:228


It leads to this conditional test in worker.c:slot_store_cstrings

    for (i = 0; i < natts; i++)
    { [...]
        if (!att->attisdropped && remoteattnum >= 0 &&
            values[remoteattnum] != NULL)

In gdb, I found remoteattnum seems to be not correctly initialized for the
latest column the scenario adds in pgbench_branches:

  (gdb) p remoteattnum
  $1 = 32639
  (gdb) p i
  $2 = 3

I hadn't time to digg further yet. However, I don't understand why this crash
is triggered when cassert is enabled.

Regards,

--MP_/9N6Uf8HyarwYF6jQy00n++P
Content-Type: application/octet-stream; name=logrep-cassert-crash.bash
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename=logrep-cassert-crash.bash

IyEvdXNyL2Jpbi9iYXNoCgojIGVudgpQR0JJTj0vdG1wL2ZpeC9iaW4KI1BHQklOPS91c3IvbGli
L3Bvc3RncmVzcWwvMTIvYmluClBVQj0vdG1wL3B1YgpTVUI9L3RtcC9zdWIKUFVCX1BPUlQ9NTQz
MgpTVUJfUE9SVD01NDMzClRTPSQoZGF0ZSArJUZUJVQpClBVQl9MT0c9IiRQVUIiLSIkVFMiLmxv
ZwpTVUJfTE9HPSIkU1VCIi0iJFRTIi5sb2cKCnVuc2V0IFBHUE9SVCBQR0hPU1QgUEdEQVRBQkFT
RSBQR0RBVEEgTERfTElCUkFSWV9QQVRICmV4cG9ydCBQR1VTRVI9cG9zdGdyZXMKZXhwb3J0IFBH
T1BUSU9OUz0iLWMgY2xpZW50X21pbl9tZXNzYWdlcz1FUlJPUiIKZXhwb3J0IFBBVEg9IiR7UEdC
SU59OiR7UEFUSH0iCgpmdW5jdGlvbiBsb2cgewogICAgTD0kMTsgc2hpZnQKICAgIGVjaG8gIiQo
ZGF0ZSArJyVGICVUJykgJHtMfTogICRAIgp9CgojIGNsZWFudXAKcGtpbGwgInBnYmVuY2giCgpm
b3IgaSBpbiAiJFBVQiIgIiRTVUIiOyBkbwogICAgdGVzdCAtZCAiJGkiIHx8IGNvbnRpbnVlCiAg
ICB0ZXN0IC1mICIkaSIvcG9zdG1hc3Rlci5waWQgJiYgIiRQR0JJTiIvcGdfY3RsIC13IC1zIC1E
ICIkaSIgLW0gaW1tZWRpYXRlIHN0b3A7CiAgICBsb2cgTE9HICJzdG9wcGVkIG9sZCBpbnN0YW5j
ZSBpbiAkaTogJD8iCiAgICBybSAtciAiJGkiCmRvbmUKCiMgY2x1c3RlcgoiJFBHQklOIi9pbml0
ZGIgLVUgcG9zdGdyZXMgLU4gIiRQVUIiICY+L2Rldi9udWxsOyBsb2cgTE9HICJpbml0IHB1Ymxp
c2hlcjogJD8iCmVjaG8gIndhbF9sZXZlbD1sb2dpY2FsIiA+PiAiJFBVQiIvcG9zdGdyZXNxbC5j
b25mCmVjaG8gImxvZ19taW5fbWVzc2FnZXM9ZGVidWcxIiA+PiAiJFBVQiIvcG9zdGdyZXNxbC5j
b25mCmVjaG8gImxvZ19lcnJvcl92ZXJib3NpdHk9dmVyYm9zZSIgPj4gIiRQVUIiL3Bvc3RncmVz
cWwuY29uZgoKIiRQR0JJTiIvaW5pdGRiIC1VIHBvc3RncmVzIC1OICIkU1VCIiAmPi9kZXYvbnVs
bDsgbG9nIExPRyAiaW5pdCBzdWJzY3JpYmVyOiAkPyIKZWNobyAibG9nX21pbl9tZXNzYWdlcz1k
ZWJ1ZzEiID4+ICIkU1VCIi9wb3N0Z3Jlc3FsLmNvbmYKZWNobyAibG9nX2Vycm9yX3ZlcmJvc2l0
eT12ZXJib3NlIiA+PiAiJFNVQiIvcG9zdGdyZXNxbC5jb25mCmVjaG8gInBvcnQ9JFNVQl9QT1JU
IiA+PiAiJFNVQiIvcG9zdGdyZXNxbC5jb25mCgoiJFBHQklOIi9wZ19jdGwgLXcgLXMgLUQgJFBV
QiAtbCAiJFBVQl9MT0ciIHN0YXJ0OyBsb2cgTE9HICJwdWJsaXNoZWQgc3RhcnRlZDogJD8iCiIk
UEdCSU4iL3BnX2N0bCAtdyAtcyAtRCAkU1VCIC1sICIkU1VCX0xPRyIgc3RhcnQ7IGxvZyBMT0cg
InN1YnNjcmliZXIgc3RhcnRlZDogJD8iCiIkUEdCSU4iL3BnYmVuY2ggLXAgIiRQVUJfUE9SVCIg
LXFpcyA1CiIkUEdCSU4iL3BnX2R1bXAgLXAgIiRQVUJfUE9SVCIgLXMgfCAiJFBHQklOIi9wc3Fs
IC1xWHAgIiRTVUJfUE9SVCIgLW8gL2Rldi9udWxsCgojIGZha2UgYWN0aXZpdHkKIiRQR0JJTiIv
cGdiZW5jaCAtcCAiJFBVQl9QT1JUIiAtVCAzMDAgLWMgMSAmPiAvZGV2L251bGwgJgoKIyByZXBs
aWNhdGlvbiBzZXR1cAoiJFBHQklOIi9wc3FsIC1wICIkUFVCX1BPUlQiIC1YYyAiQ1JFQVRFIFBV
QkxJQ0FUSU9OIHByb3YgRk9SIEFMTCBUQUJMRVMiCiIkUEdCSU4iL3BzcWwgLXAgIiRTVUJfUE9S
VCIgLVhjICJDUkVBVEUgU1VCU0NSSVBUSU9OIHN1YgogICAgICAgICAgICAgICAgICBDT05ORUNU
SU9OICdob3N0PTEyNy4wLjAuMSBwb3J0PTU0MzInCiAgICAgICAgICAgICAgICAgIFBVQkxJQ0FU
SU9OIHByb3YiCgojIHdhaXQgZm9yIHRoZSBzdHJlYW1pbmcKdW5zZXQgVjsKdW50aWwgWyAiJFYi
ID09ICIwIiBdOyBkbwogICAgVj0kKCIkUEdCSU4iL3BzcWwgLXAgIiRTVUJfUE9SVCIgLUF0WGMg
IlNFTEVDVCBjb3VudChzcnN1YnN0YXRlKQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgIEZST00gcGdfc3Vic2NyaXB0aW9uX3JlbAogICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgIFdIRVJFIHNyc3Vic3RhdGUgPD4gJ3InIgogICAgKQog
ICAgbG9nIExPRyAid2FpdGluZyBmb3IgYWxsIHRhYmxlIHRvIGJlIHJlYWR5OiAkViIKICAgIHNs
ZWVwIDEKZG9uZQoKZ3JlcCAtRSAic3Vic2NyaXB0aW9uLipoYXMgZmluaXNoZWQiICIkU1VCX0xP
RyIKCnNsZWVwIDIKCmxvZyBMT0cgImJ1Z2dpbmcgaXQiCiMgdGhpcyBzaG91bGQgdHJpZ2dlciB0
aGUgYnVnCiIkUEdCSU4iL3BzcWwgLWggMTI3LjAuMC4xIC1wICIkU1VCX1BPUlQiIC1YYyBcCiAg
ICAiQUxURVIgVEFCTEUgcGdiZW5jaF9icmFuY2hlcyBBREQgaWQgSU5UIE5PVCBOVUxMIERFRkFV
TFQgMCIKCnNsZWVwIDUKCmdyZXAgLUUgIkVSUk9SfFdBUk5JTkciICIkUFVCX0xPRyIgIiRTVUJf
TE9HIgoK

--MP_/9N6Uf8HyarwYF6jQy00n++P--