Date: Thu, 9 Sep 2021 16:04:29 +0200
From: Christoph Berg <christoph.berg@credativ.de>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Subject: Re: trap instead of error on 32 TiB table
Message-ID: <YToUbWq+jvwehMCO@msg.credativ.de>
Mail-Followup-To: Christoph Berg <christoph.berg@credativ.de>,
	Tom Lane <tgl@sss.pgh.pa.us>,
	PostgreSQL Hackers <pgsql-hackers@postgresql.org>
References: <YTn1iTkUYBZfcODk@msg.credativ.de>
 <3568228.1631195095@sss.pgh.pa.us>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <3568228.1631195095@sss.pgh.pa.us>
Archived-At: 
 <https://www.postgresql.org/message-id/YToUbWq%2BjvwehMCO%40msg.credativ.de>
Precedence: bulk

Re: Tom Lane
> Can you provide a stack trace from that?

PG log:

TRAP: FailedAssertion("tagPtr->blockNum != P_NEW", File: "./build/../src/backend/storage/buffer/buf_table.c", Line: 125)
postgres: 13/main: cbe postgres [local] INSERT(ExceptionalCondition+0x7d)[0x558b6223d44d]
postgres: 13/main: cbe postgres [local] INSERT(BufTableInsert+0x89)[0x558b620bafb9]
postgres: 13/main: cbe postgres [local] INSERT(+0x441827)[0x558b620bf827]
postgres: 13/main: cbe postgres [local] INSERT(ReadBufferExtended+0x7a)[0x558b620c021a]
postgres: 13/main: cbe postgres [local] INSERT(RelationGetBufferForTuple+0x250)[0x558b61da7850]
postgres: 13/main: cbe postgres [local] INSERT(heap_insert+0x8b)[0x558b61d965cb]
postgres: 13/main: cbe postgres [local] INSERT(+0x123b89)[0x558b61da1b89]
postgres: 13/main: cbe postgres [local] INSERT(+0x30294c)[0x558b61f8094c]
postgres: 13/main: cbe postgres [local] INSERT(+0x303660)[0x558b61f81660]
postgres: 13/main: cbe postgres [local] INSERT(standard_ExecutorRun+0x115)[0x558b61f4eaa5]
postgres: 13/main: cbe postgres [local] INSERT(+0x47fb72)[0x558b620fdb72]
postgres: 13/main: cbe postgres [local] INSERT(+0x4809be)[0x558b620fe9be]
postgres: 13/main: cbe postgres [local] INSERT(PortalRun+0x1c2)[0x558b620fee72]
postgres: 13/main: cbe postgres [local] INSERT(+0x47c2e0)[0x558b620fa2e0]
postgres: 13/main: cbe postgres [local] INSERT(PostgresMain+0x1a53)[0x558b620fc153]
postgres: 13/main: cbe postgres [local] INSERT(+0x3e7f74)[0x558b62065f74]
postgres: 13/main: cbe postgres [local] INSERT(PostmasterMain+0xd78)[0x558b62066e68]
postgres: 13/main: cbe postgres [local] INSERT(main+0x796)[0x558b61d40356]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x7f57071fad0a]
postgres: 13/main: cbe postgres [local] INSERT(_start+0x2a)[0x558b61d403fa]
2021-09-09 13:19:31.024 CEST [1533530] LOG:  server process (PID 1534108) was terminated by signal 6: Aborted
2021-09-09 13:19:31.024 CEST [1533530] DETAIL:  Failed process was running: insert into huge select generate_series(1,10);

gdb bt:

Core was generated by `postgres: 13/main: cbe postgres [local] INSERT                                '.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50	../sysdeps/unix/sysv/linux/raise.c: Datei oder Verzeichnis nicht gefunden.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f57071f9537 in __GI_abort () at abort.c:79
#2  0x0000558b6223d46e in ExceptionalCondition (conditionName=conditionName@entry=0x558b623b2577 "tagPtr->blockNum != P_NEW",
    errorType=errorType@entry=0x558b6229b016 "FailedAssertion",
    fileName=fileName@entry=0x558b623b2598 "./build/../src/backend/storage/buffer/buf_table.c", lineNumber=lineNumber@entry=125)
    at ./build/../src/backend/utils/error/assert.c:67
#3  0x0000558b620bafb9 in BufTableInsert (tagPtr=tagPtr@entry=0x7ffec8919330, hashcode=hashcode@entry=960067002, buf_id=<optimized out>)
    at ./build/../src/backend/storage/buffer/buf_table.c:125
#4  0x0000558b620bf827 in BufferAlloc (foundPtr=0x7ffec891932b, strategy=0x0, blockNum=4294967295, forkNum=MAIN_FORKNUM,
    relpersistence=112 'p', smgr=0x558b62ed4b38) at ./build/../src/backend/storage/buffer/bufmgr.c:1234
#5  ReadBuffer_common (smgr=0x558b62ed4b38, relpersistence=<optimized out>, forkNum=forkNum@entry=MAIN_FORKNUM,
    blockNum=blockNum@entry=4294967295, mode=mode@entry=RBM_ZERO_AND_LOCK, strategy=0x0, hit=0x7ffec89193d7)
    at ./build/../src/backend/storage/buffer/bufmgr.c:761
#6  0x0000558b620c021a in ReadBufferExtended (reln=0x7f56fb4f8120, forkNum=forkNum@entry=MAIN_FORKNUM,
    blockNum=blockNum@entry=4294967295, mode=mode@entry=RBM_ZERO_AND_LOCK, strategy=strategy@entry=0x0)
    at ./build/../src/backend/storage/buffer/bufmgr.c:677
#7  0x0000558b61da7056 in ReadBufferBI (relation=relation@entry=0x7f56fb4f8120, targetBlock=targetBlock@entry=4294967295,
    mode=mode@entry=RBM_ZERO_AND_LOCK, bistate=bistate@entry=0x0) at ./build/../src/backend/access/heap/hio.c:87
#8  0x0000558b61da7850 in RelationGetBufferForTuple (relation=relation@entry=0x7f56fb4f8120, len=32, otherBuffer=otherBuffer@entry=0,
    options=options@entry=0, bistate=bistate@entry=0x0, vmbuffer=vmbuffer@entry=0x7ffec89194c8, vmbuffer_other=0x0)
    at ./build/../src/backend/access/heap/hio.c:598
#9  0x0000558b61d965cb in heap_insert (relation=relation@entry=0x7f56fb4f8120, tup=tup@entry=0x558b62f0b780, cid=cid@entry=0,
    options=options@entry=0, bistate=bistate@entry=0x0) at ./build/../src/backend/access/heap/heapam.c:1868
#10 0x0000558b61da1b89 in heapam_tuple_insert (relation=0x7f56fb4f8120, slot=0x558b62f0b6b0, cid=0, options=0, bistate=0x0)
    at ./build/../src/backend/access/heap/heapam_handler.c:251
#11 0x0000558b61f8094c in table_tuple_insert (bistate=0x0, options=0, cid=<optimized out>, slot=0x558b62f0b6b0, rel=0x7f56fb4f8120)
    at ./build/../src/include/access/tableam.h:1156
#12 ExecInsert (mtstate=0x558b62f0ab50, slot=0x558b62f0b6b0, planSlot=0x558b62f0b6b0, srcSlot=0x0, returningRelInfo=0x558b62f0aa38,
    estate=0x558b62f0a7c0, canSetTag=true) at ./build/../src/backend/executor/nodeModifyTable.c:642
#13 0x0000558b61f81660 in ExecModifyTable (pstate=0x558b62f0ab50) at ./build/../src/backend/executor/nodeModifyTable.c:2321
#14 0x0000558b61f4eaa5 in ExecProcNode (node=0x558b62f0ab50) at ./build/../src/include/executor/executor.h:248
#15 ExecutePlan (execute_once=<optimized out>, dest=0x558b62e6d908, direction=<optimized out>, numberTuples=0,
    sendTuples=<optimized out>, operation=CMD_INSERT, use_parallel_mode=<optimized out>, planstate=0x558b62f0ab50, estate=0x558b62f0a7c0)
    at ./build/../src/backend/executor/execMain.c:1632
#16 standard_ExecutorRun (queryDesc=0x558b62e68920, direction=<optimized out>, count=0, execute_once=<optimized out>)
    at ./build/../src/backend/executor/execMain.c:350
#17 0x0000558b620fdb72 in ProcessQuery (plan=0x558b62e6d798, sourceText=0x558b62e22000 "insert into huge values ('boom');", params=0x0,
    queryEnv=0x0, dest=0x558b62e6d908, qc=0x7ffec8919a00) at ./build/../src/backend/tcop/pquery.c:160
#18 0x0000558b620fe9be in PortalRunMulti (portal=portal@entry=0x558b62eade60, isTopLevel=isTopLevel@entry=true,
    setHoldSnapshot=setHoldSnapshot@entry=false, dest=dest@entry=0x558b62e6d908, altdest=altdest@entry=0x558b62e6d908,
    qc=qc@entry=0x7ffec8919a00) at ./build/../src/backend/tcop/pquery.c:1263
#19 0x0000558b620fee72 in PortalRun (portal=portal@entry=0x558b62eade60, count=count@entry=9223372036854775807,
    isTopLevel=isTopLevel@entry=true, run_once=run_once@entry=true, dest=dest@entry=0x558b62e6d908,
    altdest=altdest@entry=0x558b62e6d908, qc=0x7ffec8919a00) at ./build/../src/backend/tcop/pquery.c:786
#20 0x0000558b620fa2e0 in exec_simple_query (query_string=0x558b62e22000 "insert into huge values ('boom');")
    at ./build/../src/backend/tcop/postgres.c:1239
#21 0x0000558b620fc153 in PostgresMain (argc=<optimized out>, argv=argv@entry=0x558b62e71f90, dbname=<optimized out>,
    username=<optimized out>) at ./build/../src/backend/tcop/postgres.c:4339
#22 0x0000558b62065f74 in BackendRun (port=0x558b62e6b360) at ./build/../src/backend/postmaster/postmaster.c:4526
#23 BackendStartup (port=0x558b62e6b360) at ./build/../src/backend/postmaster/postmaster.c:4210
#24 ServerLoop () at ./build/../src/backend/postmaster/postmaster.c:1739
#25 0x0000558b62066e68 in PostmasterMain (argc=argc@entry=5, argv=argv@entry=0x558b62e1ae80)
    at ./build/../src/backend/postmaster/postmaster.c:1412
#26 0x0000558b61d40356 in main (argc=5, argv=0x558b62e1ae80) at ./build/../src/backend/main/main.c:210

(Both should be from the same instance.)

> (or else a recipe for reproducing the bug ... I'm not excited
> about reverse-engineering the details of the method)

Create a table, note the relfilenode, shut down PG, and

fn=53332
dd if=/dev/zero bs=8k of=$fn seek=131071 count=1
for i in {1..32766}; do dd if=/dev/zero bs=8k of=$fn.$i seek=131071 count=1; done
dd if=/dev/zero bs=8k of=$fn.32767 seek=131070 count=1
rm ${fn}_fsm

... and then start PG and insert enough to fill the last block.

Christoph
-- 
Senior Consultant, Tel.: +49 2166 9901 187
credativ GmbH, HRB Mönchengladbach 12080, USt-ID-Nummer: DE204566209
Trompeterallee 108, 41189 Mönchengladbach
Geschäftsführung: Dr. Michael Meskes, Sascha Heuer, Geoff Richardson,
Peter Lilley; Unser Umgang mit personenbezogenen Daten unterliegt
folgenden Bestimmungen: https://www.credativ.de/datenschutz