pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedConnections hang indefinitely while taking a gin index's LWLock buffer_content lock
67+ messages / 11 participants
[nested] [flat]
Commits from this thread:
2019-11-19 e14641197a 051c50c01c 9f29279899 21ad61ab31 Fix deadlock between ginDeletePage() and ginStepRight()
2019-11-19 99f5888d35 4fc4856849 1414821e16 d5ad7a09af ee437ca740 c0bf354215 ab64b474d9 Fix traversing to the deleted GIN page via downlink
2018-12-13 bf0e5a73be Fix wrong backpatching of ginRedoDeletePage() deadlock fix
2018-12-13 dd951dc34a a0696d2955 cd24b4eaec 1cf175c74f ad6ebcfcbb 52ac6cd2d0 Prevent GIN deleted pages from being reclaimed too early
2018-12-13 2e3bd064e5 fd83c83d09 Fix deadlock in GIN vacuum introduced by 218f51584d5
2018-12-13 80d4d8d71d 865870374a f6c44e1b55 19cf52e6cc c6ade7a8cd Prevent deadlock in ginRedoDeletePage()
* Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-10-29 19:04 chenhj <chjischj@163.com>
0 siblings, 3 replies; 67+ messages in thread
From: chenhj @ 2018-10-29 19:04 UTC (permalink / raw)
To: pgsql-hackers@lists.postgresql.org
Hi,all
In our PostgreSQL 10.2 database, two sessions of insert and autovacuum of gin index blocked in LWLock:buffer_content.
This blocked checkpoint and dead tuple recycle,and we had to restart the datebase.
According to the information collected from gcore, a deadlock occurred when acquiring the buffer lock.
This system is a citus 7.2.1 cluster. It has been running for more than a year. This problem has not happened before.
10/19 the cluster has been expanded from 8 workers to 16 workers. I don't know if it has anything to do with expansion.
The problem happend on the coordinator node of citus, but the table is an normal table(not partition table).
And the load of this coordinator node has always been heavy.
After 10/25 this issue occurred three times in the 4 days, but we can not reproduce this problem in test environment.
3 faults are stuck on 3 different tables, but the stack is the same.
## stack of insert：Acquire lock 0x2aaab4009564 and hold 0x2aaab670dfe4, 0x2aaac587ae64
----------------------------------------------------------------------------
(gdb) bt
#0 0x00007fe11552379b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1 0x00007fe11552382f in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2 0x00007fe1155238cb in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3 0x000000000069d362 in PGSemaphoreLock (sema=0x2aaaaac02958) at pg_sema.c:310
#4 0x00000000007095ac in LWLockAcquire (lock=0x2aaab4009564, mode=LW_EXCLUSIVE) at lwlock.c:1233
#5 0x0000000000490a32 in ginStepRight (buffer=6713826, index=<optimized out>, lockmode=lockmode@entry=2) at ginbtree.c:174
#6 0x0000000000490c1c in ginFinishSplit (btree=btree@entry=0x7ffd81e4f950, stack=0x28eebf8, freestack=freestack@entry=1 '\001', buildStats=buildStats@entry=0x0) at ginbtree.c:701
#7 0x0000000000491396 in ginInsertValue (btree=btree@entry=0x7ffd81e4f950, stack=<optimized out>, insertdata=insertdata@entry=0x7ffd81e4f940, buildStats=buildStats@entry=0x0)
at ginbtree.c:773
#8 0x000000000048fb42 in ginInsertItemPointers (index=<optimized out>, rootBlkno=rootBlkno@entry=644, items=items@entry=0x2916598, nitem=nitem@entry=353, buildStats=buildStats@entry=0x0)
at gindatapage.c:1907
#9 0x000000000048a9ea in ginEntryInsert (ginstate=ginstate@entry=0x28e7ef8, attnum=<optimized out>, key=42920440, category=<optimized out>, items=0x2916598, nitem=353,
buildStats=buildStats@entry=0x0) at gininsert.c:214
#10 0x0000000000496d94 in ginInsertCleanup (ginstate=ginstate@entry=0x28e7ef8, full_clean=full_clean@entry=0 '\000', fill_fsm=fill_fsm@entry=1 '\001',
forceCleanup=forceCleanup@entry=0 '\000', stats=stats@entry=0x0) at ginfast.c:883
#11 0x0000000000497727 in ginHeapTupleFastInsert (ginstate=ginstate@entry=0x28e7ef8, collector=collector@entry=0x7ffd81e4fe60) at ginfast.c:448
#12 0x000000000048b209 in gininsert (index=<optimized out>, values=0x7ffd81e4ff40, isnull=0x7ffd81e50040 "", ht_ctid=0x280d98c, heapRel=<optimized out>, checkUnique=<optimized out>,
indexInfo=0x28b5aa8) at gininsert.c:522
#13 0x00000000005ee8dd in ExecInsertIndexTuples (slot=slot@entry=0x28b5d58, tupleid=tupleid@entry=0x280d98c, estate=estate@entry=0x28b5288, noDupErr=noDupErr@entry=1 '\001',
specConflict=specConflict@entry=0x7ffd81e5013b "", arbiterIndexes=arbiterIndexes@entry=0x28c6dd8) at execIndexing.c:386
#14 0x000000000060ccf5 in ExecInsert (canSetTag=1 '\001', estate=0x28b5288, onconflict=ONCONFLICT_UPDATE, arbiterIndexes=0x28c6dd8, planSlot=0x28b5d58, slot=0x28b5d58, mtstate=0x28b5628)
at nodeModifyTable.c:564
#15 ExecModifyTable (pstate=0x28b5628) at nodeModifyTable.c:1766
#16 0x00000000005ef612 in standard_ExecutorRun (queryDesc=<optimized out>, direction=<optimized out>, count=<optimized out>, execute_once=<optimized out>)
at ../../../src/include/executor/executor.h:250
#17 0x00007fe10607d15d in pgss_ExecutorRun (queryDesc=0x28cc1d8, direction=ForwardScanDirection, count=0, execute_once=<optimized out>) at pg_stat_statements.c:889
#18 0x000000000071a7ba in ProcessQuery (plan=<optimized out>,
sourceText=0x2849058 "INSERT INTO bi_dm.tdm_sncity_workorder_analytic_statistics (CITYCODE,CITYNAME,DISTRICT,DISTRICT_NAME,ZL_WERKS,ZL_WERKS_NAME,AREA_CD,AREA_NM,ZSIZE,DEPT_CD,TMALL_FLG,ORDER_DATE,DISTRIBUTE,WORKORDER,FNSH"..., params=0x28b2260, queryEnv=0x0, dest=0xc9e2a0 <donothingDR>, completionTag=0x7ffd81e507b0 "") at pquery.c:161
#19 0x000000000071a9f7 in PortalRunMulti (portal=portal@entry=0x289fdd8, isTopLevel=isTopLevel@entry=1 '\001', setHoldSnapshot=setHoldSnapshot@entry=0 '\000', dest=0xc9e2a0 <donothingDR>,
dest@entry=0x284d778, altdest=0xc9e2a0 <donothingDR>, altdest@entry=0x284d778, completionTag=completionTag@entry=0x7ffd81e507b0 "") at pquery.c:1286
#20 0x000000000071b535 in PortalRun (portal=<optimized out>, count=1, isTopLevel=<optimized out>, run_once=<optimized out>, dest=0x284d778, altdest=0x284d778,
completionTag=0x7ffd81e507b0 "") at pquery.c:799
#21 0x0000000000718f84 in PostgresMain (argc=<optimized out>, argv=<optimized out>, dbname=<optimized out>, username=<optimized out>) at postgres.c:1984
#22 0x000000000047ad3c in BackendRun (port=0x27cbc40) at postmaster.c:4405
#23 BackendStartup (port=0x27cbc40) at postmaster.c:4077
#24 ServerLoop () at postmaster.c:1755
#25 0x00000000006afacf in PostmasterMain (argc=argc@entry=3, argv=argv@entry=0x277ba20) at postmaster.c:1363
#26 0x000000000047bb6f in main (argc=3, argv=0x277ba20) at main.c:228
(gdb) f 4
#4 0x00000000007095ac in LWLockAcquire (lock=0x2aaab4009564, mode=LW_EXCLUSIVE) at lwlock.c:1233
1233PGSemaphoreLock(proc->sem);
(gdb) p num_held_lwlocks
$16 = 2
(gdb) p InterruptHoldoffCount
$17 = 3
(gdb) p held_lwlocks
$18 = {{lock = 0x2aaab670dfe4, mode = LW_EXCLUSIVE}, {lock = 0x2aaac587ae64, mode = LW_EXCLUSIVE}, {lock = 0x2aaaaac0a380, mode = LW_SHARED}, {lock = 0x2aaab670a2e4, mode = LW_EXCLUSIVE}, {
lock = 0x2aaab670a2e4, mode = LW_EXCLUSIVE}, {lock = 0x2aaab670a2e4, mode = LW_EXCLUSIVE}, {lock = 0x2aaab670a2e4, mode = LW_EXCLUSIVE}, {lock = 0x2aaab670a2e4, mode = LW_EXCLUSIVE}, {
lock = 0x2aaab670a2e4, mode = LW_EXCLUSIVE}, {lock = 0x2aaab670a2e4, mode = LW_EXCLUSIVE}, {lock = 0x2aaaaac15a80, mode = LW_EXCLUSIVE}, {lock = 0x0,
mode = LW_EXCLUSIVE} <repeats 189 times>}
## stack of autovacuum:Acquire lock 0x2aaab670dfe4 and hold 0x2aaab4009564
--------------------------------------
(gdb) bt
#0 0x00007fe11552379b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1 0x00007fe11552382f in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2 0x00007fe1155238cb in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3 0x000000000069d362 in PGSemaphoreLock (sema=0x2aaaaac07eb8) at pg_sema.c:310
#4 0x00000000007095ac in LWLockAcquire (lock=0x2aaab670dfe4, mode=LW_EXCLUSIVE) at lwlock.c:1233
#5 0x00000000004947ef in ginScanToDelete (gvs=gvs@entry=0x7ffd81e4d0f0, blkno=701, isRoot=isRoot@entry=0 '\000', parent=parent@entry=0x7ffd81e4c790, myoff=myoff@entry=37) at ginvacuum.c:262
#6 0x0000000000494874 in ginScanToDelete (gvs=gvs@entry=0x7ffd81e4d0f0, blkno=blkno@entry=9954, isRoot=isRoot@entry=1 '\001', parent=parent@entry=0x7ffd81e4c790, myoff=myoff@entry=0)
at ginvacuum.c:277
#7 0x0000000000494ed1 in ginVacuumPostingTreeLeaves (gvs=gvs@entry=0x7ffd81e4d0f0, blkno=9954, isRoot=isRoot@entry=0 '\000') at ginvacuum.c:404
#8 0x0000000000494e21 in ginVacuumPostingTreeLeaves (gvs=gvs@entry=0x7ffd81e4d0f0, blkno=644, isRoot=isRoot@entry=1 '\001') at ginvacuum.c:372
#9 0x0000000000495540 in ginVacuumPostingTree (rootBlkno=<optimized out>, gvs=0x7ffd81e4d0f0) at ginvacuum.c:426
#10 ginbulkdelete (info=0x7ffd81e4f720, stats=<optimized out>, callback=<optimized out>, callback_state=<optimized out>) at ginvacuum.c:649
#11 0x00000000005e1194 in lazy_vacuum_index (indrel=0x3146e28, stats=stats@entry=0x28ec200, vacrelstats=vacrelstats@entry=0x28ebc28) at vacuumlazy.c:1621
#12 0x00000000005e214d in lazy_scan_heap (aggressive=<optimized out>, nindexes=<optimized out>, Irel=<optimized out>, vacrelstats=<optimized out>, options=16, onerel=0x28ec1f8)
at vacuumlazy.c:1311
#13 lazy_vacuum_rel (onerel=onerel@entry=0x3144fa8, options=options@entry=99, params=params@entry=0x289f270, bstrategy=<optimized out>) at vacuumlazy.c:258
#14 0x00000000005dfc35 in vacuum_rel (relid=relid@entry=2060396948, relation=relation@entry=0x7ffd81e4fe50, options=options@entry=99, params=params@entry=0x289f270) at vacuum.c:1463
#15 0x00000000005e0ad2 in vacuum (options=99, relation=relation@entry=0x7ffd81e4fe50, relid=<optimized out>, params=params@entry=0x289f270, va_cols=va_cols@entry=0x0,
bstrategy=<optimized out>, bstrategy@entry=0x289e988, isTopLevel=isTopLevel@entry=1 '\001') at vacuum.c:306
#16 0x00000000006a01ff in autovacuum_do_vac_analyze (bstrategy=0x289e988, tab=0x289f268) at autovacuum.c:3095
#17 do_autovacuum () at autovacuum.c:2450
#18 0x00000000006a06a9 in AutoVacWorkerMain (argv=0x0, argc=0) at autovacuum.c:1693
#19 0x00000000006a0f49 in StartAutoVacWorker () at autovacuum.c:1498
#20 0x00000000006aeaea in StartAutovacuumWorker () at postmaster.c:5462
#21 sigusr1_handler (postgres_signal_arg=<optimized out>) at postmaster.c:5159
#22 <signal handler called>
#23 0x00007fe11334bb83 in __select_nocancel () from /lib64/libc.so.6
#24 0x000000000047a3a3 in ServerLoop () at postmaster.c:1719
#25 0x00000000006afacf in PostmasterMain (argc=argc@entry=3, argv=argv@entry=0x277ba20) at postmaster.c:1363
#26 0x000000000047bb6f in main (argc=3, argv=0x277ba20) at main.c:228
(gdb) f 4
#4 0x00000000007095ac in LWLockAcquire (lock=0x2aaab670dfe4, mode=LW_EXCLUSIVE) at lwlock.c:1233
1233PGSemaphoreLock(proc->sem);
(gdb)
(gdb) p num_held_lwlocks
$1 = 1
(gdb) p InterruptHoldoffCount
$2 = 2
(gdb) p held_lwlocks
$3 = {{lock = 0x2aaab4009564, mode = LW_EXCLUSIVE}, {lock = 0x2aaaaac0aa00, mode = LW_SHARED}, {lock = 0x2aaaba8da364, mode = LW_EXCLUSIVE}, {lock = 0x2aaaaac15900, mode = LW_EXCLUSIVE}, {
lock = 0x2aaab66eb064, mode = LW_EXCLUSIVE}, {lock = 0x2aaaaac15900, mode = LW_EXCLUSIVE}, {lock = 0x2aaaaac15c80, mode = LW_EXCLUSIVE}, {lock = 0x0,
mode = LW_EXCLUSIVE} <repeats 193 times>}
# env
- CentOS 7.3 64bit
- PostgreSQL 10.2(installed via rpm download from postgresql.org)
# application SQL
begin;
delete from bi_dm.tdm_sncity_workorder_analytic_statistics where order_date between '2018-10-09' and '2018-10-09';
INSERT INTO bi_dm.tdm_sncity_workorder_analytic_statistics (CITYCODE,CITYNAME,DISTRICT,DISTRICT_NAME,ZL_WERKS,ZL_WERKS_NAME,AREA_CD,AREA_NM,ZSIZE,DEPT_CD,TMALL_FLG,ORDER_DATE,DISTRIBUTE,WORKORDER,FNSH
WORKORDER,STATISFACTION_CHILD,STATISFACTION_MOTHER,ONLYONCE_FNSH,REPLY_1H,FNSH_24H,FNSH_48H,ETL_TIME,SALE_PTY,SALE_PTY_FLAG,VBELN_TP,DEALCOUNT_3,NOT_FNSH_NUM) VALUES($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11
,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24,$25,$26,$27) ON CONFLICT(order_date,citycode,cityname,district,district_name,zl_werks,zl_werks_name,area_cd,area_nm,zsize,dept_cd,tmall_flg,sale_pty
,sale_pty_flag,vbeln_tp) DO UPDATE SET CITYCODE = excluded.CITYCODE,CITYNAME = excluded.CITYNAME,DISTRICT = excluded.DISTRICT,DISTRICT_NAME = excluded.DISTRICT_NAME,ZL_WERKS = excluded.ZL_WERKS,
ZL_WERKS_NAME = excluded.ZL_WERKS_NAME,AREA_CD = excluded.AREA_CD,AREA_NM = excluded.AREA_NM,ZSIZE = excluded.ZSIZE,DEPT_CD = excluded.DEPT_CD,TMALL_FLG = excluded.TMALL_FLG,ORDER_DATE =
excluded.ORDER_DATE,DISTRIBUTE = excluded.DISTRIBUTE,WORKORDER = excluded.WORKORDER,FNSHWORKORDER = excluded.FNSHWORKORDER,STATISFACTION_CHILD = excluded.STATISFACTION_CHILD,STATISFACTION_MOTHER = excluded.STATISFACTIOH_24H
,FNSH_48H = excluded.FNSH_48H,ETL_TIME = excluded.ETL_TIME,SALE_PTY = excluded.SALE_PTY,SALE_PTY_FLAG = excluded.SALE_PTY_FLAG,VBELN_TP = excluded.VBELN_TP,DEALCOUNT_3 = excluded.DEALCOUNT_3,
NOT_N_MOTHER,ONLYONCE_FNSH = excluded.ONLYONCE_FNSH,REPLY_1H = excluded.REPLY_1H,FNSH_24H = excluded.FNSH_24H,FNSH_48H = excluded.FNSH_48H,ETL_TIME = excluded.ETL_TIME,SALE_PTY = excluded.SALE_PTY,
SALE_PTY_FLAG = excluded.SALE_PTY_FLAG,VBELN_TP = excluded.VBELN_TP,DEALCOUNT_3 = excluded.DEALCOUNT_3,NOT_FNSH_NUM = excluded.NOT_FNSH_NUM
-- use jdbc batch update to execute insert
commit;
# table def
db1=# \d bi_dm.tdm_sncity_workorder_analytic_statistics
Table "bi_dm.tdm_sncity_workorder_analytic_statistics"
Column | Type | Collation | Nullable | Default
----------------------+------------------------+-----------+----------+---------
citycode | character varying(100) | | |
cityname | character varying(100) | | |
district | character varying(100) | | |
district_name | character varying(100) | | |
zl_werks | character varying(100) | | |
zl_werks_name | character varying(100) | | |
area_cd | character varying(100) | | |
area_nm | character varying(100) | | |
zsize | character varying(100) | | |
dept_cd | character varying(100) | | |
tmall_flg | character varying(100) | | |
order_date | character varying(100) | | |
distribute | integer | | |
workorder | integer | | |
fnshworkorder | integer | | |
statisfaction_child | integer | | |
statisfaction_mother | integer | | |
onlyonce_fnsh | integer | | |
reply_1h | integer | | |
fnsh_24h | integer | | |
fnsh_48h | integer | | |
etl_time | character varying(100) | | |
sale_pty | character varying(100) | | |
sale_pty_flag | character varying(100) | |
vbeln_tp | character varying(100) | | |
dealcount_3 | integer | | |
not_fnsh_num | integer | | |
Indexes:
"idx1_tdm_sncity_workorder_analytic_statistics" UNIQUE, btree (order_date, citycode, cityname, district, district_name, zl_werks, zl_werks_name, area_cd, area_nm, zsize, dept_cd, tmall_flg, sale_pty, sale_pty_flag, vbeln_tp)
"gin_tdm_sncity_workorder_analytic_statistics" gin (order_date)
# another phenomenon
The above phenomenon just happened on the primary node. The standby node hang on another stack four times.
## stack of recovery process
--------------------------------------
#0 0x00007f7aaf72a79b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1 0x00007f7aaf72a82f in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2 0x00007f7aaf72a8cb in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3 0x000000000069d362 in PGSemaphoreLock (sema=0x2aaaaac07ff8) at pg_sema.c:310
#4 0x00000000007095ac in LWLockAcquire (lock=0x2aaaad50e2e4, mode=LW_EXCLUSIVE) at lwlock.c:1233
#5 0x0000000000503c1a in XLogReadBufferForRedoExtended (record=record@entry=0x105ce58, block_id=block_id@entry=2 '\002', mode=mode@entry=RBM_NORMAL, get_cleanup_lock=get_cleanup_lock@entry=0 '\000', buf=buf@entry=0x7fff550d93c0) at xlogutils.c:399
#6 0x0000000000503d00 in XLogReadBufferForRedo (record=record@entry=0x105ce58, block_id=block_id@entry=2 '\002', buf=buf@entry=0x7fff550d93c0) at xlogutils.c:293
#7 0x000000000048bcd5 in ginRedoDeletePage (record=0x105ce58) at ginxlog.c:482
#8 gin_redo (record=0x105ce58) at ginxlog.c:731
#9 0x00000000004f8b83 in StartupXLOG () at xlog.c:7183
#10 0x00000000006b0091 in StartupProcessMain () at startup.c:217
#11 0x000000000050689a in AuxiliaryProcessMain (argc=argc@entry=2, argv=argv@entry=0x7fff550dcbe0) at bootstrap.c:426
#12 0x00000000006ad160 in StartChildProcess (type=StartupProcess) at postmaster.c:5361
#13 0x00000000006afaab in PostmasterMain (argc=argc@entry=1, argv=argv@entry=0x101ba20) at postmaster.c:1355
#14 0x000000000047bb6f in main (argc=1, argv=0x101ba20) at main.c:228
## stack of backend process(tens of such process)
--------------------------------------
#0 0x00007f7aaf72a79b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1 0x00007f7aaf72a82f in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2 0x00007f7aaf72a8cb in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3 0x000000000069d362 in PGSemaphoreLock (sema=0x2aaaaac074b8) at pg_sema.c:310
#4 0x00000000007095ac in LWLockAcquire (lock=0x2aaac6c081e4, mode=LW_SHARED) at lwlock.c:1233
#5 0x0000000000490943 in ginTraverseLock (buffer=7034160, searchMode=searchMode@entry=1 '\001') at ginbtree.c:40
#6 0x00000000004910e0 in ginFindLeafPage (btree=btree@entry=0x1fd6318, searchMode=searchMode@entry=1 '\001', snapshot=snapshot@entry=0x200a958) at ginbtree.c:97
#7 0x000000000048fe5b in ginScanBeginPostingTree (btree=btree@entry=0x1fd6318, index=<optimized out>, rootBlkno=rootBlkno@entry=2246, snapshot=snapshot@entry=0x200a958) at gindatapage.c:1924
#8 0x000000000049319c in startScanEntry (ginstate=ginstate@entry=0x1eade90, entry=<optimized out>, snapshot=0x200a958) at ginget.c:390
#9 0x0000000000493a92 in startScan (scan=0x1e6c348, scan=0x1e6c348) at ginget.c:543
#10 gingetbitmap (scan=0x1e6c348, tbm=0x20df978) at ginget.c:1829
#11 0x00000000004c245a in index_getbitmap (scan=scan@entry=0x1e6c348, bitmap=bitmap@entry=0x20df978) at indexam.c:726
#12 0x0000000000600c52 in MultiExecBitmapIndexScan (node=0x1ebfcd0) at nodeBitmapIndexscan.c:104
#13 0x00000000005f40b9 in MultiExecProcNode (node=<optimized out>) at execProcnode.c:490
#14 0x00000000006004c7 in BitmapHeapNext (node=node@entry=0x1a0c618) at nodeBitmapHeapscan.c:117
#15 0x00000000005f5094 in ExecScanFetch (recheckMtd=0x5ff990 <BitmapHeapRecheck>, accessMtd=0x5ffa20 <BitmapHeapNext>, node=0x1a0c618) at execScan.c:97
#16 ExecScan (node=0x1a0c618, accessMtd=0x5ffa20 <BitmapHeapNext>, recheckMtd=0x5ff990 <BitmapHeapRecheck>) at execScan.c:164
#17 0x0000000000604a3f in ExecProcNode (node=0x1a0c618) at ../../../src/include/executor/executor.h:250
#18 ExecHashJoin (pstate=0x1ebf670) at nodeHashjoin.c:145
#19 0x00000000005fad89 in ExecProcNode (node=0x1ebf670) at ../../../src/include/executor/executor.h:250
#20 fetch_input_tuple (aggstate=aggstate@entry=0x1ebef78) at nodeAgg.c:695
#21 0x00000000005fcfd3 in agg_fill_hash_table (aggstate=0x1ebef78) at nodeAgg.c:2539
#22 ExecAgg (pstate=0x1ebef78) at nodeAgg.c:2151
#23 0x0000000000610636 in ExecProcNode (node=0x1ebef78) at ../../../src/include/executor/executor.h:250
#24 ExecSort (pstate=0x1ebed08) at nodeSort.c:106
#25 0x0000000000607cfd in ExecProcNode (node=0x1ebed08) at ../../../src/include/executor/executor.h:250
#26 ExecLimit (pstate=0x1ebe0f8) at nodeLimit.c:95
#27 0x00000000005ef612 in ExecProcNode (node=0x1ebe0f8) at ../../../src/include/executor/executor.h:250
#28 ExecutePlan (execute_once=<optimized out>, dest=0x10e34f8, direction=<optimized out>, numberTuples=0, sendTuples=1 '\001', operation=CMD_SELECT, use_parallel_mode=<optimized out>, planstate=0x1ebe0f8, estate=0x1ebde98) at execMain.c:1722
#29 standard_ExecutorRun (queryDesc=0x18caf88, direction=<optimized out>, count=0, execute_once=<optimized out>) at execMain.c:363
#30 0x00007f7aa028415d in pgss_ExecutorRun (queryDesc=0x18caf88, direction=ForwardScanDirection, count=0, execute_once=<optimized out>) at pg_stat_statements.c:889
#31 0x000000000071a28b in PortalRunSelect (portal=portal@entry=0x1111d88, forward=forward@entry=1 '\001', count=0, count@entry=9223372036854775807, dest=dest@entry=0x10e34f8) at pquery.c:932
#32 0x000000000071b63f in PortalRun (portal=<optimized out>, count=9223372036854775807, isTopLevel=<optimized out>, run_once=<optimized out>, dest=0x10e34f8, altdest=0x10e34f8, completionTag=0x7fff550dc220 "") at pquery.c:773
#33 0x0000000000718f84 in PostgresMain (argc=<optimized out>, argv=<optimized out>, dbname=<optimized out>, username=<optimized out>) at postgres.c:1984
#34 0x000000000047ad3c in BackendRun (port=0x106a590) at postmaster.c:4405
#35 BackendStartup (port=0x106a590) at postmaster.c:4077
#36 ServerLoop () at postmaster.c:1755
#37 0x00000000006afacf in PostmasterMain (argc=argc@entry=1, argv=argv@entry=0x101ba20) at postmaster.c:1363
#38 0x000000000047bb6f in main (argc=1, argv=0x101ba20) at main.c:228
# Similar problem
I noticed that there were some similar bug reports before.
- https://www.postgresql.org/message-id/CAK7KUdBjF4TN5Z2Rf0fv1MVUa%3DuqQssKMhoTXUa2y88R4PPjsg%40mail.g...
- https://www.postgresql.org/message-id/1509467408126-0.post@n3.nabble.com
- https://www.postgresql-archive.org/Stalls-on-PGSemaphoreLock-td5797385.html
According to this https://www.postgresql-archive.org/Stalls-on-PGSemaphoreLock-td5797385.html, this issue may be related to the huge page.
So we turn off the huge page, and I don't know if this problem will happen again, but it hasn't happened yet to now.
Maybe should only turn off transparent hugepage
BTW:
Related configuration of our machine is as following:
cat /sys/kernel/mm/transparent_hugepage/defrag
[always] madvise never
cat /proc/sys/kernel/sem
250 32000 32 128
/proc/meminfo
AnonHugePages: 4446208 kB
HugePages_Total: 38912
HugePages_Free: 5907
HugePages_Rsvd: 40
HugePages_Surp: 0
Maybe we should only trun off transparent hugepage instead of huge page.
Now, I am not sure if this problem is a kernel configuration issue or a gin index implementation bug, any tips?
Regard,
Chen Huajun
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re:Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-11-06 17:57 chenhj <chjischj@163.com>
parent: chenhj <chjischj@163.com>
2 siblings, 1 reply; 67+ messages in thread
From: chenhj @ 2018-11-06 17:57 UTC (permalink / raw)
To: chenhj <chjischj@163.com>; +Cc: pgsql-hackers@lists.postgresql.org
Hi,all
I analyzed the btree block where lwlock deadlock occurred, as follows:
## insert process(ginInsertValue())
644(root blkno)
|
7054(2. held LWLock:0x2aaac587ae64) ----rightlink----> xxxx(3. Acquire LWLock:0x2aaab4009564，buffer = 2119038,blkno should be 9954)
|
701(1. held LWLock:0x2aaab670dfe4)
The ginInsertValue() function above gets the lwlock in the order described in the README.
src/backend/access/gin/README
---------------------------------------------------------------
To avoid deadlocks, B-tree pages must always be locked in the same order:
left to right, and bottom to top.
...
-----------------------------------------------------------------
## autovacuum process(ginScanToDelete())
644(root blkno)
|
... 9954(1. held LWLock:0x2aaab4009564) ...
|
701(2. Acquire LWLock:0x2aaab670dfe4)
note：according to autovacuum's core 701's parent is 9954;while insert's core shows 701's parent is 7054, rightlink of 7054 is 9954！
However, ginScanToDelete() depth-first scans the btree and gets the EXCLUSIVE lock, which creates a deadlock.
Is the above statement correct? If so, deadlocks should easily happen.
static bool
ginScanToDelete(GinVacuumState *gvs, BlockNumber blkno, bool isRoot,
DataPageDeleteStack *parent, OffsetNumber myoff)
{
...
if (!isRoot)
LockBuffer(buffer, GIN_EXCLUSIVE);
...
for (i = FirstOffsetNumber; i <= GinPageGetOpaque(page)->maxoff; i++)
{
PostingItem *pitem = GinDataPageGetPostingItem(page, i);
if (ginScanToDelete(gvs, PostingItemGetBlockNumber(pitem), FALSE, me, i))
i--;
}
...
}
src/backend/access/gin/README
-----------------------------------------------------------------
The previous paragraph's reasoning only applies to searches, and only to
posting trees. To protect from inserters following a downlink to a deleted
page, vacuum simply locks out all concurrent insertions to the posting tree,
by holding a super-exclusive lock on the parent page of subtree with deletable
pages.
...
posting tree. To exclude interference with readers vacuum takes exclusive
locks in a depth-first scan in left-to-right order of page tuples.
...
-----------------------------------------------------------------
# stacks
## stack of insert：Acquire lock 0x2aaab4009564(blkno:9954) and hold 0x2aaab670dfe4(blkno:701), 0x2aaac587ae64(blkno:7054)
-----------------------------------------------------------------
(gdb) bt
#0 0x00007fe11552379b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1 0x00007fe11552382f in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2 0x00007fe1155238cb in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3 0x000000000069d362 in PGSemaphoreLock (sema=0x2aaaaac02958) at pg_sema.c:310
#4 0x00000000007095ac in LWLockAcquire (lock=0x2aaab4009564, mode=LW_EXCLUSIVE) at lwlock.c:1233
#5 0x0000000000490a32 in ginStepRight (buffer=6713826, index=<optimized out>, lockmode=lockmode@entry=2) at ginbtree.c:174
#6 0x0000000000490c1c in ginFinishSplit (btree=btree@entry=0x7ffd81e4f950, stack=0x28eebf8, freestack=freestack@entry=1 '\001', buildStats=buildStats@entry=0x0) at ginbtree.c:701
#7 0x0000000000491396 in ginInsertValue (btree=btree@entry=0x7ffd81e4f950, stack=<optimized out>, insertdata=insertdata@entry=0x7ffd81e4f940, buildStats=buildStats@entry=0x0)
at ginbtree.c:773
#8 0x000000000048fb42 in ginInsertItemPointers (index=<optimized out>, rootBlkno=rootBlkno@entry=644, items=items@entry=0x2916598, nitem=nitem@entry=353, buildStats=buildStats@entry=0x0)
at gindatapage.c:1907
#9 0x000000000048a9ea in ginEntryInsert (ginstate=ginstate@entry=0x28e7ef8, attnum=<optimized out>, key=42920440, category=<optimized out>, items=0x2916598, nitem=353,
buildStats=buildStats@entry=0x0) at gininsert.c:214
#10 0x0000000000496d94 in ginInsertCleanup (ginstate=ginstate@entry=0x28e7ef8, full_clean=full_clean@entry=0 '\000', fill_fsm=fill_fsm@entry=1 '\001',
forceCleanup=forceCleanup@entry=0 '\000', stats=stats@entry=0x0) at ginfast.c:883
#11 0x0000000000497727 in ginHeapTupleFastInsert (ginstate=ginstate@entry=0x28e7ef8, collector=collector@entry=0x7ffd81e4fe60) at ginfast.c:448
#12 0x000000000048b209 in gininsert (index=<optimized out>, values=0x7ffd81e4ff40, isnull=0x7ffd81e50040 "", ht_ctid=0x280d98c, heapRel=<optimized out>, checkUnique=<optimized out>,
indexInfo=0x28b5aa8) at gininsert.c:522
#13 0x00000000005ee8dd in ExecInsertIndexTuples (slot=slot@entry=0x28b5d58, tupleid=tupleid@entry=0x280d98c, estate=estate@entry=0x28b5288, noDupErr=noDupErr@entry=1 '\001',
specConflict=specConflict@entry=0x7ffd81e5013b "", arbiterIndexes=arbiterIndexes@entry=0x28c6dd8) at execIndexing.c:386
#14 0x000000000060ccf5 in ExecInsert (canSetTag=1 '\001', estate=0x28b5288, onconflict=ONCONFLICT_UPDATE, arbiterIndexes=0x28c6dd8, planSlot=0x28b5d58, slot=0x28b5d58, mtstate=0x28b5628)
at nodeModifyTable.c:564
#15 ExecModifyTable (pstate=0x28b5628) at nodeModifyTable.c:1766
#16 0x00000000005ef612 in standard_ExecutorRun (queryDesc=<optimized out>, direction=<optimized out>, count=<optimized out>, execute_once=<optimized out>)
at ../../../src/include/executor/executor.h:250
#17 0x00007fe10607d15d in pgss_ExecutorRun (queryDesc=0x28cc1d8, direction=ForwardScanDirection, count=0, execute_once=<optimized out>) at pg_stat_statements.c:889
#18 0x000000000071a7ba in ProcessQuery (plan=<optimized out>,
sourceText=0x2849058 "INSERT INTO bi_dm.tdm_sncity_workorder_analytic_statistics (CITYCODE,CITYNAME,DISTRICT,DISTRICT_NAME,ZL_WERKS,ZL_WERKS_NAME,AREA_CD,AREA_NM,ZSIZE,DEPT_CD,TMALL_FLG,ORDER_DATE,DISTRIBUTE,WORKORDER,FNSH"..., params=0x28b2260, queryEnv=0x0, dest=0xc9e2a0 <donothingDR>, completionTag=0x7ffd81e507b0 "") at pquery.c:161
#19 0x000000000071a9f7 in PortalRunMulti (portal=portal@entry=0x289fdd8, isTopLevel=isTopLevel@entry=1 '\001', setHoldSnapshot=setHoldSnapshot@entry=0 '\000', dest=0xc9e2a0 <donothingDR>,
dest@entry=0x284d778, altdest=0xc9e2a0 <donothingDR>, altdest@entry=0x284d778, completionTag=completionTag@entry=0x7ffd81e507b0 "") at pquery.c:1286
#20 0x000000000071b535 in PortalRun (portal=<optimized out>, count=1, isTopLevel=<optimized out>, run_once=<optimized out>, dest=0x284d778, altdest=0x284d778,
completionTag=0x7ffd81e507b0 "") at pquery.c:799
#21 0x0000000000718f84 in PostgresMain (argc=<optimized out>, argv=<optimized out>, dbname=<optimized out>, username=<optimized out>) at postgres.c:1984
#22 0x000000000047ad3c in BackendRun (port=0x27cbc40) at postmaster.c:4405
#23 BackendStartup (port=0x27cbc40) at postmaster.c:4077
#24 ServerLoop () at postmaster.c:1755
#25 0x00000000006afacf in PostmasterMain (argc=argc@entry=3, argv=argv@entry=0x277ba20) at postmaster.c:1363
#26 0x000000000047bb6f in main (argc=3, argv=0x277ba20) at main.c:228
(gdb) p num_held_lwlocks
$1 = 2
(gdb) p lwlocks
$2 = {{lock = 0x2aaab670dfe4, mode = LW_EXCLUSIVE}, {lock = 0x2aaac587ae64, mode = LW_EXCLUSIVE}, {lock = 0x2aaaaac0a380, mode = LW_SHARED}, {lock = 0x2aaab670a2e4, mode = LW_EXCLUSIVE}, {
lock = 0x2aaab670a2e4, mode = LW_EXCLUSIVE}, {lock = 0x2aaab670a2e4, mode = LW_EXCLUSIVE}, {lock = 0x2aaab670a2e4, mode = LW_EXCLUSIVE}, {lock = 0x2aaab670a2e4, mode = LW_EXCLUSIVE}, {
lock = 0x2aaab670a2e4, mode = LW_EXCLUSIVE}, {lock = 0x2aaab670a2e4, mode = LW_EXCLUSIVE}, {lock = 0x2aaaaac15a80, mode = LW_EXCLUSIVE}, {lock = 0x0,
mode = LW_EXCLUSIVE} <repeats 189 times>}
(gdb) f 6
#6 0x0000000000490c1c in ginFinishSplit (btree=btree@entry=0x7ffd81e4f950, stack=0x28eebf8, freestack=freestack@entry=1 '\001', buildStats=buildStats@entry=0x0) at ginbtree.c:701
701 parent->buffer = ginStepRight(parent->buffer, btree->index, GIN_EXCLUSIVE);
(gdb) p *stack
$27 = {blkno = 701, buffer = 2758312, off = 0, iptr = {ip_blkid = {bi_hi = 5, bi_lo = 0}, ip_posid = 40884}, predictNumber = 1, parent = 0x28eea68}
(gdb) p ((LWLock*) (&((&BufferDescriptors[(2758312 - 1)].bufferdesc))->content_lock))
$28 = (LWLock *) 0x2aaab670dfe4
(gdb) p *parent
$29 = {blkno = 7054, buffer = 6713826, off = 0, iptr = {ip_blkid = {bi_hi = 8, bi_lo = 0}, ip_posid = 40886}, predictNumber = 1, parent = 0x28eeab8}
(gdb) p ((LWLock*) (&((&BufferDescriptors[(6713826 - 1)].bufferdesc))->content_lock))
$32 = (LWLock *) 0x2aab4fa5cfe4
(gdb) p *(GinBtreeStack *)0x28eeab8
$31 = {blkno = 644, buffer = 6713825, off = 1, iptr = {ip_blkid = {bi_hi = 35, bi_lo = 0}, ip_posid = 40884}, predictNumber = 1, parent = 0x0}
(gdb) p ((LWLock*) (&((&BufferDescriptors[(6713825 - 1)].bufferdesc))->content_lock))
$33 = (LWLock *) 0x2aaac587ae24
-----------------------------------------------------------------
## stack of autovacuum:Acquire lock 0x2aaab670dfe4(blkno:701) and hold 0x2aaab4009564(blkno:9954)
-----------------------------------------------------------------
#0 0x00007fe11552379b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1 0x00007fe11552382f in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2 0x00007fe1155238cb in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3 0x000000000069d362 in PGSemaphoreLock (sema=0x2aaaaac07eb8) at pg_sema.c:310
#4 0x00000000007095ac in LWLockAcquire (lock=0x2aaab670dfe4, mode=LW_EXCLUSIVE) at lwlock.c:1233
#5 0x00000000004947ef in ginScanToDelete (gvs=gvs@entry=0x7ffd81e4d0f0, blkno=701, isRoot=isRoot@entry=0 '\000',
parent=parent@entry=0x7ffd81e4c790, myoff=myoff@entry=37) at ginvacuum.c:262
#6 0x0000000000494874 in ginScanToDelete (gvs=gvs@entry=0x7ffd81e4d0f0, blkno=blkno@entry=9954, isRoot=isRoot@entry=1 '\001',
parent=parent@entry=0x7ffd81e4c790, myoff=myoff@entry=0) at ginvacuum.c:277
#7 0x0000000000494ed1 in ginVacuumPostingTreeLeaves (gvs=gvs@entry=0x7ffd81e4d0f0, blkno=9954, isRoot=isRoot@entry=0 '\000') at ginvacuum.c:404
#8 0x0000000000494e21 in ginVacuumPostingTreeLeaves (gvs=gvs@entry=0x7ffd81e4d0f0, blkno=644, isRoot=isRoot@entry=1 '\001') at ginvacuum.c:372
#9 0x0000000000495540 in ginVacuumPostingTree (rootBlkno=<optimized out>, gvs=0x7ffd81e4d0f0) at ginvacuum.c:426
#10 ginbulkdelete (info=0x7ffd81e4f720, stats=<optimized out>, callback=<optimized out>, callback_state=<optimized out>) at ginvacuum.c:649
#11 0x00000000005e1194 in lazy_vacuum_index (indrel=0x3146e28, stats=stats@entry=0x28ec200, vacrelstats=vacrelstats@entry=0x28ebc28)
at vacuumlazy.c:1621
#12 0x00000000005e214d in lazy_scan_heap (aggressive=<optimized out>, nindexes=<optimized out>, Irel=<optimized out>,
vacrelstats=<optimized out>, options=16, onerel=0x28ec1f8) at vacuumlazy.c:1311
#13 lazy_vacuum_rel (onerel=onerel@entry=0x3144fa8, options=options@entry=99, params=params@entry=0x289f270, bstrategy=<optimized out>)
at vacuumlazy.c:258
#14 0x00000000005dfc35 in vacuum_rel (relid=relid@entry=2060396948, relation=relation@entry=0x7ffd81e4fe50, options=options@entry=99,
---Type <return> to continue, or q <return> to quit---
params=params@entry=0x289f270) at vacuum.c:1463
#15 0x00000000005e0ad2 in vacuum (options=99, relation=relation@entry=0x7ffd81e4fe50, relid=<optimized out>, params=params@entry=0x289f270,
va_cols=va_cols@entry=0x0, bstrategy=<optimized out>, bstrategy@entry=0x289e988, isTopLevel=isTopLevel@entry=1 '\001') at vacuum.c:306
#16 0x00000000006a01ff in autovacuum_do_vac_analyze (bstrategy=0x289e988, tab=0x289f268) at autovacuum.c:3095
#17 do_autovacuum () at autovacuum.c:2450
#18 0x00000000006a06a9 in AutoVacWorkerMain (argv=0x0, argc=0) at autovacuum.c:1693
#19 0x00000000006a0f49 in StartAutoVacWorker () at autovacuum.c:1498
#20 0x00000000006aeaea in StartAutovacuumWorker () at postmaster.c:5462
#21 sigusr1_handler (postgres_signal_arg=<optimized out>) at postmaster.c:5159
#22 <signal handler called>
#23 0x00007fe11334bb83 in __select_nocancel () from /lib64/libc.so.6
#24 0x000000000047a3a3 in ServerLoop () at postmaster.c:1719
#25 0x00000000006afacf in PostmasterMain (argc=argc@entry=3, argv=argv@entry=0x277ba20) at postmaster.c:1363
#26 0x000000000047bb6f in main (argc=3, argv=0x277ba20) at main.c:228
(gdb) f 4
#4 0x00000000007095ac in LWLockAcquire (lock=0x2aaab670dfe4, mode=LW_EXCLUSIVE) at lwlock.c:1233
1233PGSemaphoreLock(proc->sem);
(gdb) p num_held_lwlocks
$1 = 1
(gdb) p InterruptHoldoffCount
$2 = 2
(gdb) p held_lwlocks
$3 = {{lock = 0x2aaab4009564, mode = LW_EXCLUSIVE}, {lock = 0x2aaaaac0aa00, mode = LW_SHARED}, {lock = 0x2aaaba8da364, mode = LW_EXCLUSIVE}, {lock = 0x2aaaaac15900, mode = LW_EXCLUSIVE}, {
lock = 0x2aaab66eb064, mode = LW_EXCLUSIVE}, {lock = 0x2aaaaac15900, mode = LW_EXCLUSIVE}, {lock = 0x2aaaaac15c80, mode = LW_EXCLUSIVE}, {lock = 0x0,
mode = LW_EXCLUSIVE} <repeats 193 times>}
(gdb) f 8
#8 0x0000000000494e21 in ginVacuumPostingTreeLeaves (gvs=gvs@entry=0x7ffd81e4d0f0, blkno=644, isRoot=isRoot@entry=1 '\001') at ginvacuum.c:372
372 if (ginVacuumPostingTreeLeaves(gvs, children[i], FALSE))
(gdb) p i
$13 = 2
(gdb) p children[1]
$14 = 7054
(gdb) p children[2]
$15 = 9954
#7 0x0000000000494ed1 in ginVacuumPostingTreeLeaves (gvs=gvs@entry=0x7ffd81e4d0f0, blkno=9954, isRoot=isRoot@entry=0 '\000') at ginvacuum.c:404
404 ginScanToDelete(gvs, blkno, TRUE, &root, InvalidOffsetNumber);
(gdb) p buffer
$16 = 2119038
(gdb) p ((LWLock*) (&((&BufferDescriptors[(2119038 - 1)].bufferdesc))->content_lock))
$17 = (LWLock *) 0x2aaab4009564
(gdb) f 5
#5 0x00000000004947ef in ginScanToDelete (gvs=gvs@entry=0x7ffd81e4d0f0, blkno=701, isRoot=isRoot@entry=0 '\000',
parent=parent@entry=0x7ffd81e4c790, myoff=myoff@entry=37) at ginvacuum.c:262
262 LockBuffer(buffer, GIN_EXCLUSIVE);
(gdb) p buffer
$29 = 2758312
(gdb) p ((LWLock*) (&((&BufferDescriptors[(2758312 - 1)].bufferdesc))->content_lock))
$32 = (LWLock *) 0x2aaab670dfe4
-----------------------------------------------------------------
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-11-07 20:52 Peter Geoghegan <pg@bowt.ie>
parent: chenhj <chjischj@163.com>
0 siblings, 0 replies; 67+ messages in thread
From: Peter Geoghegan @ 2018-11-07 20:52 UTC (permalink / raw)
To: chjischj@163.com; +Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Tue, Nov 6, 2018 at 10:05 AM chenhj <chjischj@163.com> wrote:
> I analyzed the btree block where lwlock deadlock occurred, as follows:
Thank you for doing this important work.
You're using Postgres 10.2. While that version isn't current with all
GIN bug fixes, it does have this important one:
"Ensure that vacuum will always clean up the pending-insertions list
of a GIN index (Masahiko Sawada)"
Later GIN fixes seem unlikely to be relevant to your issue. I think
that this is probably a genuine, new bug.
> The ginInsertValue() function above gets the lwlock in the order described in the README.
> However, ginScanToDelete() depth-first scans the btree and gets the EXCLUSIVE lock, which creates a deadlock.
> Is the above statement correct? If so, deadlocks should easily happen.
I have been suspicious of deadlock hazards in the code for some time
-- particularly around pending list cleanup. I go into a lot of detail
on my suspicions here:
https://www.postgresql.org/message-id/flat/CAH2-WzmfUpRjWcUq3%2B9ijyum4AJ%2Bk-meGT8_HnxBMpKz1aNS-g%4...
I note that your first deadlock involve the following kinds of backends:
* ginInsertCleanup() calls from a regular backend, which will have a
backend do things that VACUUM generally only gets to do, like call
RecordFreeIndexPage().
* (auto)VACUUM processes.
Your second/recovery deadlock involves:
* Regular read-only queries.
* Recovery code.
Quite a lot of stuff is involved here!
The code in this area is way too complicated, and I haven't thought
about it in about a year, so it's hard for me to be sure of anything
at the moment. My guess is that there is confusion about the type of
page expected within one or more blocks (e.g. entry tree vs. pending
list), due to a race condition in block deletion and/or recycling --
again, I've suspected something like this could happen for some time.
The fact that you get a distinct deadlock during recovery is
consistent with that theory.
It's safe to say that promoting the asserts on gin page type into
"can't happen" elog errors in places like scanPendingInsert() and
ginInsertCleanup() would be a good start. Note that we already did
similar Assert-elog(ERROR) promotion this for posting tree code, when
similar bugs appeared way back in 2013.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-11-07 23:18 Peter Geoghegan <pg@bowt.ie>
parent: chenhj <chjischj@163.com>
2 siblings, 1 reply; 67+ messages in thread
From: Peter Geoghegan @ 2018-11-07 23:18 UTC (permalink / raw)
To: chjischj@163.com; +Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Mon, Oct 29, 2018 at 12:04 PM chenhj <chjischj@163.com> wrote:
> #4 0x00000000007095ac in LWLockAcquire (lock=0x2aaab4009564, mode=LW_EXCLUSIVE) at lwlock.c:1233
> #5 0x0000000000490a32 in ginStepRight (buffer=6713826, index=<optimized out>, lockmode=lockmode@entry=2) at ginbtree.c:174
> #6 0x0000000000490c1c in ginFinishSplit (btree=btree@entry=0x7ffd81e4f950, stack=0x28eebf8, freestack=freestack@entry=1 '\001', buildStats=buildStats@entry=0x0) at ginbtree.c:701
> #7 0x0000000000491396 in ginInsertValue (btree=btree@entry=0x7ffd81e4f950, stack=<optimized out>, insertdata=insertdata@entry=0x7ffd81e4f940, buildStats=buildStats@entry=0x0)
> at ginbtree.c:773
> #8 0x000000000048fb42 in ginInsertItemPointers (index=<optimized out>, rootBlkno=rootBlkno@entry=644, items=items@entry=0x2916598, nitem=nitem@entry=353, buildStats=buildStats@entry=0x0)
> at gindatapage.c:1907
> #9 0x000000000048a9ea in ginEntryInsert (ginstate=ginstate@entry=0x28e7ef8, attnum=<optimized out>, key=42920440, category=<optimized out>, items=0x2916598, nitem=353,
> buildStats=buildStats@entry=0x0) at gininsert.c:214
> #10 0x0000000000496d94 in ginInsertCleanup (ginstate=ginstate@entry=0x28e7ef8, full_clean=full_clean@entry=0 '\000', fill_fsm=fill_fsm@entry=1 '\001',
> forceCleanup=forceCleanup@entry=0 '\000', stats=stats@entry=0x0) at ginfast.c:883
> #11 0x0000000000497727 in ginHeapTupleFastInsert (ginstate=ginstate@entry=0x28e7ef8, collector=collector@entry=0x7ffd81e4fe60) at ginfast.c:448
> #12 0x000000000048b209 in gininsert (index=<optimized out>, values=0x7ffd81e4ff40, isnull=0x7ffd81e50040 "", ht_ctid=0x280d98c, heapRel=<optimized out>, checkUnique=<optimized out>,
> indexInfo=0x28b5aa8) at gininsert.c:522
I want to point something out about the above inserter represented by
this stack trace. It deadlocks against VACUUM from within the
following ginEntryInsert() call:
/*
* While we left the page unlocked, more stuff might have gotten
* added to it. If so, process those entries immediately. There
* shouldn't be very many, so we don't worry about the fact that
* we're doing this with exclusive lock. Insertion algorithm
* guarantees that inserted row(s) will not continue on next page.
* NOTE: intentionally no vacuum_delay_point in this loop.
*/
if (PageGetMaxOffsetNumber(page) != maxoff)
{
ginInitBA(&accum);
processPendingPage(&accum, &datums, page, maxoff + 1);
ginBeginBAScan(&accum);
while ((list = ginGetBAEntry(&accum,
&attnum, &key, &category,
&nlist)) != NULL)
ginEntryInsert(ginstate, attnum, key, category, //
<--- deadlocks
list, nlist, NULL);
}
Clearly this particular call to ginEntryInsert() from within
ginInsertCleanup() is generally supposed to be a rare occurrence. It's
not surprising that it's hard for you to hit this issue.
BTW, I strongly doubt that disabling hugepages has fixed anything,
though it may have accidentally masked the problem. This is an issue
around the basic correctness of the locking protocols used by GIN.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-11-08 01:46 Peter Geoghegan <pg@bowt.ie>
parent: chenhj <chjischj@163.com>
2 siblings, 3 replies; 67+ messages in thread
From: Peter Geoghegan @ 2018-11-08 01:46 UTC (permalink / raw)
To: chjischj@163.com, Teodor Sigaev <teodor@sigaev.ru>; +Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Mon, Oct 29, 2018 at 12:04 PM chenhj <chjischj@163.com> wrote:
> ## stack of autovacuum:Acquire lock 0x2aaab670dfe4 and hold 0x2aaab4009564
> --------------------------------------
> (gdb) bt
> #0 0x00007fe11552379b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
> #1 0x00007fe11552382f in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
> #2 0x00007fe1155238cb in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
> #3 0x000000000069d362 in PGSemaphoreLock (sema=0x2aaaaac07eb8) at pg_sema.c:310
> #4 0x00000000007095ac in LWLockAcquire (lock=0x2aaab670dfe4, mode=LW_EXCLUSIVE) at lwlock.c:1233
> #5 0x00000000004947ef in ginScanToDelete (gvs=gvs@entry=0x7ffd81e4d0f0, blkno=701, isRoot=isRoot@entry=0 '\000', parent=parent@entry=0x7ffd81e4c790, myoff=myoff@entry=37) at ginvacuum.c:262
> #6 0x0000000000494874 in ginScanToDelete (gvs=gvs@entry=0x7ffd81e4d0f0, blkno=blkno@entry=9954, isRoot=isRoot@entry=1 '\001', parent=parent@entry=0x7ffd81e4c790, myoff=myoff@entry=0)
> at ginvacuum.c:277
> #7 0x0000000000494ed1 in ginVacuumPostingTreeLeaves (gvs=gvs@entry=0x7ffd81e4d0f0, blkno=9954, isRoot=isRoot@entry=0 '\000') at ginvacuum.c:404
> #8 0x0000000000494e21 in ginVacuumPostingTreeLeaves (gvs=gvs@entry=0x7ffd81e4d0f0, blkno=644, isRoot=isRoot@entry=1 '\001') at ginvacuum.c:372
> #9 0x0000000000495540 in ginVacuumPostingTree (rootBlkno=<optimized out>, gvs=0x7ffd81e4d0f0) at ginvacuum.c:426
> #10 ginbulkdelete (info=0x7ffd81e4f720, stats=<optimized out>, callback=<optimized out>, callback_state=<optimized out>) at ginvacuum.c:649
> #11 0x00000000005e1194 in lazy_vacuum_index (indrel=0x3146e28, stats=stats@entry=0x28ec200, vacrelstats=vacrelstats@entry=0x28ebc28) at vacuumlazy.c:1621
> #12 0x00000000005e214d in lazy_scan_heap (aggressive=<optimized out>, nindexes=<optimized out>, Irel=<optimized out>, vacrelstats=<optimized out>, options=16, onerel=0x28ec1f8)
> at vacuumlazy.c:1311
> #13 lazy_vacuum_rel (onerel=onerel@entry=0x3144fa8, options=options@entry=99, params=params@entry=0x289f270, bstrategy=<optimized out>) at vacuumlazy.c:258
Actually, the bigger problem is on this side of the deadlock, within
VACUUM. ginInsertCleanup() (the first/other side of the deadlock) may
have problems, but this seems worse. Commit 218f51584d5 appears to be
at fault here.
First things first: ginScanToDelete() *maintains* buffer locks on
multiple levels of a posting tree, meaning that there may be cases
where quite a few exclusive buffer locks may be held all at once (one
per level). MAX_SIMUL_LWLOCKS is 200 these days, and in practice a
B-Tree can never get that tall, but having the number of buffer locks
acquired be determined by the height of the tree is not okay, on
general principle. The nbtree code's page deletion goes to a lot of
effort to keep the number of buffer locks fixed, but nothing like that
is is attempted for GIN posting trees.
Chen's original analysis of the problem seems to be more or less
accurate: the order that ginScanToDelete() acquires buffer locks as it
descends the tree (following commit 218f51584d5) is not compatible
with the order within ginFinishSplit(). The faulty code within
ginScanToDelete() crabs/couples buffer locks while descending the
tree, whereas the code within ginFinishSplit() crabs them as it
ascends the same tree.
Teodor: Do you think that the issue is fixable? It looks like there
are serious issues with the design of 218f51584d5 to me. I don't think
the general "there can't be any inserters at this subtree" thing works
given that we have to couple buffer locks when moving right for other
reasons. We call ginStepRight() within ginFinishSplit(), for reasons
that date back to bug fix commit ac4ab97e from 2013 -- that detail is
probably important, because it seems to be what breaks the subtree
design (we don't delete in two phases or anything in GIN).
I think that you have to be doing a multi-level delete for a deadlock
to take place, which isn't particularly likely to coincide with a
concurrent insertion in general. That may explain why it's taken a
year to get a high-quality bug report.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-11-08 01:49 Peter Geoghegan <pg@bowt.ie>
parent: Peter Geoghegan <pg@bowt.ie>
2 siblings, 0 replies; 67+ messages in thread
From: Peter Geoghegan @ 2018-11-08 01:49 UTC (permalink / raw)
To: chjischj@163.com, Teodor Sigaev <teodor@sigaev.ru>; +Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Wed, Nov 7, 2018 at 5:46 PM Peter Geoghegan <pg@bowt.ie> wrote:
> I think that you have to be doing a multi-level delete for a deadlock
> to take place, which isn't particularly likely to coincide with a
> concurrent insertion in general. That may explain why it's taken a
> year to get a high-quality bug report.
BTW, it's notable that Chen's query uses ON CONFLICT DO UPDATE.
Speculative insertion might cause just the wrong kind of churn,
involving continual recycling of heap TIDs.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-11-11 01:42 Peter Geoghegan <pg@bowt.ie>
parent: Peter Geoghegan <pg@bowt.ie>
2 siblings, 2 replies; 67+ messages in thread
From: Peter Geoghegan @ 2018-11-11 01:42 UTC (permalink / raw)
To: chjischj@163.com, Teodor Sigaev <teodor@sigaev.ru>; +Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Wed, Nov 7, 2018 at 5:46 PM Peter Geoghegan <pg@bowt.ie> wrote:
> Teodor: Do you think that the issue is fixable? It looks like there
> are serious issues with the design of 218f51584d5 to me. I don't think
> the general "there can't be any inserters at this subtree" thing works
> given that we have to couple buffer locks when moving right for other
> reasons. We call ginStepRight() within ginFinishSplit(), for reasons
> that date back to bug fix commit ac4ab97e from 2013 -- that detail is
> probably important, because it seems to be what breaks the subtree
> design (we don't delete in two phases or anything in GIN).
Ping?
This is a thorny problem, and I'd like to get your input soon. I
suspect that reverting 218f51584d5 may be the ultimate outcome, even
though it's a v10 feature.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re:Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-11-11 17:10 chenhj <chjischj@163.com>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 0 replies; 67+ messages in thread
From: chenhj @ 2018-11-11 17:10 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
>
>BTW, I strongly doubt that disabling hugepages has fixed anything,
>though it may have accidentally masked the problem. This is an issue
>around the basic correctness of the locking protocols used by GIN.
>
We had disable hugepages for some days, but the bug still occurs.
The attachment is the gcore files that was collected when the issue occurred on the primary, and I hope to help analyze the problem.
Regard,
Chen Huajun
Attachments:
primary_hang_20181026_52.tar.gz (application/x-gzip, 4.8M) download
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re:Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-11-11 17:33 chenhj <chjischj@163.com>
parent: Peter Geoghegan <pg@bowt.ie>
1 sibling, 1 reply; 67+ messages in thread
From: chenhj @ 2018-11-11 17:33 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi
Before we only discussed the connection hang on the primary, the connection hang on the standby database should be another problem.
When the recovery process replays the gin's delete WAL record, the order of get lwlock is not the same as the select process,
resulting in a deadlock. Accord infomation form gcore, the details are as follows:
## select process
2246(rootBlkno=2246)
|
3254(1. Held buffer=366260,LWLock=0x2aaaad50e2e4) --rightlink--> 483(2. Acquire buffer=2201739,LWLock=0x2aaab45158a4)
The ginStepRight() function in select process gets the lwlock in the order of left to right.
## recovey process
2246(2. Held buffer=7034160,LWLock=0x2aaac6c081e4,rootBlkno)
|
3254(3. Acquire buffer=366260,LWLock=0x2aaaad50e2e4) --rightlink--> 483(1. Held buffer=2201739,LWLock=0x2aaab45158a4)
But, the ginRedoDeletePage() function in recovery process gets the lwlock in the order of current to parent and to left.
So, i think inconsistent lock order in ginRedoDeletePage() is the reason for hang in the standby.
static void
ginRedoDeletePage(XLogReaderState *record)
{
XLogRecPtrlsn = record->EndRecPtr;
ginxlogDeletePage *data = (ginxlogDeletePage *) XLogRecGetData(record);
Bufferdbuffer;
Bufferpbuffer;
Bufferlbuffer;
Pagepage;
if (XLogReadBufferForRedo(record, 0, &dbuffer) == BLK_NEEDS_REDO)
{
page = BufferGetPage(dbuffer);
Assert(GinPageIsData(page));
GinPageGetOpaque(page)->flags = GIN_DELETED;
PageSetLSN(page, lsn);
MarkBufferDirty(dbuffer);
}
if (XLogReadBufferForRedo(record, 1, &pbuffer) == BLK_NEEDS_REDO)
{
page = BufferGetPage(pbuffer);
Assert(GinPageIsData(page));
Assert(!GinPageIsLeaf(page));
GinPageDeletePostingItem(page, data->parentOffset);
PageSetLSN(page, lsn);
MarkBufferDirty(pbuffer);
}
if (XLogReadBufferForRedo(record, 2, &lbuffer) == BLK_NEEDS_REDO)
{
page = BufferGetPage(lbuffer);
Assert(GinPageIsData(page));
GinPageGetOpaque(page)->rightlink = data->rightLink;
PageSetLSN(page, lsn);
MarkBufferDirty(lbuffer);
}
if (BufferIsValid(lbuffer))
UnlockReleaseBuffer(lbuffer);
if (BufferIsValid(pbuffer))
UnlockReleaseBuffer(pbuffer);
if (BufferIsValid(dbuffer))
UnlockReleaseBuffer(dbuffer);
}
The order of get lwlock in ginRedoDeletePage() may should be change from "dbuffer->pbuffer->lbuffer" to "lbuffer->dbuffer->pbuffer" . Is this right?
## How to reproduct this issue
1. modify ginxlog.c and build(add "sleep(60)" )
if (XLogReadBufferForRedo(record, 1, &pbuffer) == BLK_NEEDS_REDO)
{
page = BufferGetPage(pbuffer);
Assert(GinPageIsData(page));
Assert(!GinPageIsLeaf(page));
GinPageDeletePostingItem(page, data->parentOffset);
PageSetLSN(page, lsn);
MarkBufferDirty(pbuffer);
}
==>
if (XLogReadBufferForRedo(record, 1, &pbuffer) == BLK_NEEDS_REDO)
{
page = BufferGetPage(pbuffer);
Assert(GinPageIsData(page));
Assert(!GinPageIsLeaf(page));
GinPageDeletePostingItem(page, data->parentOffset);
PageSetLSN(page, lsn);
MarkBufferDirty(pbuffer);
sleep(60);//add for debug
}
2. run test SQL on the primary
create table tb1(id int);
create index ON tb1 using gin(id);
insert into tb1 select 1 from generate_series(1,1000000)id;
delete from tb1;
3. check recovery process in standby had enter "sleep()" branch
$ ps -ef|grep reco
postgres 13418 13417 0 22:23 ? 00:00:00 postgres: startup process recovering 00000001000000000000005E
postgres 13425 31505 0 22:23 pts/8 00:00:00 grep --color=auto reco
$ pstack 13418
#0 0x00007f2166d39650 in __nanosleep_nocancel () from /lib64/libc.so.6
#1 0x00007f2166d39504 in sleep () from /lib64/libc.so.6
#2 0x000000000048614f in ginRedoDeletePage (record=0x127cbe8) at ginxlog.c:480
#3 gin_redo (record=0x127cbe8) at ginxlog.c:732
#4 0x00000000004efec3 in StartupXLOG ()
#5 0x0000000000697c51 in StartupProcessMain ()
#6 0x00000000004fd22a in AuxiliaryProcessMain ()
#7 0x0000000000694e49 in StartChildProcess ()
#8 0x0000000000697665 in PostmasterMain ()
#9 0x00000000004766e1 in main ()
4. execute select SQL
set enable_seqscan = false;
select count(*) from tb1 where id =2;
5. check result
recovery process block in LWLock
$ pstack 13418
#0 0x00007f216775779b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1 0x00007f216775782f in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2 0x00007f21677578cb in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3 0x0000000000685df2 in PGSemaphoreLock ()
#4 0x00000000006edd64 in LWLockAcquire ()
#5 0x00000000004fa66a in XLogReadBufferForRedoExtended ()
#6 0x0000000000486161 in ginRedoDeletePage (record=0x127cbe8) at ginxlog.c:483
#7 gin_redo (record=0x127cbe8) at ginxlog.c:732
#8 0x00000000004efec3 in StartupXLOG ()
#9 0x0000000000697c51 in StartupProcessMain ()
#10 0x00000000004fd22a in AuxiliaryProcessMain ()
#11 0x0000000000694e49 in StartChildProcess ()
#12 0x0000000000697665 in PostmasterMain ()
#13 0x00000000004766e1 in main ()
and select also block in LWLock(deadlock ocuurs)
postgres=# select pid,wait_event_type,wait_event,state,query from pg_stat_activity where state<>'idle';
pid | wait_event_type | wait_event | state | query
-------+-----------------+----------------+--------+----------------------------------------------------------------------------------------------
13472 | LWLock | buffer_content | active | select count(*) from tb1 where id =1;
13526 | | | active | select pid,wait_event_type,wait_event,state,query from pg_stat_activity where state<>'idle';
(2 rows)
## appendix
### select process gcore analysis
---------------------------------------------------------------
[New LWP 14468]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `postgres: lma lma 10.246.5.80(54684) SELECT'.
#0 0x00007f7aaf72a79b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
Missing separate debuginfos, use: debuginfo-install audit-libs-2.6.5-3.el7.x86_64 bzip2-libs-1.0.6-13.el7.x86_64 cyrus-sasl-lib-2.1.26-20.el7_2.x86_64 elfutils-libelf-0.166-2.el7.x86_64 elfutils-libs-0.166-2.el7.x86_64 glibc-2.17-157.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.14.1-26.el7.x86_64 libattr-2.4.46-12.el7.x86_64 libcap-2.22-8.el7.x86_64 libcap-ng-0.7.5-4.el7.x86_64 libcom_err-1.42.9-9.el7.x86_64 libcurl-7.29.0-35.el7.centos.x86_64 libgcc-4.8.5-11.el7.x86_64 libgcrypt-1.5.3-12.el7_1.1.x86_64 libgpg-error-1.12-3.el7.x86_64 libicu-50.1.2-15.el7.x86_64 libidn-1.28-4.el7.x86_64 libselinux-2.5-6.el7.x86_64 libssh2-1.4.3-10.el7_2.1.x86_64 libstdc++-4.8.5-11.el7.x86_64 libxml2-2.9.1-6.el7_2.3.x86_64 nspr-4.11.0-1.el7_2.x86_64 nss-3.21.0-17.el7.x86_64 nss-softokn-freebl-3.16.2.3-14.4.el7.x86_64 nss-util-3.21.0-2.2.el7_2.x86_64 openldap-2.4.40-13.el7.x86_64 openssl-libs-1.0.1e-60.el7.x86_64 pam-1.1.8-18.el7.x86_64 pcre-8.32-15.el7_2.1.x86_64 systemd-libs-219-30.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) bt
#0 0x00007f7aaf72a79b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1 0x00007f7aaf72a82f in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2 0x00007f7aaf72a8cb in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3 0x000000000069d362 in PGSemaphoreLock (sema=0x2aaaaac072d8) at pg_sema.c:310
#4 0x00000000007095ac in LWLockAcquire (lock=0x2aaab45158a4, mode=LW_SHARED) at lwlock.c:1233
#5 0x0000000000490a32 in ginStepRight (buffer=366260, index=<optimized out>, lockmode=lockmode@entry=1) at ginbtree.c:174
#6 0x000000000049253c in entryLoadMoreItems (ginstate=0x1da3ad0, snapshot=0x167c798, advancePast=..., entry=0x159e218) at ginget.c:677
#7 entryGetItem (ginstate=ginstate@entry=0x1da3ad0, entry=entry@entry=0x159e218, advancePast=..., snapshot=snapshot@entry=0x167c798) at ginget.c:867
#8 0x0000000000493f21 in keyGetItem (snapshot=0x167c798, advancePast=..., key=<optimized out>, tempCtx=0x107ea68, ginstate=<optimized out>) at ginget.c:953
#9 scanGetItem (scan=<optimized out>, scan=<optimized out>, recheck=<synthetic pointer>, item=0x7fff550db820, advancePast=...) at ginget.c:1222
#10 gingetbitmap (scan=<optimized out>, tbm=<optimized out>) at ginget.c:1837
#11 0x00000000004c245a in index_getbitmap (scan=scan@entry=0x2236168, bitmap=bitmap@entry=0x2240f08) at indexam.c:726
#12 0x0000000000600c52 in MultiExecBitmapIndexScan (node=0x22358e8) at nodeBitmapIndexscan.c:104
#13 0x00000000005f40b9 in MultiExecProcNode (node=<optimized out>) at execProcnode.c:490
#14 0x00000000006004c7 in BitmapHeapNext (node=node@entry=0x1ec9788) at nodeBitmapHeapscan.c:117
#15 0x00000000005f5094 in ExecScanFetch (recheckMtd=0x5ff990 <BitmapHeapRecheck>, accessMtd=0x5ffa20 <BitmapHeapNext>, node=0x1ec9788) at execScan.c:97
#16 ExecScan (node=0x1ec9788, accessMtd=0x5ffa20 <BitmapHeapNext>, recheckMtd=0x5ff990 <BitmapHeapRecheck>) at execScan.c:164
#17 0x000000000060e10c in ExecProcNode (node=0x1ec9788) at ../../../src/include/executor/executor.h:250
#18 ExecNestLoop (pstate=0x1ec73c0) at nodeNestloop.c:160
#19 0x000000000060e05f in ExecProcNode (node=0x1ec73c0) at ../../../src/include/executor/executor.h:250
#20 ExecNestLoop (pstate=0x1ec7200) at nodeNestloop.c:109
#21 0x0000000000610636 in ExecProcNode (node=0x1ec7200) at ../../../src/include/executor/executor.h:250
#22 ExecSort (pstate=0x1ec6f90) at nodeSort.c:106
#23 0x00000000005fad89 in ExecProcNode (node=0x1ec6f90) at ../../../src/include/executor/executor.h:250
#24 fetch_input_tuple (aggstate=aggstate@entry=0x1ec6948) at nodeAgg.c:695
#25 0x00000000005fcf27 in agg_retrieve_direct (aggstate=0x1ec6948) at nodeAgg.c:2347
#26 ExecAgg (pstate=0x1ec6948) at nodeAgg.c:2158
#27 0x00000000005faa42 in ExecProcNode (node=0x1ec6948) at ../../../src/include/executor/executor.h:250
#28 ExecAppend (pstate=0x20843e8) at nodeAppend.c:222
#29 0x000000000060eee9 in ExecProcNode (node=0x20843e8) at ../../../src/include/executor/executor.h:250
#30 ExecResult (pstate=0x2084178) at nodeResult.c:115
#31 0x0000000000610636 in ExecProcNode (node=0x2084178) at ../../../src/include/executor/executor.h:250
#32 ExecSort (pstate=0x2083f08) at nodeSort.c:106
#33 0x0000000000607cfd in ExecProcNode (node=0x2083f08) at ../../../src/include/executor/executor.h:250
#34 ExecLimit (pstate=0x2083798) at nodeLimit.c:95
#35 0x00000000005ef612 in ExecProcNode (node=0x2083798) at ../../../src/include/executor/executor.h:250
#36 ExecutePlan (execute_once=<optimized out>, dest=0x10e34f8, direction=<optimized out>, numberTuples=0, sendTuples=1 '\001', operation=CMD_SELECT, use_parallel_mode=<optimized out>,
planstate=0x2083798, estate=0x20834e8) at execMain.c:1722
#37 standard_ExecutorRun (queryDesc=0x197b6e0, direction=<optimized out>, count=0, execute_once=<optimized out>) at execMain.c:363
---Type <return> to continue, or q <return> to quit---
#38 0x00007f7aa028415d in pgss_ExecutorRun (queryDesc=0x197b6e0, direction=ForwardScanDirection, count=0, execute_once=<optimized out>) at pg_stat_statements.c:889
#39 0x000000000071a28b in PortalRunSelect (portal=portal@entry=0x1112198, forward=forward@entry=1 '\001', count=0, count@entry=9223372036854775807, dest=dest@entry=0x10e34f8) at pquery.c:932
#40 0x000000000071b63f in PortalRun (portal=<optimized out>, count=9223372036854775807, isTopLevel=<optimized out>, run_once=<optimized out>, dest=0x10e34f8, altdest=0x10e34f8,
completionTag=0x7fff550dc220 "") at pquery.c:773
#41 0x0000000000718f84 in PostgresMain (argc=<optimized out>, argv=<optimized out>, dbname=<optimized out>, username=<optimized out>) at postgres.c:1984
#42 0x000000000047ad3c in BackendRun (port=0x106a590) at postmaster.c:4405
#43 BackendStartup (port=0x106a590) at postmaster.c:4077
#44 ServerLoop () at postmaster.c:1755
#45 0x00000000006afacf in PostmasterMain (argc=argc@entry=1, argv=argv@entry=0x101ba20) at postmaster.c:1363
#46 0x000000000047bb6f in main (argc=1, argv=0x101ba20) at main.c:228
(gdb) p num_held_lwlocks
$1 = 1
(gdb) p held_lwlocks
$2 = {{lock = 0x2aaaad50e2e4, mode = LW_SHARED}, {lock = 0x2aaaaac09e80, mode = LW_SHARED}, {lock = 0x2aaaaac09280, mode = LW_EXCLUSIVE}, {lock = 0x2aaaaac09280, mode = LW_EXCLUSIVE}, {
lock = 0x0, mode = LW_EXCLUSIVE} <repeats 196 times>}
(gdb) f 5
#5 0x0000000000490a32 in ginStepRight (buffer=366260, index=<optimized out>, lockmode=lockmode@entry=1) at ginbtree.c:174
174LockBuffer(nextbuffer, lockmode);
(gdb) p ((LWLock*) (&((&BufferDescriptors[(366260 - 1)].bufferdesc))->content_lock))
$3 = (LWLock *) 0x2aaaad50e2e4
(gdb) p nextbuffer
$4 = 2201739
(gdb) p ((LWLock*) (&((&BufferDescriptors[(2201739 - 1)].bufferdesc))->content_lock))
$5 = (LWLock *) 0x2aaab45158a4
#6 0x000000000049253c in entryLoadMoreItems (ginstate=0x1da3ad0, snapshot=0x167c798, advancePast=..., entry=0x159e218) at ginget.c:677
677entry->buffer = ginStepRight(entry->buffer,
(gdb) p *entry
$2 = {queryKey = 20181026, queryCategory = 0 '\000', isPartialMatch = 0 '\000', extra_data = 0x159e170 "\003", strategy = 3, searchMode = 0, attnum = 1,
buffer = 366260, curItem = {ip_blkid = {bi_hi = 0, bi_lo = 0}, ip_posid = 0}, matchBitmap = 0x0, matchIterator = 0x0, matchResult = 0x0, list = 0x0, nlist = 0,
offset = 0, isFinished = 0 '\000', reduceResult = 0 '\000', predictNumberResult = 0, btree = {findChildPage = 0x48d890 <dataLocateItem>,
getLeftMostChild = 0x48db40 <dataGetLeftMostPage>, isMoveRight = 0x48d830 <dataIsMoveRight>, findItem = 0x0, findChildPtr = 0x48da10 <dataFindChildPtr>,
beginPlaceToPage = 0x48e320 <dataBeginPlaceToPage>, execPlaceToPage = 0x48f4c0 <dataExecPlaceToPage>, prepareDownlink = 0x48dfd0 <dataPrepareDownlink>,
fillRoot = 0x48f420 <ginDataFillRoot>, isData = 1 '\001', index = 0x11f1b08, rootBlkno = 2246, ginstate = 0x0, fullScan = 1 '\001', isBuild = 0 '\000',
entryAttnum = 0, entryKey = 0, entryCategory = 0 '\000', itemptr = {ip_blkid = {bi_hi = 0, bi_lo = 0}, ip_posid = 0}}}
---------------------------------------------------------------
### recovery process gcore analysis
---------------------------------------------------------------
[New LWP 37811]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `postgres: startup process recovering 000000040001042900000050'.
#0 0x00007f7aaf72a79b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
Missing separate debuginfos, use: debuginfo-install audit-libs-2.6.5-3.el7.x86_64 bzip2-libs-1.0.6-13.el7.x86_64 cyrus-sasl-lib-2.1.26-20.el7_2.x86_64 elfutils-libelf-0.166-2.el7.x86_64 elfutils-libs-0.166-2.el7.x86_64 glibc-2.17-157.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.14.1-26.el7.x86_64 libattr-2.4.46-12.el7.x86_64 libcap-2.22-8.el7.x86_64 libcap-ng-0.7.5-4.el7.x86_64 libcom_err-1.42.9-9.el7.x86_64 libcurl-7.29.0-35.el7.centos.x86_64 libgcc-4.8.5-11.el7.x86_64 libgcrypt-1.5.3-12.el7_1.1.x86_64 libgpg-error-1.12-3.el7.x86_64 libicu-50.1.2-15.el7.x86_64 libidn-1.28-4.el7.x86_64 libselinux-2.5-6.el7.x86_64 libssh2-1.4.3-10.el7_2.1.x86_64 libstdc++-4.8.5-11.el7.x86_64 libxml2-2.9.1-6.el7_2.3.x86_64 nspr-4.11.0-1.el7_2.x86_64 nss-3.21.0-17.el7.x86_64 nss-softokn-freebl-3.16.2.3-14.4.el7.x86_64 nss-util-3.21.0-2.2.el7_2.x86_64 openldap-2.4.40-13.el7.x86_64 openssl-libs-1.0.1e-60.el7.x86_64 pam-1.1.8-18.el7.x86_64 pcre-8.32-15.el7_2.1.x86_64 systemd-libs-219-30.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) bt
#0 0x00007f7aaf72a79b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1 0x00007f7aaf72a82f in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2 0x00007f7aaf72a8cb in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3 0x000000000069d362 in PGSemaphoreLock (sema=0x2aaaaac07ff8) at pg_sema.c:310
#4 0x00000000007095ac in LWLockAcquire (lock=0x2aaaad50e2e4, mode=LW_EXCLUSIVE) at lwlock.c:1233
#5 0x0000000000503c1a in XLogReadBufferForRedoExtended (record=record@entry=0x105ce58, block_id=block_id@entry=2 '\002', mode=mode@entry=RBM_NORMAL,
get_cleanup_lock=get_cleanup_lock@entry=0 '\000', buf=buf@entry=0x7fff550d93c0) at xlogutils.c:399
#6 0x0000000000503d00 in XLogReadBufferForRedo (record=record@entry=0x105ce58, block_id=block_id@entry=2 '\002', buf=buf@entry=0x7fff550d93c0) at xlogutils.c:293
#7 0x000000000048bcd5 in ginRedoDeletePage (record=0x105ce58) at ginxlog.c:482
#8 gin_redo (record=0x105ce58) at ginxlog.c:731
#9 0x00000000004f8b83 in StartupXLOG () at xlog.c:7183
#10 0x00000000006b0091 in StartupProcessMain () at startup.c:217
#11 0x000000000050689a in AuxiliaryProcessMain (argc=argc@entry=2, argv=argv@entry=0x7fff550dcbe0) at bootstrap.c:426
#12 0x00000000006ad160 in StartChildProcess (type=StartupProcess) at postmaster.c:5361
#13 0x00000000006afaab in PostmasterMain (argc=argc@entry=1, argv=argv@entry=0x101ba20) at postmaster.c:1355
#14 0x000000000047bb6f in main (argc=1, argv=0x101ba20) at main.c:228
(gdb) p num_held_lwlocks
$1 = 2
(gdb) p held_lwlocks
$2 = {{lock = 0x2aaab45158a4, mode = LW_EXCLUSIVE}, {lock = 0x2aaac6c081e4, mode = LW_EXCLUSIVE}, {lock = 0x2aaaaac0a780, mode = LW_SHARED}, {lock = 0x2aaabf16e7e4, mode = LW_EXCLUSIVE}, {
lock = 0x2aaaaac08a00, mode = LW_EXCLUSIVE}, {lock = 0x0, mode = LW_EXCLUSIVE} <repeats 195 times>}
(gdb) f 7
#7 0x000000000048bcd5 in ginRedoDeletePage (record=0x105ce58) at ginxlog.c:482
482if (XLogReadBufferForRedo(record, 2, &lbuffer) == BLK_NEEDS_REDO)
(gdb) p *record
$6 = {read_page = 0x4f40f0 <XLogPageRead>, system_identifier = 6605939872368807293, private_data = 0x7fff550d94a0, ReadRecPtr = 286050462744944, EndRecPtr = 286050462745016,
decoded_record = 0x105dfd8, main_data = 0x11026f8 "\002", main_data_len = 8, main_data_bufsz = 53792, record_origin = 0, blocks = {{in_use = 1 '\001', rnode = {spcNode = 1663,
dbNode = 16400, relNode = 2079769372}, forknum = MAIN_FORKNUM, blkno = 483, flags = 0 '\000', has_image = 0 '\000', apply_image = 0 '\000', bkp_image = 0x105e7d5 "\232\204",
hole_offset = 256, hole_length = 7680, bimg_len = 222, bimg_info = 7 '\a', has_data = 0 '\000', data = 0x1084ea8 "", data_len = 0, data_bufsz = 8100}, {in_use = 1 '\001', rnode = {
spcNode = 1663, dbNode = 16400, relNode = 2079769372}, forknum = MAIN_FORKNUM, blkno = 2246, flags = 128 '\200', has_image = 0 '\000', apply_image = 0 '\000',
bkp_image = 0x10b9168 "", hole_offset = 4084, hole_length = 4100, bimg_len = 4092, bimg_info = 5 '\005', has_data = 0 '\000', data = 0x1081a58 "\001", data_len = 0,
data_bufsz = 4496}, {in_use = 1 '\001', rnode = {spcNode = 1663, dbNode = 16400, relNode = 2079769372}, forknum = MAIN_FORKNUM, blkno = 3253, flags = 128 '\200', has_image = 0 '\000',
apply_image = 0 '\000',
bkp_image = 0x10b943f "\001\351\017\021\203\020\361\017\201\020\377\017\374\017\n\003\003\002\210\020\345\037\025\003\352\017\227\020\364\037\361\037\233\020\362\017\022\200\020\354\037\t\360/\027\351\017\002\016\203\020\370\017\001\001\367\017\021\001\361\017\r\364\017\020\365\017\003\200\020\003\205\020\357\017\374\017", hole_offset = 500, hole_length = 60,
bimg_len = 1421, bimg_info = 7 '\a', has_data = 0 '\000', data = 0x1083da8 "\234\001", data_len = 0, data_bufsz = 16}, {in_use = 0 '\000', rnode = {spcNode = 1663, dbNode = 16400,
relNode = 2080872819}, forknum = MAIN_FORKNUM, blkno = 6758, flags = 192 '\300', has_image = 0 '\000', apply_image = 0 '\000', bkp_image = 0x0, hole_offset = 0, hole_length = 0,
bimg_len = 0, bimg_info = 0 '\000', has_data = 0 '\000', data = 0x0, data_len = 0, data_bufsz = 0}, {in_use = 0 '\000', rnode = {spcNode = 1663, dbNode = 16400, relNode = 2080872819},
forknum = MAIN_FORKNUM, blkno = 6780, flags = 192 '\300', has_image = 0 '\000', apply_image = 0 '\000', bkp_image = 0x0, hole_offset = 0, hole_length = 0, bimg_len = 0,
bimg_info = 0 '\000', has_data = 0 '\000', data = 0x10e7038 "\017\001", data_len = 0, data_bufsz = 16}, {in_use = 0 '\000', rnode = {spcNode = 1663, dbNode = 16400,
relNode = 2080872819}, forknum = MAIN_FORKNUM, blkno = 6781, flags = 192 '\300', has_image = 0 '\000', apply_image = 0 '\000', bkp_image = 0x0, hole_offset = 0, hole_length = 0,
bimg_len = 0, bimg_info = 0 '\000', has_data = 0 '\000', data = 0x0, data_len = 0, data_bufsz = 0}, {in_use = 0 '\000', rnode = {spcNode = 1663, dbNode = 16400, relNode = 2080872819},
forknum = MAIN_FORKNUM, blkno = 6792, flags = 192 '\300', has_image = 0 '\000', apply_image = 0 '\000', bkp_image = 0x0, hole_offset = 0, hole_length = 0, bimg_len = 0,
bimg_info = 0 '\000', has_data = 0 '\000', data = 0x0, data_len = 0, data_bufsz = 0}, {in_use = 0 '\000', rnode = {spcNode = 1663, dbNode = 16400, relNode = 2080872819},
forknum = MAIN_FORKNUM, blkno = 6824, flags = 192 '\300', has_image = 0 '\000', apply_image = 0 '\000', bkp_image = 0x0, hole_offset = 0, hole_length = 0, bimg_len = 0,
bimg_info = 0 '\000', has_data = 0 '\000', data = 0x0, data_len = 0, data_bufsz = 0}, {in_use = 0 '\000', rnode = {spcNode = 1663, dbNode = 16400, relNode = 2080872819},
forknum = MAIN_FORKNUM, blkno = 6832, flags = 192 '\300', has_image = 0 '\000', apply_image = 0 '\000', bkp_image = 0x0, hole_offset = 0, hole_length = 0, bimg_len = 0,
bimg_info = 0 '\000', has_data = 0 '\000', data = 0x0, data_len = 0, data_bufsz = 0}, {in_use = 0 '\000', rnode = {spcNode = 1663, dbNode = 16400, relNode = 2080872819},
forknum = MAIN_FORKNUM, blkno = 6833, flags = 192 '\300', has_image = 0 '\000', apply_image = 0 '\000', bkp_image = 0x0, hole_offset = 0, hole_length = 0, bimg_len = 0,
bimg_info = 0 '\000', has_data = 0 '\000', data = 0x0, data_len = 0, data_bufsz = 0}, {in_use = 0 '\000', rnode = {spcNode = 1663, dbNode = 16400, relNode = 2080872819},
forknum = MAIN_FORKNUM, blkno = 6903, flags = 192 '\300', has_image = 0 '\000', apply_image = 0 '\000', bkp_image = 0x0, hole_offset = 0, hole_length = 0, bimg_len = 0,
bimg_info = 0 '\000', has_data = 0 '\000', data = 0x0, data_len = 0, data_bufsz = 0}, {in_use = 0 '\000', rnode = {spcNode = 1663, dbNode = 16400, relNode = 2080872819},
forknum = MAIN_FORKNUM, blkno = 262, flags = 192 '\300', has_image = 0 '\000', apply_image = 0 '\000', bkp_image = 0x0, hole_offset = 0, hole_length = 0, bimg_len = 0,
bimg_info = 0 '\000', has_data = 0 '\000', data = 0x0, data_len = 0, data_bufsz = 0}, {in_use = 0 '\000', rnode = {spcNode = 1663, dbNode = 16400, relNode = 2080872819},
forknum = MAIN_FORKNUM, blkno = 8759, flags = 192 '\300', has_image = 0 '\000', apply_image = 0 '\000', bkp_image = 0x0, hole_offset = 0, hole_length = 0, bimg_len = 0,
bimg_info = 0 '\000', has_data = 0 '\000', data = 0x0, data_len = 0, data_bufsz = 0}, {in_use = 0 '\000', rnode = {spcNode = 1663, dbNode = 16400, relNode = 2080872819},
forknum = MAIN_FORKNUM, blkno = 8766, flags = 192 '\300', has_image = 0 '\000', apply_image = 0 '\000', bkp_image = 0x0, hole_offset = 0, hole_length = 0, bimg_len = 0,
bimg_info = 0 '\000', has_data = 0 '\000', data = 0x0, data_len = 0, data_bufsz = 0}, {in_use = 0 '\000', rnode = {spcNode = 1663, dbNode = 16400, relNode = 2080872819},
forknum = MAIN_FORKNUM, blkno = 8927, flags = 192 '\300', has_image = 0 '\000', apply_image = 0 '\000', bkp_image = 0x0, hole_offset = 0, hole_length = 0, bimg_len = 0,
bimg_info = 0 '\000', has_data = 0 '\000', data = 0x0, data_len = 0, data_bufsz = 0}, {in_use = 0 '\000', rnode = {spcNode = 1663, dbNode = 16400, relNode = 2080872819},
forknum = MAIN_FORKNUM, blkno = 8941, flags = 192 '\300', has_image = 0 '\000', apply_image = 0 '\000', bkp_image = 0x0, hole_offset = 0, hole_length = 0, bimg_len = 0,
bimg_info = 0 '\000', has_data = 0 '\000', data = 0x0, data_len = 0, data_bufsz = 0}, {in_use = 0 '\000', rnode = {spcNode = 1663, dbNode = 16400, relNode = 2080872819},
forknum = MAIN_FORKNUM, blkno = 9207, flags = 192 '\300', has_image = 0 '\000', apply_image = 0 '\000', bkp_image = 0x0, hole_offset = 0, hole_length = 0, bimg_len = 0,
bimg_info = 0 '\000', has_data = 0 '\000', data = 0x0, data_len = 0, data_bufsz = 0}, {in_use = 0 '\000', rnode = {spcNode = 0, dbNode = 0, relNode = 0}, forknum = MAIN_FORKNUM,
---Type <return> to continue, or q <return> to quit---
blkno = 0, flags = 0 '\000', has_image = 0 '\000', apply_image = 0 '\000', bkp_image = 0x0, hole_offset = 0, hole_length = 0, bimg_len = 0, bimg_info = 0 '\000', has_data = 0 '\000',
data = 0x0, data_len = 0, data_bufsz = 0} <repeats 16 times>}, max_block_id = 2, readBuf = 0x105de68 "\227\320\005", readLen = 8192, readSegNo = 17049936, readOff = 3686400,
readPageTLI = 4, latestPagePtr = 286050462744576, latestPageTLI = 4, currRecPtr = 286050462744944, currTLI = 0, currTLIValidUntil = 0, nextTLI = 0, readRecordBuf = 0x10b80f8 "F",
readRecordBufSize = 57344, errormsg_buf = 0x105fe78 ""}
#7 0x000000000048bcd5 in ginRedoDeletePage (record=0x105ce58) at ginxlog.c:482
482if (XLogReadBufferForRedo(record, 2, &lbuffer) == BLK_NEEDS_REDO)
(gdb) p dbuffer
$1 = 2201739
(gdb) p pbuffer
$2 = 7034160
(gdb) p lbuffer
$3 = 366260
(gdb) p ((LWLock*) (&((&BufferDescriptors[(2201739 - 1)].bufferdesc))->content_lock))
$4 = (LWLock *) 0x2aaab45158a4
(gdb) p ((LWLock*) (&((&BufferDescriptors[(7034160 - 1)].bufferdesc))->content_lock))
$5 = (LWLock *) 0x2aaac6c081e4
(gdb) p ((LWLock*) (&((&BufferDescriptors[(366260 - 1)].bufferdesc))->content_lock))
$6 = (LWLock *) 0x2aaaad50e2e4
---------------------------------------------------------------
Regard,
Chen Huajun
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re:Re: Connections hang indefinitely while taking a gin index's
LWLock buffer_content lock
@ 2018-11-12 01:46 chjischj@163.com <chjischj@163.com>
parent: chenhj <chjischj@163.com>
0 siblings, 0 replies; 67+ messages in thread
From: chjischj@163.com @ 2018-11-12 01:46 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"></head><body><div class="quote" style="line-height: 1.5"><br></div><div class="quote" style="line-height: 1.5">There's a mistake in the last mail.</div><div class="quote" style="line-height: 1.5">About how to reproduct the issue,the following SQL should be:</div><div class="quote" style="line-height: 1.5"><br></div><div class="quote" style="line-height: 1.5">set enable_seqscan = false;</div><div class="quote" style="line-height: 1.5"><chjischj@163.com><pg@bowt.ie><teodor@sigaev.ru><pgsql-hackers@lists.postgresql.org>select count(*) from tb1 where id =1;<br><br><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial"><br><div>4. execute select SQL</div><div><br></div><div> set enable_seqscan = false;</div><div> select count(*) from tb1 where id =2;</div><div><br></div><div>Regard,</div><div>Chen Huajun</div></div></blockquote></pgsql-hackers@lists.postgresql.org></teodor@sigaev.ru></pg@bowt.ie></chjischj@163.com></div></body></html>
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-11-14 18:11 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Peter Geoghegan <pg@bowt.ie>
2 siblings, 1 reply; 67+ messages in thread
From: Andrey Borodin @ 2018-11-14 18:11 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: chjischj@163.com, Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
<html><head></head><body dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div class="ApplePlainTextBody">Hi everyone!<br><br>I didn't noticed this thread for too long somehow, sorry.<br><br><blockquote type="cite">8 нояб. 2018 г., в 6:46, Peter Geoghegan <pg@bowt.ie> написал(а):<br><br>I don't think<br>the general "there can't be any inserters at this subtree" thing works<br>given that we have to couple buffer locks when moving right for other<br>reasons. We call ginStepRight() within ginFinishSplit(), for reasons<br>that date back to bug fix commit ac4ab97e from 2013 -- that detail is<br>probably important, because it seems to be what breaks the subtree<br>design (we don't delete in two phases or anything in GIN).<br></blockquote><br>ginVacuumPostingTreeLeaves() holds LockBufferForCleanup() on subtree root b.<br>Thus there may be no GinBtreeStack's holding pin on b at the moment.<br>When you ginStepRight(b) to the parent in ginFinishSplit(), you always get to the buffer from your stack.<br>Hence you can never have ginFinishSplit() deadlock with cleanup of subtree whose root is LockBufferForCleanup()'d.<br><br>Is this correct or did I miss something?<br><br>But we have a deadlock at hand, I'll think more about it. Something with locking protocol is clearly wrong.<br><br><blockquote type="cite">11 нояб. 2018 г., в 22:33, chenhj <chjischj@163.com> написал(а):<br><br>The order of get lwlock in ginRedoDeletePage() may should be change from "dbuffer->pbuffer->lbuffer" to "lbuffer->dbuffer->pbuffer" . Is this right?<br></blockquote><br>This looks correct to me.<br><br>Best regards, Andrey Borodin.</div></body></html>
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-11-15 12:30 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 67+ messages in thread
From: Andrey Borodin @ 2018-11-15 12:30 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: chjischj@163.com <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi!
Mail.App somehow borken my reponse with <div class="ApplePlainTextBody">, which is far beyond of my understanding of what is plain text, that's why I'll quote all my previous message here. Hope this mail client is OK.
14.11.2018, 23:13, "Andrey Borodin" <x4mmm@yandex-team.ru>:
> Hi everyone!
>
> I didn't noticed this thread for too long somehow, sorry.
>
>> 8 нояб. 2018 г., в 6:46, Peter Geoghegan <pg@bowt.ie> написал(а):
>>
>> I don't think
>> the general "there can't be any inserters at this subtree" thing works
>> given that we have to couple buffer locks when moving right for other
>> reasons. We call ginStepRight() within ginFinishSplit(), for reasons
>> that date back to bug fix commit ac4ab97e from 2013 -- that detail is
>> probably important, because it seems to be what breaks the subtree
>> design (we don't delete in two phases or anything in GIN).
>
> ginVacuumPostingTreeLeaves() holds LockBufferForCleanup() on subtree root b.
> Thus there may be no GinBtreeStack's holding pin on b at the moment.
> When you ginStepRight(b) to the parent in ginFinishSplit(), you always get to the buffer from your stack.
> Hence you can never have ginFinishSplit() deadlock with cleanup of subtree whose root is LockBufferForCleanup()'d.
>
> Is this correct or did I miss something?
>
> But we have a deadlock at hand, I'll think more about it. Something with locking protocol is clearly wrong.
>
>> 11 нояб. 2018 г., в 22:33, chenhj <chjischj@163.com> написал(а):
>>
>> The order of get lwlock in ginRedoDeletePage() may should be change from "dbuffer->pbuffer->lbuffer" to "lbuffer->dbuffer->pbuffer" . Is this right?
>
> This looks correct to me.
>
> Best regards, Andrey Borodin.
I've reread Chen's reports and understood his findings.
> When you ginStepRight(b) to the parent in ginFinishSplit(), you always get to the buffer from your stack.
This statement does not hold.
When you have a GinBtreeStack S, one of it's internal pages may split into new page P. If in this moment you start a multi-level delete from P and cascade split from S, P is not in S and we may deadlock.
Correct solution seems to replace lock-coupling descent by usual B-tree searches for parent pages as in B-tree.
I think I can compose patch for consideration.
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-11-23 10:18 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 67+ messages in thread
From: Andrey Borodin @ 2018-11-23 10:18 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: chjischj@163.com <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi!
> 15 нояб. 2018 г., в 17:30, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
>
> I think I can compose patch for consideration.
I found no practical way to fix concept of "subtree-lock". Pre-locking all left siblings for cleanup would suffice, but does not look very practical.
GIN Posting tree has no all useful B-tree inventory like left links and high keys for concurrent deletes without "subtree-lock".
Let's consider concurrent deletion.
As far as I understand, absence of high-key makes unsafe concurrent deletion of highest keys from internal pages. Thus, internal pages cannot be deleted at all.
PFA patch with most simple deletion algorithm:
1. Descend to leftmost leaf
2. Scan by rightlinks until rightmost page
3. Delete empty leaf pages, if they are not highest keys in corresponding parent page
Parent page is found by rightlink scan for leftmost parent with cached previous result.
Deleted page is stamped by ReadNewTransactionId()'s xid, page cannot be reused until deletion xid is beyond RecentGlobalDataXmin.
DeleteXid is WAL-logged.
I've refactored btree->getLeftMostChild() and btree->findChildPtr() to reuse their code (removed unnecessary parameter).
Please note, that attached patch do not fix 2nd bug found by Chen in page delete redo.
I understand that reverting commit 218f51584d5 and returning to long but finite root locks is safest solution. In this description of proposed patch I've tried to put concepts in a minimum of words. Please ping me if some concepts sound cryptic and require more clarification.
Best regards, Andrey Borodin.
Attachments:
0001-Use-correct-locking-protocol-during-GIN-posting-tree.patch (application/octet-stream, 20.5K) download
From 71e5703357fc99028cfeb8404eae99d98f62b36f Mon Sep 17 00:00:00 2001
From: Andrey <amborodin@acm.org>
Date: Thu, 22 Nov 2018 23:14:46 +0500
Subject: [PATCH] Use correct locking protocol during GIN posting tree vacuum
---
src/backend/access/gin/README | 17 +-
src/backend/access/gin/ginbtree.c | 6 +-
src/backend/access/gin/gindatapage.c | 10 +-
src/backend/access/gin/ginentrypage.c | 6 +-
src/backend/access/gin/ginutil.c | 7 +-
src/backend/access/gin/ginvacuum.c | 321 +++++++++++---------------
src/backend/access/gin/ginxlog.c | 1 +
src/include/access/gin_private.h | 9 +-
src/include/access/ginxlog.h | 1 +
9 files changed, 159 insertions(+), 219 deletions(-)
diff --git a/src/backend/access/gin/README b/src/backend/access/gin/README
index cc434b1feb..947c8355e5 100644
--- a/src/backend/access/gin/README
+++ b/src/backend/access/gin/README
@@ -311,20 +311,9 @@ insert location). So as long as we don't delete the leftmost page on each
level, a search can never follow a downlink to page that's about to be
deleted.
-The previous paragraph's reasoning only applies to searches, and only to
-posting trees. To protect from inserters following a downlink to a deleted
-page, vacuum simply locks out all concurrent insertions to the posting tree,
-by holding a super-exclusive lock on the parent page of subtree with deletable
-pages. Inserters hold a pin on the root page, but searches do not, so while
-new searches cannot begin while root page is locked, any already-in-progress
-scans can continue concurrently with vacuum in corresponding subtree of
-posting tree. To exclude interference with readers vacuum takes exclusive
-locks in a depth-first scan in left-to-right order of page tuples. Leftmost
-page is never deleted. Thus before deleting any page we obtain exclusive
-lock on any left page, effectively excluding deadlock with any reader, despite
-taking parent lock before current and left lock after current. We take left
-lock not for a concurrency reasons, but rather in need to mark page dirty.
-In the entry tree, we never delete pages.
+During vacuum of posting tree leaf pages of posting tree are scanned with
+lock coupling, iterating by rightlink. If leaf if empty, we try to lookup
+parent page for it and delete it.
(This is quite different from the mechanism the btree indexam uses to make
page-deletions safe; it stamps the deleted pages with an XID and keeps the
diff --git a/src/backend/access/gin/ginbtree.c b/src/backend/access/gin/ginbtree.c
index d5a568106c..ec59c342d9 100644
--- a/src/backend/access/gin/ginbtree.c
+++ b/src/backend/access/gin/ginbtree.c
@@ -271,9 +271,9 @@ ginFindParents(GinBtree btree, GinBtreeStack *stack)
ginFinishSplit(btree, ptr, false, NULL);
}
- leftmostBlkno = btree->getLeftMostChild(btree, page);
+ leftmostBlkno = btree->getLeftMostChild(page);
- while ((offset = btree->findChildPtr(btree, page, stack->blkno, InvalidOffsetNumber)) == InvalidOffsetNumber)
+ while ((offset = btree->findChildPtr(page, stack->blkno, InvalidOffsetNumber)) == InvalidOffsetNumber)
{
blkno = GinPageGetOpaque(page)->rightlink;
if (blkno == InvalidBlockNumber)
@@ -708,7 +708,7 @@ ginFinishSplit(GinBtree btree, GinBtreeStack *stack, bool freestack,
/* move right if it's needed */
page = BufferGetPage(parent->buffer);
- while ((parent->off = btree->findChildPtr(btree, page, stack->blkno, parent->off)) == InvalidOffsetNumber)
+ while ((parent->off = btree->findChildPtr(page, stack->blkno, parent->off)) == InvalidOffsetNumber)
{
if (GinPageRightMost(page))
{
diff --git a/src/backend/access/gin/gindatapage.c b/src/backend/access/gin/gindatapage.c
index 9f20513811..529269a2ad 100644
--- a/src/backend/access/gin/gindatapage.c
+++ b/src/backend/access/gin/gindatapage.c
@@ -262,7 +262,7 @@ dataLocateItem(GinBtree btree, GinBtreeStack *stack)
{
stack->off = FirstOffsetNumber;
stack->predictNumber *= GinPageGetOpaque(page)->maxoff;
- return btree->getLeftMostChild(btree, page);
+ return btree->getLeftMostChild(page);
}
low = FirstOffsetNumber;
@@ -312,8 +312,8 @@ dataLocateItem(GinBtree btree, GinBtreeStack *stack)
/*
* Find link to blkno on non-leaf page, returns offset of PostingItem
*/
-static OffsetNumber
-dataFindChildPtr(GinBtree btree, Page page, BlockNumber blkno, OffsetNumber storedOff)
+OffsetNumber
+dataFindChildPtr(Page page, BlockNumber blkno, OffsetNumber storedOff)
{
OffsetNumber i,
maxoff = GinPageGetOpaque(page)->maxoff;
@@ -357,8 +357,8 @@ dataFindChildPtr(GinBtree btree, Page page, BlockNumber blkno, OffsetNumber stor
/*
* Return blkno of leftmost child
*/
-static BlockNumber
-dataGetLeftMostPage(GinBtree btree, Page page)
+BlockNumber
+dataGetLeftMostPage(Page page)
{
PostingItem *pitem;
diff --git a/src/backend/access/gin/ginentrypage.c b/src/backend/access/gin/ginentrypage.c
index 184cc0af3e..df3868820a 100644
--- a/src/backend/access/gin/ginentrypage.c
+++ b/src/backend/access/gin/ginentrypage.c
@@ -284,7 +284,7 @@ entryLocateEntry(GinBtree btree, GinBtreeStack *stack)
{
stack->off = FirstOffsetNumber;
stack->predictNumber *= PageGetMaxOffsetNumber(page);
- return btree->getLeftMostChild(btree, page);
+ return btree->getLeftMostChild(page);
}
low = FirstOffsetNumber;
@@ -403,7 +403,7 @@ entryLocateLeafEntry(GinBtree btree, GinBtreeStack *stack)
}
static OffsetNumber
-entryFindChildPtr(GinBtree btree, Page page, BlockNumber blkno, OffsetNumber storedOff)
+entryFindChildPtr(Page page, BlockNumber blkno, OffsetNumber storedOff)
{
OffsetNumber i,
maxoff = PageGetMaxOffsetNumber(page);
@@ -444,7 +444,7 @@ entryFindChildPtr(GinBtree btree, Page page, BlockNumber blkno, OffsetNumber sto
}
static BlockNumber
-entryGetLeftMostPage(GinBtree btree, Page page)
+entryGetLeftMostPage(Page page)
{
IndexTuple itup;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 0a32182dd7..bf354417ee 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -314,7 +314,12 @@ GinNewBuffer(Relation index)
if (PageIsNew(page))
return buffer; /* OK to use, if never initialized */
- if (GinPageIsDeleted(page))
+ /*
+ * We should not reuse page until every transaction started before
+ * deletion is over
+ */
+ if (GinPageIsDeleted(page)
+ && TransactionIdPrecedes(GinPageGetDeleteXid(page), RecentGlobalDataXmin))
return buffer; /* OK to use */
LockBuffer(buffer, GIN_UNLOCK);
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index 3104bc12b6..217e214982 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -126,33 +126,14 @@ typedef struct DataPageDeleteStack
* Delete a posting tree page.
*/
static void
-ginDeletePage(GinVacuumState *gvs, BlockNumber deleteBlkno, BlockNumber leftBlkno,
- BlockNumber parentBlkno, OffsetNumber myoff, bool isParentRoot)
+ginDeletePage(GinVacuumState *gvs, BlockNumber deleteBlkno, Buffer dBuffer,
+ Buffer lBuffer, Buffer pBuffer, OffsetNumber myoff)
{
- Buffer dBuffer;
- Buffer lBuffer;
- Buffer pBuffer;
Page page,
parentPage;
BlockNumber rightlink;
- /*
- * This function MUST be called only if someone of parent pages hold
- * exclusive cleanup lock. This guarantees that no insertions currently
- * happen in this subtree. Caller also acquire Exclusive lock on deletable
- * page and is acquiring and releasing exclusive lock on left page before.
- * Left page was locked and released. Then parent and this page are
- * locked. We acquire left page lock here only to mark page dirty after
- * changing right pointer.
- */
- lBuffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM, leftBlkno,
- RBM_NORMAL, gvs->strategy);
- dBuffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM, deleteBlkno,
- RBM_NORMAL, gvs->strategy);
- pBuffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM, parentBlkno,
- RBM_NORMAL, gvs->strategy);
-
- LockBuffer(lBuffer, GIN_EXCLUSIVE);
+ /* We epect that buffers are correctly locked */
page = BufferGetPage(dBuffer);
rightlink = GinPageGetOpaque(page)->rightlink;
@@ -169,6 +150,9 @@ ginDeletePage(GinVacuumState *gvs, BlockNumber deleteBlkno, BlockNumber leftBlkn
page = BufferGetPage(lBuffer);
GinPageGetOpaque(page)->rightlink = rightlink;
+ /* for each deletead page we remember last xid which could knew it's address */
+ GinPageSetDeleteXid(page, ReadNewTransactionId());
+
/* Delete downlink from parent */
parentPage = BufferGetPage(pBuffer);
#ifdef USE_ASSERT_CHECKING
@@ -213,6 +197,7 @@ ginDeletePage(GinVacuumState *gvs, BlockNumber deleteBlkno, BlockNumber leftBlkn
data.parentOffset = myoff;
data.rightLink = GinPageGetOpaque(page)->rightlink;
+ data.deleteXid = GinPageGetDeleteXid(page);
XLogRegisterData((char *) &data, sizeof(ginxlogDeletePage));
@@ -222,215 +207,169 @@ ginDeletePage(GinVacuumState *gvs, BlockNumber deleteBlkno, BlockNumber leftBlkn
PageSetLSN(BufferGetPage(lBuffer), recptr);
}
- ReleaseBuffer(pBuffer);
- UnlockReleaseBuffer(lBuffer);
- ReleaseBuffer(dBuffer);
+ UnlockReleaseBuffer(pBuffer);
+ UnlockReleaseBuffer(dBuffer);
END_CRIT_SECTION();
gvs->result->pages_deleted++;
}
-
-/*
- * scans posting tree and deletes empty pages
- */
-static bool
-ginScanToDelete(GinVacuumState *gvs, BlockNumber blkno, bool isRoot,
- DataPageDeleteStack *parent, OffsetNumber myoff)
+/* Descent from root to leftmost leaf page, remembering parent */
+static Buffer
+getPostingTreeLeftmostLeaf(GinVacuumState *gvs,
+ BlockNumber blkno,
+ BlockNumber *leftmost_parent_blkno)
{
- DataPageDeleteStack *me;
- Buffer buffer;
- Page page;
- bool meDelete = false;
- bool isempty;
-
- if (isRoot)
- {
- me = parent;
- }
- else
+ *leftmost_parent_blkno = InvalidBlockNumber;
+ /* Probably, we can have sanity check here? */
+ while (true)
{
- if (!parent->child)
- {
- me = (DataPageDeleteStack *) palloc0(sizeof(DataPageDeleteStack));
- me->parent = parent;
- parent->child = me;
- me->leftBlkno = InvalidBlockNumber;
- }
- else
- me = parent->child;
- }
-
- buffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM, blkno,
- RBM_NORMAL, gvs->strategy);
-
- if (!isRoot)
- LockBuffer(buffer, GIN_EXCLUSIVE);
-
- page = BufferGetPage(buffer);
-
- Assert(GinPageIsData(page));
-
- if (!GinPageIsLeaf(page))
- {
- OffsetNumber i;
+ Buffer buffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM,
+ blkno,
+ RBM_NORMAL, gvs->strategy);
+ Page page = BufferGetPage(buffer);
+ LockBuffer(buffer, GIN_SHARE);
- me->blkno = blkno;
- for (i = FirstOffsetNumber; i <= GinPageGetOpaque(page)->maxoff; i++)
+ if (GinPageIsLeaf(page))
{
- PostingItem *pitem = GinDataPageGetPostingItem(page, i);
-
- if (ginScanToDelete(gvs, PostingItemGetBlockNumber(pitem), false, me, i))
- i--;
+ LockBuffer(buffer, GIN_UNLOCK);
+ LockBufferForCleanup(buffer);
+ /* Root can become internal page, recheck */
+ if (GinPageIsLeaf(page))
+ {
+ return buffer;
+ }
}
- }
- if (GinPageIsLeaf(page))
- isempty = GinDataLeafPageIsEmpty(page);
- else
- isempty = GinPageGetOpaque(page)->maxoff < FirstOffsetNumber;
+ *leftmost_parent_blkno = blkno;
+ blkno = dataGetLeftMostPage(page);
+ UnlockReleaseBuffer(buffer);
+ }
+}
- if (isempty)
+/*
+ * Find parent for leaf page using information about previously known parent
+ * Returned buffer must be exclusively locked
+ */
+static Buffer tryFindParentForLeaf(GinVacuumState *gvs,
+ BlockNumber *parent_cache,
+ BlockNumber leaf_blkno,
+ OffsetNumber *myoff)
+{
+ Buffer buffer;
+ Page page;
+ OffsetNumber offnum;
+ /* Potentially we scan through all lowest-level internal pages */
+ while (true)
{
- /* we never delete the left- or rightmost branch */
- if (me->leftBlkno != InvalidBlockNumber && !GinPageRightMost(page))
+ /* we have nowhere to search anymore */
+ if (!BlockNumberIsValid(*parent_cache))
+ return InvalidBuffer;
+
+ buffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM,
+ *parent_cache,
+ RBM_NORMAL, gvs->strategy);
+ page = BufferGetPage(buffer);
+ LockBuffer(buffer, GIN_EXCLUSIVE);
+ Assert(!GinPageIsLeaf(page));
+
+ /* try to find downlink to deletable page */
+ offnum = dataFindChildPtr(page, leaf_blkno, InvalidOffsetNumber);
+ if (OffsetNumberIsValid(offnum))
{
- Assert(!isRoot);
- ginDeletePage(gvs, blkno, me->leftBlkno, me->parent->blkno, myoff, me->parent->isRoot);
- meDelete = true;
+ /*
+ * we should not delete highest keys on page
+ * posting tree uses them for concurrency reasons
+ * instead of highkeys
+ */
+ if (offnum == PageGetMaxOffsetNumber(page))
+ {
+ UnlockReleaseBuffer(buffer);
+ return InvalidBuffer;
+ }
+ /* OK, return it */
+ *myoff = offnum;
+ return buffer;
}
+ /*
+ * downlink to deletable page not found - try next
+ * in case of races vacuum will not be able to delete leaves,
+ * but this is acceptable
+ */
+ *parent_cache = GinPageGetOpaque(page)->rightlink;
+ UnlockReleaseBuffer(buffer);
}
-
- if (!isRoot)
- LockBuffer(buffer, GIN_UNLOCK);
-
- ReleaseBuffer(buffer);
-
- if (!meDelete)
- me->leftBlkno = blkno;
-
- return meDelete;
}
-
-/*
- * Scan through posting tree, delete empty tuples from leaf pages.
- * Also, this function collects empty subtrees (with all empty leafs).
- * For parents of these subtrees CleanUp lock is taken, then we call
- * ScanToDelete. This is done for every inner page, which points to
- * empty subtree.
- */
-static bool
-ginVacuumPostingTreeLeaves(GinVacuumState *gvs, BlockNumber blkno, bool isRoot)
+static void
+ginVacuumPostingTree(GinVacuumState *gvs, BlockNumber rootBlkno)
{
- Buffer buffer;
- Page page;
- bool hasVoidPage = false;
+ BlockNumber parentBlkno;
+ Buffer buffer, prevBuffer;
+ Page page, prevPage;
MemoryContext oldCxt;
- buffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM, blkno,
- RBM_NORMAL, gvs->strategy);
- page = BufferGetPage(buffer);
+ /*
+ * Find leftmost page, then move right with lock coupling to guarantee that all
+ * tuples are removed.
+ */
+ prevBuffer = getPostingTreeLeftmostLeaf(gvs, rootBlkno, &parentBlkno);
+ prevPage = BufferGetPage(prevBuffer);
- ginTraverseLock(buffer, false);
+ Assert(GinPageIsLeaf(prevPage));
- Assert(GinPageIsData(page));
+ oldCxt = MemoryContextSwitchTo(gvs->tmpCxt);
+ /* leftmost page is undeletable, just clean it */
+ ginVacuumPostingTreeLeaf(gvs->index, prevBuffer, gvs);
+ MemoryContextSwitchTo(oldCxt);
+ MemoryContextReset(gvs->tmpCxt);
- if (GinPageIsLeaf(page))
+ while (!GinPageRightMost(prevPage))
{
+ BlockNumber blkno = GinPageGetOpaque(prevPage)->rightlink;
+ buffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM,
+ blkno,
+ RBM_NORMAL, gvs->strategy);
+ page = BufferGetPage(buffer);
+ LockBufferForCleanup(buffer);
+
+ /* Cleanup tuples on page */
oldCxt = MemoryContextSwitchTo(gvs->tmpCxt);
- ginVacuumPostingTreeLeaf(gvs->index, buffer, gvs);
+ ginVacuumPostingTreeLeaf(gvs->index, prevBuffer, gvs);
MemoryContextSwitchTo(oldCxt);
MemoryContextReset(gvs->tmpCxt);
- /* if root is a leaf page, we don't desire further processing */
- if (GinDataLeafPageIsEmpty(page))
- hasVoidPage = true;
-
- UnlockReleaseBuffer(buffer);
-
- return hasVoidPage;
- }
- else
- {
- OffsetNumber i;
- bool hasEmptyChild = false;
- bool hasNonEmptyChild = false;
- OffsetNumber maxoff = GinPageGetOpaque(page)->maxoff;
- BlockNumber *children = palloc(sizeof(BlockNumber) * (maxoff + 1));
-
- /*
- * Read all children BlockNumbers. Not sure it is safe if there are
- * many concurrent vacuums.
- */
-
- for (i = FirstOffsetNumber; i <= maxoff; i++)
+ if (!GinPageRightMost(page)
+ && GinDataLeafPageIsEmpty(page)
+ && BlockNumberIsValid(parentBlkno))
{
- PostingItem *pitem = GinDataPageGetPostingItem(page, i);
-
- children[i] = PostingItemGetBlockNumber(pitem);
- }
+ /* try to delete this page */
+ OffsetNumber myoff;
+ Buffer parentBuffer = tryFindParentForLeaf(gvs, &parentBlkno, blkno, &myoff);
+ if (BufferIsValid(parentBuffer))
+ {
+ /* All pages are properly locked */
+ ginDeletePage(gvs, blkno, buffer, prevBuffer, parentBuffer, myoff);
- UnlockReleaseBuffer(buffer);
+ /* prevPage now has new rightlink, be careful not to release prevPage */
- for (i = FirstOffsetNumber; i <= maxoff; i++)
- {
- if (ginVacuumPostingTreeLeaves(gvs, children[i], false))
- hasEmptyChild = true;
- else
- hasNonEmptyChild = true;
+ continue;
+ }
}
+ UnlockReleaseBuffer(prevBuffer);
- pfree(children);
-
- vacuum_delay_point();
+ prevBuffer = buffer;
+ prevPage = BufferGetPage(prevBuffer);
/*
- * All subtree is empty - just return true to indicate that parent
- * must do a cleanup, unless we are ROOT and there is way to go upper.
+ * During delaypoint we hold one cleanup lock. We cannot avoid this lock
+ * to guaranty that all tuples were vacuumed.
*/
-
- if (hasEmptyChild && !hasNonEmptyChild && !isRoot)
- return true;
-
- if (hasEmptyChild)
- {
- DataPageDeleteStack root,
- *ptr,
- *tmp;
-
- buffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM, blkno,
- RBM_NORMAL, gvs->strategy);
- LockBufferForCleanup(buffer);
-
- memset(&root, 0, sizeof(DataPageDeleteStack));
- root.leftBlkno = InvalidBlockNumber;
- root.isRoot = true;
-
- ginScanToDelete(gvs, blkno, true, &root, InvalidOffsetNumber);
-
- ptr = root.child;
-
- while (ptr)
- {
- tmp = ptr->child;
- pfree(ptr);
- ptr = tmp;
- }
-
- UnlockReleaseBuffer(buffer);
- }
-
- /* Here we have deleted all empty subtrees */
- return false;
+ vacuum_delay_point();
}
-}
-static void
-ginVacuumPostingTree(GinVacuumState *gvs, BlockNumber rootBlkno)
-{
- ginVacuumPostingTreeLeaves(gvs, rootBlkno, true);
+ UnlockReleaseBuffer(prevBuffer);
}
/*
diff --git a/src/backend/access/gin/ginxlog.c b/src/backend/access/gin/ginxlog.c
index 7701a2d6bf..7821f69077 100644
--- a/src/backend/access/gin/ginxlog.c
+++ b/src/backend/access/gin/ginxlog.c
@@ -518,6 +518,7 @@ ginRedoDeletePage(XLogReaderState *record)
page = BufferGetPage(dbuffer);
Assert(GinPageIsData(page));
GinPageGetOpaque(page)->flags = GIN_DELETED;
+ GinPageSetDeleteXid(page, data->deleteXid);
PageSetLSN(page, lsn);
MarkBufferDirty(dbuffer);
}
diff --git a/src/include/access/gin_private.h b/src/include/access/gin_private.h
index 81bf8734ce..f360196af4 100644
--- a/src/include/access/gin_private.h
+++ b/src/include/access/gin_private.h
@@ -44,6 +44,9 @@ typedef struct GinOptions
#define GIN_SHARE BUFFER_LOCK_SHARE
#define GIN_EXCLUSIVE BUFFER_LOCK_EXCLUSIVE
+#define GinPageGetDeleteXid(page) ( ((PageHeader) (page))->pd_prune_xid )
+#define GinPageSetDeleteXid(page, val) ( ((PageHeader) (page))->pd_prune_xid = val)
+
/*
* GinState: working data structure describing the index being worked on
@@ -144,12 +147,12 @@ typedef struct GinBtreeData
{
/* search methods */
BlockNumber (*findChildPage) (GinBtree, GinBtreeStack *);
- BlockNumber (*getLeftMostChild) (GinBtree, Page);
+ BlockNumber (*getLeftMostChild) (Page);
bool (*isMoveRight) (GinBtree, Page);
bool (*findItem) (GinBtree, GinBtreeStack *);
/* insert methods */
- OffsetNumber (*findChildPtr) (GinBtree, Page, BlockNumber, OffsetNumber);
+ OffsetNumber (*findChildPtr) (Page, BlockNumber, OffsetNumber);
GinPlaceToPageRC (*beginPlaceToPage) (GinBtree, Buffer, GinBtreeStack *, void *, BlockNumber, void **, Page *, Page *);
void (*execPlaceToPage) (GinBtree, Buffer, GinBtreeStack *, void *, BlockNumber, void *);
void *(*prepareDownlink) (GinBtree, Buffer);
@@ -225,6 +228,8 @@ extern void ginInsertItemPointers(Relation index, BlockNumber rootBlkno,
GinStatsData *buildStats);
extern GinBtreeStack *ginScanBeginPostingTree(GinBtree btree, Relation index, BlockNumber rootBlkno, Snapshot snapshot);
extern void ginDataFillRoot(GinBtree btree, Page root, BlockNumber lblkno, Page lpage, BlockNumber rblkno, Page rpage);
+extern BlockNumber dataGetLeftMostPage(Page page);
+extern OffsetNumber dataFindChildPtr(Page page, BlockNumber blkno, OffsetNumber storedOff);
/*
* This is declared in ginvacuum.c, but is passed between ginVacuumItemPointers
diff --git a/src/include/access/ginxlog.h b/src/include/access/ginxlog.h
index 64a3c9e18b..d7f44d6b83 100644
--- a/src/include/access/ginxlog.h
+++ b/src/include/access/ginxlog.h
@@ -158,6 +158,7 @@ typedef struct ginxlogDeletePage
{
OffsetNumber parentOffset;
BlockNumber rightLink;
+ TransactionId deleteXid; /* last Xid which could see page in scan */
} ginxlogDeletePage;
#define XLOG_GIN_UPDATE_META_PAGE 0x60
--
2.17.2 (Apple Git-113)
=
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-11-24 00:18 Peter Geoghegan <pg@bowt.ie>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 0 replies; 67+ messages in thread
From: Peter Geoghegan @ 2018-11-24 00:18 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: chjischj@163.com, Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Fri, Nov 23, 2018 at 2:18 AM Andrey Borodin <x4mmm@yandex-team.ru> wrote:
> I found no practical way to fix concept of "subtree-lock". Pre-locking all left siblings for cleanup would suffice, but does not look very practical.
> GIN Posting tree has no all useful B-tree inventory like left links and high keys for concurrent deletes without "subtree-lock".
Significantly changing the design of GIN vacuuming in back branches
for a release that's more than a year old is absolutely out of the
question. I think that your patch should be evaluated as a new v12
feature, following the revert of 218f51584d5.
> Please note, that attached patch do not fix 2nd bug found by Chen in page delete redo.
>
> I understand that reverting commit 218f51584d5 and returning to long but finite root locks is safest solution
Whatever happens with the master branch, I think that reverting commit
218f51584d5 is the only option on the table for the back branches. Its
design is based on ideas on locking protocols that are fundamentally
incorrect and unworkable. I don't have a lot of faith in our ability
to retrofit a design that fixes the issue without causing problems
elsewhere.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-12-04 17:01 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
1 sibling, 1 reply; 67+ messages in thread
From: Andres Freund @ 2018-12-04 17:01 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: chjischj@163.com, Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>; Alexander Korotkov <aekorotkov@gmail.com>
Hi,
On 2018-11-10 17:42:16 -0800, Peter Geoghegan wrote:
> On Wed, Nov 7, 2018 at 5:46 PM Peter Geoghegan <pg@bowt.ie> wrote:
> > Teodor: Do you think that the issue is fixable? It looks like there
> > are serious issues with the design of 218f51584d5 to me. I don't think
> > the general "there can't be any inserters at this subtree" thing works
> > given that we have to couple buffer locks when moving right for other
> > reasons. We call ginStepRight() within ginFinishSplit(), for reasons
> > that date back to bug fix commit ac4ab97e from 2013 -- that detail is
> > probably important, because it seems to be what breaks the subtree
> > design (we don't delete in two phases or anything in GIN).
>
> Ping?
>
> This is a thorny problem, and I'd like to get your input soon. I
> suspect that reverting 218f51584d5 may be the ultimate outcome, even
> though it's a v10 feature.
Teodor, any chance for a response here? It's not OK to commit something
and then just not respond to bug-reports, especially when they're well
diagnose and clearly point towards issues in a commit of yours.
Alexander, CCing you, perhaps you could point the thread out to Teodor?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-12-06 19:56 Alexander Korotkov <aekorotkov@gmail.com>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 67+ messages in thread
From: Alexander Korotkov @ 2018-12-06 19:56 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Peter Geoghegan <pg@bowt.ie>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Tue, Dec 4, 2018 at 8:01 PM Andres Freund <andres@anarazel.de> wrote:
> On 2018-11-10 17:42:16 -0800, Peter Geoghegan wrote:
> > On Wed, Nov 7, 2018 at 5:46 PM Peter Geoghegan <pg@bowt.ie> wrote:
> > > Teodor: Do you think that the issue is fixable? It looks like there
> > > are serious issues with the design of 218f51584d5 to me. I don't think
> > > the general "there can't be any inserters at this subtree" thing works
> > > given that we have to couple buffer locks when moving right for other
> > > reasons. We call ginStepRight() within ginFinishSplit(), for reasons
> > > that date back to bug fix commit ac4ab97e from 2013 -- that detail is
> > > probably important, because it seems to be what breaks the subtree
> > > design (we don't delete in two phases or anything in GIN).
> >
> > Ping?
> >
> > This is a thorny problem, and I'd like to get your input soon. I
> > suspect that reverting 218f51584d5 may be the ultimate outcome, even
> > though it's a v10 feature.
>
> Teodor, any chance for a response here? It's not OK to commit something
> and then just not respond to bug-reports, especially when they're well
> diagnose and clearly point towards issues in a commit of yours.
Unfortunately, Teodor is unreachable at least till next week.
> Alexander, CCing you, perhaps you could point the thread out to Teodor?
I'll take a look at this issue.
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-12-06 20:51 Alexander Korotkov <aekorotkov@gmail.com>
parent: Alexander Korotkov <aekorotkov@gmail.com>
0 siblings, 1 reply; 67+ messages in thread
From: Alexander Korotkov @ 2018-12-06 20:51 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Peter Geoghegan <pg@bowt.ie>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Thu, Dec 6, 2018 at 10:56 PM Alexander Korotkov <aekorotkov@gmail.com> wrote:
> On Tue, Dec 4, 2018 at 8:01 PM Andres Freund <andres@anarazel.de> wrote:
> > On 2018-11-10 17:42:16 -0800, Peter Geoghegan wrote:
> > > On Wed, Nov 7, 2018 at 5:46 PM Peter Geoghegan <pg@bowt.ie> wrote:
> > > > Teodor: Do you think that the issue is fixable? It looks like there
> > > > are serious issues with the design of 218f51584d5 to me. I don't think
> > > > the general "there can't be any inserters at this subtree" thing works
> > > > given that we have to couple buffer locks when moving right for other
> > > > reasons. We call ginStepRight() within ginFinishSplit(), for reasons
> > > > that date back to bug fix commit ac4ab97e from 2013 -- that detail is
> > > > probably important, because it seems to be what breaks the subtree
> > > > design (we don't delete in two phases or anything in GIN).
> > >
> > > Ping?
> > >
> > > This is a thorny problem, and I'd like to get your input soon. I
> > > suspect that reverting 218f51584d5 may be the ultimate outcome, even
> > > though it's a v10 feature.
> >
> > Teodor, any chance for a response here? It's not OK to commit something
> > and then just not respond to bug-reports, especially when they're well
> > diagnose and clearly point towards issues in a commit of yours.
>
> Unfortunately, Teodor is unreachable at least till next week.
>
> > Alexander, CCing you, perhaps you could point the thread out to Teodor?
>
> I'll take a look at this issue.
I've run through the thread.
So, algorithm introduced by 218f51584d5 is broken. It tries to
guarantee that there are no inserters in the subtree by placing
cleanup lock to subtree root, assuming that inserter holds pins on the
path from root to leaf. But due to concurrent splits of internal
pages the pins held can be not relevant to actual path. I don't see
the way to fix this easily. So, I think we should revert it from back
branches and try to reimplement that in master.
However, I'd like to note that 218f51584d5 introduces two changes:
1) Cleanup locking only if there pages to delete
2) Cleanup locking only subtree root
The 2nd one is broken. But the 1st one seems still good for me and
useful, because in vast majority of cases vacuum doesn't delete any
index pages. So, I propose to revert 218f51584d5, but leave there
logic, which locks root for cleanup only once there are pages to
delete. Any thoughts?
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-12-06 21:50 Peter Geoghegan <pg@bowt.ie>
parent: Alexander Korotkov <aekorotkov@gmail.com>
0 siblings, 2 replies; 67+ messages in thread
From: Peter Geoghegan @ 2018-12-06 21:50 UTC (permalink / raw)
To: aekorotkov@gmail.com; +Cc: Andres Freund <andres@anarazel.de>; chjischj@163.com, Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Thu, Dec 6, 2018 at 12:51 PM Alexander Korotkov <aekorotkov@gmail.com> wrote:
> So, algorithm introduced by 218f51584d5 is broken. It tries to
> guarantee that there are no inserters in the subtree by placing
> cleanup lock to subtree root, assuming that inserter holds pins on the
> path from root to leaf. But due to concurrent splits of internal
> pages the pins held can be not relevant to actual path. I don't see
> the way to fix this easily. So, I think we should revert it from back
> branches and try to reimplement that in master.
>
> However, I'd like to note that 218f51584d5 introduces two changes:
> 1) Cleanup locking only if there pages to delete
> 2) Cleanup locking only subtree root
> The 2nd one is broken. But the 1st one seems still good for me and
> useful, because in vast majority of cases vacuum doesn't delete any
> index pages. So, I propose to revert 218f51584d5, but leave there
> logic, which locks root for cleanup only once there are pages to
> delete. Any thoughts?
Can you post a patch that just removes the 2nd part, leaving the
still-correct first part?
Your proposal sounds reasonable, but I'd like to see what you have in
mind in detail before commenting.
Thanks
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-12-07 09:14 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Peter Geoghegan <pg@bowt.ie>
1 sibling, 1 reply; 67+ messages in thread
From: Andrey Borodin @ 2018-12-07 09:14 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: aekorotkov@gmail.com, Andres Freund <andres@anarazel.de>; chjischj@163.com, Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi!
> 7 дек. 2018 г., в 2:50, Peter Geoghegan <pg@bowt.ie> написал(а):
>
> On Thu, Dec 6, 2018 at 12:51 PM Alexander Korotkov <aekorotkov@gmail.com> wrote:
>>
>> However, I'd like to note that 218f51584d5 introduces two changes:
>> 1) Cleanup locking only if there pages to delete
>> 2) Cleanup locking only subtree root
>> The 2nd one is broken. But the 1st one seems still good for me and
>> useful, because in vast majority of cases vacuum doesn't delete any
>> index pages. So, I propose to revert 218f51584d5, but leave there
>> logic, which locks root for cleanup only once there are pages to
>> delete. Any thoughts?
>
> Can you post a patch that just removes the 2nd part, leaving the
> still-correct first part?
I like the idea of keeping cleanup lock only if there are pages to delete. It will still solve the original problem that caused proposals for GIN VACUUM enhancements.
But I must note that there is one more problem: ginVacuumPostingTreeLeaves() do not ensure that all splitted pages are visited. It copies downlink block numbers to a temp array and then unlocks parent. It is not correct way to scan posting tree for cleanup.
PFA diff with following changes:
1. Always take root cleanup lock before deleting pages
2. Check for concurrent splits after scanning page
Please note, that neither applying this diff nor reverting 218f51584d5 will solve bug of page delete redo lock on standby.
Best regards, Andrey Borodin.
Attachments:
gin_root_lock.diff (application/octet-stream, 2.6K) download
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index 3104bc12b6..3d5f6a4ca5 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -316,10 +316,8 @@ ginScanToDelete(GinVacuumState *gvs, BlockNumber blkno, bool isRoot,
/*
* Scan through posting tree, delete empty tuples from leaf pages.
- * Also, this function collects empty subtrees (with all empty leafs).
- * For parents of these subtrees CleanUp lock is taken, then we call
- * ScanToDelete. This is done for every inner page, which points to
- * empty subtree.
+ * If there are empty pages root CleanUp lock is taken, then we call
+ * ScanToDelete.
*/
static bool
ginVacuumPostingTreeLeaves(GinVacuumState *gvs, BlockNumber blkno, bool isRoot)
@@ -356,13 +354,11 @@ ginVacuumPostingTreeLeaves(GinVacuumState *gvs, BlockNumber blkno, bool isRoot)
{
OffsetNumber i;
bool hasEmptyChild = false;
- bool hasNonEmptyChild = false;
OffsetNumber maxoff = GinPageGetOpaque(page)->maxoff;
BlockNumber *children = palloc(sizeof(BlockNumber) * (maxoff + 1));
/*
- * Read all children BlockNumbers. Not sure it is safe if there are
- * many concurrent vacuums.
+ * Read all children BlockNumbers.
*/
for (i = FirstOffsetNumber; i <= maxoff; i++)
@@ -372,26 +368,38 @@ ginVacuumPostingTreeLeaves(GinVacuumState *gvs, BlockNumber blkno, bool isRoot)
children[i] = PostingItemGetBlockNumber(pitem);
}
- UnlockReleaseBuffer(buffer);
+ LockBuffer(buffer, GIN_UNLOCK);
for (i = FirstOffsetNumber; i <= maxoff; i++)
{
if (ginVacuumPostingTreeLeaves(gvs, children[i], false))
hasEmptyChild = true;
- else
- hasNonEmptyChild = true;
}
+ LockBuffer(buffer, GIN_SHARE);
+ /* There might be concurrent splits of downlinks*/
+ if ((GinPageGetOpaque(page)->maxoff) != maxoff)
+ {
+ OffsetNumber maxoff = GinPageGetOpaque(page)->maxoff;
+ for (i = FirstOffsetNumber; i <= maxoff; i++)
+ {
+ PostingItem *pitem = GinDataPageGetPostingItem(page, i);
+ if (ginVacuumPostingTreeLeaves(gvs, PostingItemGetBlockNumber(pitem), false))
+ hasEmptyChild = true;
+ }
+ }
+ UnlockReleaseBuffer(buffer);
+
pfree(children);
vacuum_delay_point();
/*
- * All subtree is empty - just return true to indicate that parent
- * must do a cleanup, unless we are ROOT and there is way to go upper.
+ * We signal upper that there are empty subtrees, root call will clean
+ * them up, after taking neccesary lock.
*/
- if (hasEmptyChild && !hasNonEmptyChild && !isRoot)
+ if (hasEmptyChild && !isRoot)
return true;
if (hasEmptyChild)
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-12-07 15:34 Alexander Korotkov <aekorotkov@gmail.com>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 0 replies; 67+ messages in thread
From: Alexander Korotkov @ 2018-12-07 15:34 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Fri, Dec 7, 2018 at 12:14 PM Andrey Borodin <x4mmm@yandex-team.ru> wrote:
> > 7 дек. 2018 г., в 2:50, Peter Geoghegan <pg@bowt.ie> написал(а):
> > On Thu, Dec 6, 2018 at 12:51 PM Alexander Korotkov <aekorotkov@gmail.com> wrote:
> >> However, I'd like to note that 218f51584d5 introduces two changes:
> >> 1) Cleanup locking only if there pages to delete
> >> 2) Cleanup locking only subtree root
> >> The 2nd one is broken. But the 1st one seems still good for me and
> >> useful, because in vast majority of cases vacuum doesn't delete any
> >> index pages. So, I propose to revert 218f51584d5, but leave there
> >> logic, which locks root for cleanup only once there are pages to
> >> delete. Any thoughts?
> >
> > Can you post a patch that just removes the 2nd part, leaving the
> > still-correct first part?
>
> I like the idea of keeping cleanup lock only if there are pages to delete. It will still solve the original problem that caused proposals for GIN VACUUM enhancements.
>
> But I must note that there is one more problem: ginVacuumPostingTreeLeaves() do not ensure that all splitted pages are visited. It copies downlink block numbers to a temp array and then unlocks parent. It is not correct way to scan posting tree for cleanup.
Hmm... In ginVacuumPostingTreeLeaves() we visit pages from parent to
children. This is why splitted pages might be unvisited. Should we
visit leafpages by rightlinks instead? Then, if we have candidates
for delete, ginScanToDelete() can work the same as before 218f51584d5.
> PFA diff with following changes:
> 1. Always take root cleanup lock before deleting pages
> 2. Check for concurrent splits after scanning page
>
> Please note, that neither applying this diff nor reverting 218f51584d5 will solve bug of page delete redo lock on standby.
ginRedoDeletePage() visits pages in following order: right(to be
deleted), parent, left. I didn't find any description of why it has
been done so. Any guesses don't we visit parent page first here?
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-12-08 01:54 Alexander Korotkov <aekorotkov@gmail.com>
parent: Peter Geoghegan <pg@bowt.ie>
1 sibling, 1 reply; 67+ messages in thread
From: Alexander Korotkov @ 2018-12-08 01:54 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Andres Freund <andres@anarazel.de>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Fri, Dec 7, 2018 at 12:50 AM Peter Geoghegan <pg@bowt.ie> wrote:
> On Thu, Dec 6, 2018 at 12:51 PM Alexander Korotkov <aekorotkov@gmail.com> wrote:
> > So, algorithm introduced by 218f51584d5 is broken. It tries to
> > guarantee that there are no inserters in the subtree by placing
> > cleanup lock to subtree root, assuming that inserter holds pins on the
> > path from root to leaf. But due to concurrent splits of internal
> > pages the pins held can be not relevant to actual path. I don't see
> > the way to fix this easily. So, I think we should revert it from back
> > branches and try to reimplement that in master.
> >
> > However, I'd like to note that 218f51584d5 introduces two changes:
> > 1) Cleanup locking only if there pages to delete
> > 2) Cleanup locking only subtree root
> > The 2nd one is broken. But the 1st one seems still good for me and
> > useful, because in vast majority of cases vacuum doesn't delete any
> > index pages. So, I propose to revert 218f51584d5, but leave there
> > logic, which locks root for cleanup only once there are pages to
> > delete. Any thoughts?
>
> Can you post a patch that just removes the 2nd part, leaving the
> still-correct first part?
>
> Your proposal sounds reasonable, but I'd like to see what you have in
> mind in detail before commenting.
Yep, please find attached draft patch.
BTW, it seems that I've another bug in GIN. README says that
"However, posting trees are only
fully searched from left to right, starting from the leftmost leaf. (The
tree-structure is only needed by insertions, to quickly find the correct
insert location). So as long as we don't delete the leftmost page on each
level, a search can never follow a downlink to page that's about to be
deleted."
But that's not really true once we teach GIN to skip parts of posting
trees in PostgreSQL 9.4. So, there might be a risk to visit page to
be deleted using downlink. But in order to get real problem, vacuum
should past cleanup stage and someone else should reuse this page,
before we traverse downlink. Thus, the risk of real problem is very
low. But it still should be addressed.
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
Attachments:
gin-vacuum-fix-1.patch (application/octet-stream, 7.9K) download
commit 4ebb1abad475c16877f4390c3c7ba082f3945c0e
Author: Alexander Korotkov <akorotkov@postgresql.org>
Date: Sat Dec 8 04:30:54 2018 +0300
Fix deadlock in GIN vacuum introduced by 218f51584d5
Before 218f51584d5 when posting tree page is about to be deleted, then the whole
posting tree is locked by LockBufferForCleanup() on root preventing all
concurrent inserts. 218f51584d5 changed reduces locking to subtree containing
page to be deleted. However, due to concurrent parent split, inserter doesn't
always holds pins on pages constituting path from root to the leaf page. That
could cause a deadlock between GIN vacuum process and GIN inserter.
This commit reverts VACUUM behavior to lock the whole posting tree before
delete any page. However, we keep another change by 218f51584d5: the tree is
locked only if there are pages to delete.
diff --git a/src/backend/access/gin/README b/src/backend/access/gin/README
index cc434b1feb7..421b5b26d5b 100644
--- a/src/backend/access/gin/README
+++ b/src/backend/access/gin/README
@@ -314,17 +314,10 @@ deleted.
The previous paragraph's reasoning only applies to searches, and only to
posting trees. To protect from inserters following a downlink to a deleted
page, vacuum simply locks out all concurrent insertions to the posting tree,
-by holding a super-exclusive lock on the parent page of subtree with deletable
-pages. Inserters hold a pin on the root page, but searches do not, so while
-new searches cannot begin while root page is locked, any already-in-progress
-scans can continue concurrently with vacuum in corresponding subtree of
-posting tree. To exclude interference with readers vacuum takes exclusive
-locks in a depth-first scan in left-to-right order of page tuples. Leftmost
-page is never deleted. Thus before deleting any page we obtain exclusive
-lock on any left page, effectively excluding deadlock with any reader, despite
-taking parent lock before current and left lock after current. We take left
-lock not for a concurrency reasons, but rather in need to mark page dirty.
-In the entry tree, we never delete pages.
+by holding a super-exclusive lock on the posting tree root. Inserters hold a
+pin on the root page, but searches do not, so while new searches cannot begin
+while root page is locked, any already-in-progress scans can continue
+concurrently with vacuum. In the entry tree, we never delete pages.
(This is quite different from the mechanism the btree indexam uses to make
page-deletions safe; it stamps the deleted pages with an XID and keeps the
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index 3104bc12b63..bdeb0bf4f52 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -315,124 +315,114 @@ ginScanToDelete(GinVacuumState *gvs, BlockNumber blkno, bool isRoot,
/*
- * Scan through posting tree, delete empty tuples from leaf pages.
- * Also, this function collects empty subtrees (with all empty leafs).
- * For parents of these subtrees CleanUp lock is taken, then we call
- * ScanToDelete. This is done for every inner page, which points to
- * empty subtree.
+ * Scan through posting tree leafs, delete empty tuples. Returns true if there
+ * is at least one empty page.
*/
static bool
-ginVacuumPostingTreeLeaves(GinVacuumState *gvs, BlockNumber blkno, bool isRoot)
+ginVacuumPostingTreeLeaves(GinVacuumState *gvs, BlockNumber blkno)
{
Buffer buffer;
Page page;
bool hasVoidPage = false;
MemoryContext oldCxt;
- buffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM, blkno,
- RBM_NORMAL, gvs->strategy);
- page = BufferGetPage(buffer);
+ /* Find leftmost leaf page of posting tree and lock it in exclusive mode */
+ while (true)
+ {
+ PostingItem *pitem;
- ginTraverseLock(buffer, false);
+ buffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM, blkno,
+ RBM_NORMAL, gvs->strategy);
+ LockBuffer(buffer, GIN_SHARE);
+ page = BufferGetPage(buffer);
- Assert(GinPageIsData(page));
+ Assert(GinPageIsData(page));
- if (GinPageIsLeaf(page))
+ if (GinPageIsLeaf(page))
+ {
+ LockBuffer(buffer, GIN_UNLOCK);
+ LockBuffer(buffer, GIN_EXCLUSIVE);
+ break;
+ }
+
+ Assert(PageGetMaxOffsetNumber(page) >= FirstOffsetNumber);
+
+ pitem = GinDataPageGetPostingItem(page, FirstOffsetNumber);
+ blkno = PostingItemGetBlockNumber(pitem);
+ Assert(blkno != InvalidBlockNumber);
+
+ UnlockReleaseBuffer(buffer);
+ }
+
+ /* Iterate all posting tree leaves using rightlinks and vacuum them */
+ while (true)
{
oldCxt = MemoryContextSwitchTo(gvs->tmpCxt);
ginVacuumPostingTreeLeaf(gvs->index, buffer, gvs);
MemoryContextSwitchTo(oldCxt);
MemoryContextReset(gvs->tmpCxt);
- /* if root is a leaf page, we don't desire further processing */
if (GinDataLeafPageIsEmpty(page))
hasVoidPage = true;
- UnlockReleaseBuffer(buffer);
-
- return hasVoidPage;
- }
- else
- {
- OffsetNumber i;
- bool hasEmptyChild = false;
- bool hasNonEmptyChild = false;
- OffsetNumber maxoff = GinPageGetOpaque(page)->maxoff;
- BlockNumber *children = palloc(sizeof(BlockNumber) * (maxoff + 1));
-
- /*
- * Read all children BlockNumbers. Not sure it is safe if there are
- * many concurrent vacuums.
- */
-
- for (i = FirstOffsetNumber; i <= maxoff; i++)
- {
- PostingItem *pitem = GinDataPageGetPostingItem(page, i);
-
- children[i] = PostingItemGetBlockNumber(pitem);
- }
+ blkno = GinPageGetOpaque(page)->rightlink;
UnlockReleaseBuffer(buffer);
- for (i = FirstOffsetNumber; i <= maxoff; i++)
- {
- if (ginVacuumPostingTreeLeaves(gvs, children[i], false))
- hasEmptyChild = true;
- else
- hasNonEmptyChild = true;
- }
+ if (blkno == InvalidBlockNumber)
+ break;
- pfree(children);
+ buffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM, blkno,
+ RBM_NORMAL, gvs->strategy);
+ LockBuffer(buffer, GIN_EXCLUSIVE);
+ page = BufferGetPage(buffer);
+ }
- vacuum_delay_point();
+ return hasVoidPage;
+}
+static void
+ginVacuumPostingTree(GinVacuumState *gvs, BlockNumber rootBlkno)
+{
+ if (ginVacuumPostingTreeLeaves(gvs, rootBlkno))
+ {
/*
- * All subtree is empty - just return true to indicate that parent
- * must do a cleanup, unless we are ROOT and there is way to go upper.
+ * There is at least one empty page. So we have to rescan the tree
+ * deleting empty pages.
*/
+ Buffer buffer;
+ DataPageDeleteStack root,
+ *ptr,
+ *tmp;
- if (hasEmptyChild && !hasNonEmptyChild && !isRoot)
- return true;
+ buffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM, rootBlkno,
+ RBM_NORMAL, gvs->strategy);
- if (hasEmptyChild)
- {
- DataPageDeleteStack root,
- *ptr,
- *tmp;
-
- buffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM, blkno,
- RBM_NORMAL, gvs->strategy);
- LockBufferForCleanup(buffer);
-
- memset(&root, 0, sizeof(DataPageDeleteStack));
- root.leftBlkno = InvalidBlockNumber;
- root.isRoot = true;
+ /*
+ * Lock posting tree root for cleanup to ensure there are no concurrent
+ * inserts.
+ */
+ LockBufferForCleanup(buffer);
- ginScanToDelete(gvs, blkno, true, &root, InvalidOffsetNumber);
+ memset(&root, 0, sizeof(DataPageDeleteStack));
+ root.leftBlkno = InvalidBlockNumber;
+ root.isRoot = true;
- ptr = root.child;
+ ginScanToDelete(gvs, rootBlkno, true, &root, InvalidOffsetNumber);
- while (ptr)
- {
- tmp = ptr->child;
- pfree(ptr);
- ptr = tmp;
- }
+ ptr = root.child;
- UnlockReleaseBuffer(buffer);
+ while (ptr)
+ {
+ tmp = ptr->child;
+ pfree(ptr);
+ ptr = tmp;
}
- /* Here we have deleted all empty subtrees */
- return false;
+ UnlockReleaseBuffer(buffer);
}
}
-static void
-ginVacuumPostingTree(GinVacuumState *gvs, BlockNumber rootBlkno)
-{
- ginVacuumPostingTreeLeaves(gvs, rootBlkno, true);
-}
-
/*
* returns modified page or NULL if page isn't modified.
* Function works with original page until first change is occurred,
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-12-08 09:48 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Alexander Korotkov <aekorotkov@gmail.com>
0 siblings, 1 reply; 67+ messages in thread
From: Andrey Borodin @ 2018-12-08 09:48 UTC (permalink / raw)
To: Alexander Korotkov <aekorotkov@gmail.com>; +Cc: Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi!
Thanks, Alexander!
> 8 дек. 2018 г., в 6:54, Alexander Korotkov <aekorotkov@gmail.com> написал(а):
>
> Yep, please find attached draft patch.
Patch seems good to me, I'll check it in more detail.
The patch gets posting item at FirstOffsetNumber instead of btree->getLeftMostChild(). This seem OK, since dataGetLeftMostPage() is doing just the same, but with few Assert()s.
> BTW, it seems that I've another bug in GIN. README says that
>
> "However, posting trees are only
> fully searched from left to right, starting from the leftmost leaf. (The
> tree-structure is only needed by insertions, to quickly find the correct
> insert location). So as long as we don't delete the leftmost page on each
> level, a search can never follow a downlink to page that's about to be
> deleted."
>
> But that's not really true once we teach GIN to skip parts of posting
> trees in PostgreSQL 9.4. So, there might be a risk to visit page to
> be deleted using downlink. But in order to get real problem, vacuum
> should past cleanup stage and someone else should reuse this page,
> before we traverse downlink. Thus, the risk of real problem is very
> low. But it still should be addressed.
There's a patch above in this thread 0001-Use-correct-locking-protocol-during-GIN-posting-tree.patch where I propose stamping every deleted page with GinPageSetDeleteXid(page, ReadNewTransactionId()); and avoid reusing the page before TransactionIdPrecedes(GinPageGetDeleteXid(page), RecentGlobalDataXmin).
Should we leave alone this bug for future fixes to keep current fix noninvasive?
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-12-09 19:25 Alexander Korotkov <aekorotkov@gmail.com>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 2 replies; 67+ messages in thread
From: Alexander Korotkov @ 2018-12-09 19:25 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Sat, Dec 8, 2018 at 12:48 PM Andrey Borodin <x4mmm@yandex-team.ru> wrote:
> > 8 дек. 2018 г., в 6:54, Alexander Korotkov <aekorotkov@gmail.com> написал(а):
> >
> > Yep, please find attached draft patch.
>
> Patch seems good to me, I'll check it in more detail.
> The patch gets posting item at FirstOffsetNumber instead of btree->getLeftMostChild(). This seem OK, since dataGetLeftMostPage() is doing just the same, but with few Assert()s.
I'd like to evade creating GinBtree for just calling
getLeftMostChild(). Also, few more places in ginvacuum.c do the same.
We have the same amount of Assert()s in ginVacuumPostingTreeLeaves().
So, let's keep it uniform.
I would also like Peter Geoghegan to take a look at this patch before
committing it.
> > BTW, it seems that I've another bug in GIN. README says that
> >
> > "However, posting trees are only
> > fully searched from left to right, starting from the leftmost leaf. (The
> > tree-structure is only needed by insertions, to quickly find the correct
> > insert location). So as long as we don't delete the leftmost page on each
> > level, a search can never follow a downlink to page that's about to be
> > deleted."
> >
> > But that's not really true once we teach GIN to skip parts of posting
> > trees in PostgreSQL 9.4. So, there might be a risk to visit page to
> > be deleted using downlink. But in order to get real problem, vacuum
> > should past cleanup stage and someone else should reuse this page,
> > before we traverse downlink. Thus, the risk of real problem is very
> > low. But it still should be addressed.
>
> There's a patch above in this thread 0001-Use-correct-locking-protocol-during-GIN-posting-tree.patch where I propose stamping every deleted page with GinPageSetDeleteXid(page, ReadNewTransactionId()); and avoid reusing the page before TransactionIdPrecedes(GinPageGetDeleteXid(page), RecentGlobalDataXmin).
> Should we leave alone this bug for future fixes to keep current fix noninvasive?
I think since is separate bug introduced in PostgreSQL 9.4, which
should be backpatched with separate commit. Could you please extract
patch dealing with GinPageSetDeleteXid() and GinPageGetDeleteXid().
The rest of work made in your patch should be considered for master.
BTW, what do you think about locking order in ginRedoDeletePage()? Do
you have any explanation of current behavior? I'll try to reach
Teodor tomorrow with this question.
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-12-10 13:56 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Alexander Korotkov <aekorotkov@gmail.com>
1 sibling, 1 reply; 67+ messages in thread
From: Andrey Borodin @ 2018-12-10 13:56 UTC (permalink / raw)
To: Alexander Korotkov <aekorotkov@gmail.com>; +Cc: Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi!
> 10 дек. 2018 г., в 0:25, Alexander Korotkov <aekorotkov@gmail.com> написал(а):
>>
>> There's a patch above in this thread 0001-Use-correct-locking-protocol-during-GIN-posting-tree.patch where I propose stamping every deleted page with GinPageSetDeleteXid(page, ReadNewTransactionId()); and avoid reusing the page before TransactionIdPrecedes(GinPageGetDeleteXid(page), RecentGlobalDataXmin).
>> Should we leave alone this bug for future fixes to keep current fix noninvasive?
>
> I think since is separate bug introduced in PostgreSQL 9.4, which
> should be backpatched with separate commit. Could you please extract
> patch dealing with GinPageSetDeleteXid() and GinPageGetDeleteXid().
> The rest of work made in your patch should be considered for master.
PFA this part. In thread about GiST vacuum Heikki was not very happy with reusing PageHeader->pd_prune_xid for this. But we somewhat concluded that this might be OK.
Anyway, there's whole page, we can store those bits anywhere.
> BTW, what do you think about locking order in ginRedoDeletePage()? Do
> you have any explanation of current behavior?
It seems to me that locking order must be left->deleted->parent.
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-12-10 13:58 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 67+ messages in thread
From: Andrey Borodin @ 2018-12-10 13:58 UTC (permalink / raw)
To: Alexander Korotkov <aekorotkov@gmail.com>; +Cc: Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
> 10 дек. 2018 г., в 18:56, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
>
> PFA this part. In thread about GiST vacuum Heikki was not very happy with reusing PageHeader->pd_prune_xid for this. But we somewhat concluded that this might be OK.
> Anyway, there's whole page, we can store those bits anywhere.
Errrmm... I'm very sorry for the noise. I've forgot the actual patch.
Best regards, Andrey Borodin.
Attachments:
0001-Stamp-deleted-GIN-page-with-xid-to-prevent-early-reu.patch (application/octet-stream, 2.5K) download
From 389ccf54b12cf32d49222e7c2b5b6b89083b98c7 Mon Sep 17 00:00:00 2001
From: Andrey <amborodin@acm.org>
Date: Mon, 10 Dec 2018 18:49:47 +0500
Subject: [PATCH] Stamp deleted GIN page with xid to prevent early reuse
---
src/backend/access/gin/ginutil.c | 7 ++++++-
src/backend/access/gin/ginxlog.c | 1 +
src/include/access/gin_private.h | 3 +++
src/include/access/ginxlog.h | 1 +
4 files changed, 11 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index d7696a1ad0..9072ee6260 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -314,7 +314,12 @@ GinNewBuffer(Relation index)
if (PageIsNew(page))
return buffer; /* OK to use, if never initialized */
- if (GinPageIsDeleted(page))
+ /*
+ * We should not reuse page until every transaction started before
+ * deletion is over
+ */
+ if (GinPageIsDeleted(page)
+ && TransactionIdPrecedes(GinPageGetDeleteXid(page), RecentGlobalDataXmin))
return buffer; /* OK to use */
LockBuffer(buffer, GIN_UNLOCK);
diff --git a/src/backend/access/gin/ginxlog.c b/src/backend/access/gin/ginxlog.c
index 7701a2d6bf..7821f69077 100644
--- a/src/backend/access/gin/ginxlog.c
+++ b/src/backend/access/gin/ginxlog.c
@@ -518,6 +518,7 @@ ginRedoDeletePage(XLogReaderState *record)
page = BufferGetPage(dbuffer);
Assert(GinPageIsData(page));
GinPageGetOpaque(page)->flags = GIN_DELETED;
+ GinPageSetDeleteXid(page, data->deleteXid);
PageSetLSN(page, lsn);
MarkBufferDirty(dbuffer);
}
diff --git a/src/include/access/gin_private.h b/src/include/access/gin_private.h
index 81bf8734ce..83d2621488 100644
--- a/src/include/access/gin_private.h
+++ b/src/include/access/gin_private.h
@@ -44,6 +44,9 @@ typedef struct GinOptions
#define GIN_SHARE BUFFER_LOCK_SHARE
#define GIN_EXCLUSIVE BUFFER_LOCK_EXCLUSIVE
+#define GinPageGetDeleteXid(page) ( ((PageHeader) (page))->pd_prune_xid )
+#define GinPageSetDeleteXid(page, val) ( ((PageHeader) (page))->pd_prune_xid = val)
+
/*
* GinState: working data structure describing the index being worked on
diff --git a/src/include/access/ginxlog.h b/src/include/access/ginxlog.h
index 64a3c9e18b..d7f44d6b83 100644
--- a/src/include/access/ginxlog.h
+++ b/src/include/access/ginxlog.h
@@ -158,6 +158,7 @@ typedef struct ginxlogDeletePage
{
OffsetNumber parentOffset;
BlockNumber rightLink;
+ TransactionId deleteXid; /* last Xid which could see page in scan */
} ginxlogDeletePage;
#define XLOG_GIN_UPDATE_META_PAGE 0x60
--
2.17.2 (Apple Git-113)
=
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-12-10 22:43 Alexander Korotkov <aekorotkov@gmail.com>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 67+ messages in thread
From: Alexander Korotkov @ 2018-12-10 22:43 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Mon, Dec 10, 2018 at 4:58 PM Andrey Borodin <x4mmm@yandex-team.ru> wrote:
> > 10 дек. 2018 г., в 18:56, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
> >
> > PFA this part. In thread about GiST vacuum Heikki was not very happy with reusing PageHeader->pd_prune_xid for this. But we somewhat concluded that this might be OK.
> > Anyway, there's whole page, we can store those bits anywhere.
We might allocate special field in opaque to store stat xid like
B-tree or SP-GiST. But we have to be binary compatible. And this
change shouldn't require upgrade procedure, since we're going to
backpatch that. We could allocate space elsewhere on the page, but
that would be a bit cumbersome. So, let's use pd_prune_xid once it's
available.
> Errrmm... I'm very sorry for the noise. I've forgot the actual patch.
Attached patch appears to be incomplete. GinPageSetDeleteXid() is
called only in ginRedoDeletePage(), so only in recovery, while it
should be set during normal work too. deleteXid field of
ginxlogDeletePage is never set.
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-12-10 22:50 Alexander Korotkov <aekorotkov@gmail.com>
parent: Alexander Korotkov <aekorotkov@gmail.com>
1 sibling, 1 reply; 67+ messages in thread
From: Alexander Korotkov @ 2018-12-10 22:50 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Sun, Dec 9, 2018 at 10:25 PM Alexander Korotkov <aekorotkov@gmail.com> wrote:
> On Sat, Dec 8, 2018 at 12:48 PM Andrey Borodin <x4mmm@yandex-team.ru> wrote:
> > > 8 дек. 2018 г., в 6:54, Alexander Korotkov <aekorotkov@gmail.com> написал(а):
> > >
> > > Yep, please find attached draft patch.
> >
> > Patch seems good to me, I'll check it in more detail.
> > The patch gets posting item at FirstOffsetNumber instead of btree->getLeftMostChild(). This seem OK, since dataGetLeftMostPage() is doing just the same, but with few Assert()s.
>
> I'd like to evade creating GinBtree for just calling
> getLeftMostChild(). Also, few more places in ginvacuum.c do the same.
> We have the same amount of Assert()s in ginVacuumPostingTreeLeaves().
> So, let's keep it uniform.
>
> I would also like Peter Geoghegan to take a look at this patch before
> committing it.
I've slightly adjusted commit message. I'm going to commit this fix
if no objections.
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
Attachments:
gin-vacuum-fix-2.patch (application/octet-stream, 7.9K) download
commit 74ecb6649ccff589d1ddd1c8d4961f1d88609d80
Author: Alexander Korotkov <akorotkov@postgresql.org>
Date: Sat Dec 8 04:30:54 2018 +0300
Fix deadlock in GIN vacuum introduced by 218f51584d5
Before 218f51584d5 if posting tree page is about to be deleted, then the whole
posting tree is locked by LockBufferForCleanup() on root preventing all
concurrent inserts. 218f51584d5 reduced locking to the subtree containing
page to be deleted. However, due to concurrent parent split, inserter doesn't
always holds pins on all the pages constituting path from root to the leaf page.
That could cause a deadlock between GIN vacuum process and GIN inserter. We
didn't find non-invasive way to fix this.
This commit reverts VACUUM behavior to lock the whole posting tree before
delete any page. However, we keep another change by 218f51584d5: the tree is
locked only if there are pages to delete.
diff --git a/src/backend/access/gin/README b/src/backend/access/gin/README
index cc434b1feb7..421b5b26d5b 100644
--- a/src/backend/access/gin/README
+++ b/src/backend/access/gin/README
@@ -314,17 +314,10 @@ deleted.
The previous paragraph's reasoning only applies to searches, and only to
posting trees. To protect from inserters following a downlink to a deleted
page, vacuum simply locks out all concurrent insertions to the posting tree,
-by holding a super-exclusive lock on the parent page of subtree with deletable
-pages. Inserters hold a pin on the root page, but searches do not, so while
-new searches cannot begin while root page is locked, any already-in-progress
-scans can continue concurrently with vacuum in corresponding subtree of
-posting tree. To exclude interference with readers vacuum takes exclusive
-locks in a depth-first scan in left-to-right order of page tuples. Leftmost
-page is never deleted. Thus before deleting any page we obtain exclusive
-lock on any left page, effectively excluding deadlock with any reader, despite
-taking parent lock before current and left lock after current. We take left
-lock not for a concurrency reasons, but rather in need to mark page dirty.
-In the entry tree, we never delete pages.
+by holding a super-exclusive lock on the posting tree root. Inserters hold a
+pin on the root page, but searches do not, so while new searches cannot begin
+while root page is locked, any already-in-progress scans can continue
+concurrently with vacuum. In the entry tree, we never delete pages.
(This is quite different from the mechanism the btree indexam uses to make
page-deletions safe; it stamps the deleted pages with an XID and keeps the
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index 3104bc12b63..bdeb0bf4f52 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -315,124 +315,114 @@ ginScanToDelete(GinVacuumState *gvs, BlockNumber blkno, bool isRoot,
/*
- * Scan through posting tree, delete empty tuples from leaf pages.
- * Also, this function collects empty subtrees (with all empty leafs).
- * For parents of these subtrees CleanUp lock is taken, then we call
- * ScanToDelete. This is done for every inner page, which points to
- * empty subtree.
+ * Scan through posting tree leafs, delete empty tuples. Returns true if there
+ * is at least one empty page.
*/
static bool
-ginVacuumPostingTreeLeaves(GinVacuumState *gvs, BlockNumber blkno, bool isRoot)
+ginVacuumPostingTreeLeaves(GinVacuumState *gvs, BlockNumber blkno)
{
Buffer buffer;
Page page;
bool hasVoidPage = false;
MemoryContext oldCxt;
- buffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM, blkno,
- RBM_NORMAL, gvs->strategy);
- page = BufferGetPage(buffer);
+ /* Find leftmost leaf page of posting tree and lock it in exclusive mode */
+ while (true)
+ {
+ PostingItem *pitem;
- ginTraverseLock(buffer, false);
+ buffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM, blkno,
+ RBM_NORMAL, gvs->strategy);
+ LockBuffer(buffer, GIN_SHARE);
+ page = BufferGetPage(buffer);
- Assert(GinPageIsData(page));
+ Assert(GinPageIsData(page));
- if (GinPageIsLeaf(page))
+ if (GinPageIsLeaf(page))
+ {
+ LockBuffer(buffer, GIN_UNLOCK);
+ LockBuffer(buffer, GIN_EXCLUSIVE);
+ break;
+ }
+
+ Assert(PageGetMaxOffsetNumber(page) >= FirstOffsetNumber);
+
+ pitem = GinDataPageGetPostingItem(page, FirstOffsetNumber);
+ blkno = PostingItemGetBlockNumber(pitem);
+ Assert(blkno != InvalidBlockNumber);
+
+ UnlockReleaseBuffer(buffer);
+ }
+
+ /* Iterate all posting tree leaves using rightlinks and vacuum them */
+ while (true)
{
oldCxt = MemoryContextSwitchTo(gvs->tmpCxt);
ginVacuumPostingTreeLeaf(gvs->index, buffer, gvs);
MemoryContextSwitchTo(oldCxt);
MemoryContextReset(gvs->tmpCxt);
- /* if root is a leaf page, we don't desire further processing */
if (GinDataLeafPageIsEmpty(page))
hasVoidPage = true;
- UnlockReleaseBuffer(buffer);
-
- return hasVoidPage;
- }
- else
- {
- OffsetNumber i;
- bool hasEmptyChild = false;
- bool hasNonEmptyChild = false;
- OffsetNumber maxoff = GinPageGetOpaque(page)->maxoff;
- BlockNumber *children = palloc(sizeof(BlockNumber) * (maxoff + 1));
-
- /*
- * Read all children BlockNumbers. Not sure it is safe if there are
- * many concurrent vacuums.
- */
-
- for (i = FirstOffsetNumber; i <= maxoff; i++)
- {
- PostingItem *pitem = GinDataPageGetPostingItem(page, i);
-
- children[i] = PostingItemGetBlockNumber(pitem);
- }
+ blkno = GinPageGetOpaque(page)->rightlink;
UnlockReleaseBuffer(buffer);
- for (i = FirstOffsetNumber; i <= maxoff; i++)
- {
- if (ginVacuumPostingTreeLeaves(gvs, children[i], false))
- hasEmptyChild = true;
- else
- hasNonEmptyChild = true;
- }
+ if (blkno == InvalidBlockNumber)
+ break;
- pfree(children);
+ buffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM, blkno,
+ RBM_NORMAL, gvs->strategy);
+ LockBuffer(buffer, GIN_EXCLUSIVE);
+ page = BufferGetPage(buffer);
+ }
- vacuum_delay_point();
+ return hasVoidPage;
+}
+static void
+ginVacuumPostingTree(GinVacuumState *gvs, BlockNumber rootBlkno)
+{
+ if (ginVacuumPostingTreeLeaves(gvs, rootBlkno))
+ {
/*
- * All subtree is empty - just return true to indicate that parent
- * must do a cleanup, unless we are ROOT and there is way to go upper.
+ * There is at least one empty page. So we have to rescan the tree
+ * deleting empty pages.
*/
+ Buffer buffer;
+ DataPageDeleteStack root,
+ *ptr,
+ *tmp;
- if (hasEmptyChild && !hasNonEmptyChild && !isRoot)
- return true;
+ buffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM, rootBlkno,
+ RBM_NORMAL, gvs->strategy);
- if (hasEmptyChild)
- {
- DataPageDeleteStack root,
- *ptr,
- *tmp;
-
- buffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM, blkno,
- RBM_NORMAL, gvs->strategy);
- LockBufferForCleanup(buffer);
-
- memset(&root, 0, sizeof(DataPageDeleteStack));
- root.leftBlkno = InvalidBlockNumber;
- root.isRoot = true;
+ /*
+ * Lock posting tree root for cleanup to ensure there are no concurrent
+ * inserts.
+ */
+ LockBufferForCleanup(buffer);
- ginScanToDelete(gvs, blkno, true, &root, InvalidOffsetNumber);
+ memset(&root, 0, sizeof(DataPageDeleteStack));
+ root.leftBlkno = InvalidBlockNumber;
+ root.isRoot = true;
- ptr = root.child;
+ ginScanToDelete(gvs, rootBlkno, true, &root, InvalidOffsetNumber);
- while (ptr)
- {
- tmp = ptr->child;
- pfree(ptr);
- ptr = tmp;
- }
+ ptr = root.child;
- UnlockReleaseBuffer(buffer);
+ while (ptr)
+ {
+ tmp = ptr->child;
+ pfree(ptr);
+ ptr = tmp;
}
- /* Here we have deleted all empty subtrees */
- return false;
+ UnlockReleaseBuffer(buffer);
}
}
-static void
-ginVacuumPostingTree(GinVacuumState *gvs, BlockNumber rootBlkno)
-{
- ginVacuumPostingTreeLeaves(gvs, rootBlkno, true);
-}
-
/*
* returns modified page or NULL if page isn't modified.
* Function works with original page until first change is occurred,
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-12-11 07:14 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Alexander Korotkov <aekorotkov@gmail.com>
0 siblings, 1 reply; 67+ messages in thread
From: Andrey Borodin @ 2018-12-11 07:14 UTC (permalink / raw)
To: Alexander Korotkov <aekorotkov@gmail.com>; +Cc: Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi!
> 11 дек. 2018 г., в 3:43, Alexander Korotkov <aekorotkov@gmail.com> написал(а):
>
> Attached patch appears to be incomplete. GinPageSetDeleteXid() is
> called only in ginRedoDeletePage(), so only in recovery, while it
> should be set during normal work too. deleteXid field of
> ginxlogDeletePage is never set.
Sorry, I've messed it again. Forgot to include ginvacuum.c changes. Here it is.
Best regards, Andrey Borodin.
Attachments:
0001-Stamp-deleted-GIN-page-with-xid-v2.patch (application/octet-stream, 3.4K) download
From 8cf07e104490a201fbd07886855c644bf5bcf3e9 Mon Sep 17 00:00:00 2001
From: Andrey <amborodin@acm.org>
Date: Mon, 10 Dec 2018 18:49:47 +0500
Subject: [PATCH] Stamp deleted GIN page with xid v2
---
src/backend/access/gin/ginutil.c | 7 ++++++-
src/backend/access/gin/ginvacuum.c | 4 ++++
src/backend/access/gin/ginxlog.c | 1 +
src/include/access/gin_private.h | 3 +++
src/include/access/ginxlog.h | 1 +
5 files changed, 15 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index d7696a1ad0..9072ee6260 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -314,7 +314,12 @@ GinNewBuffer(Relation index)
if (PageIsNew(page))
return buffer; /* OK to use, if never initialized */
- if (GinPageIsDeleted(page))
+ /*
+ * We should not reuse page until every transaction started before
+ * deletion is over
+ */
+ if (GinPageIsDeleted(page)
+ && TransactionIdPrecedes(GinPageGetDeleteXid(page), RecentGlobalDataXmin))
return buffer; /* OK to use */
LockBuffer(buffer, GIN_UNLOCK);
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index 3104bc12b6..c53e13edeb 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -16,6 +16,7 @@
#include "access/gin_private.h"
#include "access/ginxlog.h"
+#include "access/transam.h"
#include "access/xloginsert.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
@@ -157,6 +158,9 @@ ginDeletePage(GinVacuumState *gvs, BlockNumber deleteBlkno, BlockNumber leftBlkn
page = BufferGetPage(dBuffer);
rightlink = GinPageGetOpaque(page)->rightlink;
+ /* for each deleted page we remember last xid which could knew it's address */
+ GinPageSetDeleteXid(page, ReadNewTransactionId());
+
/*
* Any insert which would have gone on the leaf block will now go to its
* right sibling.
diff --git a/src/backend/access/gin/ginxlog.c b/src/backend/access/gin/ginxlog.c
index 7701a2d6bf..7821f69077 100644
--- a/src/backend/access/gin/ginxlog.c
+++ b/src/backend/access/gin/ginxlog.c
@@ -518,6 +518,7 @@ ginRedoDeletePage(XLogReaderState *record)
page = BufferGetPage(dbuffer);
Assert(GinPageIsData(page));
GinPageGetOpaque(page)->flags = GIN_DELETED;
+ GinPageSetDeleteXid(page, data->deleteXid);
PageSetLSN(page, lsn);
MarkBufferDirty(dbuffer);
}
diff --git a/src/include/access/gin_private.h b/src/include/access/gin_private.h
index 81bf8734ce..83d2621488 100644
--- a/src/include/access/gin_private.h
+++ b/src/include/access/gin_private.h
@@ -44,6 +44,9 @@ typedef struct GinOptions
#define GIN_SHARE BUFFER_LOCK_SHARE
#define GIN_EXCLUSIVE BUFFER_LOCK_EXCLUSIVE
+#define GinPageGetDeleteXid(page) ( ((PageHeader) (page))->pd_prune_xid )
+#define GinPageSetDeleteXid(page, val) ( ((PageHeader) (page))->pd_prune_xid = val)
+
/*
* GinState: working data structure describing the index being worked on
diff --git a/src/include/access/ginxlog.h b/src/include/access/ginxlog.h
index 64a3c9e18b..d7f44d6b83 100644
--- a/src/include/access/ginxlog.h
+++ b/src/include/access/ginxlog.h
@@ -158,6 +158,7 @@ typedef struct ginxlogDeletePage
{
OffsetNumber parentOffset;
BlockNumber rightLink;
+ TransactionId deleteXid; /* last Xid which could see page in scan */
} ginxlogDeletePage;
#define XLOG_GIN_UPDATE_META_PAGE 0x60
--
2.17.2 (Apple Git-113)
=
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-12-11 22:26 Alexander Korotkov <aekorotkov@gmail.com>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 67+ messages in thread
From: Alexander Korotkov @ 2018-12-11 22:26 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Tue, Dec 11, 2018 at 10:14 AM Andrey Borodin <x4mmm@yandex-team.ru> wrote:
> > 11 дек. 2018 г., в 3:43, Alexander Korotkov <aekorotkov@gmail.com> написал(а):
> >
> > Attached patch appears to be incomplete. GinPageSetDeleteXid() is
> > called only in ginRedoDeletePage(), so only in recovery, while it
> > should be set during normal work too. deleteXid field of
> > ginxlogDeletePage is never set.
>
> Sorry, I've messed it again. Forgot to include ginvacuum.c changes. Here it is.
BTW, I still can't see you're setting deleteXid field of
ginxlogDeletePage struct anywhere.
Also, I note that protection against re-usage of recently deleted
pages is implemented differently than it is in B-tree.
1) You check TransactionIdPrecedes(GinPageGetDeleteXid(page),
RecentGlobalDataXmin)), while B-tree checks
TransactionIdPrecedes(opaque->btpo.xact, RecentGlobalXmin). Could you
clarify why do we use RecentGlobalDataXmin instead of RecentGlobalXmin
here?
2) B-tree checks this condition both before putting page into FSM and
after getting page from FSM. You're checking only after getting page
from FSM. Approach of B-tree looks better for me. It's seems more
consistent when FSM pages are really free for usage.
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-12-12 00:22 Alexander Korotkov <aekorotkov@gmail.com>
parent: Alexander Korotkov <aekorotkov@gmail.com>
0 siblings, 1 reply; 67+ messages in thread
From: Alexander Korotkov @ 2018-12-12 00:22 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Tue, Dec 11, 2018 at 1:50 AM Alexander Korotkov <aekorotkov@gmail.com> wrote:
> On Sun, Dec 9, 2018 at 10:25 PM Alexander Korotkov <aekorotkov@gmail.com> wrote:
> > On Sat, Dec 8, 2018 at 12:48 PM Andrey Borodin <x4mmm@yandex-team.ru> wrote:
> > > > 8 дек. 2018 г., в 6:54, Alexander Korotkov <aekorotkov@gmail.com> написал(а):
> > > >
> > > > Yep, please find attached draft patch.
> > >
> > > Patch seems good to me, I'll check it in more detail.
> > > The patch gets posting item at FirstOffsetNumber instead of btree->getLeftMostChild(). This seem OK, since dataGetLeftMostPage() is doing just the same, but with few Assert()s.
> >
> > I'd like to evade creating GinBtree for just calling
> > getLeftMostChild(). Also, few more places in ginvacuum.c do the same.
> > We have the same amount of Assert()s in ginVacuumPostingTreeLeaves().
> > So, let's keep it uniform.
> >
> > I would also like Peter Geoghegan to take a look at this patch before
> > committing it.
>
> I've slightly adjusted commit message. I'm going to commit this fix
> if no objections.
Please also find patch changing lock order in ginRedoDeletePage()
preventing deadlock on standby. I'm going to commit it too.
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
Attachments:
gin-redo-delete-page-locking-order-1.patch (application/octet-stream, 2.0K) download
commit b1ba9d8a5a647af3088a21597dc65af1631919ea
Author: Alexander Korotkov <akorotkov@postgresql.org>
Date: Wed Dec 12 03:06:07 2018 +0300
Prevent deadlock in ginRedoDeletePage()
On standby ginRedoDeletePage() can work concurrently with read-only queries.
Those queries can traverse posting tree in two ways.
1) Using rightlinks by ginStepRight(), which locks next page before unlocking
its left sibling.
2) Using downlinks by ginFindLeafPage(), which locks at most one page at time.
Original lock order was: page, parent, left sibling. That lock order can
deadlock with ginStepRight(). In order to prevent deadlock this commit changes
lock order to: left sibling, page, parent. Note, that position of parent in
locking order seems insignificant, because we only lock one page at time while
traversing downlinks.
diff --git a/src/backend/access/gin/ginxlog.c b/src/backend/access/gin/ginxlog.c
index 7701a2d6bfc..b626a219dec 100644
--- a/src/backend/access/gin/ginxlog.c
+++ b/src/backend/access/gin/ginxlog.c
@@ -513,6 +513,19 @@ ginRedoDeletePage(XLogReaderState *record)
Buffer lbuffer;
Page page;
+ /*
+ * Lock left page first in order to prevent possible deadlock with
+ * ginStepRight().
+ */
+ if (XLogReadBufferForRedo(record, 2, &lbuffer) == BLK_NEEDS_REDO)
+ {
+ page = BufferGetPage(lbuffer);
+ Assert(GinPageIsData(page));
+ GinPageGetOpaque(page)->rightlink = data->rightLink;
+ PageSetLSN(page, lsn);
+ MarkBufferDirty(lbuffer);
+ }
+
if (XLogReadBufferForRedo(record, 0, &dbuffer) == BLK_NEEDS_REDO)
{
page = BufferGetPage(dbuffer);
@@ -532,15 +545,6 @@ ginRedoDeletePage(XLogReaderState *record)
MarkBufferDirty(pbuffer);
}
- if (XLogReadBufferForRedo(record, 2, &lbuffer) == BLK_NEEDS_REDO)
- {
- page = BufferGetPage(lbuffer);
- Assert(GinPageIsData(page));
- GinPageGetOpaque(page)->rightlink = data->rightLink;
- PageSetLSN(page, lsn);
- MarkBufferDirty(lbuffer);
- }
-
if (BufferIsValid(lbuffer))
UnlockReleaseBuffer(lbuffer);
if (BufferIsValid(pbuffer))
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-12-12 13:08 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Alexander Korotkov <aekorotkov@gmail.com>
0 siblings, 1 reply; 67+ messages in thread
From: Andrey Borodin @ 2018-12-12 13:08 UTC (permalink / raw)
To: Alexander Korotkov <aekorotkov@gmail.com>; +Cc: Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi!
> 12 дек. 2018 г., в 3:26, Alexander Korotkov <aekorotkov@gmail.com> написал(а):
>
> BTW, I still can't see you're setting deleteXid field of
> ginxlogDeletePage struct anywhere.
Oops. Fixed.
>
> Also, I note that protection against re-usage of recently deleted
> pages is implemented differently than it is in B-tree.
> 1) You check TransactionIdPrecedes(GinPageGetDeleteXid(page),
> RecentGlobalDataXmin)), while B-tree checks
> TransactionIdPrecedes(opaque->btpo.xact, RecentGlobalXmin). Could you
> clarify why do we use RecentGlobalDataXmin instead of RecentGlobalXmin
> here?
As far as I understand RecentGlobalDataXmin may be slightly farther than RecentGlobalXmin in case when replication_slot_catalog_xmin is holding RecentGlobalXmin. And GIN is never used by catalog tables. But let's use RecentGlobalXmin like in B-tree.
> 2) B-tree checks this condition both before putting page into FSM and
> after getting page from FSM. You're checking only after getting page
> from FSM. Approach of B-tree looks better for me. It's seems more
> consistent when FSM pages are really free for usage.
Fixed.
Best regards, Andrey Borodin.
Attachments:
0001-Stamp-deleted-GIN-page-with-xid-v3.patch (application/octet-stream, 4.1K) download
From 0560b46f96e82c7082b8fcf434a3c6c9f46a94b8 Mon Sep 17 00:00:00 2001
From: Andrey <amborodin@acm.org>
Date: Mon, 10 Dec 2018 18:49:47 +0500
Subject: [PATCH] Stamp deleted GIN page with xid v3
---
src/backend/access/gin/ginutil.c | 7 ++++++-
src/backend/access/gin/ginvacuum.c | 7 ++++++-
src/backend/access/gin/ginxlog.c | 1 +
src/include/access/gin_private.h | 3 +++
src/include/access/ginxlog.h | 1 +
5 files changed, 17 insertions(+), 2 deletions(-)
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index d7696a1ad0..5ce39d821b 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -314,7 +314,12 @@ GinNewBuffer(Relation index)
if (PageIsNew(page))
return buffer; /* OK to use, if never initialized */
- if (GinPageIsDeleted(page))
+ /*
+ * We should not reuse page until every transaction started before
+ * deletion is over
+ */
+ if (GinPageIsDeleted(page)
+ && TransactionIdPrecedes(GinPageGetDeleteXid(page), RecentGlobalXmin))
return buffer; /* OK to use */
LockBuffer(buffer, GIN_UNLOCK);
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index 3104bc12b6..75dc03f495 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -16,6 +16,7 @@
#include "access/gin_private.h"
#include "access/ginxlog.h"
+#include "access/transam.h"
#include "access/xloginsert.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
@@ -157,6 +158,9 @@ ginDeletePage(GinVacuumState *gvs, BlockNumber deleteBlkno, BlockNumber leftBlkn
page = BufferGetPage(dBuffer);
rightlink = GinPageGetOpaque(page)->rightlink;
+ /* for each deleted page we remember last xid which could knew it's address */
+ GinPageSetDeleteXid(page, ReadNewTransactionId());
+
/*
* Any insert which would have gone on the leaf block will now go to its
* right sibling.
@@ -213,6 +217,7 @@ ginDeletePage(GinVacuumState *gvs, BlockNumber deleteBlkno, BlockNumber leftBlkn
data.parentOffset = myoff;
data.rightLink = GinPageGetOpaque(page)->rightlink;
+ data.deleteXid = GinPageGetDeleteXid(page);
XLogRegisterData((char *) &data, sizeof(ginxlogDeletePage));
@@ -742,7 +747,7 @@ ginvacuumcleanup(IndexVacuumInfo *info, IndexBulkDeleteResult *stats)
LockBuffer(buffer, GIN_SHARE);
page = (Page) BufferGetPage(buffer);
- if (PageIsNew(page) || GinPageIsDeleted(page))
+ if (PageIsNew(page) || (GinPageIsDeleted(page) && TransactionIdPrecedes(GinPageGetDeleteXid(page), RecentGlobalXmin)))
{
Assert(blkno != GIN_ROOT_BLKNO);
RecordFreeIndexPage(index, blkno);
diff --git a/src/backend/access/gin/ginxlog.c b/src/backend/access/gin/ginxlog.c
index 7701a2d6bf..7821f69077 100644
--- a/src/backend/access/gin/ginxlog.c
+++ b/src/backend/access/gin/ginxlog.c
@@ -518,6 +518,7 @@ ginRedoDeletePage(XLogReaderState *record)
page = BufferGetPage(dbuffer);
Assert(GinPageIsData(page));
GinPageGetOpaque(page)->flags = GIN_DELETED;
+ GinPageSetDeleteXid(page, data->deleteXid);
PageSetLSN(page, lsn);
MarkBufferDirty(dbuffer);
}
diff --git a/src/include/access/gin_private.h b/src/include/access/gin_private.h
index 81bf8734ce..83d2621488 100644
--- a/src/include/access/gin_private.h
+++ b/src/include/access/gin_private.h
@@ -44,6 +44,9 @@ typedef struct GinOptions
#define GIN_SHARE BUFFER_LOCK_SHARE
#define GIN_EXCLUSIVE BUFFER_LOCK_EXCLUSIVE
+#define GinPageGetDeleteXid(page) ( ((PageHeader) (page))->pd_prune_xid )
+#define GinPageSetDeleteXid(page, val) ( ((PageHeader) (page))->pd_prune_xid = val)
+
/*
* GinState: working data structure describing the index being worked on
diff --git a/src/include/access/ginxlog.h b/src/include/access/ginxlog.h
index 64a3c9e18b..d7f44d6b83 100644
--- a/src/include/access/ginxlog.h
+++ b/src/include/access/ginxlog.h
@@ -158,6 +158,7 @@ typedef struct ginxlogDeletePage
{
OffsetNumber parentOffset;
BlockNumber rightLink;
+ TransactionId deleteXid; /* last Xid which could see page in scan */
} ginxlogDeletePage;
#define XLOG_GIN_UPDATE_META_PAGE 0x60
--
2.17.2 (Apple Git-113)
=
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-12-13 12:03 Alexander Korotkov <aekorotkov@gmail.com>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 2 replies; 67+ messages in thread
From: Alexander Korotkov @ 2018-12-13 12:03 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Wed, Dec 12, 2018 at 4:08 PM Andrey Borodin <x4mmm@yandex-team.ru> wrote:
> > 12 дек. 2018 г., в 3:26, Alexander Korotkov <aekorotkov@gmail.com> написал(а):
> >
> > BTW, I still can't see you're setting deleteXid field of
> > ginxlogDeletePage struct anywhere.
> Oops. Fixed.
> >
> > Also, I note that protection against re-usage of recently deleted
> > pages is implemented differently than it is in B-tree.
> > 1) You check TransactionIdPrecedes(GinPageGetDeleteXid(page),
> > RecentGlobalDataXmin)), while B-tree checks
> > TransactionIdPrecedes(opaque->btpo.xact, RecentGlobalXmin). Could you
> > clarify why do we use RecentGlobalDataXmin instead of RecentGlobalXmin
> > here?
> As far as I understand RecentGlobalDataXmin may be slightly farther than RecentGlobalXmin in case when replication_slot_catalog_xmin is holding RecentGlobalXmin. And GIN is never used by catalog tables. But let's use RecentGlobalXmin like in B-tree.
>
> > 2) B-tree checks this condition both before putting page into FSM and
> > after getting page from FSM. You're checking only after getting page
> > from FSM. Approach of B-tree looks better for me. It's seems more
> > consistent when FSM pages are really free for usage.
> Fixed.
Thank you. I've revised your patch and pushed it. As long as two
other patches in this thread.
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re:Connections hang indefinitely while taking a gin index's LWLock
buffer_content lock
@ 2018-12-13 15:12 chjischj@163.com <chjischj@163.com>
parent: Alexander Korotkov <aekorotkov@gmail.com>
0 siblings, 0 replies; 67+ messages in thread
From: chjischj@163.com @ 2018-12-13 15:12 UTC (permalink / raw)
To: Alexander Korotkov <aekorotkov@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"></head><body><div class="quote" style="line-height: 1.5"><aekorotkov@gmail.com><x4mmm@yandex-team.ru><pg@bowt.ie><andres@anarazel.de><chjischj@163.com><teodor@sigaev.ru><pgsql-hackers@lists.postgresql.org><br><div class="quote" style="line-height: 1.5">hi</div><div class="quote" style="line-height: 1.5">I Have a question. Why the order of unlocking is not adjusted in this patch? like this:</div><div class="quote" style="line-height: 1.5"><br></div><div class="quote" style="line-height: 1.5">if (BufferIsValid(lbuffer))</div><div class="quote" style="line-height: 1.5">UnlockReleaseBuffer(lbuffer);</div><div class="quote" style="line-height: 1.5">if (BufferIsValid(pbuffer))</div><div class="quote" style="line-height: 1.5">UnlockReleaseBuffer(pbuffer);</div><div class="quote" style="line-height: 1.5">if (BufferIsValid(dbuffer))</div><div class="quote" style="line-height: 1.5">UnlockReleaseBuffer(dbuffer);</div><div class="quote" style="line-height: 1.5">==></div><div class="quote" style="line-height: 1.5">if (BufferIsValid(pbuffer))</div><div class="quote" style="line-height: 1.5">UnlockReleaseBuffer(pbuffer);</div><div class="quote" style="line-height: 1.5">if (BufferIsValid(dbuffer))</div><div class="quote" style="line-height: 1.5">UnlockReleaseBuffer(dbuffer);</div><div class="quote" style="line-height: 1.5">if (BufferIsValid(lbuffer))</div><div class="quote" style="line-height: 1.5">UnlockReleaseBuffer(lbuffer);</div><div class="quote" style="line-height: 1.5"><br></div><br type="attribution"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Tue, Dec 11, 2018 at 1:50 AM Alexander Korotkov <aekorotkov@gmail.com> wrote:<br>> On Sun, Dec 9, 2018 at 10:25 PM Alexander Korotkov <aekorotkov@gmail.com> wrote:<br>> > On Sat, Dec 8, 2018 at 12:48 PM Andrey Borodin <x4mmm@yandex-team.ru> wrote:<br>> > > > 8 дек. 2018 г., в 6:54, Alexander Korotkov <aekorotkov@gmail.com> написал(а):<br>> > > ><br>> > > > Yep, please find attached draft patch.<br>> > ><br>> > > Patch seems good to me, I'll check it in more detail.<br>> > > The patch gets posting item at FirstOffsetNumber instead of btree->getLeftMostChild(). This seem OK, since dataGetLeftMostPage() is doing just the same, but with few Assert()s.<br>> ><br>> > I'd like to evade creating GinBtree for just calling<br>> > getLeftMostChild(). Also, few more places in ginvacuum.c do the same.<br>> > We have the same amount of Assert()s in ginVacuumPostingTreeLeaves().<br>> > So, let's keep it uniform.<br>> ><br>> > I would also like Peter Geoghegan to take a look at this patch before<br>> > committing it.<br>><br>> I've slightly adjusted commit message. I'm going to commit this fix<br>> if no objections.<br><br>Please also find patch changing lock order in ginRedoDeletePage()<br>preventing deadlock on standby. I'm going to commit it too.<br><br>------<br>Alexander Korotkov<br>Postgres Professional: http://www.postgrespro.com<br>The Russian Postgres Company</aekorotkov@gmail.com></x4mmm@yandex-team.ru></aekorotkov@gmail.com></aekorotkov@gmail.com></blockquote><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br></blockquote><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br></blockquote></pgsql-hackers@lists.postgresql.org></teodor@sigaev.ru></chjischj@163.com></andres@anarazel.de></pg@bowt.ie></x4mmm@yandex-team.ru></aekorotkov@gmail.com></div></body></html>
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-12-13 15:32 Bruce Momjian <bruce@momjian.us>
parent: Alexander Korotkov <aekorotkov@gmail.com>
1 sibling, 1 reply; 67+ messages in thread
From: Bruce Momjian @ 2018-12-13 15:32 UTC (permalink / raw)
To: Alexander Korotkov <aekorotkov@gmail.com>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Thu, Dec 13, 2018 at 03:03:54PM +0300, Alexander Korotkov wrote:
> On Wed, Dec 12, 2018 at 4:08 PM Andrey Borodin <x4mmm@yandex-team.ru> wrote:
> > > 12 дек. 2018 г., в 3:26, Alexander Korotkov <aekorotkov@gmail.com> написал(а):
> > >
> > > BTW, I still can't see you're setting deleteXid field of
> > > ginxlogDeletePage struct anywhere.
> > Oops. Fixed.
> > >
> > > Also, I note that protection against re-usage of recently deleted
> > > pages is implemented differently than it is in B-tree.
> > > 1) You check TransactionIdPrecedes(GinPageGetDeleteXid(page),
> > > RecentGlobalDataXmin)), while B-tree checks
> > > TransactionIdPrecedes(opaque->btpo.xact, RecentGlobalXmin). Could you
> > > clarify why do we use RecentGlobalDataXmin instead of RecentGlobalXmin
> > > here?
> > As far as I understand RecentGlobalDataXmin may be slightly farther than RecentGlobalXmin in case when replication_slot_catalog_xmin is holding RecentGlobalXmin. And GIN is never used by catalog tables. But let's use RecentGlobalXmin like in B-tree.
> >
> > > 2) B-tree checks this condition both before putting page into FSM and
> > > after getting page from FSM. You're checking only after getting page
> > > from FSM. Approach of B-tree looks better for me. It's seems more
> > > consistent when FSM pages are really free for usage.
> > Fixed.
>
> Thank you. I've revised your patch and pushed it. As long as two
> other patches in this thread.
I am seeing this warning in the 9.4 branch:
ginxlog.c:756:5: warning: ‘lbuffer’ may be used uninitialized
in this function [-Wmaybe-uninitialized]
from this commit:
commit 19cf52e6cc33b9e126775f28269ccccb6ddf7e30
Author: Alexander Korotkov <akorotkov@postgresql.org>
Date: Thu Dec 13 06:12:25 2018 +0300
Prevent deadlock in ginRedoDeletePage()
On standby ginRedoDeletePage() can work concurrently with read-only queries.
Those queries can traverse posting tree in two ways.
1) Using rightlinks by ginStepRight(), which locks the next page before
unlocking its left sibling.
2) Using downlinks by ginFindLeafPage(), which locks at most one page at time.
Original lock order was: page, parent, left sibling. That lock order can
deadlock with ginStepRight(). In order to prevent deadlock this commit changes
lock order to: left sibling, page, parent. Note, that position of parent in
locking order seems insignificant, because we only lock one page at time while
traversing downlinks.
Reported-by: Chen Huajun
Diagnosed-by: Chen Huajun, Peter Geoghegan, Andrey Borodin
Discussion: https://postgr.es/m/31a702a.14dd.166c1366ac1.Coremail.chjischj%40163.com
Author: Alexander Korotkov
Backpatch-through: 9.4
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-12-13 16:48 Tom Lane <tgl@sss.pgh.pa.us>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 0 replies; 67+ messages in thread
From: Tom Lane @ 2018-12-13 16:48 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Alexander Korotkov <aekorotkov@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Bruce Momjian <bruce@momjian.us> writes:
> I am seeing this warning in the 9.4 branch:
> ginxlog.c:756:5: warning: ‘lbuffer’ may be used uninitialized
> in this function [-Wmaybe-uninitialized]
I'm also getting that, just in 9.4, but at a different line number:
ginxlog.c: In function 'ginRedoDeletePage':
ginxlog.c:693: warning: 'lbuffer' may be used uninitialized in this function
That's with gcc version 4.4.7 20120313 (Red Hat 4.4.7-23)
regards, tom lane
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-12-13 17:06 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Alexander Korotkov <aekorotkov@gmail.com>
1 sibling, 1 reply; 67+ messages in thread
From: Andrey Borodin @ 2018-12-13 17:06 UTC (permalink / raw)
To: Alexander Korotkov <aekorotkov@gmail.com>; +Cc: Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi!
> 13 дек. 2018 г., в 17:03, Alexander Korotkov <aekorotkov@gmail.com> написал(а):
>
> Thank you. I've revised your patch and pushed it. As long as two
> other patches in this thread.
That's great! Thanks!
> 13 дек. 2018 г., в 20:12, chjischj@163.com написал(а):
>
>
> hi
> I Have a question. Why the order of unlocking is not adjusted in this patch? like this:
>
> if (BufferIsValid(lbuffer))
> UnlockReleaseBuffer(lbuffer);
> if (BufferIsValid(pbuffer))
> UnlockReleaseBuffer(pbuffer);
> if (BufferIsValid(dbuffer))
> UnlockReleaseBuffer(dbuffer);
> ==>
> if (BufferIsValid(pbuffer))
> UnlockReleaseBuffer(pbuffer);
> if (BufferIsValid(dbuffer))
> UnlockReleaseBuffer(dbuffer);
> if (BufferIsValid(lbuffer))
> UnlockReleaseBuffer(lbuffer);
I think that unlock order does not matter. But I may be wrong. May be just for uniformity?
> 13 дек. 2018 г., в 21:48, Tom Lane <tgl@sss.pgh.pa.us> написал(а):
>
> Bruce Momjian <bruce@momjian.us> writes:
>> I am seeing this warning in the 9.4 branch:
>> ginxlog.c:756:5: warning: ‘lbuffer’ may be used uninitialized
>> in this function [-Wmaybe-uninitialized]
>
> I'm also getting that, just in 9.4, but at a different line number:
>
> ginxlog.c: In function 'ginRedoDeletePage':
> ginxlog.c:693: warning: 'lbuffer' may be used uninitialized in this function
>
> That's with gcc version 4.4.7 20120313 (Red Hat 4.4.7-23)
That's the same variable, one place is definition while other is potential misuse.
Seems like these 2 lines [0]
+ if (BufferIsValid(lbuffer))
+ UnlockReleaseBuffer(lbuffer);
are superfluous: lbuffer is UnlockReleased earlier.
Best regards, Andrey Borodin.
[0] https://github.com/postgres/postgres/commit/19cf52e6cc33b9e126775f28269ccccb6ddf7e30#diff-ed6446a899...
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-12-13 19:40 Alexander Korotkov <aekorotkov@gmail.com>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 2 replies; 67+ messages in thread
From: Alexander Korotkov @ 2018-12-13 19:40 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Thu, Dec 13, 2018 at 8:06 PM Andrey Borodin <x4mmm@yandex-team.ru> wrote:
> > if (BufferIsValid(lbuffer))
> > UnlockReleaseBuffer(lbuffer);
> > if (BufferIsValid(pbuffer))
> > UnlockReleaseBuffer(pbuffer);
> > if (BufferIsValid(dbuffer))
> > UnlockReleaseBuffer(dbuffer);
> > ==>
> > if (BufferIsValid(pbuffer))
> > UnlockReleaseBuffer(pbuffer);
> > if (BufferIsValid(dbuffer))
> > UnlockReleaseBuffer(dbuffer);
> > if (BufferIsValid(lbuffer))
> > UnlockReleaseBuffer(lbuffer);
>
> I think that unlock order does not matter. But I may be wrong. May be just for uniformity?
It doesn't mater, because we release all locks on every buffer at one
time. The unlock order can have effect on what waiter will acquire
the lock next after ginRedoDeletePage(). However, I don't see why one
unlock order is better than another. Thus, I just used the rule of
thumb to not change code when it's not necessary for bug fix.
> > 13 дек. 2018 г., в 21:48, Tom Lane <tgl@sss.pgh.pa.us> написал(а):
> >
> > Bruce Momjian <bruce@momjian.us> writes:
> >> I am seeing this warning in the 9.4 branch:
> >> ginxlog.c:756:5: warning: ‘lbuffer’ may be used uninitialized
> >> in this function [-Wmaybe-uninitialized]
> >
> > I'm also getting that, just in 9.4, but at a different line number:
> >
> > ginxlog.c: In function 'ginRedoDeletePage':
> > ginxlog.c:693: warning: 'lbuffer' may be used uninitialized in this function
> >
> > That's with gcc version 4.4.7 20120313 (Red Hat 4.4.7-23)
>
>
> That's the same variable, one place is definition while other is potential misuse.
> Seems like these 2 lines [0]
>
> + if (BufferIsValid(lbuffer))
> + UnlockReleaseBuffer(lbuffer);
>
> are superfluous: lbuffer is UnlockReleased earlier.
Thanks to everybody for noticing! Speaking more generally backpatch
to 9.4 was completely wrong: there were multiple errors. It's my
oversight, I forget how much better our xlog system became since 9.4
:)
I've pushed bf0e5a73be fixing that.
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-12-13 19:46 Andres Freund <andres@anarazel.de>
parent: Alexander Korotkov <aekorotkov@gmail.com>
1 sibling, 1 reply; 67+ messages in thread
From: Andres Freund @ 2018-12-13 19:46 UTC (permalink / raw)
To: Alexander Korotkov <aekorotkov@gmail.com>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Peter Geoghegan <pg@bowt.ie>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi,
On 2018-12-13 22:40:59 +0300, Alexander Korotkov wrote:
> It doesn't mater, because we release all locks on every buffer at one
> time. The unlock order can have effect on what waiter will acquire
> the lock next after ginRedoDeletePage(). However, I don't see why one
> unlock order is better than another. Thus, I just used the rule of
> thumb to not change code when it's not necessary for bug fix.
I think it's right to not change unlock order at the same time as a
bugfix here. More generally I think it can often be useful to default
to release locks in the inverse order they've been acquired - if there's
any likelihood that somebody will acquire them in the same order, that
ensures that such a party would only need to wait for a lock once,
instead of being woken up for one lock, and then immediately having to
wait for the next one.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-12-13 19:48 Alexander Korotkov <aekorotkov@gmail.com>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 67+ messages in thread
From: Alexander Korotkov @ 2018-12-13 19:48 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Peter Geoghegan <pg@bowt.ie>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Thu, Dec 13, 2018 at 10:46 PM Andres Freund <andres@anarazel.de> wrote:
> On 2018-12-13 22:40:59 +0300, Alexander Korotkov wrote:
> > It doesn't mater, because we release all locks on every buffer at one
> > time. The unlock order can have effect on what waiter will acquire
> > the lock next after ginRedoDeletePage(). However, I don't see why one
> > unlock order is better than another. Thus, I just used the rule of
> > thumb to not change code when it's not necessary for bug fix.
>
> I think it's right to not change unlock order at the same time as a
> bugfix here. More generally I think it can often be useful to default
> to release locks in the inverse order they've been acquired - if there's
> any likelihood that somebody will acquire them in the same order, that
> ensures that such a party would only need to wait for a lock once,
> instead of being woken up for one lock, and then immediately having to
> wait for the next one.
Good point, thank you!
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2018-12-14 16:19 Bruce Momjian <bruce@momjian.us>
parent: Alexander Korotkov <aekorotkov@gmail.com>
1 sibling, 0 replies; 67+ messages in thread
From: Bruce Momjian @ 2018-12-14 16:19 UTC (permalink / raw)
To: Alexander Korotkov <aekorotkov@gmail.com>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Thu, Dec 13, 2018 at 10:40:59PM +0300, Alexander Korotkov wrote:
> On Thu, Dec 13, 2018 at 8:06 PM Andrey Borodin <x4mmm@yandex-team.ru> wrote:
> > That's the same variable, one place is definition while other is potential misuse.
> > Seems like these 2 lines [0]
> >
> > + if (BufferIsValid(lbuffer))
> > + UnlockReleaseBuffer(lbuffer);
> >
> > are superfluous: lbuffer is UnlockReleased earlier.
>
> Thanks to everybody for noticing! Speaking more generally backpatch
> to 9.4 was completely wrong: there were multiple errors. It's my
> oversight, I forget how much better our xlog system became since 9.4
> :)
>
> I've pushed bf0e5a73be fixing that.
I can confirm the compiler warning is now gone. Thanks.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2019-03-21 18:25 Simon Riggs <simon@2ndquadrant.com>
parent: Alexander Korotkov <aekorotkov@gmail.com>
0 siblings, 1 reply; 67+ messages in thread
From: Simon Riggs @ 2019-03-21 18:25 UTC (permalink / raw)
To: Alexander Korotkov <aekorotkov@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Andrey Borodin <x4mmm@yandex-team.ru>; Peter Geoghegan <pg@bowt.ie>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Thu, 13 Dec 2018 at 14:48, Alexander Korotkov <aekorotkov@gmail.com>
wrote:
> On Thu, Dec 13, 2018 at 10:46 PM Andres Freund <andres@anarazel.de> wrote:
> > On 2018-12-13 22:40:59 +0300, Alexander Korotkov wrote:
> > > It doesn't mater, because we release all locks on every buffer at one
> > > time. The unlock order can have effect on what waiter will acquire
> > > the lock next after ginRedoDeletePage(). However, I don't see why one
> > > unlock order is better than another. Thus, I just used the rule of
> > > thumb to not change code when it's not necessary for bug fix.
> >
> > I think it's right to not change unlock order at the same time as a
> > bugfix here. More generally I think it can often be useful to default
> > to release locks in the inverse order they've been acquired - if there's
> > any likelihood that somebody will acquire them in the same order, that
> > ensures that such a party would only need to wait for a lock once,
> > instead of being woken up for one lock, and then immediately having to
> > wait for the next one.
>
> Good point, thank you!
>
It's been pointed out to me that 52ac6cd2d0cd70e01291e0ac4ee6d068b69bc478
introduced a WAL incompatibility that has not been flagged.
In ginRedoDeletePage() we use the struct directly to read the WAL record,
so if a WAL record was written prior to
52ac6cd2d0cd70e01291e0ac4ee6d068b69bc478, yet read by code at
52ac6cd2d0cd70e01291e0ac4ee6d068b69bc478 or later then we will have
problems, since deleteXid will not be set correctly.
It seems this should not have been backpatched.
Please give your assessment.
--
Simon Riggs http://www.2ndQuadrant.com/
<http://www.2ndquadrant.com/;
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2019-03-21 19:18 Alexander Korotkov <aekorotkov@gmail.com>
parent: Simon Riggs <simon@2ndquadrant.com>
0 siblings, 2 replies; 67+ messages in thread
From: Alexander Korotkov @ 2019-03-21 19:18 UTC (permalink / raw)
To: Simon Riggs <simon@2ndquadrant.com>; +Cc: Andres Freund <andres@anarazel.de>; Andrey Borodin <x4mmm@yandex-team.ru>; Peter Geoghegan <pg@bowt.ie>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Thu, Mar 21, 2019 at 9:26 PM Simon Riggs <simon@2ndquadrant.com> wrote:
> On Thu, 13 Dec 2018 at 14:48, Alexander Korotkov <aekorotkov@gmail.com> wrote:
>> On Thu, Dec 13, 2018 at 10:46 PM Andres Freund <andres@anarazel.de> wrote:
>> > On 2018-12-13 22:40:59 +0300, Alexander Korotkov wrote:
>> > > It doesn't mater, because we release all locks on every buffer at one
>> > > time. The unlock order can have effect on what waiter will acquire
>> > > the lock next after ginRedoDeletePage(). However, I don't see why one
>> > > unlock order is better than another. Thus, I just used the rule of
>> > > thumb to not change code when it's not necessary for bug fix.
>> >
>> > I think it's right to not change unlock order at the same time as a
>> > bugfix here. More generally I think it can often be useful to default
>> > to release locks in the inverse order they've been acquired - if there's
>> > any likelihood that somebody will acquire them in the same order, that
>> > ensures that such a party would only need to wait for a lock once,
>> > instead of being woken up for one lock, and then immediately having to
>> > wait for the next one.
>>
>> Good point, thank you!
>
>
> It's been pointed out to me that 52ac6cd2d0cd70e01291e0ac4ee6d068b69bc478
> introduced a WAL incompatibility that has not been flagged.
>
> In ginRedoDeletePage() we use the struct directly to read the WAL record, so if a WAL record was written prior to 52ac6cd2d0cd70e01291e0ac4ee6d068b69bc478, yet read by code at 52ac6cd2d0cd70e01291e0ac4ee6d068b69bc478 or later then we will have problems, since deleteXid will not be set correctly.
>
> It seems this should not have been backpatched.
>
> Please give your assessment.
Oh, right. This is my fault.
However, I think this still can be backpatched correctly. We can
determine whether xlog record data contains deleteXid by its size.
See the attached patch. I haven't test this yet. I'm going to test
it. If OK, then push.
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
Attachments:
gin-redo-delete-page-backpatch-fix.patch (application/octet-stream, 591B) download
diff --git a/src/backend/access/gin/ginxlog.c b/src/backend/access/gin/ginxlog.c
index c467ffa346d..6203d1ca990 100644
--- a/src/backend/access/gin/ginxlog.c
+++ b/src/backend/access/gin/ginxlog.c
@@ -531,7 +531,8 @@ ginRedoDeletePage(XLogReaderState *record)
page = BufferGetPage(dbuffer);
Assert(GinPageIsData(page));
GinPageGetOpaque(page)->flags = GIN_DELETED;
- GinPageSetDeleteXid(page, data->deleteXid);
+ if (XLogRecGetDataLen(record) == sizeof(ginxlogDeletePage))
+ GinPageSetDeleteXid(page, data->deleteXid);
PageSetLSN(page, lsn);
MarkBufferDirty(dbuffer);
}
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2019-03-21 20:14 Simon Riggs <simon@2ndquadrant.com>
parent: Alexander Korotkov <aekorotkov@gmail.com>
1 sibling, 0 replies; 67+ messages in thread
From: Simon Riggs @ 2019-03-21 20:14 UTC (permalink / raw)
To: Alexander Korotkov <aekorotkov@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Andrey Borodin <x4mmm@yandex-team.ru>; Peter Geoghegan <pg@bowt.ie>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Thu, 21 Mar 2019 at 15:18, Alexander Korotkov <aekorotkov@gmail.com>
wrote:
> On Thu, Mar 21, 2019 at 9:26 PM Simon Riggs <simon@2ndquadrant.com> wrote:
>
>
> > It's been pointed out to me that 52ac6cd2d0cd70e01291e0ac4ee6d068b69bc478
> > introduced a WAL incompatibility that has not been flagged.
> >
> > In ginRedoDeletePage() we use the struct directly to read the WAL
> record, so if a WAL record was written prior to
> 52ac6cd2d0cd70e01291e0ac4ee6d068b69bc478, yet read by code at
> 52ac6cd2d0cd70e01291e0ac4ee6d068b69bc478 or later then we will have
> problems, since deleteXid will not be set correctly.
> >
> > It seems this should not have been backpatched.
> >
> > Please give your assessment.
>
> Oh, right. This is my fault.
>
> However, I think this still can be backpatched correctly. We can
> determine whether xlog record data contains deleteXid by its size.
> See the attached patch. I haven't test this yet. I'm going to test
> it. If OK, then push.
>
Patch looks like it will work.
I'd prefer to see a tighter test, since the length should either be the old
or new length, no other.
Thanks
--
Simon Riggs http://www.2ndQuadrant.com/
<http://www.2ndquadrant.com/;
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2019-03-21 21:06 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Alexander Korotkov <aekorotkov@gmail.com>
1 sibling, 1 reply; 67+ messages in thread
From: Alvaro Herrera @ 2019-03-21 21:06 UTC (permalink / raw)
To: Alexander Korotkov <aekorotkov@gmail.com>; +Cc: Simon Riggs <simon@2ndquadrant.com>; Andres Freund <andres@anarazel.de>; Andrey Borodin <x4mmm@yandex-team.ru>; Peter Geoghegan <pg@bowt.ie>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On 2019-Mar-21, Alexander Korotkov wrote:
> However, I think this still can be backpatched correctly. We can
> determine whether xlog record data contains deleteXid by its size.
> See the attached patch. I haven't test this yet. I'm going to test
> it. If OK, then push.
Wow, this is so magical that I think it merits a long comment explaining
what is going on.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2019-03-22 08:05 Alexander Korotkov <aekorotkov@gmail.com>
parent: Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 1 reply; 67+ messages in thread
From: Alexander Korotkov @ 2019-03-22 08:05 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Simon Riggs <simon@2ndquadrant.com>; Andres Freund <andres@anarazel.de>; Andrey Borodin <x4mmm@yandex-team.ru>; Peter Geoghegan <pg@bowt.ie>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Fri, Mar 22, 2019 at 12:06 AM Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:
> On 2019-Mar-21, Alexander Korotkov wrote:
>
> > However, I think this still can be backpatched correctly. We can
> > determine whether xlog record data contains deleteXid by its size.
> > See the attached patch. I haven't test this yet. I'm going to test
> > it. If OK, then push.
>
> Wow, this is so magical that I think it merits a long comment explaining
> what is going on.
Yeah, have to fo magic to fix such weird things :)
Please, find next revision of patch attached. It uses static
assertion to check that data structure size has changed. Also,
assertion checks that record data length match on of structures
length.
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
Attachments:
gin-redo-delete-page-backpatch-fix-2.patch (application/octet-stream, 1.9K) download
diff --git a/src/backend/access/gin/ginxlog.c b/src/backend/access/gin/ginxlog.c
index b84ecf2ab15..f3c82ac836c 100644
--- a/src/backend/access/gin/ginxlog.c
+++ b/src/backend/access/gin/ginxlog.c
@@ -531,7 +531,24 @@ ginRedoDeletePage(XLogReaderState *record)
page = BufferGetPage(dbuffer);
Assert(GinPageIsData(page));
GinPageGetOpaque(page)->flags = GIN_DELETED;
- GinPageSetDeleteXid(page, data->deleteXid);
+
+ /*
+ * deleteXid field of ginxlogDeletePage was added during backpatching.
+ * But, non-backpatched instances will continue generate WAL without
+ * this field. We should be able to correctly apply that. We can
+ * distinguish new WAL records by size their data, because
+ * ginxlogDeletePage changes its size on both 32-bit and 64-bit
+ * platforms.
+ */
+ StaticAssertStmt(sizeof(ginxlogDeletePage) !=
+ sizeof(ginxlogDeletePageOld),
+ "ginxlogDeletePage size should be changed "
+ "with addition of deleteXid field");
+ Assert(XLogRecGetDataLen(record) == sizeof(ginxlogDeletePage) ||
+ XLogRecGetDataLen(record) == sizeof(ginxlogDeletePageOld));
+ if (XLogRecGetDataLen(record) == sizeof(ginxlogDeletePage))
+ GinPageSetDeleteXid(page, data->deleteXid);
+
PageSetLSN(page, lsn);
MarkBufferDirty(dbuffer);
}
diff --git a/src/include/access/ginxlog.h b/src/include/access/ginxlog.h
index b2f3126aa8a..87b1e7bbda3 100644
--- a/src/include/access/ginxlog.h
+++ b/src/include/access/ginxlog.h
@@ -161,6 +161,16 @@ typedef struct ginxlogDeletePage
TransactionId deleteXid; /* last Xid which could see this page in scan */
} ginxlogDeletePage;
+/*
+ * Previous version of ginxlogDeletePage struct, which didn't have deleteXid
+ * field. Used for size comparison (see ginRedoDeletePage()).
+ */
+typedef struct ginxlogDeletePageOld
+{
+ OffsetNumber parentOffset;
+ BlockNumber rightLink;
+} ginxlogDeletePageOld;
+
#define XLOG_GIN_UPDATE_META_PAGE 0x60
/*
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2019-03-23 08:43 Alexander Korotkov <aekorotkov@gmail.com>
parent: Alexander Korotkov <aekorotkov@gmail.com>
0 siblings, 1 reply; 67+ messages in thread
From: Alexander Korotkov @ 2019-03-23 08:43 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Simon Riggs <simon@2ndquadrant.com>; Andres Freund <andres@anarazel.de>; Andrey Borodin <x4mmm@yandex-team.ru>; Peter Geoghegan <pg@bowt.ie>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Fri, Mar 22, 2019 at 11:05 AM Alexander Korotkov
<aekorotkov@gmail.com> wrote:
> On Fri, Mar 22, 2019 at 12:06 AM Alvaro Herrera
> <alvherre@2ndquadrant.com> wrote:
> > On 2019-Mar-21, Alexander Korotkov wrote:
> >
> > > However, I think this still can be backpatched correctly. We can
> > > determine whether xlog record data contains deleteXid by its size.
> > > See the attached patch. I haven't test this yet. I'm going to test
> > > it. If OK, then push.
> >
> > Wow, this is so magical that I think it merits a long comment explaining
> > what is going on.
>
> Yeah, have to fo magic to fix such weird things :)
> Please, find next revision of patch attached. It uses static
> assertion to check that data structure size has changed. Also,
> assertion checks that record data length match on of structures
> length.
I'm going to backpatch this on no objections.
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock
@ 2019-03-24 13:35 Alexander Korotkov <aekorotkov@gmail.com>
parent: Alexander Korotkov <aekorotkov@gmail.com>
0 siblings, 0 replies; 67+ messages in thread
From: Alexander Korotkov @ 2019-03-24 13:35 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Simon Riggs <simon@2ndquadrant.com>; Andres Freund <andres@anarazel.de>; Andrey Borodin <x4mmm@yandex-team.ru>; Peter Geoghegan <pg@bowt.ie>; chenhj <chjischj@163.com>; Teodor Sigaev <teodor@sigaev.ru>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Sat, Mar 23, 2019 at 11:43 AM Alexander Korotkov
<aekorotkov@gmail.com> wrote:
> On Fri, Mar 22, 2019 at 11:05 AM Alexander Korotkov
> <aekorotkov@gmail.com> wrote:
> > On Fri, Mar 22, 2019 at 12:06 AM Alvaro Herrera
> > <alvherre@2ndquadrant.com> wrote:
> > > On 2019-Mar-21, Alexander Korotkov wrote:
> > >
> > > > However, I think this still can be backpatched correctly. We can
> > > > determine whether xlog record data contains deleteXid by its size.
> > > > See the attached patch. I haven't test this yet. I'm going to test
> > > > it. If OK, then push.
> > >
> > > Wow, this is so magical that I think it merits a long comment explaining
> > > what is going on.
> >
> > Yeah, have to fo magic to fix such weird things :)
> > Please, find next revision of patch attached. It uses static
> > assertion to check that data structure size has changed. Also,
> > assertion checks that record data length match on of structures
> > length.
>
> I'm going to backpatch this on no objections.
So, pushed.
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 67+ messages in thread
* Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7)
@ 2019-09-29 08:16 chenhj <chjischj@163.com>
0 siblings, 1 reply; 67+ messages in thread
From: chenhj @ 2019-09-29 08:16 UTC (permalink / raw)
To: pgsql-hackers@lists.postgresql.org
Hi,all
In our PostgreSQL 10.7(rhel 6.3) database, autovacuum process and many insert processes blocked in gin index's LWLock:buffer_content for long time.
In other words, the following gin index lwlock deadlock phenomenon has occurred again. Since the following bug in 10.7 has been fixed. So this should be a new bug.
https://www.postgresql.org/message-id/flat/31a702a.14dd.166c1366ac1.Coremail.chjischj%40163.com
We have already obtained coredump files of autovacuum process and one of insert processes.
Unfortunately the insert process(run by gcore) held no lwlock, it should be another process(we did not fetch core file) that hold the lwlock needed for autovacuum process.
the stack is as following:
## stack of one insert process: Acquire lock 0x7f6c517dbfa4 which was held by vacuum process
----------------------------------------------------------------------------
(gdb) bt
#0 0x000000369ea0da00 in sem_wait () from /lib64/libpthread.so.0
#1 0x00000000006a7910 in PGSemaphoreLock (sema=0x7f6c4f76a7b8) at pg_sema.c:316
#2 0x0000000000718225 in LWLockAcquire (lock=0x7f6c517dbfa4, mode=LW_SHARED) at lwlock.c:1233
#3 0x000000000048b622 in ginTraverseLock (buffer=224225, searchMode=0 '\000') at ginbtree.c:40
#4 0x000000000048ca13 in ginFindLeafPage (btree=0x7fffc71c4ea0, searchMode=0 '\000', snapshot=0x0) at ginbtree.c:97
#5 0x00000000004894db in ginInsertItemPointers (index=<value optimized out>, rootBlkno=<value optimized out>, items=<value optimized out>, nitem=<value optimized out>, buildStats=0x0)
at gindatapage.c:1909
#6 0x00000000004863a7 in ginEntryInsert (ginstate=0x1c72158, attnum=1, key=20190913, category=0 '\000', items=0x1c81508, nitem=72, buildStats=0x0) at gininsert.c:214
#7 0x000000000049219a in ginInsertCleanup (ginstate=0x1c72158, full_clean=0 '\000', fill_fsm=1 '\001', forceCleanup=<value optimized out>, stats=<value optimized out>) at ginfast.c:878
#8 0x000000000049308e in ginHeapTupleFastInsert (ginstate=0x1c72158, collector=<value optimized out>) at ginfast.c:443
#9 0x0000000000486749 in gininsert (index=<value optimized out>, values=0x7fffc71c54f0, isnull=0x7fffc71c5600 "", ht_ctid=0x1c6d3a4, heapRel=<value optimized out>,
checkUnique=<value optimized out>, indexInfo=0x1c61da8) at gininsert.c:522
#10 0x00000000005f75f0 in ExecInsertIndexTuples (slot=0x1c62168, tupleid=0x1c6d3a4, estate=0x1c61768, noDupErr=0 '\000', specConflict=0x0, arbiterIndexes=0x0) at execIndexing.c:387
#11 0x0000000000616497 in ExecInsert (pstate=0x1c61ab8) at nodeModifyTable.c:519
#12 ExecModifyTable (pstate=0x1c61ab8) at nodeModifyTable.c:1779
#13 0x00000000005fb6bf in ExecProcNode (queryDesc=0x1c67760, direction=<value optimized out>, count=0, execute_once=-72 '\270') at ../../../src/include/executor/executor.h:250
#14 ExecutePlan (queryDesc=0x1c67760, direction=<value optimized out>, count=0, execute_once=-72 '\270') at execMain.c:1723
#15 standard_ExecutorRun (queryDesc=0x1c67760, direction=<value optimized out>, count=0, execute_once=-72 '\270') at execMain.c:364
#16 0x00007f6e226aa6f8 in pgss_ExecutorRun (queryDesc=0x1c67760, direction=ForwardScanDirection, count=0, execute_once=1 '\001') at pg_stat_statements.c:889
#17 0x00007f6e224a474d in explain_ExecutorRun (queryDesc=0x1c67760, direction=ForwardScanDirection, count=0, execute_once=1 '\001') at auto_explain.c:267
#18 0x000000000072a15b in ProcessQuery (plan=<value optimized out>,
sourceText=0x1c21458 "INSERT INTO bi_dm.tdm_wh_shopgds_fnsh_rt (STATIS_DATE,SITE_CD,LGORT,ZSIZE,ZVTWEG,VSBED,TOTAL_CNT,FNSH_CNT,UNFNSH_CNT,ETL_TIME,DEPT_CD,TMALL_FLG,BUSS_TP,ZCKYWLX) VALUES($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$"..., params=0x1c21580, queryEnv=0x0, dest=<value optimized out>, completionTag=0x7fffc71c5de0 "") at pquery.c:161
#19 0x000000000072a395 in PortalRunMulti (portal=0x1c57f18, isTopLevel=1 '\001', setHoldSnapshot=0 '\000', dest=0xc9b480, altdest=0xc9b480, completionTag=0x7fffc71c5de0 "") at pquery.c:1286
#20 0x000000000072aa98 in PortalRun (portal=0x1c57f18, count=1, isTopLevel=1 '\001', run_once=1 '\001', dest=0x1c25768, altdest=0x1c25768, completionTag=0x7fffc71c5de0 "") at pquery.c:799
#21 0x0000000000728c9a in exec_execute_message (argc=<value optimized out>, argv=<value optimized out>, dbname=0x1bbb800 "lbiwhdb", username=<value optimized out>) at postgres.c:2007
#22 PostgresMain (argc=<value optimized out>, argv=<value optimized out>, dbname=0x1bbb800 "lbiwhdb", username=<value optimized out>) at postgres.c:4180
#23 0x00000000006bb43a in BackendRun (argc=<value optimized out>, argv=<value optimized out>) at postmaster.c:4405
---Type <return> to continue, or q <return> to quit---
#24 BackendStartup (argc=<value optimized out>, argv=<value optimized out>) at postmaster.c:4077
#25 ServerLoop (argc=<value optimized out>, argv=<value optimized out>) at postmaster.c:1755
#26 PostmasterMain (argc=<value optimized out>, argv=<value optimized out>) at postmaster.c:1363
#27 0x000000000063b4d0 in main (argc=3, argv=0x1b839e0) at main.c:228
(gdb) f 2
#2 0x0000000000718225 in LWLockAcquire (lock=0x7f6c517dbfa4, mode=LW_SHARED) at lwlock.c:1233
1233lwlock.c: No such file or directory.
in lwlock.c
(gdb) p num_held_lwlocks
$1 = 0
(gdb)
## stack of autovacuum:Acquire lock 0x7f6c519ba5a4 and hold 0x7f6c517dbfa4, 0x7f6c51684f64
--------------------------------------
(gdb) bt
#0 0x000000369ea0da00 in sem_wait () from /lib64/libpthread.so.0
#1 0x00000000006a7910 in PGSemaphoreLock (sema=0x7f6c4f77fdb8) at pg_sema.c:316
#2 0x0000000000718225 in LWLockAcquire (lock=0x7f6c519ba5a4, mode=LW_EXCLUSIVE) at lwlock.c:1233
#3 0x00000000004900fb in ginDeletePage (gvs=0x7fffc71c26d0, blkno=3709, isRoot=0 '\000', parent=<value optimized out>, myoff=2) at ginvacuum.c:154
#4 ginScanToDelete (gvs=0x7fffc71c26d0, blkno=3709, isRoot=0 '\000', parent=<value optimized out>, myoff=2) at ginvacuum.c:297
#5 0x000000000049004c in ginScanToDelete (gvs=0x7fffc71c26d0, blkno=6418, isRoot=1 '\001', parent=<value optimized out>, myoff=0) at ginvacuum.c:281
#6 0x0000000000490d1f in ginVacuumPostingTree (info=0x7fffc71c4da0, stats=<value optimized out>, callback=<value optimized out>, callback_state=<value optimized out>) at ginvacuum.c:408
#7 ginbulkdelete (info=0x7fffc71c4da0, stats=<value optimized out>, callback=<value optimized out>, callback_state=<value optimized out>) at ginvacuum.c:643
#8 0x00000000005e9d05 in lazy_vacuum_index (indrel=0x7f6c4f396fe8, stats=0x1c25ad8, vacrelstats=0x1c25578) at vacuumlazy.c:1621
#9 0x00000000005eaa29 in lazy_scan_heap (onerel=<value optimized out>, options=<value optimized out>, params=0x1ca7550, bstrategy=<value optimized out>) at vacuumlazy.c:1311
#10 lazy_vacuum_rel (onerel=<value optimized out>, options=<value optimized out>, params=0x1ca7550, bstrategy=<value optimized out>) at vacuumlazy.c:258
#11 0x00000000005e8d68 in vacuum_rel (relid=176178, relation=<value optimized out>, options=97, params=0x1ca7550) at vacuum.c:1445
#12 0x00000000005e9187 in vacuum (options=97, relation=0x7fffc71c5640, relid=<value optimized out>, params=0x1ca7550, va_cols=0x0, bstrategy=<value optimized out>, isTopLevel=1 '\001')
at vacuum.c:306
#13 0x00000000006ab262 in autovacuum_do_vac_analyze () at autovacuum.c:3135
#14 do_autovacuum () at autovacuum.c:2490
#15 0x00000000006aba70 in AutoVacWorkerMain (argc=<value optimized out>, argv=<value optimized out>) at autovacuum.c:1707
#16 0x00000000006abb46 in StartAutoVacWorker () at autovacuum.c:1504
#17 0x00000000006b972a in StartAutovacuumWorker (postgres_signal_arg=<value optimized out>) at postmaster.c:5462
#18 sigusr1_handler (postgres_signal_arg=<value optimized out>) at postmaster.c:5159
#19 <signal handler called>
#20 0x000000369e2e1393 in __select_nocancel () from /lib64/libc.so.6
#21 0x00000000006baa66 in ServerLoop (argc=<value optimized out>, argv=<value optimized out>) at postmaster.c:1719
#22 PostmasterMain (argc=<value optimized out>, argv=<value optimized out>) at postmaster.c:1363
#23 0x000000000063b4d0 in main (argc=3, argv=0x1b839e0) at main.c:228
(gdb) f 2
#2 0x0000000000718225 in LWLockAcquire (lock=0x7f6c519ba5a4, mode=LW_EXCLUSIVE) at lwlock.c:1233
1233lwlock.c: No such file or directory.
in lwlock.c
(gdb) p num_held_lwlocks
$1 = 2
(gdb) p InterruptHoldoffCount
$2 = 3
(gdb) p held_lwlocks
$3 = {{lock = 0x7f6c517dbfa4, mode = LW_EXCLUSIVE}, {lock = 0x7f6c51684f64, mode = LW_EXCLUSIVE}, {lock = 0x7f6c4f782a00, mode = LW_SHARED}, {lock = 0x7f6c4f78df80, mode = LW_EXCLUSIVE}, {
lock = 0x7f6c51a7a1e4, mode = LW_EXCLUSIVE}, {lock = 0x7f6c4f78df80, mode = LW_EXCLUSIVE}, {lock = 0x7f6c53a7eb64, mode = LW_EXCLUSIVE}, {lock = 0x7f6c4f78df80, mode = LW_EXCLUSIVE}, {
lock = 0x0, mode = LW_EXCLUSIVE} <repeats 192 times>}
(gdb)
(gdb) f 4
#4 ginScanToDelete (gvs=0x7fffc71c26d0, blkno=3709, isRoot=0 '\000', parent=<value optimized out>, myoff=2) at ginvacuum.c:297
297in ginvacuum.c
(gdb) p *me
$15 = {child = 0x0, parent = 0x7fffc71c26a0, blkno = 0, leftBlkno = 5649, isRoot = 0 '\000'}
(gdb) p *(&BufferDescriptors[(buffer - 1)].bufferdesc)
$3 = {tag = {rnode = {spcNode = 1663, dbNode = 16397, relNode = 692113}, forkNum = MAIN_FORKNUM, blockNum = 3709}, buf_id = 202271, state = {value = 3549691907}, wait_backend_pid = 0,
freeNext = -2, content_lock = {tranche = 54, state = {value = 1627389952}, waiters = {head = 539, tail = 1223}}}
(gdb) p &((&BufferDescriptors[(buffer - 1)].bufferdesc)->content_lock)
$5 = (LWLock *) 0x7f6c51684f64
(gdb) f 5
#5 0x000000000049004c in ginScanToDelete (gvs=0x7fffc71c26d0, blkno=6418, isRoot=1 '\001', parent=<value optimized out>, myoff=0) at ginvacuum.c:281
281in ginvacuum.c
(gdb) p *(&BufferDescriptors[(buffer - 1)].bufferdesc)
$6 = {tag = {rnode = {spcNode = 1663, dbNode = 16397, relNode = 692113}, forkNum = MAIN_FORKNUM, blockNum = 6418}, buf_id = 224224, state = {value = 3549692473}, wait_backend_pid = 14261,
freeNext = -2, content_lock = {tranche = 54, state = {value = 1627389952}, waiters = {head = 527, tail = 370}}}
(gdb) p &((&BufferDescriptors[(buffer - 1)].bufferdesc)->content_lock)
$7 = (LWLock *) 0x7f6c517dbfa4
According to above information in core file and source code, autovacuum process is trying to lock the block 5649,and be blocked.
6418(1. held LWLock:0x7f6c517dbfa4) root
|
5649(3. Acquire LWLock:0x7f6c519ba5a4) --> 3709(2.held LWLock:0x7f6c51684f64)
Does the locking order of autovacuum process(root->right->left) correct? While insert process lock gin buffer by order of bottom->top and left->right.
1. vacuum(root->right->left):
---------------------------------------------------------------------
static void
ginVacuumPostingTree(GinVacuumState *gvs, BlockNumber rootBlkno)
{
if (ginVacuumPostingTreeLeaves(gvs, rootBlkno))
{
...
LockBufferForCleanup(buffer);//1. (1*)lock the root
...
ginScanToDelete(gvs, rootBlkno, true, &root, InvalidOffsetNumber);//2. (2)scan to delete for root
}
}
static bool
ginScanToDelete(GinVacuumState *gvs, BlockNumber blkno, bool isRoot,
DataPageDeleteStack *parent, OffsetNumber myoff)
{
...
if (!isRoot)
LockBuffer(buffer, GIN_EXCLUSIVE);//2.1.1 (4)lock the first child(left);
//2.2.1 (7*)lock the second child(right);
...
if (!GinPageIsLeaf(page))
{
...
for (i = FirstOffsetNumber; i <= GinPageGetOpaque(page)->maxoff; i++)
{
PostingItem *pitem = GinDataPageGetPostingItem(page, i);
if (ginScanToDelete(gvs, PostingItemGetBlockNumber(pitem), FALSE, me, i))// 2.1 (3)scan to delete for the first child(left);
// 2.2 (6)scan to delete for the second child(right);
i--;
}
}
...
if (isempty)
{
/* we never delete the left- or rightmost branch */
if (me->leftBlkno != InvalidBlockNumber && !GinPageRightMost(page))
{
Assert(!isRoot);
ginDeletePage(gvs, blkno, me->leftBlkno, me->parent->blkno, myoff, me->parent->isRoot);//2.2.2 (8)delete the second child(right)
meDelete = TRUE;
}
}
if (!isRoot)
LockBuffer(buffer, GIN_UNLOCK);//2.1.2 (5)unlock the first child(left)
...
}
static void
ginDeletePage(GinVacuumState *gvs, BlockNumber deleteBlkno, BlockNumber leftBlkno,
BlockNumber parentBlkno, OffsetNumber myoff, bool isParentRoot)
{
...
lBuffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM, leftBlkno,
RBM_NORMAL, gvs->strategy);
...
LockBuffer(lBuffer, GIN_EXCLUSIVE);// 2.2.2.1 (9*)lock the first child(left);
...
}
---------------------------------------------------------------------
2. insert(bottom->top,left->right):
https://www.postgresql.org/message-id/19e4290b.3bb.166ea2c6831.Coremail.chjischj%40163.com
> # insert process(ginInsertValue())
>
>
>
> 644(root blkno)
> |
> 7054(2. held LWLock:0x2aaac587ae64) ----rightlink----> xxxx(3. Acquire LWLock:0x2aaab4009564，buffer = 2119038,blkno should be 9954)
> |
> 701(1. held LWLock:0x2aaab670dfe4)
>
>
> The ginInsertValue() function above gets the lwlock in the order described in the README.
>
>
> src/backend/access/gin/README
> ---------------------------------------------------------------
> To avoid deadlocks, B-tree pages must always be locked in the same order:
> left to right, and bottom to top.
> ...
> -----------------------------------------------------------------
Regards,
Chen Huajun
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7)
@ 2019-09-29 14:38 Alexander Korotkov <a.korotkov@postgrespro.ru>
parent: chenhj <chjischj@163.com>
0 siblings, 2 replies; 67+ messages in thread
From: Alexander Korotkov @ 2019-09-29 14:38 UTC (permalink / raw)
To: chenhj <chjischj@163.com>; +Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi!
Thank you for reporting.
On Sun, Sep 29, 2019 at 11:17 AM chenhj <chjischj@163.com> wrote:
> Does the locking order of autovacuum process(root->right->left) correct? While insert process lock gin buffer by order of bottom->top and left->right.
>
> 1. vacuum(root->right->left):
Starting from root seems OK for me, because vacuum blocks all
concurrent inserts before doing this. But this needs to be properly
documented in readme.
Locking from right to left is clearly wrong. It could deadlock with
concurrent ginStepRight(), which locks from left to right. I expect
this happened in your case. I'm going to reproduce this and fix.
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7)
@ 2019-09-29 15:12 Alexander Korotkov <a.korotkov@postgrespro.ru>
parent: Alexander Korotkov <a.korotkov@postgrespro.ru>
1 sibling, 2 replies; 67+ messages in thread
From: Alexander Korotkov @ 2019-09-29 15:12 UTC (permalink / raw)
To: chenhj <chjischj@163.com>; +Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Sun, Sep 29, 2019 at 5:38 PM Alexander Korotkov
<a.korotkov@postgrespro.ru> wrote:
> On Sun, Sep 29, 2019 at 11:17 AM chenhj <chjischj@163.com> wrote:
> > Does the locking order of autovacuum process(root->right->left) correct? While insert process lock gin buffer by order of bottom->top and left->right.
> >
> > 1. vacuum(root->right->left):
>
> Starting from root seems OK for me, because vacuum blocks all
> concurrent inserts before doing this. But this needs to be properly
> documented in readme.
>
> Locking from right to left is clearly wrong. It could deadlock with
> concurrent ginStepRight(), which locks from left to right. I expect
> this happened in your case. I'm going to reproduce this and fix.
I just managed to reproduce this using two sessions on master branch.
session 1
session 2
# create table test with (autovacuum_enabled = false) as (select
array[1] ar from generate_series(1,20000) i);
# create index test_ar_idx on test using gin (ar);
# vacuum analyze test;
# delete from test;
# set enable_seqscan = off;
gdb> b ginbtree.c:150
# select * from test where ar @> '{1}'::integer[];
Step in gdb just before ReadBuffer() in ReleaseAndReadBuffer().
gdb> b ginvacuum.c:155
# vacuum test;
gdb > continue
gdb> continue
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7)
@ 2019-09-29 16:27 Alexander Korotkov <a.korotkov@postgrespro.ru>
parent: Alexander Korotkov <a.korotkov@postgrespro.ru>
1 sibling, 0 replies; 67+ messages in thread
From: Alexander Korotkov @ 2019-09-29 16:27 UTC (permalink / raw)
To: chenhj <chjischj@163.com>; +Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Sun, Sep 29, 2019 at 6:12 PM Alexander Korotkov
<a.korotkov@postgrespro.ru> wrote:
> On Sun, Sep 29, 2019 at 5:38 PM Alexander Korotkov
> <a.korotkov@postgrespro.ru> wrote:
> > On Sun, Sep 29, 2019 at 11:17 AM chenhj <chjischj@163.com> wrote:
> > > Does the locking order of autovacuum process(root->right->left) correct? While insert process lock gin buffer by order of bottom->top and left->right.
> > >
> > > 1. vacuum(root->right->left):
> >
> > Starting from root seems OK for me, because vacuum blocks all
> > concurrent inserts before doing this. But this needs to be properly
> > documented in readme.
> >
> > Locking from right to left is clearly wrong. It could deadlock with
> > concurrent ginStepRight(), which locks from left to right. I expect
> > this happened in your case. I'm going to reproduce this and fix.
>
> I just managed to reproduce this using two sessions on master branch.
>
> session 1
> session 2
>
> # create table test with (autovacuum_enabled = false) as (select
> array[1] ar from generate_series(1,20000) i);
> # create index test_ar_idx on test using gin (ar);
> # vacuum analyze test;
> # delete from test;
>
> # set enable_seqscan = off;
> gdb> b ginbtree.c:150
> # select * from test where ar @> '{1}'::integer[];
> Step in gdb just before ReadBuffer() in ReleaseAndReadBuffer().
>
> gdb> b ginvacuum.c:155
> # vacuum test;
>
> gdb > continue
> gdb> continue
Patch with fix is attached. Idea is simple: ginScanToDelete() now
keeps exclusive lock on left page eliminating the need to relock it.
So, we preserve left-to-right locking order and can't deadlock with
ginStepRight().
Also, we need to adjust Concurrency section in GIN README. For me the
description looks vague and inconsistent even with current behavior.
I'm going to post this later.
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
Attachments:
gin_ginDeletePage_ginStepRight_deadlock_fix-1.patch (application/octet-stream, 4.7K) download
commit 56d380a08f1d5ba23fd676a9372c5ce409a5a208
Author: Alexander Korotkov <akorotkov@postgresql.org>
Date: Sun Sep 29 19:19:06 2019 +0300
Fix deadlock between ginDeletePage() and ginStepRight()
Reported-by:
Bug:
Discussion:
Author:
Reviewed-by:
Tested-by:
Backpatch-through:
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index dc46f2460e2..a1db48f211f 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -117,7 +117,8 @@ typedef struct DataPageDeleteStack
struct DataPageDeleteStack *parent;
BlockNumber blkno; /* current block number */
- BlockNumber leftBlkno; /* rightest non-deleted page on left */
+ Buffer leftBuffer; /* pinned and locked rightest
+ * non-deleted page on left */
bool isRoot;
} DataPageDeleteStack;
@@ -139,11 +140,8 @@ ginDeletePage(GinVacuumState *gvs, BlockNumber deleteBlkno, BlockNumber leftBlkn
/*
* This function MUST be called only if someone of parent pages hold
* exclusive cleanup lock. This guarantees that no insertions currently
- * happen in this subtree. Caller also acquire Exclusive lock on deletable
- * page and is acquiring and releasing exclusive lock on left page before.
- * Left page was locked and released. Then parent and this page are
- * locked. We acquire left page lock here only to mark page dirty after
- * changing right pointer.
+ * happen in this subtree. Caller also acquires Exclusive locks on deletable
+ * and left pages.
*/
lBuffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM, leftBlkno,
RBM_NORMAL, gvs->strategy);
@@ -152,8 +150,6 @@ ginDeletePage(GinVacuumState *gvs, BlockNumber deleteBlkno, BlockNumber leftBlkn
pBuffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM, parentBlkno,
RBM_NORMAL, gvs->strategy);
- LockBuffer(lBuffer, GIN_EXCLUSIVE);
-
page = BufferGetPage(dBuffer);
rightlink = GinPageGetOpaque(page)->rightlink;
@@ -227,7 +223,7 @@ ginDeletePage(GinVacuumState *gvs, BlockNumber deleteBlkno, BlockNumber leftBlkn
}
ReleaseBuffer(pBuffer);
- UnlockReleaseBuffer(lBuffer);
+ ReleaseBuffer(lBuffer);
ReleaseBuffer(dBuffer);
END_CRIT_SECTION();
@@ -237,7 +233,11 @@ ginDeletePage(GinVacuumState *gvs, BlockNumber deleteBlkno, BlockNumber leftBlkn
/*
- * scans posting tree and deletes empty pages
+ * Scans posting tree and deletes empty pages. Caller must lock root page for
+ * cleanup. During scan path from root to current page is kept exclusively
+ * locked. Also keep left page exclusively locked, because ginDeletePage()
+ * needs it. If we try to relock left page later, it could deadlock with
+ * ginStepRight().
*/
static bool
ginScanToDelete(GinVacuumState *gvs, BlockNumber blkno, bool isRoot,
@@ -260,7 +260,7 @@ ginScanToDelete(GinVacuumState *gvs, BlockNumber blkno, bool isRoot,
me = (DataPageDeleteStack *) palloc0(sizeof(DataPageDeleteStack));
me->parent = parent;
parent->child = me;
- me->leftBlkno = InvalidBlockNumber;
+ me->leftBuffer = InvalidBuffer;
}
else
me = parent->child;
@@ -288,6 +288,12 @@ ginScanToDelete(GinVacuumState *gvs, BlockNumber blkno, bool isRoot,
if (ginScanToDelete(gvs, PostingItemGetBlockNumber(pitem), false, me, i))
i--;
}
+
+ if (BufferIsValid(me->child->leftBuffer))
+ {
+ UnlockReleaseBuffer(me->child->leftBuffer);
+ me->child->leftBuffer = InvalidBuffer;
+ }
}
if (GinPageIsLeaf(page))
@@ -298,21 +304,31 @@ ginScanToDelete(GinVacuumState *gvs, BlockNumber blkno, bool isRoot,
if (isempty)
{
/* we never delete the left- or rightmost branch */
- if (me->leftBlkno != InvalidBlockNumber && !GinPageRightMost(page))
+ if (BufferIsValid(me->leftBuffer) && !GinPageRightMost(page))
{
Assert(!isRoot);
- ginDeletePage(gvs, blkno, me->leftBlkno, me->parent->blkno, myoff, me->parent->isRoot);
+ ginDeletePage(gvs, blkno, BufferGetBlockNumber(me->leftBuffer),
+ me->parent->blkno, myoff, me->parent->isRoot);
meDelete = true;
}
}
- if (!isRoot)
- LockBuffer(buffer, GIN_UNLOCK);
+ if (!meDelete)
+ {
+ if (BufferIsValid(me->leftBuffer))
+ UnlockReleaseBuffer(me->leftBuffer);
+ me->leftBuffer = buffer;
+ }
+ else
+ {
+ if (!isRoot)
+ LockBuffer(buffer, GIN_UNLOCK);
- ReleaseBuffer(buffer);
+ ReleaseBuffer(buffer);
+ }
- if (!meDelete)
- me->leftBlkno = blkno;
+ if (isRoot)
+ ReleaseBuffer(buffer);
return meDelete;
}
@@ -409,7 +425,7 @@ ginVacuumPostingTree(GinVacuumState *gvs, BlockNumber rootBlkno)
LockBufferForCleanup(buffer);
memset(&root, 0, sizeof(DataPageDeleteStack));
- root.leftBlkno = InvalidBlockNumber;
+ root.leftBuffer = InvalidBuffer;
root.isRoot = true;
ginScanToDelete(gvs, rootBlkno, true, &root, InvalidOffsetNumber);
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7)
@ 2019-09-29 19:52 Peter Geoghegan <pg@bowt.ie>
parent: Alexander Korotkov <a.korotkov@postgrespro.ru>
1 sibling, 2 replies; 67+ messages in thread
From: Peter Geoghegan @ 2019-09-29 19:52 UTC (permalink / raw)
To: Alexander Korotkov <a.korotkov@postgrespro.ru>; +Cc: chenhj <chjischj@163.com>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Sun, Sep 29, 2019 at 7:38 AM Alexander Korotkov
<a.korotkov@postgrespro.ru> wrote:
> Starting from root seems OK for me, because vacuum blocks all
> concurrent inserts before doing this. But this needs to be properly
> documented in readme.
I never got an adequate answer to this closely related question almost
two years ago:
https://www.postgresql.org/message-id/CAH2-Wz=GTnAPzEEZqYELOv3h1Fxpo5xhMrBP6aMGEKLKv95csQ@mail.gmail...
In general, ginInsertCleanup() seems badly designed. Why is it okay
that there is no nbtree-style distinction between page deletion and
page recycling?
> Locking from right to left is clearly wrong. It could deadlock with
> concurrent ginStepRight(), which locks from left to right. I expect
> this happened in your case. I'm going to reproduce this and fix.
I am sick and tired of seeing extremely basic errors like this within
GIN's locking protocols. Bugs happen, but these are not ordinary bugs.
They're more or less all a result of giving no thought to the high
level design. I'm not blaming you for this, or any one person. But
this is not okay.
Anything around index concurrency needs to be explained in
excruciating detail, while taking a top-down approach that applies
general rules (e.g. you can only do lock coupling left to right, or
bottom to top in nbtree). Anything less than that should be assumed to
be wrong on general principle.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7)
@ 2019-09-30 05:38 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Peter Geoghegan <pg@bowt.ie>
1 sibling, 2 replies; 67+ messages in thread
From: Andrey Borodin @ 2019-09-30 05:38 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Alexander Korotkov <a.korotkov@postgrespro.ru>; chenhj <chjischj@163.com>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
> 29 сент. 2019 г., в 21:27, Alexander Korotkov <a.korotkov@postgrespro.ru> написал(а):
>
> Patch with fix is attached. Idea is simple: ginScanToDelete() now
> keeps exclusive lock on left page eliminating the need to relock it.
> So, we preserve left-to-right locking order and can't deadlock with
> ginStepRight().
In this function ginDeletePage(gvs, blkno, BufferGetBlockNumber(me->leftBuffer),...)
we are going to reread buffer
lBuffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM, leftBlkno,
RBM_NORMAL, gvs->strategy);
Is it OK?
> 30 сент. 2019 г., в 0:52, Peter Geoghegan <pg@bowt.ie> написал(а):
>
> Why is it okay
> that there is no nbtree-style distinction between page deletion and
> page recycling?
As far as I understand deleted page is stamped with
GinPageSetDeleteXid(page, ReadNewTransactionId());
It will not be recycled until that Xid is far behind.
BTW we found a small bug (wraparound) in similar GiST and B-tree implementations.
Probably, it's there in GIN too.
--
Andrey Borodin
Open source RDBMS development team leader
Yandex.Cloud
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7)
@ 2019-09-30 17:37 Alexander Korotkov <a.korotkov@postgrespro.ru>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
1 sibling, 0 replies; 67+ messages in thread
From: Alexander Korotkov @ 2019-09-30 17:37 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Peter Geoghegan <pg@bowt.ie>; chenhj <chjischj@163.com>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Mon, Sep 30, 2019 at 8:38 AM Andrey Borodin <x4mmm@yandex-team.ru> wrote:
>
> > 29 сент. 2019 г., в 21:27, Alexander Korotkov <a.korotkov@postgrespro.ru> написал(а):
> >
> > Patch with fix is attached. Idea is simple: ginScanToDelete() now
> > keeps exclusive lock on left page eliminating the need to relock it.
> > So, we preserve left-to-right locking order and can't deadlock with
> > ginStepRight().
>
> In this function ginDeletePage(gvs, blkno, BufferGetBlockNumber(me->leftBuffer),...)
> we are going to reread buffer
> lBuffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM, leftBlkno,
> RBM_NORMAL, gvs->strategy);
> Is it OK?
That's related not only to left buffer. We also could pass buffer
instead of deleteBlkno. And with some code changes it's also possible
to pass buffer instead of parentBlkno. But I decided to keep code
changes minimal at least for backpatch version. We could apply such
optimization to master as a separate patch.
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7)
@ 2019-09-30 17:59 Alexander Korotkov <a.korotkov@postgrespro.ru>
parent: Peter Geoghegan <pg@bowt.ie>
1 sibling, 1 reply; 67+ messages in thread
From: Alexander Korotkov @ 2019-09-30 17:59 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: chenhj <chjischj@163.com>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Sun, Sep 29, 2019 at 10:53 PM Peter Geoghegan <pg@bowt.ie> wrote:
> On Sun, Sep 29, 2019 at 7:38 AM Alexander Korotkov
> <a.korotkov@postgrespro.ru> wrote:
> > Starting from root seems OK for me, because vacuum blocks all
> > concurrent inserts before doing this. But this needs to be properly
> > documented in readme.
>
> I never got an adequate answer to this closely related question almost
> two years ago:
>
> https://www.postgresql.org/message-id/CAH2-Wz=GTnAPzEEZqYELOv3h1Fxpo5xhMrBP6aMGEKLKv95csQ@mail.gmail...
>
> In general, ginInsertCleanup() seems badly designed. Why is it okay
> that there is no nbtree-style distinction between page deletion and
> page recycling?
Thank you for pointing. I remember this thread, but don't remember
details. I'll reread it.
> > Locking from right to left is clearly wrong. It could deadlock with
> > concurrent ginStepRight(), which locks from left to right. I expect
> > this happened in your case. I'm going to reproduce this and fix.
>
> I am sick and tired of seeing extremely basic errors like this within
> GIN's locking protocols. Bugs happen, but these are not ordinary bugs.
> They're more or less all a result of giving no thought to the high
> level design. I'm not blaming you for this, or any one person. But
> this is not okay.
>
> Anything around index concurrency needs to be explained in
> excruciating detail, while taking a top-down approach that applies
> general rules (e.g. you can only do lock coupling left to right, or
> bottom to top in nbtree). Anything less than that should be assumed to
> be wrong on general principle.
Frankly speaking I'm not very happy with special version of B-tree,
which is builtin to GIN. This version of B-tree is lacking of high
keys. AFAIR because of lack of high keys, we can't implement the same
concurrency model as nbtree. Instead we have to do super-locking for
page deletion and so on. That looks ridiculous for me. I think in
future we should somehow reimplement GIN on top of nbtree.
But we have to provide some way less radical fixes for our stable
releases. In particular, I believe patch I've posted in this thread
makes situation better not worse. That is it fixes one bug without
introducing mode bugs. But I'm going to analyze more on this and
document GIN concurrency better in the README. Probably, I'll spot
more details.
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7)
@ 2019-09-30 19:07 Peter Geoghegan <pg@bowt.ie>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
1 sibling, 0 replies; 67+ messages in thread
From: Peter Geoghegan @ 2019-09-30 19:07 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Alexander Korotkov <a.korotkov@postgrespro.ru>; chenhj <chjischj@163.com>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Sun, Sep 29, 2019 at 10:38 PM Andrey Borodin <x4mmm@yandex-team.ru> wrote:
> As far as I understand deleted page is stamped with
> GinPageSetDeleteXid(page, ReadNewTransactionId());
> It will not be recycled until that Xid is far behind.
That only gets used within posting tree pages, though.
ginInsertCleanup() is concerned with pending list pages.
> BTW we found a small bug (wraparound) in similar GiST and B-tree implementations.
> Probably, it's there in GIN too.
Probably, but that's much less of a problem to me.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7)
@ 2019-09-30 19:25 Peter Geoghegan <pg@bowt.ie>
parent: Alexander Korotkov <a.korotkov@postgrespro.ru>
0 siblings, 0 replies; 67+ messages in thread
From: Peter Geoghegan @ 2019-09-30 19:25 UTC (permalink / raw)
To: Alexander Korotkov <a.korotkov@postgrespro.ru>; +Cc: chenhj <chjischj@163.com>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Mon, Sep 30, 2019 at 11:00 AM Alexander Korotkov
<a.korotkov@postgrespro.ru> wrote:
> Thank you for pointing. I remember this thread, but don't remember
> details. I'll reread it.
I think that doing this now would be a good idea:
"""
Defensively checking the page type (pending, posting, etc) within
scanPendingInsert() would be a good start. That's already something
that we do for posting trees. If we're following the same rules as
posting trees (which sounds like a good idea), then we should have the
same defenses. Same applies to using elogs within ginInsertCleanup()
-- we should promote those Assert()s to elog()s.
"""
In other words, ginInsertCleanup() should have defensive "Page types
match?" checks that are similar to the existing checks in
ginStepRight(). In both cases we're stepping right while coupling
locks, to avoid concurrent page deletion.
> > > Locking from right to left is clearly wrong. It could deadlock with
> > > concurrent ginStepRight(), which locks from left to right. I expect
> > > this happened in your case. I'm going to reproduce this and fix.
> Frankly speaking I'm not very happy with special version of B-tree,
> which is builtin to GIN. This version of B-tree is lacking of high
> keys. AFAIR because of lack of high keys, we can't implement the same
> concurrency model as nbtree. Instead we have to do super-locking for
> page deletion and so on. That looks ridiculous for me. I think in
> future we should somehow reimplement GIN on top of nbtree.
I think that that could work on top of the new nbtree posting list
stuff, provided that it was acceptable to not use posting list
compression in the main tree -- the way that posting list splits work
there needs to be able to think about free space in a very simple way
that is broken even by GIN's varbyte compression. Compression could
still be used in posting trees, though.
> But we have to provide some way less radical fixes for our stable
> releases. In particular, I believe patch I've posted in this thread
> makes situation better not worse. That is it fixes one bug without
> introducing mode bugs. But I'm going to analyze more on this and
> document GIN concurrency better in the README. Probably, I'll spot
> more details.
Thanks.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7)
@ 2019-09-30 19:54 Peter Geoghegan <pg@bowt.ie>
parent: Alexander Korotkov <a.korotkov@postgrespro.ru>
1 sibling, 1 reply; 67+ messages in thread
From: Peter Geoghegan @ 2019-09-30 19:54 UTC (permalink / raw)
To: Alexander Korotkov <a.korotkov@postgrespro.ru>; +Cc: chenhj <chjischj@163.com>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Sun, Sep 29, 2019 at 8:12 AM Alexander Korotkov
<a.korotkov@postgrespro.ru> wrote:
> I just managed to reproduce this using two sessions on master branch.
>
> session 1
> session 2
Was the involvement of the pending list stuff in Chen's example just a
coincidence? Can you recreate the problem while eliminating that
factor (i.e. while setting fastupdate to off)?
Chen's example involved an INSERT that deadlocked against VACUUM --
not a SELECT. Is this just a coincidence?
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7)
@ 2019-10-01 02:55 Alexander Korotkov <a.korotkov@postgrespro.ru>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 67+ messages in thread
From: Alexander Korotkov @ 2019-10-01 02:55 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: chenhj <chjischj@163.com>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Mon, Sep 30, 2019 at 10:54 PM Peter Geoghegan <pg@bowt.ie> wrote:
>
> On Sun, Sep 29, 2019 at 8:12 AM Alexander Korotkov
> <a.korotkov@postgrespro.ru> wrote:
> > I just managed to reproduce this using two sessions on master branch.
> >
> > session 1
> > session 2
>
> Was the involvement of the pending list stuff in Chen's example just a
> coincidence? Can you recreate the problem while eliminating that
> factor (i.e. while setting fastupdate to off)?
>
> Chen's example involved an INSERT that deadlocked against VACUUM --
> not a SELECT. Is this just a coincidence?
Chen wrote.
> Unfortunately the insert process(run by gcore) held no lwlock, it should be another process(we did not fetch core file) that hold the lwlock needed for autovacuum process.
So, he catched backtrace for INSERT and post it for information. But
since INSERT has no lwlocks held, it couldn't participate deadlock.
It was just side waiter.
I've rerun my reproduction case and it still deadlocks. Just the same
steps but GIN index with (fastupdate = off).
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7)
@ 2019-10-03 21:05 Alexander Korotkov <a.korotkov@postgrespro.ru>
parent: Alexander Korotkov <a.korotkov@postgrespro.ru>
0 siblings, 1 reply; 67+ messages in thread
From: Alexander Korotkov @ 2019-10-03 21:05 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: chenhj <chjischj@163.com>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Tue, Oct 1, 2019 at 5:55 AM Alexander Korotkov
<a.korotkov@postgrespro.ru> wrote:
> On Mon, Sep 30, 2019 at 10:54 PM Peter Geoghegan <pg@bowt.ie> wrote:
> >
> > On Sun, Sep 29, 2019 at 8:12 AM Alexander Korotkov
> > <a.korotkov@postgrespro.ru> wrote:
> > > I just managed to reproduce this using two sessions on master branch.
> > >
> > > session 1
> > > session 2
> >
> > Was the involvement of the pending list stuff in Chen's example just a
> > coincidence? Can you recreate the problem while eliminating that
> > factor (i.e. while setting fastupdate to off)?
> >
> > Chen's example involved an INSERT that deadlocked against VACUUM --
> > not a SELECT. Is this just a coincidence?
>
> Chen wrote.
>
> > Unfortunately the insert process(run by gcore) held no lwlock, it should be another process(we did not fetch core file) that hold the lwlock needed for autovacuum process.
>
> So, he catched backtrace for INSERT and post it for information. But
> since INSERT has no lwlocks held, it couldn't participate deadlock.
> It was just side waiter.
>
> I've rerun my reproduction case and it still deadlocks. Just the same
> steps but GIN index with (fastupdate = off).
BTW, while trying to revise README I found another bug. It appears to
be possible to reach deleted page from downlink. The reproduction
case is following.
session 1
session 2
# create table tmp (ar int[]) with (autovacuum_enabled = false);
# insert into tmp (select '{1}' from generate_series(1,10000000) i);
# insert into tmp values ('{1,2}');
# insert into tmp (select '{1}' from generate_series(1,10000000) i);
# create index tmp_idx on tmp using gin(ar);
# delete from tmp;
# set max_parallel_workers_per_gather = 0;
/* Breakpoint where entyLoadMoreItems() calls ginFindLeafPage() to
search GIN posting tree */
gdb> b ginget.c:682
gdb> select * from tmp where ar @> '{1,2}';
gdb> /* step till ReleaseAndReadBuffer() releases a buffer */
# vacuum tmp;
# continue
It also appears that previous version of deadlock fix didn't supply
left sibling to leftmost child of any page. As result, internal pages
were never deleted. The first attached patch is revised fix is
attached.
The second patch fix traversing to deleted page using downlink.
Similarly to nbtree, we just always move right if landed on deleted
page. Also, it appears that we clear all other flags while marking
page as deleted. That cause assert to fire. With patch, we just add
deleted flag without erasing others. Also, I have to remove assert
that ginStepRight() never steps to deleted page. If we landed to
deleted page from downlink, then we can find other deleted page by
rightlink.
I'm planning to continue work on README, comments and commit messages.
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
Attachments: download all (2 files, .tar.gz)
0001-gin_ginDeletePage_ginStepRight_deadlock_fix-2.patch (application/octet-stream, 10.5K) download
commit 7dd844da20395992e5fd9c1c2e48582d0a2864d1
Author: Alexander Korotkov <akorotkov@postgresql.org>
Date: Sun Sep 29 19:19:06 2019 +0300
Fix deadlock between ginDeletePage() and ginStepRight()
Reported-by:
Bug:
Discussion:
Author:
Reviewed-by:
Tested-by:
Backpatch-through:
diff --git a/src/backend/access/gin/README b/src/backend/access/gin/README
index 76c12ae2f6b..e4152500cc6 100644
--- a/src/backend/access/gin/README
+++ b/src/backend/access/gin/README
@@ -277,50 +277,63 @@ followed by the packed items.
Concurrency
-----------
-The entry tree and each posting tree is a B-tree, with right-links connecting
+The entry tree and each posting tree are B-trees, with right-links connecting
sibling pages at the same level. This is the same structure that is used in
the regular B-tree indexam (invented by Lehman & Yao), but we don't support
scanning a GIN trees backwards, so we don't need left-links.
To avoid deadlocks, B-tree pages must always be locked in the same order:
-left to right, and bottom to top. When searching, the tree is traversed from
-top to bottom, so the lock on the parent page must be released before
+left to right, and bottom to top. The exception is page deletion during vacuum,
+which would be considered separately. When searching, the tree is traversed
+from top to bottom, so the lock on the parent page must be released before
descending to the next level. Concurrent page splits move the keyspace to
right, so after following a downlink, the page actually containing the key
we're looking for might be somewhere to the right of the page we landed on.
In that case, we follow the right-links until we find the page we're looking
-for.
-
-To delete a page, the page's left sibling, the target page, and its parent,
-are locked in that order, and the page is marked as deleted. However, a
-concurrent search might already have read a pointer to the page, and might be
-just about to follow it. A page can be reached via the right-link of its left
-sibling, or via its downlink in the parent.
+for. In spite of searches, insertions keeps pins on all pages in path from
+root to the current page. These pages potentially requies downlink insertion,
+while pins prevent them from being deleted.
+
+Vacuum never deletes tuples or pages from the entry tree. It traverses entry
+tree leafs in logical order by rightlinks and removes deletable TIDs from
+posting lists. Posting trees are processed by links from entry tree leafs. They
+are vacuumed in two stages. At first stage, deletable TIDs are removed from
+leafs. If first stage detects at least one empty page, then at the second stage
+ginScanToDelete() deletes empty pages.
+
+ginScanToDelete() scans posting tree to delete empty pages, while vacuum holds
+cleanup lock on the posting tree root. This lock prevent concurrent inserts,
+because inserters hold a pin on the root page. In spite of inserters searches
+don't hold pin on root page. So, while new searches cannot begin while root page
+is locked, any already-in-progress scans can continue concurrently with vacuum.
+
+ginScanToDelete() does depth-first tree scanning while keeping each page in path
+from root to current page exclusively locked. It also keeps left sibling of
+each page in the path locked. Thus, if current page is to be removed, all
+required pages to remove both downlink and rightlink are already locked.
+Therefore, page deletion locks pages top to bottom and left to right breaking
+our general rule. But assuming there is no concurrent insertions, this can't
+cause a deadlock.
+
+During replay od page deletion at standby, the page's left sibling, the target
+page, and its parent, are locked in that order. So it follows bottom to top and
+left to right rule.
+
+A search concurrent to page deletion might already have read a pointer to the
+page to be deleted, and might be just about to follow it. A page can be reached
+via the right-link of its left sibling, or via its downlink in the parent.
To prevent a backend from reaching a deleted page via a right-link, when
-following a right-link the lock on the previous page is not released until
-the lock on next page has been acquired.
-
-The downlink is more tricky. A search descending the tree must release the
-lock on the parent page before locking the child, or it could deadlock with
-a concurrent split of the child page; a page split locks the parent, while
-already holding a lock on the child page. So, deleted page cannot be reclaimed
-immediately. Instead, we have to wait for every transaction, which might wait
-to reference this page, to finish. Corresponding processes must observe that
-the page is marked deleted and recover accordingly.
-
-The previous paragraph's reasoning only applies to searches, and only to
-posting trees. To protect from inserters following a downlink to a deleted
-page, vacuum simply locks out all concurrent insertions to the posting tree,
-by holding a super-exclusive lock on the posting tree root. Inserters hold a
-pin on the root page, but searches do not, so while new searches cannot begin
-while root page is locked, any already-in-progress scans can continue
-concurrently with vacuum. In the entry tree, we never delete pages.
-
-(This is quite different from the mechanism the btree indexam uses to make
-page-deletions safe; it stamps the deleted pages with an XID and keeps the
-deleted pages around with the right-link intact until all concurrent scans
-have finished.)
+following a right-link the lock on the previous page is not released until the
+lock on next page has been acquired.
+
+The downlink is more tricky. A search descending the tree must release the lock
+on the parent page before locking the child, or it could deadlock with a
+concurrent split of the child page; a page split locks the parent, while already
+holding a lock on the child page. So, deleted page cannot be reclaimed
+immediately. Instead, we have to wait for every transaction, which might wait to
+reference this page, to finish. Corresponding processes must observe that the
+page is marked deleted and recover accordingly.
Predicate Locking
-----------------
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index dc46f2460e2..72169bb0bdc 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -117,7 +117,8 @@ typedef struct DataPageDeleteStack
struct DataPageDeleteStack *parent;
BlockNumber blkno; /* current block number */
- BlockNumber leftBlkno; /* rightest non-deleted page on left */
+ Buffer leftBuffer; /* pinned and locked rightest non-deleted page
+ * on left */
bool isRoot;
} DataPageDeleteStack;
@@ -139,11 +140,8 @@ ginDeletePage(GinVacuumState *gvs, BlockNumber deleteBlkno, BlockNumber leftBlkn
/*
* This function MUST be called only if someone of parent pages hold
* exclusive cleanup lock. This guarantees that no insertions currently
- * happen in this subtree. Caller also acquire Exclusive lock on deletable
- * page and is acquiring and releasing exclusive lock on left page before.
- * Left page was locked and released. Then parent and this page are
- * locked. We acquire left page lock here only to mark page dirty after
- * changing right pointer.
+ * happen in this subtree. Caller also acquires Exclusive locks on
+ * deletable and left pages.
*/
lBuffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM, leftBlkno,
RBM_NORMAL, gvs->strategy);
@@ -152,8 +150,6 @@ ginDeletePage(GinVacuumState *gvs, BlockNumber deleteBlkno, BlockNumber leftBlkn
pBuffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM, parentBlkno,
RBM_NORMAL, gvs->strategy);
- LockBuffer(lBuffer, GIN_EXCLUSIVE);
-
page = BufferGetPage(dBuffer);
rightlink = GinPageGetOpaque(page)->rightlink;
@@ -227,7 +223,7 @@ ginDeletePage(GinVacuumState *gvs, BlockNumber deleteBlkno, BlockNumber leftBlkn
}
ReleaseBuffer(pBuffer);
- UnlockReleaseBuffer(lBuffer);
+ ReleaseBuffer(lBuffer);
ReleaseBuffer(dBuffer);
END_CRIT_SECTION();
@@ -237,7 +233,11 @@ ginDeletePage(GinVacuumState *gvs, BlockNumber deleteBlkno, BlockNumber leftBlkn
/*
- * scans posting tree and deletes empty pages
+ * Scans posting tree and deletes empty pages. Caller must lock root page for
+ * cleanup. During scan path from root to current page is kept exclusively
+ * locked. Also keep left page exclusively locked, because ginDeletePage()
+ * needs it. If we try to relock left page later, it could deadlock with
+ * ginStepRight().
*/
static bool
ginScanToDelete(GinVacuumState *gvs, BlockNumber blkno, bool isRoot,
@@ -260,7 +260,7 @@ ginScanToDelete(GinVacuumState *gvs, BlockNumber blkno, bool isRoot,
me = (DataPageDeleteStack *) palloc0(sizeof(DataPageDeleteStack));
me->parent = parent;
parent->child = me;
- me->leftBlkno = InvalidBlockNumber;
+ me->leftBuffer = InvalidBuffer;
}
else
me = parent->child;
@@ -288,6 +288,12 @@ ginScanToDelete(GinVacuumState *gvs, BlockNumber blkno, bool isRoot,
if (ginScanToDelete(gvs, PostingItemGetBlockNumber(pitem), false, me, i))
i--;
}
+
+ if (GinPageRightMost(page) && BufferIsValid(me->child->leftBuffer))
+ {
+ UnlockReleaseBuffer(me->child->leftBuffer);
+ me->child->leftBuffer = InvalidBuffer;
+ }
}
if (GinPageIsLeaf(page))
@@ -298,21 +304,31 @@ ginScanToDelete(GinVacuumState *gvs, BlockNumber blkno, bool isRoot,
if (isempty)
{
/* we never delete the left- or rightmost branch */
- if (me->leftBlkno != InvalidBlockNumber && !GinPageRightMost(page))
+ if (BufferIsValid(me->leftBuffer) && !GinPageRightMost(page))
{
Assert(!isRoot);
- ginDeletePage(gvs, blkno, me->leftBlkno, me->parent->blkno, myoff, me->parent->isRoot);
+ ginDeletePage(gvs, blkno, BufferGetBlockNumber(me->leftBuffer),
+ me->parent->blkno, myoff, me->parent->isRoot);
meDelete = true;
}
}
- if (!isRoot)
- LockBuffer(buffer, GIN_UNLOCK);
+ if (!meDelete)
+ {
+ if (BufferIsValid(me->leftBuffer))
+ UnlockReleaseBuffer(me->leftBuffer);
+ me->leftBuffer = buffer;
+ }
+ else
+ {
+ if (!isRoot)
+ LockBuffer(buffer, GIN_UNLOCK);
- ReleaseBuffer(buffer);
+ ReleaseBuffer(buffer);
+ }
- if (!meDelete)
- me->leftBlkno = blkno;
+ if (isRoot)
+ ReleaseBuffer(buffer);
return meDelete;
}
@@ -409,7 +425,7 @@ ginVacuumPostingTree(GinVacuumState *gvs, BlockNumber rootBlkno)
LockBufferForCleanup(buffer);
memset(&root, 0, sizeof(DataPageDeleteStack));
- root.leftBlkno = InvalidBlockNumber;
+ root.leftBuffer = InvalidBuffer;
root.isRoot = true;
ginScanToDelete(gvs, rootBlkno, true, &root, InvalidOffsetNumber);
0002-gin-fix-traversing-to-deleted-page-by-downlink-2.patch (application/octet-stream, 2.4K) download
commit 7709e22db542c7a453b766180003234a4931f8da
Author: Alexander Korotkov <akorotkov@postgresql.org>
Date: Thu Oct 3 23:49:16 2019 +0300
Fix traversing to the deleted page from downlink
Reported-by:
Bug:
Discussion:
Author:
Reviewed-by:
Tested-by:
Backpatch-through:
diff --git a/src/backend/access/gin/ginbtree.c b/src/backend/access/gin/ginbtree.c
index 4c29261256a..b18056cf894 100644
--- a/src/backend/access/gin/ginbtree.c
+++ b/src/backend/access/gin/ginbtree.c
@@ -187,13 +187,6 @@ ginStepRight(Buffer buffer, Relation index, int lockmode)
if (isLeaf != GinPageIsLeaf(page) || isData != GinPageIsData(page))
elog(ERROR, "right sibling of GIN page is of different type");
- /*
- * Given the proper lock sequence above, we should never land on a deleted
- * page.
- */
- if (GinPageIsDeleted(page))
- elog(ERROR, "right sibling of GIN page was deleted");
-
return nextbuffer;
}
diff --git a/src/backend/access/gin/gindatapage.c b/src/backend/access/gin/gindatapage.c
index e8c34d6b1f6..e497210b4c0 100644
--- a/src/backend/access/gin/gindatapage.c
+++ b/src/backend/access/gin/gindatapage.c
@@ -238,6 +238,9 @@ dataIsMoveRight(GinBtree btree, Page page)
if (GinPageRightMost(page))
return false;
+ if (GinPageIsDeleted(page))
+ return true;
+
return (ginCompareItemPointers(&btree->itemptr, iptr) > 0) ? true : false;
}
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index 72169bb0bdc..11e050171ad 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -186,7 +186,7 @@ ginDeletePage(GinVacuumState *gvs, BlockNumber deleteBlkno, BlockNumber leftBlkn
* we shouldn't change rightlink field to save workability of running
* search scan
*/
- GinPageGetOpaque(page)->flags = GIN_DELETED;
+ GinPageSetDeleted(page);
MarkBufferDirty(pBuffer);
MarkBufferDirty(lBuffer);
diff --git a/src/backend/access/gin/ginxlog.c b/src/backend/access/gin/ginxlog.c
index c945b282721..387102f21a1 100644
--- a/src/backend/access/gin/ginxlog.c
+++ b/src/backend/access/gin/ginxlog.c
@@ -500,7 +500,7 @@ ginRedoDeletePage(XLogReaderState *record)
{
page = BufferGetPage(dbuffer);
Assert(GinPageIsData(page));
- GinPageGetOpaque(page)->flags = GIN_DELETED;
+ GinPageSetDeleted(page);
GinPageSetDeleteXid(page, data->deleteXid);
PageSetLSN(page, lsn);
MarkBufferDirty(dbuffer);
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7)
@ 2019-10-27 19:03 Alexander Korotkov <a.korotkov@postgrespro.ru>
parent: Alexander Korotkov <a.korotkov@postgrespro.ru>
0 siblings, 2 replies; 67+ messages in thread
From: Alexander Korotkov @ 2019-10-27 19:03 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: chenhj <chjischj@163.com>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi, Peter!
On Fri, Oct 4, 2019 at 12:05 AM Alexander Korotkov
<a.korotkov@postgrespro.ru> wrote:
> I'm planning to continue work on README, comments and commit messages.
It tool me a lot of efforts to revise a concurrency section in README.
I can't judge the result, but I probably did my best. I'd like to
commit this patchset including both bug fixes and README update. But
I'd like you to take a look on the README patch first.
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
Attachments: download all (3 files, .tar.gz)
0001-Fix-deadlock-between-ginDeletePage-and-ginStepRigh-3.patch (application/octet-stream, 11.3K) download
From e9db2bafe15d0c736f7b5652ee91a0d9104eef38 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <akorotkov@postgresql.org>
Date: Sun, 29 Sep 2019 19:19:06 +0300
Subject: [PATCH 1/3] Fix deadlock between ginDeletePage() and ginStepRight()
When ginDeletePage() is about to delete page it locks its left sibling to revise
the rightlink. So, it locks pages in right to left manner. Int he same time
ginStepRight() locks pages in left to right manner, and that could cause a
deadlock.
This commit makes ginScanToDelete() keep exclusive lock on left siblings of
currently investigated path. That elimites need to relock left sibling in
ginDeletePage(). Thus, deadlock with ginStepRight() can't happen anymore.
Reported-by: Chen Huajun
Discussion: https://postgr.es/m/5c332bd1.87b6.16d7c17aa98.Coremail.chjischj%40163.com
Author: Alexander Korotkov
Reviewed-by: Peter Geoghegan
Backpatch-through: 9.4
---
src/backend/access/gin/README | 79 ++++++++++++++++++++++----------------
src/backend/access/gin/ginvacuum.c | 54 +++++++++++++++++---------
2 files changed, 81 insertions(+), 52 deletions(-)
diff --git a/src/backend/access/gin/README b/src/backend/access/gin/README
index 76c12ae2f6b..e4152500cc6 100644
--- a/src/backend/access/gin/README
+++ b/src/backend/access/gin/README
@@ -277,50 +277,63 @@ followed by the packed items.
Concurrency
-----------
-The entry tree and each posting tree is a B-tree, with right-links connecting
+The entry tree and each posting tree are B-trees, with right-links connecting
sibling pages at the same level. This is the same structure that is used in
the regular B-tree indexam (invented by Lehman & Yao), but we don't support
scanning a GIN trees backwards, so we don't need left-links.
To avoid deadlocks, B-tree pages must always be locked in the same order:
-left to right, and bottom to top. When searching, the tree is traversed from
-top to bottom, so the lock on the parent page must be released before
+left to right, and bottom to top. The exception is page deletion during vacuum,
+which would be considered separately. When searching, the tree is traversed
+from top to bottom, so the lock on the parent page must be released before
descending to the next level. Concurrent page splits move the keyspace to
right, so after following a downlink, the page actually containing the key
we're looking for might be somewhere to the right of the page we landed on.
In that case, we follow the right-links until we find the page we're looking
-for.
-
-To delete a page, the page's left sibling, the target page, and its parent,
-are locked in that order, and the page is marked as deleted. However, a
-concurrent search might already have read a pointer to the page, and might be
-just about to follow it. A page can be reached via the right-link of its left
-sibling, or via its downlink in the parent.
+for. In spite of searches, insertions keeps pins on all pages in path from
+root to the current page. These pages potentially requies downlink insertion,
+while pins prevent them from being deleted.
+
+Vacuum never deletes tuples or pages from the entry tree. It traverses entry
+tree leafs in logical order by rightlinks and removes deletable TIDs from
+posting lists. Posting trees are processed by links from entry tree leafs. They
+are vacuumed in two stages. At first stage, deletable TIDs are removed from
+leafs. If first stage detects at least one empty page, then at the second stage
+ginScanToDelete() deletes empty pages.
+
+ginScanToDelete() scans posting tree to delete empty pages, while vacuum holds
+cleanup lock on the posting tree root. This lock prevent concurrent inserts,
+because inserters hold a pin on the root page. In spite of inserters searches
+don't hold pin on root page. So, while new searches cannot begin while root page
+is locked, any already-in-progress scans can continue concurrently with vacuum.
+
+ginScanToDelete() does depth-first tree scanning while keeping each page in path
+from root to current page exclusively locked. It also keeps left sibling of
+each page in the path locked. Thus, if current page is to be removed, all
+required pages to remove both downlink and rightlink are already locked.
+Therefore, page deletion locks pages top to bottom and left to right breaking
+our general rule. But assuming there is no concurrent insertions, this can't
+cause a deadlock.
+
+During replay od page deletion at standby, the page's left sibling, the target
+page, and its parent, are locked in that order. So it follows bottom to top and
+left to right rule.
+
+A search concurrent to page deletion might already have read a pointer to the
+page to be deleted, and might be just about to follow it. A page can be reached
+via the right-link of its left sibling, or via its downlink in the parent.
To prevent a backend from reaching a deleted page via a right-link, when
-following a right-link the lock on the previous page is not released until
-the lock on next page has been acquired.
-
-The downlink is more tricky. A search descending the tree must release the
-lock on the parent page before locking the child, or it could deadlock with
-a concurrent split of the child page; a page split locks the parent, while
-already holding a lock on the child page. So, deleted page cannot be reclaimed
-immediately. Instead, we have to wait for every transaction, which might wait
-to reference this page, to finish. Corresponding processes must observe that
-the page is marked deleted and recover accordingly.
-
-The previous paragraph's reasoning only applies to searches, and only to
-posting trees. To protect from inserters following a downlink to a deleted
-page, vacuum simply locks out all concurrent insertions to the posting tree,
-by holding a super-exclusive lock on the posting tree root. Inserters hold a
-pin on the root page, but searches do not, so while new searches cannot begin
-while root page is locked, any already-in-progress scans can continue
-concurrently with vacuum. In the entry tree, we never delete pages.
-
-(This is quite different from the mechanism the btree indexam uses to make
-page-deletions safe; it stamps the deleted pages with an XID and keeps the
-deleted pages around with the right-link intact until all concurrent scans
-have finished.)
+following a right-link the lock on the previous page is not released until the
+lock on next page has been acquired.
+
+The downlink is more tricky. A search descending the tree must release the lock
+on the parent page before locking the child, or it could deadlock with a
+concurrent split of the child page; a page split locks the parent, while already
+holding a lock on the child page. So, deleted page cannot be reclaimed
+immediately. Instead, we have to wait for every transaction, which might wait to
+reference this page, to finish. Corresponding processes must observe that the
+page is marked deleted and recover accordingly.
Predicate Locking
-----------------
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index dc46f2460e2..72169bb0bdc 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -117,7 +117,8 @@ typedef struct DataPageDeleteStack
struct DataPageDeleteStack *parent;
BlockNumber blkno; /* current block number */
- BlockNumber leftBlkno; /* rightest non-deleted page on left */
+ Buffer leftBuffer; /* pinned and locked rightest non-deleted page
+ * on left */
bool isRoot;
} DataPageDeleteStack;
@@ -139,11 +140,8 @@ ginDeletePage(GinVacuumState *gvs, BlockNumber deleteBlkno, BlockNumber leftBlkn
/*
* This function MUST be called only if someone of parent pages hold
* exclusive cleanup lock. This guarantees that no insertions currently
- * happen in this subtree. Caller also acquire Exclusive lock on deletable
- * page and is acquiring and releasing exclusive lock on left page before.
- * Left page was locked and released. Then parent and this page are
- * locked. We acquire left page lock here only to mark page dirty after
- * changing right pointer.
+ * happen in this subtree. Caller also acquires Exclusive locks on
+ * deletable and left pages.
*/
lBuffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM, leftBlkno,
RBM_NORMAL, gvs->strategy);
@@ -152,8 +150,6 @@ ginDeletePage(GinVacuumState *gvs, BlockNumber deleteBlkno, BlockNumber leftBlkn
pBuffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM, parentBlkno,
RBM_NORMAL, gvs->strategy);
- LockBuffer(lBuffer, GIN_EXCLUSIVE);
-
page = BufferGetPage(dBuffer);
rightlink = GinPageGetOpaque(page)->rightlink;
@@ -227,7 +223,7 @@ ginDeletePage(GinVacuumState *gvs, BlockNumber deleteBlkno, BlockNumber leftBlkn
}
ReleaseBuffer(pBuffer);
- UnlockReleaseBuffer(lBuffer);
+ ReleaseBuffer(lBuffer);
ReleaseBuffer(dBuffer);
END_CRIT_SECTION();
@@ -237,7 +233,11 @@ ginDeletePage(GinVacuumState *gvs, BlockNumber deleteBlkno, BlockNumber leftBlkn
/*
- * scans posting tree and deletes empty pages
+ * Scans posting tree and deletes empty pages. Caller must lock root page for
+ * cleanup. During scan path from root to current page is kept exclusively
+ * locked. Also keep left page exclusively locked, because ginDeletePage()
+ * needs it. If we try to relock left page later, it could deadlock with
+ * ginStepRight().
*/
static bool
ginScanToDelete(GinVacuumState *gvs, BlockNumber blkno, bool isRoot,
@@ -260,7 +260,7 @@ ginScanToDelete(GinVacuumState *gvs, BlockNumber blkno, bool isRoot,
me = (DataPageDeleteStack *) palloc0(sizeof(DataPageDeleteStack));
me->parent = parent;
parent->child = me;
- me->leftBlkno = InvalidBlockNumber;
+ me->leftBuffer = InvalidBuffer;
}
else
me = parent->child;
@@ -288,6 +288,12 @@ ginScanToDelete(GinVacuumState *gvs, BlockNumber blkno, bool isRoot,
if (ginScanToDelete(gvs, PostingItemGetBlockNumber(pitem), false, me, i))
i--;
}
+
+ if (GinPageRightMost(page) && BufferIsValid(me->child->leftBuffer))
+ {
+ UnlockReleaseBuffer(me->child->leftBuffer);
+ me->child->leftBuffer = InvalidBuffer;
+ }
}
if (GinPageIsLeaf(page))
@@ -298,21 +304,31 @@ ginScanToDelete(GinVacuumState *gvs, BlockNumber blkno, bool isRoot,
if (isempty)
{
/* we never delete the left- or rightmost branch */
- if (me->leftBlkno != InvalidBlockNumber && !GinPageRightMost(page))
+ if (BufferIsValid(me->leftBuffer) && !GinPageRightMost(page))
{
Assert(!isRoot);
- ginDeletePage(gvs, blkno, me->leftBlkno, me->parent->blkno, myoff, me->parent->isRoot);
+ ginDeletePage(gvs, blkno, BufferGetBlockNumber(me->leftBuffer),
+ me->parent->blkno, myoff, me->parent->isRoot);
meDelete = true;
}
}
- if (!isRoot)
- LockBuffer(buffer, GIN_UNLOCK);
+ if (!meDelete)
+ {
+ if (BufferIsValid(me->leftBuffer))
+ UnlockReleaseBuffer(me->leftBuffer);
+ me->leftBuffer = buffer;
+ }
+ else
+ {
+ if (!isRoot)
+ LockBuffer(buffer, GIN_UNLOCK);
- ReleaseBuffer(buffer);
+ ReleaseBuffer(buffer);
+ }
- if (!meDelete)
- me->leftBlkno = blkno;
+ if (isRoot)
+ ReleaseBuffer(buffer);
return meDelete;
}
@@ -409,7 +425,7 @@ ginVacuumPostingTree(GinVacuumState *gvs, BlockNumber rootBlkno)
LockBufferForCleanup(buffer);
memset(&root, 0, sizeof(DataPageDeleteStack));
- root.leftBlkno = InvalidBlockNumber;
+ root.leftBuffer = InvalidBuffer;
root.isRoot = true;
ginScanToDelete(gvs, rootBlkno, true, &root, InvalidOffsetNumber);
--
2.14.3
0002-Fix-traversing-to-the-deleted-GIN-page-via-downlin-3.patch (application/octet-stream, 3.1K) download
From 7988aa1c18a475f7f7b7eaa0a54b07ab177ddb59 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <akorotkov@postgresql.org>
Date: Thu, 3 Oct 2019 23:49:16 +0300
Subject: [PATCH 2/3] Fix traversing to the deleted GIN page via downlink
Current GIN code appears to don't handle traversing to the deleted page via
downlink. This commit fixes that by stepping right from the delete page like
we do in nbtree.
This commit also fixes setting 'deleted' flag to the GIN pages. Now other page
flags are not erased once page is deleted. That helps to keep our assertions
true if we arrive deleted page via downlink.
Discussion: https://postgr.es/m/CAPpHfdvMvsw-NcE5bRS7R1BbvA4BxoDnVVjkXC5W0Czvy9LVrg%40mail.gmail.com
Author: Alexander Korotkov
Backpatch-through: 9.4
---
src/backend/access/gin/ginbtree.c | 7 -------
src/backend/access/gin/gindatapage.c | 3 +++
src/backend/access/gin/ginvacuum.c | 2 +-
src/backend/access/gin/ginxlog.c | 2 +-
4 files changed, 5 insertions(+), 9 deletions(-)
diff --git a/src/backend/access/gin/ginbtree.c b/src/backend/access/gin/ginbtree.c
index 4c29261256a..b18056cf894 100644
--- a/src/backend/access/gin/ginbtree.c
+++ b/src/backend/access/gin/ginbtree.c
@@ -187,13 +187,6 @@ ginStepRight(Buffer buffer, Relation index, int lockmode)
if (isLeaf != GinPageIsLeaf(page) || isData != GinPageIsData(page))
elog(ERROR, "right sibling of GIN page is of different type");
- /*
- * Given the proper lock sequence above, we should never land on a deleted
- * page.
- */
- if (GinPageIsDeleted(page))
- elog(ERROR, "right sibling of GIN page was deleted");
-
return nextbuffer;
}
diff --git a/src/backend/access/gin/gindatapage.c b/src/backend/access/gin/gindatapage.c
index e8c34d6b1f6..e497210b4c0 100644
--- a/src/backend/access/gin/gindatapage.c
+++ b/src/backend/access/gin/gindatapage.c
@@ -238,6 +238,9 @@ dataIsMoveRight(GinBtree btree, Page page)
if (GinPageRightMost(page))
return false;
+ if (GinPageIsDeleted(page))
+ return true;
+
return (ginCompareItemPointers(&btree->itemptr, iptr) > 0) ? true : false;
}
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index 72169bb0bdc..11e050171ad 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -186,7 +186,7 @@ ginDeletePage(GinVacuumState *gvs, BlockNumber deleteBlkno, BlockNumber leftBlkn
* we shouldn't change rightlink field to save workability of running
* search scan
*/
- GinPageGetOpaque(page)->flags = GIN_DELETED;
+ GinPageSetDeleted(page);
MarkBufferDirty(pBuffer);
MarkBufferDirty(lBuffer);
diff --git a/src/backend/access/gin/ginxlog.c b/src/backend/access/gin/ginxlog.c
index c945b282721..387102f21a1 100644
--- a/src/backend/access/gin/ginxlog.c
+++ b/src/backend/access/gin/ginxlog.c
@@ -500,7 +500,7 @@ ginRedoDeletePage(XLogReaderState *record)
{
page = BufferGetPage(dbuffer);
Assert(GinPageIsData(page));
- GinPageGetOpaque(page)->flags = GIN_DELETED;
+ GinPageSetDeleted(page);
GinPageSetDeleteXid(page, data->deleteXid);
PageSetLSN(page, lsn);
MarkBufferDirty(dbuffer);
--
2.14.3
0003-Revise-concurrency-section-in-GIN-README-3.patch (application/octet-stream, 10.6K) download
From d3ad98a5b793c331f298d7da2a97055cb0ee4863 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <akorotkov@postgresql.org>
Date: Mon, 21 Oct 2019 23:01:21 +0300
Subject: [PATCH 3/3] Revise concurrency section in GIN README
We find GIN concurrency bugs from time to time. One of the problems here is
that concurrency of GIN isn't well-documented in README. So, it might be even
hard to distinguish design bugs from implementation bugs.
This commit revised concurrency section in GIN README providing more details.
Some examples are illustrated in ASCII art.
Discussion: https://postgr.es/m/5c332bd1.87b6.16d7c17aa98.Coremail.chjischj%40163.com
Author: Alexander Korotkov
Backpatch-through: 9.4
---
src/backend/access/gin/README | 185 +++++++++++++++++++++++++++++++++---------
1 file changed, 147 insertions(+), 38 deletions(-)
diff --git a/src/backend/access/gin/README b/src/backend/access/gin/README
index e4152500cc6..2d5c167d223 100644
--- a/src/backend/access/gin/README
+++ b/src/backend/access/gin/README
@@ -278,21 +278,86 @@ Concurrency
-----------
The entry tree and each posting tree are B-trees, with right-links connecting
-sibling pages at the same level. This is the same structure that is used in
+sibling pages at the same level. This is the same structure that is used in
the regular B-tree indexam (invented by Lehman & Yao), but we don't support
-scanning a GIN trees backwards, so we don't need left-links.
-
-To avoid deadlocks, B-tree pages must always be locked in the same order:
-left to right, and bottom to top. The exception is page deletion during vacuum,
-which would be considered separately. When searching, the tree is traversed
-from top to bottom, so the lock on the parent page must be released before
-descending to the next level. Concurrent page splits move the keyspace to
-right, so after following a downlink, the page actually containing the key
-we're looking for might be somewhere to the right of the page we landed on.
-In that case, we follow the right-links until we find the page we're looking
-for. In spite of searches, insertions keeps pins on all pages in path from
-root to the current page. These pages potentially requies downlink insertion,
-while pins prevent them from being deleted.
+scanning a GIN trees backwards, so we don't need left-links. The entry tree
+leaves don't have dedicated high keys, instead greatest leaf tuple serves as
+high key. That works because tuples are never deleted from the entry tree.
+
+The algorithms used to operate entry and posting trees are considered below.
+
+### Locating the leaf page
+
+When we search for leaf page in GIN btree to perform a read, we descend from
+the root page to the leaf through using downlinks taking pin and shared lock on
+one page at once. So, we release pin and shared lock on previous page before
+getting them on the next page.
+
+The picture below shows tree state after finding the leaf page. Lower case
+letters depicts tree pages. 'S' depicts shared lock on the page.
+
+ a
+ / | \
+ b c d
+ / | \ | \ | \
+ eS f g h i j k
+
+### Steping right
+
+Concurrent page splits move the keyspace to right, so after following a
+downlink, the page actually containing the key we're looking for might be
+somewhere to the right of the page we landed on. In that case, we follow the
+right-links until we find the page we're looking for.
+
+During stepping right we take pin and shared lock on the right sibling before
+releasing them from the current page. This mechanism was designed to protect
+from stepping to delete page. We step to the right sibling while hold lock on
+the rightlink pointing there. So, it's guaranteed that nobody updates rightlink
+concurrently and doesn't delete right sibling accordingly.
+
+The picture below shows two pages locked at once during stepping right.
+
+ a
+ / | \
+ b c d
+ / | \ | \ | \
+ eS fS g h i j k
+
+### Insert
+
+While finding appropriate leaf for insertion we also descend from the root to
+leaf, while shared locking one page at once in. But during insertion we don't
+release pins from root and internal pages. That could save us some lookups to
+the buffers hash table for downlinks insertion assuming parents are not changed
+due to concurrent splits. Once we reach leaf we re-lock the page in exclusive
+mode.
+
+The picture below shows leaf page locked in exclusive mode and ready for
+insertion. 'P' and 'E' depict pin and exclusive lock correspondingly.
+
+
+ aP
+ / | \
+ b cP d
+ / | \ | \ | \
+ e f g hE i j k
+
+
+If insert causes a page split, the parent is locked in exclusive mode before
+unlocking the left child. So, insertion algorithm can exclusively lock both
+parent and child pages at once starting from child.
+
+The picture below shows tree state after leaf page split. 'q' is new page
+produced by split. Parent 'c' is about to have downlink inserted.
+
+ aP
+ / | \
+ b cE d
+ / | \ / | \ | \
+ e f g hE q i j k
+
+
+### Page deletion
Vacuum never deletes tuples or pages from the entry tree. It traverses entry
tree leafs in logical order by rightlinks and removes deletable TIDs from
@@ -301,39 +366,83 @@ are vacuumed in two stages. At first stage, deletable TIDs are removed from
leafs. If first stage detects at least one empty page, then at the second stage
ginScanToDelete() deletes empty pages.
-ginScanToDelete() scans posting tree to delete empty pages, while vacuum holds
-cleanup lock on the posting tree root. This lock prevent concurrent inserts,
-because inserters hold a pin on the root page. In spite of inserters searches
-don't hold pin on root page. So, while new searches cannot begin while root page
-is locked, any already-in-progress scans can continue concurrently with vacuum.
+ginScanToDelete() traverses the whole tree in depth-first manner. It starts
+from the super-exclusive lock on the tree root. This lock prevents all the
+concurrent insertions into this tree while we're deleting pages. However,
+there are still might be some in-progress readers, who traversed root before
+we locked it.
+
+The picture below shows tree state after page deletion algorithm traversed to
+leftmost leaf of the tree.
-ginScanToDelete() does depth-first tree scanning while keeping each page in path
-from root to current page exclusively locked. It also keeps left sibling of
-each page in the path locked. Thus, if current page is to be removed, all
-required pages to remove both downlink and rightlink are already locked.
-Therefore, page deletion locks pages top to bottom and left to right breaking
-our general rule. But assuming there is no concurrent insertions, this can't
-cause a deadlock.
+ aE
+ / | \
+ bE c d
+ / | \ | \ | \
+ eE f g h i j k
-During replay od page deletion at standby, the page's left sibling, the target
-page, and its parent, are locked in that order. So it follows bottom to top and
-left to right rule.
+Deletion algorithm keeps exclusive locks on left siblings of pages comprising
+currently investigated path. Thus, if current page is to be removed, all
+required pages to remove both downlink and rightlink are already locked. That
+evades potential right to left page locking order, which could deadlock with
+concurrent stepping right.
A search concurrent to page deletion might already have read a pointer to the
-page to be deleted, and might be just about to follow it. A page can be reached
+page to be deleted, and might be just about to follow it. A page can be reached
via the right-link of its left sibling, or via its downlink in the parent.
-To prevent a backend from reaching a deleted page via a right-link, when
-following a right-link the lock on the previous page is not released until the
-lock on next page has been acquired.
+To prevent a backend from reaching a deleted page via a right-link, stepping
+right algorithm doesn't release lock on the current page until lock of the
+right page is acquired.
-The downlink is more tricky. A search descending the tree must release the lock
+The downlink is more tricky. A search descending the tree must release the lock
on the parent page before locking the child, or it could deadlock with a
concurrent split of the child page; a page split locks the parent, while already
-holding a lock on the child page. So, deleted page cannot be reclaimed
-immediately. Instead, we have to wait for every transaction, which might wait to
-reference this page, to finish. Corresponding processes must observe that the
-page is marked deleted and recover accordingly.
+holding a lock on the child page. So, deleted page cannot be reclaimed
+immediately. Instead, we have to wait for every transaction, which might wait
+to reference this page, to finish. Corresponding processes must observe that
+the page is marked deleted and recover accordingly.
+
+The picture below shows tree state after page deletion algorithm further
+traversed the tree. Currently investigated path is 'a-c-h'. Left siblings 'b'
+and 'g' of 'c' and 'h' correspondingly are also exclusively locked.
+
+ aE
+ / | \
+ bE cE d
+ / | \ | \ | \
+ e f gE hE i j k
+
+The next picture shows tree state after page 'h' was deleted. It's marked with
+'deleted' flag and newest xid, which might visit it. Downlink from 'c' to 'h'
+is also deleted.
+
+ aE
+ / | \
+ bE cE d
+ / | \ \ | \
+ e f gE hD iE j k
+
+However, it's still possible that concurrent reader has seen downlink from 'c'
+to 'h' before we deleted it. In that case this reader will step right from 'h'
+to till find non-deleted page. Xid-marking of page 'h' guarantees that this
+page wouldn't be reused till all such readers gone. Next leaf page under
+investigation is 'i'. 'g' remains locked as it becomes left sibling of 'i'.
+
+The next picture shows tree state after 'i' and 'c' was deleted. Internal page
+'c' was deleted because it appeared to have no downlinks. The path under
+investigation is 'a-d-j'. Pages 'b' and 'g' are locked as self siblings of 'd'
+and 'j'.
+
+ aE
+ / \
+ bE cD dE
+ / | \ | \
+ e f gE hD iD jE k
+
+During the replay of page deletion at standby, the page's left sibling, the
+target page, and its parent, are locked in that order. This order guarantees
+no deadlock with concurrent reads.
Predicate Locking
-----------------
--
2.14.3
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7)
@ 2019-10-28 11:00 Peter Geoghegan <pg@bowt.ie>
parent: Alexander Korotkov <a.korotkov@postgrespro.ru>
1 sibling, 1 reply; 67+ messages in thread
From: Peter Geoghegan @ 2019-10-28 11:00 UTC (permalink / raw)
To: Alexander Korotkov <a.korotkov@postgrespro.ru>; +Cc: chenhj <chjischj@163.com>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi Alexander,
On Sun, Oct 27, 2019 at 7:04 PM Alexander Korotkov
<a.korotkov@postgrespro.ru> wrote:
> It tool me a lot of efforts to revise a concurrency section in README.
> I can't judge the result, but I probably did my best. I'd like to
> commit this patchset including both bug fixes and README update. But
> I'd like you to take a look on the README patch first.
Thank you for working on this.
I am flying back to the USA today, and will try to take a look at what
you came up with on the way. I will definitely have some feedback in
the next few days.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7)
@ 2019-10-28 17:32 Alexander Korotkov <a.korotkov@postgrespro.ru>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 0 replies; 67+ messages in thread
From: Alexander Korotkov @ 2019-10-28 17:32 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: chenhj <chjischj@163.com>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Mon, Oct 28, 2019 at 2:00 PM Peter Geoghegan <pg@bowt.ie> wrote:
> On Sun, Oct 27, 2019 at 7:04 PM Alexander Korotkov
> <a.korotkov@postgrespro.ru> wrote:
> > It tool me a lot of efforts to revise a concurrency section in README.
> > I can't judge the result, but I probably did my best. I'd like to
> > commit this patchset including both bug fixes and README update. But
> > I'd like you to take a look on the README patch first.
>
> Thank you for working on this.
>
> I am flying back to the USA today, and will try to take a look at what
> you came up with on the way. I will definitely have some feedback in
> the next few days.
Thank you so much!
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7)
@ 2019-10-29 23:34 Peter Geoghegan <pg@bowt.ie>
parent: Alexander Korotkov <a.korotkov@postgrespro.ru>
1 sibling, 1 reply; 67+ messages in thread
From: Peter Geoghegan @ 2019-10-29 23:34 UTC (permalink / raw)
To: Alexander Korotkov <a.korotkov@postgrespro.ru>; +Cc: chenhj <chjischj@163.com>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Sun, Oct 27, 2019 at 12:04 PM Alexander Korotkov
<a.korotkov@postgrespro.ru> wrote:
> (0001-Fix-deadlock-between-ginDeletePage-and-ginStepRigh-3.patch)
Some thoughts on the first patch:
* How do comparisons of items in each type of B-Tree work?
I would like to see a statement about invariants similar to nbtree's
"subtree" invariant. Looks like the "high key" is <= (not <) in each
case (i.e. for both entry trees and posting trees), just like nbtree.
nbtree has an explicit high key rather than using the current highest
data item on the page (maybe this can be called a "pseudo high key" or
something). That is one important difference, but in general GIN seems
to copy nbtree. I think.
However, GIN never explains stuff that must be affected by invariants
in binary search routines like entryLocateLeafEntry() and
dataLocateItem(). GIN never makes the similarities and differences
clear. Maybe you could do more on that -- it's too hard to tell why
entryLocateLeafEntry() and entryLocateEntry() (i.e. leaf and internal
page binary search variants for entry B-Trees) do things differently
to each other (and do things differently to nbtree's binary search
routine).
This code from entryLocateEntry() is a good example of how this can be
confusing:
if (result == 0)
{
stack->off = mid;
Assert(GinGetDownlink(itup) != GIN_ROOT_BLKNO);
return GinGetDownlink(itup);
}
else if (result > 0)
low = mid + 1;
else
high = mid;
Some questions about this code:
* Why is the "if (result == 0)" branch using the block number/downlink
from the "mid" tuple as its return value?
Why not follow nbtree's _bt_binsrch() here too, by returning "the last
key < scan key" on internal pages? If your high key from one level
down is <=, and also a pseudo high key (i.e. both a "real" entry and a
high key), how can it be correct to go to the block/downlink from an
equal "mid" tuple like this? Won't that make index scans miss the
pseudo high key, which they have to return to the scan?
* Why is it okay that there is no "negative infinity" item in internal
pages for the entry tree?
GIN does not have to copy nbtree in every detail to be correct, but it
confuses me that it *mostly* copies nbtree, but doesn't do so *fully*.
As I wrote just now (or at least implied), entryIsMoveRight() works in
a similar way to _bt_moveright(), and yet we still have these apparent
differences with how binary searches work that seems to not be
compatible with that. Maybe this is okay, but I cannot understand why
that is right now. It looks wrong to me -- so wrong that I suppose I
must be mistaken.
I also spotted some fairly minor issues:
* s/rightest/rightmost/
* It's weird that GinBtree/GinBtreeData is a set of callbacks for
both posting trees and main entry trees, since the rules are now quite
different in each case. Not sure if it's worth saying something about
that.
* "Exclusive lock" makes me think of "ExclusiveLock", which is a kind
of heavyweight lock (not a buffer lock). I suggest changing that
wording to avoid confusion.
In general, it seems very important to be clear about exactly how the
key space works. The nbtree work for v12 greatly benefitted from
defining comparisons in a way that didn't really change how nbtree
worked, while at the same time minimizing I/O and making nbtree
faithful to Lehman & Yao's original design. It isn't obvious how
valuable it is to really carefully define how invariants and key
comparisons work, but it seems possible to solve a lot of problems
that way.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7)
@ 2019-11-10 23:42 Alexander Korotkov <a.korotkov@postgrespro.ru>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 67+ messages in thread
From: Alexander Korotkov @ 2019-11-10 23:42 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: chenhj <chjischj@163.com>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi!
I'm sorry for late reply. I was busy with various things. Also
digging into these details took some time. Please find my explanation
below.
On Wed, Oct 30, 2019 at 2:34 AM Peter Geoghegan <pg@bowt.ie> wrote:
> In general, it seems very important to be clear about exactly how the
> key space works. The nbtree work for v12 greatly benefitted from
> defining comparisons in a way that didn't really change how nbtree
> worked, while at the same time minimizing I/O and making nbtree
> faithful to Lehman & Yao's original design. It isn't obvious how
> valuable it is to really carefully define how invariants and key
> comparisons work, but it seems possible to solve a lot of problems
> that way.
gin packs downlinks and pivot key into tuples in the different way
than nbtree does. Let's see the logical structure of internal B-tree
page. We can represent it as following.
downlink_1, key_1_2, downlink_2, key_2_3, .... , downlink_n, highkey
key_{i-1}_i is pivot key, which splits key space between
downlink_{i-1} and downlink_i. So, every key under downlink_{i-1} is
<= key_{i-1}_i. Every key under downlink_i is > key_{i-1}_i. And all
underlying keys are <= highkey.
nbtree packs that into tuples as followings (tuples are shown in parentheses).
(highkey), (-inf, downlink_1), (key_1_2, downlink_2), ...
(key_{n-1}_n, downlink_n)
So, we store highkey separately. key_{i-1}_i and downlink_i form a
tuple together. downlink_1 is coupled with -inf key.
gin packs tuples in a different way.
(downlink_1, key_1_2), (downlink_2, key_2_3), ... , (downlink_n, highkey)
So, in GIN key_{i-1}_i and downlink_{i-1} form a tuple. Highkey is
coupled with downlink_n. And -inf key is not needed here.
But there is couple notes about highkey:
1) In entry tree rightmost page, a key coupled with downlink_n doesn't
really matter. Highkey is assumed to be infinity.
2) In posting tree, a key coupled with downlink_n always doesn't
matter. Highkey for non-rightmost pages is stored separately and
accessed via GinDataPageGetRightBound().
I think this explains following:
1) The invariants in gin btree are same as they are in nbtree. Just
page layout is different.
2) The way entryLocateEntry() works. In particular why it's OK to go
the mid downlink if corresponding key equals.
3) There is no "negative infinity" item in internal pages.
Does the explanation of above looks clear for you? If so, I can put
it into the README.
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7)
@ 2019-11-17 18:18 Alexander Korotkov <a.korotkov@postgrespro.ru>
parent: Alexander Korotkov <a.korotkov@postgrespro.ru>
0 siblings, 1 reply; 67+ messages in thread
From: Alexander Korotkov @ 2019-11-17 18:18 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: chenhj <chjischj@163.com>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Mon, Nov 11, 2019 at 2:42 AM Alexander Korotkov
<a.korotkov@postgrespro.ru> wrote:
> I'm sorry for late reply. I was busy with various things. Also
> digging into these details took some time. Please find my explanation
> below.
>
> On Wed, Oct 30, 2019 at 2:34 AM Peter Geoghegan <pg@bowt.ie> wrote:
> > In general, it seems very important to be clear about exactly how the
> > key space works. The nbtree work for v12 greatly benefitted from
> > defining comparisons in a way that didn't really change how nbtree
> > worked, while at the same time minimizing I/O and making nbtree
> > faithful to Lehman & Yao's original design. It isn't obvious how
> > valuable it is to really carefully define how invariants and key
> > comparisons work, but it seems possible to solve a lot of problems
> > that way.
>
> gin packs downlinks and pivot key into tuples in the different way
> than nbtree does. Let's see the logical structure of internal B-tree
> page. We can represent it as following.
>
> downlink_1, key_1_2, downlink_2, key_2_3, .... , downlink_n, highkey
>
> key_{i-1}_i is pivot key, which splits key space between
> downlink_{i-1} and downlink_i. So, every key under downlink_{i-1} is
> <= key_{i-1}_i. Every key under downlink_i is > key_{i-1}_i. And all
> underlying keys are <= highkey.
>
> nbtree packs that into tuples as followings (tuples are shown in parentheses).
>
> (highkey), (-inf, downlink_1), (key_1_2, downlink_2), ...
> (key_{n-1}_n, downlink_n)
>
> So, we store highkey separately. key_{i-1}_i and downlink_i form a
> tuple together. downlink_1 is coupled with -inf key.
>
> gin packs tuples in a different way.
>
> (downlink_1, key_1_2), (downlink_2, key_2_3), ... , (downlink_n, highkey)
>
> So, in GIN key_{i-1}_i and downlink_{i-1} form a tuple. Highkey is
> coupled with downlink_n. And -inf key is not needed here.
>
> But there is couple notes about highkey:
> 1) In entry tree rightmost page, a key coupled with downlink_n doesn't
> really matter. Highkey is assumed to be infinity.
> 2) In posting tree, a key coupled with downlink_n always doesn't
> matter. Highkey for non-rightmost pages is stored separately and
> accessed via GinDataPageGetRightBound().
>
> I think this explains following:
> 1) The invariants in gin btree are same as they are in nbtree. Just
> page layout is different.
> 2) The way entryLocateEntry() works. In particular why it's OK to go
> the mid downlink if corresponding key equals.
> 3) There is no "negative infinity" item in internal pages.
>
> Does the explanation of above looks clear for you? If so, I can put
> it into the README.
So, I've put this explanation into README patch. I just change
designation to better match with Lehman & Yao paper and did some minor
enchantments.
I'm going to push this patchset if no objections.
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
Attachments: download all (3 files, .tar.gz)
0002-Fix-traversing-to-the-deleted-GIN-page-via-downlin-4.patch (application/octet-stream, 3.1K) download
From 5e8f16e120ebf4f388bf16efc29e0cc27428ced7 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <akorotkov@postgresql.org>
Date: Sun, 17 Nov 2019 21:10:45 +0300
Subject: [PATCH 2/3] Fix traversing to the deleted GIN page via downlink
Current GIN code appears to don't handle traversing to the deleted page via
downlink. This commit fixes that by stepping right from the delete page like
we do in nbtree.
This commit also fixes setting 'deleted' flag to the GIN pages. Now other page
flags are not erased once page is deleted. That helps to keep our assertions
true if we arrive deleted page via downlink.
Discussion: https://postgr.es/m/CAPpHfdvMvsw-NcE5bRS7R1BbvA4BxoDnVVjkXC5W0Czvy9LVrg%40mail.gmail.com
Author: Alexander Korotkov
Reviewed-by: Peter Geoghegan
Backpatch-through: 9.4
---
src/backend/access/gin/ginbtree.c | 7 -------
src/backend/access/gin/gindatapage.c | 3 +++
src/backend/access/gin/ginvacuum.c | 2 +-
src/backend/access/gin/ginxlog.c | 2 +-
4 files changed, 5 insertions(+), 9 deletions(-)
diff --git a/src/backend/access/gin/ginbtree.c b/src/backend/access/gin/ginbtree.c
index 20adfa21948..8f8cd85db6a 100644
--- a/src/backend/access/gin/ginbtree.c
+++ b/src/backend/access/gin/ginbtree.c
@@ -187,13 +187,6 @@ ginStepRight(Buffer buffer, Relation index, int lockmode)
if (isLeaf != GinPageIsLeaf(page) || isData != GinPageIsData(page))
elog(ERROR, "right sibling of GIN page is of different type");
- /*
- * Given the proper lock sequence above, we should never land on a deleted
- * page.
- */
- if (GinPageIsDeleted(page))
- elog(ERROR, "right sibling of GIN page was deleted");
-
return nextbuffer;
}
diff --git a/src/backend/access/gin/gindatapage.c b/src/backend/access/gin/gindatapage.c
index e8c34d6b1f6..e497210b4c0 100644
--- a/src/backend/access/gin/gindatapage.c
+++ b/src/backend/access/gin/gindatapage.c
@@ -238,6 +238,9 @@ dataIsMoveRight(GinBtree btree, Page page)
if (GinPageRightMost(page))
return false;
+ if (GinPageIsDeleted(page))
+ return true;
+
return (ginCompareItemPointers(&btree->itemptr, iptr) > 0) ? true : false;
}
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index 72169bb0bdc..11e050171ad 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -186,7 +186,7 @@ ginDeletePage(GinVacuumState *gvs, BlockNumber deleteBlkno, BlockNumber leftBlkn
* we shouldn't change rightlink field to save workability of running
* search scan
*/
- GinPageGetOpaque(page)->flags = GIN_DELETED;
+ GinPageSetDeleted(page);
MarkBufferDirty(pBuffer);
MarkBufferDirty(lBuffer);
diff --git a/src/backend/access/gin/ginxlog.c b/src/backend/access/gin/ginxlog.c
index c945b282721..387102f21a1 100644
--- a/src/backend/access/gin/ginxlog.c
+++ b/src/backend/access/gin/ginxlog.c
@@ -500,7 +500,7 @@ ginRedoDeletePage(XLogReaderState *record)
{
page = BufferGetPage(dbuffer);
Assert(GinPageIsData(page));
- GinPageGetOpaque(page)->flags = GIN_DELETED;
+ GinPageSetDeleted(page);
GinPageSetDeleteXid(page, data->deleteXid);
PageSetLSN(page, lsn);
MarkBufferDirty(dbuffer);
--
2.14.3
0001-Fix-deadlock-between-ginDeletePage-and-ginStepRigh-4.patch (application/octet-stream, 11.3K) download
From a73d0636c9c0c8363a6cae2815c938cf7d76ef5c Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <akorotkov@postgresql.org>
Date: Sun, 17 Nov 2019 21:10:34 +0300
Subject: [PATCH 1/3] Fix deadlock between ginDeletePage() and ginStepRight()
When ginDeletePage() is about to delete page it locks its left sibling to revise
the rightlink. So, it locks pages in right to left manner. Int he same time
ginStepRight() locks pages in left to right manner, and that could cause a
deadlock.
This commit makes ginScanToDelete() keep exclusive lock on left siblings of
currently investigated path. That elimites need to relock left sibling in
ginDeletePage(). Thus, deadlock with ginStepRight() can't happen anymore.
Reported-by: Chen Huajun
Discussion: https://postgr.es/m/5c332bd1.87b6.16d7c17aa98.Coremail.chjischj%40163.com
Author: Alexander Korotkov
Reviewed-by: Peter Geoghegan
Backpatch-through: 9.4
---
src/backend/access/gin/README | 79 ++++++++++++++++++++++----------------
src/backend/access/gin/ginvacuum.c | 54 +++++++++++++++++---------
2 files changed, 81 insertions(+), 52 deletions(-)
diff --git a/src/backend/access/gin/README b/src/backend/access/gin/README
index 76c12ae2f6b..e4152500cc6 100644
--- a/src/backend/access/gin/README
+++ b/src/backend/access/gin/README
@@ -277,50 +277,63 @@ followed by the packed items.
Concurrency
-----------
-The entry tree and each posting tree is a B-tree, with right-links connecting
+The entry tree and each posting tree are B-trees, with right-links connecting
sibling pages at the same level. This is the same structure that is used in
the regular B-tree indexam (invented by Lehman & Yao), but we don't support
scanning a GIN trees backwards, so we don't need left-links.
To avoid deadlocks, B-tree pages must always be locked in the same order:
-left to right, and bottom to top. When searching, the tree is traversed from
-top to bottom, so the lock on the parent page must be released before
+left to right, and bottom to top. The exception is page deletion during vacuum,
+which would be considered separately. When searching, the tree is traversed
+from top to bottom, so the lock on the parent page must be released before
descending to the next level. Concurrent page splits move the keyspace to
right, so after following a downlink, the page actually containing the key
we're looking for might be somewhere to the right of the page we landed on.
In that case, we follow the right-links until we find the page we're looking
-for.
-
-To delete a page, the page's left sibling, the target page, and its parent,
-are locked in that order, and the page is marked as deleted. However, a
-concurrent search might already have read a pointer to the page, and might be
-just about to follow it. A page can be reached via the right-link of its left
-sibling, or via its downlink in the parent.
+for. In spite of searches, insertions keeps pins on all pages in path from
+root to the current page. These pages potentially requies downlink insertion,
+while pins prevent them from being deleted.
+
+Vacuum never deletes tuples or pages from the entry tree. It traverses entry
+tree leafs in logical order by rightlinks and removes deletable TIDs from
+posting lists. Posting trees are processed by links from entry tree leafs. They
+are vacuumed in two stages. At first stage, deletable TIDs are removed from
+leafs. If first stage detects at least one empty page, then at the second stage
+ginScanToDelete() deletes empty pages.
+
+ginScanToDelete() scans posting tree to delete empty pages, while vacuum holds
+cleanup lock on the posting tree root. This lock prevent concurrent inserts,
+because inserters hold a pin on the root page. In spite of inserters searches
+don't hold pin on root page. So, while new searches cannot begin while root page
+is locked, any already-in-progress scans can continue concurrently with vacuum.
+
+ginScanToDelete() does depth-first tree scanning while keeping each page in path
+from root to current page exclusively locked. It also keeps left sibling of
+each page in the path locked. Thus, if current page is to be removed, all
+required pages to remove both downlink and rightlink are already locked.
+Therefore, page deletion locks pages top to bottom and left to right breaking
+our general rule. But assuming there is no concurrent insertions, this can't
+cause a deadlock.
+
+During replay od page deletion at standby, the page's left sibling, the target
+page, and its parent, are locked in that order. So it follows bottom to top and
+left to right rule.
+
+A search concurrent to page deletion might already have read a pointer to the
+page to be deleted, and might be just about to follow it. A page can be reached
+via the right-link of its left sibling, or via its downlink in the parent.
To prevent a backend from reaching a deleted page via a right-link, when
-following a right-link the lock on the previous page is not released until
-the lock on next page has been acquired.
-
-The downlink is more tricky. A search descending the tree must release the
-lock on the parent page before locking the child, or it could deadlock with
-a concurrent split of the child page; a page split locks the parent, while
-already holding a lock on the child page. So, deleted page cannot be reclaimed
-immediately. Instead, we have to wait for every transaction, which might wait
-to reference this page, to finish. Corresponding processes must observe that
-the page is marked deleted and recover accordingly.
-
-The previous paragraph's reasoning only applies to searches, and only to
-posting trees. To protect from inserters following a downlink to a deleted
-page, vacuum simply locks out all concurrent insertions to the posting tree,
-by holding a super-exclusive lock on the posting tree root. Inserters hold a
-pin on the root page, but searches do not, so while new searches cannot begin
-while root page is locked, any already-in-progress scans can continue
-concurrently with vacuum. In the entry tree, we never delete pages.
-
-(This is quite different from the mechanism the btree indexam uses to make
-page-deletions safe; it stamps the deleted pages with an XID and keeps the
-deleted pages around with the right-link intact until all concurrent scans
-have finished.)
+following a right-link the lock on the previous page is not released until the
+lock on next page has been acquired.
+
+The downlink is more tricky. A search descending the tree must release the lock
+on the parent page before locking the child, or it could deadlock with a
+concurrent split of the child page; a page split locks the parent, while already
+holding a lock on the child page. So, deleted page cannot be reclaimed
+immediately. Instead, we have to wait for every transaction, which might wait to
+reference this page, to finish. Corresponding processes must observe that the
+page is marked deleted and recover accordingly.
Predicate Locking
-----------------
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index dc46f2460e2..72169bb0bdc 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -117,7 +117,8 @@ typedef struct DataPageDeleteStack
struct DataPageDeleteStack *parent;
BlockNumber blkno; /* current block number */
- BlockNumber leftBlkno; /* rightest non-deleted page on left */
+ Buffer leftBuffer; /* pinned and locked rightest non-deleted page
+ * on left */
bool isRoot;
} DataPageDeleteStack;
@@ -139,11 +140,8 @@ ginDeletePage(GinVacuumState *gvs, BlockNumber deleteBlkno, BlockNumber leftBlkn
/*
* This function MUST be called only if someone of parent pages hold
* exclusive cleanup lock. This guarantees that no insertions currently
- * happen in this subtree. Caller also acquire Exclusive lock on deletable
- * page and is acquiring and releasing exclusive lock on left page before.
- * Left page was locked and released. Then parent and this page are
- * locked. We acquire left page lock here only to mark page dirty after
- * changing right pointer.
+ * happen in this subtree. Caller also acquires Exclusive locks on
+ * deletable and left pages.
*/
lBuffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM, leftBlkno,
RBM_NORMAL, gvs->strategy);
@@ -152,8 +150,6 @@ ginDeletePage(GinVacuumState *gvs, BlockNumber deleteBlkno, BlockNumber leftBlkn
pBuffer = ReadBufferExtended(gvs->index, MAIN_FORKNUM, parentBlkno,
RBM_NORMAL, gvs->strategy);
- LockBuffer(lBuffer, GIN_EXCLUSIVE);
-
page = BufferGetPage(dBuffer);
rightlink = GinPageGetOpaque(page)->rightlink;
@@ -227,7 +223,7 @@ ginDeletePage(GinVacuumState *gvs, BlockNumber deleteBlkno, BlockNumber leftBlkn
}
ReleaseBuffer(pBuffer);
- UnlockReleaseBuffer(lBuffer);
+ ReleaseBuffer(lBuffer);
ReleaseBuffer(dBuffer);
END_CRIT_SECTION();
@@ -237,7 +233,11 @@ ginDeletePage(GinVacuumState *gvs, BlockNumber deleteBlkno, BlockNumber leftBlkn
/*
- * scans posting tree and deletes empty pages
+ * Scans posting tree and deletes empty pages. Caller must lock root page for
+ * cleanup. During scan path from root to current page is kept exclusively
+ * locked. Also keep left page exclusively locked, because ginDeletePage()
+ * needs it. If we try to relock left page later, it could deadlock with
+ * ginStepRight().
*/
static bool
ginScanToDelete(GinVacuumState *gvs, BlockNumber blkno, bool isRoot,
@@ -260,7 +260,7 @@ ginScanToDelete(GinVacuumState *gvs, BlockNumber blkno, bool isRoot,
me = (DataPageDeleteStack *) palloc0(sizeof(DataPageDeleteStack));
me->parent = parent;
parent->child = me;
- me->leftBlkno = InvalidBlockNumber;
+ me->leftBuffer = InvalidBuffer;
}
else
me = parent->child;
@@ -288,6 +288,12 @@ ginScanToDelete(GinVacuumState *gvs, BlockNumber blkno, bool isRoot,
if (ginScanToDelete(gvs, PostingItemGetBlockNumber(pitem), false, me, i))
i--;
}
+
+ if (GinPageRightMost(page) && BufferIsValid(me->child->leftBuffer))
+ {
+ UnlockReleaseBuffer(me->child->leftBuffer);
+ me->child->leftBuffer = InvalidBuffer;
+ }
}
if (GinPageIsLeaf(page))
@@ -298,21 +304,31 @@ ginScanToDelete(GinVacuumState *gvs, BlockNumber blkno, bool isRoot,
if (isempty)
{
/* we never delete the left- or rightmost branch */
- if (me->leftBlkno != InvalidBlockNumber && !GinPageRightMost(page))
+ if (BufferIsValid(me->leftBuffer) && !GinPageRightMost(page))
{
Assert(!isRoot);
- ginDeletePage(gvs, blkno, me->leftBlkno, me->parent->blkno, myoff, me->parent->isRoot);
+ ginDeletePage(gvs, blkno, BufferGetBlockNumber(me->leftBuffer),
+ me->parent->blkno, myoff, me->parent->isRoot);
meDelete = true;
}
}
- if (!isRoot)
- LockBuffer(buffer, GIN_UNLOCK);
+ if (!meDelete)
+ {
+ if (BufferIsValid(me->leftBuffer))
+ UnlockReleaseBuffer(me->leftBuffer);
+ me->leftBuffer = buffer;
+ }
+ else
+ {
+ if (!isRoot)
+ LockBuffer(buffer, GIN_UNLOCK);
- ReleaseBuffer(buffer);
+ ReleaseBuffer(buffer);
+ }
- if (!meDelete)
- me->leftBlkno = blkno;
+ if (isRoot)
+ ReleaseBuffer(buffer);
return meDelete;
}
@@ -409,7 +425,7 @@ ginVacuumPostingTree(GinVacuumState *gvs, BlockNumber rootBlkno)
LockBufferForCleanup(buffer);
memset(&root, 0, sizeof(DataPageDeleteStack));
- root.leftBlkno = InvalidBlockNumber;
+ root.leftBuffer = InvalidBuffer;
root.isRoot = true;
ginScanToDelete(gvs, rootBlkno, true, &root, InvalidOffsetNumber);
--
2.14.3
0003-Revise-GIN-README-4.patch (application/octet-stream, 12.1K) download
From addee9d8a1c0d596377fa5989ce6d6daa876a6a1 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <akorotkov@postgresql.org>
Date: Sun, 17 Nov 2019 21:11:22 +0300
Subject: [PATCH 3/3] Revise GIN README
We find GIN concurrency bugs from time to time. One of the problems here is
that concurrency of GIN isn't well-documented in README. So, it might be even
hard to distinguish design bugs from implementation bugs.
This commit revised concurrency section in GIN README providing more details.
Some examples are illustrated in ASCII art.
Also, this commit add the explanation of how is tuple layout in internal GIN
B-tree page different in comparison with nbtree.
Discussion: https://postgr.es/m/5c332bd1.87b6.16d7c17aa98.Coremail.chjischj%40163.com
Author: Alexander Korotkov
Reviewed-by: Peter Geoghegan
Backpatch-through: 9.4
---
src/backend/access/gin/README | 214 ++++++++++++++++++++++++++++++++++--------
1 file changed, 176 insertions(+), 38 deletions(-)
diff --git a/src/backend/access/gin/README b/src/backend/access/gin/README
index e4152500cc6..125a82219b9 100644
--- a/src/backend/access/gin/README
+++ b/src/backend/access/gin/README
@@ -215,6 +215,35 @@ fit on one pending-list page must have those pages to itself, even if this
results in wasting much of the space on the preceding page and the last
page for the tuple.)
+GIN packs downlinks and pivot keys into internal page tuples in a different way
+than nbtree does. Lehman & Yao defines it as following.
+
+P_0, K_1, P_1, K_2, P_2, ... , K_n, P_n, K_{n+1}
+
+There P_i is a downlink and K_i is a key. K_i splits key space between P_{i-1}
+and P_i (0 <= i <= n). K_{n+1} is high key.
+
+In internal page tuple is key and downlink grouped together. nbtree packs
+keys and downlinks into tuples as following.
+
+(K_{n+1}, None), (-Inf, P_0), (K_1, P_1), ... , (K_n, P_n)
+
+There tuples are shown in parentheses. So, highkey is stored separately. P_i
+is grouped with K_i. P_0 is grouped with -Inf key.
+
+GIN packs keys and downlinks into tuples in a different way.
+
+(P_0, K_1), (P_1, K_2), ... , (P_n, K_{n+1})
+
+P_i is grouped with K_{i+1}. -Inf key is not needed.
+
+There are couple of additional notes regarding K_{n+1} key.
+1) In entry tree rightmost page, a key coupled with P_n doesn't really matter.
+Highkey is assumed to be infinity.
+2) In posting tree, a key coupled with P_n always doesn't matter. Highkey for
+non-rightmost pages is stored separately and accessed via
+GinDataPageGetRightBound().
+
Posting tree
------------
@@ -278,21 +307,86 @@ Concurrency
-----------
The entry tree and each posting tree are B-trees, with right-links connecting
-sibling pages at the same level. This is the same structure that is used in
+sibling pages at the same level. This is the same structure that is used in
the regular B-tree indexam (invented by Lehman & Yao), but we don't support
-scanning a GIN trees backwards, so we don't need left-links.
-
-To avoid deadlocks, B-tree pages must always be locked in the same order:
-left to right, and bottom to top. The exception is page deletion during vacuum,
-which would be considered separately. When searching, the tree is traversed
-from top to bottom, so the lock on the parent page must be released before
-descending to the next level. Concurrent page splits move the keyspace to
-right, so after following a downlink, the page actually containing the key
-we're looking for might be somewhere to the right of the page we landed on.
-In that case, we follow the right-links until we find the page we're looking
-for. In spite of searches, insertions keeps pins on all pages in path from
-root to the current page. These pages potentially requies downlink insertion,
-while pins prevent them from being deleted.
+scanning a GIN trees backwards, so we don't need left-links. The entry tree
+leaves don't have dedicated high keys, instead greatest leaf tuple serves as
+high key. That works because tuples are never deleted from the entry tree.
+
+The algorithms used to operate entry and posting trees are considered below.
+
+### Locating the leaf page
+
+When we search for leaf page in GIN btree to perform a read, we descend from
+the root page to the leaf through using downlinks taking pin and shared lock on
+one page at once. So, we release pin and shared lock on previous page before
+getting them on the next page.
+
+The picture below shows tree state after finding the leaf page. Lower case
+letters depicts tree pages. 'S' depicts shared lock on the page.
+
+ a
+ / | \
+ b c d
+ / | \ | \ | \
+ eS f g h i j k
+
+### Steping right
+
+Concurrent page splits move the keyspace to right, so after following a
+downlink, the page actually containing the key we're looking for might be
+somewhere to the right of the page we landed on. In that case, we follow the
+right-links until we find the page we're looking for.
+
+During stepping right we take pin and shared lock on the right sibling before
+releasing them from the current page. This mechanism was designed to protect
+from stepping to delete page. We step to the right sibling while hold lock on
+the rightlink pointing there. So, it's guaranteed that nobody updates rightlink
+concurrently and doesn't delete right sibling accordingly.
+
+The picture below shows two pages locked at once during stepping right.
+
+ a
+ / | \
+ b c d
+ / | \ | \ | \
+ eS fS g h i j k
+
+### Insert
+
+While finding appropriate leaf for insertion we also descend from the root to
+leaf, while shared locking one page at once in. But during insertion we don't
+release pins from root and internal pages. That could save us some lookups to
+the buffers hash table for downlinks insertion assuming parents are not changed
+due to concurrent splits. Once we reach leaf we re-lock the page in exclusive
+mode.
+
+The picture below shows leaf page locked in exclusive mode and ready for
+insertion. 'P' and 'E' depict pin and exclusive lock correspondingly.
+
+
+ aP
+ / | \
+ b cP d
+ / | \ | \ | \
+ e f g hE i j k
+
+
+If insert causes a page split, the parent is locked in exclusive mode before
+unlocking the left child. So, insertion algorithm can exclusively lock both
+parent and child pages at once starting from child.
+
+The picture below shows tree state after leaf page split. 'q' is new page
+produced by split. Parent 'c' is about to have downlink inserted.
+
+ aP
+ / | \
+ b cE d
+ / | \ / | \ | \
+ e f g hE q i j k
+
+
+### Page deletion
Vacuum never deletes tuples or pages from the entry tree. It traverses entry
tree leafs in logical order by rightlinks and removes deletable TIDs from
@@ -301,39 +395,83 @@ are vacuumed in two stages. At first stage, deletable TIDs are removed from
leafs. If first stage detects at least one empty page, then at the second stage
ginScanToDelete() deletes empty pages.
-ginScanToDelete() scans posting tree to delete empty pages, while vacuum holds
-cleanup lock on the posting tree root. This lock prevent concurrent inserts,
-because inserters hold a pin on the root page. In spite of inserters searches
-don't hold pin on root page. So, while new searches cannot begin while root page
-is locked, any already-in-progress scans can continue concurrently with vacuum.
+ginScanToDelete() traverses the whole tree in depth-first manner. It starts
+from the super-exclusive lock on the tree root. This lock prevents all the
+concurrent insertions into this tree while we're deleting pages. However,
+there are still might be some in-progress readers, who traversed root before
+we locked it.
+
+The picture below shows tree state after page deletion algorithm traversed to
+leftmost leaf of the tree.
-ginScanToDelete() does depth-first tree scanning while keeping each page in path
-from root to current page exclusively locked. It also keeps left sibling of
-each page in the path locked. Thus, if current page is to be removed, all
-required pages to remove both downlink and rightlink are already locked.
-Therefore, page deletion locks pages top to bottom and left to right breaking
-our general rule. But assuming there is no concurrent insertions, this can't
-cause a deadlock.
+ aE
+ / | \
+ bE c d
+ / | \ | \ | \
+ eE f g h i j k
-During replay od page deletion at standby, the page's left sibling, the target
-page, and its parent, are locked in that order. So it follows bottom to top and
-left to right rule.
+Deletion algorithm keeps exclusive locks on left siblings of pages comprising
+currently investigated path. Thus, if current page is to be removed, all
+required pages to remove both downlink and rightlink are already locked. That
+evades potential right to left page locking order, which could deadlock with
+concurrent stepping right.
A search concurrent to page deletion might already have read a pointer to the
-page to be deleted, and might be just about to follow it. A page can be reached
+page to be deleted, and might be just about to follow it. A page can be reached
via the right-link of its left sibling, or via its downlink in the parent.
-To prevent a backend from reaching a deleted page via a right-link, when
-following a right-link the lock on the previous page is not released until the
-lock on next page has been acquired.
+To prevent a backend from reaching a deleted page via a right-link, stepping
+right algorithm doesn't release lock on the current page until lock of the
+right page is acquired.
-The downlink is more tricky. A search descending the tree must release the lock
+The downlink is more tricky. A search descending the tree must release the lock
on the parent page before locking the child, or it could deadlock with a
concurrent split of the child page; a page split locks the parent, while already
-holding a lock on the child page. So, deleted page cannot be reclaimed
-immediately. Instead, we have to wait for every transaction, which might wait to
-reference this page, to finish. Corresponding processes must observe that the
-page is marked deleted and recover accordingly.
+holding a lock on the child page. So, deleted page cannot be reclaimed
+immediately. Instead, we have to wait for every transaction, which might wait
+to reference this page, to finish. Corresponding processes must observe that
+the page is marked deleted and recover accordingly.
+
+The picture below shows tree state after page deletion algorithm further
+traversed the tree. Currently investigated path is 'a-c-h'. Left siblings 'b'
+and 'g' of 'c' and 'h' correspondingly are also exclusively locked.
+
+ aE
+ / | \
+ bE cE d
+ / | \ | \ | \
+ e f gE hE i j k
+
+The next picture shows tree state after page 'h' was deleted. It's marked with
+'deleted' flag and newest xid, which might visit it. Downlink from 'c' to 'h'
+is also deleted.
+
+ aE
+ / | \
+ bE cE d
+ / | \ \ | \
+ e f gE hD iE j k
+
+However, it's still possible that concurrent reader has seen downlink from 'c'
+to 'h' before we deleted it. In that case this reader will step right from 'h'
+to till find non-deleted page. Xid-marking of page 'h' guarantees that this
+page wouldn't be reused till all such readers gone. Next leaf page under
+investigation is 'i'. 'g' remains locked as it becomes left sibling of 'i'.
+
+The next picture shows tree state after 'i' and 'c' was deleted. Internal page
+'c' was deleted because it appeared to have no downlinks. The path under
+investigation is 'a-d-j'. Pages 'b' and 'g' are locked as self siblings of 'd'
+and 'j'.
+
+ aE
+ / \
+ bE cD dE
+ / | \ | \
+ e f gE hD iD jE k
+
+During the replay of page deletion at standby, the page's left sibling, the
+target page, and its parent, are locked in that order. This order guarantees
+no deadlock with concurrent reads.
Predicate Locking
-----------------
--
2.14.3
* Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7)
@ 2019-11-19 21:20 Alexander Korotkov <a.korotkov@postgrespro.ru>
parent: Alexander Korotkov <a.korotkov@postgrespro.ru>
0 siblings, 0 replies; 67+ messages in thread
From: Alexander Korotkov @ 2019-11-19 21:20 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: chenhj <chjischj@163.com>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Sun, Nov 17, 2019 at 9:18 PM Alexander Korotkov
<a.korotkov@postgrespro.ru> wrote:
> So, I've put this explanation into README patch. I just change
> designation to better match with Lehman & Yao paper and did some minor
> enchantments.
>
> I'm going to push this patchset if no objections.
So, pushed with minor changes during backpatching.
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 67+ messages in thread
end of thread, other threads:[~2019-11-19 21:20 UTC | newest]
Thread overview: 67+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2018-10-29 19:04 Connections hang indefinitely while taking a gin index's LWLock buffer_content lock chenhj <chjischj@163.com>
2018-11-06 17:57 ` chenhj <chjischj@163.com>
2018-11-07 20:52 ` Peter Geoghegan <pg@bowt.ie>
2018-11-07 23:18 ` Peter Geoghegan <pg@bowt.ie>
2018-11-11 17:10 ` chenhj <chjischj@163.com>
2018-11-08 01:46 ` Peter Geoghegan <pg@bowt.ie>
2018-11-08 01:49 ` Peter Geoghegan <pg@bowt.ie>
2018-11-11 01:42 ` Peter Geoghegan <pg@bowt.ie>
2018-11-11 17:33 ` chenhj <chjischj@163.com>
2018-11-12 01:46 ` Re:Re: Connections hang indefinitely while taking a gin index's
LWLock buffer_content lock chjischj@163.com <chjischj@163.com>
2018-12-04 17:01 ` Andres Freund <andres@anarazel.de>
2018-12-06 19:56 ` Alexander Korotkov <aekorotkov@gmail.com>
2018-12-06 20:51 ` Alexander Korotkov <aekorotkov@gmail.com>
2018-12-06 21:50 ` Peter Geoghegan <pg@bowt.ie>
2018-12-07 09:14 ` Andrey Borodin <x4mmm@yandex-team.ru>
2018-12-07 15:34 ` Alexander Korotkov <aekorotkov@gmail.com>
2018-12-08 01:54 ` Alexander Korotkov <aekorotkov@gmail.com>
2018-12-08 09:48 ` Andrey Borodin <x4mmm@yandex-team.ru>
2018-12-09 19:25 ` Alexander Korotkov <aekorotkov@gmail.com>
2018-12-10 13:56 ` Andrey Borodin <x4mmm@yandex-team.ru>
2018-12-10 13:58 ` Andrey Borodin <x4mmm@yandex-team.ru>
2018-12-10 22:43 ` Alexander Korotkov <aekorotkov@gmail.com>
2018-12-11 07:14 ` Andrey Borodin <x4mmm@yandex-team.ru>
2018-12-11 22:26 ` Alexander Korotkov <aekorotkov@gmail.com>
2018-12-12 13:08 ` Andrey Borodin <x4mmm@yandex-team.ru>
2018-12-13 12:03 ` Alexander Korotkov <aekorotkov@gmail.com>
2018-12-13 15:32 ` Bruce Momjian <bruce@momjian.us>
2018-12-13 16:48 ` Tom Lane <tgl@sss.pgh.pa.us>
2018-12-13 17:06 ` Andrey Borodin <x4mmm@yandex-team.ru>
2018-12-13 19:40 ` Alexander Korotkov <aekorotkov@gmail.com>
2018-12-13 19:46 ` Andres Freund <andres@anarazel.de>
2018-12-13 19:48 ` Alexander Korotkov <aekorotkov@gmail.com>
2019-03-21 18:25 ` Simon Riggs <simon@2ndquadrant.com>
2019-03-21 19:18 ` Alexander Korotkov <aekorotkov@gmail.com>
2019-03-21 20:14 ` Simon Riggs <simon@2ndquadrant.com>
2019-03-21 21:06 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-03-22 08:05 ` Alexander Korotkov <aekorotkov@gmail.com>
2019-03-23 08:43 ` Alexander Korotkov <aekorotkov@gmail.com>
2019-03-24 13:35 ` Alexander Korotkov <aekorotkov@gmail.com>
2018-12-14 16:19 ` Bruce Momjian <bruce@momjian.us>
2018-12-10 22:50 ` Alexander Korotkov <aekorotkov@gmail.com>
2018-12-12 00:22 ` Alexander Korotkov <aekorotkov@gmail.com>
2018-12-13 15:12 ` Re:Connections hang indefinitely while taking a gin index's LWLock
buffer_content lock chjischj@163.com <chjischj@163.com>
2018-11-14 18:11 ` Andrey Borodin <x4mmm@yandex-team.ru>
2018-11-15 12:30 ` Andrey Borodin <x4mmm@yandex-team.ru>
2018-11-23 10:18 ` Andrey Borodin <x4mmm@yandex-team.ru>
2018-11-24 00:18 ` Peter Geoghegan <pg@bowt.ie>
2019-09-29 08:16 Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7) chenhj <chjischj@163.com>
2019-09-29 14:38 ` Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7) Alexander Korotkov <a.korotkov@postgrespro.ru>
2019-09-29 15:12 ` Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7) Alexander Korotkov <a.korotkov@postgrespro.ru>
2019-09-29 16:27 ` Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7) Alexander Korotkov <a.korotkov@postgrespro.ru>
2019-09-30 19:54 ` Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7) Peter Geoghegan <pg@bowt.ie>
2019-10-01 02:55 ` Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7) Alexander Korotkov <a.korotkov@postgrespro.ru>
2019-10-03 21:05 ` Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7) Alexander Korotkov <a.korotkov@postgrespro.ru>
2019-10-27 19:03 ` Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7) Alexander Korotkov <a.korotkov@postgrespro.ru>
2019-10-28 11:00 ` Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7) Peter Geoghegan <pg@bowt.ie>
2019-10-28 17:32 ` Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7) Alexander Korotkov <a.korotkov@postgrespro.ru>
2019-10-29 23:34 ` Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7) Peter Geoghegan <pg@bowt.ie>
2019-11-10 23:42 ` Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7) Alexander Korotkov <a.korotkov@postgrespro.ru>
2019-11-17 18:18 ` Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7) Alexander Korotkov <a.korotkov@postgrespro.ru>
2019-11-19 21:20 ` Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7) Alexander Korotkov <a.korotkov@postgrespro.ru>
2019-09-29 19:52 ` Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7) Peter Geoghegan <pg@bowt.ie>
2019-09-30 05:38 ` Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7) Andrey Borodin <x4mmm@yandex-team.ru>
2019-09-30 17:37 ` Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7) Alexander Korotkov <a.korotkov@postgrespro.ru>
2019-09-30 19:07 ` Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7) Peter Geoghegan <pg@bowt.ie>
2019-09-30 17:59 ` Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7) Alexander Korotkov <a.korotkov@postgrespro.ru>
2019-09-30 19:25 ` Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock(PG10.7) Peter Geoghegan <pg@bowt.ie>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox