pg.ddx.io pgsql-committers@postgresql.org mailing list archivehelp / color / mirror / Atom feed
pgsql: Fix an undetected deadlock due to apply worker. 6+ messages / 1 participants [nested] [flat]
* pgsql: Fix an undetected deadlock due to apply worker. @ 2023-12-11 03:33 Amit Kapila <akapila@postgresql.org> 0 siblings, 0 replies; 6+ messages in thread From: Amit Kapila @ 2023-12-11 03:33 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix an undetected deadlock due to apply worker. The apply worker needs to update the state of the subscription tables to 'READY' during the synchronization phase which requires locking the corresponding subscription. The apply worker also waits for the subscription tables to reach the 'SYNCDONE' state after holding the locks on the subscription and the wait is done using WaitLatch. The 'SYNCDONE' state is changed by tablesync workers again by locking the corresponding subscription. Both the state updates use AccessShareLock mode to lock the subscription, so they can't block each other. However, a backend can simultaneously try to acquire a lock on the same subscription using AccessExclusiveLock mode to alter the subscription. Now, the backend's wait on a lock can sneak in between the apply worker and table sync worker causing deadlock. In other words, apply_worker waits for tablesync worker which waits for backend, and backend waits for apply worker. This is not detected by the deadlock detector because apply worker uses WaitLatch. The fix is to release existing locks in apply worker before it starts to wait for tablesync worker to change the state. Reported-by: Tomas Vondra Author: Shlok Kyal Reviewed-by: Amit Kapila, Peter Smith Backpatch-through: 12 Discussion: https://postgr.es/m/d291bb50-12c4-e8af-2af2-7bb9bb4d8e3e@enterprisedb.com Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/8d7d2197f31c1839db4726d1a12cb29016f4fa36 Modified Files -------------- src/backend/replication/logical/tablesync.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Fix an undetected deadlock due to apply worker. @ 2023-12-11 03:33 Amit Kapila <akapila@postgresql.org> 0 siblings, 0 replies; 6+ messages in thread From: Amit Kapila @ 2023-12-11 03:33 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix an undetected deadlock due to apply worker. The apply worker needs to update the state of the subscription tables to 'READY' during the synchronization phase which requires locking the corresponding subscription. The apply worker also waits for the subscription tables to reach the 'SYNCDONE' state after holding the locks on the subscription and the wait is done using WaitLatch. The 'SYNCDONE' state is changed by tablesync workers again by locking the corresponding subscription. Both the state updates use AccessShareLock mode to lock the subscription, so they can't block each other. However, a backend can simultaneously try to acquire a lock on the same subscription using AccessExclusiveLock mode to alter the subscription. Now, the backend's wait on a lock can sneak in between the apply worker and table sync worker causing deadlock. In other words, apply_worker waits for tablesync worker which waits for backend, and backend waits for apply worker. This is not detected by the deadlock detector because apply worker uses WaitLatch. The fix is to release existing locks in apply worker before it starts to wait for tablesync worker to change the state. Reported-by: Tomas Vondra Author: Shlok Kyal Reviewed-by: Amit Kapila, Peter Smith Backpatch-through: 12 Discussion: https://postgr.es/m/d291bb50-12c4-e8af-2af2-7bb9bb4d8e3e@enterprisedb.com Branch ------ REL_16_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/01cc92fa6251c4d1a0ac128806db6f7d8f31ec7f Modified Files -------------- src/backend/replication/logical/tablesync.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Fix an undetected deadlock due to apply worker. @ 2023-12-11 03:33 Amit Kapila <akapila@postgresql.org> 0 siblings, 0 replies; 6+ messages in thread From: Amit Kapila @ 2023-12-11 03:33 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix an undetected deadlock due to apply worker. The apply worker needs to update the state of the subscription tables to 'READY' during the synchronization phase which requires locking the corresponding subscription. The apply worker also waits for the subscription tables to reach the 'SYNCDONE' state after holding the locks on the subscription and the wait is done using WaitLatch. The 'SYNCDONE' state is changed by tablesync workers again by locking the corresponding subscription. Both the state updates use AccessShareLock mode to lock the subscription, so they can't block each other. However, a backend can simultaneously try to acquire a lock on the same subscription using AccessExclusiveLock mode to alter the subscription. Now, the backend's wait on a lock can sneak in between the apply worker and table sync worker causing deadlock. In other words, apply_worker waits for tablesync worker which waits for backend, and backend waits for apply worker. This is not detected by the deadlock detector because apply worker uses WaitLatch. The fix is to release existing locks in apply worker before it starts to wait for tablesync worker to change the state. Reported-by: Tomas Vondra Author: Shlok Kyal Reviewed-by: Amit Kapila, Peter Smith Backpatch-through: 12 Discussion: https://postgr.es/m/d291bb50-12c4-e8af-2af2-7bb9bb4d8e3e@enterprisedb.com Branch ------ REL_15_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/332b43063351758055f68c6227444e6003dba769 Modified Files -------------- src/backend/replication/logical/tablesync.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Fix an undetected deadlock due to apply worker. @ 2023-12-11 03:33 Amit Kapila <akapila@postgresql.org> 0 siblings, 0 replies; 6+ messages in thread From: Amit Kapila @ 2023-12-11 03:33 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix an undetected deadlock due to apply worker. The apply worker needs to update the state of the subscription tables to 'READY' during the synchronization phase which requires locking the corresponding subscription. The apply worker also waits for the subscription tables to reach the 'SYNCDONE' state after holding the locks on the subscription and the wait is done using WaitLatch. The 'SYNCDONE' state is changed by tablesync workers again by locking the corresponding subscription. Both the state updates use AccessShareLock mode to lock the subscription, so they can't block each other. However, a backend can simultaneously try to acquire a lock on the same subscription using AccessExclusiveLock mode to alter the subscription. Now, the backend's wait on a lock can sneak in between the apply worker and table sync worker causing deadlock. In other words, apply_worker waits for tablesync worker which waits for backend, and backend waits for apply worker. This is not detected by the deadlock detector because apply worker uses WaitLatch. The fix is to release existing locks in apply worker before it starts to wait for tablesync worker to change the state. Reported-by: Tomas Vondra Author: Shlok Kyal Reviewed-by: Amit Kapila, Peter Smith Backpatch-through: 12 Discussion: https://postgr.es/m/d291bb50-12c4-e8af-2af2-7bb9bb4d8e3e@enterprisedb.com Branch ------ REL_14_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/3f86867d560fbcdc717ad301c1a7142992f311c2 Modified Files -------------- src/backend/replication/logical/tablesync.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Fix an undetected deadlock due to apply worker. @ 2023-12-11 03:33 Amit Kapila <akapila@postgresql.org> 0 siblings, 0 replies; 6+ messages in thread From: Amit Kapila @ 2023-12-11 03:33 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix an undetected deadlock due to apply worker. The apply worker needs to update the state of the subscription tables to 'READY' during the synchronization phase which requires locking the corresponding subscription. The apply worker also waits for the subscription tables to reach the 'SYNCDONE' state after holding the locks on the subscription and the wait is done using WaitLatch. The 'SYNCDONE' state is changed by tablesync workers again by locking the corresponding subscription. Both the state updates use AccessShareLock mode to lock the subscription, so they can't block each other. However, a backend can simultaneously try to acquire a lock on the same subscription using AccessExclusiveLock mode to alter the subscription. Now, the backend's wait on a lock can sneak in between the apply worker and table sync worker causing deadlock. In other words, apply_worker waits for tablesync worker which waits for backend, and backend waits for apply worker. This is not detected by the deadlock detector because apply worker uses WaitLatch. The fix is to release existing locks in apply worker before it starts to wait for tablesync worker to change the state. Reported-by: Tomas Vondra Author: Shlok Kyal Reviewed-by: Amit Kapila, Peter Smith Backpatch-through: 12 Discussion: https://postgr.es/m/d291bb50-12c4-e8af-2af2-7bb9bb4d8e3e@enterprisedb.com Branch ------ REL_13_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/b47ea4f3d3056d70dd5dc0940b37f843c62739c6 Modified Files -------------- src/backend/replication/logical/tablesync.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Fix an undetected deadlock due to apply worker. @ 2023-12-11 03:33 Amit Kapila <akapila@postgresql.org> 0 siblings, 0 replies; 6+ messages in thread From: Amit Kapila @ 2023-12-11 03:33 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix an undetected deadlock due to apply worker. The apply worker needs to update the state of the subscription tables to 'READY' during the synchronization phase which requires locking the corresponding subscription. The apply worker also waits for the subscription tables to reach the 'SYNCDONE' state after holding the locks on the subscription and the wait is done using WaitLatch. The 'SYNCDONE' state is changed by tablesync workers again by locking the corresponding subscription. Both the state updates use AccessShareLock mode to lock the subscription, so they can't block each other. However, a backend can simultaneously try to acquire a lock on the same subscription using AccessExclusiveLock mode to alter the subscription. Now, the backend's wait on a lock can sneak in between the apply worker and table sync worker causing deadlock. In other words, apply_worker waits for tablesync worker which waits for backend, and backend waits for apply worker. This is not detected by the deadlock detector because apply worker uses WaitLatch. The fix is to release existing locks in apply worker before it starts to wait for tablesync worker to change the state. Reported-by: Tomas Vondra Author: Shlok Kyal Reviewed-by: Amit Kapila, Peter Smith Backpatch-through: 12 Discussion: https://postgr.es/m/d291bb50-12c4-e8af-2af2-7bb9bb4d8e3e@enterprisedb.com Branch ------ REL_12_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/e81e617f3273b74c8eb46f7a7c51387509bc8d8c Modified Files -------------- src/backend/replication/logical/tablesync.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2023-12-11 03:33 UTC | newest] Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2023-12-11 03:33 pgsql: Fix an undetected deadlock due to apply worker. Amit Kapila <akapila@postgresql.org> 2023-12-11 03:33 pgsql: Fix an undetected deadlock due to apply worker. Amit Kapila <akapila@postgresql.org> 2023-12-11 03:33 pgsql: Fix an undetected deadlock due to apply worker. Amit Kapila <akapila@postgresql.org> 2023-12-11 03:33 pgsql: Fix an undetected deadlock due to apply worker. Amit Kapila <akapila@postgresql.org> 2023-12-11 03:33 pgsql: Fix an undetected deadlock due to apply worker. Amit Kapila <akapila@postgresql.org> 2023-12-11 03:33 pgsql: Fix an undetected deadlock due to apply worker. Amit Kapila <akapila@postgresql.org>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox