pg.ddx.io pgsql-committers@postgresql.org mailing list archivehelp / color / mirror / Atom feed
pgsql: Fix race in Parallel Hash Join batch cleanup. 10+ messages / 1 participants [nested] [flat]
* pgsql: Fix race in Parallel Hash Join batch cleanup. @ 2021-03-17 05:14 Thomas Munro <tmunro@postgresql.org> 0 siblings, 0 replies; 10+ messages in thread From: Thomas Munro @ 2021-03-17 05:14 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix race in Parallel Hash Join batch cleanup. With very unlucky timing and parallel_leader_participation off, PHJ could attempt to access per-batch state just as it was being freed. There was code intended to prevent that by checking for a cleared pointer, but it was buggy. Fix, by introducing an extra barrier phase. The new phase PHJ_BUILD_RUNNING means that it's safe to access the per-batch state to find a batch to help with, and PHJ_BUILD_DONE means that it is too late. The last to detach will free the array of per-batch state as before, but now it will also atomically advance the phase at the same time, so that late attachers can avoid the hazard, without the data race. This mirrors the way per-batch hash tables are freed (see phases PHJ_BATCH_PROBING and PHJ_BATCH_DONE). Revealed by a one-off build farm failure, where BarrierAttach() failed a sanity check assertion, because the memory had been clobbered by dsa_free(). Back-patch to 11, where the code arrived. Reported-by: Michael Paquier <michael@paquier.xyz> Discussion: https://postgr.es/m/20200929061142.GA29096%40paquier.xyz Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/3b8981b6e1a2aea0f18384c803e21e9391de669a Modified Files -------------- src/backend/executor/nodeHash.c | 47 +++++++++++++++++++++++++------------ src/backend/executor/nodeHashjoin.c | 40 ++++++++++++++++++------------- src/include/executor/hashjoin.h | 3 ++- 3 files changed, 58 insertions(+), 32 deletions(-) ^ permalink raw reply [nested|flat] 10+ messages in thread
* pgsql: Fix race in Parallel Hash Join batch cleanup. @ 2021-03-17 05:15 Thomas Munro <tmunro@postgresql.org> 0 siblings, 0 replies; 10+ messages in thread From: Thomas Munro @ 2021-03-17 05:15 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix race in Parallel Hash Join batch cleanup. With very unlucky timing and parallel_leader_participation off, PHJ could attempt to access per-batch state just as it was being freed. There was code intended to prevent that by checking for a cleared pointer, but it was buggy. Fix, by introducing an extra barrier phase. The new phase PHJ_BUILD_RUNNING means that it's safe to access the per-batch state to find a batch to help with, and PHJ_BUILD_DONE means that it is too late. The last to detach will free the array of per-batch state as before, but now it will also atomically advance the phase at the same time, so that late attachers can avoid the hazard, without the data race. This mirrors the way per-batch hash tables are freed (see phases PHJ_BATCH_PROBING and PHJ_BATCH_DONE). Revealed by a one-off build farm failure, where BarrierAttach() failed a sanity check assertion, because the memory had been clobbered by dsa_free(). Back-patch to 11, where the code arrived. Reported-by: Michael Paquier <michael@paquier.xyz> Discussion: https://postgr.es/m/20200929061142.GA29096%40paquier.xyz Branch ------ REL_13_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/4e0f0995e923948631c4114ab353b256b51b58ad Modified Files -------------- src/backend/executor/nodeHash.c | 47 +++++++++++++++++++++++++------------ src/backend/executor/nodeHashjoin.c | 40 ++++++++++++++++++------------- src/include/executor/hashjoin.h | 3 ++- 3 files changed, 58 insertions(+), 32 deletions(-) ^ permalink raw reply [nested|flat] 10+ messages in thread
* pgsql: Fix race in Parallel Hash Join batch cleanup. @ 2021-03-17 05:15 Thomas Munro <tmunro@postgresql.org> 0 siblings, 0 replies; 10+ messages in thread From: Thomas Munro @ 2021-03-17 05:15 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix race in Parallel Hash Join batch cleanup. With very unlucky timing and parallel_leader_participation off, PHJ could attempt to access per-batch state just as it was being freed. There was code intended to prevent that by checking for a cleared pointer, but it was buggy. Fix, by introducing an extra barrier phase. The new phase PHJ_BUILD_RUNNING means that it's safe to access the per-batch state to find a batch to help with, and PHJ_BUILD_DONE means that it is too late. The last to detach will free the array of per-batch state as before, but now it will also atomically advance the phase at the same time, so that late attachers can avoid the hazard, without the data race. This mirrors the way per-batch hash tables are freed (see phases PHJ_BATCH_PROBING and PHJ_BATCH_DONE). Revealed by a one-off build farm failure, where BarrierAttach() failed a sanity check assertion, because the memory had been clobbered by dsa_free(). Back-patch to 11, where the code arrived. Reported-by: Michael Paquier <michael@paquier.xyz> Discussion: https://postgr.es/m/20200929061142.GA29096%40paquier.xyz Branch ------ REL_12_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/8fa2478b407ef867d501fafcdea45fd827f70799 Modified Files -------------- src/backend/executor/nodeHash.c | 47 +++++++++++++++++++++++++------------ src/backend/executor/nodeHashjoin.c | 40 ++++++++++++++++++------------- src/include/executor/hashjoin.h | 3 ++- 3 files changed, 58 insertions(+), 32 deletions(-) ^ permalink raw reply [nested|flat] 10+ messages in thread
* pgsql: Fix race in Parallel Hash Join batch cleanup. @ 2021-03-17 05:15 Thomas Munro <tmunro@postgresql.org> 0 siblings, 0 replies; 10+ messages in thread From: Thomas Munro @ 2021-03-17 05:15 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix race in Parallel Hash Join batch cleanup. With very unlucky timing and parallel_leader_participation off, PHJ could attempt to access per-batch state just as it was being freed. There was code intended to prevent that by checking for a cleared pointer, but it was buggy. Fix, by introducing an extra barrier phase. The new phase PHJ_BUILD_RUNNING means that it's safe to access the per-batch state to find a batch to help with, and PHJ_BUILD_DONE means that it is too late. The last to detach will free the array of per-batch state as before, but now it will also atomically advance the phase at the same time, so that late attachers can avoid the hazard, without the data race. This mirrors the way per-batch hash tables are freed (see phases PHJ_BATCH_PROBING and PHJ_BATCH_DONE). Revealed by a one-off build farm failure, where BarrierAttach() failed a sanity check assertion, because the memory had been clobbered by dsa_free(). Back-patch to 11, where the code arrived. Reported-by: Michael Paquier <michael@paquier.xyz> Discussion: https://postgr.es/m/20200929061142.GA29096%40paquier.xyz Branch ------ REL_11_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/0129c56fbe5c26bfec91bfc2c8a3b8818f441d6e Modified Files -------------- src/backend/executor/nodeHash.c | 47 +++++++++++++++++++++++++------------ src/backend/executor/nodeHashjoin.c | 40 ++++++++++++++++++------------- src/include/executor/hashjoin.h | 3 ++- 3 files changed, 58 insertions(+), 32 deletions(-) ^ permalink raw reply [nested|flat] 10+ messages in thread
* pgsql: Fix race in parallel hash join batch cleanup, take II. @ 2023-03-21 01:57 Thomas Munro <tmunro@postgresql.org> 0 siblings, 0 replies; 10+ messages in thread From: Thomas Munro @ 2023-03-21 01:57 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix race in parallel hash join batch cleanup, take II. With unlucky timing and parallel_leader_participation=off (not the default), PHJ could attempt to access per-batch shared state just as it was being freed. There was code intended to prevent that by checking for a cleared pointer, but it was racy. Fix, by introducing an extra barrier phase. The new phase PHJ_BUILD_RUNNING means that it's safe to access the per-batch state to find a batch to help with, and PHJ_BUILD_DONE means that it is too late. The last to detach will free the array of per-batch state as before, but now it will also atomically advance the phase, so that late attachers can avoid the hazard. This mirrors the way per-batch hash tables are freed (see phases PHJ_BATCH_PROBING and PHJ_BATCH_DONE). An earlier attempt to fix this (commit 3b8981b6, later reverted) missed one special case. When the inner side is empty (the "empty inner optimization), the build barrier would only make it to PHJ_BUILD_HASHING_INNER phase before workers attempted to detach from the hashtable. In that case, fast-forward the build barrier to PHJ_BUILD_RUNNING before proceeding, so that our later assertions hold and we can still negotiate who is cleaning up. Revealed by build farm failures, where BarrierAttach() failed a sanity check assertion, because the memory had been clobbered by dsa_free(). In non-assert builds, the result could be a segmentation fault. Back-patch to all supported releases. Author: Thomas Munro <thomas.munro@gmail.com> Author: Melanie Plageman <melanieplageman@gmail.com> Reported-by: Michael Paquier <michael@paquier.xyz> Reported-by: David Geier <geidav.pg@gmail.com> Tested-by: David Geier <geidav.pg@gmail.com> Discussion: https://postgr.es/m/20200929061142.GA29096%40paquier.xyz Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/8d578b9b2e37a4d9d6f422ced5126acec62365a7 Modified Files -------------- src/backend/executor/nodeHash.c | 50 +++++++++++++++++++++++----------- src/backend/executor/nodeHashjoin.c | 54 ++++++++++++++++++++++++++----------- src/include/executor/hashjoin.h | 3 ++- 3 files changed, 74 insertions(+), 33 deletions(-) ^ permalink raw reply [nested|flat] 10+ messages in thread
* pgsql: Fix race in parallel hash join batch cleanup, take II. @ 2023-03-21 01:58 Thomas Munro <tmunro@postgresql.org> 0 siblings, 0 replies; 10+ messages in thread From: Thomas Munro @ 2023-03-21 01:58 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix race in parallel hash join batch cleanup, take II. With unlucky timing and parallel_leader_participation=off (not the default), PHJ could attempt to access per-batch shared state just as it was being freed. There was code intended to prevent that by checking for a cleared pointer, but it was racy. Fix, by introducing an extra barrier phase. The new phase PHJ_BUILD_RUNNING means that it's safe to access the per-batch state to find a batch to help with, and PHJ_BUILD_DONE means that it is too late. The last to detach will free the array of per-batch state as before, but now it will also atomically advance the phase, so that late attachers can avoid the hazard. This mirrors the way per-batch hash tables are freed (see phases PHJ_BATCH_PROBING and PHJ_BATCH_DONE). An earlier attempt to fix this (commit 3b8981b6, later reverted) missed one special case. When the inner side is empty (the "empty inner optimization), the build barrier would only make it to PHJ_BUILD_HASHING_INNER phase before workers attempted to detach from the hashtable. In that case, fast-forward the build barrier to PHJ_BUILD_RUNNING before proceeding, so that our later assertions hold and we can still negotiate who is cleaning up. Revealed by build farm failures, where BarrierAttach() failed a sanity check assertion, because the memory had been clobbered by dsa_free(). In non-assert builds, the result could be a segmentation fault. Back-patch to all supported releases. Author: Thomas Munro <thomas.munro@gmail.com> Author: Melanie Plageman <melanieplageman@gmail.com> Reported-by: Michael Paquier <michael@paquier.xyz> Reported-by: David Geier <geidav.pg@gmail.com> Tested-by: David Geier <geidav.pg@gmail.com> Discussion: https://postgr.es/m/20200929061142.GA29096%40paquier.xyz Branch ------ REL_15_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/c03c6e8cf6a9118a3d1219ec0cb06b439db54100 Modified Files -------------- src/backend/executor/nodeHash.c | 50 +++++++++++++++++++++++----------- src/backend/executor/nodeHashjoin.c | 54 ++++++++++++++++++++++++++----------- src/include/executor/hashjoin.h | 3 ++- 3 files changed, 74 insertions(+), 33 deletions(-) ^ permalink raw reply [nested|flat] 10+ messages in thread
* pgsql: Fix race in parallel hash join batch cleanup, take II. @ 2023-03-21 01:58 Thomas Munro <tmunro@postgresql.org> 0 siblings, 0 replies; 10+ messages in thread From: Thomas Munro @ 2023-03-21 01:58 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix race in parallel hash join batch cleanup, take II. With unlucky timing and parallel_leader_participation=off (not the default), PHJ could attempt to access per-batch shared state just as it was being freed. There was code intended to prevent that by checking for a cleared pointer, but it was racy. Fix, by introducing an extra barrier phase. The new phase PHJ_BUILD_RUNNING means that it's safe to access the per-batch state to find a batch to help with, and PHJ_BUILD_DONE means that it is too late. The last to detach will free the array of per-batch state as before, but now it will also atomically advance the phase, so that late attachers can avoid the hazard. This mirrors the way per-batch hash tables are freed (see phases PHJ_BATCH_PROBING and PHJ_BATCH_DONE). An earlier attempt to fix this (commit 3b8981b6, later reverted) missed one special case. When the inner side is empty (the "empty inner optimization), the build barrier would only make it to PHJ_BUILD_HASHING_INNER phase before workers attempted to detach from the hashtable. In that case, fast-forward the build barrier to PHJ_BUILD_RUNNING before proceeding, so that our later assertions hold and we can still negotiate who is cleaning up. Revealed by build farm failures, where BarrierAttach() failed a sanity check assertion, because the memory had been clobbered by dsa_free(). In non-assert builds, the result could be a segmentation fault. Back-patch to all supported releases. Author: Thomas Munro <thomas.munro@gmail.com> Author: Melanie Plageman <melanieplageman@gmail.com> Reported-by: Michael Paquier <michael@paquier.xyz> Reported-by: David Geier <geidav.pg@gmail.com> Tested-by: David Geier <geidav.pg@gmail.com> Discussion: https://postgr.es/m/20200929061142.GA29096%40paquier.xyz Branch ------ REL_14_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/1b9e42e82ae62f2a53e4a3da19b5fff994af5d72 Modified Files -------------- src/backend/executor/nodeHash.c | 50 +++++++++++++++++++++++----------- src/backend/executor/nodeHashjoin.c | 54 ++++++++++++++++++++++++++----------- src/include/executor/hashjoin.h | 3 ++- 3 files changed, 74 insertions(+), 33 deletions(-) ^ permalink raw reply [nested|flat] 10+ messages in thread
* pgsql: Fix race in parallel hash join batch cleanup, take II. @ 2023-03-21 01:58 Thomas Munro <tmunro@postgresql.org> 0 siblings, 0 replies; 10+ messages in thread From: Thomas Munro @ 2023-03-21 01:58 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix race in parallel hash join batch cleanup, take II. With unlucky timing and parallel_leader_participation=off (not the default), PHJ could attempt to access per-batch shared state just as it was being freed. There was code intended to prevent that by checking for a cleared pointer, but it was racy. Fix, by introducing an extra barrier phase. The new phase PHJ_BUILD_RUNNING means that it's safe to access the per-batch state to find a batch to help with, and PHJ_BUILD_DONE means that it is too late. The last to detach will free the array of per-batch state as before, but now it will also atomically advance the phase, so that late attachers can avoid the hazard. This mirrors the way per-batch hash tables are freed (see phases PHJ_BATCH_PROBING and PHJ_BATCH_DONE). An earlier attempt to fix this (commit 3b8981b6, later reverted) missed one special case. When the inner side is empty (the "empty inner optimization), the build barrier would only make it to PHJ_BUILD_HASHING_INNER phase before workers attempted to detach from the hashtable. In that case, fast-forward the build barrier to PHJ_BUILD_RUNNING before proceeding, so that our later assertions hold and we can still negotiate who is cleaning up. Revealed by build farm failures, where BarrierAttach() failed a sanity check assertion, because the memory had been clobbered by dsa_free(). In non-assert builds, the result could be a segmentation fault. Back-patch to all supported releases. Author: Thomas Munro <thomas.munro@gmail.com> Author: Melanie Plageman <melanieplageman@gmail.com> Reported-by: Michael Paquier <michael@paquier.xyz> Reported-by: David Geier <geidav.pg@gmail.com> Tested-by: David Geier <geidav.pg@gmail.com> Discussion: https://postgr.es/m/20200929061142.GA29096%40paquier.xyz Branch ------ REL_13_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/6e94d62e3f04b03c95bd707e43ba32147d35908e Modified Files -------------- src/backend/executor/nodeHash.c | 50 +++++++++++++++++++++++----------- src/backend/executor/nodeHashjoin.c | 54 ++++++++++++++++++++++++++----------- src/include/executor/hashjoin.h | 3 ++- 3 files changed, 74 insertions(+), 33 deletions(-) ^ permalink raw reply [nested|flat] 10+ messages in thread
* pgsql: Fix race in parallel hash join batch cleanup, take II. @ 2023-03-21 01:58 Thomas Munro <tmunro@postgresql.org> 0 siblings, 0 replies; 10+ messages in thread From: Thomas Munro @ 2023-03-21 01:58 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix race in parallel hash join batch cleanup, take II. With unlucky timing and parallel_leader_participation=off (not the default), PHJ could attempt to access per-batch shared state just as it was being freed. There was code intended to prevent that by checking for a cleared pointer, but it was racy. Fix, by introducing an extra barrier phase. The new phase PHJ_BUILD_RUNNING means that it's safe to access the per-batch state to find a batch to help with, and PHJ_BUILD_DONE means that it is too late. The last to detach will free the array of per-batch state as before, but now it will also atomically advance the phase, so that late attachers can avoid the hazard. This mirrors the way per-batch hash tables are freed (see phases PHJ_BATCH_PROBING and PHJ_BATCH_DONE). An earlier attempt to fix this (commit 3b8981b6, later reverted) missed one special case. When the inner side is empty (the "empty inner optimization), the build barrier would only make it to PHJ_BUILD_HASHING_INNER phase before workers attempted to detach from the hashtable. In that case, fast-forward the build barrier to PHJ_BUILD_RUNNING before proceeding, so that our later assertions hold and we can still negotiate who is cleaning up. Revealed by build farm failures, where BarrierAttach() failed a sanity check assertion, because the memory had been clobbered by dsa_free(). In non-assert builds, the result could be a segmentation fault. Back-patch to all supported releases. Author: Thomas Munro <thomas.munro@gmail.com> Author: Melanie Plageman <melanieplageman@gmail.com> Reported-by: Michael Paquier <michael@paquier.xyz> Reported-by: David Geier <geidav.pg@gmail.com> Tested-by: David Geier <geidav.pg@gmail.com> Discussion: https://postgr.es/m/20200929061142.GA29096%40paquier.xyz Branch ------ REL_12_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/44d44aa9712f315b125c9c5f2a1640ebb70e1d2f Modified Files -------------- src/backend/executor/nodeHash.c | 50 +++++++++++++++++++++++----------- src/backend/executor/nodeHashjoin.c | 54 ++++++++++++++++++++++++++----------- src/include/executor/hashjoin.h | 3 ++- 3 files changed, 74 insertions(+), 33 deletions(-) ^ permalink raw reply [nested|flat] 10+ messages in thread
* pgsql: Fix race in parallel hash join batch cleanup, take II. @ 2023-03-21 01:59 Thomas Munro <tmunro@postgresql.org> 0 siblings, 0 replies; 10+ messages in thread From: Thomas Munro @ 2023-03-21 01:59 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix race in parallel hash join batch cleanup, take II. With unlucky timing and parallel_leader_participation=off (not the default), PHJ could attempt to access per-batch shared state just as it was being freed. There was code intended to prevent that by checking for a cleared pointer, but it was racy. Fix, by introducing an extra barrier phase. The new phase PHJ_BUILD_RUNNING means that it's safe to access the per-batch state to find a batch to help with, and PHJ_BUILD_DONE means that it is too late. The last to detach will free the array of per-batch state as before, but now it will also atomically advance the phase, so that late attachers can avoid the hazard. This mirrors the way per-batch hash tables are freed (see phases PHJ_BATCH_PROBING and PHJ_BATCH_DONE). An earlier attempt to fix this (commit 3b8981b6, later reverted) missed one special case. When the inner side is empty (the "empty inner optimization), the build barrier would only make it to PHJ_BUILD_HASHING_INNER phase before workers attempted to detach from the hashtable. In that case, fast-forward the build barrier to PHJ_BUILD_RUNNING before proceeding, so that our later assertions hold and we can still negotiate who is cleaning up. Revealed by build farm failures, where BarrierAttach() failed a sanity check assertion, because the memory had been clobbered by dsa_free(). In non-assert builds, the result could be a segmentation fault. Back-patch to all supported releases. Author: Thomas Munro <thomas.munro@gmail.com> Author: Melanie Plageman <melanieplageman@gmail.com> Reported-by: Michael Paquier <michael@paquier.xyz> Reported-by: David Geier <geidav.pg@gmail.com> Tested-by: David Geier <geidav.pg@gmail.com> Discussion: https://postgr.es/m/20200929061142.GA29096%40paquier.xyz Branch ------ REL_11_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/ef16d27245f6167acd878bea49cdde0a8dcec49c Modified Files -------------- src/backend/executor/nodeHash.c | 50 +++++++++++++++++++++++----------- src/backend/executor/nodeHashjoin.c | 54 ++++++++++++++++++++++++++----------- src/include/executor/hashjoin.h | 3 ++- 3 files changed, 74 insertions(+), 33 deletions(-) ^ permalink raw reply [nested|flat] 10+ messages in thread
end of thread, other threads:[~2023-03-21 01:59 UTC | newest] Thread overview: 10+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2021-03-17 05:14 pgsql: Fix race in Parallel Hash Join batch cleanup. Thomas Munro <tmunro@postgresql.org> 2021-03-17 05:15 pgsql: Fix race in Parallel Hash Join batch cleanup. Thomas Munro <tmunro@postgresql.org> 2021-03-17 05:15 pgsql: Fix race in Parallel Hash Join batch cleanup. Thomas Munro <tmunro@postgresql.org> 2021-03-17 05:15 pgsql: Fix race in Parallel Hash Join batch cleanup. Thomas Munro <tmunro@postgresql.org> 2023-03-21 01:57 pgsql: Fix race in parallel hash join batch cleanup, take II. Thomas Munro <tmunro@postgresql.org> 2023-03-21 01:58 pgsql: Fix race in parallel hash join batch cleanup, take II. Thomas Munro <tmunro@postgresql.org> 2023-03-21 01:58 pgsql: Fix race in parallel hash join batch cleanup, take II. Thomas Munro <tmunro@postgresql.org> 2023-03-21 01:58 pgsql: Fix race in parallel hash join batch cleanup, take II. Thomas Munro <tmunro@postgresql.org> 2023-03-21 01:58 pgsql: Fix race in parallel hash join batch cleanup, take II. Thomas Munro <tmunro@postgresql.org> 2023-03-21 01:59 pgsql: Fix race in parallel hash join batch cleanup, take II. Thomas Munro <tmunro@postgresql.org>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox