pg.ddx.io pgsql-committers@postgresql.org mailing list archivehelp / color / mirror / Atom feed
pgsql: Fix partition pruning setup during DETACH CONCURRENTLY 5+ messages / 1 participants [nested] [flat]
* pgsql: Fix partition pruning setup during DETACH CONCURRENTLY @ 2024-06-24 13:58 Alvaro Herrera <alvherre@alvh.no-ip.org> 0 siblings, 0 replies; 5+ messages in thread From: Alvaro Herrera @ 2024-06-24 13:58 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix partition pruning setup during DETACH CONCURRENTLY When detaching partition in concurrent mode, it's possible for partition descriptors to not match the set that was recently seen when the plan was made, causing an assertion failure or (in production builds) failure to construct a working plan. The case that was reported involves prepared statements, but I think it may be possible to hit this bug without that too. The problem is that CreatePartitionPruneState is constructing a PartitionPruneState under the assumption that new partitions can be added, but never removed, but it turns out that this isn't true: a prepared statement gets replanned when the DETACH CONCURRENTLY session sends out its invalidation message, but if the invalidation message arrives after ExecInitAppend started, we would build a partition descriptor without the partition, and then CreatePartitionPruneState would refuse to work with it. CreatePartitionPruneState already contains code to deal with the new descriptor having more partitions than before (and behaving for the extra partitions as if they had been pruned), but doesn't have code to deal with less partitions than before, and it is naïve about the case where the number of partitions is the same. We could simply add that a new stanza for less partitions than before, and in simple testing it works to do that; but it's possible to press the test scripts even further and hit the case where one partition is added and a partition is removed quickly enough that we see the same number of partitions, but they don't actually match, causing hangs during execution. To cope with both these problems, we now memcmp() the arrays of partition OIDs, and do a more elaborate mapping (relying on the fact that both OID arrays are in partition-bounds order) if they're not identical. Backpatch to 14, where DETACH CONCURRENTLY appeared. Reported-by: yajun Hu <1026592243@qq.com> Reviewed-by: Tender Wang <tndrwang@gmail.com> Discussion: https://postgr.es/m/18377-e0324601cfebdfe5@postgresql.org Branch ------ REL_14_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/66e569f5022beb987c6335d5d9d27d7da98fbb75 Modified Files -------------- src/backend/executor/execPartition.c | 110 +++++++++++++++++++---------------- 1 file changed, 60 insertions(+), 50 deletions(-) ^ permalink raw reply [nested|flat] 5+ messages in thread
* pgsql: Fix partition pruning setup during DETACH CONCURRENTLY @ 2024-06-24 13:58 Alvaro Herrera <alvherre@alvh.no-ip.org> 0 siblings, 0 replies; 5+ messages in thread From: Alvaro Herrera @ 2024-06-24 13:58 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix partition pruning setup during DETACH CONCURRENTLY When detaching partition in concurrent mode, it's possible for partition descriptors to not match the set that was recently seen when the plan was made, causing an assertion failure or (in production builds) failure to construct a working plan. The case that was reported involves prepared statements, but I think it may be possible to hit this bug without that too. The problem is that CreatePartitionPruneState is constructing a PartitionPruneState under the assumption that new partitions can be added, but never removed, but it turns out that this isn't true: a prepared statement gets replanned when the DETACH CONCURRENTLY session sends out its invalidation message, but if the invalidation message arrives after ExecInitAppend started, we would build a partition descriptor without the partition, and then CreatePartitionPruneState would refuse to work with it. CreatePartitionPruneState already contains code to deal with the new descriptor having more partitions than before (and behaving for the extra partitions as if they had been pruned), but doesn't have code to deal with less partitions than before, and it is naïve about the case where the number of partitions is the same. We could simply add that a new stanza for less partitions than before, and in simple testing it works to do that; but it's possible to press the test scripts even further and hit the case where one partition is added and a partition is removed quickly enough that we see the same number of partitions, but they don't actually match, causing hangs during execution. To cope with both these problems, we now memcmp() the arrays of partition OIDs, and do a more elaborate mapping (relying on the fact that both OID arrays are in partition-bounds order) if they're not identical. Backpatch to 14, where DETACH CONCURRENTLY appeared. Reported-by: yajun Hu <1026592243@qq.com> Reviewed-by: Tender Wang <tndrwang@gmail.com> Discussion: https://postgr.es/m/18377-e0324601cfebdfe5@postgresql.org Branch ------ REL_16_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/96105ebfe21848c58e009c388da4ee7f513c0033 Modified Files -------------- src/backend/executor/execPartition.c | 110 +++++++++++++++++++---------------- 1 file changed, 60 insertions(+), 50 deletions(-) ^ permalink raw reply [nested|flat] 5+ messages in thread
* pgsql: Fix partition pruning setup during DETACH CONCURRENTLY @ 2024-06-24 13:58 Alvaro Herrera <alvherre@alvh.no-ip.org> 0 siblings, 0 replies; 5+ messages in thread From: Alvaro Herrera @ 2024-06-24 13:58 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix partition pruning setup during DETACH CONCURRENTLY When detaching partition in concurrent mode, it's possible for partition descriptors to not match the set that was recently seen when the plan was made, causing an assertion failure or (in production builds) failure to construct a working plan. The case that was reported involves prepared statements, but I think it may be possible to hit this bug without that too. The problem is that CreatePartitionPruneState is constructing a PartitionPruneState under the assumption that new partitions can be added, but never removed, but it turns out that this isn't true: a prepared statement gets replanned when the DETACH CONCURRENTLY session sends out its invalidation message, but if the invalidation message arrives after ExecInitAppend started, we would build a partition descriptor without the partition, and then CreatePartitionPruneState would refuse to work with it. CreatePartitionPruneState already contains code to deal with the new descriptor having more partitions than before (and behaving for the extra partitions as if they had been pruned), but doesn't have code to deal with less partitions than before, and it is naïve about the case where the number of partitions is the same. We could simply add that a new stanza for less partitions than before, and in simple testing it works to do that; but it's possible to press the test scripts even further and hit the case where one partition is added and a partition is removed quickly enough that we see the same number of partitions, but they don't actually match, causing hangs during execution. To cope with both these problems, we now memcmp() the arrays of partition OIDs, and do a more elaborate mapping (relying on the fact that both OID arrays are in partition-bounds order) if they're not identical. Backpatch to 14, where DETACH CONCURRENTLY appeared. Reported-by: yajun Hu <1026592243@qq.com> Reviewed-by: Tender Wang <tndrwang@gmail.com> Discussion: https://postgr.es/m/18377-e0324601cfebdfe5@postgresql.org Branch ------ REL_15_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/fb0fb0740fb6b7f51db6ad7cd40d0459ae1a8b98 Modified Files -------------- src/backend/executor/execPartition.c | 110 +++++++++++++++++++---------------- 1 file changed, 60 insertions(+), 50 deletions(-) ^ permalink raw reply [nested|flat] 5+ messages in thread
* pgsql: Fix partition pruning setup during DETACH CONCURRENTLY @ 2024-06-24 13:58 Alvaro Herrera <alvherre@alvh.no-ip.org> 0 siblings, 0 replies; 5+ messages in thread From: Alvaro Herrera @ 2024-06-24 13:58 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix partition pruning setup during DETACH CONCURRENTLY When detaching partition in concurrent mode, it's possible for partition descriptors to not match the set that was recently seen when the plan was made, causing an assertion failure or (in production builds) failure to construct a working plan. The case that was reported involves prepared statements, but I think it may be possible to hit this bug without that too. The problem is that CreatePartitionPruneState is constructing a PartitionPruneState under the assumption that new partitions can be added, but never removed, but it turns out that this isn't true: a prepared statement gets replanned when the DETACH CONCURRENTLY session sends out its invalidation message, but if the invalidation message arrives after ExecInitAppend started, we would build a partition descriptor without the partition, and then CreatePartitionPruneState would refuse to work with it. CreatePartitionPruneState already contains code to deal with the new descriptor having more partitions than before (and behaving for the extra partitions as if they had been pruned), but doesn't have code to deal with less partitions than before, and it is naïve about the case where the number of partitions is the same. We could simply add that a new stanza for less partitions than before, and in simple testing it works to do that; but it's possible to press the test scripts even further and hit the case where one partition is added and a partition is removed quickly enough that we see the same number of partitions, but they don't actually match, causing hangs during execution. To cope with both these problems, we now memcmp() the arrays of partition OIDs, and do a more elaborate mapping (relying on the fact that both OID arrays are in partition-bounds order) if they're not identical. Backpatch to 14, where DETACH CONCURRENTLY appeared. Reported-by: yajun Hu <1026592243@qq.com> Reviewed-by: Tender Wang <tndrwang@gmail.com> Discussion: https://postgr.es/m/18377-e0324601cfebdfe5@postgresql.org Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/27162a64b386a1f639a4f2b96c7cc3b1db192d92 Modified Files -------------- src/backend/executor/execPartition.c | 110 +++++++++++++++++++---------------- 1 file changed, 60 insertions(+), 50 deletions(-) ^ permalink raw reply [nested|flat] 5+ messages in thread
* pgsql: Fix partition pruning setup during DETACH CONCURRENTLY @ 2024-06-26 11:43 Alvaro Herrera <alvherre@alvh.no-ip.org> 0 siblings, 0 replies; 5+ messages in thread From: Alvaro Herrera @ 2024-06-26 11:43 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix partition pruning setup during DETACH CONCURRENTLY When detaching partition in concurrent mode, it's possible for partition descriptors to not match the set that was recently seen when the plan was made, causing an assertion failure or (in production builds) failure to construct a working plan. The case that was reported involves prepared statements, but I think it may be possible to hit this bug without that too. The problem is that CreatePartitionPruneState is constructing a PartitionPruneState under the assumption that new partitions can be added, but never removed, but it turns out that this isn't true: a prepared statement gets replanned when the DETACH CONCURRENTLY session sends out its invalidation message, but if the invalidation message arrives after ExecInitAppend started, we would build a partition descriptor without the partition, and then CreatePartitionPruneState would refuse to work with it. CreatePartitionPruneState already contains code to deal with the new descriptor having more partitions than before (and behaving for the extra partitions as if they had been pruned), but doesn't have code to deal with less partitions than before, and it is naïve about the case where the number of partitions is the same. We could simply add that a new stanza for less partitions than before, and in simple testing it works to do that; but it's possible to press the test scripts even further and hit the case where one partition is added and a partition is removed quickly enough that we see the same number of partitions, but they don't actually match, causing hangs during execution. To cope with both these problems, we now memcmp() the arrays of partition OIDs, and do a more elaborate mapping (relying on the fact that both OID arrays are in partition-bounds order) if they're not identical. This fix was already pushed in backbranches earlier. Reported-by: yajun Hu <1026592243@qq.com> Reviewed-by: Tender Wang <tndrwang@gmail.com> Discussion: https://postgr.es/m/18377-e0324601cfebdfe5@postgresql.org Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/dbca3469ebf8ac523f401dd0c2eaffd9df64078f Modified Files -------------- src/backend/executor/execPartition.c | 110 +++++++++++++++++++---------------- 1 file changed, 60 insertions(+), 50 deletions(-) ^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2024-06-26 11:43 UTC | newest] Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2024-06-24 13:58 pgsql: Fix partition pruning setup during DETACH CONCURRENTLY Alvaro Herrera <alvherre@alvh.no-ip.org> 2024-06-24 13:58 pgsql: Fix partition pruning setup during DETACH CONCURRENTLY Alvaro Herrera <alvherre@alvh.no-ip.org> 2024-06-24 13:58 pgsql: Fix partition pruning setup during DETACH CONCURRENTLY Alvaro Herrera <alvherre@alvh.no-ip.org> 2024-06-24 13:58 pgsql: Fix partition pruning setup during DETACH CONCURRENTLY Alvaro Herrera <alvherre@alvh.no-ip.org> 2024-06-26 11:43 pgsql: Fix partition pruning setup during DETACH CONCURRENTLY Alvaro Herrera <alvherre@alvh.no-ip.org>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox