agora inbox for [email protected]  

help / color / mirror / Atom feed

[PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
107+ messages / 5 participants
[nested] [flat]

* [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  5 +++--
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 19 insertions(+), 15 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6d357565a8 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -147,8 +147,9 @@ The following infomask bits are applicable:
 
 - HEAP_KEYS_UPDATED
   This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  this tuple and changed the key values, or it deleted the tuple.  It can also
+  be set in combination of HEAP_XMAX_LOCK_ONLY.  It's set regardless of whether
+  the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--+HP7ph2BbKc20aGI--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  5 +++--
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 19 insertions(+), 15 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6d357565a8 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -147,8 +147,9 @@ The following infomask bits are applicable:
 
 - HEAP_KEYS_UPDATED
   This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  this tuple and changed the key values, or it deleted the tuple.  It can also
+  be set in combination of HEAP_XMAX_LOCK_ONLY.  It's set regardless of whether
+  the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--+HP7ph2BbKc20aGI--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  7 ++++---
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6441e8baf0 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -146,9 +146,10 @@ The following infomask bits are applicable:
   FOR UPDATE; this is implemented by the HEAP_KEYS_UPDATED bit.
 
 - HEAP_KEYS_UPDATED
-  This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  This bit lives in t_infomask2.  If set, indicates that the operation(s) done
+  by the XMAX compromise the tuple key, such as a SELECT FOR UPDATE, an UPDATE
+  that modifies the columns of the key, or a DELETE.  It's set regardless of
+  whether the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--0ntfKIWw70PvrIHh--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  7 ++++---
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6441e8baf0 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -146,9 +146,10 @@ The following infomask bits are applicable:
   FOR UPDATE; this is implemented by the HEAP_KEYS_UPDATED bit.
 
 - HEAP_KEYS_UPDATED
-  This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  This bit lives in t_infomask2.  If set, indicates that the operation(s) done
+  by the XMAX compromise the tuple key, such as a SELECT FOR UPDATE, an UPDATE
+  that modifies the columns of the key, or a DELETE.  It's set regardless of
+  whether the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--0ntfKIWw70PvrIHh--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  7 ++++---
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6441e8baf0 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -146,9 +146,10 @@ The following infomask bits are applicable:
   FOR UPDATE; this is implemented by the HEAP_KEYS_UPDATED bit.
 
 - HEAP_KEYS_UPDATED
-  This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  This bit lives in t_infomask2.  If set, indicates that the operation(s) done
+  by the XMAX compromise the tuple key, such as a SELECT FOR UPDATE, an UPDATE
+  that modifies the columns of the key, or a DELETE.  It's set regardless of
+  whether the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--0ntfKIWw70PvrIHh--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  7 ++++---
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6441e8baf0 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -146,9 +146,10 @@ The following infomask bits are applicable:
   FOR UPDATE; this is implemented by the HEAP_KEYS_UPDATED bit.
 
 - HEAP_KEYS_UPDATED
-  This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  This bit lives in t_infomask2.  If set, indicates that the operation(s) done
+  by the XMAX compromise the tuple key, such as a SELECT FOR UPDATE, an UPDATE
+  that modifies the columns of the key, or a DELETE.  It's set regardless of
+  whether the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--0ntfKIWw70PvrIHh--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  7 ++++---
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6441e8baf0 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -146,9 +146,10 @@ The following infomask bits are applicable:
   FOR UPDATE; this is implemented by the HEAP_KEYS_UPDATED bit.
 
 - HEAP_KEYS_UPDATED
-  This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  This bit lives in t_infomask2.  If set, indicates that the operation(s) done
+  by the XMAX compromise the tuple key, such as a SELECT FOR UPDATE, an UPDATE
+  that modifies the columns of the key, or a DELETE.  It's set regardless of
+  whether the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--0ntfKIWw70PvrIHh--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  7 ++++---
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6441e8baf0 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -146,9 +146,10 @@ The following infomask bits are applicable:
   FOR UPDATE; this is implemented by the HEAP_KEYS_UPDATED bit.
 
 - HEAP_KEYS_UPDATED
-  This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  This bit lives in t_infomask2.  If set, indicates that the operation(s) done
+  by the XMAX compromise the tuple key, such as a SELECT FOR UPDATE, an UPDATE
+  that modifies the columns of the key, or a DELETE.  It's set regardless of
+  whether the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--0ntfKIWw70PvrIHh--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  5 +++--
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 19 insertions(+), 15 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6d357565a8 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -147,8 +147,9 @@ The following infomask bits are applicable:
 
 - HEAP_KEYS_UPDATED
   This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  this tuple and changed the key values, or it deleted the tuple.  It can also
+  be set in combination of HEAP_XMAX_LOCK_ONLY.  It's set regardless of whether
+  the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--+HP7ph2BbKc20aGI--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/hio.c          |  8 +++-----
 3 files changed, 16 insertions(+), 13 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--CE+1k2dSO48ffgeK--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  7 ++++---
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6441e8baf0 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -146,9 +146,10 @@ The following infomask bits are applicable:
   FOR UPDATE; this is implemented by the HEAP_KEYS_UPDATED bit.
 
 - HEAP_KEYS_UPDATED
-  This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  This bit lives in t_infomask2.  If set, indicates that the operation(s) done
+  by the XMAX compromise the tuple key, such as a SELECT FOR UPDATE, an UPDATE
+  that modifies the columns of the key, or a DELETE.  It's set regardless of
+  whether the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--0ntfKIWw70PvrIHh--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/hio.c          |  8 +++-----
 3 files changed, 16 insertions(+), 13 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--CE+1k2dSO48ffgeK--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  5 +++--
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 19 insertions(+), 15 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6d357565a8 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -147,8 +147,9 @@ The following infomask bits are applicable:
 
 - HEAP_KEYS_UPDATED
   This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  this tuple and changed the key values, or it deleted the tuple.  It can also
+  be set in combination of HEAP_XMAX_LOCK_ONLY.  It's set regardless of whether
+  the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--+HP7ph2BbKc20aGI--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  5 +++--
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 19 insertions(+), 15 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6d357565a8 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -147,8 +147,9 @@ The following infomask bits are applicable:
 
 - HEAP_KEYS_UPDATED
   This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  this tuple and changed the key values, or it deleted the tuple.  It can also
+  be set in combination of HEAP_XMAX_LOCK_ONLY.  It's set regardless of whether
+  the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--+HP7ph2BbKc20aGI--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  5 +++--
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 19 insertions(+), 15 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6d357565a8 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -147,8 +147,9 @@ The following infomask bits are applicable:
 
 - HEAP_KEYS_UPDATED
   This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  this tuple and changed the key values, or it deleted the tuple.  It can also
+  be set in combination of HEAP_XMAX_LOCK_ONLY.  It's set regardless of whether
+  the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--+HP7ph2BbKc20aGI--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  5 +++--
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 19 insertions(+), 15 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6d357565a8 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -147,8 +147,9 @@ The following infomask bits are applicable:
 
 - HEAP_KEYS_UPDATED
   This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  this tuple and changed the key values, or it deleted the tuple.  It can also
+  be set in combination of HEAP_XMAX_LOCK_ONLY.  It's set regardless of whether
+  the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--+HP7ph2BbKc20aGI--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  7 ++++---
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6441e8baf0 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -146,9 +146,10 @@ The following infomask bits are applicable:
   FOR UPDATE; this is implemented by the HEAP_KEYS_UPDATED bit.
 
 - HEAP_KEYS_UPDATED
-  This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  This bit lives in t_infomask2.  If set, indicates that the operation(s) done
+  by the XMAX compromise the tuple key, such as a SELECT FOR UPDATE, an UPDATE
+  that modifies the columns of the key, or a DELETE.  It's set regardless of
+  whether the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--0ntfKIWw70PvrIHh--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  7 ++++---
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6441e8baf0 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -146,9 +146,10 @@ The following infomask bits are applicable:
   FOR UPDATE; this is implemented by the HEAP_KEYS_UPDATED bit.
 
 - HEAP_KEYS_UPDATED
-  This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  This bit lives in t_infomask2.  If set, indicates that the operation(s) done
+  by the XMAX compromise the tuple key, such as a SELECT FOR UPDATE, an UPDATE
+  that modifies the columns of the key, or a DELETE.  It's set regardless of
+  whether the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--0ntfKIWw70PvrIHh--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  7 ++++---
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6441e8baf0 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -146,9 +146,10 @@ The following infomask bits are applicable:
   FOR UPDATE; this is implemented by the HEAP_KEYS_UPDATED bit.
 
 - HEAP_KEYS_UPDATED
-  This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  This bit lives in t_infomask2.  If set, indicates that the operation(s) done
+  by the XMAX compromise the tuple key, such as a SELECT FOR UPDATE, an UPDATE
+  that modifies the columns of the key, or a DELETE.  It's set regardless of
+  whether the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--0ntfKIWw70PvrIHh--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  5 +++--
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 19 insertions(+), 15 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6d357565a8 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -147,8 +147,9 @@ The following infomask bits are applicable:
 
 - HEAP_KEYS_UPDATED
   This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  this tuple and changed the key values, or it deleted the tuple.  It can also
+  be set in combination of HEAP_XMAX_LOCK_ONLY.  It's set regardless of whether
+  the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--+HP7ph2BbKc20aGI--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  7 ++++---
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6441e8baf0 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -146,9 +146,10 @@ The following infomask bits are applicable:
   FOR UPDATE; this is implemented by the HEAP_KEYS_UPDATED bit.
 
 - HEAP_KEYS_UPDATED
-  This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  This bit lives in t_infomask2.  If set, indicates that the operation(s) done
+  by the XMAX compromise the tuple key, such as a SELECT FOR UPDATE, an UPDATE
+  that modifies the columns of the key, or a DELETE.  It's set regardless of
+  whether the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--0ntfKIWw70PvrIHh--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  5 +++--
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 19 insertions(+), 15 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6d357565a8 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -147,8 +147,9 @@ The following infomask bits are applicable:
 
 - HEAP_KEYS_UPDATED
   This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  this tuple and changed the key values, or it deleted the tuple.  It can also
+  be set in combination of HEAP_XMAX_LOCK_ONLY.  It's set regardless of whether
+  the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--+HP7ph2BbKc20aGI--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  7 ++++---
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6441e8baf0 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -146,9 +146,10 @@ The following infomask bits are applicable:
   FOR UPDATE; this is implemented by the HEAP_KEYS_UPDATED bit.
 
 - HEAP_KEYS_UPDATED
-  This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  This bit lives in t_infomask2.  If set, indicates that the operation(s) done
+  by the XMAX compromise the tuple key, such as a SELECT FOR UPDATE, an UPDATE
+  that modifies the columns of the key, or a DELETE.  It's set regardless of
+  whether the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--0ntfKIWw70PvrIHh--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  5 +++--
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 19 insertions(+), 15 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6d357565a8 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -147,8 +147,9 @@ The following infomask bits are applicable:
 
 - HEAP_KEYS_UPDATED
   This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  this tuple and changed the key values, or it deleted the tuple.  It can also
+  be set in combination of HEAP_XMAX_LOCK_ONLY.  It's set regardless of whether
+  the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--+HP7ph2BbKc20aGI--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  7 ++++---
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6441e8baf0 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -146,9 +146,10 @@ The following infomask bits are applicable:
   FOR UPDATE; this is implemented by the HEAP_KEYS_UPDATED bit.
 
 - HEAP_KEYS_UPDATED
-  This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  This bit lives in t_infomask2.  If set, indicates that the operation(s) done
+  by the XMAX compromise the tuple key, such as a SELECT FOR UPDATE, an UPDATE
+  that modifies the columns of the key, or a DELETE.  It's set regardless of
+  whether the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--0ntfKIWw70PvrIHh--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/hio.c          |  8 +++-----
 3 files changed, 16 insertions(+), 13 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--CE+1k2dSO48ffgeK--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/hio.c          |  8 +++-----
 3 files changed, 16 insertions(+), 13 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--CE+1k2dSO48ffgeK--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/hio.c          |  8 +++-----
 3 files changed, 16 insertions(+), 13 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--CE+1k2dSO48ffgeK--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/hio.c          |  8 +++-----
 3 files changed, 16 insertions(+), 13 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--CE+1k2dSO48ffgeK--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  7 ++++---
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6441e8baf0 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -146,9 +146,10 @@ The following infomask bits are applicable:
   FOR UPDATE; this is implemented by the HEAP_KEYS_UPDATED bit.
 
 - HEAP_KEYS_UPDATED
-  This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  This bit lives in t_infomask2.  If set, indicates that the operation(s) done
+  by the XMAX compromise the tuple key, such as a SELECT FOR UPDATE, an UPDATE
+  that modifies the columns of the key, or a DELETE.  It's set regardless of
+  whether the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--0ntfKIWw70PvrIHh--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  7 ++++---
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6441e8baf0 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -146,9 +146,10 @@ The following infomask bits are applicable:
   FOR UPDATE; this is implemented by the HEAP_KEYS_UPDATED bit.
 
 - HEAP_KEYS_UPDATED
-  This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  This bit lives in t_infomask2.  If set, indicates that the operation(s) done
+  by the XMAX compromise the tuple key, such as a SELECT FOR UPDATE, an UPDATE
+  that modifies the columns of the key, or a DELETE.  It's set regardless of
+  whether the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--0ntfKIWw70PvrIHh--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  5 +++--
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 19 insertions(+), 15 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6d357565a8 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -147,8 +147,9 @@ The following infomask bits are applicable:
 
 - HEAP_KEYS_UPDATED
   This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  this tuple and changed the key values, or it deleted the tuple.  It can also
+  be set in combination of HEAP_XMAX_LOCK_ONLY.  It's set regardless of whether
+  the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--+HP7ph2BbKc20aGI--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  7 ++++---
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6441e8baf0 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -146,9 +146,10 @@ The following infomask bits are applicable:
   FOR UPDATE; this is implemented by the HEAP_KEYS_UPDATED bit.
 
 - HEAP_KEYS_UPDATED
-  This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  This bit lives in t_infomask2.  If set, indicates that the operation(s) done
+  by the XMAX compromise the tuple key, such as a SELECT FOR UPDATE, an UPDATE
+  that modifies the columns of the key, or a DELETE.  It's set regardless of
+  whether the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--0ntfKIWw70PvrIHh--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/hio.c          |  8 +++-----
 3 files changed, 16 insertions(+), 13 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--CE+1k2dSO48ffgeK--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/hio.c          |  8 +++-----
 3 files changed, 16 insertions(+), 13 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--CE+1k2dSO48ffgeK--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  7 ++++---
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6441e8baf0 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -146,9 +146,10 @@ The following infomask bits are applicable:
   FOR UPDATE; this is implemented by the HEAP_KEYS_UPDATED bit.
 
 - HEAP_KEYS_UPDATED
-  This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  This bit lives in t_infomask2.  If set, indicates that the operation(s) done
+  by the XMAX compromise the tuple key, such as a SELECT FOR UPDATE, an UPDATE
+  that modifies the columns of the key, or a DELETE.  It's set regardless of
+  whether the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--0ntfKIWw70PvrIHh--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/hio.c          |  8 +++-----
 3 files changed, 16 insertions(+), 13 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--CE+1k2dSO48ffgeK--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/hio.c          |  8 +++-----
 3 files changed, 16 insertions(+), 13 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--CE+1k2dSO48ffgeK--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  7 ++++---
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6441e8baf0 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -146,9 +146,10 @@ The following infomask bits are applicable:
   FOR UPDATE; this is implemented by the HEAP_KEYS_UPDATED bit.
 
 - HEAP_KEYS_UPDATED
-  This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  This bit lives in t_infomask2.  If set, indicates that the operation(s) done
+  by the XMAX compromise the tuple key, such as a SELECT FOR UPDATE, an UPDATE
+  that modifies the columns of the key, or a DELETE.  It's set regardless of
+  whether the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--0ntfKIWw70PvrIHh--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/hio.c          |  8 +++-----
 3 files changed, 16 insertions(+), 13 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--CE+1k2dSO48ffgeK--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/hio.c          |  8 +++-----
 3 files changed, 16 insertions(+), 13 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--CE+1k2dSO48ffgeK--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/hio.c          |  8 +++-----
 3 files changed, 16 insertions(+), 13 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--CE+1k2dSO48ffgeK--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  7 ++++---
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6441e8baf0 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -146,9 +146,10 @@ The following infomask bits are applicable:
   FOR UPDATE; this is implemented by the HEAP_KEYS_UPDATED bit.
 
 - HEAP_KEYS_UPDATED
-  This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  This bit lives in t_infomask2.  If set, indicates that the operation(s) done
+  by the XMAX compromise the tuple key, such as a SELECT FOR UPDATE, an UPDATE
+  that modifies the columns of the key, or a DELETE.  It's set regardless of
+  whether the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--0ntfKIWw70PvrIHh--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/hio.c          |  8 +++-----
 3 files changed, 16 insertions(+), 13 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--CE+1k2dSO48ffgeK--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  7 ++++---
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6441e8baf0 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -146,9 +146,10 @@ The following infomask bits are applicable:
   FOR UPDATE; this is implemented by the HEAP_KEYS_UPDATED bit.
 
 - HEAP_KEYS_UPDATED
-  This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  This bit lives in t_infomask2.  If set, indicates that the operation(s) done
+  by the XMAX compromise the tuple key, such as a SELECT FOR UPDATE, an UPDATE
+  that modifies the columns of the key, or a DELETE.  It's set regardless of
+  whether the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--0ntfKIWw70PvrIHh--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  5 +++--
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 19 insertions(+), 15 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6d357565a8 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -147,8 +147,9 @@ The following infomask bits are applicable:
 
 - HEAP_KEYS_UPDATED
   This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  this tuple and changed the key values, or it deleted the tuple.  It can also
+  be set in combination of HEAP_XMAX_LOCK_ONLY.  It's set regardless of whether
+  the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--+HP7ph2BbKc20aGI--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/hio.c          |  8 +++-----
 3 files changed, 16 insertions(+), 13 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--CE+1k2dSO48ffgeK--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/hio.c          |  8 +++-----
 3 files changed, 16 insertions(+), 13 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--CE+1k2dSO48ffgeK--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  5 +++--
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 19 insertions(+), 15 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6d357565a8 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -147,8 +147,9 @@ The following infomask bits are applicable:
 
 - HEAP_KEYS_UPDATED
   This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  this tuple and changed the key values, or it deleted the tuple.  It can also
+  be set in combination of HEAP_XMAX_LOCK_ONLY.  It's set regardless of whether
+  the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--+HP7ph2BbKc20aGI--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  5 +++--
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 19 insertions(+), 15 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6d357565a8 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -147,8 +147,9 @@ The following infomask bits are applicable:
 
 - HEAP_KEYS_UPDATED
   This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  this tuple and changed the key values, or it deleted the tuple.  It can also
+  be set in combination of HEAP_XMAX_LOCK_ONLY.  It's set regardless of whether
+  the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--+HP7ph2BbKc20aGI--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  5 +++--
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 19 insertions(+), 15 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6d357565a8 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -147,8 +147,9 @@ The following infomask bits are applicable:
 
 - HEAP_KEYS_UPDATED
   This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  this tuple and changed the key values, or it deleted the tuple.  It can also
+  be set in combination of HEAP_XMAX_LOCK_ONLY.  It's set regardless of whether
+  the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--+HP7ph2BbKc20aGI--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/hio.c          |  8 +++-----
 3 files changed, 16 insertions(+), 13 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--CE+1k2dSO48ffgeK--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  5 +++--
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 19 insertions(+), 15 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6d357565a8 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -147,8 +147,9 @@ The following infomask bits are applicable:
 
 - HEAP_KEYS_UPDATED
   This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  this tuple and changed the key values, or it deleted the tuple.  It can also
+  be set in combination of HEAP_XMAX_LOCK_ONLY.  It's set regardless of whether
+  the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--+HP7ph2BbKc20aGI--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/hio.c          |  8 +++-----
 3 files changed, 16 insertions(+), 13 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--CE+1k2dSO48ffgeK--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  5 +++--
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 19 insertions(+), 15 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6d357565a8 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -147,8 +147,9 @@ The following infomask bits are applicable:
 
 - HEAP_KEYS_UPDATED
   This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  this tuple and changed the key values, or it deleted the tuple.  It can also
+  be set in combination of HEAP_XMAX_LOCK_ONLY.  It's set regardless of whether
+  the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--+HP7ph2BbKc20aGI--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  5 +++--
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 19 insertions(+), 15 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6d357565a8 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -147,8 +147,9 @@ The following infomask bits are applicable:
 
 - HEAP_KEYS_UPDATED
   This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  this tuple and changed the key values, or it deleted the tuple.  It can also
+  be set in combination of HEAP_XMAX_LOCK_ONLY.  It's set regardless of whether
+  the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--+HP7ph2BbKc20aGI--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  5 +++--
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 19 insertions(+), 15 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6d357565a8 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -147,8 +147,9 @@ The following infomask bits are applicable:
 
 - HEAP_KEYS_UPDATED
   This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  this tuple and changed the key values, or it deleted the tuple.  It can also
+  be set in combination of HEAP_XMAX_LOCK_ONLY.  It's set regardless of whether
+  the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--+HP7ph2BbKc20aGI--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  5 +++--
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 19 insertions(+), 15 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6d357565a8 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -147,8 +147,9 @@ The following infomask bits are applicable:
 
 - HEAP_KEYS_UPDATED
   This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  this tuple and changed the key values, or it deleted the tuple.  It can also
+  be set in combination of HEAP_XMAX_LOCK_ONLY.  It's set regardless of whether
+  the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--+HP7ph2BbKc20aGI--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/hio.c          |  8 +++-----
 3 files changed, 16 insertions(+), 13 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--CE+1k2dSO48ffgeK--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  5 +++--
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 19 insertions(+), 15 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6d357565a8 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -147,8 +147,9 @@ The following infomask bits are applicable:
 
 - HEAP_KEYS_UPDATED
   This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  this tuple and changed the key values, or it deleted the tuple.  It can also
+  be set in combination of HEAP_XMAX_LOCK_ONLY.  It's set regardless of whether
+  the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--+HP7ph2BbKc20aGI--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  7 ++++---
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6441e8baf0 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -146,9 +146,10 @@ The following infomask bits are applicable:
   FOR UPDATE; this is implemented by the HEAP_KEYS_UPDATED bit.
 
 - HEAP_KEYS_UPDATED
-  This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  This bit lives in t_infomask2.  If set, indicates that the operation(s) done
+  by the XMAX compromise the tuple key, such as a SELECT FOR UPDATE, an UPDATE
+  that modifies the columns of the key, or a DELETE.  It's set regardless of
+  whether the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--0ntfKIWw70PvrIHh--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/hio.c          |  8 +++-----
 3 files changed, 16 insertions(+), 13 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--CE+1k2dSO48ffgeK--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/hio.c          |  8 +++-----
 3 files changed, 16 insertions(+), 13 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--CE+1k2dSO48ffgeK--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  5 +++--
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 19 insertions(+), 15 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6d357565a8 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -147,8 +147,9 @@ The following infomask bits are applicable:
 
 - HEAP_KEYS_UPDATED
   This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  this tuple and changed the key values, or it deleted the tuple.  It can also
+  be set in combination of HEAP_XMAX_LOCK_ONLY.  It's set regardless of whether
+  the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--+HP7ph2BbKc20aGI--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/README.tuplock |  7 ++++---
 src/backend/access/heap/hio.c          |  8 +++-----
 4 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/README.tuplock b/src/backend/access/heap/README.tuplock
index d03ddf6cdc..6441e8baf0 100644
--- a/src/backend/access/heap/README.tuplock
+++ b/src/backend/access/heap/README.tuplock
@@ -146,9 +146,10 @@ The following infomask bits are applicable:
   FOR UPDATE; this is implemented by the HEAP_KEYS_UPDATED bit.
 
 - HEAP_KEYS_UPDATED
-  This bit lives in t_infomask2.  If set, indicates that the XMAX updated
-  this tuple and changed the key values, or it deleted the tuple.
-  It's set regardless of whether the XMAX is a TransactionId or a MultiXactId.
+  This bit lives in t_infomask2.  If set, indicates that the operation(s) done
+  by the XMAX compromise the tuple key, such as a SELECT FOR UPDATE, an UPDATE
+  that modifies the columns of the key, or a DELETE.  It's set regardless of
+  whether the XMAX is a TransactionId or a MultiXactId.
 
 We currently never set the HEAP_XMAX_COMMITTED when the HEAP_XMAX_IS_MULTI bit
 is set.
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--0ntfKIWw70PvrIHh--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination.
@ 2021-01-21 16:06 Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Julien Rouhaud @ 2021-01-21 16:06 UTC (permalink / raw)

This combination of hint bits was previously detected as a form of corruption,
but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE
queries.

Discussion: https://postgr.es/m/20210124061758.GA11756@nol
---
 contrib/amcheck/t/001_verify_heapam.pl | 14 +++++++++++++-
 contrib/amcheck/verify_heapam.c        |  7 -------
 src/backend/access/heap/hio.c          |  8 +++-----
 3 files changed, 16 insertions(+), 13 deletions(-)

diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.pl
index a2f65b826d..6050feb712 100644
--- a/contrib/amcheck/t/001_verify_heapam.pl
+++ b/contrib/amcheck/t/001_verify_heapam.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use TestLib;
 
-use Test::More tests => 79;
+use Test::More tests => 80;
 
 my ($node, $result);
 
@@ -47,6 +47,9 @@ detects_heap_corruption(
 #
 fresh_test_table('test');
 $node->safe_psql('postgres', q(VACUUM (FREEZE, DISABLE_PAGE_SKIPPING) test));
+detects_no_corruption(
+	"verify_heapam('test')",
+	"all-frozen not corrupted table");
 corrupt_first_page('test');
 detects_heap_corruption("verify_heapam('test')",
 	"all-frozen corrupted table");
@@ -92,6 +95,15 @@ sub fresh_test_table
 		ALTER TABLE $relname ALTER b SET STORAGE external;
 		INSERT INTO $relname (a, b)
 			(SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);
+		BEGIN;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		RELEASE s1;
+		SAVEPOINT s1;
+		SELECT 1 FROM $relname WHERE a = 42 FOR UPDATE;
+		UPDATE $relname SET b = b WHERE a = 42;
+		COMMIT;
 	));
 }
 
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 88ab32490c..49f5ca0ef2 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -608,13 +608,6 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
 								   ctx->tuphdr->t_hoff, ctx->lp_len));
 		header_garbled = true;
 	}
-	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-		(ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))
-	{
-		report_corruption(ctx,
-						  pstrdup("tuple is marked as only locked, but also claims key columns were updated"));
-		header_garbled = true;
-	}
 
 	if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&
 		(ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index fb7ad0bab4..75223c9bea 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -49,12 +49,10 @@ RelationPutHeapTuple(Relation relation,
 
 	/*
 	 * Do not allow tuples with invalid combinations of hint bits to be placed
-	 * on a page.  These combinations are detected as corruption by the
-	 * contrib/amcheck logic, so if you disable one or both of these
-	 * assertions, make corresponding changes there.
+	 * on a page.  This combination is detected as corruption by the
+	 * contrib/amcheck logic, so if you disable this assertion, make
+	 * corresponding changes there.
 	 */
-	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&
-			 (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));
 	Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&
 			 (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));
 
-- 
2.20.1


--CE+1k2dSO48ffgeK--





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
@ 2024-09-05 04:52 David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: David Rowley @ 2024-09-05 04:52 UTC (permalink / raw)
  To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; pgsql-hackers

On Wed, 10 Jul 2024 at 21:36, Tatsuo Ishii <[email protected]> wrote:
> v2-0005-Add-memory-disk-usage-for-Window-Aggregate-nodes-.patch: This
> adds memory/disk usage for Window Aggregate nodes in EXPLAIN (ANALYZE)
> command. Note that if David's proposal
> https://www.postgresql.org/message-id/CAHoyFK9n-QCXKTUWT_xxtXninSMEv%2BgbJN66-y6prM3f4WkEHw%40mail.g...
> is committed, this will need to be adjusted.

Hi,

I pushed the changes to WindowAgg so as not to call tuplestore_end()
on every partition.  Can you rebase this patch over that change?

It would be good to do this in a way that does not add any new state
to WindowAggState, you can see that I had to shuffle fields around in
that struct because the next_parition field would have caused the
struct to become larger. I've not looked closely, but I expect this
can be done by adding more code to tuplestore_updatemax() to also
track the disk space used if the current storage has gone to disk. I
expect the maxSpace field can be used for both, but we'd need another
bool field to track if the max used was by disk or memory.

I think the performance of this would also need to be tested as it
means doing an lseek() on every tuplestore_clear() when we've gone to
disk. Probably that will be dominated by all the other overheads of a
tuplestore going to disk (i.e. dumptuples() etc), but it would be good
to check this. I suggest setting work_mem = 64 and making a test case
that only just spills to disk. Maybe do a few thousand partitions
worth of that and see if you can measure any slowdown.

David





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
@ 2024-09-05 05:38 ` Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: Tatsuo Ishii @ 2024-09-05 05:38 UTC (permalink / raw)
  To: [email protected]; +Cc: [email protected]; pgsql-hackers

Hi,

> On Wed, 10 Jul 2024 at 21:36, Tatsuo Ishii <[email protected]> wrote:
>> v2-0005-Add-memory-disk-usage-for-Window-Aggregate-nodes-.patch: This
>> adds memory/disk usage for Window Aggregate nodes in EXPLAIN (ANALYZE)
>> command. Note that if David's proposal
>> https://www.postgresql.org/message-id/CAHoyFK9n-QCXKTUWT_xxtXninSMEv%2BgbJN66-y6prM3f4WkEHw%40mail.g...
>> is committed, this will need to be adjusted.
> 
> Hi,
> 
> I pushed the changes to WindowAgg so as not to call tuplestore_end()
> on every partition.  Can you rebase this patch over that change?
> 
> It would be good to do this in a way that does not add any new state
> to WindowAggState, you can see that I had to shuffle fields around in
> that struct because the next_parition field would have caused the
> struct to become larger. I've not looked closely, but I expect this
> can be done by adding more code to tuplestore_updatemax() to also
> track the disk space used if the current storage has gone to disk. I
> expect the maxSpace field can be used for both, but we'd need another
> bool field to track if the max used was by disk or memory.
> 
> I think the performance of this would also need to be tested as it
> means doing an lseek() on every tuplestore_clear() when we've gone to
> disk. Probably that will be dominated by all the other overheads of a
> tuplestore going to disk (i.e. dumptuples() etc), but it would be good
> to check this. I suggest setting work_mem = 64 and making a test case
> that only just spills to disk. Maybe do a few thousand partitions
> worth of that and see if you can measure any slowdown.

Thank you for the suggestion. I will look into this.

Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
@ 2024-09-06 04:21   ` Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: Tatsuo Ishii @ 2024-09-06 04:21 UTC (permalink / raw)
  To: [email protected]; +Cc: [email protected]; pgsql-hackers

Hi David,

>> I pushed the changes to WindowAgg so as not to call tuplestore_end()
>> on every partition.  Can you rebase this patch over that change?
>> 
>> It would be good to do this in a way that does not add any new state
>> to WindowAggState, you can see that I had to shuffle fields around in
>> that struct because the next_parition field would have caused the
>> struct to become larger. I've not looked closely, but I expect this
>> can be done by adding more code to tuplestore_updatemax() to also
>> track the disk space used if the current storage has gone to disk. I
>> expect the maxSpace field can be used for both, but we'd need another
>> bool field to track if the max used was by disk or memory.

I have created a patch in the direction you suggested.  See attached
patch (v1-0001-Enhance-tuplestore.txt). To not confuse CFbot, the
extension is "txt", not "patch".

>> I think the performance of this would also need to be tested as it
>> means doing an lseek() on every tuplestore_clear() when we've gone to
>> disk. Probably that will be dominated by all the other overheads of a
>> tuplestore going to disk (i.e. dumptuples() etc), but it would be good
>> to check this. I suggest setting work_mem = 64 and making a test case
>> that only just spills to disk. Maybe do a few thousand partitions
>> worth of that and see if you can measure any slowdown.

I copied your shell script and slightly modified it then ran pgbench
with (1 10 100 1000 5000 10000) window partitions (see attached shell
script). In the script I set work_mem to 64kB. It seems for 10000,
1000 and 100 partitions, the performance difference seems
noises. However, for 10, 2, 1 partitions. I see large performance
degradation with the patched version: patched is slower than stock
master in 1.5% (10 partitions), 41% (2 partitions) and 55.7% (1
partition). See the attached graph.


^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
@ 2024-09-06 05:07     ` David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: David Rowley @ 2024-09-06 05:07 UTC (permalink / raw)
  To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; pgsql-hackers

On Fri, 6 Sept 2024 at 16:21, Tatsuo Ishii <[email protected]> wrote:
> However, for 10, 2, 1 partitions. I see large performance
> degradation with the patched version: patched is slower than stock
> master in 1.5% (10 partitions), 41% (2 partitions) and 55.7% (1
> partition). See the attached graph.

Thanks for making the adjustments to this.

I don't think there is any need to call tuplestore_updatemax() from
within writetup_heap(). That means having to update the maximum space
used every time a tuple is written to disk. That's a fairly massive
overhead.

Instead, it should be fine to modify tuplestore_updatemax() to set a
flag to true if state->status != TSS_INMEM and then record the disk
space used. That flag won't ever be set to false again.
tuplestore_storage_type_name() should just return "Disk" if the new
disk flag is set, even if state->status == TSS_INMEM. Since the
work_mem size won't change between tuplestore_clear() calls, if we've
once spilt to disk, then we shouldn't care about the memory used for
runs that didn't. Those will always have used less memory.

I did this quickly, but playing around with the attached, I didn't see
any slowdown.

Here's the results I got on my Zen2 AMD machine:

parts    master    yours    mine    mine_v_master
10000    5.01    5.12    5.09    99%
1000    4.30    4.25    4.24    101%
100    4.17    4.13    4.12    101%
10    4.16    4.12    4.10    101%
2    4.75    7.64    4.73    100%
1    4.75    8.57    4.73    100%

David

diff --git a/src/backend/utils/sort/tuplestore.c b/src/backend/utils/sort/tuplestore.c
index 444c8e25c2..03d745c609 100644
--- a/src/backend/utils/sort/tuplestore.c
+++ b/src/backend/utils/sort/tuplestore.c
@@ -107,9 +107,10 @@ struct Tuplestorestate
 	bool		backward;		/* store extra length words in file? */
 	bool		interXact;		/* keep open through transactions? */
 	bool		truncated;		/* tuplestore_trim has removed tuples? */
+	bool		usedDisk;		/* true if tuple store has ever gone to disk */
 	int64		availMem;		/* remaining memory available, in bytes */
 	int64		allowedMem;		/* total memory allowed, in bytes */
-	int64		maxSpace;		/* maximum space used in memory */
+	int64		maxSpace;		/* maximum space used in memory or disk */
 	int64		tuples;			/* number of tuples added */
 	BufFile    *myfile;			/* underlying file, or NULL if none */
 	MemoryContext context;		/* memory context for holding tuples */
@@ -262,6 +263,7 @@ tuplestore_begin_common(int eflags, bool interXact, int maxKBytes)
 	state->eflags = eflags;
 	state->interXact = interXact;
 	state->truncated = false;
+	state->usedDisk = false;
 	state->allowedMem = maxKBytes * 1024L;
 	state->availMem = state->allowedMem;
 	state->maxSpace = 0;
@@ -1499,17 +1501,25 @@ tuplestore_updatemax(Tuplestorestate *state)
 	if (state->status == TSS_INMEM)
 		state->maxSpace = Max(state->maxSpace,
 							  state->allowedMem - state->availMem);
+	else
+	{
+		state->maxSpace = Max(state->maxSpace,
+							  BufFileSize(state->myfile));
+		state->usedDisk = true;
+	}
 }
 
 /*
  * tuplestore_storage_type_name
  *		Return a string description of the storage method used to store the
- *		tuples.
+ *		tuples.  We don't use the current status as if the tuplestore went to
+ *		disk and a tuplestore_clear() allowed it to switch back to memory
+ *		again, we still want to know that it has spilled to disk.
  */
 const char *
 tuplestore_storage_type_name(Tuplestorestate *state)
 {
-	if (state->status == TSS_INMEM)
+	if (!state->usedDisk)
 		return "Memory";
 	else
 		return "Disk";
@@ -1517,8 +1527,7 @@ tuplestore_storage_type_name(Tuplestorestate *state)
 
 /*
  * tuplestore_space_used
- *		Return the maximum space used in memory unless the tuplestore has spilled
- *		to disk, in which case, return the disk space used.
+ *		Return the maximum space used in memory or disk.
  */
 int64
 tuplestore_space_used(Tuplestorestate *state)
@@ -1526,10 +1535,7 @@ tuplestore_space_used(Tuplestorestate *state)
 	/* First, update the maxSpace field */
 	tuplestore_updatemax(state);
 
-	if (state->status == TSS_INMEM)
-		return state->maxSpace;
-	else
-		return BufFileSize(state->myfile);
+	return state->maxSpace;
 }
 
 /*
@@ -1601,7 +1607,6 @@ writetup_heap(Tuplestorestate *state, void *tup)
 	if (state->backward)		/* need trailing length word? */
 		BufFileWrite(state->myfile, &tuplen, sizeof(tuplen));
 
-	/* no need to call tuplestore_updatemax() when not in TSS_INMEM */
 	FREEMEM(state, GetMemoryChunkSpace(tuple));
 	heap_free_minimal_tuple(tuple);
 }


Attachments:

  [text/plain] tuplestore_track_max_disk_space_used.patch.txt (3.0K, ../../CAApHDvqgBcwV8iVZY0f1+D2TQrRAbjFA5AmkX-BHL2z9PhTQSQ@mail.gmail.com/2-tuplestore_track_max_disk_space_used.patch.txt)
  download | inline diff:
diff --git a/src/backend/utils/sort/tuplestore.c b/src/backend/utils/sort/tuplestore.c
index 444c8e25c2..03d745c609 100644
--- a/src/backend/utils/sort/tuplestore.c
+++ b/src/backend/utils/sort/tuplestore.c
@@ -107,9 +107,10 @@ struct Tuplestorestate
 	bool		backward;		/* store extra length words in file? */
 	bool		interXact;		/* keep open through transactions? */
 	bool		truncated;		/* tuplestore_trim has removed tuples? */
+	bool		usedDisk;		/* true if tuple store has ever gone to disk */
 	int64		availMem;		/* remaining memory available, in bytes */
 	int64		allowedMem;		/* total memory allowed, in bytes */
-	int64		maxSpace;		/* maximum space used in memory */
+	int64		maxSpace;		/* maximum space used in memory or disk */
 	int64		tuples;			/* number of tuples added */
 	BufFile    *myfile;			/* underlying file, or NULL if none */
 	MemoryContext context;		/* memory context for holding tuples */
@@ -262,6 +263,7 @@ tuplestore_begin_common(int eflags, bool interXact, int maxKBytes)
 	state->eflags = eflags;
 	state->interXact = interXact;
 	state->truncated = false;
+	state->usedDisk = false;
 	state->allowedMem = maxKBytes * 1024L;
 	state->availMem = state->allowedMem;
 	state->maxSpace = 0;
@@ -1499,17 +1501,25 @@ tuplestore_updatemax(Tuplestorestate *state)
 	if (state->status == TSS_INMEM)
 		state->maxSpace = Max(state->maxSpace,
 							  state->allowedMem - state->availMem);
+	else
+	{
+		state->maxSpace = Max(state->maxSpace,
+							  BufFileSize(state->myfile));
+		state->usedDisk = true;
+	}
 }
 
 /*
  * tuplestore_storage_type_name
  *		Return a string description of the storage method used to store the
- *		tuples.
+ *		tuples.  We don't use the current status as if the tuplestore went to
+ *		disk and a tuplestore_clear() allowed it to switch back to memory
+ *		again, we still want to know that it has spilled to disk.
  */
 const char *
 tuplestore_storage_type_name(Tuplestorestate *state)
 {
-	if (state->status == TSS_INMEM)
+	if (!state->usedDisk)
 		return "Memory";
 	else
 		return "Disk";
@@ -1517,8 +1527,7 @@ tuplestore_storage_type_name(Tuplestorestate *state)
 
 /*
  * tuplestore_space_used
- *		Return the maximum space used in memory unless the tuplestore has spilled
- *		to disk, in which case, return the disk space used.
+ *		Return the maximum space used in memory or disk.
  */
 int64
 tuplestore_space_used(Tuplestorestate *state)
@@ -1526,10 +1535,7 @@ tuplestore_space_used(Tuplestorestate *state)
 	/* First, update the maxSpace field */
 	tuplestore_updatemax(state);
 
-	if (state->status == TSS_INMEM)
-		return state->maxSpace;
-	else
-		return BufFileSize(state->myfile);
+	return state->maxSpace;
 }
 
 /*
@@ -1601,7 +1607,6 @@ writetup_heap(Tuplestorestate *state, void *tup)
 	if (state->backward)		/* need trailing length word? */
 		BufFileWrite(state->myfile, &tuplen, sizeof(tuplen));
 
-	/* no need to call tuplestore_updatemax() when not in TSS_INMEM */
 	FREEMEM(state, GetMemoryChunkSpace(tuple));
 	heap_free_minimal_tuple(tuple);
 }


^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
@ 2024-09-06 06:02       ` Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: Tatsuo Ishii @ 2024-09-06 06:02 UTC (permalink / raw)
  To: [email protected]; +Cc: [email protected]; pgsql-hackers

> Thanks for making the adjustments to this.
> 
> I don't think there is any need to call tuplestore_updatemax() from
> within writetup_heap(). That means having to update the maximum space
> used every time a tuple is written to disk. That's a fairly massive
> overhead.
> 
> Instead, it should be fine to modify tuplestore_updatemax() to set a
> flag to true if state->status != TSS_INMEM and then record the disk
> space used. That flag won't ever be set to false again.
> tuplestore_storage_type_name() should just return "Disk" if the new
> disk flag is set, even if state->status == TSS_INMEM. Since the
> work_mem size won't change between tuplestore_clear() calls, if we've
> once spilt to disk, then we shouldn't care about the memory used for
> runs that didn't. Those will always have used less memory.
> 
> I did this quickly, but playing around with the attached, I didn't see
> any slowdown.

Your patch looks good to me and I confirmed that with your patch I
didn't see any slowdown either. Thanks!

Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
@ 2024-09-06 07:07         ` Ashutosh Bapat <[email protected]>
  2024-09-06 07:50           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  0 siblings, 2 replies; 107+ messages in thread

From: Ashutosh Bapat @ 2024-09-06 07:07 UTC (permalink / raw)
  To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; pgsql-hackers

On Fri, Sep 6, 2024 at 11:32 AM Tatsuo Ishii <[email protected]> wrote:
>
> > Thanks for making the adjustments to this.
> >
> > I don't think there is any need to call tuplestore_updatemax() from
> > within writetup_heap(). That means having to update the maximum space
> > used every time a tuple is written to disk. That's a fairly massive
> > overhead.
> >
> > Instead, it should be fine to modify tuplestore_updatemax() to set a
> > flag to true if state->status != TSS_INMEM and then record the disk
> > space used. That flag won't ever be set to false again.
> > tuplestore_storage_type_name() should just return "Disk" if the new
> > disk flag is set, even if state->status == TSS_INMEM. Since the
> > work_mem size won't change between tuplestore_clear() calls, if we've
> > once spilt to disk, then we shouldn't care about the memory used for
> > runs that didn't. Those will always have used less memory.
> >
> > I did this quickly, but playing around with the attached, I didn't see
> > any slowdown.
>
> Your patch looks good to me and I confirmed that with your patch I
> didn't see any slowdown either. Thanks!

The changes look better. A nitpick though. With their definitions
changed, I think it's better to change the names of the functions
since their purpose has changed. Right now they report the storage
type and size used, respectively, at the time of calling the function.
With this patch, they report maximum space ever used and the storage
corresponding to the maximum space. tuplestore_space_used() may be
changed to tuplestore_maxspace_used(). I am having difficulty with
tuplestore_storage_type_name(); tuplestore_largest_storage_type_name()
seems mouthful and yet not doing justice to the functionality. It
might be better to just have one funciton tuplestore_maxspace_used()
which returns both the maximum space used as well as the storage type
when maximum space was used.

The comments need a bit of grammar fixes, but that can be done when
finalizing the patches.

-- 
Best Wishes,
Ashutosh Bapat





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
@ 2024-09-06 07:50           ` Tatsuo Ishii <[email protected]>
  1 sibling, 0 replies; 107+ messages in thread

From: Tatsuo Ishii @ 2024-09-06 07:50 UTC (permalink / raw)
  To: [email protected]; +Cc: [email protected]; pgsql-hackers

> The changes look better. A nitpick though. With their definitions
> changed, I think it's better to change the names of the functions
> since their purpose has changed. Right now they report the storage
> type and size used, respectively, at the time of calling the function.
> With this patch, they report maximum space ever used and the storage
> corresponding to the maximum space. tuplestore_space_used() may be
> changed to tuplestore_maxspace_used(). I am having difficulty with
> tuplestore_storage_type_name(); tuplestore_largest_storage_type_name()
> seems mouthful and yet not doing justice to the functionality. It
> might be better to just have one funciton tuplestore_maxspace_used()
> which returns both the maximum space used as well as the storage type
> when maximum space was used.

+1. Returning the storage type by the same function, not by a separate
function looks more natural.

Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
@ 2024-09-06 13:51           ` David Rowley <[email protected]>
  2024-09-09 10:12             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  1 sibling, 2 replies; 107+ messages in thread

From: David Rowley @ 2024-09-06 13:51 UTC (permalink / raw)
  To: Ashutosh Bapat <[email protected]>; +Cc: Tatsuo Ishii <[email protected]>; pgsql-hackers

On Fri, 6 Sept 2024 at 19:08, Ashutosh Bapat
<[email protected]> wrote:
> The changes look better. A nitpick though. With their definitions
> changed, I think it's better to change the names of the functions
> since their purpose has changed. Right now they report the storage
> type and size used, respectively, at the time of calling the function.
> With this patch, they report maximum space ever used and the storage
> corresponding to the maximum space. tuplestore_space_used() may be
> changed to tuplestore_maxspace_used(). I am having difficulty with
> tuplestore_storage_type_name(); tuplestore_largest_storage_type_name()
> seems mouthful and yet not doing justice to the functionality. It
> might be better to just have one funciton tuplestore_maxspace_used()
> which returns both the maximum space used as well as the storage type
> when maximum space was used.

How about just removing tuplestore_storage_type_name() and
tuplestore_space_used() and adding tuplestore_get_stats(). I did take
some inspiration from tuplesort.c for this, so maybe we can defer back
there for further guidance. I'm not so sure it's worth having a stats
struct type like tuplesort.c has. All we need is a char ** and an
int64 * output parameter to pass to the stats function.  I don't think
we need to copy the tuplesort_method_name(). It seems fine just to
point the output parameter of the stats function at the statically
allocated constant.

David





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
@ 2024-09-09 10:12             ` Ashutosh Bapat <[email protected]>
  1 sibling, 0 replies; 107+ messages in thread

From: Ashutosh Bapat @ 2024-09-09 10:12 UTC (permalink / raw)
  To: David Rowley <[email protected]>; +Cc: Tatsuo Ishii <[email protected]>; pgsql-hackers

On Fri, Sep 6, 2024 at 7:21 PM David Rowley <[email protected]> wrote:
>
> On Fri, 6 Sept 2024 at 19:08, Ashutosh Bapat
> <[email protected]> wrote:
> > The changes look better. A nitpick though. With their definitions
> > changed, I think it's better to change the names of the functions
> > since their purpose has changed. Right now they report the storage
> > type and size used, respectively, at the time of calling the function.
> > With this patch, they report maximum space ever used and the storage
> > corresponding to the maximum space. tuplestore_space_used() may be
> > changed to tuplestore_maxspace_used(). I am having difficulty with
> > tuplestore_storage_type_name(); tuplestore_largest_storage_type_name()
> > seems mouthful and yet not doing justice to the functionality. It
> > might be better to just have one funciton tuplestore_maxspace_used()
> > which returns both the maximum space used as well as the storage type
> > when maximum space was used.
>
> How about just removing tuplestore_storage_type_name() and
> tuplestore_space_used() and adding tuplestore_get_stats(). I did take
> some inspiration from tuplesort.c for this, so maybe we can defer back
> there for further guidance. I'm not so sure it's worth having a stats
> struct type like tuplesort.c has. All we need is a char ** and an
> int64 * output parameter to pass to the stats function.  I don't think
> we need to copy the tuplesort_method_name(). It seems fine just to
> point the output parameter of the stats function at the statically
> allocated constant.

tuplestore_get_stats() similar to tuplesort_get_stats() looks fine. In
future the stats reported by this function might expand e.g. maximum
number of readers may be included in the stats. If it expands beyond
two values, we could think of a separate structure, but for now it
looks fine given its limited use. A comment explaining why we aren't
using a stats structure and some guidance on when that would be
appropriate will be better.

-- 
Best Wishes,
Ashutosh Bapat





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
@ 2024-09-12 02:03             ` Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  1 sibling, 1 reply; 107+ messages in thread

From: Tatsuo Ishii @ 2024-09-12 02:03 UTC (permalink / raw)
  To: [email protected]; +Cc: [email protected]; pgsql-hackers

> On Fri, 6 Sept 2024 at 19:08, Ashutosh Bapat
> <[email protected]> wrote:
>> The changes look better. A nitpick though. With their definitions
>> changed, I think it's better to change the names of the functions
>> since their purpose has changed. Right now they report the storage
>> type and size used, respectively, at the time of calling the function.
>> With this patch, they report maximum space ever used and the storage
>> corresponding to the maximum space. tuplestore_space_used() may be
>> changed to tuplestore_maxspace_used(). I am having difficulty with
>> tuplestore_storage_type_name(); tuplestore_largest_storage_type_name()
>> seems mouthful and yet not doing justice to the functionality. It
>> might be better to just have one funciton tuplestore_maxspace_used()
>> which returns both the maximum space used as well as the storage type
>> when maximum space was used.
> 
> How about just removing tuplestore_storage_type_name() and
> tuplestore_space_used() and adding tuplestore_get_stats(). I did take
> some inspiration from tuplesort.c for this, so maybe we can defer back
> there for further guidance. I'm not so sure it's worth having a stats
> struct type like tuplesort.c has. All we need is a char ** and an
> int64 * output parameter to pass to the stats function.  I don't think
> we need to copy the tuplesort_method_name(). It seems fine just to
> point the output parameter of the stats function at the statically
> allocated constant.

Are you going to push the changes to tuplestore.c anytime soon? I
would like to rebase my patch[1] but the patch could be affected by
the tuplestore API change.

Best reagards,

[1] https://www.postgresql.org/message-id/CAApHDvoY8cibGcicLV0fNh%3D9JVx9PANcWvhkdjBnDCc9Quqytg%40mail.g...
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
@ 2024-09-12 02:28               ` David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: David Rowley @ 2024-09-12 02:28 UTC (permalink / raw)
  To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; pgsql-hackers

On Thu, 12 Sept 2024 at 14:04, Tatsuo Ishii <[email protected]> wrote:
> Are you going to push the changes to tuplestore.c anytime soon? I
> would like to rebase my patch[1] but the patch could be affected by
> the tuplestore API change.

Ok, I'll look at that. I had thought you were taking care of writing the patch.

David





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
@ 2024-09-12 02:42                 ` Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: Tatsuo Ishii @ 2024-09-12 02:42 UTC (permalink / raw)
  To: [email protected]; +Cc: [email protected]; pgsql-hackers

> On Thu, 12 Sept 2024 at 14:04, Tatsuo Ishii <[email protected]> wrote:
>> Are you going to push the changes to tuplestore.c anytime soon? I
>> would like to rebase my patch[1] but the patch could be affected by
>> the tuplestore API change.
> 
> Ok, I'll look at that.

Thanks.

I had thought you were taking care of writing the patch.

Sorry, I should have asked you first if you are going to write the API
change patch.
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp






^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
@ 2024-09-12 05:07                   ` David Rowley <[email protected]>
  2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: David Rowley @ 2024-09-12 05:07 UTC (permalink / raw)
  To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; pgsql-hackers

On Thu, 12 Sept 2024 at 14:42, Tatsuo Ishii <[email protected]> wrote:
> Sorry, I should have asked you first if you are going to write the API
> change patch.

I pushed a patch to change the API.

David





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
@ 2024-09-12 05:09                     ` Tatsuo Ishii <[email protected]>
  2024-09-12 12:12                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: Tatsuo Ishii @ 2024-09-12 05:09 UTC (permalink / raw)
  To: [email protected]; +Cc: [email protected]; pgsql-hackers

> I pushed a patch to change the API.

Thank you!
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
@ 2024-09-12 12:12                       ` Tatsuo Ishii <[email protected]>
  2024-09-12 21:20                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: Tatsuo Ishii @ 2024-09-12 12:12 UTC (permalink / raw)
  To: [email protected]; [email protected]; [email protected]; [email protected]; +Cc: pgsql-hackers

Here is the v3 patch. This time I only include patch for the Window
Aggregate node. Patches for other node types will come after this
patch getting committed or come close to commitable state.

David,
In this patch I refactored show_material_info. I divided it into
show_material_info and show_storage_info so that the latter can be
used by other node types including window aggregate node. What do you
think?

I also added a test case in explain.sql per discussion with Maxim
Orlov.

Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp


^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 12:12                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
@ 2024-09-12 21:20                         ` David Rowley <[email protected]>
  2024-09-13 06:11                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: David Rowley @ 2024-09-12 21:20 UTC (permalink / raw)
  To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers

On Fri, 13 Sept 2024 at 00:12, Tatsuo Ishii <[email protected]> wrote:
> In this patch I refactored show_material_info. I divided it into
> show_material_info and show_storage_info so that the latter can be
> used by other node types including window aggregate node. What do you
> think?

Yes, I think it's a good idea to move that into a helper function. If
you do the other node types, without that helper the could would have
to be repeated quite a few times. Maybe show_storage_info() can be
moved up with the other helper functions, say below
show_sortorder_options() ?  It might be a good idea to keep the "if
(!es->analyze || tupstore == NULL)" checks in the calling function
rather than the helper too.

I thought about the location of the test for a while and read the
"This file is concerned with testing EXPLAIN in its own right."
comment at the top of that explain.out.  I was trying to decide if
testing output of a specific node type met this or not. I can't pick
out any other tests there which are specific to a node type, so I'm
unsure if this is the location for it or not. However, to put it
anywhere else means having to add a plpgsql function to mask out the
unstable parts of EXPLAIN, so maybe the location is good as it saves
from having to do that. I'm 50/50 on this, so I'm happy to let you
decide.  You could also shrink that 100 rows into a smaller number for
the generate_series without losing any coverage.

Aside from that, I think the patch is good.  Thanks for working on it.

David





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 12:12                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 21:20                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
@ 2024-09-13 06:11                           ` Tatsuo Ishii <[email protected]>
  2024-09-13 08:57                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Maxim Orlov <[email protected]>
  2024-09-13 09:06                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  0 siblings, 2 replies; 107+ messages in thread

From: Tatsuo Ishii @ 2024-09-13 06:11 UTC (permalink / raw)
  To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers

David,

Thank you for your review.

> Yes, I think it's a good idea to move that into a helper function. If
> you do the other node types, without that helper the could would have
> to be repeated quite a few times. Maybe show_storage_info() can be
> moved up with the other helper functions, say below
> show_sortorder_options() ?

Yeah, that makes sense. Looks less random.

> It might be a good idea to keep the "if
> (!es->analyze || tupstore == NULL)" checks in the calling function
> rather than the helper too.

I agree with this. This kind of check should be done in the calling
function.

> I thought about the location of the test for a while and read the
> "This file is concerned with testing EXPLAIN in its own right."
> comment at the top of that explain.out.  I was trying to decide if
> testing output of a specific node type met this or not. I can't pick
> out any other tests there which are specific to a node type, so I'm
> unsure if this is the location for it or not. However, to put it
> anywhere else means having to add a plpgsql function to mask out the
> unstable parts of EXPLAIN, so maybe the location is good as it saves
> from having to do that. I'm 50/50 on this, so I'm happy to let you
> decide.

Yeah. Maybe we should move the function to elsewhere so that it can be
shared by other tests. However in this case it's purpose is testing an
additional output in an explain command. I think this is not far from
"This file is concerned with testing EXPLAIN in its own right.". So I
would like to keep the test in explain.sql.

> You could also shrink that 100 rows into a smaller number for
> the generate_series without losing any coverage.

Right. I will make the change.

> Aside from that, I think the patch is good.  Thanks for working on it.

Thanks. Attached is the v4 patch. I am going push it if there's no
objection.

After this, I will work on remaining node types.

Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp


^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 12:12                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 21:20                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-13 06:11                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
@ 2024-09-13 08:57                             ` Maxim Orlov <[email protected]>
  1 sibling, 0 replies; 107+ messages in thread

From: Maxim Orlov @ 2024-09-13 08:57 UTC (permalink / raw)
  To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers

On Fri, 13 Sept 2024 at 09:11, Tatsuo Ishii <[email protected]> wrote:

> Thanks. Attached is the v4 patch. I am going push it if there's no
> objection.
>

Looks good to me. Thank you for your work.

-- 
Best regards,
Maxim Orlov.


^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 12:12                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 21:20                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-13 06:11                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
@ 2024-09-13 09:06                             ` Ashutosh Bapat <[email protected]>
  2024-09-13 09:31                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  1 sibling, 1 reply; 107+ messages in thread

From: Ashutosh Bapat @ 2024-09-13 09:06 UTC (permalink / raw)
  To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers

The patch looks fine but it doesn't add a test case where Storage is
Disk or the case when the last usage fit in memory but an earlier
usage spilled to disk. Do we want to cover those. This test would be
the only one where those code paths could be tested.

On Fri, Sep 13, 2024 at 11:41 AM Tatsuo Ishii <[email protected]> wrote:
>
> David,
>
> Thank you for your review.
>
> > Yes, I think it's a good idea to move that into a helper function. If
> > you do the other node types, without that helper the could would have
> > to be repeated quite a few times. Maybe show_storage_info() can be
> > moved up with the other helper functions, say below
> > show_sortorder_options() ?
>
> Yeah, that makes sense. Looks less random.
>
> > It might be a good idea to keep the "if
> > (!es->analyze || tupstore == NULL)" checks in the calling function
> > rather than the helper too.
>
> I agree with this. This kind of check should be done in the calling
> function.
>
> > I thought about the location of the test for a while and read the
> > "This file is concerned with testing EXPLAIN in its own right."
> > comment at the top of that explain.out.  I was trying to decide if
> > testing output of a specific node type met this or not. I can't pick
> > out any other tests there which are specific to a node type, so I'm
> > unsure if this is the location for it or not. However, to put it
> > anywhere else means having to add a plpgsql function to mask out the
> > unstable parts of EXPLAIN, so maybe the location is good as it saves
> > from having to do that. I'm 50/50 on this, so I'm happy to let you
> > decide.
>
> Yeah. Maybe we should move the function to elsewhere so that it can be
> shared by other tests. However in this case it's purpose is testing an
> additional output in an explain command. I think this is not far from
> "This file is concerned with testing EXPLAIN in its own right.". So I
> would like to keep the test in explain.sql.
>
> > You could also shrink that 100 rows into a smaller number for
> > the generate_series without losing any coverage.
>
> Right. I will make the change.
>
> > Aside from that, I think the patch is good.  Thanks for working on it.
>
> Thanks. Attached is the v4 patch. I am going push it if there's no
> objection.
>
> After this, I will work on remaining node types.
>
> Best reagards,
> --
> Tatsuo Ishii
> SRA OSS K.K.
> English: http://www.sraoss.co.jp/index_en/
> Japanese:http://www.sraoss.co.jp



-- 
Best Wishes,
Ashutosh Bapat





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 12:12                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 21:20                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-13 06:11                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 09:06                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
@ 2024-09-13 09:31                               ` Tatsuo Ishii <[email protected]>
  2024-09-13 10:44                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: Tatsuo Ishii @ 2024-09-13 09:31 UTC (permalink / raw)
  To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers

> The patch looks fine but it doesn't add a test case where Storage is
> Disk

We can test the case by setting work_mem to the minimum size (64kB)
and giving slightly larger "stop" parameter to generate_series.

> or the case when the last usage fit in memory but an earlier
> usage spilled to disk.

In my understanding once tuplestore changes the storage type to disk,
it never returns to the memory storage type in terms of
tuplestore_get_stats.  i.e. once state->usedDisk is set to true, it
never goes back to false. So the test case is not necessary.
David, am I correct?

> Do we want to cover those. This test would be
> the only one where those code paths could be tested.

I am fine to add the first test case.

Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 12:12                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 21:20                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-13 06:11                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 09:06                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-13 09:31                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
@ 2024-09-13 10:44                                 ` Ashutosh Bapat <[email protected]>
  2024-09-14 08:12                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: Ashutosh Bapat @ 2024-09-13 10:44 UTC (permalink / raw)
  To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers

On Fri, Sep 13, 2024 at 3:02 PM Tatsuo Ishii <[email protected]> wrote:
>
> > The patch looks fine but it doesn't add a test case where Storage is
> > Disk
>
> We can test the case by setting work_mem to the minimum size (64kB)
> and giving slightly larger "stop" parameter to generate_series.
>

WFM

> > or the case when the last usage fit in memory but an earlier
> > usage spilled to disk.
>
> In my understanding once tuplestore changes the storage type to disk,
> it never returns to the memory storage type in terms of
> tuplestore_get_stats.  i.e. once state->usedDisk is set to true, it
> never goes back to false. So the test case is not necessary.
> David, am I correct?

I understand that. I am requesting a testcase to test that same logic.
-- 
Best Wishes,
Ashutosh Bapat





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 12:12                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 21:20                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-13 06:11                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 09:06                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-13 09:31                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 10:44                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
@ 2024-09-14 08:12                                   ` Tatsuo Ishii <[email protected]>
  2024-09-16 11:11                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: Tatsuo Ishii @ 2024-09-14 08:12 UTC (permalink / raw)
  To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers

>> > or the case when the last usage fit in memory but an earlier
>> > usage spilled to disk.
>>
>> In my understanding once tuplestore changes the storage type to disk,
>> it never returns to the memory storage type in terms of
>> tuplestore_get_stats.  i.e. once state->usedDisk is set to true, it
>> never goes back to false. So the test case is not necessary.
>> David, am I correct?
> 
> I understand that. I am requesting a testcase to test that same logic.

Maybe something like this? In the example below there are 2
partitions. the first one has 1998 rows and the second one has 2
rows. Assuming that work_mem is 64kB, the first one does not fit the
memory and spills to disk. The second partition fits memory. However as
state->usedDisk remains true, explain shows "Storage: Disk".

test=# explain (analyze,costs off) select sum(n) over(partition by m) from (SELECT n < 3 as m, n from generate_series(1,2000) a(n));
n                                         QUERY PLAN                             
             
--------------------------------------------------------------------------------
-------------
 WindowAgg (actual time=1.958..473328.589 rows=2000 loops=1)
   Storage: Disk  Maximum Storage: 65kB
   ->  Sort (actual time=1.008..1.277 rows=2000 loops=1)
         Sort Key: ((a.n < 3))
         Sort Method: external merge  Disk: 48kB
         ->  Function Scan on generate_series a (actual time=0.300..0.633 rows=2
000 loops=1)
 Planning Time: 0.069 ms
 Execution Time: 474515.476 ms
(8 rows)

Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 12:12                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 21:20                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-13 06:11                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 09:06                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-13 09:31                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 10:44                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-14 08:12                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
@ 2024-09-16 11:11                                     ` Ashutosh Bapat <[email protected]>
  2024-09-16 12:13                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: Ashutosh Bapat @ 2024-09-16 11:11 UTC (permalink / raw)
  To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers

On Sat, Sep 14, 2024 at 1:42 PM Tatsuo Ishii <[email protected]> wrote:
>
> >> > or the case when the last usage fit in memory but an earlier
> >> > usage spilled to disk.
> >>
> >> In my understanding once tuplestore changes the storage type to disk,
> >> it never returns to the memory storage type in terms of
> >> tuplestore_get_stats.  i.e. once state->usedDisk is set to true, it
> >> never goes back to false. So the test case is not necessary.
> >> David, am I correct?
> >
> > I understand that. I am requesting a testcase to test that same logic.
>
> Maybe something like this? In the example below there are 2
> partitions. the first one has 1998 rows and the second one has 2
> rows. Assuming that work_mem is 64kB, the first one does not fit the
> memory and spills to disk. The second partition fits memory. However as
> state->usedDisk remains true, explain shows "Storage: Disk".
>
> test=# explain (analyze,costs off) select sum(n) over(partition by m) from (SELECT n < 3 as m, n from generate_series(1,2000) a(n));
> n                                         QUERY PLAN
>
> --------------------------------------------------------------------------------
> -------------
>  WindowAgg (actual time=1.958..473328.589 rows=2000 loops=1)
>    Storage: Disk  Maximum Storage: 65kB
>    ->  Sort (actual time=1.008..1.277 rows=2000 loops=1)
>          Sort Key: ((a.n < 3))
>          Sort Method: external merge  Disk: 48kB
>          ->  Function Scan on generate_series a (actual time=0.300..0.633 rows=2
> 000 loops=1)
>  Planning Time: 0.069 ms
>  Execution Time: 474515.476 ms
> (8 rows)

Thanks. This will do. Is there a way to force the larger partition to
be computed first? That way we definitely know that the last
computation was done when all the tuples in the tuplestore were in
memory.

-- 
Best Wishes,
Ashutosh Bapat





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 12:12                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 21:20                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-13 06:11                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 09:06                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-13 09:31                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 10:44                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-14 08:12                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-16 11:11                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
@ 2024-09-16 12:13                                       ` Tatsuo Ishii <[email protected]>
  2024-09-17 02:40                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: Tatsuo Ishii @ 2024-09-16 12:13 UTC (permalink / raw)
  To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers

Hi Ashutosh,

Thank you for the review.

> Thanks. This will do. Is there a way to force the larger partition to
> be computed first? That way we definitely know that the last
> computation was done when all the tuples in the tuplestore were in
> memory.

Not sure if there's any way to force it in the SQL standard.  However
in term of implementation, PostgreSQL sorts the function
(generate_series) scan result using a sort key "a.n < 3", which
results in rows being >= 2 first (as false == 0), then rows being < 3
(as true == 1). So unless PostgreSQL changes the way to sort boolean
data type, I think the result should be stable.

Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 12:12                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 21:20                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-13 06:11                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 09:06                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-13 09:31                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 10:44                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-14 08:12                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-16 11:11                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-16 12:13                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
@ 2024-09-17 02:40                                         ` Tatsuo Ishii <[email protected]>
  2024-09-17 03:16                                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: Tatsuo Ishii @ 2024-09-17 02:40 UTC (permalink / raw)
  To: [email protected]; [email protected]; [email protected]; [email protected]; +Cc: pgsql-hackers

> Not sure if there's any way to force it in the SQL standard.  However
> in term of implementation, PostgreSQL sorts the function
> (generate_series) scan result using a sort key "a.n < 3", which
> results in rows being >= 2 first (as false == 0), then rows being < 3
> (as true == 1). So unless PostgreSQL changes the way to sort boolean
> data type, I think the result should be stable.

Attached is the v5 patch. The difference from v4 is addtion of two
more tests to explain.sql:

1) spils to disk case
2) splis to disk then switch back to memory case

Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp


^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 12:12                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 21:20                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-13 06:11                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 09:06                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-13 09:31                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 10:44                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-14 08:12                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-16 11:11                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-16 12:13                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 02:40                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
@ 2024-09-17 03:16                                           ` David Rowley <[email protected]>
  2024-09-17 06:06                                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: David Rowley @ 2024-09-17 03:16 UTC (permalink / raw)
  To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers

On Tue, 17 Sept 2024 at 14:40, Tatsuo Ishii <[email protected]> wrote:
> Attached is the v5 patch. The difference from v4 is addtion of two
> more tests to explain.sql:
>
> 1) spils to disk case
> 2) splis to disk then switch back to memory case

Looks ok to me, aside from the missing "reset work_mem;" after you're
done with testing the disk spilling code.

David





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 12:12                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 21:20                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-13 06:11                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 09:06                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-13 09:31                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 10:44                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-14 08:12                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-16 11:11                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-16 12:13                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 02:40                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 03:16                                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
@ 2024-09-17 06:06                                             ` Tatsuo Ishii <[email protected]>
  2024-09-18 12:12                                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: Tatsuo Ishii @ 2024-09-17 06:06 UTC (permalink / raw)
  To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers

> On Tue, 17 Sept 2024 at 14:40, Tatsuo Ishii <[email protected]> wrote:
>> Attached is the v5 patch. The difference from v4 is addtion of two
>> more tests to explain.sql:
>>
>> 1) spils to disk case
>> 2) splis to disk then switch back to memory case
> 
> Looks ok to me, aside from the missing "reset work_mem;" after you're
> done with testing the disk spilling code.

Thanks. I have added it and pushed the patch.

Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 12:12                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 21:20                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-13 06:11                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 09:06                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-13 09:31                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 10:44                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-14 08:12                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-16 11:11                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-16 12:13                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 02:40                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 03:16                                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-17 06:06                                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
@ 2024-09-18 12:12                                               ` Tatsuo Ishii <[email protected]>
  2024-09-18 21:57                                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: Tatsuo Ishii @ 2024-09-18 12:12 UTC (permalink / raw)
  To: [email protected]; [email protected]; [email protected]; [email protected]; +Cc: pgsql-hackers

> Thanks. I have added it and pushed the patch.

So I have created patches to do the same for CTE scan and table
function scan node. Patch attached.

Actually there's one more executor node type that uses tuplestore:
recursive union (used in "with recursive"). The particular node type
uses two tuplestore and we cannot simply apply tuplestore_get_stats()
to the node type. We need to modify RecursiveUnionState to track the
maximum tuplestore usage. I am not sure this would be worth the
effort. Opinion?

Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp


^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 12:12                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 21:20                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-13 06:11                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 09:06                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-13 09:31                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 10:44                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-14 08:12                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-16 11:11                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-16 12:13                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 02:40                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 03:16                                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-17 06:06                                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-18 12:12                                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
@ 2024-09-18 21:57                                                 ` David Rowley <[email protected]>
  2024-09-19 00:01                                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: David Rowley @ 2024-09-18 21:57 UTC (permalink / raw)
  To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers

On Thu, 19 Sept 2024 at 00:13, Tatsuo Ishii <[email protected]> wrote:
> Actually there's one more executor node type that uses tuplestore:
> recursive union (used in "with recursive"). The particular node type
> uses two tuplestore and we cannot simply apply tuplestore_get_stats()
> to the node type. We need to modify RecursiveUnionState to track the
> maximum tuplestore usage. I am not sure this would be worth the
> effort. Opinion?

Could you add the two sizes together and take the storage type from
the tuplestore with the highest storage size?

David





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 12:12                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 21:20                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-13 06:11                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 09:06                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-13 09:31                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 10:44                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-14 08:12                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-16 11:11                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-16 12:13                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 02:40                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 03:16                                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-17 06:06                                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-18 12:12                                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-18 21:57                                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
@ 2024-09-19 00:01                                                   ` Tatsuo Ishii <[email protected]>
  2024-09-19 00:49                                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: Tatsuo Ishii @ 2024-09-19 00:01 UTC (permalink / raw)
  To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers

> On Thu, 19 Sept 2024 at 00:13, Tatsuo Ishii <[email protected]> wrote:
>> Actually there's one more executor node type that uses tuplestore:
>> recursive union (used in "with recursive"). The particular node type
>> uses two tuplestore and we cannot simply apply tuplestore_get_stats()
>> to the node type. We need to modify RecursiveUnionState to track the
>> maximum tuplestore usage. I am not sure this would be worth the
>> effort. Opinion?
> 
> Could you add the two sizes together and take the storage type from
> the tuplestore with the highest storage size?

I don't think this works because tuplestore_begin/tuplestore_end are
called while executing the node (ExecRecursiveUnion).

I think the way you are proposing only shows the stats last time when
those tuplestore are created.

Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 12:12                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 21:20                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-13 06:11                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 09:06                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-13 09:31                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 10:44                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-14 08:12                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-16 11:11                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-16 12:13                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 02:40                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 03:16                                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-17 06:06                                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-18 12:12                                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-18 21:57                                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 00:01                                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
@ 2024-09-19 00:49                                                     ` David Rowley <[email protected]>
  2024-09-19 01:49                                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: David Rowley @ 2024-09-19 00:49 UTC (permalink / raw)
  To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers

On Thu, 19 Sept 2024 at 12:01, Tatsuo Ishii <[email protected]> wrote:
> > Could you add the two sizes together and take the storage type from
> > the tuplestore with the highest storage size?
>
> I don't think this works because tuplestore_begin/tuplestore_end are
> called while executing the node (ExecRecursiveUnion).
>
> I think the way you are proposing only shows the stats last time when
> those tuplestore are created.

That code could be modified to swap the tuplestores and do a
tuplestore_clear() instead of tuplestore_end() followed by
tuplestore_begin_heap().

It's likely worthwhile from a performance point of view.  Here's a
small test as an example:

master:
postgres=# with recursive cte (a) as (select 1 union all select
cte.a+1 from cte where cte.a+1 <= 1000000) select count(*) from cte;
Time: 219.023 ms
Time: 218.828 ms
Time: 219.093 ms

with attached patched:
postgres=# with recursive cte (a) as (select 1 union all select
cte.a+1 from cte where cte.a+1 <= 1000000) select count(*) from cte;
Time: 169.734 ms
Time: 164.841 ms
Time: 169.168 ms

David

diff --git a/src/backend/executor/nodeRecursiveunion.c b/src/backend/executor/nodeRecursiveunion.c
index c7f8a19fa4..8aa7a78d0c 100644
--- a/src/backend/executor/nodeRecursiveunion.c
+++ b/src/backend/executor/nodeRecursiveunion.c
@@ -115,19 +115,26 @@ ExecRecursiveUnion(PlanState *pstate)
 		slot = ExecProcNode(innerPlan);
 		if (TupIsNull(slot))
 		{
+			Tuplestorestate *swaptemp;
+
 			/* Done if there's nothing in the intermediate table */
 			if (node->intermediate_empty)
 				break;
 
-			/* done with old working table ... */
-			tuplestore_end(node->working_table);
+			/*
+			 * Now we let the intermediate table become the work table.  We
+			 * need a fresh intermediate table, so delete the tuples from
+			 * the current working table and use that as the new intermediate
+			 * table.  This saves a round of free/malloc from creating a new
+			 * tuple store.
+			 */
+			tuplestore_clear(node->working_table);
 
-			/* intermediate table becomes working table */
+			swaptemp = node->working_table;
 			node->working_table = node->intermediate_table;
+			node->intermediate_table = swaptemp;
 
-			/* create new empty intermediate table */
-			node->intermediate_table = tuplestore_begin_heap(false, false,
-															 work_mem);
+			/* mark the intermediate table as empty */
 			node->intermediate_empty = true;
 
 			/* reset the recursive term */


Attachments:

  [text/plain] reuse_tuplestore_in_recursive_union.patch.txt (1.3K, ../../CAApHDvr9yW0YRiK8A2J7nvyT8g17YzbSfOviEWrghazKZbHbig@mail.gmail.com/2-reuse_tuplestore_in_recursive_union.patch.txt)
  download | inline diff:
diff --git a/src/backend/executor/nodeRecursiveunion.c b/src/backend/executor/nodeRecursiveunion.c
index c7f8a19fa4..8aa7a78d0c 100644
--- a/src/backend/executor/nodeRecursiveunion.c
+++ b/src/backend/executor/nodeRecursiveunion.c
@@ -115,19 +115,26 @@ ExecRecursiveUnion(PlanState *pstate)
 		slot = ExecProcNode(innerPlan);
 		if (TupIsNull(slot))
 		{
+			Tuplestorestate *swaptemp;
+
 			/* Done if there's nothing in the intermediate table */
 			if (node->intermediate_empty)
 				break;
 
-			/* done with old working table ... */
-			tuplestore_end(node->working_table);
+			/*
+			 * Now we let the intermediate table become the work table.  We
+			 * need a fresh intermediate table, so delete the tuples from
+			 * the current working table and use that as the new intermediate
+			 * table.  This saves a round of free/malloc from creating a new
+			 * tuple store.
+			 */
+			tuplestore_clear(node->working_table);
 
-			/* intermediate table becomes working table */
+			swaptemp = node->working_table;
 			node->working_table = node->intermediate_table;
+			node->intermediate_table = swaptemp;
 
-			/* create new empty intermediate table */
-			node->intermediate_table = tuplestore_begin_heap(false, false,
-															 work_mem);
+			/* mark the intermediate table as empty */
 			node->intermediate_empty = true;
 
 			/* reset the recursive term */


^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 12:12                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 21:20                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-13 06:11                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 09:06                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-13 09:31                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 10:44                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-14 08:12                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-16 11:11                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-16 12:13                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 02:40                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 03:16                                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-17 06:06                                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-18 12:12                                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-18 21:57                                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 00:01                                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-19 00:49                                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
@ 2024-09-19 01:49                                                       ` Tatsuo Ishii <[email protected]>
  2024-09-19 02:11                                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 03:22                                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  0 siblings, 2 replies; 107+ messages in thread

From: Tatsuo Ishii @ 2024-09-19 01:49 UTC (permalink / raw)
  To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers

> That code could be modified to swap the tuplestores and do a
> tuplestore_clear() instead of tuplestore_end() followed by
> tuplestore_begin_heap().
> 
> It's likely worthwhile from a performance point of view.  Here's a
> small test as an example:
> 
> master:
> postgres=# with recursive cte (a) as (select 1 union all select
> cte.a+1 from cte where cte.a+1 <= 1000000) select count(*) from cte;
> Time: 219.023 ms
> Time: 218.828 ms
> Time: 219.093 ms
> 
> with attached patched:
> postgres=# with recursive cte (a) as (select 1 union all select
> cte.a+1 from cte where cte.a+1 <= 1000000) select count(*) from cte;
> Time: 169.734 ms
> Time: 164.841 ms
> Time: 169.168 ms

Impressive result. I also ran your query with count 1000.

without the patch:
Time: 3.655 ms
Time: 4.123 ms
Time: 2.163 ms

wit the patch:
Time: 3.641 ms
Time: 2.356 ms
Time: 2.347 ms

It seems with the patch the performance is slightly better or almost
same. I think the patch improves the performance without sacrificing
the smaller iteration case.

Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 12:12                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 21:20                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-13 06:11                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 09:06                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-13 09:31                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 10:44                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-14 08:12                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-16 11:11                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-16 12:13                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 02:40                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 03:16                                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-17 06:06                                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-18 12:12                                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-18 21:57                                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 00:01                                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-19 00:49                                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 01:49                                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
@ 2024-09-19 02:11                                                         ` David Rowley <[email protected]>
  1 sibling, 0 replies; 107+ messages in thread

From: David Rowley @ 2024-09-19 02:11 UTC (permalink / raw)
  To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers

On Thu, 19 Sept 2024 at 13:49, Tatsuo Ishii <[email protected]> wrote:
> I also ran your query with count 1000.
>
> without the patch:
> Time: 3.655 ms
> Time: 4.123 ms
> Time: 2.163 ms
>
> wit the patch:
> Time: 3.641 ms
> Time: 2.356 ms
> Time: 2.347 ms
>
> It seems with the patch the performance is slightly better or almost
> same. I think the patch improves the performance without sacrificing
> the smaller iteration case.

You might need to use pgbench to get more stable results with such a
small test. If your CPU clocks down when idle, it's not going to clock
up as fast as you might like it to when you give it something to do.

Here's what I got when running 1000 iterations:

$ cat bench.sql
with recursive cte (a) as (select 1 union all select cte.a+1 from cte
where cte.a+1 <= 1000) select count(*) from cte;

master
$ for i in {1..5}; do pgbench -n -f bench.sql -T 10 -M prepared
postgres | grep latency; done
latency average = 0.251 ms
latency average = 0.254 ms
latency average = 0.251 ms
latency average = 0.252 ms
latency average = 0.253 ms

patched
$ for i in {1..5}; do pgbench -n -f bench.sql -T 10 -M prepared
postgres | grep latency; done
latency average = 0.202 ms
latency average = 0.202 ms
latency average = 0.207 ms
latency average = 0.202 ms
latency average = 0.202 ms (~24.2% faster)

David





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 12:12                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 21:20                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-13 06:11                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 09:06                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-13 09:31                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 10:44                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-14 08:12                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-16 11:11                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-16 12:13                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 02:40                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 03:16                                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-17 06:06                                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-18 12:12                                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-18 21:57                                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 00:01                                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-19 00:49                                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 01:49                                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
@ 2024-09-19 03:22                                                         ` David Rowley <[email protected]>
  2024-09-19 04:21                                                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  1 sibling, 1 reply; 107+ messages in thread

From: David Rowley @ 2024-09-19 03:22 UTC (permalink / raw)
  To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers

On Thu, 19 Sept 2024 at 13:49, Tatsuo Ishii <[email protected]> wrote:
>
> > That code could be modified to swap the tuplestores and do a
> > tuplestore_clear() instead of tuplestore_end() followed by
> > tuplestore_begin_heap().
> >
> Impressive result. I also ran your query with count 1000.

I've pushed that patch. That should now unblock you on the
nodeRecursiveunion.c telemetry.

David





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 12:12                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 21:20                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-13 06:11                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 09:06                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-13 09:31                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 10:44                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-14 08:12                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-16 11:11                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-16 12:13                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 02:40                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 03:16                                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-17 06:06                                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-18 12:12                                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-18 21:57                                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 00:01                                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-19 00:49                                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 01:49                                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-19 03:22                                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
@ 2024-09-19 04:21                                                           ` Tatsuo Ishii <[email protected]>
  2024-09-19 05:37                                                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: Tatsuo Ishii @ 2024-09-19 04:21 UTC (permalink / raw)
  To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers

>> > That code could be modified to swap the tuplestores and do a
>> > tuplestore_clear() instead of tuplestore_end() followed by
>> > tuplestore_begin_heap().
>> >
>> Impressive result. I also ran your query with count 1000.
> 
> I've pushed that patch. That should now unblock you on the
> nodeRecursiveunion.c telemetry.

Thanks. Attached is a patch for CTE scan, table function scan and
recursive union scan nodes.

Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp


^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 12:12                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 21:20                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-13 06:11                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 09:06                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-13 09:31                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 10:44                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-14 08:12                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-16 11:11                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-16 12:13                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 02:40                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 03:16                                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-17 06:06                                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-18 12:12                                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-18 21:57                                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 00:01                                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-19 00:49                                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 01:49                                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-19 03:22                                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 04:21                                                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
@ 2024-09-19 05:37                                                             ` David Rowley <[email protected]>
  2024-09-19 10:17                                                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: David Rowley @ 2024-09-19 05:37 UTC (permalink / raw)
  To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers

On Thu, 19 Sept 2024 at 16:21, Tatsuo Ishii <[email protected]> wrote:
> Thanks. Attached is a patch for CTE scan, table function scan and
> recursive union scan nodes.

1. It's probably a minor detail, but in show_recursive_union_info(), I
don't think the tuplestores can ever be NULL.

+ if (working_table != NULL)
+ tuplestore_get_stats(working_table, &tempStorageType, &tempSpaceUsed);
+
+ if (intermediate_table != NULL)
+ tuplestore_get_stats(intermediate_table, &maxStorageType, &maxSpaceUsed);

I added the NULL tests for the Materialize case as the tuplestore is
created in ExecMaterial() rather than ExecInitMaterial(). For the two
tuplestorestates above, they're both created in
ExecInitRecursiveUnion().

2. I imagined you'd always do maxSpaceUsed += tempSpaceUsed; or
maxSpaceUsedKB = BYTES_TO_KILOBYTES(maxSpaceUsed + tempSpaceUsed);

+ if (tempSpaceUsed > maxSpaceUsed)
+ {
+ maxStorageType = tempStorageType;
+ maxSpaceUsed = tempSpaceUsed;
+ }

Why do you think the the space used by the smaller tuplestore should
be ignored in the storage size output?

David





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 12:12                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 21:20                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-13 06:11                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 09:06                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-13 09:31                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 10:44                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-14 08:12                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-16 11:11                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-16 12:13                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 02:40                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 03:16                                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-17 06:06                                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-18 12:12                                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-18 21:57                                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 00:01                                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-19 00:49                                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 01:49                                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-19 03:22                                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 04:21                                                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-19 05:37                                                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
@ 2024-09-19 10:17                                                               ` Tatsuo Ishii <[email protected]>
  2024-09-22 23:55                                                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: Tatsuo Ishii @ 2024-09-19 10:17 UTC (permalink / raw)
  To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers

> 1. It's probably a minor detail, but in show_recursive_union_info(), I
> don't think the tuplestores can ever be NULL.
> 
> + if (working_table != NULL)
> + tuplestore_get_stats(working_table, &tempStorageType, &tempSpaceUsed);
> +
> + if (intermediate_table != NULL)
> + tuplestore_get_stats(intermediate_table, &maxStorageType, &maxSpaceUsed);
> 
> I added the NULL tests for the Materialize case as the tuplestore is
> created in ExecMaterial() rather than ExecInitMaterial(). For the two
> tuplestorestates above, they're both created in
> ExecInitRecursiveUnion().

You are right. Also I checked other Exec* in nodeRecursiveunion.c and
did not find any place where working_table and intermediate_table are
set to NULL.

> 2. I imagined you'd always do maxSpaceUsed += tempSpaceUsed; or
> maxSpaceUsedKB = BYTES_TO_KILOBYTES(maxSpaceUsed + tempSpaceUsed);
> 
> + if (tempSpaceUsed > maxSpaceUsed)
> + {
> + maxStorageType = tempStorageType;
> + maxSpaceUsed = tempSpaceUsed;
> + }
> 
> Why do you think the the space used by the smaller tuplestore should
> be ignored in the storage size output?

I thought about the case when the two tuplestores have different
storage types. But I remember that we already use disk storage type
even if the storage type was changed from disk to memory[1]. So
probably we don't need to much worry about the storage kind difference
in the two tuplestores.

Attached patch fixes 1 & 2.

[1] https://www.postgresql.org/message-id/CAExHW5vRPRLvsZYLmNGcDLkPDWDHXGSWYjox-to-OsCVFETd3w%40mail.gma...

Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp


^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 12:12                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 21:20                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-13 06:11                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 09:06                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-13 09:31                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 10:44                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-14 08:12                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-16 11:11                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-16 12:13                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 02:40                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 03:16                                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-17 06:06                                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-18 12:12                                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-18 21:57                                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 00:01                                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-19 00:49                                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 01:49                                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-19 03:22                                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 04:21                                                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-19 05:37                                                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 10:17                                                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
@ 2024-09-22 23:55                                                                 ` David Rowley <[email protected]>
  2024-09-23 06:28                                                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: David Rowley @ 2024-09-22 23:55 UTC (permalink / raw)
  To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers

On Thu, 19 Sept 2024 at 22:17, Tatsuo Ishii <[email protected]> wrote:
> Attached patch fixes 1 & 2.

I looked at this and thought that one thing you might want to consider
is adjusting show_storage_info() to accept the size and type
parameters so you don't have to duplicate the formatting code in
show_recursive_union_info().

The first of the new tests also isn't testing what you want it to
test. Maybe you could add a "materialized" in there to stop the CTE
being inlined:

explain (analyze,costs off) with w(n) as materialized (select n from
generate_series(1,10) a(n)) select sum(n) from w

Also, I'm on the fence about if the new tests are worthwhile. I won't
object to them, however. I just wanted to note that most of the
complexity is in tuplestore.c of which there's already coverage for.
The test's value is reduced by the fact that most of the interesting
details have to be masked out due to possible platform variations in
the number of bytes. Really the new tests are only testing that we
display the storage details and maybe that the storage type came out
as expected. It seems unlikely these would get broken. I'd say it's
committers preference, however. I just wanted to add my thoughts. You
have to offset the value against the fact that the expected output is
likely to change over the years which adds to the burden of making
changes to the EXPLAIN output.

David





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 12:12                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 21:20                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-13 06:11                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 09:06                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-13 09:31                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 10:44                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-14 08:12                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-16 11:11                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-16 12:13                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 02:40                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 03:16                                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-17 06:06                                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-18 12:12                                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-18 21:57                                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 00:01                                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-19 00:49                                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 01:49                                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-19 03:22                                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 04:21                                                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-19 05:37                                                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 10:17                                                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-22 23:55                                                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
@ 2024-09-23 06:28                                                                   ` Tatsuo Ishii <[email protected]>
  2024-09-23 07:05                                                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: Tatsuo Ishii @ 2024-09-23 06:28 UTC (permalink / raw)
  To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers

> I looked at this and thought that one thing you might want to consider
> is adjusting show_storage_info() to accept the size and type
> parameters so you don't have to duplicate the formatting code in
> show_recursive_union_info().

I agree and made necessary changes. See attached v4 patches.

> The first of the new tests also isn't testing what you want it to
> test. Maybe you could add a "materialized" in there to stop the CTE
> being inlined:
> 
> explain (analyze,costs off) with w(n) as materialized (select n from
> generate_series(1,10) a(n)) select sum(n) from w
> 
> Also, I'm on the fence about if the new tests are worthwhile. I won't
> object to them, however. I just wanted to note that most of the
> complexity is in tuplestore.c of which there's already coverage for.
> The test's value is reduced by the fact that most of the interesting
> details have to be masked out due to possible platform variations in
> the number of bytes. Really the new tests are only testing that we
> display the storage details and maybe that the storage type came out
> as expected. It seems unlikely these would get broken. I'd say it's
> committers preference, however. I just wanted to add my thoughts. You
> have to offset the value against the fact that the expected output is
> likely to change over the years which adds to the burden of making
> changes to the EXPLAIN output.

After thinking more, I lean toward to your opinion. The new tests do
not give big value, but on the other hand they could become a burden
over the years. I do not include the new tests in the v4 patches.

Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp


^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 12:12                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 21:20                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-13 06:11                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 09:06                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-13 09:31                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 10:44                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-14 08:12                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-16 11:11                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-16 12:13                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 02:40                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 03:16                                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-17 06:06                                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-18 12:12                                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-18 21:57                                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 00:01                                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-19 00:49                                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 01:49                                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-19 03:22                                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 04:21                                                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-19 05:37                                                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 10:17                                                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-22 23:55                                                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-23 06:28                                                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
@ 2024-09-23 07:05                                                                     ` David Rowley <[email protected]>
  2024-09-23 07:52                                                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  0 siblings, 1 reply; 107+ messages in thread

From: David Rowley @ 2024-09-23 07:05 UTC (permalink / raw)
  To: Tatsuo Ishii <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers

On Mon, 23 Sept 2024 at 18:28, Tatsuo Ishii <[email protected]> wrote:
> I agree and made necessary changes. See attached v4 patches.

Looks good to me.

David





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

* Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN
  2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 12:12                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-12 21:20                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-13 06:11                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 09:06                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-13 09:31                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-13 10:44                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-14 08:12                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-16 11:11                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
  2024-09-16 12:13                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 02:40                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-17 03:16                                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-17 06:06                                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-18 12:12                                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-18 21:57                                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 00:01                                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-19 00:49                                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 01:49                                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-19 03:22                                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 04:21                                                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-19 05:37                                                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-19 10:17                                                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-22 23:55                                                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
  2024-09-23 06:28                                                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
  2024-09-23 07:05                                                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
@ 2024-09-23 07:52                                                                       ` Tatsuo Ishii <[email protected]>
  0 siblings, 0 replies; 107+ messages in thread

From: Tatsuo Ishii @ 2024-09-23 07:52 UTC (permalink / raw)
  To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers

> On Mon, 23 Sept 2024 at 18:28, Tatsuo Ishii <[email protected]> wrote:
>> I agree and made necessary changes. See attached v4 patches.
> 
> Looks good to me.

Thank you for the review! I have pushed the patch.

Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp





^ permalink  raw  reply  [nested|flat] 107+ messages in thread

end of thread, other threads:[~2024-09-23 07:52 UTC | newest]

Thread overview: 107+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-01-21 16:06 [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v2] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v3] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2021-01-21 16:06 [PATCH v1] Allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. Julien Rouhaud <[email protected]>
2024-09-05 04:52 Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
2024-09-05 05:38 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
2024-09-06 04:21   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
2024-09-06 05:07     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
2024-09-06 06:02       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
2024-09-06 07:07         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
2024-09-06 07:50           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
2024-09-06 13:51           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
2024-09-09 10:12             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
2024-09-12 02:03             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
2024-09-12 02:28               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
2024-09-12 02:42                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
2024-09-12 05:07                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
2024-09-12 05:09                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
2024-09-12 12:12                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
2024-09-12 21:20                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
2024-09-13 06:11                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
2024-09-13 08:57                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Maxim Orlov <[email protected]>
2024-09-13 09:06                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
2024-09-13 09:31                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
2024-09-13 10:44                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
2024-09-14 08:12                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
2024-09-16 11:11                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Ashutosh Bapat <[email protected]>
2024-09-16 12:13                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
2024-09-17 02:40                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
2024-09-17 03:16                                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
2024-09-17 06:06                                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
2024-09-18 12:12                                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
2024-09-18 21:57                                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
2024-09-19 00:01                                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
2024-09-19 00:49                                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
2024-09-19 01:49                                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
2024-09-19 02:11                                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
2024-09-19 03:22                                                         ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
2024-09-19 04:21                                                           ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
2024-09-19 05:37                                                             ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
2024-09-19 10:17                                                               ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
2024-09-22 23:55                                                                 ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
2024-09-23 06:28                                                                   ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>
2024-09-23 07:05                                                                     ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN David Rowley <[email protected]>
2024-09-23 07:52                                                                       ` Re: Add memory/disk usage for WindowAgg nodes in EXPLAIN Tatsuo Ishii <[email protected]>

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox