From: Mark Dilger <mark.dilger@enterprisedb.com>
To: Robert Haas <robertmhaas@gmail.com>
To: Peter Geoghegan <pg@bowt.ie>
To: Andrey M. Borodin <x4mmm@yandex-team.ru>
To: Stephen Frost <sfrost@snowman.net>
To: Michael Paquier <michael@paquier.xyz>
Cc: Amul Sul <sulamul@gmail.com>
Cc: Dilip Kumar <dilipbalaut@gmail.com>
Cc: Andres Freund <andres@anarazel.de>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: new heapcheck contrib module
Date: Tue, 6 Oct 2020 16:20:22 -0700
Message-ID: <3894CAFE-E391-4EE0-9599-37C5D90BE740@enterprisedb.com> (raw)
In-Reply-To: <A4C9E0AE-46BA-457F-A445-4527E5AA5E3C@enterprisedb.com>
References: <12ED3DA8-25F0-4B68-937D-D907CFBF08E7@enterprisedb.com>
<20200420194210.sevnpi4lyghoanxi@alap3.anarazel.de>
<23E60C3C-C20A-4508-84AC-E0CEA59E784A@enterprisedb.com>
<CA+TgmobAm8vB1MNQ1taO1et+HxTN8ULnQvEG+grNDqNQTBjbtA@mail.gmail.com>
<F845AC50-A479-43F3-ADFB-861A4D2AD287@enterprisedb.com>
<5F743835-3399-419C-8324-2D424237E999@enterprisedb.com>
<CAFiTN-syyHc3jZoou51v0SR8z0POoNfktqEO6MaGig4YS8mosA@mail.gmail.com>
<E6F4258A-B661-4F2C-8A35-278C9436930A@enterprisedb.com>
<CAFiTN-s-VX3DLfqv7OOo0zGCTOZ3xVU9UZ6GgzBZdJTPNnYFNg@mail.gmail.com>
<885D2EC9-7D06-4903-A17D-72C184A1E1C5@enterprisedb.com>
<CAFiTN-uYg_PiZ854=5R-D9skratapAHKhaDAG7fhJv-Vn5UhaQ@mail.gmail.com>
<5CC85F28-F210-47AC-8DA8-EC4E82AF3FD5@enterprisedb.com>
<CAFiTN-sP5iFJGERTq4a4OLTCzjZKhLC5P6_k2bxQKXnuvdmzGA@mail.gmail.com>
<CAFiTN-tTg0R5AHvXC1ro9XphroX1wUtc5t91hU_+7DeGTpE5VQ@mail.gmail.com>
<0C0349EB-DD85-4DA8-82C7-DF70AB714592@enterprisedb.com>
<CAFiTN-uD3WSeggU20yU6voGou6BDb-cQcfkE8B1rAzWKZVK-Bw@mail.gmail.com>
<BA6E870F-241C-4DDB-B92D-DB39E679C7D4@enterprisedb.com>
<CA+TgmobQUdfM2Q7WAR606RR2YehCZ5GHRKBc-VWfRj-1ZUKqDA@mail.gmail.com>
<C4A3E5AE-CBB2-4E3F-8BAE-29EE216762C2@enterprisedb.com>
<CAAJ_b96_Xr2a+D0j8xKbOnCzURcVQm0V2DS+T_pfc+wt3BpL8g@mail.gmail.com>
<9DF93B83-FE46-4EC4-A99D-77A60729FD2D@enterprisedb.com>
<CA+TgmoZT3kCOaKq6=m5wHGKh_9XQDbCC1LZXL6w_+NMB-n1+mw@mail.gmail.com>
<CAAJ_b94hsrF4x7UKQj03416fAEXJorUpZ9WhsZZjqgv-4+Wg9A@mail.gmail.com>
<50F6468F-5A5E-412A-86EC-DF24B549D733@enterprisedb.com>
<CAAJ_b94+A1Gk5r5sBYbYM2K5hH_gyp_o=+gwJuJhn5ZEm5_rqw@mail.gmail.com>
<CAAJ_b94Wh_7kM9JuJk7wnNsYSNPdSzz35cOEaE_ZRnPcqvupCg@mail.gmail.com>
<7D603B53-8078-4AF0-851B-803FF4416501@enterprisedb.com>
<CA+TgmoZbPtTgkQBGoco39uqKUWuv=0+TOkiNcCuqWWvUkJVuxQ@mail.gmail.com>
<A4C9E0AE-46BA-457F-A445-4527E5AA5E3C@enterprisedb.com>
> On Oct 5, 2020, at 5:24 PM, Mark Dilger <mark.dilger@enterprisedb.com> wrote:
>
> - This version does not change clog handling, which leaves Andrey's concern unaddressed. Peter also showed some support for (or perhaps just a lack of opposition to) doing more of what Andrey suggests. I may come back to this issue, depending on time available and further feedback.
Attached is a patch set that includes the clog handling as discussed. The 0001 and 0002 are effectively unchanged since version 16 posted yesterday, but this now includes 0003 which creates a non-throwing interface to clog, and 0004 which uses the non-throwing interface from within amcheck's heap checking functions.
I think this is a pretty good sketch for discussion, though I am unsatisfied with the lack of regression test coverage of verify_heapam in the presence of clog truncation. I was hoping to have that as part of v17, but since it is taking a bit longer than I anticipated, I'll have to come back with that in a later patch.
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:download all (4 files, .tar.gz)v17-0001-Adding-function-verify_heapam-to-amcheck-module.patch (application/octet-stream, 88.2K) download
From 19f858fb20f29cd716a0ce55f65193a0bde7c0c8 Mon Sep 17 00:00:00 2001
From: Mark Dilger <mark.dilger@enterprisedb.com>
Date: Mon, 5 Oct 2020 15:42:18 -0700
Subject: [PATCH v17 1/4] Adding function verify_heapam to amcheck module
Adding new function verify_heapam for checking a heap relation and
optionally its associated toast relation, if any.
---
contrib/amcheck/Makefile | 7 +-
contrib/amcheck/amcheck--1.2--1.3.sql | 30 +
contrib/amcheck/amcheck.control | 2 +-
contrib/amcheck/expected/check_heap.out | 200 +++
contrib/amcheck/sql/check_heap.sql | 123 ++
contrib/amcheck/t/001_verify_heapam.pl | 242 ++++
contrib/amcheck/verify_heapam.c | 1537 +++++++++++++++++++++++
doc/src/sgml/amcheck.sgml | 237 +++-
src/backend/access/heap/hio.c | 11 +
src/backend/access/transam/multixact.c | 19 +
src/include/access/multixact.h | 1 +
src/tools/pgindent/typedefs.list | 4 +
12 files changed, 2404 insertions(+), 9 deletions(-)
create mode 100644 contrib/amcheck/amcheck--1.2--1.3.sql
create mode 100644 contrib/amcheck/expected/check_heap.out
create mode 100644 contrib/amcheck/sql/check_heap.sql
create mode 100644 contrib/amcheck/t/001_verify_heapam.pl
create mode 100644 contrib/amcheck/verify_heapam.c
diff --git a/contrib/amcheck/Makefile b/contrib/amcheck/Makefileindex a2b1b1036b..b82f221e50 100644--- a/contrib/amcheck/Makefile+++ b/contrib/amcheck/Makefile@@ -3,13 +3,16 @@
MODULE_big = amcheck
OBJS = \
$(WIN32RES) \
+ verify_heapam.o \
verify_nbtree.o
EXTENSION = amcheck
-DATA = amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql+DATA = amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
PGFILEDESC = "amcheck - function for verifying relation integrity"
-REGRESS = check check_btree+REGRESS = check check_btree check_heap++TAP_TESTS = 1
ifdef USE_PGXS
PG_CONFIG = pg_config
diff --git a/contrib/amcheck/amcheck--1.2--1.3.sql b/contrib/amcheck/amcheck--1.2--1.3.sqlnew file mode 100644index 0000000000..7237ab738c--- /dev/null+++ b/contrib/amcheck/amcheck--1.2--1.3.sql@@ -0,0 +1,30 @@+/* contrib/amcheck/amcheck--1.2--1.3.sql */++-- complain if script is sourced in psql, rather than via CREATE EXTENSION+\echo Use "ALTER EXTENSION amcheck UPDATE TO '1.3'" to load this file. \quit++--+-- verify_heapam()+--+CREATE FUNCTION verify_heapam(relation regclass,+ on_error_stop boolean default false,+ check_toast boolean default false,+ skip text default 'none',+ startblock bigint default null,+ endblock bigint default null,+ blkno OUT bigint,+ offnum OUT integer,+ attnum OUT integer,+ msg OUT text)+RETURNS SETOF record+AS 'MODULE_PATHNAME', 'verify_heapam'+LANGUAGE C;++-- Don't want this to be available to public+REVOKE ALL ON FUNCTION verify_heapam(regclass,+ boolean,+ boolean,+ text,+ bigint,+ bigint)+FROM PUBLIC;diff --git a/contrib/amcheck/amcheck.control b/contrib/amcheck/amcheck.controlindex c6e310046d..ab50931f75 100644--- a/contrib/amcheck/amcheck.control+++ b/contrib/amcheck/amcheck.control@@ -1,5 +1,5 @@
# amcheck extension
comment = 'functions for verifying relation integrity'
-default_version = '1.2'+default_version = '1.3'
module_pathname = '$libdir/amcheck'
relocatable = true
diff --git a/contrib/amcheck/expected/check_heap.out b/contrib/amcheck/expected/check_heap.outnew file mode 100644index 0000000000..41cdc6435c--- /dev/null+++ b/contrib/amcheck/expected/check_heap.out@@ -0,0 +1,200 @@+CREATE TABLE heaptest (a integer, b text);+REVOKE ALL ON heaptest FROM PUBLIC;+-- Check that invalid skip option is rejected+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'rope');+ERROR: invalid skip option+HINT: Valid skip options are "all-visible", "all-frozen", and "none".+-- Check specifying invalid block ranges when verifying an empty table+SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 0, endblock := 0);+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 5, endblock := 8);+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++-- Check that valid options are not rejected nor corruption reported+-- for an empty table, and that skip enum-like parameter is case-insensitive+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'none');+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-frozen');+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-visible');+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'None');+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'All-Frozen');+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'All-Visible');+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'NONE');+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'ALL-FROZEN');+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'ALL-VISIBLE');+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++-- Add some data so subsequent tests are not entirely trivial+INSERT INTO heaptest (a, b)+ (SELECT gs, repeat('x', gs)+ FROM generate_series(1,50) gs);+-- Check that valid options are not rejected nor corruption reported+-- for a non-empty table+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'none');+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-frozen');+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-visible');+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 0, endblock := 0);+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++CREATE ROLE regress_heaptest_role;+-- verify permissions are checked (error due to function not callable)+SET ROLE regress_heaptest_role;+SELECT * FROM verify_heapam(relation := 'heaptest');+ERROR: permission denied for function verify_heapam+RESET ROLE;+GRANT EXECUTE ON FUNCTION verify_heapam(regclass, boolean, boolean, text, bigint, bigint) TO regress_heaptest_role;+-- verify permissions are checked (error due to no select privileges on relation)+SET ROLE regress_heaptest_role;+SELECT * FROM verify_heapam(relation := 'heaptest');+ERROR: permission denied for table heaptest+RESET ROLE;+GRANT SELECT ON heaptest TO regress_heaptest_role;+-- verify permissions are now sufficient+SET ROLE regress_heaptest_role;+SELECT * FROM verify_heapam(relation := 'heaptest');+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++RESET ROLE;+-- Check specifying invalid block ranges when verifying a non-empty table.+SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 0, endblock := 10000);+ERROR: ending block number must be between 0 and 0+SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 10000, endblock := 11000);+ERROR: starting block number must be between 0 and 0+-- Vacuum freeze to change the xids encountered in subsequent tests+VACUUM FREEZE heaptest;+-- Check that valid options are not rejected nor corruption reported+-- for a non-empty frozen table+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'none');+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-frozen');+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-visible');+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 0, endblock := 0);+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++-- Check that partitioned tables (the parent ones) which don't have visibility+-- maps are rejected+CREATE TABLE test_partitioned (a int, b text default repeat('x', 5000))+ PARTITION BY list (a);+SELECT * FROM verify_heapam('test_partitioned',+ startblock := NULL,+ endblock := NULL);+ERROR: "test_partitioned" is not a table, materialized view, or TOAST table+-- Check that valid options are not rejected nor corruption reported+-- for an empty partition table (the child one)+CREATE TABLE test_partition partition OF test_partitioned FOR VALUES IN (1);+SELECT * FROM verify_heapam('test_partition',+ startblock := NULL,+ endblock := NULL);+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++-- Check that valid options are not rejected nor corruption reported+-- for a non-empty partition table (the child one)+INSERT INTO test_partitioned (a) (SELECT 1 FROM generate_series(1,1000) gs);+SELECT * FROM verify_heapam('test_partition',+ startblock := NULL,+ endblock := NULL);+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++-- Check that indexes are rejected+CREATE INDEX test_index ON test_partition (a);+SELECT * FROM verify_heapam('test_index',+ startblock := NULL,+ endblock := NULL);+ERROR: "test_index" is not a table, materialized view, or TOAST table+-- Check that views are rejected+CREATE VIEW test_view AS SELECT 1;+SELECT * FROM verify_heapam('test_view',+ startblock := NULL,+ endblock := NULL);+ERROR: "test_view" is not a table, materialized view, or TOAST table+-- Check that sequences are rejected+CREATE SEQUENCE test_sequence;+SELECT * FROM verify_heapam('test_sequence',+ startblock := NULL,+ endblock := NULL);+ERROR: "test_sequence" is not a table, materialized view, or TOAST table+-- Check that foreign tables are rejected+CREATE FOREIGN DATA WRAPPER dummy;+CREATE SERVER dummy_server FOREIGN DATA WRAPPER dummy;+CREATE FOREIGN TABLE test_foreign_table () SERVER dummy_server;+SELECT * FROM verify_heapam('test_foreign_table',+ startblock := NULL,+ endblock := NULL);+ERROR: "test_foreign_table" is not a table, materialized view, or TOAST table+-- cleanup+DROP TABLE heaptest;+DROP TABLE test_partition;+DROP TABLE test_partitioned;+DROP OWNED BY regress_heaptest_role; -- permissions+DROP ROLE regress_heaptest_role;diff --git a/contrib/amcheck/sql/check_heap.sql b/contrib/amcheck/sql/check_heap.sqlnew file mode 100644index 0000000000..c8397a46f0--- /dev/null+++ b/contrib/amcheck/sql/check_heap.sql@@ -0,0 +1,123 @@+CREATE TABLE heaptest (a integer, b text);+REVOKE ALL ON heaptest FROM PUBLIC;++-- Check that invalid skip option is rejected+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'rope');++-- Check specifying invalid block ranges when verifying an empty table+SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 0, endblock := 0);+SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 5, endblock := 8);++-- Check that valid options are not rejected nor corruption reported+-- for an empty table, and that skip enum-like parameter is case-insensitive+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'none');+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-frozen');+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-visible');+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'None');+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'All-Frozen');+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'All-Visible');+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'NONE');+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'ALL-FROZEN');+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'ALL-VISIBLE');++-- Add some data so subsequent tests are not entirely trivial+INSERT INTO heaptest (a, b)+ (SELECT gs, repeat('x', gs)+ FROM generate_series(1,50) gs);++-- Check that valid options are not rejected nor corruption reported+-- for a non-empty table+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'none');+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-frozen');+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-visible');+SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 0, endblock := 0);++CREATE ROLE regress_heaptest_role;++-- verify permissions are checked (error due to function not callable)+SET ROLE regress_heaptest_role;+SELECT * FROM verify_heapam(relation := 'heaptest');+RESET ROLE;++GRANT EXECUTE ON FUNCTION verify_heapam(regclass, boolean, boolean, text, bigint, bigint) TO regress_heaptest_role;++-- verify permissions are checked (error due to no select privileges on relation)+SET ROLE regress_heaptest_role;+SELECT * FROM verify_heapam(relation := 'heaptest');+RESET ROLE;++GRANT SELECT ON heaptest TO regress_heaptest_role;++-- verify permissions are now sufficient+SET ROLE regress_heaptest_role;+SELECT * FROM verify_heapam(relation := 'heaptest');+RESET ROLE;++-- Check specifying invalid block ranges when verifying a non-empty table.+SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 0, endblock := 10000);+SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 10000, endblock := 11000);++-- Vacuum freeze to change the xids encountered in subsequent tests+VACUUM FREEZE heaptest;++-- Check that valid options are not rejected nor corruption reported+-- for a non-empty frozen table+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'none');+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-frozen');+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-visible');+SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 0, endblock := 0);++-- Check that partitioned tables (the parent ones) which don't have visibility+-- maps are rejected+CREATE TABLE test_partitioned (a int, b text default repeat('x', 5000))+ PARTITION BY list (a);+SELECT * FROM verify_heapam('test_partitioned',+ startblock := NULL,+ endblock := NULL);++-- Check that valid options are not rejected nor corruption reported+-- for an empty partition table (the child one)+CREATE TABLE test_partition partition OF test_partitioned FOR VALUES IN (1);+SELECT * FROM verify_heapam('test_partition',+ startblock := NULL,+ endblock := NULL);++-- Check that valid options are not rejected nor corruption reported+-- for a non-empty partition table (the child one)+INSERT INTO test_partitioned (a) (SELECT 1 FROM generate_series(1,1000) gs);+SELECT * FROM verify_heapam('test_partition',+ startblock := NULL,+ endblock := NULL);++-- Check that indexes are rejected+CREATE INDEX test_index ON test_partition (a);+SELECT * FROM verify_heapam('test_index',+ startblock := NULL,+ endblock := NULL);++-- Check that views are rejected+CREATE VIEW test_view AS SELECT 1;+SELECT * FROM verify_heapam('test_view',+ startblock := NULL,+ endblock := NULL);++-- Check that sequences are rejected+CREATE SEQUENCE test_sequence;+SELECT * FROM verify_heapam('test_sequence',+ startblock := NULL,+ endblock := NULL);++-- Check that foreign tables are rejected+CREATE FOREIGN DATA WRAPPER dummy;+CREATE SERVER dummy_server FOREIGN DATA WRAPPER dummy;+CREATE FOREIGN TABLE test_foreign_table () SERVER dummy_server;+SELECT * FROM verify_heapam('test_foreign_table',+ startblock := NULL,+ endblock := NULL);++-- cleanup+DROP TABLE heaptest;+DROP TABLE test_partition;+DROP TABLE test_partitioned;+DROP OWNED BY regress_heaptest_role; -- permissions+DROP ROLE regress_heaptest_role;diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.plnew file mode 100644index 0000000000..e7526c17b8--- /dev/null+++ b/contrib/amcheck/t/001_verify_heapam.pl@@ -0,0 +1,242 @@+use strict;+use warnings;++use PostgresNode;+use TestLib;++use Test::More tests => 65;++my ($node, $result);++#+# Test set-up+#+$node = get_new_node('test');+$node->init;+$node->append_conf('postgresql.conf', 'autovacuum=off');+$node->start;+$node->safe_psql('postgres', q(CREATE EXTENSION amcheck));++#+# Check a table with data loaded but no corruption, freezing, etc.+#+fresh_test_table('test');+check_all_options_uncorrupted('test', 'plain');++#+# Check a corrupt table+#+fresh_test_table('test');+corrupt_first_page('test');+detects_corruption(+ "verify_heapam('test')",+ "plain corrupted table");+detects_corruption(+ "verify_heapam('test', skip := 'all-visible')",+ "plain corrupted table skipping all-visible");+detects_corruption(+ "verify_heapam('test', skip := 'all-frozen')",+ "plain corrupted table skipping all-frozen");+detects_corruption(+ "verify_heapam('test', check_toast := false)",+ "plain corrupted table skipping toast");+detects_corruption(+ "verify_heapam('test', startblock := 0, endblock := 0)",+ "plain corrupted table checking only block zero");++#+# Check a corrupt table with all-frozen data+#+fresh_test_table('test');+$node->safe_psql('postgres', q(VACUUM FREEZE test));+corrupt_first_page('test');+detects_corruption(+ "verify_heapam('test')",+ "all-frozen corrupted table");+detects_no_corruption(+ "verify_heapam('test', skip := 'all-frozen')",+ "all-frozen corrupted table skipping all-frozen");++#+# Check a corrupt table with corrupt page header+#+fresh_test_table('test');+corrupt_first_page_and_header('test');+detects_corruption(+ "verify_heapam('test')",+ "corrupted test table with bad page header");++#+# Check an uncorrupted table with corrupt toast page header+#+fresh_test_table('test');+my $toast = get_toast_for('test');+corrupt_first_page_and_header($toast);+detects_corruption(+ "verify_heapam('test', check_toast := true)",+ "table with corrupted toast page header checking toast");+detects_no_corruption(+ "verify_heapam('test', check_toast := false)",+ "table with corrupted toast page header skipping toast");+detects_corruption(+ "verify_heapam('$toast')",+ "corrupted toast page header");++#+# Check an uncorrupted table with corrupt toast+#+fresh_test_table('test');+$toast = get_toast_for('test');+corrupt_first_page($toast);+detects_corruption(+ "verify_heapam('test', check_toast := true)",+ "table with corrupted toast checking toast");+detects_no_corruption(+ "verify_heapam('test', check_toast := false)",+ "table with corrupted toast skipping toast");+detects_corruption(+ "verify_heapam('$toast')",+ "corrupted toast table");++#+# Check an uncorrupted all-frozen table with corrupt toast+#+fresh_test_table('test');+$node->safe_psql('postgres', q(VACUUM FREEZE test));+$toast = get_toast_for('test');+corrupt_first_page($toast);+detects_corruption(+ "verify_heapam('test', check_toast := true)",+ "all-frozen table with corrupted toast checking toast");+detects_no_corruption(+ "verify_heapam('test', check_toast := false)",+ "all-frozen table with corrupted toast skipping toast");+detects_corruption(+ "verify_heapam('$toast')",+ "corrupted toast table of all-frozen table");++# Returns the filesystem path for the named relation.+sub relation_filepath+{+ my ($relname) = @_;++ my $pgdata = $node->data_dir;+ my $rel = $node->safe_psql('postgres',+ qq(SELECT pg_relation_filepath('$relname')));+ die "path not found for relation $relname" unless defined $rel;+ return "$pgdata/$rel";+}++# Returns the fully qualified name of the toast table for the named relation+sub get_toast_for+{+ my ($relname) = @_;+ $node->safe_psql('postgres', qq(+ SELECT 'pg_toast.' || t.relname+ FROM pg_catalog.pg_class c, pg_catalog.pg_class t+ WHERE c.relname = '$relname'+ AND c.reltoastrelid = t.oid));+}++# (Re)create and populate a test table of the given name.+sub fresh_test_table+{+ my ($relname) = @_;+ $node->safe_psql('postgres', qq(+ DROP TABLE IF EXISTS $relname CASCADE;+ CREATE TABLE $relname (a integer, b text);+ ALTER TABLE $relname SET (autovacuum_enabled=false);+ ALTER TABLE $relname ALTER b SET STORAGE external;+ INSERT INTO $relname (a, b)+ (SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);+ ));+}++# Stops the test node, corrupts the first page of the named relation, and+# restarts the node.+sub corrupt_first_page_internal+{+ my ($relname, $corrupt_header) = @_;+ my $relpath = relation_filepath($relname);++ $node->stop;+ my $fh;+ open($fh, '+<', $relpath);+ binmode $fh;++ # If we corrupt the header, postgres won't allow the page into the buffer.+ syswrite($fh, '\xFF\xFF\xFF\xFF', 8) if ($corrupt_header);++ # Corrupt at least the line pointers. Exactly what this corrupts will+ # depend on the page, as it may run past the line pointers into the user+ # data. We stop short of writing 2048 bytes (2k), the smallest supported+ # page size, as we don't want to corrupt the next page.+ seek($fh, 32, 0);+ syswrite($fh, '\x77\x77\x77\x77', 500);+ close($fh);+ $node->start;+}++sub corrupt_first_page+{+ corrupt_first_page_internal($_[0], undef);+}++sub corrupt_first_page_and_header+{+ corrupt_first_page_internal($_[0], 1);+}++sub detects_corruption+{+ my ($function, $testname) = @_;++ my $result = $node->safe_psql('postgres',+ qq(SELECT COUNT(*) > 0 FROM $function));+ is($result, 't', $testname);+}++sub detects_no_corruption+{+ my ($function, $testname) = @_;++ my $result = $node->safe_psql('postgres',+ qq(SELECT COUNT(*) = 0 FROM $function));+ is($result, 't', $testname);+}++# Check various options are stable (don't abort) and do not report corruption+# when running verify_heapam on an uncorrupted test table.+#+# The relname *must* be an uncorrupted table, or this will fail.+#+# The prefix is used to identify the test, along with the options,+# and should be unique.+sub check_all_options_uncorrupted+{+ my ($relname, $prefix) = @_;+ for my $stop (qw(true false))+ {+ for my $check_toast (qw(true false))+ {+ for my $skip ("'none'", "'all-frozen'", "'all-visible'")+ {+ for my $startblock (qw(NULL 0))+ {+ for my $endblock (qw(NULL 0))+ {+ my $opts = "on_error_stop := $stop, " .+ "check_toast := $check_toast, " .+ "skip := $skip, " .+ "startblock := $startblock, " .+ "endblock := $endblock";++ detects_no_corruption(+ "verify_heapam('$relname', $opts)",+ "$prefix: $opts");+ }+ }+ }+ }+ }+}diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.cnew file mode 100644index 0000000000..b46cce798e--- /dev/null+++ b/contrib/amcheck/verify_heapam.c@@ -0,0 +1,1537 @@+/*-------------------------------------------------------------------------+ *+ * verify_heapam.c+ * Functions to check postgresql heap relations for corruption+ *+ * Copyright (c) 2016-2020, PostgreSQL Global Development Group+ *+ * contrib/amcheck/verify_heapam.c+ *-------------------------------------------------------------------------+ */+#include "postgres.h"++#include "access/detoast.h"+#include "access/genam.h"+#include "access/heapam.h"+#include "access/heaptoast.h"+#include "access/multixact.h"+#include "access/toast_internals.h"+#include "access/visibilitymap.h"+#include "catalog/pg_am.h"+#include "funcapi.h"+#include "miscadmin.h"+#include "storage/bufmgr.h"+#include "storage/procarray.h"+#include "utils/acl.h"+#include "utils/builtins.h"+#include "utils/fmgroids.h"++PG_FUNCTION_INFO_V1(verify_heapam);++/* The number of columns in tuples returned by verify_heapam */+#define HEAPCHECK_RELATION_COLS 4++typedef enum XidBoundsViolation+{+ XID_INVALID,+ XID_IN_FUTURE,+ XID_PRECEDES_DATMIN,+ XID_PRECEDES_RELMIN,+ XID_BOUNDS_OK+} XidBoundsViolation;++typedef enum XidCommitStatus+{+ XID_COMMITTED,+ XID_IN_PROGRESS,+ XID_ABORTED+} XidCommitStatus;++typedef enum SkipPages+{+ SKIP_PAGES_ALL_FROZEN,+ SKIP_PAGES_ALL_VISIBLE,+ SKIP_PAGES_NONE+} SkipPages;++/*+ * Struct holding the running context information during+ * a lifetime of a verify_heapam execution.+ */+typedef struct HeapCheckContext+{+ /*+ * Cached copies of values from ShmemVariableCache and computed values+ * from them.+ */+ FullTransactionId next_fxid; /* ShmemVariableCache->nextXid */+ TransactionId next_xid; /* 32-bit version of next_fxid */+ TransactionId oldest_xid; /* ShmemVariableCache->oldestXid */+ FullTransactionId oldest_fxid; /* 64-bit version of oldest_xid, computed+ * relative to next_fxid */++ /*+ * Cached copy of value from MultiXactState+ */+ MultiXactId next_mxact; /* MultiXactState->nextMXact */+ MultiXactId oldest_mxact; /* MultiXactState->oldestMultiXactId */++ /*+ * Cached copies of the most recently checked xid and its status.+ */+ TransactionId cached_xid;+ XidCommitStatus cached_status;++ /* Values concerning the heap relation being checked */+ Relation rel;+ TransactionId relfrozenxid;+ FullTransactionId relfrozenfxid;+ TransactionId relminmxid;+ Relation toast_rel;+ Relation *toast_indexes;+ Relation valid_toast_index;+ int num_toast_indexes;++ /* Values for iterating over pages in the relation */+ BlockNumber blkno;+ BufferAccessStrategy bstrategy;+ Buffer buffer;+ Page page;++ /* Values for iterating over tuples within a page */+ OffsetNumber offnum;+ ItemId itemid;+ uint16 lp_len;+ HeapTupleHeader tuphdr;+ int natts;++ /* Values for iterating over attributes within the tuple */+ uint32 offset; /* offset in tuple data */+ AttrNumber attnum;++ /* Values for iterating over toast for the attribute */+ int32 chunkno;+ int32 attrsize;+ int32 endchunk;+ int32 totalchunks;++ /* Whether verify_heapam has yet encountered any corrupt tuples */+ bool is_corrupt;++ /* The descriptor and tuplestore for verify_heapam's result tuples */+ TupleDesc tupdesc;+ Tuplestorestate *tupstore;+} HeapCheckContext;++/* Internal implementation */+static void sanity_check_relation(Relation rel);+static void check_tuple(HeapCheckContext *ctx);+static void check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx);++static bool check_tuple_attribute(HeapCheckContext *ctx);+static bool check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx);++static void report_corruption(HeapCheckContext *ctx, char *msg);+static TupleDesc verify_heapam_tupdesc(void);+static FullTransactionId FullTransactionIdFromXidAndCtx(TransactionId xid, const HeapCheckContext *ctx);+static void update_cached_xid_range(HeapCheckContext *ctx);+static void update_cached_mxid_range(HeapCheckContext *ctx);+static XidBoundsViolation check_mxid_in_range(MultiXactId mxid, HeapCheckContext *ctx);+static XidBoundsViolation check_mxid_valid_in_rel(MultiXactId mxid, HeapCheckContext *ctx);+static XidBoundsViolation get_xid_status(TransactionId xid, HeapCheckContext *ctx, XidCommitStatus *status);++/*+ * Scan and report corruption in heap pages, optionally reconciling toasted+ * attributes with entries in the associated toast table. Intended to be+ * called from SQL with the following parameters:+ *+ * relation+ * The Oid of the heap relation to be checked.+ *+ * on_error_stop:+ * Whether to stop at the end of the first page for which errors are+ * detected. Note that multiple rows may be returned.+ *+ * check_toast:+ * Whether to check each toasted attribute against the toast table to+ * verify that it can be found there.+ *+ * skip:+ * What kinds of pages in the heap relation should be skipped. Valid+ * options are "all-visible", "all-frozen", and "none".+ *+ * Returns to the SQL caller a set of tuples, each containing the location+ * and a description of a corruption found in the heap.+ *+ * Note that if check_toast is true, it is the caller's responsibility to+ * provide that the toast table and index are not corrupt, and that they+ * do not become corrupt while this function is running.+ */+Datum+verify_heapam(PG_FUNCTION_ARGS)+{+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;+ MemoryContext old_context;+ bool random_access;+ HeapCheckContext ctx;+ Buffer vmbuffer = InvalidBuffer;+ Oid relid;+ bool on_error_stop;+ bool check_toast;+ SkipPages skip_option = SKIP_PAGES_NONE;+ BlockNumber first_block;+ BlockNumber last_block;+ BlockNumber nblocks;+ const char *skip;++ /* Check to see if caller supports us returning a tuplestore */+ if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))+ ereport(ERROR,+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),+ errmsg("set-valued function called in context that cannot accept a set")));+ if (!(rsinfo->allowedModes & SFRM_Materialize))+ ereport(ERROR,+ (errcode(ERRCODE_SYNTAX_ERROR),+ errmsg("materialize mode required, but it is not allowed in this context")));++ /* Check supplied arguments */+ if (PG_ARGISNULL(0))+ ereport(ERROR,+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),+ errmsg("relation cannot be null")));+ relid = PG_GETARG_OID(0);++ if (PG_ARGISNULL(1))+ ereport(ERROR,+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),+ errmsg("on_error_stop cannot be null")));+ on_error_stop = PG_GETARG_BOOL(1);++ if (PG_ARGISNULL(2))+ ereport(ERROR,+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),+ errmsg("check_toast cannot be null")));+ check_toast = PG_GETARG_BOOL(2);++ if (PG_ARGISNULL(3))+ ereport(ERROR,+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),+ errmsg("skip cannot be null")));+ skip = text_to_cstring(PG_GETARG_TEXT_PP(3));+ if (pg_strcasecmp(skip, "all-visible") == 0)+ skip_option = SKIP_PAGES_ALL_VISIBLE;+ else if (pg_strcasecmp(skip, "all-frozen") == 0)+ skip_option = SKIP_PAGES_ALL_FROZEN;+ else if (pg_strcasecmp(skip, "none") == 0)+ skip_option = SKIP_PAGES_NONE;+ else+ ereport(ERROR,+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),+ errmsg("invalid skip option"),+ errhint("Valid skip options are \"all-visible\", \"all-frozen\", and \"none\".")));++ memset(&ctx, 0, sizeof(HeapCheckContext));+ ctx.cached_xid = InvalidTransactionId;++ /* The tupdesc and tuplestore must be created in ecxt_per_query_memory */+ old_context = MemoryContextSwitchTo(rsinfo->econtext->ecxt_per_query_memory);+ random_access = (rsinfo->allowedModes & SFRM_Materialize_Random) != 0;+ ctx.tupdesc = verify_heapam_tupdesc();+ ctx.tupstore = tuplestore_begin_heap(random_access, false, work_mem);+ rsinfo->returnMode = SFRM_Materialize;+ rsinfo->setResult = ctx.tupstore;+ rsinfo->setDesc = ctx.tupdesc;+ MemoryContextSwitchTo(old_context);++ /* Open relation, check relkind and access method, and check privileges */+ ctx.rel = relation_open(relid, AccessShareLock);+ sanity_check_relation(ctx.rel);++ /* Early exit if the relation is empty */+ nblocks = RelationGetNumberOfBlocks(ctx.rel);+ if (!nblocks)+ {+ relation_close(ctx.rel, AccessShareLock);+ PG_RETURN_NULL();+ }++ ctx.bstrategy = GetAccessStrategy(BAS_BULKREAD);+ ctx.buffer = InvalidBuffer;+ ctx.page = NULL;++ /* Validate block numbers, or handle nulls. */+ if (PG_ARGISNULL(4))+ first_block = 0;+ else+ {+ int64 fb = PG_GETARG_INT64(4);++ if (fb < 0 || fb >= nblocks)+ ereport(ERROR,+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),+ errmsg("starting block number must be between 0 and %u",+ nblocks - 1)));+ first_block = (BlockNumber) fb;+ }+ if (PG_ARGISNULL(5))+ last_block = nblocks - 1;+ else+ {+ int64 lb = PG_GETARG_INT64(5);++ if (lb < 0 || lb >= nblocks)+ ereport(ERROR,+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),+ errmsg("ending block number must be between 0 and %u",+ nblocks - 1)));+ last_block = (BlockNumber) lb;+ }++ /* Optionally open the toast relation, if any. */+ if (ctx.rel->rd_rel->reltoastrelid && check_toast)+ {+ int offset;++ /* Main relation has associated toast relation */+ ctx.toast_rel = table_open(ctx.rel->rd_rel->reltoastrelid,+ AccessShareLock);+ offset = toast_open_indexes(ctx.toast_rel,+ AccessShareLock,+ &(ctx.toast_indexes),+ &(ctx.num_toast_indexes));+ ctx.valid_toast_index = ctx.toast_indexes[offset];+ }+ else+ {+ /*+ * Main relation has no associated toast relation, or we're+ * intentionally skipping it.+ */+ ctx.toast_rel = NULL;+ ctx.toast_indexes = NULL;+ ctx.num_toast_indexes = 0;+ }++ update_cached_xid_range(&ctx);+ update_cached_mxid_range(&ctx);+ ctx.relfrozenxid = ctx.rel->rd_rel->relfrozenxid;+ ctx.relfrozenfxid = FullTransactionIdFromXidAndCtx(ctx.relfrozenxid, &ctx);+ ctx.relminmxid = ctx.rel->rd_rel->relminmxid;++ if (TransactionIdIsNormal(ctx.relfrozenxid))+ ctx.oldest_xid = ctx.relfrozenxid;++ for (ctx.blkno = first_block; ctx.blkno <= last_block; ctx.blkno++)+ {+ OffsetNumber maxoff;++ /* Optionally skip over all-frozen or all-visible blocks */+ if (skip_option != SKIP_PAGES_NONE)+ {+ int32 mapbits;++ mapbits = (int32) visibilitymap_get_status(ctx.rel, ctx.blkno,+ &vmbuffer);+ if (skip_option == SKIP_PAGES_ALL_FROZEN)+ {+ if ((mapbits & VISIBILITYMAP_ALL_FROZEN) != 0)+ continue;+ }++ if (skip_option == SKIP_PAGES_ALL_VISIBLE)+ {+ if ((mapbits & VISIBILITYMAP_ALL_VISIBLE) != 0)+ continue;+ }+ }++ /* Read and lock the next page. */+ ctx.buffer = ReadBufferExtended(ctx.rel, MAIN_FORKNUM, ctx.blkno,+ RBM_NORMAL, ctx.bstrategy);+ LockBuffer(ctx.buffer, BUFFER_LOCK_SHARE);+ ctx.page = BufferGetPage(ctx.buffer);++ /* Perform tuple checks */+ maxoff = PageGetMaxOffsetNumber(ctx.page);+ for (ctx.offnum = FirstOffsetNumber; ctx.offnum <= maxoff;+ ctx.offnum = OffsetNumberNext(ctx.offnum))+ {+ ctx.itemid = PageGetItemId(ctx.page, ctx.offnum);++ /* Skip over unused/dead line pointers */+ if (!ItemIdIsUsed(ctx.itemid) || ItemIdIsDead(ctx.itemid))+ continue;++ /*+ * If this line pointer has been redirected, check that it+ * redirects to a valid offset within the line pointer array.+ */+ if (ItemIdIsRedirected(ctx.itemid))+ {+ OffsetNumber rdoffnum = ItemIdGetRedirect(ctx.itemid);+ ItemId rditem;++ if (rdoffnum < FirstOffsetNumber || rdoffnum > maxoff)+ {+ report_corruption(&ctx,+ /*------+ translator: the %u is an offset */+ psprintf(_("line pointer redirection to item at offset %u exceeds maximum offset %u"),+ (unsigned) rdoffnum,+ (unsigned) maxoff));+ continue;+ }+ rditem = PageGetItemId(ctx.page, rdoffnum);+ if (!ItemIdIsUsed(rditem))+ report_corruption(&ctx,+ /*------+ translator: the %u is an offset */+ psprintf(_("line pointer redirection to unused item at offset %u"),+ (unsigned) rdoffnum));+ continue;+ }++ /* Set up context information about this next tuple */+ ctx.lp_len = ItemIdGetLength(ctx.itemid);+ ctx.tuphdr = (HeapTupleHeader) PageGetItem(ctx.page, ctx.itemid);+ ctx.natts = HeapTupleHeaderGetNatts(ctx.tuphdr);++ /* Ok, ready to check this next tuple */+ check_tuple(&ctx);+ }++ /* clean up */+ UnlockReleaseBuffer(ctx.buffer);++ if (on_error_stop && ctx.is_corrupt)+ break;+ }++ if (vmbuffer != InvalidBuffer)+ ReleaseBuffer(vmbuffer);++ /* Close the associated toast table and indexes, if any. */+ if (ctx.toast_indexes)+ toast_close_indexes(ctx.toast_indexes, ctx.num_toast_indexes,+ AccessShareLock);+ if (ctx.toast_rel)+ table_close(ctx.toast_rel, AccessShareLock);++ /* Close the main relation */+ relation_close(ctx.rel, AccessShareLock);++ PG_RETURN_NULL();+}++/*+ * Check that a relation's relkind and access method are both supported,+ * and that the caller has select privilege on the relation.+ */+static void+sanity_check_relation(Relation rel)+{+ AclResult aclresult;++ if (rel->rd_rel->relkind != RELKIND_RELATION &&+ rel->rd_rel->relkind != RELKIND_MATVIEW &&+ rel->rd_rel->relkind != RELKIND_TOASTVALUE)+ ereport(ERROR,+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),+ /*------+ translator: %s is a user supplied object name */+ errmsg("\"%s\" is not a table, materialized view, or TOAST table",+ RelationGetRelationName(rel))));+ if (rel->rd_rel->relam != HEAP_TABLE_AM_OID)+ ereport(ERROR,+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),+ errmsg("only heap AM is supported")));+ aclresult = pg_class_aclcheck(rel->rd_id, GetUserId(), ACL_SELECT);+ if (aclresult != ACLCHECK_OK)+ aclcheck_error(aclresult,+ get_relkind_objtype(rel->rd_rel->relkind),+ RelationGetRelationName(rel));+}++/*+ * Record a single corruption found in the table. The values in ctx should+ * reflect the location of the corruption, and the msg argument should contain+ * a human readable description of the corruption.+ *+ * The msg argument is pfree'd by this function.+ */+static void+report_corruption(HeapCheckContext *ctx, char *msg)+{+ Datum values[HEAPCHECK_RELATION_COLS];+ bool nulls[HEAPCHECK_RELATION_COLS];+ HeapTuple tuple;++ MemSet(values, 0, sizeof(values));+ MemSet(nulls, 0, sizeof(nulls));+ values[0] = Int64GetDatum(ctx->blkno);+ values[1] = Int32GetDatum(ctx->offnum);+ values[2] = Int32GetDatum(ctx->attnum);+ nulls[2] = (ctx->attnum < 0);+ values[3] = CStringGetTextDatum(msg);++ /*+ * In principle, there is nothing to prevent a scan over a large, highly+ * corrupted table from using work_mem worth of memory building up the+ * tuplestore. That's ok, but if we also leak the msg argument memory+ * until the end of the query, we could exceed work_mem by more than a+ * trivial amount. Therefore, free the msg argument each time we are+ * called rather than waiting for our current memory context to be freed.+ */+ pfree(msg);++ tuple = heap_form_tuple(ctx->tupdesc, values, nulls);+ tuplestore_puttuple(ctx->tupstore, tuple);+ ctx->is_corrupt = true;+}++/*+ * Construct the TupleDesc used to report messages about corruptions found+ * while scanning the heap.+ */+static TupleDesc+verify_heapam_tupdesc(void)+{+ TupleDesc tupdesc;+ AttrNumber a = 0;++ tupdesc = CreateTemplateTupleDesc(HEAPCHECK_RELATION_COLS);+ TupleDescInitEntry(tupdesc, ++a, "blkno", INT8OID, -1, 0);+ TupleDescInitEntry(tupdesc, ++a, "offnum", INT4OID, -1, 0);+ TupleDescInitEntry(tupdesc, ++a, "attnum", INT4OID, -1, 0);+ TupleDescInitEntry(tupdesc, ++a, "msg", TEXTOID, -1, 0);+ Assert(a == HEAPCHECK_RELATION_COLS);++ return BlessTupleDesc(tupdesc);+}++/*+ * Check for tuple header corruption and tuple visibility.+ *+ * Since we do not hold a snapshot, tuple visibility is not a question of+ * whether we should be able to see the tuple relative to any particular+ * snapshot, but rather a question of whether it is safe and reasonable to+ * to check the tuple attributes.+ *+ * Some kinds of corruption make it unsafe to check the tuple attributes, for+ * example when the line pointer refers to a range of bytes outside the page.+ * In such cases, we return false (not visible) after recording appropriate+ * corruption messages.+ *+ * Some other kinds of tuple header corruption confuse the question of where+ * the tuple attributes begin, or how long the nulls bitmap is, etc., making it+ * unreasonable to attempt to check attributes, even if all candidate answers+ * to those questions would not result in reading past the end of the line+ * pointer or page. In such cases, like above, we record corruption messages+ * about the header and then return false.+ *+ * Other kinds of tuple header corruption do not bear on the question of+ * whether the tuple attributes can be checked, so we record corruption+ * messages for them but do not base our visibility determination on them. (In+ * other words, we do not return false merely because we detected them.)+ *+ * For visibility determination not specifically related to corruption, what we+ * want to know is if a tuple is potentially visible to any running+ * transaction. If you are tempted to replace this function's visibility logic+ * with a call to another visibility checking function, keep in mind that this+ * function does not update hint bits, as it seems imprudent to write hint bits+ * (or anything at all) to a table during a corruption check. Nor does this+ * function bother classifying tuple visibility beyond a boolean visible vs.+ * not visible.+ *+ * The caller should already have checked that xmin and xmax are not out of+ * bounds for the relation.+ *+ * Returns whether the tuple is both visible and sufficiently sensible to+ * undergo attribute checks.+ */+static bool+check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)+{+ uint16 infomask = tuphdr->t_infomask;+ bool header_garbled = false;+ unsigned expected_hoff;;++ if (ctx->tuphdr->t_hoff > ctx->lp_len)+ {+ report_corruption(ctx,+ /*------+ translator: first %u is an offset, second %u is a length */+ psprintf(_("data begins at offset %u beyond the tuple length %u"),+ ctx->tuphdr->t_hoff, ctx->lp_len));+ header_garbled = true;+ }+ if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&+ (ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))+ {+ report_corruption(ctx,+ pstrdup(_("tuple is marked as only locked, but also claims key columns were updated")));+ header_garbled = true;+ }++ if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&+ (ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))+ {+ report_corruption(ctx,+ pstrdup(_("multixact should not be marked committed")));++ /*+ * This condition is clearly wrong, but we do not consider the header+ * garbled, because we don't rely on this property for determining if+ * the tuple is visible or for interpreting other relevant header+ * fields.+ */+ }++ if (infomask & HEAP_HASNULL)+ expected_hoff = MAXALIGN(SizeofHeapTupleHeader + BITMAPLEN(ctx->natts));+ else+ expected_hoff = MAXALIGN(SizeofHeapTupleHeader);+ if (ctx->tuphdr->t_hoff != expected_hoff)+ {+ if ((infomask & HEAP_HASNULL) && ctx->natts == 1)+ report_corruption(ctx,+ /*------+ translator: both %u represent an offset */+ psprintf(_("tuple data should begin at byte %u, but actually begins at byte %u (1 attribute, has nulls)"),+ expected_hoff, ctx->tuphdr->t_hoff));+ else if ((infomask & HEAP_HASNULL))+ report_corruption(ctx,+ /*------+ translator: first and second %u represent an offset, third %u+ represents the number of attributes */+ psprintf(_("tuple data should begin at byte %u, but actually begins at byte %u (%u attributes, has nulls)"),+ expected_hoff, ctx->tuphdr->t_hoff, ctx->natts));+ else if (ctx->natts == 1)+ report_corruption(ctx,+ /*------+ translator: both %u represent an offset */+ psprintf(_("tuple data should begin at byte %u, but actually begins at byte %u (1 attribute, no nulls)"),+ expected_hoff, ctx->tuphdr->t_hoff));+ else+ report_corruption(ctx,+ /*------+ translator: first and second %u represent an offset, third %u+ represents the number of attributes */+ psprintf(_("tuple data should begin at byte %u, but actually begins at byte %u (%u attributes, no nulls)"),+ expected_hoff, ctx->tuphdr->t_hoff, ctx->natts));+ header_garbled = true;+ }++ if (header_garbled)+ return false; /* checking of this tuple should not continue */++ /*+ * Ok, we can examine the header for tuple visibility purposes, though we+ * still need to be careful about a few remaining types of header+ * corruption. This logic roughly follows that of+ * HeapTupleSatisfiesVacuum. Where possible the comments indicate which+ * HTSV_Result we think that function might return for this tuple.+ */+ if (!HeapTupleHeaderXminCommitted(tuphdr))+ {+ TransactionId raw_xmin = HeapTupleHeaderGetRawXmin(tuphdr);++ if (HeapTupleHeaderXminInvalid(tuphdr))+ return false; /* HEAPTUPLE_DEAD */+ /* Used by pre-9.0 binary upgrades */+ else if (infomask & HEAP_MOVED_OFF ||+ infomask & HEAP_MOVED_IN)+ {+ XidCommitStatus status;+ TransactionId xvac = HeapTupleHeaderGetXvac(tuphdr);++ switch (get_xid_status(xvac, ctx, &status))+ {+ case XID_INVALID:+ report_corruption(ctx,+ pstrdup(_("old-style VACUUM FULL transaction ID is invalid")));+ return false; /* corrupt */+ case XID_IN_FUTURE:+ report_corruption(ctx,+ /*------+ translator: first and third %u are a transaction+ identifier, second %u is an epoch */+ psprintf(_("old-style VACUUM FULL transaction ID %u equals or exceeds next valid transaction ID %u:%u"),+ xvac,+ EpochFromFullTransactionId(ctx->next_fxid),+ XidFromFullTransactionId(ctx->next_fxid)));+ return false; /* corrupt */+ case XID_PRECEDES_RELMIN:+ report_corruption(ctx,+ /*------+ translator: first and third %u are a transaction+ identifier, second %u is an epoch */+ psprintf(_("old-style VACUUM FULL transaction ID %u precedes relation freeze threshold %u:%u"),+ xvac,+ EpochFromFullTransactionId(ctx->relfrozenfxid),+ XidFromFullTransactionId(ctx->relfrozenfxid)));+ return false; /* corrupt */+ break;+ case XID_PRECEDES_DATMIN:+ report_corruption(ctx,+ /*------+ translator: first and third %u are a transaction+ identifier, second %u is an epoch */+ psprintf(_("old-style VACUUM FULL transaction ID %u precedes oldest valid transaction ID %u:%u"),+ xvac,+ EpochFromFullTransactionId(ctx->oldest_fxid),+ XidFromFullTransactionId(ctx->oldest_fxid)));+ return false; /* corrupt */+ break;+ case XID_BOUNDS_OK:+ switch (status)+ {+ case XID_IN_PROGRESS:+ return true; /* HEAPTUPLE_DELETE_IN_PROGRESS */+ case XID_COMMITTED:+ case XID_ABORTED:+ return false; /* HEAPTUPLE_DEAD */+ }+ }+ }+ else+ {+ XidCommitStatus status;++ switch (get_xid_status(raw_xmin, ctx, &status))+ {+ case XID_INVALID:+ report_corruption(ctx,+ pstrdup(_("raw xmin is invalid")));+ return false;+ case XID_IN_FUTURE:+ report_corruption(ctx,+ /*------+ translator: first and third %u are a transaction+ identifier, second %u is an epoch */+ psprintf(_("raw xmin %u equals or exceeds next valid transaction ID %u:%u"),+ raw_xmin,+ EpochFromFullTransactionId(ctx->next_fxid),+ XidFromFullTransactionId(ctx->next_fxid)));+ return false; /* corrupt */+ case XID_PRECEDES_RELMIN:+ report_corruption(ctx,+ /*------+ translator: first and third %u are a transaction+ identifier, second %u is an epoch */+ psprintf(_("raw xmin %u precedes relation freeze threshold %u:%u"),+ raw_xmin,+ EpochFromFullTransactionId(ctx->relfrozenfxid),+ XidFromFullTransactionId(ctx->relfrozenfxid)));+ return false; /* corrupt */+ case XID_PRECEDES_DATMIN:+ report_corruption(ctx,+ /*------+ translator: first and third %u are a transaction+ identifier, second %u is an epoch */+ psprintf(_("raw xmin %u precedes oldest valid transaction ID %u:%u"),+ raw_xmin,+ EpochFromFullTransactionId(ctx->oldest_fxid),+ XidFromFullTransactionId(ctx->oldest_fxid)));+ return false; /* corrupt */+ case XID_BOUNDS_OK:+ switch (status)+ {+ case XID_COMMITTED:+ break;+ case XID_IN_PROGRESS:+ return true; /* insert or delete in progress */+ case XID_ABORTED:+ return false; /* HEAPTUPLE_DEAD */+ }+ }+ }+ }++ if (!(infomask & HEAP_XMAX_INVALID) && !HEAP_XMAX_IS_LOCKED_ONLY(infomask))+ {+ if (infomask & HEAP_XMAX_IS_MULTI)+ {+ XidCommitStatus status;+ TransactionId xmax = HeapTupleGetUpdateXid(tuphdr);++ switch (get_xid_status(xmax, ctx, &status))+ {+ /* not LOCKED_ONLY, so it has to have an xmax */+ case XID_INVALID:+ report_corruption(ctx,+ pstrdup(_("xmax is invalid")));+ return false; /* corrupt */+ case XID_IN_FUTURE:+ report_corruption(ctx,+ /*------+ translator: first and third %u are a transaction+ identifier, second %u is an epoch */+ psprintf(_("xmax %u equals or exceeds next valid transaction ID %u:%u"),+ xmax,+ EpochFromFullTransactionId(ctx->next_fxid),+ XidFromFullTransactionId(ctx->next_fxid)));+ return false; /* corrupt */+ case XID_PRECEDES_RELMIN:+ report_corruption(ctx,+ /*------+ translator: first and third %u are a transaction+ identifier, second %u is an epoch */+ psprintf(_("xmax %u precedes relation freeze threshold %u:%u"),+ xmax,+ EpochFromFullTransactionId(ctx->relfrozenfxid),+ XidFromFullTransactionId(ctx->relfrozenfxid)));+ return false; /* corrupt */+ case XID_PRECEDES_DATMIN:+ report_corruption(ctx,+ /*------+ translator: first and third %u are a transaction+ identifier, second %u is an epoch */+ psprintf(_("xmax %u precedes oldest valid transaction ID %u:%u"),+ xmax,+ EpochFromFullTransactionId(ctx->oldest_fxid),+ XidFromFullTransactionId(ctx->oldest_fxid)));+ return false; /* corrupt */+ case XID_BOUNDS_OK:+ switch (status)+ {+ case XID_IN_PROGRESS:+ return true; /* HEAPTUPLE_DELETE_IN_PROGRESS */+ case XID_COMMITTED:+ case XID_ABORTED:+ return false; /* HEAPTUPLE_RECENTLY_DEAD or+ * HEAPTUPLE_DEAD */+ }+ }++ /* Ok, the tuple is live */+ }+ else if (!(infomask & HEAP_XMAX_COMMITTED))+ return true; /* HEAPTUPLE_DELETE_IN_PROGRESS or+ * HEAPTUPLE_LIVE */+ else+ return false; /* HEAPTUPLE_RECENTLY_DEAD or HEAPTUPLE_DEAD */+ }+ return true; /* not dead */+}++/*+ * Check the current toast tuple against the state tracked in ctx, recording+ * any corruption found in ctx->tupstore.+ *+ * This is not equivalent to running verify_heapam on the toast table itself,+ * and is not hardened against corruption of the toast table. Rather, when+ * validating a toasted attribute in the main table, the sequence of toast+ * tuples that store the toasted value are retrieved and checked in order, with+ * each toast tuple being checked against where we are in the sequence, as well+ * as each toast tuple having its varlena structure sanity checked.+ */+static void+check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx)+{+ int32 curchunk;+ Pointer chunk;+ bool isnull;+ int32 chunksize;+ int32 expected_size;++ /*+ * Have a chunk, extract the sequence number and the data+ */+ curchunk = DatumGetInt32(fastgetattr(toasttup, 2,+ ctx->toast_rel->rd_att, &isnull));+ if (isnull)+ {+ report_corruption(ctx,+ pstrdup(_("toast chunk sequence number is null")));+ return;+ }+ chunk = DatumGetPointer(fastgetattr(toasttup, 3,+ ctx->toast_rel->rd_att, &isnull));+ if (isnull)+ {+ report_corruption(ctx,+ pstrdup(_("toast chunk data is null")));+ return;+ }+ if (!VARATT_IS_EXTENDED(chunk))+ chunksize = VARSIZE(chunk) - VARHDRSZ;+ else if (VARATT_IS_SHORT(chunk))+ {+ /*+ * could happen due to heap_form_tuple doing its thing+ */+ chunksize = VARSIZE_SHORT(chunk) - VARHDRSZ_SHORT;+ }+ else+ {+ /* should never happen */+ uint32 header = ((varattrib_4b *) chunk)->va_4byte.va_header;++ report_corruption(ctx,+ /*------+ translator: %0x represents a bit pattern in hexadecimal, %d represents+ the sequence number */+ psprintf(_("corrupt extended toast chunk has invalid varlena header: %0x (sequence number %d)"),+ header, curchunk));+ return;+ }++ /*+ * Some checks on the data we've found+ */+ if (curchunk != ctx->chunkno)+ {+ report_corruption(ctx,+ /*------+ translator: both %u represent a sequence number */+ psprintf(_("toast chunk sequence number %u does not match the expected sequence number %u"),+ curchunk, ctx->chunkno));+ return;+ }+ if (curchunk > ctx->endchunk)+ {+ report_corruption(ctx,+ /*------+ translator: both %u represent a sequence number */+ psprintf(_("toast chunk sequence number %u exceeds the end chunk sequence number %u"),+ curchunk, ctx->endchunk));+ return;+ }++ expected_size = curchunk < ctx->totalchunks - 1 ? TOAST_MAX_CHUNK_SIZE+ : ctx->attrsize - ((ctx->totalchunks - 1) * TOAST_MAX_CHUNK_SIZE);+ if (chunksize != expected_size)+ {+ report_corruption(ctx,+ /*------+ translator: both %u represent a chunk size */+ psprintf(_("toast chunk size %u differs from the expected size %u"),+ chunksize, expected_size));+ return;+ }+}++/*+ * Check the current attribute as tracked in ctx, recording any corruption+ * found in ctx->tupstore.+ *+ * This function follows the logic performed by heap_deform_tuple(), and in the+ * case of a toasted value, optionally continues along the logic of+ * detoast_external_attr(), checking for any conditions that would result in+ * either of those functions Asserting or crashing the backend. The checks+ * performed by Asserts present in those two functions are also performed here.+ * In cases where those two functions are a bit cavalier in their assumptions+ * about data being correct, we perform additional checks not present in either+ * of those two functions. Where some condition is checked in both of those+ * functions, we perform it here twice, as we parallel the logical flow of+ * those two functions. The presence of duplicate checks seems a reasonable+ * price to pay for keeping this code tightly coupled with the code it+ * protects.+ *+ * Returns true if the tuple attribute is sane enough for processing to+ * continue on to the next attribute, false otherwise.+ */+static bool+check_tuple_attribute(HeapCheckContext *ctx)+{+ struct varatt_external toast_pointer;+ ScanKeyData toastkey;+ SysScanDesc toastscan;+ SnapshotData SnapshotToast;+ HeapTuple toasttup;+ bool found_toasttup;+ Datum attdatum;+ struct varlena *attr;+ char *tp; /* pointer to the tuple data */+ uint16 infomask;+ Form_pg_attribute thisatt;++ infomask = ctx->tuphdr->t_infomask;+ thisatt = TupleDescAttr(RelationGetDescr(ctx->rel), ctx->attnum);++ tp = (char *) ctx->tuphdr + ctx->tuphdr->t_hoff;++ if (ctx->tuphdr->t_hoff + ctx->offset > ctx->lp_len)+ {+ report_corruption(ctx,+ /*------+ translator: first %u represents an attribute number; second and fourth+ %u represent a length; third %u represents an offset */+ psprintf(_("attribute %u with length %u starts at offset %u beyond total tuple length %u"),+ ctx->attnum,+ thisatt->attlen,+ ctx->tuphdr->t_hoff + ctx->offset,+ ctx->lp_len));+ return false;+ }++ /* Skip null values */+ if (infomask & HEAP_HASNULL && att_isnull(ctx->attnum, ctx->tuphdr->t_bits))+ return true;++ /* Skip non-varlena values, but update offset first */+ if (thisatt->attlen != -1)+ {+ ctx->offset = att_align_nominal(ctx->offset, thisatt->attalign);+ ctx->offset = att_addlength_pointer(ctx->offset, thisatt->attlen,+ tp + ctx->offset);+ if (ctx->tuphdr->t_hoff + ctx->offset > ctx->lp_len)+ {+ report_corruption(ctx,+ /*------+ translator: first %u represents an attribute number; second and+ fourth %u represent a length; third %u represents an offset */+ psprintf(_("attribute %u with length %u ends at offset %u beyond total tuple length %u"),+ ctx->attnum,+ thisatt->attlen,+ ctx->tuphdr->t_hoff + ctx->offset,+ ctx->lp_len));+ return false;+ }+ return true;+ }++ /* Ok, we're looking at a varlena attribute. */+ ctx->offset = att_align_pointer(ctx->offset, thisatt->attalign, -1,+ tp + ctx->offset);++ /* Get the (possibly corrupt) varlena datum */+ attdatum = fetchatt(thisatt, tp + ctx->offset);++ /*+ * We have the datum, but we cannot decode it carelessly, as it may still+ * be corrupt.+ */++ /*+ * Check that VARTAG_SIZE won't hit a TrapMacro on a corrupt va_tag before+ * risking a call into att_addlength_pointer+ */+ if (VARATT_IS_EXTERNAL(tp + ctx->offset))+ {+ uint8 va_tag = VARTAG_EXTERNAL(tp + ctx->offset);++ if (va_tag != VARTAG_ONDISK)+ {+ report_corruption(ctx,+ /*------+ translator: first %u represents an attribute number, second %u+ represents an enumeration value */+ psprintf(_("toasted attribute %u has unexpected TOAST tag %u"),+ ctx->attnum,+ va_tag));+ /* We can't know where the next attribute begins */+ return false;+ }+ }++ /* Ok, should be safe now */+ ctx->offset = att_addlength_pointer(ctx->offset, thisatt->attlen,+ tp + ctx->offset);++ if (ctx->tuphdr->t_hoff + ctx->offset > ctx->lp_len)+ {+ report_corruption(ctx,+ /*------+ translator: first %u represents an attribute number; second and fourth+ %u represent a length; third %u represents an offset */+ psprintf(_("attribute %u with length %u ends at offset %u beyond total tuple length %u"),+ ctx->attnum,+ thisatt->attlen,+ ctx->tuphdr->t_hoff + ctx->offset,+ ctx->lp_len));++ return false;+ }++ /*+ * heap_deform_tuple would be done with this attribute at this point,+ * having stored it in values[], and would continue to the next attribute.+ * We go further, because we need to check if the toast datum is corrupt.+ */++ attr = (struct varlena *) DatumGetPointer(attdatum);++ /*+ * Now we follow the logic of detoast_external_attr(), with the same+ * caveats about being paranoid about corruption.+ */++ /* Skip values that are not external */+ if (!VARATT_IS_EXTERNAL(attr))+ return true;++ /* It is external, and we're looking at a page on disk */++ /* The tuple header better claim to contain toasted values */+ if (!(infomask & HEAP_HASEXTERNAL))+ {+ report_corruption(ctx,+ /*------+ translator: %u represents the attribute number */+ psprintf(_("attribute %u is external but tuple header flag HEAP_HASEXTERNAL not set"),+ ctx->attnum));+ return true;+ }++ /* The relation better have a toast table */+ if (!ctx->rel->rd_rel->reltoastrelid)+ {+ report_corruption(ctx,+ /*------+ translator: %u represents the attribute number */+ psprintf(_("attribute %u is external but relation has no toast relation"),+ ctx->attnum));+ return true;+ }++ /* If we were told to skip toast checking, then we're done. */+ if (ctx->toast_rel == NULL)+ return true;++ /*+ * Must copy attr into toast_pointer for alignment considerations+ */+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);++ ctx->attrsize = toast_pointer.va_extsize;+ ctx->endchunk = (ctx->attrsize - 1) / TOAST_MAX_CHUNK_SIZE;+ ctx->totalchunks = ctx->endchunk + 1;++ /*+ * Setup a scan key to find chunks in toast table with matching va_valueid+ */+ ScanKeyInit(&toastkey,+ (AttrNumber) 1,+ BTEqualStrategyNumber, F_OIDEQ,+ ObjectIdGetDatum(toast_pointer.va_valueid));++ /*+ * Check if any chunks for this toasted object exist in the toast table,+ * accessible via the index.+ */+ init_toast_snapshot(&SnapshotToast);+ toastscan = systable_beginscan_ordered(ctx->toast_rel,+ ctx->valid_toast_index,+ &SnapshotToast, 1,+ &toastkey);+ ctx->chunkno = 0;+ found_toasttup = false;+ while ((toasttup =+ systable_getnext_ordered(toastscan,+ ForwardScanDirection)) != NULL)+ {+ found_toasttup = true;+ check_toast_tuple(toasttup, ctx);+ ctx->chunkno++;+ }+ if (ctx->chunkno != (ctx->endchunk + 1))+ report_corruption(ctx,+ /*------+ translator: both %u represent a chunk number */+ psprintf(_("final toast chunk number %u differs from expected value %u"),+ ctx->chunkno, (ctx->endchunk + 1)));+ if (!found_toasttup)+ report_corruption(ctx,+ /*------+ translator: %u represents the attribute number */+ psprintf(_("toasted value for attribute %u missing from toast table"),+ ctx->attnum));+ systable_endscan_ordered(toastscan);++ return true;+}++/*+ * Check the current tuple as tracked in ctx, recording any corruption found in+ * ctx->tupstore.+ */+static void+check_tuple(HeapCheckContext *ctx)+{+ TransactionId xmin;+ TransactionId xmax;+ bool fatal = false;+ uint16 infomask = ctx->tuphdr->t_infomask;++ /*+ * If we report corruption before iterating over individual attributes, we+ * need attnum to be reported as NULL. Set that up before any corruption+ * reporting might happen.+ */+ ctx->attnum = -1;++ /*+ * If the line pointer for this tuple does not reserve enough space for a+ * complete tuple header, we dare not read the tuple header.+ */+ if (ctx->lp_len < MAXALIGN(SizeofHeapTupleHeader))+ {+ report_corruption(ctx,+ /*------+ translator: first %u represents a length, second %u represents a size+ */+ psprintf(_("line pointer length %u is less than the minimum tuple header size %u"),+ ctx->lp_len, (uint32) MAXALIGN(SizeofHeapTupleHeader)));+ return;+ }++ /* If xmin is normal, it should be within valid range */+ xmin = HeapTupleHeaderGetXmin(ctx->tuphdr);+ switch (get_xid_status(xmin, ctx, NULL))+ {+ case XID_INVALID:+ case XID_BOUNDS_OK:+ break;+ case XID_IN_FUTURE:+ report_corruption(ctx,+ /*------+ translator: first and third %u are a transaction identifier, second+ %u is an epoch */+ psprintf(_("xmin %u equals or exceeds next valid transaction ID %u:%u"),+ xmin,+ EpochFromFullTransactionId(ctx->next_fxid),+ XidFromFullTransactionId(ctx->next_fxid)));+ fatal = true;+ break;+ case XID_PRECEDES_DATMIN:+ report_corruption(ctx,+ /*------+ translator: first and third %u are a transaction identifier, second+ %u is an epoch */+ psprintf(_("xmin %u precedes oldest valid transaction ID %u:%u"),+ xmin,+ EpochFromFullTransactionId(ctx->oldest_fxid),+ XidFromFullTransactionId(ctx->oldest_fxid)));+ fatal = true;+ break;+ case XID_PRECEDES_RELMIN:+ report_corruption(ctx,+ /*------+ translator: first and third %u are a transaction identifier, second+ %u is an epoch */+ psprintf(_("xmin %u precedes relation freeze threshold %u:%u"),+ xmin,+ EpochFromFullTransactionId(ctx->relfrozenfxid),+ XidFromFullTransactionId(ctx->relfrozenfxid)));+ fatal = true;+ break;+ }++ xmax = HeapTupleHeaderGetRawXmax(ctx->tuphdr);++ /* If xmax is a multixact, it should be within valid range */+ if (infomask & HEAP_XMAX_IS_MULTI)+ {+ switch (check_mxid_valid_in_rel(xmax, ctx))+ {+ case XID_INVALID:+ report_corruption(ctx,+ pstrdup(_("multitransaction ID is invalid")));+ fatal = true;+ break;+ case XID_PRECEDES_RELMIN:+ report_corruption(ctx,+ /*------+ translator: both %u are multitransaction IDs */+ psprintf(_("multitransaction ID %u precedes relation minimum multitransaction ID threshold %u"),+ xmax, ctx->relminmxid));+ fatal = true;+ break;+ case XID_PRECEDES_DATMIN:+ report_corruption(ctx,+ /*------+ translator: Both %u are multitransaction IDs */+ psprintf(_("multitransaction ID %u precedes oldest valid multitransaction ID threshold %u"),+ xmax, ctx->oldest_mxact));+ fatal = true;+ break;+ case XID_IN_FUTURE:+ report_corruption(ctx,+ /*------+ translator: %u is a multitransaction ID */+ psprintf(_("multitransaction ID %u equals or exceeds next valid multitransaction ID %u"),+ xmax,+ ctx->next_mxact));+ fatal = true;+ break;+ case XID_BOUNDS_OK:+ break;+ }+ }+ /* If xmax is not a multixact and is normal, it should be within valid range */+ else+ {+ switch (get_xid_status(xmax, ctx, NULL))+ {+ case XID_INVALID:+ case XID_BOUNDS_OK:+ break;+ case XID_IN_FUTURE:+ report_corruption(ctx,+ /*------+ translator: first and third %u are a transaction identifier,+ second %u is an epoch */+ psprintf(_("xmax %u equals or exceeds next valid transaction ID %u:%u"),+ xmax,+ EpochFromFullTransactionId(ctx->next_fxid),+ XidFromFullTransactionId(ctx->next_fxid)));+ fatal = true;+ break;+ case XID_PRECEDES_DATMIN:+ report_corruption(ctx,+ /*------+ translator: first and third %u are a transaction identifier,+ second %u is an epoch */+ psprintf(_("xmax %u precedes oldest valid transaction ID %u:%u"),+ xmax,+ EpochFromFullTransactionId(ctx->oldest_fxid),+ XidFromFullTransactionId(ctx->oldest_fxid)));+ fatal = true;+ break;+ case XID_PRECEDES_RELMIN:+ report_corruption(ctx,+ /*------+ translator: first and third %u are a transaction identifier,+ second %u is an epoch */+ psprintf(_("xmax %u precedes relation freeze threshold %u:%u"),+ xmax,+ EpochFromFullTransactionId(ctx->relfrozenfxid),+ XidFromFullTransactionId(ctx->relfrozenfxid)));+ fatal = true;+ }+ }++ /*+ * Cannot process tuple data if tuple header was corrupt, as the offsets+ * within the page cannot be trusted, leaving too much risk of reading+ * garbage if we continue.+ *+ * We also cannot process the tuple if the xmin or xmax were invalid+ * relative to relfrozenxid or relminmxid, as clog entries for the xids+ * may already be gone.+ */+ if (fatal)+ return;++ /*+ * Check various forms of tuple header corruption. If the header is too+ * corrupt to continue checking, or if the tuple is not visible to anyone,+ * we cannot continue with other checks.+ */+ if (!check_tuple_header_and_visibilty(ctx->tuphdr, ctx))+ return;++ /*+ * The tuple is visible, so it must be compatible with the current version+ * of the relation descriptor. It might have fewer columns than are+ * present in the relation descriptor, but it cannot have more.+ */+ if (RelationGetDescr(ctx->rel)->natts < ctx->natts)+ {+ report_corruption(ctx,+ /*------+ translator: both %u are a number */+ psprintf(_("number of attributes %u exceeds maximum expected for table %u"),+ ctx->natts,+ RelationGetDescr(ctx->rel)->natts));+ return;+ }++ /*+ * Check each attribute unless we hit corruption that confuses what to do+ * next, at which point we abort further attribute checks for this tuple.+ * Note that we don't abort for all types of corruption, only for those+ * types where we don't know how to continue.+ */+ ctx->offset = 0;+ for (ctx->attnum = 0; ctx->attnum < ctx->natts; ctx->attnum++)+ if (!check_tuple_attribute(ctx))+ break; /* cannot continue */+}++/*+ * Convert a TransactionId into a FullTransactionId using our cached values of+ * the valid transaction ID range. It is the caller's responsibility to have+ * already updated the cached values, if necessary.+ */+static FullTransactionId+FullTransactionIdFromXidAndCtx(TransactionId xid, const HeapCheckContext *ctx)+{+ uint32 epoch;++ if (!TransactionIdIsNormal(xid))+ return FullTransactionIdFromEpochAndXid(0, xid);+ epoch = EpochFromFullTransactionId(ctx->next_fxid);+ if (xid > ctx->next_xid)+ epoch--;+ return FullTransactionIdFromEpochAndXid(epoch, xid);+}++/*+ * Update our cached range of valid transaction IDs.+ */+static void+update_cached_xid_range(HeapCheckContext *ctx)+{+ /* Make cached copies */+ LWLockAcquire(XidGenLock, LW_SHARED);+ ctx->next_fxid = ShmemVariableCache->nextXid;+ ctx->oldest_xid = ShmemVariableCache->oldestXid;+ LWLockRelease(XidGenLock);++ /* And compute alternate versions of the same */+ ctx->oldest_fxid = FullTransactionIdFromXidAndCtx(ctx->oldest_xid, ctx);+ ctx->next_xid = XidFromFullTransactionId(ctx->next_fxid);+}++/*+ * Update our cached range of valid multitransaction IDs.+ */+static void+update_cached_mxid_range(HeapCheckContext *ctx)+{+ ReadMultiXactIdRange(&ctx->oldest_mxact, &ctx->next_mxact);+}++/*+ * Return whether the given FullTransactionId is within our cached valid+ * transaction ID range.+ */+static inline bool+fxid_in_cached_range(FullTransactionId fxid, const HeapCheckContext *ctx)+{+ return (FullTransactionIdPrecedesOrEquals(ctx->oldest_fxid, fxid) &&+ FullTransactionIdPrecedes(fxid, ctx->next_fxid));+}++/*+ * Checks wheter a multitransaction ID is in the cached valid range, returning+ * the nature of the range violation, if any.+ */+static XidBoundsViolation+check_mxid_in_range(MultiXactId mxid, HeapCheckContext *ctx)+{+ if (!TransactionIdIsValid(mxid))+ return XID_INVALID;+ if (MultiXactIdPrecedes(mxid, ctx->relminmxid))+ return XID_PRECEDES_RELMIN;+ if (MultiXactIdPrecedes(mxid, ctx->oldest_mxact))+ return XID_PRECEDES_DATMIN;+ if (MultiXactIdPrecedesOrEquals(ctx->next_mxact, mxid))+ return XID_IN_FUTURE;+ return XID_BOUNDS_OK;+}++/*+ * Checks whether the given mxid is valid to appear in the heap being checked,+ * returning the nature of the range violation, if any.+ *+ * This function attempts to return quickly by caching the known valid mxid+ * range in ctx. Callers should already have performed the initial setup of+ * the cache prior to the first call to this function.+ */+static XidBoundsViolation+check_mxid_valid_in_rel(MultiXactId mxid, HeapCheckContext *ctx)+{+ XidBoundsViolation result;++ result = check_mxid_in_range(mxid, ctx);+ if (result == XID_BOUNDS_OK)+ return XID_BOUNDS_OK;++ /* The range may have advanced. Recheck. */+ update_cached_mxid_range(ctx);+ return check_mxid_in_range(mxid, ctx);+}++/*+ * Checks whether the given transaction ID is (or was recently) valid to appear+ * in the heap being checked, or whether it is too old or too new to appear in+ * the relation, returning information about the nature of the bounds violation.+ *+ * We cache the range of valid transaction IDs. If xid is in that range, we+ * conclude that it is valid, even though concurrent changes to the table might+ * invalidate it under certain corrupt conditions. (For example, if the table+ * contains corrupt all-frozen bits, a concurrent vacuum might skip the page(s)+ * containing the xid and then truncate clog and advance the relfrozenxid+ * beyond xid.) Reporting the xid as valid under such conditions seems+ * acceptable, since if we had checked it earlier in our scan it would have+ * truly been valid at that time.+ *+ * If the status argument is not NULL, and if and only if the transaction ID+ * appears to be valid in this relation, clog will be consulted and the commit+ * status argument will be set with the status of the transaction ID.+ */+static XidBoundsViolation+get_xid_status(TransactionId xid, HeapCheckContext *ctx, XidCommitStatus *status)+{+ XidBoundsViolation result;+ FullTransactionId fxid;+ FullTransactionId clog_horizon;++ /* Quick check for special xids */+ if (!TransactionIdIsValid(xid))+ result = XID_INVALID;+ else if (xid == BootstrapTransactionId || xid == FrozenTransactionId)+ result = XID_BOUNDS_OK;+ else+ {+ /* Check if the xid is within bounds */+ fxid = FullTransactionIdFromXidAndCtx(xid, ctx);+ if (!fxid_in_cached_range(fxid, ctx))+ {+ /*+ * We may have been checking against stale values. Update the+ * cached range to be sure, and since we relied on the cached+ * range when we performed the full xid conversion, reconvert.+ */+ update_cached_xid_range(ctx);+ fxid = FullTransactionIdFromXidAndCtx(xid, ctx);+ }++ if (FullTransactionIdPrecedesOrEquals(ctx->next_fxid, fxid))+ result = XID_IN_FUTURE;+ else if (FullTransactionIdPrecedes(fxid, ctx->oldest_fxid))+ result = XID_PRECEDES_DATMIN;+ else if (FullTransactionIdPrecedes(fxid, ctx->relfrozenfxid))+ result = XID_PRECEDES_RELMIN;+ else+ result = XID_BOUNDS_OK;+ }++ /*+ * Early return if the caller does not request clog checking, or if the+ * xid is already known to be out of bounds. We dare not check clog for+ * out of bounds transaction IDs.+ */+ if (status == NULL || result != XID_BOUNDS_OK)+ return result;++ /* Early return if we just checked this xid in a prior call */+ if (xid == ctx->cached_xid)+ {+ *status = ctx->cached_status;+ return result;+ }++ *status = XID_COMMITTED;+ LWLockAcquire(XactTruncationLock, LW_SHARED);+ clog_horizon = FullTransactionIdFromXidAndCtx(ShmemVariableCache->oldestClogXid, ctx);+ if (FullTransactionIdPrecedesOrEquals(clog_horizon, fxid))+ {+ if (TransactionIdIsCurrentTransactionId(xid))+ *status = XID_IN_PROGRESS;+ else if (TransactionIdDidCommit(xid))+ *status = XID_COMMITTED;+ else if (TransactionIdDidAbort(xid))+ *status = XID_ABORTED;+ else+ *status = XID_IN_PROGRESS;+ }+ LWLockRelease(XactTruncationLock);+ ctx->cached_xid = xid;+ ctx->cached_status = *status;+ return result;+}diff --git a/doc/src/sgml/amcheck.sgml b/doc/src/sgml/amcheck.sgmlindex a9df2c1a9d..a57781992a 100644--- a/doc/src/sgml/amcheck.sgml+++ b/doc/src/sgml/amcheck.sgml@@ -9,12 +9,11 @@
<para>
The <filename>amcheck</filename> module provides functions that allow you to
- verify the logical consistency of the structure of relations. If the- structure appears to be valid, no error is raised.+ verify the logical consistency of the structure of relations.
</para>
<para>
- The functions verify various <emphasis>invariants</emphasis> in the+ The B-Tree checking functions verify various <emphasis>invariants</emphasis> in the
structure of the representation of particular relations. The
correctness of the access method functions behind index scans and
other important operations relies on these invariants always
@@ -24,7 +23,7 @@
collated lexical order). If that particular invariant somehow fails
to hold, we can expect binary searches on the affected page to
incorrectly guide index scans, resulting in wrong answers to SQL
- queries.+ queries. If the structure appears to be valid, no error is raised.
</para>
<para>
Verification is performed using the same procedures as those used by
@@ -35,7 +34,22 @@
functions.
</para>
<para>
- <filename>amcheck</filename> functions may only be used by superusers.+ Unlike the B-Tree checking functions which report corruption by raising+ errors, the heap checking function <function>verify_heapam</function> checks+ a table and attempts to return a set of rows, one row per corruption+ detected. Despite this, if facilities that+ <function>verify_heapam</function> relies upon are themselves corrupted, the+ function may be unable to continue and may instead raise an error.+ </para>+ <para>+ Permission to execute <filename>amcheck</filename> functions may be granted+ to non-superusers, but before granting such permissions careful consideration+ should be given to data security and privacy concerns. Although the+ corruption reports generated by these functions do not focus on the contents+ of the corrupted data so much as on the structure of that data and the nature+ of the corruptions found, an attacker who gains permission to execute these+ functions, particularly if the attacker can also induce corruption, might be+ able to infer something of the data itself from such messages.
</para>
<sect2>
@@ -187,12 +201,223 @@ SET client_min_messages = DEBUG1;
</para>
</tip>
+ <variablelist>+ <varlistentry>+ <term>+ <function>+ verify_heapam(relation regclass,+ on_error_stop boolean,+ check_toast boolean,+ skip cstring,+ startblock bigint,+ endblock bigint,+ blkno OUT bigint,+ offnum OUT integer,+ attnum OUT integer,+ msg OUT text)+ returns record+ </function>+ </term>+ <listitem>+ <para>+ Checks a table for structural corruption, where pages in the relation+ contain data that is invalidly formatted, and for logical corruption,+ where pages are structurally valid but inconsistent with the rest of the+ database cluster. Example usage:+<screen>+test=# select * from verify_heapam('mytable', check_toast := true);+ blkno | offnum | attnum | msg+-------+--------+--------+--------------------------------------------------------------------------------------------------+ 17 | 12 | | xmin 4294967295 precedes relation freeze threshold 17:1134217582+ 960 | 4 | | data begins at offset 152 beyond the tuple length 58+ 960 | 4 | | tuple data should begin at byte 24, but actually begins at byte 152 (3 attributes, no nulls)+ 960 | 5 | | tuple data should begin at byte 24, but actually begins at byte 27 (3 attributes, no nulls)+ 960 | 6 | | tuple data should begin at byte 24, but actually begins at byte 16 (3 attributes, no nulls)+ 960 | 7 | | tuple data should begin at byte 24, but actually begins at byte 21 (3 attributes, no nulls)+ 1147 | 2 | | number of attributes 2047 exceeds maximum expected for table 3+ 1147 | 10 | | tuple data should begin at byte 280, but actually begins at byte 24 (2047 attributes, has nulls)+ 1147 | 15 | | number of attributes 67 exceeds maximum expected for table 3+ 1147 | 16 | 1 | attribute 1 with length 4294967295 ends at offset 416848000 beyond total tuple length 58+ 1147 | 18 | 2 | final toast chunk number 0 differs from expected value 6+ 1147 | 19 | 2 | toasted value for attribute 2 missing from toast table+ 1147 | 21 | | tuple is marked as only locked, but also claims key columns were updated+ 1147 | 22 | | multitransaction ID 1775655 is from before relation cutoff 2355572+(14 rows)+</screen>+ As this example shows, the Tuple ID (TID) of the corrupt tuple is given+ in the (<literal>blkno</literal>, <literal>offnum</literal>) columns, and+ for corruptions specific to a particular attribute in the tuple, the+ <literal>attnum</literal> field shows which one.+ </para>+ <para>+ Structural corruption can happen due to faulty storage hardware, or+ relation files being overwritten or modified by unrelated software.+ This kind of corruption can also be detected with+ <link linkend="app-initdb-data-checksums"><application>data page+ checksums</application></link>.+ </para>+ <para>+ Relation pages which are correctly formatted, internally consistent, and+ correct relative to their own internal checksums may still contain+ logical corruption. As such, this kind of corruption cannot be detected+ with <application>checksums</application>. Examples include toasted+ values in the main table which lack a corresponding entry in the toast+ table, and tuples in the main table with a Transaction ID that is older+ than the oldest valid Transaction ID in the database or cluster.+ </para>+ <para>+ Multiple causes of logical corruption have been observed in production+ systems, including bugs in the <productname>PostgreSQL</productname>+ server software, faulty and ill-conceived backup and restore tools, and+ user error.+ </para>+ <para>+ Corrupt relations are most concerning in live production environments,+ precisely the same environments where high risk activities are least+ welcome. For this reason, <function>verify_heapam</function> has been+ designed to diagnose corruption without undue risk. It cannot guard+ against all causes of backend crashes, as even executing the calling+ query could be unsafe on a badly corrupted system. Access to <link+ linkend="catalogs-overview">catalog tables</link> are performed and could+ be problematic if the catalogs themselves are corrupted.+ </para>+ <para>+ The design principle adhered to in <function>verify_heapam</function> is+ that, if the rest of the system and server hardware are correct, under+ default options, <function>verify_heapam</function> will not crash the+ server due merely to structural or logical corruption in the target+ table.+ </para>+ <para>+ An experimental option, <literal>check_toast</literal>, exists to+ reconcile the target table against entries in its corresponding toast+ table. This option may change in future, is disabled by default, and is+ known to be slow. It is also unsafe under some conditions. If the+ target relation's corresponding toast table or toast index are corrupt,+ reconciling the target table against toast values may be unsafe. If the+ catalogs, toast table and toast index are uncorrupted, and remain so+ during the check of the target table, reconciling the target table+ against its toast table should be safe.+ </para>+ <para>+ The following optional arguments are recognized:+ </para>+ <variablelist>+ <varlistentry>+ <term>on_error_stop</term>+ <listitem>+ <para>+ If true, corruption checking stops at the end of the first block on+ which any corruptions are found.+ </para>+ <para>+ Defaults to false.+ </para>+ </listitem>+ </varlistentry>+ <varlistentry>+ <term>check_toast</term>+ <listitem>+ <para>+ If this experimental option is true, toasted values are checked gainst+ the corresponding TOAST table.+ </para>+ <para>+ Defaults to false.+ </para>+ </listitem>+ </varlistentry>+ <varlistentry>+ <term>skip</term>+ <listitem>+ <para>+ If not <literal>none</literal>, corruption checking skips blocks that+ are marked as all-visible or all-frozen, as given.+ Valid options are <literal>all-visible</literal>,+ <literal>all-frozen</literal> and <literal>none</literal>.+ </para>+ <para>+ Defaults to <literal>none</literal>.+ </para>+ </listitem>+ </varlistentry>+ <varlistentry>+ <term>startblock</term>+ <listitem>+ <para>+ If specified, corruption checking begins at the specified block,+ skipping all previous blocks. It is an error to specify a+ <literal>startblock</literal> outside the range of blocks in the+ target table.+ </para>+ <para>+ By default, does not skip any blocks.+ </para>+ </listitem>+ </varlistentry>+ <varlistentry>+ <term>endblock</term>+ <listitem>+ <para>+ If specified, corruption checking ends at the specified block,+ skipping all remaining blocks. It is an error to specify an+ <literal>endblock</literal> outside the range of blocks in the target+ table.+ </para>+ <para>+ By default, does not skip any blocks.+ </para>+ </listitem>+ </varlistentry>+ </variablelist>+ <para>+ For each corruption detected, <function>verify_heapam</function> returns+ a row with the following columns:+ </para>+ <variablelist>+ <varlistentry>+ <term>blkno</term>+ <listitem>+ <para>+ The number of the block containing the corrupt page.+ </para>+ </listitem>+ </varlistentry>+ <varlistentry>+ <term>offnum</term>+ <listitem>+ <para>+ The OffsetNumber of the corrupt tuple.+ </para>+ </listitem>+ </varlistentry>+ <varlistentry>+ <term>attnum</term>+ <listitem>+ <para>+ The attribute number of the corrupt column in the tuple, if the+ corruption is specific to a column and not the tuple as a whole.+ </para>+ </listitem>+ </varlistentry>+ <varlistentry>+ <term>msg</term>+ <listitem>+ <para>+ A human readable message describing the corruption in the page.+ </para>+ </listitem>+ </varlistentry>+ </variablelist>+ </listitem>+ </varlistentry>+ </variablelist>
</sect2>
<sect2>
<title>Optional <parameter>heapallindexed</parameter> Verification</title>
<para>
- When the <parameter>heapallindexed</parameter> argument to+ When the <parameter>heapallindexed</parameter> argument to B-Tree
verification functions is <literal>true</literal>, an additional
phase of verification is performed against the table associated with
the target index relation. This consists of a <quote>dummy</quote>
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.cindex aa3f14c019..ca357410a2 100644--- a/src/backend/access/heap/hio.c+++ b/src/backend/access/heap/hio.c@@ -47,6 +47,17 @@ RelationPutHeapTuple(Relation relation,
*/
Assert(!token || HeapTupleHeaderIsSpeculative(tuple->t_data));
+ /*+ * Do not allow tuples with invalid combinations of hint bits to be placed+ * on a page. These combinations are detected as corruption by the+ * contrib/amcheck logic, so if you disable one or both of these+ * assertions, make corresponding changes there.+ */+ Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&+ (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));+ Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&+ (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));+
/* Add the tuple to the page */
pageHeader = BufferGetPage(buffer);
diff --git a/src/backend/access/transam/multixact.c b/src/backend/access/transam/multixact.cindex a2ce617c8c..81752b68eb 100644--- a/src/backend/access/transam/multixact.c+++ b/src/backend/access/transam/multixact.c@@ -735,6 +735,25 @@ ReadNextMultiXactId(void)
return mxid;
}
+/*+ * ReadMultiXactIdRange+ * Get the range of IDs that may still be referenced by a relation.+ */+void+ReadMultiXactIdRange(MultiXactId *oldest, MultiXactId *next)+{+ LWLockAcquire(MultiXactGenLock, LW_SHARED);+ *oldest = MultiXactState->oldestMultiXactId;+ *next = MultiXactState->nextMXact;+ LWLockRelease(MultiXactGenLock);++ if (*oldest < FirstMultiXactId)+ *oldest = FirstMultiXactId;+ if (*next < FirstMultiXactId)+ *next = FirstMultiXactId;+}++
/*
* MultiXactIdCreateFromMembers
* Make a new MultiXactId from the specified set of members
diff --git a/src/include/access/multixact.h b/src/include/access/multixact.hindex 58c42ffe1f..9a30380901 100644--- a/src/include/access/multixact.h+++ b/src/include/access/multixact.h@@ -109,6 +109,7 @@ extern MultiXactId MultiXactIdCreateFromMembers(int nmembers,
MultiXactMember *members);
extern MultiXactId ReadNextMultiXactId(void);
+extern void ReadMultiXactIdRange(MultiXactId *oldest, MultiXactId *next);
extern bool MultiXactIdIsRunning(MultiXactId multi, bool isLockOnly);
extern void MultiXactIdSetOldestMember(void);
extern int GetMultiXactIdMembers(MultiXactId multi, MultiXactMember **xids,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.listindex 4191f94869..bca30f3dde 100644--- a/src/tools/pgindent/typedefs.list+++ b/src/tools/pgindent/typedefs.list@@ -1020,6 +1020,7 @@ HbaToken
HeadlineJsonState
HeadlineParsedText
HeadlineWordEntry
+HeapCheckContext
HeapScanDesc
HeapTuple
HeapTupleData
@@ -2287,6 +2288,7 @@ SimpleStringList
SimpleStringListCell
SingleBoundSortItem
Size
+SkipPages
SlabBlock
SlabChunk
SlabContext
@@ -2788,6 +2790,8 @@ XactCallback
XactCallbackItem
XactEvent
XactLockTableWaitInfo
+XidBoundsViolation+XidCommitStatus
XidHorizonPrefetchState
XidStatus
XmlExpr
--
2.21.1 (Apple Git-122.3)
=
v17-0002-Adding-contrib-module-pg_amcheck.patch (application/octet-stream, 84.9K) download
From 2f4b08a77308c003db5a49020649d6fc61c2352e Mon Sep 17 00:00:00 2001
From: Mark Dilger <mark.dilger@enterprisedb.com>
Date: Mon, 5 Oct 2020 15:43:00 -0700
Subject: [PATCH v17 2/4] Adding contrib module pg_amcheck
Adding new contrib module pg_amcheck, which is a command line
interface for running amcheck's verifications against tables and
indexes.
---
contrib/Makefile | 1 +
contrib/pg_amcheck/.gitignore | 2 +
contrib/pg_amcheck/Makefile | 28 +
contrib/pg_amcheck/pg_amcheck.c | 1281 ++++++++++++++++++++
contrib/pg_amcheck/pg_amcheck.control | 5 +
contrib/pg_amcheck/t/001_basic.pl | 9 +
contrib/pg_amcheck/t/002_nonesuch.pl | 60 +
contrib/pg_amcheck/t/003_check.pl | 231 ++++
contrib/pg_amcheck/t/004_verify_heapam.pl | 489 ++++++++
contrib/pg_amcheck/t/005_opclass_damage.pl | 52 +
doc/src/sgml/contrib.sgml | 1 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/pgamcheck.sgml | 228 ++++
src/tools/msvc/Mkvcbuild.pm | 6 +-
src/tools/pgindent/typedefs.list | 2 +
15 files changed, 2393 insertions(+), 3 deletions(-)
create mode 100644 contrib/pg_amcheck/.gitignore
create mode 100644 contrib/pg_amcheck/Makefile
create mode 100644 contrib/pg_amcheck/pg_amcheck.c
create mode 100644 contrib/pg_amcheck/pg_amcheck.control
create mode 100644 contrib/pg_amcheck/t/001_basic.pl
create mode 100644 contrib/pg_amcheck/t/002_nonesuch.pl
create mode 100644 contrib/pg_amcheck/t/003_check.pl
create mode 100644 contrib/pg_amcheck/t/004_verify_heapam.pl
create mode 100644 contrib/pg_amcheck/t/005_opclass_damage.pl
create mode 100644 doc/src/sgml/pgamcheck.sgml
diff --git a/contrib/Makefile b/contrib/Makefileindex 7a4866e338..0fd4125902 100644--- a/contrib/Makefile+++ b/contrib/Makefile@@ -30,6 +30,7 @@ SUBDIRS = \
old_snapshot \
pageinspect \
passwordcheck \
+ pg_amcheck \
pg_buffercache \
pg_freespacemap \
pg_prewarm \
diff --git a/contrib/pg_amcheck/.gitignore b/contrib/pg_amcheck/.gitignorenew file mode 100644index 0000000000..f8eecf70bf--- /dev/null+++ b/contrib/pg_amcheck/.gitignore@@ -0,0 +1,2 @@+/pg_amcheck+/tmp_check/diff --git a/contrib/pg_amcheck/Makefile b/contrib/pg_amcheck/Makefilenew file mode 100644index 0000000000..74554b9e8d--- /dev/null+++ b/contrib/pg_amcheck/Makefile@@ -0,0 +1,28 @@+# contrib/pg_amcheck/Makefile++PGFILEDESC = "pg_amcheck - detects corruption within database relations"+PGAPPICON = win32++PROGRAM = pg_amcheck+OBJS = \+ $(WIN32RES) \+ pg_amcheck.o++REGRESS_OPTS += --load-extension=amcheck --load-extension=pageinspect+EXTRA_INSTALL += contrib/amcheck contrib/pageinspect++TAP_TESTS = 1++PG_CPPFLAGS = -I$(libpq_srcdir)+PG_LIBS_INTERNAL = -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport)++ifdef USE_PGXS+PG_CONFIG = pg_config+PGXS := $(shell $(PG_CONFIG) --pgxs)+include $(PGXS)+else+subdir = contrib/pg_amcheck+top_builddir = ../..+include $(top_builddir)/src/Makefile.global+include $(top_srcdir)/contrib/contrib-global.mk+endifdiff --git a/contrib/pg_amcheck/pg_amcheck.c b/contrib/pg_amcheck/pg_amcheck.cnew file mode 100644index 0000000000..324cf1cfc8--- /dev/null+++ b/contrib/pg_amcheck/pg_amcheck.c@@ -0,0 +1,1281 @@+/*-------------------------------------------------------------------------+ *+ * pg_amcheck.c+ * Detects corruption within database relations.+ *+ * Copyright (c) 2017-2020, PostgreSQL Global Development Group+ *+ * IDENTIFICATION+ * contrib/pg_amcheck/pg_amcheck.c+ *+ *-------------------------------------------------------------------------+ */+#include "postgres_fe.h"++#include "catalog/pg_am.h"+#include "catalog/pg_class.h"+#include "common/logging.h"+#include "common/username.h"+#include "common/connect.h"+#include "common/string.h"+#include "fe_utils/print.h"+#include "fe_utils/simple_list.h"+#include "fe_utils/string_utils.h"+#include "pg_getopt.h"++const char *usage_text[] = {+ "pg_amcheck is the PostgreSQL command line frontend for the amcheck database corruption checker.",+ "",+ "Usage:",+ " pg_amcheck [OPTION]... [DBNAME [USERNAME]]",+ "",+ "General options:",+ " -V, --version output version information, then exit",+ " -?, --help show this help, then exit",+ " -s, --strict-names require include patterns to match at least one entity each",+ " -o, --on-error-stop stop checking at end of first corrupt page",+ "",+ "Schema checking options:",+ " -n, --schema=PATTERN check relations in the specified schema(s) only",+ " -N, --exclude-schema=PATTERN do NOT check relations in the specified schema(s)",+ "",+ "Table checking options:",+ " -t, --table=PATTERN check the specified table(s) only",+ " -T, --exclude-table=PATTERN do NOT check the specified table(s)",+ " -b, --startblock begin checking table(s) at the given starting block number",+ " -e, --endblock check table(s) only up to the given ending block number",+ " -f, --skip-all-frozen do NOT check blocks marked as all frozen",+ " -v, --skip-all-visible do NOT check blocks marked as all visible",+ "",+ "TOAST table checking options:",+ " -z, --check-toast check associated toast tables and toast indexes",+ " -Z, --skip-toast do NOT check associated toast tables and toast indexes",+ " -B, --toast-startblock begin checking toast table(s) at the given starting block",+ " -E, --toast-endblock check toast table(s) only up to the given ending block",+ "",+ "Index checking options:",+ " -x, --check-indexes check btree indexes associated with tables being checked",+ " -X, --skip-indexes do NOT check any btree indexes",+ " -i, --index=PATTERN check the specified index(es) only",+ " -I, --exclude-index=PATTERN do NOT check the specified index(es)",+ " -c, --check-corrupt check indexes even if their associated table is corrupt",+ " -C, --skip-corrupt do NOT check indexes if their associated table is corrupt",+ " -a, --heapallindexed check index tuples against the table tuples",+ " -A, --no-heapallindexed do NOT check index tuples against the table tuples",+ " -r, --rootdescend search from the root page for each index tuple",+ " -R, --no-rootdescend do NOT search from the root page for each index tuple",+ "",+ "Connection options:",+ " -d, --dbname=DBNAME database name to connect to",+ " -h, --host=HOSTNAME database server host or socket directory",+ " -p, --port=PORT database server port",+ " -U, --username=USERNAME database user name",+ " -w, --no-password never prompt for password",+ " -W, --password force password prompt (should happen automatically)",+ "",+ NULL /* sentinel */+};++typedef struct+AmCheckSettings+{+ char *dbname;+ char *host;+ char *port;+ char *username;+} ConnectOptions;++typedef enum trivalue+{+ TRI_DEFAULT,+ TRI_NO,+ TRI_YES+} trivalue;++typedef struct+{+ PGconn *db; /* connection to backend */+ bool notty; /* stdin or stdout is not a tty (as determined+ * on startup) */+ trivalue getPassword; /* prompt for a username and password */+ const char *progname; /* in case you renamed pg_amcheck */+ bool strict_names; /* The specified names/patterns should to+ * match at least one entity */+ bool on_error_stop; /* The checking of each table should stop+ * after the first corrupt page is found. */+ bool skip_frozen; /* Do not check pages marked all frozen */+ bool skip_visible; /* Do not check pages marked all visible */+ bool check_indexes; /* Check btree indexes */+ bool check_toast; /* Check associated toast tables and indexes */+ bool check_corrupt; /* Check indexes even if table is corrupt */+ bool heapallindexed; /* Perform index to table reconciling checks */+ bool rootdescend; /* Perform index rootdescend checks */+ char *startblock; /* Block number where checking begins */+ char *endblock; /* Block number where checking ends, inclusive */+ char *toaststart; /* Block number where toast checking begins */+ char *toastend; /* Block number where toast checking ends,+ * inclusive */+} AmCheckSettings;++static AmCheckSettings settings;++/*+ * Object inclusion/exclusion lists+ *+ * The string lists record the patterns given by command-line switches,+ * which we then convert to lists of Oids of matching objects.+ */+static SimpleStringList schema_include_patterns = {NULL, NULL};+static SimpleOidList schema_include_oids = {NULL, NULL};+static SimpleStringList schema_exclude_patterns = {NULL, NULL};+static SimpleOidList schema_exclude_oids = {NULL, NULL};++static SimpleStringList table_include_patterns = {NULL, NULL};+static SimpleOidList table_include_oids = {NULL, NULL};+static SimpleStringList table_exclude_patterns = {NULL, NULL};+static SimpleOidList table_exclude_oids = {NULL, NULL};++static SimpleStringList index_include_patterns = {NULL, NULL};+static SimpleOidList index_include_oids = {NULL, NULL};+static SimpleStringList index_exclude_patterns = {NULL, NULL};+static SimpleOidList index_exclude_oids = {NULL, NULL};++/*+ * List of tables to be checked, compiled from above lists.+ */+static SimpleOidList checklist = {NULL, NULL};++/*+ * Strings to be constructed once upon first use. These could be made+ * string constants instead, but that would require embedding knowledge+ * of the single character values for each relkind, such as 'm' for+ * materialized views, which we'd rather not embed here.+ */+static char *table_relkind_quals = NULL;+static char *index_relkind_quals = NULL;++/*+ * Functions to get pointers to the two strings, above, after initializing+ * them upon the first call to the function.+ */+static const char *get_table_relkind_quals(void);+static const char *get_index_relkind_quals(void);++/*+ * Functions for running the various corruption checks.+ */+static void check_tables(SimpleOidList *checklist);+static uint64 check_toast(Oid tbloid);+static uint64 check_table(Oid tbloid, const char *startblock,+ const char *endblock, bool on_error_stop,+ bool check_toast);+static uint64 check_indexes(Oid tbloid, const SimpleOidList *include_oids,+ const SimpleOidList *exclude_oids);+static uint64 check_index(const char *idxoid, const char *idxname,+ const char *tblname);++/*+ * Functions implementing standard command line behaviors.+ */+static void parse_cli_options(int argc, char *argv[],+ ConnectOptions *connOpts);+static void usage(void);+static void showVersion(void);+static void NoticeProcessor(void *arg, const char *message);++/*+ * Functions for converting command line options that include or exclude+ * schemas, tables, and indexes by pattern into internally useful lists of+ * Oids for objects that match those patterns.+ */+static void expand_schema_name_patterns(const SimpleStringList *patterns,+ const SimpleOidList *exclude_oids,+ SimpleOidList *oids,+ bool strict_names);+static void expand_relkind_name_patterns(const SimpleStringList *patterns,+ const SimpleOidList *exclude_nsp_oids,+ const SimpleOidList *exclude_oids,+ SimpleOidList *oids,+ bool strict_names,+ const char *missing_errtext,+ const char *relkind_quals);+static void expand_table_name_patterns(const SimpleStringList *patterns,+ const SimpleOidList *exclude_nsp_oids,+ const SimpleOidList *exclude_oids,+ SimpleOidList *oids,+ bool strict_names);+static void expand_index_name_patterns(const SimpleStringList *patterns,+ const SimpleOidList *exclude_nsp_oids,+ const SimpleOidList *exclude_oids,+ SimpleOidList *oids,+ bool strict_names);+static void get_table_check_list(const SimpleOidList *include_nsp,+ const SimpleOidList *exclude_nsp,+ const SimpleOidList *include_tbl,+ const SimpleOidList *exclude_tbl,+ SimpleOidList *checklist);+static PGresult *ExecuteSqlQuery(const char *query, char **error);+static PGresult *ExecuteSqlQueryOrDie(const char *query);++static void append_csv_oids(PQExpBuffer querybuf, const SimpleOidList *oids);+static void apply_filter(PQExpBuffer querybuf, const char *lval,+ const SimpleOidList *oids, bool include);++#define fatal(...) do { pg_log_error(__VA_ARGS__); exit(1); } while(0)++/* Like fatal(), but with a complaint about a particular query. */+static void+die_on_query_failure(const char *query)+{+ pg_log_error("query failed: %s",+ PQerrorMessage(settings.db));+ fatal("query was: %s", query);+}++#define EXIT_BADCONN 2++int+main(int argc, char **argv)+{+ ConnectOptions connOpts;+ bool have_password = false;+ char *password = NULL;+ bool new_pass;++ pg_logging_init(argv[0]);+ set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_amcheck"));++ if (argc > 1)+ {+ if ((strcmp(argv[1], "-?") == 0) ||+ (argc == 2 && (strcmp(argv[1], "--help") == 0)))+ {+ usage();+ exit(EXIT_SUCCESS);+ }+ if (strcmp(argv[1], "--version") == 0 || strcmp(argv[1], "-V") == 0)+ {+ showVersion();+ exit(EXIT_SUCCESS);+ }+ }++ memset(&settings, 0, sizeof(settings));+ settings.progname = get_progname(argv[0]);++ settings.db = NULL;+ setDecimalLocale();++ settings.notty = (!isatty(fileno(stdin)) || !isatty(fileno(stdout)));++ settings.getPassword = TRI_DEFAULT;++ /*+ * Default behaviors for user settable options. Note that these default+ * to doing all the safe checks and none of the unsafe ones, on the theory+ * that if a user says "pg_amcheck mydb" without specifying any additional+ * options, we should check everything we know how to check without+ * risking any backend aborts.+ */++ settings.on_error_stop = false;+ settings.skip_frozen = false;+ settings.skip_visible = false;++ /* Index checking options */+ settings.check_indexes = false;+ settings.check_corrupt = false;+ settings.heapallindexed = false;+ settings.rootdescend = false;++ /*+ * Reconciling toasted attributes from the main table against the toast+ * table can crash the backend if the toast table or index are corrupt.+ * We can optionally check the toast table and then the toast index prior+ * to checking the main table, but if the toast table or index are+ * concurrently corrupted after we conclude they are valid, the check of+ * the main table can crash the backend. The oneous is on any caller who+ * enables this option to make certain the environment is sufficiently+ * stable that concurrent corruptions of the toast is not possible.+ */+ settings.check_toast = false;++ parse_cli_options(argc, argv, &connOpts);++ if (settings.getPassword == TRI_YES)+ {+ /*+ * We can't be sure yet of the username that will be used, so don't+ * offer a potentially wrong one. Typical uses of this option are+ * noninteractive anyway.+ */+ password = simple_prompt("Password: ", false);+ have_password = true;+ }++ /* loop until we have a password if requested by backend */+ do+ {+#define ARRAY_SIZE 8+ const char **keywords = pg_malloc(ARRAY_SIZE * sizeof(*keywords));+ const char **values = pg_malloc(ARRAY_SIZE * sizeof(*values));++ keywords[0] = "host";+ values[0] = connOpts.host;+ keywords[1] = "port";+ values[1] = connOpts.port;+ keywords[2] = "user";+ values[2] = connOpts.username;+ keywords[3] = "password";+ values[3] = have_password ? password : NULL;+ keywords[4] = "dbname"; /* see do_connect() */+ if (connOpts.dbname == NULL)+ {+ if (getenv("PGDATABASE"))+ values[4] = getenv("PGDATABASE");+ else if (getenv("PGUSER"))+ values[4] = getenv("PGUSER");+ else+ values[4] = "postgres";+ }+ else+ values[4] = connOpts.dbname;+ keywords[5] = "fallback_application_name";+ values[5] = settings.progname;+ keywords[6] = "client_encoding";+ values[6] = (settings.notty ||+ getenv("PGCLIENTENCODING")) ? NULL : "auto";+ keywords[7] = NULL;+ values[7] = NULL;++ new_pass = false;+ settings.db = PQconnectdbParams(keywords, values, true);+ if (settings.db == NULL)+ {+ pg_log_error("no connection to server after initial attempt");+ exit(EXIT_BADCONN);+ }++ free(keywords);+ free(values);++ if (PQstatus(settings.db) == CONNECTION_BAD &&+ PQconnectionNeedsPassword(settings.db) &&+ !have_password &&+ settings.getPassword != TRI_NO)+ {+ /*+ * Before closing the old PGconn, extract the user name that was+ * actually connected with.+ */+ const char *realusername = PQuser(settings.db);+ char *password_prompt;++ if (realusername && realusername[0])+ password_prompt = psprintf(_("Password for user %s: "),+ realusername);+ else+ password_prompt = pg_strdup(_("Password: "));+ PQfinish(settings.db);++ password = simple_prompt(password_prompt, false);+ free(password_prompt);+ have_password = true;+ new_pass = true;+ }+ } while (new_pass);++ if (!settings.db)+ {+ pg_log_error("no connection to server");+ exit(EXIT_BADCONN);+ }++ if (PQstatus(settings.db) == CONNECTION_BAD)+ {+ pg_log_error("could not connect to server: %s",+ PQerrorMessage(settings.db));+ PQfinish(settings.db);+ exit(EXIT_BADCONN);+ }++ /*+ * Expand schema, table, and index exclusion patterns, if any. Note that+ * non-matching exclusion patterns are not an error, even when+ * --strict-names was specified.+ */+ expand_schema_name_patterns(&schema_exclude_patterns, NULL,+ &schema_exclude_oids, false);+ expand_table_name_patterns(&table_exclude_patterns, NULL, NULL,+ &table_exclude_oids, false);+ expand_index_name_patterns(&index_exclude_patterns, NULL, NULL,+ &index_exclude_oids, false);++ /* Expand schema selection patterns into Oid lists */+ if (schema_include_patterns.head != NULL)+ {+ expand_schema_name_patterns(&schema_include_patterns,+ &schema_exclude_oids,+ &schema_include_oids,+ settings.strict_names);+ if (schema_include_oids.head == NULL)+ fatal("no matching schemas were found");+ }++ /* Expand table selection patterns into Oid lists */+ if (table_include_patterns.head != NULL)+ {+ expand_table_name_patterns(&table_include_patterns,+ &schema_exclude_oids,+ &table_exclude_oids,+ &table_include_oids,+ settings.strict_names);+ if (table_include_oids.head == NULL)+ fatal("no matching tables were found");+ }++ /* Expand index selection patterns into Oid lists */+ if (index_include_patterns.head != NULL)+ {+ expand_index_name_patterns(&index_include_patterns,+ &schema_exclude_oids,+ &index_exclude_oids,+ &index_include_oids,+ settings.strict_names);+ if (index_include_oids.head == NULL)+ fatal("no matching indexes were found");+ }++ /*+ * Compile list of all tables to be checked based on namespace and table+ * includes and excludes.+ */+ get_table_check_list(&schema_include_oids, &schema_exclude_oids,+ &table_include_oids, &table_exclude_oids, &checklist);++ PQsetNoticeProcessor(settings.db, NoticeProcessor, NULL);++ /*+ * All information about corrupt indexes are returned via ereport, not as+ * tuples. We want all the details to report if corruption exists.+ */+ PQsetErrorVerbosity(settings.db, PQERRORS_VERBOSE);++ check_tables(&checklist);++ return 0;+}++/*+ * Conditionally add a restriction to a query such that lval must be an Oid in+ * the given list of Oids, except that for a null or empty oids list argument,+ * no filtering is done and we return without having modified the query buffer.+ *+ * The query argument must already have begun the WHERE clause and must be in a+ * state where we can append an AND clause. No checking of this requirement is+ * done here.+ *+ * On return, the query buffer will be extended with an AND clause that filters+ * only those rows where the lval is an Oid present in the given list of oids.+ */+static inline void+include_filter(PQExpBuffer querybuf, const char *lval, const SimpleOidList *oids)+{+ apply_filter(querybuf, lval, oids, true);+}++/*+ * Same as include_filter, above, except that for a non-null, non-empty oids+ * list, the lval is restricted to not be any of the values in the list.+ */+static inline void+exclude_filter(PQExpBuffer querybuf, const char *lval, const SimpleOidList *oids)+{+ apply_filter(querybuf, lval, oids, false);+}++/*+ * Check each table from the given checklist per the user specified options.+ */+static void+check_tables(SimpleOidList *checklist)+{+ const SimpleOidListCell *cell;++ for (cell = checklist->head; cell; cell = cell->next)+ {+ uint64 corruptions = 0;+ bool reconcile_toast;++ /*+ * If we skip checking the toast table, or if during the check we+ * detect any toast table corruption, the main table checks below must+ * not reconcile toasted attributes against the toast table, as such+ * accesses to the toast table might crash the backend. Instead, skip+ * such reconciliations for this table.+ *+ * This protection contains a race condition; the toast table or index+ * could become corrupted concurrently with our checks, but prevention+ * of such concurrent corruption is documented as the caller's+ * reponsibility, so we don't worry about it here.+ */+ reconcile_toast = false;+ if (settings.check_toast)+ {+ if (check_toast(cell->val) == 0)+ reconcile_toast = true;+ }++ corruptions = check_table(cell->val,+ settings.startblock,+ settings.endblock,+ settings.on_error_stop,+ reconcile_toast);++ if (settings.check_indexes)+ {+ bool old_heapallindexed;++ /* Optionally skip the index checks for a corrupt table. */+ if (corruptions && !settings.check_corrupt)+ continue;++ /*+ * The btree checking logic which optionally checks the contents+ * of an index against the corresponding table has not yet been+ * sufficiently hardened against corrupt tables. In particular,+ * when called with heapallindexed true, it segfaults if the file+ * backing the table relation has been erroneously unlinked. In+ * any event, it seems unwise to reconcile an index against its+ * table when we already know the table is corrupt.+ */+ old_heapallindexed = settings.heapallindexed;+ if (corruptions)+ settings.heapallindexed = false;++ corruptions += check_indexes(cell->val,+ &index_include_oids,+ &index_exclude_oids);++ settings.heapallindexed = old_heapallindexed;+ }+ }+}++/*+ * For a given main table relation, returns the associated toast table,+ * or InvalidOid if none exists.+ */+static Oid+get_toast_oid(Oid tbloid)+{+ PQExpBuffer querybuf = createPQExpBuffer();+ PGresult *res;+ char *error = NULL;+ Oid result = InvalidOid;++ appendPQExpBuffer(querybuf,+ "SELECT c.reltoastrelid"+ "\nFROM pg_catalog.pg_class c"+ "\nWHERE c.oid = %u",+ tbloid);+ res = ExecuteSqlQuery(querybuf->data, &error);+ if (PQresultStatus(res) == PGRES_TUPLES_OK && PQntuples(res) > 0)+ result = atooid(PQgetvalue(res, 0, 0));+ else if (error)+ die_on_query_failure(querybuf->data);++ PQclear(res);+ destroyPQExpBuffer(querybuf);++ return result;+}++/*+ * For the given main table relation, checks the associated toast table and+ * index, in any. This should be performed *before* checking the main table+ * relation, as the checks inside verify_heapam assume both the toast table and+ * toast index are usable.+ *+ * Returns the number of corruptions detected.+ */+static uint64+check_toast(Oid tbloid)+{+ Oid toastoid;+ uint64 corruption_cnt = 0;++ if (settings.db == NULL)+ fatal("no connection on entry to check_toast");++ toastoid = get_toast_oid(tbloid);+ if (OidIsValid(toastoid))+ {+ corruption_cnt = check_table(toastoid, settings.toaststart,+ settings.toastend, settings.on_error_stop,+ false);++ /*+ * If the toast table is corrupt, checking the index is not safe.+ * There is a race condition here, as the toast table could be+ * concurrently corrupted, but preventing concurrent corruption is the+ * caller's responsibility, not ours.+ */+ if (corruption_cnt == 0)+ corruption_cnt += check_indexes(toastoid, NULL, NULL);+ }++ return corruption_cnt;+}++/*+ * Checks the given table for corruption, returning the number of corruptions+ * detected and printed to the user.+ */+static uint64+check_table(Oid tbloid, const char *startblock, const char *endblock,+ bool on_error_stop, bool check_toast)+{+ PQExpBuffer querybuf;+ PGresult *res;+ int i;+ char *skip;+ char *toast;+ const char *stop;+ char *error = NULL;+ uint64 corruption_cnt = 0;++ if (settings.db == NULL)+ fatal("no connection on entry to check_table");++ if (startblock == NULL)+ startblock = "NULL";+ if (endblock == NULL)+ endblock = "NULL";+ if (settings.skip_frozen)+ skip = pg_strdup("'all frozen'");+ else if (settings.skip_visible)+ skip = pg_strdup("'all visible'");+ else+ skip = pg_strdup("'none'");+ stop = (on_error_stop) ? "true" : "false";+ toast = (check_toast) ? "true" : "false";++ querybuf = createPQExpBuffer();++ appendPQExpBuffer(querybuf,+ "SELECT c.relname, v.blkno, v.offnum, v.attnum, v.msg "+ "FROM verify_heapam("+ "relation := %u, "+ "on_error_stop := %s, "+ "skip := %s, "+ "check_toast := %s, "+ "startblock := %s, "+ "endblock := %s) v, "+ "pg_catalog.pg_class c "+ "WHERE c.oid = %u",+ tbloid, stop, skip, toast, startblock, endblock, tbloid);++ res = ExecuteSqlQuery(querybuf->data, &error);+ if (PQresultStatus(res) == PGRES_TUPLES_OK && PQntuples(res) > 0)+ {+ corruption_cnt += PQntuples(res);+ for (i = 0; i < PQntuples(res); i++)+ {+ printf("(relname=%s,blkno=%s,offnum=%s,attnum=%s)\n%s\n",+ PQgetvalue(res, i, 0), /* relname */+ PQgetvalue(res, i, 1), /* blkno */+ PQgetvalue(res, i, 2), /* offnum */+ PQgetvalue(res, i, 3), /* attnum */+ PQgetvalue(res, i, 4)); /* msg */+ }+ }+ else if (error)+ {+ corruption_cnt++;+ printf("%s\n", error);+ pfree(error);+ }++ PQclear(res);+ destroyPQExpBuffer(querybuf);+ return corruption_cnt;+}++static uint64+check_indexes(Oid tbloid, const SimpleOidList *include_oids,+ const SimpleOidList *exclude_oids)+{+ PQExpBuffer querybuf;+ PGresult *res;+ int i;+ char *error = NULL;+ uint64 corruption_cnt = 0;++ if (settings.db == NULL)+ fatal("no connection on entry to check_indexes");++ querybuf = createPQExpBuffer();+ appendPQExpBuffer(querybuf,+ "SELECT i.indexrelid, ci.relname, ct.relname"+ "\nFROM pg_catalog.pg_index i, pg_catalog.pg_class ci, "+ "pg_catalog.pg_class ct"+ "\nWHERE i.indexrelid = ci.oid"+ "\n AND i.indrelid = ct.oid"+ "\n AND ci.relam = %u"+ "\n AND i.indrelid = %u",+ BTREE_AM_OID, tbloid);+ include_filter(querybuf, "i.indexrelid", include_oids);+ exclude_filter(querybuf, "i.indexrelid", exclude_oids);++ res = ExecuteSqlQuery(querybuf->data, &error);+ if (PQresultStatus(res) == PGRES_TUPLES_OK)+ {+ for (i = 0; i < PQntuples(res); i++)+ corruption_cnt += check_index(PQgetvalue(res, i, 0),+ PQgetvalue(res, i, 1),+ PQgetvalue(res, i, 2));+ }+ else if (error)+ {+ corruption_cnt++;+ printf("%s\n", error);+ pfree(error);+ }++ PQclear(res);+ destroyPQExpBuffer(querybuf);++ return corruption_cnt;+}++static uint64+check_index(const char *idxoid, const char *idxname, const char *tblname)+{+ PQExpBuffer querybuf;+ PGresult *res;+ uint64 corruption_cnt = 0;++ if (settings.db == NULL)+ fatal("no connection on entry to check_index");+ if (idxname == NULL)+ fatal("no index name on entry to check_index");+ if (tblname == NULL)+ fatal("no table name on entry to check_index");++ querybuf = createPQExpBuffer();+ appendPQExpBuffer(querybuf,+ "SELECT bt_index_parent_check('%s'::regclass, %s, %s)",+ idxoid,+ settings.heapallindexed ? "true" : "false",+ settings.rootdescend ? "true" : "false");+ res = PQexec(settings.db, querybuf->data);+ if (PQresultStatus(res) != PGRES_TUPLES_OK)+ {+ corruption_cnt++;+ printf("index check failed for index %s of table %s:\n",+ idxname, tblname);+ printf("%s", PQerrorMessage(settings.db));+ }++ PQclear(res);+ destroyPQExpBuffer(querybuf);++ return corruption_cnt;+}++static void+parse_cli_options(int argc, char *argv[], ConnectOptions *connOpts)+{+ static struct option long_options[] =+ {+ {"check-corrupt", no_argument, NULL, 'c'},+ {"check-indexes", no_argument, NULL, 'x'},+ {"check-toast", no_argument, NULL, 'z'},+ {"dbname", required_argument, NULL, 'd'},+ {"endblock", required_argument, NULL, 'e'},+ {"exclude-index", required_argument, NULL, 'I'},+ {"exclude-schema", required_argument, NULL, 'N'},+ {"exclude-table", required_argument, NULL, 'T'},+ {"heapallindexed", no_argument, NULL, 'a'},+ {"help", optional_argument, NULL, '?'},+ {"host", required_argument, NULL, 'h'},+ {"index", required_argument, NULL, 'i'},+ {"no-heapallindexed", no_argument, NULL, 'A'},+ {"no-password", no_argument, NULL, 'w'},+ {"no-rootdescend", no_argument, NULL, 'R'},+ {"on-error-stop", no_argument, NULL, 'o'},+ {"password", no_argument, NULL, 'W'},+ {"port", required_argument, NULL, 'p'},+ {"rootdescend", no_argument, NULL, 'r'},+ {"schema", required_argument, NULL, 'n'},+ {"skip-all-frozen", no_argument, NULL, 'f'},+ {"skip-all-visible", no_argument, NULL, 'v'},+ {"skip-corrupt", no_argument, NULL, 'C'},+ {"skip-indexes", no_argument, NULL, 'X'},+ {"skip-toast", no_argument, NULL, 'Z'},+ {"startblock", required_argument, NULL, 'b'},+ {"strict-names", no_argument, NULL, 's'},+ {"table", required_argument, NULL, 't'},+ {"toast-endblock", required_argument, NULL, 'E'},+ {"toast-startblock", required_argument, NULL, 'B'},+ {"username", required_argument, NULL, 'U'},+ {"version", no_argument, NULL, 'V'},+ {NULL, 0, NULL, 0}+ };++ int optindex;+ int c;++ memset(connOpts, 0, sizeof *connOpts);++ while ((c = getopt_long(argc, argv, "aAb:B:cCd:e:E:fh:i:I:n:N:op:rRst:T:U:vVwWxXzZ?1",+ long_options, &optindex)) != -1)+ {+ switch (c)+ {+ case 'a':+ settings.heapallindexed = true;+ break;+ case 'A':+ settings.heapallindexed = false;+ break;+ case 'b':+ settings.startblock = pg_strdup(optarg);+ break;+ case 'B':+ settings.toaststart = pg_strdup(optarg);+ break;+ case 'c':+ settings.check_corrupt = true;+ break;+ case 'C':+ settings.check_corrupt = false;+ break;+ case 'd':+ connOpts->dbname = pg_strdup(optarg);+ break;+ case 'e':+ settings.endblock = pg_strdup(optarg);+ break;+ case 'E':+ settings.toastend = pg_strdup(optarg);+ break;+ case 'f':+ settings.skip_frozen = true;+ break;+ case 'h':+ connOpts->host = pg_strdup(optarg);+ break;+ case 'i':+ simple_string_list_append(&index_include_patterns, optarg);+ break;+ case 'I':+ simple_string_list_append(&index_exclude_patterns, optarg);+ break;+ case 'n': /* include schema(s) */+ simple_string_list_append(&schema_include_patterns, optarg);+ break;+ case 'N': /* exclude schema(s) */+ simple_string_list_append(&schema_exclude_patterns, optarg);+ break;+ case 'o':+ settings.on_error_stop = true;+ break;+ case 'p':+ connOpts->port = pg_strdup(optarg);+ break;+ case 's':+ settings.strict_names = true;+ break;+ case 'r':+ settings.rootdescend = true;+ break;+ case 'R':+ settings.rootdescend = false;+ break;+ case 't': /* include table(s) */+ simple_string_list_append(&table_include_patterns, optarg);+ break;+ case 'T': /* exclude table(s) */+ simple_string_list_append(&table_exclude_patterns, optarg);+ break;+ case 'U':+ connOpts->username = pg_strdup(optarg);+ break;+ case 'v':+ settings.skip_visible = true;+ break;+ case 'V':+ showVersion();+ exit(EXIT_SUCCESS);+ case 'w':+ settings.getPassword = TRI_NO;+ break;+ case 'W':+ settings.getPassword = TRI_YES;+ break;+ case 'x':+ settings.check_indexes = true;+ break;+ case 'X':+ settings.check_indexes = false;+ break;+ case 'z':+ settings.check_toast = true;+ break;+ case 'Z':+ settings.check_toast = false;+ break;+ case '?':+ if (optind <= argc &&+ strcmp(argv[optind - 1], "-?") == 0)+ {+ /* actual help option given */+ usage();+ exit(EXIT_SUCCESS);+ }+ else+ {+ /* getopt error (unknown option or missing argument) */+ goto unknown_option;+ }+ break;+ case 1:+ {+ if (!optarg || strcmp(optarg, "options") == 0)+ usage();+ else+ goto unknown_option;++ exit(EXIT_SUCCESS);+ }+ break;+ default:+ unknown_option:+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),+ settings.progname);+ exit(EXIT_FAILURE);+ break;+ }+ }++ /*+ * if we still have arguments, use it as the database name and username+ */+ while (argc - optind >= 1)+ {+ if (!connOpts->dbname)+ connOpts->dbname = argv[optind];+ else if (!connOpts->username)+ connOpts->username = argv[optind];+ else+ pg_log_warning("extra command-line argument \"%s\" ignored",+ argv[optind]);++ optind++;+ }++}++/*+ * usage+ *+ * print out command line arguments+ */+static void+usage(void)+{+ int lineno;++ for (lineno = 0; usage_text[lineno]; lineno++)+ printf("%s\n", usage_text[lineno]);+ printf("Report bugs to <%s>.\n", PACKAGE_BUGREPORT);+ printf("%s home page: <%s>\n", PACKAGE_NAME, PACKAGE_URL);+}++static void+showVersion(void)+{+ puts("pg_amcheck (PostgreSQL) " PG_VERSION);+}++/*+ * for backend Notice messages (INFO, WARNING, etc)+ */+static void+NoticeProcessor(void *arg, const char *message)+{+ (void) arg; /* not used */+ pg_log_info("%s", message);+}++/*+ * Helper function for apply_filter, below.+ */+static void+append_csv_oids(PQExpBuffer querybuf, const SimpleOidList *oids)+{+ const SimpleOidListCell *cell;+ const char *comma;++ for (comma = "", cell = oids->head; cell; comma = ", ", cell = cell->next)+ appendPQExpBuffer(querybuf, "%s%u", comma, cell->val);+}++/*+ * Internal implementation of include_filter and exclude_filter+ */+static void+apply_filter(PQExpBuffer querybuf, const char *lval, const SimpleOidList *oids,+ bool include)+{+ if (!oids || !oids->head)+ return;+ if (include)+ appendPQExpBuffer(querybuf, "\nAND %s OPERATOR(pg_catalog.=) ANY(array[", lval);+ else+ appendPQExpBuffer(querybuf, "\nAND %s OPERATOR(pg_catalog.!=) ALL(array[", lval);+ append_csv_oids(querybuf, oids);+ appendPQExpBuffer(querybuf, "]::OID[])");+}++/*+ * Find and append to the given Oid list the Oids of all schemas matching the+ * given list of patterns but not included in the given list of excluded Oids.+ */+static void+expand_schema_name_patterns(const SimpleStringList *patterns,+ const SimpleOidList *exclude_nsp,+ SimpleOidList *oids,+ bool strict_names)+{+ PQExpBuffer querybuf;+ PGresult *res;+ SimpleStringListCell *cell;+ int i;++ if (settings.db == NULL)+ fatal("no connection on entry to expand_schema_name_patterns");++ if (patterns->head == NULL)+ return; /* nothing to do */++ querybuf = createPQExpBuffer();++ /*+ * The loop below runs multiple SELECTs might sometimes result in+ * duplicate entries in the Oid list, but we don't care.+ */++ for (cell = patterns->head; cell; cell = cell->next)+ {+ appendPQExpBufferStr(querybuf,+ "SELECT oid FROM pg_catalog.pg_namespace n\n");+ processSQLNamePattern(settings.db, querybuf, cell->val, false,+ false, NULL, "n.nspname", NULL, NULL);+ exclude_filter(querybuf, "n.oid", exclude_nsp);++ res = ExecuteSqlQueryOrDie(querybuf->data);+ if (strict_names && PQntuples(res) == 0)+ fatal("no matching schemas were found for pattern \"%s\"",+ cell->val);++ for (i = 0; i < PQntuples(res); i++)+ {+ simple_oid_list_append(oids, atooid(PQgetvalue(res, i, 0)));+ }++ PQclear(res);+ resetPQExpBuffer(querybuf);+ }++ destroyPQExpBuffer(querybuf);+}++/*+ * Find and append to the given Oid list the Oids of all relations matching the+ * given list of patterns but not included in the given list of excluded Oids+ * nor in one of the given excluded namespaces. The relations are filtered by+ * the given schema_quals. They are further filtered by the given+ * relkind_quals, allowing the caller to restrict the relations to just indexes+ * or tables. The missing_errtext should be a message for use in error+ * messages if no matching relations are found and strict_names was specified.+ */+static void+expand_relkind_name_patterns(const SimpleStringList *patterns,+ const SimpleOidList *exclude_nsp_oids,+ const SimpleOidList *exclude_oids,+ SimpleOidList *oids,+ bool strict_names,+ const char *missing_errtext,+ const char *relkind_quals)+{+ PQExpBuffer querybuf;+ PGresult *res;+ SimpleStringListCell *cell;+ int i;++ if (settings.db == NULL)+ fatal("no connection on entry to expand_relkind_name_patterns");++ if (patterns->head == NULL)+ return; /* nothing to do */++ querybuf = createPQExpBuffer();++ /*+ * this might sometimes result in duplicate entries in the Oid list, but+ * we don't care.+ */++ for (cell = patterns->head; cell; cell = cell->next)+ {+ /*+ * Query must remain ABSOLUTELY devoid of unqualified names. This+ * would be unnecessary given a pg_table_is_visible() variant taking a+ * search_path argument.+ */+ appendPQExpBuffer(querybuf,+ "SELECT c.oid"+ "\nFROM pg_catalog.pg_class c"+ "\n LEFT JOIN pg_catalog.pg_namespace n"+ "\n ON n.oid OPERATOR(pg_catalog.=) c.relnamespace"+ "\nWHERE c.relkind OPERATOR(pg_catalog.=) %s\n",+ relkind_quals);+ exclude_filter(querybuf, "c.oid", exclude_oids);+ exclude_filter(querybuf, "n.oid", exclude_nsp_oids);+ processSQLNamePattern(settings.db, querybuf, cell->val, true,+ false, "n.nspname", "c.relname", NULL, NULL);+ res = ExecuteSqlQueryOrDie(querybuf->data);+ if (strict_names && PQntuples(res) == 0)+ fatal("%s \"%s\"", missing_errtext, cell->val);++ for (i = 0; i < PQntuples(res); i++)+ simple_oid_list_append(oids, atooid(PQgetvalue(res, i, 0)));++ PQclear(res);+ resetPQExpBuffer(querybuf);+ }++ destroyPQExpBuffer(querybuf);+}++/*+ * Find the Oids of all tables matching the given list of patterns,+ * and append them to the given Oid list.+ */+static void+expand_table_name_patterns(const SimpleStringList *patterns, const SimpleOidList *exclude_nsp_oids,+ const SimpleOidList *exclude_oids, SimpleOidList *oids, bool strict_names)+{+ expand_relkind_name_patterns(patterns, exclude_nsp_oids, exclude_oids, oids, strict_names,+ "no matching tables were found for pattern",+ get_table_relkind_quals());+}++/*+ * Find the Oids of all indexes matching the given list of patterns,+ * and append them to the given Oid list.+ */+static void+expand_index_name_patterns(const SimpleStringList *patterns, const SimpleOidList *exclude_nsp_oids,+ const SimpleOidList *exclude_oids, SimpleOidList *oids, bool strict_names)+{+ expand_relkind_name_patterns(patterns, exclude_nsp_oids, exclude_oids, oids, strict_names,+ "no matching indexes were found for pattern",+ get_index_relkind_quals());+}++static void+get_table_check_list(const SimpleOidList *include_nsp, const SimpleOidList *exclude_nsp,+ const SimpleOidList *include_tbl, const SimpleOidList *exclude_tbl,+ SimpleOidList *checklist)+{+ PQExpBuffer querybuf;+ PGresult *res;+ int i;++ if (settings.db == NULL)+ fatal("no connection on entry to get_table_check_list");++ querybuf = createPQExpBuffer();++ appendPQExpBuffer(querybuf,+ "SELECT c.oid"+ "\nFROM pg_catalog.pg_class c, pg_catalog.pg_namespace n"+ "\nWHERE n.oid OPERATOR(pg_catalog.=) c.relnamespace"+ "\n AND c.relkind OPERATOR(pg_catalog.=) %s\n",+ get_table_relkind_quals());+ include_filter(querybuf, "n.oid", include_nsp);+ exclude_filter(querybuf, "n.oid", exclude_nsp);+ include_filter(querybuf, "c.oid", include_tbl);+ exclude_filter(querybuf, "c.oid", exclude_tbl);++ res = ExecuteSqlQueryOrDie(querybuf->data);+ for (i = 0; i < PQntuples(res); i++)+ simple_oid_list_append(checklist, atooid(PQgetvalue(res, i, 0)));++ PQclear(res);+ destroyPQExpBuffer(querybuf);+}++static PGresult *+ExecuteSqlQueryOrDie(const char *query)+{+ PGresult *res;++ res = PQexec(settings.db, query);+ if (PQresultStatus(res) != PGRES_TUPLES_OK)+ die_on_query_failure(query);+ return res;+}++/*+ * Execute the given SQL query. This function should only be used for queries+ * which are not expected to fail under normal circumstances, as failures will+ * result in the printing of error messages, which will look a bit messy when+ * interleaved with corruption reports.+ *+ * On error, use the supplied error_context string and the error string+ * returned from the database connection to print an error message for the+ * user.+ *+ * The error_context argument is pfree'd by us at the end of the call.+ */+static PGresult *+ExecuteSqlQuery(const char *query, char **error)+{+ PGresult *res;++ res = PQexec(settings.db, query);+ if (PQresultStatus(res) != PGRES_TUPLES_OK)+ *error = pstrdup(PQerrorMessage(settings.db));+ return res;+}++/*+ * Return the cached relkind quals string for tables, computing it first if we+ * don't have one cached.+ */+static const char *+get_table_relkind_quals(void)+{+ if (!table_relkind_quals)+ table_relkind_quals = psprintf("ANY(array['%c', '%c', '%c'])",+ RELKIND_RELATION, RELKIND_MATVIEW,+ RELKIND_PARTITIONED_TABLE);+ return table_relkind_quals;+}++/*+ * Return the cached relkind quals string for indexes, computing it first if we+ * don't have one cached.+ */+static const char *+get_index_relkind_quals(void)+{+ if (!index_relkind_quals)+ index_relkind_quals = psprintf("'%c'", RELKIND_INDEX);+ return index_relkind_quals;+}diff --git a/contrib/pg_amcheck/pg_amcheck.control b/contrib/pg_amcheck/pg_amcheck.controlnew file mode 100644index 0000000000..395f368101--- /dev/null+++ b/contrib/pg_amcheck/pg_amcheck.control@@ -0,0 +1,5 @@+# pg_amcheck extension+comment = 'command-line tool for verifying relation integrity'+default_version = '1.3'+module_pathname = '$libdir/pg_amcheck'+relocatable = truediff --git a/contrib/pg_amcheck/t/001_basic.pl b/contrib/pg_amcheck/t/001_basic.plnew file mode 100644index 0000000000..dfa0ae9e06--- /dev/null+++ b/contrib/pg_amcheck/t/001_basic.pl@@ -0,0 +1,9 @@+use strict;+use warnings;++use TestLib;+use Test::More tests => 8;++program_help_ok('pg_amcheck');+program_version_ok('pg_amcheck');+program_options_handling_ok('pg_amcheck');diff --git a/contrib/pg_amcheck/t/002_nonesuch.pl b/contrib/pg_amcheck/t/002_nonesuch.plnew file mode 100644index 0000000000..68be9c6585--- /dev/null+++ b/contrib/pg_amcheck/t/002_nonesuch.pl@@ -0,0 +1,60 @@+use strict;+use warnings;++use PostgresNode;+use TestLib;+use Test::More tests => 14;++# Test set-up+my ($node, $port);+$node = get_new_node('test');+$node->init;+$node->start;+$port = $node->port;++# Load the amcheck extension, upon which pg_amcheck depends+$node->safe_psql('postgres', q(CREATE EXTENSION amcheck));++#########################################+# Test connecting to a non-existent database++command_fails_like(+ [ 'pg_amcheck', '-p', "$port", 'qqq' ],+ qr/\Qpg_amcheck: error: could not connect to server: FATAL: database "qqq" does not exist\E/,+ 'connecting to a non-existent database');++#########################################+# Test connecting with a non-existent user++command_fails_like(+ [ 'pg_amcheck', '-p', "$port", '-U=no_such_user' ],+ qr/\Qpg_amcheck: error: could not connect to server: FATAL: role "=no_such_user" does not exist\E/,+ 'connecting with a non-existent user');++#########################################+# Test checking a non-existent schema, table, and patterns with --strict-names++command_fails_like(+ [ 'pg_amcheck', '-p', "$port", '-n', 'nonexistent' ],+ qr/\Qpg_amcheck: error: no matching schemas were found\E/,+ 'checking a non-existent schema');++command_fails_like(+ [ 'pg_amcheck', '-p', "$port", '-t', 'nonexistent' ],+ qr/\Qpg_amcheck: error: no matching tables were found\E/,+ 'checking a non-existent table');++command_fails_like(+ [ 'pg_amcheck', '-p', "$port", '--strict-names', '-n', 'nonexistent*' ],+ qr/\Qpg_amcheck: error: no matching schemas were found for pattern\E/,+ 'no matching schemas');++command_fails_like(+ [ 'pg_amcheck', '-p', "$port", '--strict-names', '-t', 'nonexistent*' ],+ qr/\Qpg_amcheck: error: no matching tables were found for pattern\E/,+ 'no matching tables');++command_fails_like(+ [ 'pg_amcheck', '-p', "$port", '--strict-names', '-i', 'nonexistent*' ],+ qr/\Qpg_amcheck: error: no matching indexes were found for pattern\E/,+ 'no matching indexes');diff --git a/contrib/pg_amcheck/t/003_check.pl b/contrib/pg_amcheck/t/003_check.plnew file mode 100644index 0000000000..4d8e61d871--- /dev/null+++ b/contrib/pg_amcheck/t/003_check.pl@@ -0,0 +1,231 @@+use strict;+use warnings;++use PostgresNode;+use TestLib;+use Test::More tests => 39;++my ($node, $port);++# Returns the filesystem path for the named relation.+#+# Assumes the test node is running+sub relation_filepath($)+{+ my ($relname) = @_;++ my $pgdata = $node->data_dir;+ my $rel = $node->safe_psql('postgres',+ qq(SELECT pg_relation_filepath('$relname')));+ die "path not found for relation $relname" unless defined $rel;+ return "$pgdata/$rel";+}++# Stops the test node, corrupts the first page of the named relation, and+# restarts the node.+#+# Assumes the node is running.+sub corrupt_first_page($)+{+ my ($relname) = @_;+ my $relpath = relation_filepath($relname);++ $node->stop;+ my $fh;+ open($fh, '+<', $relpath);+ binmode $fh;+ seek($fh, 32, 0);+ syswrite($fh, '\x77\x77\x77\x77', 500);+ close($fh);+ $node->start;+}++# Stops the test node, unlinks the file from the filesystem that backs the+# relation, and restarts the node.+#+# Assumes the test node is running+sub remove_relation_file($)+{+ my ($relname) = @_;+ my $relpath = relation_filepath($relname);++ $node->stop();+ unlink($relpath);+ $node->start;+}++# Test set-up+$node = get_new_node('test');+$node->init;+$node->start;+$port = $node->port;++# Load the amcheck extension, upon which pg_amcheck depends+$node->safe_psql('postgres', q(CREATE EXTENSION amcheck));++# Create schemas and tables for checking pg_amcheck's include+# and exclude schema and table command line options+$node->safe_psql('postgres', q(+-- We'll corrupt all indexes in s1+CREATE SCHEMA s1;+CREATE TABLE s1.t1 (a TEXT);+CREATE TABLE s1.t2 (a TEXT);+CREATE INDEX i1 ON s1.t1(a);+CREATE INDEX i2 ON s1.t2(a);+INSERT INTO s1.t1 (a) (SELECT gs::TEXT FROM generate_series(1,10000) AS gs);+INSERT INTO s1.t2 (a) (SELECT gs::TEXT FROM generate_series(1,10000) AS gs);++-- We'll corrupt all tables in s2+CREATE SCHEMA s2;+CREATE TABLE s2.t1 (a TEXT);+CREATE TABLE s2.t2 (a TEXT);+CREATE INDEX i1 ON s2.t1(a);+CREATE INDEX i2 ON s2.t2(a);+INSERT INTO s2.t1 (a) (SELECT gs::TEXT FROM generate_series(1,10000) AS gs);+INSERT INTO s2.t2 (a) (SELECT gs::TEXT FROM generate_series(1,10000) AS gs);++-- We'll corrupt all tables and indexes in s3+CREATE SCHEMA s3;+CREATE TABLE s3.t1 (a TEXT);+CREATE TABLE s3.t2 (a TEXT);+CREATE INDEX i1 ON s3.t1(a);+CREATE INDEX i2 ON s3.t2(a);+INSERT INTO s3.t1 (a) (SELECT gs::TEXT FROM generate_series(1,10000) AS gs);+INSERT INTO s3.t2 (a) (SELECT gs::TEXT FROM generate_series(1,10000) AS gs);++-- We'll leave everything in s4 uncorrupted+CREATE SCHEMA s4;+CREATE TABLE s4.t1 (a TEXT);+CREATE TABLE s4.t2 (a TEXT);+CREATE INDEX i1 ON s4.t1(a);+CREATE INDEX i2 ON s4.t2(a);+INSERT INTO s4.t1 (a) (SELECT gs::TEXT FROM generate_series(1,10000) AS gs);+INSERT INTO s4.t2 (a) (SELECT gs::TEXT FROM generate_series(1,10000) AS gs);+));++# Corrupt indexes in schema "s1"+remove_relation_file('s1.i1');+corrupt_first_page('s1.i2');++# Corrupt tables in schema "s2"+remove_relation_file('s2.t1');+corrupt_first_page('s2.t2');++# Corrupt tables and indexes in schema "s3"+remove_relation_file('s3.i1');+corrupt_first_page('s3.i2');+remove_relation_file('s3.t1');+corrupt_first_page('s3.t2');++# Leave schema "s4" alone+++# The pg_amcheck command itself should return a success exit status, even+# though tables and indexes are corrupt. An error code returned would mean the+# pg_amcheck command itself failed, for example because a connection to the+# database could not be established.+#+# For these checks, we're ignoring any corruption reported and focusing+# exclusively on the exit code from pg_amcheck.+#+$node->command_ok(+ [ 'pg_amcheck', '-x', '-p', $port, 'postgres' ],+ 'pg_amcheck all schemas and tables');++$node->command_ok(+ [ 'pg_amcheck', '-p', $port, 'postgres' ],+ 'pg_amcheck all schemas, tables and indexes');++$node->command_ok(+ [ 'pg_amcheck', '-p', $port, 'postgres', '-n', 's1' ],+ 'pg_amcheck all objects in schema s1');++$node->command_ok(+ [ 'pg_amcheck', '-p', $port, 'postgres', '-n', 's*', '-t', 't1' ],+ 'pg_amcheck all tables named t1 and their indexes');++$node->command_ok(+ [ 'pg_amcheck', '-x', '-p', $port, 'postgres', '-T', 't1' ],+ 'pg_amcheck all tables not named t1');++$node->command_ok(+ [ 'pg_amcheck', '-x', '-p', $port, 'postgres', '-N', 's1', '-T', 't1' ],+ 'pg_amcheck all tables not named t1 nor in schema s1');++# Scans of indexes in s1 should detect the specific corruption that we created+# above. For missing relation forks, we know what the error message looks+# like. For corrupted index pages, the error might vary depending on how the+# page was formatted on disk, including variations due to alignment differences+# between platforms, so we accept any non-empty error message.+#+$node->command_like(+ [ 'pg_amcheck', '-x', '-p', $port, 'postgres', '-n', 's1', '-i', 'i1' ],+ qr/index "i1" lacks a main relation fork/,+ 'pg_amcheck index s1.i1 reports missing main relation fork');++$node->command_like(+ [ 'pg_amcheck', '-x', '-p', $port, 'postgres', '-n', 's1', '-i', 'i2' ],+ qr/.+/, # Any non-empty error message is acceptable+ 'pg_amcheck index s1.s2 reports index corruption');+++# In schema s3, the tables and indexes are both corrupt. Ordinarily, checking+# of indexes will not be performed for corrupt tables, but the --check-corrupt+# option (-c) forces the indexes to also be checked.+#+$node->command_like(+ [ 'pg_amcheck', '-x', '-c', '-p', $port, 'postgres', '-n', 's3', '-i', 'i1' ],+ qr/index "i1" lacks a main relation fork/,+ 'pg_amcheck index s3.i1 reports missing main relation fork');++$node->command_like(+ [ 'pg_amcheck', '-x', '-c', '-p', $port, 'postgres', '-n', 's3', '-i', 'i2' ],+ qr/.+/, # Any non-empty error message is acceptable+ 'pg_amcheck index s3.s2 reports index corruption');+++# Check that '-x' and '-X' work as expected. Since only index corruption+# (and not table corruption) exists in s1, '-X' should give no errors, and+# '-x' should give errors about index corruption.+#+$node->command_like(+ [ 'pg_amcheck', '-x', '-p', $port, 'postgres', '-n', 's1' ],+ qr/.+/, # Any non-empty error message is acceptable+ 'pg_amcheck over tables and indexes in schema s1 reports corruption');++$node->command_like(+ [ 'pg_amcheck', '-X', '-p', $port, 'postgres', '-n', 's1' ],+ qr/^$/, # Empty+ 'pg_amcheck over only tables in schema s1 reports no corruption');+++# Check that table corruption is reported as expected, with or without+# index checking+#+$node->command_like(+ [ 'pg_amcheck', '-x', '-p', $port, 'postgres', '-n', 's2' ],+ qr/could not open file/,+ 'pg_amcheck over tables in schema s2 reports table corruption');++$node->command_like(+ [ 'pg_amcheck', '-p', $port, 'postgres', '-n', 's2' ],+ qr/could not open file/,+ 'pg_amcheck over tables and indexes in schema s2 reports table corruption');++# Check that no corruption is reported in schema s4+$node->command_like(+ [ 'pg_amcheck', '-p', $port, 'postgres', '-n', 's4' ],+ qr/^$/, # Empty+ 'pg_amcheck over schema s4 reports no corruption');++# Check that no corruption is reported if we exclude corrupt schemas+$node->command_like(+ [ 'pg_amcheck', '-p', $port, 'postgres', '-N', 's1', '-N', 's2', '-N', 's3' ],+ qr/^$/, # Empty+ 'pg_amcheck excluding corrupt schemas reports no corruption');++# Check that no corruption is reported if we exclude corrupt tables+$node->command_like(+ [ 'pg_amcheck', '-p', $port, 'postgres', '-T', 't1', '-T', 't2' ],+ qr/^$/, # Empty+ 'pg_amcheck excluding corrupt tables reports no corruption');diff --git a/contrib/pg_amcheck/t/004_verify_heapam.pl b/contrib/pg_amcheck/t/004_verify_heapam.plnew file mode 100644index 0000000000..1cc36b25b7--- /dev/null+++ b/contrib/pg_amcheck/t/004_verify_heapam.pl@@ -0,0 +1,489 @@+use strict;+use warnings;++use PostgresNode;+use TestLib;++use Test::More tests => 22;++# This regression test demonstrates that the pg_amcheck binary supplied with+# the pg_amcheck contrib module correctly identifies specific kinds of+# corruption within pages. To test this, we need a mechanism to create corrupt+# pages with predictable, repeatable corruption. The postgres backend cannot+# be expected to help us with this, as its design is not consistent with the+# goal of intentionally corrupting pages.+#+# Instead, we create a table to corrupt, and with careful consideration of how+# postgresql lays out heap pages, we seek to offsets within the page and+# overwrite deliberately chosen bytes with specific values calculated to+# corrupt the page in expected ways. We then verify that pg_amcheck reports+# the corruption, and that it runs without crashing. Note that the backend+# cannot simply be started to run queries against the corrupt table, as the+# backend will crash, at least for some of the corruption types we generate.+#+# Autovacuum potentially touching the table in the background makes the exact+# behavior of this test harder to reason about. We turn it off to keep things+# simpler. We use a "belt and suspenders" approach, turning it off for the+# system generally in postgresql.conf, and turning it off specifically for the+# test table.+#+# This test depends on the table being written to the heap file exactly as we+# expect it to be, so we take care to arrange the columns of the table, and+# insert rows of the table, that give predictable sizes and locations within+# the table page.+#+# The HeapTupleHeaderData has 23 bytes of fixed size fields before the variable+# length t_bits[] array. We have exactly 3 columns in the table, so natts = 3,+# t_bits is 1 byte long, and t_hoff = MAXALIGN(23 + 1) = 24.+#+# We're not too fussy about which datatypes we use for the test, but we do care+# about some specific properties. We'd like to test both fixed size and+# varlena types. We'd like some varlena data inline and some toasted. And+# we'd like the layout of the table such that the datums land at predictable+# offsets within the tuple. We choose a structure without padding on all+# supported architectures:+#+# a BIGINT+# b TEXT+# c TEXT+#+# We always insert a 7-ascii character string into field 'b', which with a+# 1-byte varlena header gives an 8 byte inline value. We always insert a long+# text string in field 'c', long enough to force toast storage.+#+# We choose to read and write binary copies of our table's tuples, using perl's+# pack() and unpack() functions. Perl uses a packing code system in which:+#+# L = "Unsigned 32-bit Long",+# S = "Unsigned 16-bit Short",+# C = "Unsigned 8-bit Octet",+# c = "signed 8-bit octet",+# q = "signed 64-bit quadword"+#+# Each tuple in our table has a layout as follows:+#+# xx xx xx xx t_xmin: xxxx offset = 0 L+# xx xx xx xx t_xmax: xxxx offset = 4 L+# xx xx xx xx t_field3: xxxx offset = 8 L+# xx xx bi_hi: xx offset = 12 S+# xx xx bi_lo: xx offset = 14 S+# xx xx ip_posid: xx offset = 16 S+# xx xx t_infomask2: xx offset = 18 S+# xx xx t_infomask: xx offset = 20 S+# xx t_hoff: x offset = 22 C+# xx t_bits: x offset = 23 C+# xx xx xx xx xx xx xx xx 'a': xxxxxxxx offset = 24 q+# xx xx xx xx xx xx xx xx 'b': xxxxxxxx offset = 32 Cccccccc+# xx xx xx xx xx xx xx xx 'c': xxxxxxxx offset = 40 SSSS+# xx xx xx xx xx xx xx xx : xxxxxxxx ...continued SSSS+# xx xx : xx ...continued S+#+# We could choose to read and write columns 'b' and 'c' in other ways, but+# it is convenient enough to do it this way. We define packing code+# constants here, where they can be compared easily against the layout.++use constant HEAPTUPLE_PACK_CODE => 'LLLSSSSSCCqCcccccccSSSSSSSSS';+use constant HEAPTUPLE_PACK_LENGTH => 58; # Total size++# Read a tuple of our table from a heap page.+#+# Takes an open filehandle to the heap file, and the offset of the tuple.+#+# Rather than returning the binary data from the file, unpacks the data into a+# perl hash with named fields. These fields exactly match the ones understood+# by write_tuple(), below. Returns a reference to this hash.+#+sub read_tuple ($$)+{+ my ($fh, $offset) = @_;+ my ($buffer, %tup);+ seek($fh, $offset, 0);+ sysread($fh, $buffer, HEAPTUPLE_PACK_LENGTH);++ @_ = unpack(HEAPTUPLE_PACK_CODE, $buffer);+ %tup = (t_xmin => shift,+ t_xmax => shift,+ t_field3 => shift,+ bi_hi => shift,+ bi_lo => shift,+ ip_posid => shift,+ t_infomask2 => shift,+ t_infomask => shift,+ t_hoff => shift,+ t_bits => shift,+ a => shift,+ b_header => shift,+ b_body1 => shift,+ b_body2 => shift,+ b_body3 => shift,+ b_body4 => shift,+ b_body5 => shift,+ b_body6 => shift,+ b_body7 => shift,+ c1 => shift,+ c2 => shift,+ c3 => shift,+ c4 => shift,+ c5 => shift,+ c6 => shift,+ c7 => shift,+ c8 => shift,+ c9 => shift);+ # Stitch together the text for column 'b'+ $tup{b} = join('', map { chr($tup{"b_body$_"}) } (1..7));+ return \%tup;+}++# Write a tuple of our table to a heap page.+#+# Takes an open filehandle to the heap file, the offset of the tuple, and a+# reference to a hash with the tuple values, as returned by read_tuple().+# Writes the tuple fields from the hash into the heap file.+#+# The purpose of this function is to write a tuple back to disk with some+# subset of fields modified. The function does no error checking. Use+# cautiously.+#+sub write_tuple($$$)+{+ my ($fh, $offset, $tup) = @_;+ my $buffer = pack(HEAPTUPLE_PACK_CODE,+ $tup->{t_xmin},+ $tup->{t_xmax},+ $tup->{t_field3},+ $tup->{bi_hi},+ $tup->{bi_lo},+ $tup->{ip_posid},+ $tup->{t_infomask2},+ $tup->{t_infomask},+ $tup->{t_hoff},+ $tup->{t_bits},+ $tup->{a},+ $tup->{b_header},+ $tup->{b_body1},+ $tup->{b_body2},+ $tup->{b_body3},+ $tup->{b_body4},+ $tup->{b_body5},+ $tup->{b_body6},+ $tup->{b_body7},+ $tup->{c1},+ $tup->{c2},+ $tup->{c3},+ $tup->{c4},+ $tup->{c5},+ $tup->{c6},+ $tup->{c7},+ $tup->{c8},+ $tup->{c9});+ seek($fh, $offset, 0);+ syswrite($fh, $buffer, HEAPTUPLE_PACK_LENGTH);+ return;+}++# Set umask so test directories and files are created with default permissions+umask(0077);++# Set up the node. Once we create and corrupt the table,+# autovacuum workers visiting the table could crash the backend.+# Disable autovacuum so that won't happen.+my $node = get_new_node('test');+$node->init;+$node->append_conf('postgresql.conf', 'autovacuum=off');++# Start the node and load the extensions. We depend on both+# amcheck and pageinspect for this test.+$node->start;+my $port = $node->port;+my $pgdata = $node->data_dir;+$node->safe_psql('postgres', "CREATE EXTENSION amcheck");+$node->safe_psql('postgres', "CREATE EXTENSION pageinspect");++# Get a non-zero datfrozenxid+$node->safe_psql('postgres', qq(VACUUM FREEZE));++# Create the test table with precisely the schema that our corruption function+# expects.+$node->safe_psql(+ 'postgres', qq(+ CREATE TABLE public.test (a BIGINT, b TEXT, c TEXT);+ ALTER TABLE public.test SET (autovacuum_enabled=false);+ ALTER TABLE public.test ALTER COLUMN c SET STORAGE EXTERNAL;+ CREATE INDEX test_idx ON public.test(a, b);+ ));++# We want (0 < datfrozenxid < test.relfrozenxid). To achieve this, we freeze+# an otherwise unused table, public.junk, prior to inserting data and freezing+# public.test+$node->safe_psql(+ 'postgres', qq(+ CREATE TABLE public.junk AS SELECT 'junk'::TEXT AS junk_column;+ ALTER TABLE public.junk SET (autovacuum_enabled=false);+ VACUUM FREEZE public.junk+ ));++my $rel = $node->safe_psql('postgres', qq(SELECT pg_relation_filepath('public.test')));+my $relpath = "$pgdata/$rel";++# Insert data and freeze public.test+use constant ROWCOUNT => 16;+$node->safe_psql('postgres', qq(+ INSERT INTO public.test (a, b, c)+ VALUES (+ 12345678,+ 'abcdefg',+ repeat('w', 10000)+ );+ VACUUM FREEZE public.test+ )) for (1..ROWCOUNT);++my $relfrozenxid = $node->safe_psql('postgres',+ q(select relfrozenxid from pg_class where relname = 'test'));+my $datfrozenxid = $node->safe_psql('postgres',+ q(select datfrozenxid from pg_database where datname = 'postgres'));++# Find where each of the tuples is located on the page.+my @lp_off;+for my $tup (0..ROWCOUNT-1)+{+ push (@lp_off, $node->safe_psql('postgres', qq(+select lp_off from heap_page_items(get_raw_page('test', 'main', 0))+ offset $tup limit 1)));+}++# Check that pg_amcheck runs against the uncorrupted table without error.+$node->command_ok(['pg_amcheck', '-p', $port, 'postgres'],+ 'pg_amcheck test table, prior to corruption');++# Check that pg_amcheck runs against the uncorrupted table and index without error.+$node->command_ok(['pg_amcheck', '-p', $port, 'postgres'],+ 'pg_amcheck test table and index, prior to corruption');++$node->stop;++# Sanity check that our 'test' table has a relfrozenxid newer than the+# datfrozenxid for the database, and that the datfrozenxid is greater than the+# first normal xid. We rely on these invariants in some of our tests.+if ($datfrozenxid <= 3 || $datfrozenxid >= $relfrozenxid)+{+ fail('Xid thresholds not as expected');+ $node->clean_node;+ exit;+}++# Some #define constants from access/htup_details.h for use while corrupting.+use constant HEAP_HASNULL => 0x0001;+use constant HEAP_XMAX_LOCK_ONLY => 0x0080;+use constant HEAP_XMIN_COMMITTED => 0x0100;+use constant HEAP_XMIN_INVALID => 0x0200;+use constant HEAP_XMAX_COMMITTED => 0x0400;+use constant HEAP_XMAX_INVALID => 0x0800;+use constant HEAP_NATTS_MASK => 0x07FF;+use constant HEAP_XMAX_IS_MULTI => 0x1000;+use constant HEAP_KEYS_UPDATED => 0x2000;++# Helper functions+sub header+{+ my ($blkno, $offnum, $attnum) = @_;+ qr/\(relname=test,blkno=$blkno,offnum=$offnum,attnum=$attnum\)\s+/ms;+}++# Corrupt the tuples, one type of corruption per tuple. Some types of+# corruption cause verify_heapam to skip to the next tuple without+# performing any remaining checks, so we can't exercise the system properly if+# we focus all our corruption on a single tuple.+#+my @expected;+my $file;+open($file, '+<', $relpath);+binmode $file;++for (my $tupidx = 0; $tupidx < ROWCOUNT; $tupidx++)+{+ my $offnum = $tupidx + 1; # offnum is 1-based, not zero-based+ my $offset = $lp_off[$tupidx];+ my $tup = read_tuple($file, $offset);++ # Sanity-check that the data appears on the page where we expect.+ if ($tup->{a} ne '12345678' || $tup->{b} ne 'abcdefg')+ {+ fail('Page layout differs from our expectations');+ $node->clean_node;+ exit;+ }++ my $header = header(0, $offnum, '');+ if ($offnum == 1)+ {+ # Corruptly set xmin < relfrozenxid+ my $xmin = $relfrozenxid - 1;+ $tup->{t_xmin} = $xmin;+ $tup->{t_infomask} &= ~HEAP_XMIN_COMMITTED;+ $tup->{t_infomask} &= ~HEAP_XMIN_INVALID;++ # Expected corruption report+ push @expected,+ qr/${header}xmin $xmin precedes relation freeze threshold 0:\d+/;+ }+ if ($offnum == 2)+ {+ # Corruptly set xmin < datfrozenxid+ my $xmin = 3;+ $tup->{t_xmin} = $xmin;+ $tup->{t_infomask} &= ~HEAP_XMIN_COMMITTED;+ $tup->{t_infomask} &= ~HEAP_XMIN_INVALID;++ push @expected,+ qr/${$header}xmin $xmin precedes oldest valid transaction ID 0:\d+/;+ }+ elsif ($offnum == 3)+ {+ # Corruptly set xmin < datfrozenxid, further back, noting circularity+ # of xid comparison. For a new cluster with epoch = 0, the corrupt+ # xmin will be interpreted as in the future+ $tup->{t_xmin} = 4026531839;+ $tup->{t_infomask} &= ~HEAP_XMIN_COMMITTED;+ $tup->{t_infomask} &= ~HEAP_XMIN_INVALID;++ push @expected,+ qr/${$header}xmin 4026531839 equals or exceeds next valid transaction ID 0:\d+/;+ }+ elsif ($offnum == 4)+ {+ # Corruptly set xmax < relminmxid;+ $tup->{t_xmax} = 4026531839;+ $tup->{t_infomask} &= ~HEAP_XMAX_INVALID;++ push @expected,+ qr/${$header}xmax 4026531839 equals or exceeds next valid transaction ID 0:\d+/;+ }+ elsif ($offnum == 5)+ {+ # Corrupt the tuple t_hoff, but keep it aligned properly+ $tup->{t_hoff} += 128;++ push @expected,+ qr/${$header}data begins at offset 152 beyond the tuple length 58/,+ qr/${$header}tuple data should begin at byte 24, but actually begins at byte 152 \(3 attributes, no nulls\)/;+ }+ elsif ($offnum == 6)+ {+ # Corrupt the tuple t_hoff, wrong alignment+ $tup->{t_hoff} += 3;++ push @expected,+ qr/${$header}tuple data should begin at byte 24, but actually begins at byte 27 \(3 attributes, no nulls\)/;+ }+ elsif ($offnum == 7)+ {+ # Corrupt the tuple t_hoff, underflow but correct alignment+ $tup->{t_hoff} -= 8;++ push @expected,+ qr/${$header}tuple data should begin at byte 24, but actually begins at byte 16 \(3 attributes, no nulls\)/;+ }+ elsif ($offnum == 8)+ {+ # Corrupt the tuple t_hoff, underflow and wrong alignment+ $tup->{t_hoff} -= 3;++ push @expected,+ qr/${$header}tuple data should begin at byte 24, but actually begins at byte 21 \(3 attributes, no nulls\)/;+ }+ elsif ($offnum == 9)+ {+ # Corrupt the tuple to look like it has lots of attributes, not just 3+ $tup->{t_infomask2} |= HEAP_NATTS_MASK;++ push @expected,+ qr/${$header}number of attributes 2047 exceeds maximum expected for table 3/;+ }+ elsif ($offnum == 10)+ {+ # Corrupt the tuple to look like it has lots of attributes, some of+ # them null. This falsely creates the impression that the t_bits+ # array is longer than just one byte, but t_hoff still says otherwise.+ $tup->{t_infomask} |= HEAP_HASNULL;+ $tup->{t_infomask2} |= HEAP_NATTS_MASK;+ $tup->{t_bits} = 0xAA;++ push @expected,+ qr/${$header}tuple data should begin at byte 280, but actually begins at byte 24 \(2047 attributes, has nulls\)/;+ }+ elsif ($offnum == 11)+ {+ # Same as above, but this time t_hoff plays along+ $tup->{t_infomask} |= HEAP_HASNULL;+ $tup->{t_infomask2} |= (HEAP_NATTS_MASK & 0x40);+ $tup->{t_bits} = 0xAA;+ $tup->{t_hoff} = 32;++ push @expected,+ qr/${$header}number of attributes 67 exceeds maximum expected for table 3/;+ }+ elsif ($offnum == 12)+ {+ # Corrupt the bits in column 'b' 1-byte varlena header+ $tup->{b_header} = 0x80;++ $header = header(0, $offnum, 1);+ push @expected,+ qr/${header}attribute 1 with length 4294967295 ends at offset 416848000 beyond total tuple length 58/;+ }+ elsif ($offnum == 13)+ {+ # Corrupt the bits in column 'c' toast pointer+ $tup->{c6} = 41;+ $tup->{c7} = 41;++ $header = header(0, $offnum, 2);+ push @expected,+ qr/${header}final toast chunk number 0 differs from expected value 6/,+ qr/${header}toasted value for attribute 2 missing from toast table/;+ }+ elsif ($offnum == 14)+ {+ # Set both HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED+ $tup->{t_infomask} |= HEAP_XMAX_LOCK_ONLY;+ $tup->{t_infomask2} |= HEAP_KEYS_UPDATED;++ push @expected,+ qr/${header}tuple is marked as only locked, but also claims key columns were updated/;+ }+ elsif ($offnum == 15)+ {+ # Set both HEAP_XMAX_COMMITTED and HEAP_XMAX_IS_MULTI+ $tup->{t_infomask} |= HEAP_XMAX_COMMITTED;+ $tup->{t_infomask} |= HEAP_XMAX_IS_MULTI;+ $tup->{t_xmax} = 4;++ push @expected,+ qr/${header}multitransaction ID 4 equals or exceeds next valid multitransaction ID 1/;+ }+ elsif ($offnum == 16) # Last offnum must equal ROWCOUNT+ {+ # Set both HEAP_XMAX_COMMITTED and HEAP_XMAX_IS_MULTI+ $tup->{t_infomask} |= HEAP_XMAX_COMMITTED;+ $tup->{t_infomask} |= HEAP_XMAX_IS_MULTI;+ $tup->{t_xmax} = 4000000000;++ push @expected,+ qr/${header}multitransaction ID 4000000000 precedes relation minimum multitransaction ID threshold 1/;+ }+ write_tuple($file, $offset, $tup);+}+close($file);+$node->start;++# Run pg_amcheck against the corrupt table with epoch=0, comparing actual+# corruption messages against the expected messages+$node->command_checks_all(+ ['pg_amcheck', '--check-toast', '--skip-indexes', '-p', $port, 'postgres'],+ 0,+ [ @expected ],+ [ qr/^$/ ],+ 'Expected corruption message output');++$node->teardown_node;+$node->clean_node;diff --git a/contrib/pg_amcheck/t/005_opclass_damage.pl b/contrib/pg_amcheck/t/005_opclass_damage.plnew file mode 100644index 0000000000..fdbb1ea402--- /dev/null+++ b/contrib/pg_amcheck/t/005_opclass_damage.pl@@ -0,0 +1,52 @@+# This regression test checks the behavior of the btree validation in the+# presence of breaking sort order changes.+#+use strict;+use warnings;+use PostgresNode;+use TestLib;+use Test::More tests => 6;++my $node = get_new_node('test');+$node->init;+$node->start;++# Create a custom operator class and an index which uses it.+$node->safe_psql('postgres', q(+ CREATE EXTENSION amcheck;++ CREATE FUNCTION int4_asc_cmp (a int4, b int4) RETURNS int LANGUAGE sql AS $$+ SELECT CASE WHEN $1 = $2 THEN 0 WHEN $1 > $2 THEN 1 ELSE -1 END; $$;++ CREATE OPERATOR CLASS int4_fickle_ops FOR TYPE int4 USING btree AS+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),+ OPERATOR 5 > (int4, int4), FUNCTION 1 int4_asc_cmp(int4, int4);++ CREATE TABLE int4tbl (i int4);+ INSERT INTO int4tbl (SELECT * FROM generate_series(1,1000) gs);+ CREATE INDEX fickleidx ON int4tbl USING btree (i int4_fickle_ops);+));++# We have not yet broken the index, so we should get no corruption+$node->command_like(+ [ 'pg_amcheck', '-p', $node->port, 'postgres' ],+ qr/^$/,+ 'pg_amcheck all schemas, tables and indexes reports no corruption');++# Change the operator class to use a function which sorts in a different+# order to corrupt the btree index+$node->safe_psql('postgres', q(+ CREATE FUNCTION int4_desc_cmp (int4, int4) RETURNS int LANGUAGE sql AS $$+ SELECT CASE WHEN $1 = $2 THEN 0 WHEN $1 > $2 THEN -1 ELSE 1 END; $$;+ UPDATE pg_catalog.pg_amproc+ SET amproc = 'int4_desc_cmp'::regproc+ WHERE amproc = 'int4_asc_cmp'::regproc+));++# Index corruption should now be reported+$node->command_like(+ [ 'pg_amcheck', '-x', '-p', $node->port, 'postgres' ],+ qr/item order invariant violated for index "fickleidx"/,+ 'pg_amcheck all schemas, tables and indexes reports fickleidx corruption'+);diff --git a/doc/src/sgml/contrib.sgml b/doc/src/sgml/contrib.sgmlindex 4e833d79ef..1efca8adc4 100644--- a/doc/src/sgml/contrib.sgml+++ b/doc/src/sgml/contrib.sgml@@ -119,6 +119,7 @@ CREATE EXTENSION <replaceable>module_name</replaceable>;
&oldsnapshot;
&pageinspect;
&passwordcheck;
+ &pgamcheck;
&pgbuffercache;
&pgcrypto;
&pgfreespacemap;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgmlindex 38e8aa0bbf..a4e1b28b38 100644--- a/doc/src/sgml/filelist.sgml+++ b/doc/src/sgml/filelist.sgml@@ -133,6 +133,7 @@
<!ENTITY oldsnapshot SYSTEM "oldsnapshot.sgml">
<!ENTITY pageinspect SYSTEM "pageinspect.sgml">
<!ENTITY passwordcheck SYSTEM "passwordcheck.sgml">
+<!ENTITY pgamcheck SYSTEM "pgamcheck.sgml">
<!ENTITY pgbuffercache SYSTEM "pgbuffercache.sgml">
<!ENTITY pgcrypto SYSTEM "pgcrypto.sgml">
<!ENTITY pgfreespacemap SYSTEM "pgfreespacemap.sgml">
diff --git a/doc/src/sgml/pgamcheck.sgml b/doc/src/sgml/pgamcheck.sgmlnew file mode 100644index 0000000000..fc36447dda--- /dev/null+++ b/doc/src/sgml/pgamcheck.sgml@@ -0,0 +1,228 @@+<!-- doc/src/sgml/pgamcheck.sgml -->++<sect1 id="pgamcheck" xreflabel="pg_amcheck">+ <title>pg_amcheck</title>++ <indexterm zone="pgamcheck">+ <primary>pg_amcheck</primary>+ </indexterm>++ <para>+ The <filename>pg_amcheck</filename> module provides a command line interface+ to the <xref linkend="amcheck"/> corruption checking functionality.+ </para>++ <para>+ <application>pg_amcheck</application> is a regular+ <productname>PostgreSQL</productname> client application. You can perform+ corruption checks from any remote host that has access to the database+ connecting as a user with sufficient privileges to check tables and indexes.+ Currently, this requires execute privileges on <xref linkend="amcheck"/>'s+ <function>bt_index_parent_check</function> and <function>verify_heapam</function>+ functions, and on having privileges to access the relations being checked.+ </para>++<synopsis>+pg_amcheck [OPTION]... [DBNAME [USERNAME]]+ General options:+ -V, --version output version information, then exit+ -?, --help show this help, then exit+ -s, --strict-names require include patterns to match at least one entity each+ -o, --on-error-stop stop checking at end of first corrupt page++ Schema checking options:+ -n, --schema=PATTERN check relations in the specified schema(s) only+ -N, --exclude-schema=PATTERN do NOT check relations in the specified schema(s)++ Table checking options:+ -t, --table=PATTERN check the specified table(s) only+ -T, --exclude-table=PATTERN do NOT check the specified table(s)+ -b, --startblock begin checking table(s) at the given starting block number+ -e, --endblock check table(s) only up to the given ending block number+ -f, --skip-all-frozen do NOT check blocks marked as all-frozen+ -v, --skip-all-visible do NOT check blocks marked as all-visible++ TOAST table checking options:+ -z, --check-toast check associated toast tables and toast indexes+ -Z, --skip-toast do NOT check associated toast tables and toast indexes+ -B, --toast-startblock begin checking toast table(s) at the given starting block+ -E, --toast-endblock check toast table(s) only up to the given ending block++ Index checking options:+ -x, --check-indexes check btree indexes associated with tables being checked+ -X, --skip-indexes do NOT check any btree indexes+ -i, --index=PATTERN check the specified index(es) only+ -I, --exclude-index=PATTERN do NOT check the specified index(es)+ -c, --check-corrupt check indexes even if their associated table is corrupt+ -C, --skip-corrupt do NOT check indexes if their associated table is corrupt+ -a, --heapallindexed check index tuples against the table tuples+ -A, --no-heapallindexed do NOT check index tuples against the table tuples+ -r, --rootdescend search from the root page for each index tuple+ -R, --no-rootdescend do NOT search from the root page for each index tuple++ Connection options:+ -d, --dbname=DBNAME database name to connect to+ -h, --host=HOSTNAME database server host or socket directory+ -p, --port=PORT database server port+ -U, --username=USERNAME database user name+ -w, --no-password never prompt for password+ -W, --password force password prompt (should happen automatically)+</synopsis>++ <sect2>+ <title>Options</title>++ <para>+ To specify which database server <application>pg_amcheck</application> should+ contact, use the command line options <option>-h</option> or+ <option>--host</option> and <option>-p</option> or+ <option>port</option>. The default host is the local host+ or whatever your <envar>PGHOST</envar> environment variable specifies.+ Similarly, the default port is indicated by the <envar>PGPORT</envar>+ environment variable or, failing that, by the compiled-in default.+ </para>+ + <para>+ Like any other <productname>PostgreSQL</productname> client application,+ <application>pg_amcheck</application> will by default connect with the+ database user name that is equal to the current operating system user name.+ To override this, either specify the <option>-U</option> option or set the+ environment variable <envar>PGUSER</envar>. Remember that+ <application>pg_amcheck</application> connections are subject to the normal+ client authentication mechanisms (which are described in <xref+ linkend="client-authentication"/>).+ </para>+ + <para>+ To restrict checking of tables and indexes to specific schemas, specify the+ <option>-s</option> or <option>--schema</option> option with a pattern.+ To exclude checking of tables and indexes within specific schemas, specify+ the <option>-N</option> or <option>--exclude-schema</option> option with+ a pattern.+ </para>+ + <para>+ To specify which tables are checked, specify the+ <option>-t</option> or <option>--table</option> option with a pattern.+ To exclude checking of tables, specify the+ <option>-T</option> or <option>--exclude-table</option> option with a+ pattern.+ </para>+ + <para>+ To check indexes associated with checked tables, specify the+ <option>-i</option> or <option>--check-indexes</option> option. Only+ indexes on tables which are being checked will themselves be checked. To+ check all indexes in a database, all tables on which the indexes exist must+ also be checked. This restriction may be relaxed in the future.+ </para>+ + <para>+ To restrict the range of blocks within a table that are checked, specify the+ <option>-b</option> or <option>--startblock</option> and/or+ <option>-e</option> or <option>--endblock</option> options with numeric+ values for the starting and ending block numbers. Although these options+ make the most sense when applied to a single table, if specified along with+ options that select multiple tables, each table check will be restricted to+ the specified blocks. If <option>--startblock</option> is omitted, checking+ begins with the first block. If <option>--endblock</option> is omitted,+ checking continues to the end of the relation.+ </para>+ + <para>+ Some users may wish to periodically check tables without incurring the cost+ of rechecking older table blocks, presumably because those blocks have+ already been checked in the past. There is at present no perfect way to do+ this. Although the <option>--startblock</option> and <option>--endblock</option>+ options can be used to restrict blocks, the user is not expected to have+ perfect knowledge of which blocks have already been checked, and in any+ event, some blocks that were previously checked may have been subject to+ modification since the last check. As an approximation to the desired+ functionality, one can specify the+ <option>-f</option> or <option>--skip-all-frozen</option> option, or+ alternatively the+ <option>-v</option> or <option>--skip-all-visible</option> option to skip+ blocks marked in the visibility map as all-frozen or all-visible,+ respectively.+ </para>+ </sect2>++ <sect2>+ <title>Example Usage</title>++ <para>+ For table corruption, each detected corruption is reported on two lines, the+ first line shows the location and the second line shows a message describing+ the problem.+ </para>++ <para>+ Checking an entire database which contains one corrupt table, "mytable",+ along with the output:+ </para>++<screen>+% pg_amcheck --check-toast --skip-indexes mydb+(relname=mytable,blkno=17,offnum=12,attnum=)+xmin 4294967295 precedes relation freeze threshold 17:1134217582+(relname=mytable,blkno=960,offnum=4,attnum=)+data begins at offset 152 beyond the tuple length 58+(relname=mytable,blkno=960,offnum=4,attnum=)+tuple data should begin at byte 24, but actually begins at byte 152 (3 attributes, no nulls)+(relname=mytable,blkno=960,offnum=5,attnum=)+tuple data should begin at byte 24, but actually begins at byte 27 (3 attributes, no nulls)+(relname=mytable,blkno=960,offnum=6,attnum=)+tuple data should begin at byte 24, but actually begins at byte 16 (3 attributes, no nulls)+(relname=mytable,blkno=960,offnum=7,attnum=)+tuple data should begin at byte 24, but actually begins at byte 21 (3 attributes, no nulls)+(relname=mytable,blkno=1147,offnum=2,attnum=)+number of attributes 2047 exceeds maximum expected for table 3+(relname=mytable,blkno=1147,offnum=10,attnum=)+tuple data should begin at byte 280, but actually begins at byte 24 (2047 attributes, has nulls)+(relname=mytable,blkno=1147,offnum=15,attnum=)+number of attributes 67 exceeds maximum expected for table 3+(relname=mytable,blkno=1147,offnum=16,attnum=1)+attribute 1 with length 4294967295 ends at offset 416848000 beyond total tuple length 58+(relname=mytable,blkno=1147,offnum=18,attnum=2)+final toast chunk number 0 differs from expected value 6+(relname=mytable,blkno=1147,offnum=19,attnum=2)+toasted value for attribute 2 missing from toast table+(relname=mytable,blkno=1147,offnum=21,attnum=)+tuple is marked as only locked, but also claims key columns were updated+(relname=mytable,blkno=1147,offnum=22,attnum=)+multitransaction ID 1775655 is from before relation cutoff 2355572+</screen>++ <para>+ For index corruption, the output is more free-form, and may span differing+ numbers of lines per corruption detected.+ </para>++ <para>+ Checking an entire database which contains one corrupt index,+ "corrupt_index", with corruption in the page header, along with the output:+ </para>++<screen>+% pg_amcheck --check-toast --check-indexes --schema=public --table=table_with_corrupt_index mydb+index check failed for index corrupt_index of table table_with_corrupt_index:+ERROR: XX002: index "corrupt_index" is not a btree+LOCATION: _bt_getmeta, nbtpage.c:152+</screen>++ <para>+ Checking again after rebuilding the index but corrupting the contents,+ along with the output:+ </para>++<screen>+% pg_amcheck --check-toast --check-indexes --schema=public --table=table_with_corrupt_index mydb+index check failed for index corrupt_index of table table_with_corrupt_index:+ERROR: XX002: index tuple size does not equal lp_len in index "corrupt_index"+DETAIL: Index tid=(39,49) tuple size=3373 lp_len=24 page lsn=0/2B548C0.+HINT: This could be a torn page problem.+LOCATION: bt_target_page_check, verify_nbtree.c:1125+</screen>++ </sect2>+</sect1>diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pmindex 89e1b39036..8cf0554823 100644--- a/src/tools/msvc/Mkvcbuild.pm+++ b/src/tools/msvc/Mkvcbuild.pm@@ -33,9 +33,9 @@ my @unlink_on_exit;
# Set of variables for modules in contrib/ and src/test/modules/
my $contrib_defines = { 'refint' => 'REFINT_VERBOSE' };
-my @contrib_uselibpq = ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo');-my @contrib_uselibpgport = ('oid2name', 'pg_standby', 'vacuumlo');-my @contrib_uselibpgcommon = ('oid2name', 'pg_standby', 'vacuumlo');+my @contrib_uselibpq = ('dblink', 'oid2name', 'pg_amcheck', 'postgres_fdw', 'vacuumlo');+my @contrib_uselibpgport = ('oid2name', 'pg_amcheck', 'pg_standby', 'vacuumlo');+my @contrib_uselibpgcommon = ('oid2name', 'pg_amcheck', 'pg_standby', 'vacuumlo');
my $contrib_extralibs = undef;
my $contrib_extraincludes = { 'dblink' => ['src/backend'] };
my $contrib_extrasource = {
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.listindex bca30f3dde..369b8e7c6f 100644--- a/src/tools/pgindent/typedefs.list+++ b/src/tools/pgindent/typedefs.list@@ -102,6 +102,7 @@ AlterUserMappingStmt
AlteredTableInfo
AlternativeSubPlan
AlternativeSubPlanState
+AmCheckSettings
AnalyzeAttrComputeStatsFunc
AnalyzeAttrFetchFunc
AnalyzeForeignTable_function
@@ -403,6 +404,7 @@ ConfigData
ConfigVariable
ConnCacheEntry
ConnCacheKey
+ConnectOptions
ConnStatusType
ConnType
ConnectionStateEnum
--
2.21.1 (Apple Git-122.3)
=
v17-0003-Creating-non-throwing-interface-to-clog-and-slru.patch (application/octet-stream, 21.8K) download
From f34c48af2746b752d11c3d55ce7604785dc44068 Mon Sep 17 00:00:00 2001
From: Mark Dilger <mark.dilger@enterprisedb.com>
Date: Tue, 6 Oct 2020 11:28:18 -0700
Subject: [PATCH v17 3/4] Creating non-throwing interface to clog and slru.
---
src/backend/access/transam/clog.c | 21 +++---
src/backend/access/transam/commit_ts.c | 4 +-
src/backend/access/transam/multixact.c | 16 ++---
src/backend/access/transam/slru.c | 23 +++---
src/backend/access/transam/subtrans.c | 4 +-
src/backend/access/transam/transam.c | 98 +++++++++-----------------
src/backend/commands/async.c | 4 +-
src/backend/storage/lmgr/predicate.c | 4 +-
src/include/access/clog.h | 18 +----
src/include/access/clogdefs.h | 33 +++++++++
src/include/access/slru.h | 6 +-
src/include/access/transam.h | 3 +
12 files changed, 122 insertions(+), 112 deletions(-)
create mode 100644 src/include/access/clogdefs.h
diff --git a/src/backend/access/transam/clog.c b/src/backend/access/transam/clog.cindex 034349aa7b..a2eb3e2983 100644--- a/src/backend/access/transam/clog.c+++ b/src/backend/access/transam/clog.c@@ -357,7 +357,7 @@ TransactionIdSetPageStatusInternal(TransactionId xid, int nsubxids,
* write-busy, since we don't care if the update reaches disk sooner than
* we think.
*/
- slotno = SimpleLruReadPage(XactCtl, pageno, XLogRecPtrIsInvalid(lsn), xid);+ slotno = SimpleLruReadPage(XactCtl, pageno, XLogRecPtrIsInvalid(lsn), xid, true);
/*
* Set the main transaction id, if any.
@@ -631,7 +631,7 @@ TransactionIdSetStatusBit(TransactionId xid, XidStatus status, XLogRecPtr lsn, i
* for most uses; TransactionLogFetch() in transam.c is the intended caller.
*/
XidStatus
-TransactionIdGetStatus(TransactionId xid, XLogRecPtr *lsn)+TransactionIdGetStatus(TransactionId xid, XLogRecPtr *lsn, bool throwError)
{
int pageno = TransactionIdToPage(xid);
int byteno = TransactionIdToByte(xid);
@@ -643,13 +643,18 @@ TransactionIdGetStatus(TransactionId xid, XLogRecPtr *lsn)
/* lock is acquired by SimpleLruReadPage_ReadOnly */
- slotno = SimpleLruReadPage_ReadOnly(XactCtl, pageno, xid);- byteptr = XactCtl->shared->page_buffer[slotno] + byteno;+ slotno = SimpleLruReadPage_ReadOnly(XactCtl, pageno, xid, throwError);+ if (slotno == InvalidSlotNo)+ status = TRANSACTION_STATUS_UNKNOWN;+ else+ {+ byteptr = XactCtl->shared->page_buffer[slotno] + byteno;- status = (*byteptr >> bshift) & CLOG_XACT_BITMASK;+ status = (*byteptr >> bshift) & CLOG_XACT_BITMASK;- lsnindex = GetLSNIndex(slotno, xid);- *lsn = XactCtl->shared->group_lsn[lsnindex];+ lsnindex = GetLSNIndex(slotno, xid);+ *lsn = XactCtl->shared->group_lsn[lsnindex];+ }
LWLockRelease(XactSLRULock);
@@ -796,7 +801,7 @@ TrimCLOG(void)
int slotno;
char *byteptr;
- slotno = SimpleLruReadPage(XactCtl, pageno, false, xid);+ slotno = SimpleLruReadPage(XactCtl, pageno, false, xid, true);
byteptr = XactCtl->shared->page_buffer[slotno] + byteno;
/* Zero so-far-unused positions in the current byte */
diff --git a/src/backend/access/transam/commit_ts.c b/src/backend/access/transam/commit_ts.cindex cb8a968801..98c685405c 100644--- a/src/backend/access/transam/commit_ts.c+++ b/src/backend/access/transam/commit_ts.c@@ -237,7 +237,7 @@ SetXidCommitTsInPage(TransactionId xid, int nsubxids,
LWLockAcquire(CommitTsSLRULock, LW_EXCLUSIVE);
- slotno = SimpleLruReadPage(CommitTsCtl, pageno, true, xid);+ slotno = SimpleLruReadPage(CommitTsCtl, pageno, true, xid, true);
TransactionIdSetCommitTs(xid, ts, nodeid, slotno);
for (i = 0; i < nsubxids; i++)
@@ -342,7 +342,7 @@ TransactionIdGetCommitTsData(TransactionId xid, TimestampTz *ts,
}
/* lock is acquired by SimpleLruReadPage_ReadOnly */
- slotno = SimpleLruReadPage_ReadOnly(CommitTsCtl, pageno, xid);+ slotno = SimpleLruReadPage_ReadOnly(CommitTsCtl, pageno, xid, true);
memcpy(&entry,
CommitTsCtl->shared->page_buffer[slotno] +
SizeOfCommitTimestampEntry * entryno,
diff --git a/src/backend/access/transam/multixact.c b/src/backend/access/transam/multixact.cindex 81752b68eb..5c0213b06d 100644--- a/src/backend/access/transam/multixact.c+++ b/src/backend/access/transam/multixact.c@@ -881,7 +881,7 @@ RecordNewMultiXact(MultiXactId multi, MultiXactOffset offset,
* enough that a MultiXactId is really involved. Perhaps someday we'll
* take the trouble to generalize the slru.c error reporting code.
*/
- slotno = SimpleLruReadPage(MultiXactOffsetCtl, pageno, true, multi);+ slotno = SimpleLruReadPage(MultiXactOffsetCtl, pageno, true, multi, true);
offptr = (MultiXactOffset *) MultiXactOffsetCtl->shared->page_buffer[slotno];
offptr += entryno;
@@ -914,7 +914,7 @@ RecordNewMultiXact(MultiXactId multi, MultiXactOffset offset,
if (pageno != prev_pageno)
{
- slotno = SimpleLruReadPage(MultiXactMemberCtl, pageno, true, multi);+ slotno = SimpleLruReadPage(MultiXactMemberCtl, pageno, true, multi, true);
prev_pageno = pageno;
}
@@ -1345,7 +1345,7 @@ retry:
pageno = MultiXactIdToOffsetPage(multi);
entryno = MultiXactIdToOffsetEntry(multi);
- slotno = SimpleLruReadPage(MultiXactOffsetCtl, pageno, true, multi);+ slotno = SimpleLruReadPage(MultiXactOffsetCtl, pageno, true, multi, true);
offptr = (MultiXactOffset *) MultiXactOffsetCtl->shared->page_buffer[slotno];
offptr += entryno;
offset = *offptr;
@@ -1377,7 +1377,7 @@ retry:
entryno = MultiXactIdToOffsetEntry(tmpMXact);
if (pageno != prev_pageno)
- slotno = SimpleLruReadPage(MultiXactOffsetCtl, pageno, true, tmpMXact);+ slotno = SimpleLruReadPage(MultiXactOffsetCtl, pageno, true, tmpMXact, true);
offptr = (MultiXactOffset *) MultiXactOffsetCtl->shared->page_buffer[slotno];
offptr += entryno;
@@ -1418,7 +1418,7 @@ retry:
if (pageno != prev_pageno)
{
- slotno = SimpleLruReadPage(MultiXactMemberCtl, pageno, true, multi);+ slotno = SimpleLruReadPage(MultiXactMemberCtl, pageno, true, multi, true);
prev_pageno = pageno;
}
@@ -2063,7 +2063,7 @@ TrimMultiXact(void)
int slotno;
MultiXactOffset *offptr;
- slotno = SimpleLruReadPage(MultiXactOffsetCtl, pageno, true, nextMXact);+ slotno = SimpleLruReadPage(MultiXactOffsetCtl, pageno, true, nextMXact, true);
offptr = (MultiXactOffset *) MultiXactOffsetCtl->shared->page_buffer[slotno];
offptr += entryno;
@@ -2095,7 +2095,7 @@ TrimMultiXact(void)
int memberoff;
memberoff = MXOffsetToMemberOffset(offset);
- slotno = SimpleLruReadPage(MultiXactMemberCtl, pageno, true, offset);+ slotno = SimpleLruReadPage(MultiXactMemberCtl, pageno, true, offset, true);
xidptr = (TransactionId *)
(MultiXactMemberCtl->shared->page_buffer[slotno] + memberoff);
@@ -2749,7 +2749,7 @@ find_multixact_start(MultiXactId multi, MultiXactOffset *result)
return false;
/* lock is acquired by SimpleLruReadPage_ReadOnly */
- slotno = SimpleLruReadPage_ReadOnly(MultiXactOffsetCtl, pageno, multi);+ slotno = SimpleLruReadPage_ReadOnly(MultiXactOffsetCtl, pageno, multi, true);
offptr = (MultiXactOffset *) MultiXactOffsetCtl->shared->page_buffer[slotno];
offptr += entryno;
offset = *offptr;
diff --git a/src/backend/access/transam/slru.c b/src/backend/access/transam/slru.cindex 16a7898697..daa145eeff 100644--- a/src/backend/access/transam/slru.c+++ b/src/backend/access/transam/slru.c@@ -385,14 +385,15 @@ SimpleLruWaitIO(SlruCtl ctl, int slotno)
* The passed-in xid is used only for error reporting, and may be
* InvalidTransactionId if no specific xid is associated with the action.
*
- * Return value is the shared-buffer slot number now holding the page.- * The buffer's LRU access info is updated.+ * On error, when throwError is false, the return value is negative.+ * Otherwise, return value is the shared-buffer slot number now holding the+ * page, and the buffer's LRU access info is updated.
*
* Control lock must be held at entry, and will be held at exit.
*/
int
SimpleLruReadPage(SlruCtl ctl, int pageno, bool write_ok,
- TransactionId xid)+ TransactionId xid, bool throwError)
{
SlruShared shared = ctl->shared;
@@ -465,7 +466,11 @@ SimpleLruReadPage(SlruCtl ctl, int pageno, bool write_ok,
/* Now it's okay to ereport if we failed */
if (!ok)
- SlruReportIOError(ctl, pageno, xid);+ {+ if (throwError)+ SlruReportIOError(ctl, pageno, xid);+ return InvalidSlotNo;+ }
SlruRecentlyUsed(shared, slotno);
@@ -484,14 +489,16 @@ SimpleLruReadPage(SlruCtl ctl, int pageno, bool write_ok,
* The passed-in xid is used only for error reporting, and may be
* InvalidTransactionId if no specific xid is associated with the action.
*
- * Return value is the shared-buffer slot number now holding the page.- * The buffer's LRU access info is updated.+ * On error, when throwError is false, the return value is negative.+ * Otherwise, return value is the shared-buffer slot number now holding the+ * page, and the buffer's LRU access info is updated.
*
* Control lock must NOT be held at entry, but will be held at exit.
* It is unspecified whether the lock will be shared or exclusive.
*/
int
-SimpleLruReadPage_ReadOnly(SlruCtl ctl, int pageno, TransactionId xid)+SimpleLruReadPage_ReadOnly(SlruCtl ctl, int pageno, TransactionId xid,+ bool throwError)
{
SlruShared shared = ctl->shared;
int slotno;
@@ -520,7 +527,7 @@ SimpleLruReadPage_ReadOnly(SlruCtl ctl, int pageno, TransactionId xid)
LWLockRelease(shared->ControlLock);
LWLockAcquire(shared->ControlLock, LW_EXCLUSIVE);
- return SimpleLruReadPage(ctl, pageno, true, xid);+ return SimpleLruReadPage(ctl, pageno, true, xid, throwError);
}
/*
diff --git a/src/backend/access/transam/subtrans.c b/src/backend/access/transam/subtrans.cindex 0111e867c7..353b946731 100644--- a/src/backend/access/transam/subtrans.c+++ b/src/backend/access/transam/subtrans.c@@ -83,7 +83,7 @@ SubTransSetParent(TransactionId xid, TransactionId parent)
LWLockAcquire(SubtransSLRULock, LW_EXCLUSIVE);
- slotno = SimpleLruReadPage(SubTransCtl, pageno, true, xid);+ slotno = SimpleLruReadPage(SubTransCtl, pageno, true, xid, true);
ptr = (TransactionId *) SubTransCtl->shared->page_buffer[slotno];
ptr += entryno;
@@ -123,7 +123,7 @@ SubTransGetParent(TransactionId xid)
/* lock is acquired by SimpleLruReadPage_ReadOnly */
- slotno = SimpleLruReadPage_ReadOnly(SubTransCtl, pageno, xid);+ slotno = SimpleLruReadPage_ReadOnly(SubTransCtl, pageno, xid, true);
ptr = (TransactionId *) SubTransCtl->shared->page_buffer[slotno];
ptr += entryno;
diff --git a/src/backend/access/transam/transam.c b/src/backend/access/transam/transam.cindex a28918657c..88f867e5ef 100644--- a/src/backend/access/transam/transam.c+++ b/src/backend/access/transam/transam.c@@ -35,7 +35,8 @@ static XidStatus cachedFetchXidStatus;
static XLogRecPtr cachedCommitLSN;
/* Local functions */
-static XidStatus TransactionLogFetch(TransactionId transactionId);+static XidStatus TransactionLogFetch(TransactionId transactionId,+ bool throwError);
/* ----------------------------------------------------------------
@@ -49,7 +50,7 @@ static XidStatus TransactionLogFetch(TransactionId transactionId);
* TransactionLogFetch --- fetch commit status of specified transaction id
*/
static XidStatus
-TransactionLogFetch(TransactionId transactionId)+TransactionLogFetch(TransactionId transactionId, bool throwError)
{
XidStatus xidstatus;
XLogRecPtr xidlsn;
@@ -76,14 +77,16 @@ TransactionLogFetch(TransactionId transactionId)
/*
* Get the transaction status.
*/
- xidstatus = TransactionIdGetStatus(transactionId, &xidlsn);+ xidstatus = TransactionIdGetStatus(transactionId, &xidlsn, throwError);
/*
* Cache it, but DO NOT cache status for unfinished or sub-committed
* transactions! We only cache status that is guaranteed not to change.
+ * Likewise, DO NOT cache when the status is unknown.
*/
if (xidstatus != TRANSACTION_STATUS_IN_PROGRESS &&
- xidstatus != TRANSACTION_STATUS_SUB_COMMITTED)+ xidstatus != TRANSACTION_STATUS_SUB_COMMITTED &&+ xidstatus != TRANSACTION_STATUS_UNKNOWN)
{
cachedFetchXid = transactionId;
cachedFetchXidStatus = xidstatus;
@@ -96,6 +99,7 @@ TransactionLogFetch(TransactionId transactionId)
/* ----------------------------------------------------------------
* Interface functions
*
+ * TransactionIdResolveStatus
* TransactionIdDidCommit
* TransactionIdDidAbort
* ========
@@ -115,24 +119,17 @@ TransactionLogFetch(TransactionId transactionId)
*/
/*
- * TransactionIdDidCommit- * True iff transaction associated with the identifier did commit.- *- * Note:- * Assumes transaction identifier is valid and exists in clog.+ * TransactionIdResolveStatus+ * Returns the status of the transaction associated with the identifer,+ * recursively resolving sub-committed transaction status by checking+ * the parent transaction.
*/
-bool /* true if given transaction committed */-TransactionIdDidCommit(TransactionId transactionId)+XidStatus+TransactionIdResolveStatus(TransactionId transactionId, bool throwError)
{
XidStatus xidstatus;
- xidstatus = TransactionLogFetch(transactionId);-- /*- * If it's marked committed, it's committed.- */- if (xidstatus == TRANSACTION_STATUS_COMMITTED)- return true;+ xidstatus = TransactionLogFetch(transactionId, throwError);
/*
* If it's marked subcommitted, we have to check the parent recursively.
@@ -153,21 +150,31 @@ TransactionIdDidCommit(TransactionId transactionId)
TransactionId parentXid;
if (TransactionIdPrecedes(transactionId, TransactionXmin))
- return false;+ return TRANSACTION_STATUS_ABORTED;
parentXid = SubTransGetParent(transactionId);
if (!TransactionIdIsValid(parentXid))
{
elog(WARNING, "no pg_subtrans entry for subcommitted XID %u",
transactionId);
- return false;+ return TRANSACTION_STATUS_ABORTED;
}
- return TransactionIdDidCommit(parentXid);+ return TransactionIdResolveStatus(parentXid, throwError);
}
+ return xidstatus;+}- /*- * It's not committed.- */- return false;+/*+ * TransactionIdDidCommit+ * True iff transaction associated with the identifier did commit.+ *+ * Note:+ * Assumes transaction identifier is valid and exists in clog.+ */+bool /* true if given transaction committed */+TransactionIdDidCommit(TransactionId transactionId)+{+ return (TransactionIdResolveStatus(transactionId, true) ==+ TRANSACTION_STATUS_COMMITTED);
}
/*
@@ -180,43 +187,8 @@ TransactionIdDidCommit(TransactionId transactionId)
bool /* true if given transaction aborted */
TransactionIdDidAbort(TransactionId transactionId)
{
- XidStatus xidstatus;-- xidstatus = TransactionLogFetch(transactionId);-- /*- * If it's marked aborted, it's aborted.- */- if (xidstatus == TRANSACTION_STATUS_ABORTED)- return true;-- /*- * If it's marked subcommitted, we have to check the parent recursively.- * However, if it's older than TransactionXmin, we can't look at- * pg_subtrans; instead assume that the parent crashed without cleaning up- * its children.- */- if (xidstatus == TRANSACTION_STATUS_SUB_COMMITTED)- {- TransactionId parentXid;-- if (TransactionIdPrecedes(transactionId, TransactionXmin))- return true;- parentXid = SubTransGetParent(transactionId);- if (!TransactionIdIsValid(parentXid))- {- /* see notes in TransactionIdDidCommit */- elog(WARNING, "no pg_subtrans entry for subcommitted XID %u",- transactionId);- return true;- }- return TransactionIdDidAbort(parentXid);- }-- /*- * It's not aborted.- */- return false;+ return (TransactionIdResolveStatus(transactionId, true) ==+ TRANSACTION_STATUS_ABORTED);
}
/*
@@ -419,7 +391,7 @@ TransactionIdGetCommitLSN(TransactionId xid)
/*
* Get the transaction status.
*/
- (void) TransactionIdGetStatus(xid, &result);+ (void) TransactionIdGetStatus(xid, &result, true);
return result;
}
diff --git a/src/backend/commands/async.c b/src/backend/commands/async.cindex 8dbcace3f9..a49126dba0 100644--- a/src/backend/commands/async.c+++ b/src/backend/commands/async.c@@ -1477,7 +1477,7 @@ asyncQueueAddEntries(ListCell *nextNotify)
slotno = SimpleLruZeroPage(NotifyCtl, pageno);
else
slotno = SimpleLruReadPage(NotifyCtl, pageno, true,
- InvalidTransactionId);+ InvalidTransactionId, true);
/* Note we mark the page dirty before writing in it */
NotifyCtl->shared->page_dirty[slotno] = true;
@@ -2010,7 +2010,7 @@ asyncQueueReadAllNotifications(void)
* part of the page we will actually inspect.
*/
slotno = SimpleLruReadPage_ReadOnly(NotifyCtl, curpage,
- InvalidTransactionId);+ InvalidTransactionId, true);
if (curpage == QUEUE_POS_PAGE(head))
{
/* we only want to read as far as head */
diff --git a/src/backend/storage/lmgr/predicate.c b/src/backend/storage/lmgr/predicate.cindex 8a365b400c..6cf12e46f6 100644--- a/src/backend/storage/lmgr/predicate.c+++ b/src/backend/storage/lmgr/predicate.c@@ -904,7 +904,7 @@ SerialAdd(TransactionId xid, SerCommitSeqNo minConflictCommitSeqNo)
slotno = SimpleLruZeroPage(SerialSlruCtl, targetPage);
}
else
- slotno = SimpleLruReadPage(SerialSlruCtl, targetPage, true, xid);+ slotno = SimpleLruReadPage(SerialSlruCtl, targetPage, true, xid, true);
SerialValue(slotno, xid) = minConflictCommitSeqNo;
SerialSlruCtl->shared->page_dirty[slotno] = true;
@@ -946,7 +946,7 @@ SerialGetMinConflictCommitSeqNo(TransactionId xid)
* but will return with that lock held, which must then be released.
*/
slotno = SimpleLruReadPage_ReadOnly(SerialSlruCtl,
- SerialPage(xid), xid);+ SerialPage(xid), xid, true);
val = SerialValue(slotno, xid);
LWLockRelease(SerialSLRULock);
return val;
diff --git a/src/include/access/clog.h b/src/include/access/clog.hindex 6c840cbf29..cf299cd8f6 100644--- a/src/include/access/clog.h+++ b/src/include/access/clog.h@@ -11,24 +11,11 @@
#ifndef CLOG_H
#define CLOG_H
+#include "access/clogdefs.h"
#include "access/xlogreader.h"
#include "storage/sync.h"
#include "lib/stringinfo.h"
-/*- * Possible transaction statuses --- note that all-zeroes is the initial- * state.- *- * A "subcommitted" transaction is a committed subtransaction whose parent- * hasn't committed or aborted yet.- */-typedef int XidStatus;--#define TRANSACTION_STATUS_IN_PROGRESS 0x00-#define TRANSACTION_STATUS_COMMITTED 0x01-#define TRANSACTION_STATUS_ABORTED 0x02-#define TRANSACTION_STATUS_SUB_COMMITTED 0x03-
typedef struct xl_clog_truncate
{
int pageno;
@@ -38,7 +25,8 @@ typedef struct xl_clog_truncate
extern void TransactionIdSetTreeStatus(TransactionId xid, int nsubxids,
TransactionId *subxids, XidStatus status, XLogRecPtr lsn);
-extern XidStatus TransactionIdGetStatus(TransactionId xid, XLogRecPtr *lsn);+extern XidStatus TransactionIdGetStatus(TransactionId xid, XLogRecPtr *lsn,+ bool throwError);
extern Size CLOGShmemBuffers(void);
extern Size CLOGShmemSize(void);
diff --git a/src/include/access/clogdefs.h b/src/include/access/clogdefs.hnew file mode 100644index 0000000000..0f9996bb08--- /dev/null+++ b/src/include/access/clogdefs.h@@ -0,0 +1,33 @@+/*+ * clogdefs.h+ *+ * PostgreSQL transaction-commit-log manager+ *+ * Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group+ * Portions Copyright (c) 1994, Regents of the University of California+ *+ * src/include/access/clogdefs.h+ */+#ifndef CLOGDEFS_H+#define CLOGDEFS_H++/*+ * Possible transaction statuses --- note that all-zeroes is the initial+ * state.+ *+ * A "subcommitted" transaction is a committed subtransaction whose parent+ * hasn't committed or aborted yet.+ *+ * An "unknown" status indicates an error condition, such as when the clog has+ * been erroneously truncated and the commit status of a transaction cannot be+ * determined.+ */+typedef enum XidStatus {+ TRANSACTION_STATUS_IN_PROGRESS = 0x00,+ TRANSACTION_STATUS_COMMITTED = 0x01,+ TRANSACTION_STATUS_ABORTED = 0x02,+ TRANSACTION_STATUS_SUB_COMMITTED = 0x03,+ TRANSACTION_STATUS_UNKNOWN = 0x04 /* error condition */+} XidStatus;++#endif /* CLOG_H */diff --git a/src/include/access/slru.h b/src/include/access/slru.hindex b39b43504d..0b6a5669d8 100644--- a/src/include/access/slru.h+++ b/src/include/access/slru.h@@ -133,6 +133,8 @@ typedef struct SlruCtlData
typedef SlruCtlData *SlruCtl;
+#define InvalidSlotNo ((int) -1)+
extern Size SimpleLruShmemSize(int nslots, int nlsns);
extern void SimpleLruInit(SlruCtl ctl, const char *name, int nslots, int nlsns,
@@ -140,9 +142,9 @@ extern void SimpleLruInit(SlruCtl ctl, const char *name, int nslots, int nlsns,
SyncRequestHandler sync_handler);
extern int SimpleLruZeroPage(SlruCtl ctl, int pageno);
extern int SimpleLruReadPage(SlruCtl ctl, int pageno, bool write_ok,
- TransactionId xid);+ TransactionId xid, bool throwError);
extern int SimpleLruReadPage_ReadOnly(SlruCtl ctl, int pageno,
- TransactionId xid);+ TransactionId xid, bool throwError);
extern void SimpleLruWritePage(SlruCtl ctl, int slotno);
extern void SimpleLruWriteAll(SlruCtl ctl, bool allow_redirtied);
extern void SimpleLruTruncate(SlruCtl ctl, int cutoffPage);
diff --git a/src/include/access/transam.h b/src/include/access/transam.hindex 2f1f144db4..7d5e2f614d 100644--- a/src/include/access/transam.h+++ b/src/include/access/transam.h@@ -14,6 +14,7 @@
#ifndef TRANSAM_H
#define TRANSAM_H
+#include "access/clogdefs.h"
#include "access/xlogdefs.h"
@@ -264,6 +265,8 @@ extern PGDLLIMPORT VariableCache ShmemVariableCache;
/*
* prototypes for functions in transam/transam.c
*/
+extern XidStatus TransactionIdResolveStatus(TransactionId transactionId,+ bool throwError);
extern bool TransactionIdDidCommit(TransactionId transactionId);
extern bool TransactionIdDidAbort(TransactionId transactionId);
extern bool TransactionIdIsKnownCompleted(TransactionId transactionId);
--
2.21.1 (Apple Git-122.3)
=
v17-0004-Using-non-throwing-clog-interface-from-amcheck.patch (application/octet-stream, 7.5K) download
From 574d7cb2d8a169fcd50653ec36a7ad919540d808 Mon Sep 17 00:00:00 2001
From: Mark Dilger <mark.dilger@enterprisedb.com>
Date: Tue, 6 Oct 2020 13:33:23 -0700
Subject: [PATCH v17 4/4] Using non-throwing clog interface from amcheck
Converting the heap checking functions to use the recently introduced
non-throwing interface to clog when checking transaction commit status, and
adding corruption reports about missing clog rather than aborting.
---
contrib/amcheck/verify_heapam.c | 84 ++++++++++++++++++++------------
src/tools/pgindent/typedefs.list | 1 -
2 files changed, 54 insertions(+), 31 deletions(-)
diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.cindex b46cce798e..9885a9d55c 100644--- a/contrib/amcheck/verify_heapam.c+++ b/contrib/amcheck/verify_heapam.c@@ -10,6 +10,7 @@
*/
#include "postgres.h"
+#include "access/clogdefs.h"
#include "access/detoast.h"
#include "access/genam.h"
#include "access/heapam.h"
@@ -40,13 +41,6 @@ typedef enum XidBoundsViolation
XID_BOUNDS_OK
} XidBoundsViolation;
-typedef enum XidCommitStatus-{- XID_COMMITTED,- XID_IN_PROGRESS,- XID_ABORTED-} XidCommitStatus;-
typedef enum SkipPages
{
SKIP_PAGES_ALL_FROZEN,
@@ -80,7 +74,7 @@ typedef struct HeapCheckContext
* Cached copies of the most recently checked xid and its status.
*/
TransactionId cached_xid;
- XidCommitStatus cached_status;+ XidStatus cached_status;
/* Values concerning the heap relation being checked */
Relation rel;
@@ -138,7 +132,7 @@ static void update_cached_xid_range(HeapCheckContext *ctx);
static void update_cached_mxid_range(HeapCheckContext *ctx);
static XidBoundsViolation check_mxid_in_range(MultiXactId mxid, HeapCheckContext *ctx);
static XidBoundsViolation check_mxid_valid_in_rel(MultiXactId mxid, HeapCheckContext *ctx);
-static XidBoundsViolation get_xid_status(TransactionId xid, HeapCheckContext *ctx, XidCommitStatus *status);+static XidBoundsViolation get_xid_status(TransactionId xid, HeapCheckContext *ctx, XidStatus *status);
/*
* Scan and report corruption in heap pages, optionally reconciling toasted
@@ -642,7 +636,7 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
else if (infomask & HEAP_MOVED_OFF ||
infomask & HEAP_MOVED_IN)
{
- XidCommitStatus status;+ XidStatus status;
TransactionId xvac = HeapTupleHeaderGetXvac(tuphdr);
switch (get_xid_status(xvac, ctx, &status))
@@ -686,17 +680,27 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
case XID_BOUNDS_OK:
switch (status)
{
- case XID_IN_PROGRESS:+ case TRANSACTION_STATUS_IN_PROGRESS:
return true; /* HEAPTUPLE_DELETE_IN_PROGRESS */
- case XID_COMMITTED:- case XID_ABORTED:+ case TRANSACTION_STATUS_COMMITTED:+ case TRANSACTION_STATUS_ABORTED:
return false; /* HEAPTUPLE_DEAD */
+ case TRANSACTION_STATUS_UNKNOWN:+ report_corruption(ctx,+ /*------+ translator: %u is a transaction identifier */+ psprintf(_("old-style VACUUM FULL transaction ID %u commit status is lost"),+ xvac));+ return false; /* corruption */+ case TRANSACTION_STATUS_SUB_COMMITTED:+ elog(ERROR, "get_xid_status failed to resolve parent transaction status");+ return false; /* not reached */
}
}
}
else
{
- XidCommitStatus status;+ XidStatus status;
switch (get_xid_status(raw_xmin, ctx, &status))
{
@@ -737,12 +741,22 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
case XID_BOUNDS_OK:
switch (status)
{
- case XID_COMMITTED:+ case TRANSACTION_STATUS_COMMITTED:
break;
- case XID_IN_PROGRESS:+ case TRANSACTION_STATUS_IN_PROGRESS:
return true; /* insert or delete in progress */
- case XID_ABORTED:+ case TRANSACTION_STATUS_ABORTED:
return false; /* HEAPTUPLE_DEAD */
+ case TRANSACTION_STATUS_UNKNOWN:+ report_corruption(ctx,+ /*------+ translator: %u is a transaction identifier */+ psprintf(_("raw xmin %u commit status is lost"),+ raw_xmin));+ return false; /* corruption */+ case TRANSACTION_STATUS_SUB_COMMITTED:+ elog(ERROR, "get_xid_status failed to resolve parent transaction status");+ return false; /* not reached */
}
}
}
@@ -752,7 +766,7 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
{
if (infomask & HEAP_XMAX_IS_MULTI)
{
- XidCommitStatus status;+ XidStatus status;
TransactionId xmax = HeapTupleGetUpdateXid(tuphdr);
switch (get_xid_status(xmax, ctx, &status))
@@ -795,12 +809,22 @@ check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)
case XID_BOUNDS_OK:
switch (status)
{
- case XID_IN_PROGRESS:+ case TRANSACTION_STATUS_IN_PROGRESS:
return true; /* HEAPTUPLE_DELETE_IN_PROGRESS */
- case XID_COMMITTED:- case XID_ABORTED:+ case TRANSACTION_STATUS_COMMITTED:+ case TRANSACTION_STATUS_ABORTED:
return false; /* HEAPTUPLE_RECENTLY_DEAD or
* HEAPTUPLE_DEAD */
+ case TRANSACTION_STATUS_UNKNOWN:+ report_corruption(ctx,+ /*------+ translator: %u is a transaction identifier */+ psprintf(_("xmax %u commit status is lost"),+ xmax));+ return false; /* corruption */+ case TRANSACTION_STATUS_SUB_COMMITTED:+ elog(ERROR, "get_xid_status failed to resolve parent transaction status");+ return false; /* not reached */
}
}
@@ -1260,7 +1284,11 @@ check_tuple(HeapCheckContext *ctx)
break;
}
}
- /* If xmax is not a multixact and is normal, it should be within valid range */++ /*+ * If xmax is not a multixact and is normal, it should be within valid+ * range+ */
else
{
switch (get_xid_status(xmax, ctx, NULL))
@@ -1465,7 +1493,7 @@ check_mxid_valid_in_rel(MultiXactId mxid, HeapCheckContext *ctx)
* status argument will be set with the status of the transaction ID.
*/
static XidBoundsViolation
-get_xid_status(TransactionId xid, HeapCheckContext *ctx, XidCommitStatus *status)+get_xid_status(TransactionId xid, HeapCheckContext *ctx, XidStatus *status)
{
XidBoundsViolation result;
FullTransactionId fxid;
@@ -1516,19 +1544,15 @@ get_xid_status(TransactionId xid, HeapCheckContext *ctx, XidCommitStatus *status
return result;
}
- *status = XID_COMMITTED;+ *status = TRANSACTION_STATUS_COMMITTED;
LWLockAcquire(XactTruncationLock, LW_SHARED);
clog_horizon = FullTransactionIdFromXidAndCtx(ShmemVariableCache->oldestClogXid, ctx);
if (FullTransactionIdPrecedesOrEquals(clog_horizon, fxid))
{
if (TransactionIdIsCurrentTransactionId(xid))
- *status = XID_IN_PROGRESS;- else if (TransactionIdDidCommit(xid))- *status = XID_COMMITTED;- else if (TransactionIdDidAbort(xid))- *status = XID_ABORTED;+ *status = TRANSACTION_STATUS_IN_PROGRESS;
else
- *status = XID_IN_PROGRESS;+ *status = TransactionIdResolveStatus(xid, false);
}
LWLockRelease(XactTruncationLock);
ctx->cached_xid = xid;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.listindex 369b8e7c6f..6ca1bac21a 100644--- a/src/tools/pgindent/typedefs.list+++ b/src/tools/pgindent/typedefs.list@@ -2793,7 +2793,6 @@ XactCallbackItem
XactEvent
XactLockTableWaitInfo
XidBoundsViolation
-XidCommitStatus
XidHorizonPrefetchState
XidStatus
XmlExpr
--
2.21.1 (Apple Git-122.3)
=
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: mark.dilger@enterprisedb.com, robertmhaas@gmail.com, pg@bowt.ie, x4mmm@yandex-team.ru, sfrost@snowman.net, michael@paquier.xyz, sulamul@gmail.com, dilipbalaut@gmail.com, andres@anarazel.de, tgl@sss.pgh.pa.us
Subject: Re: new heapcheck contrib module
In-Reply-To: <3894CAFE-E391-4EE0-9599-37C5D90BE740@enterprisedb.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox