From: Mark Dilger <mark.dilger@enterprisedb.com>
To: Amul Sul <sulamul@gmail.com>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: Dilip Kumar <dilipbalaut@gmail.com>
Cc: Peter Geoghegan <pg@bowt.ie>
Cc: Andres Freund <andres@anarazel.de>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: new heapcheck contrib module
Date: Wed, 19 Aug 2020 19:30:27 -0700
Message-ID: <50F6468F-5A5E-412A-86EC-DF24B549D733@enterprisedb.com> (raw)
In-Reply-To: <CAAJ_b94hsrF4x7UKQj03416fAEXJorUpZ9WhsZZjqgv-4+Wg9A@mail.gmail.com>
References: <12ED3DA8-25F0-4B68-937D-D907CFBF08E7@enterprisedb.com>
<20200420194210.sevnpi4lyghoanxi@alap3.anarazel.de>
<23E60C3C-C20A-4508-84AC-E0CEA59E784A@enterprisedb.com>
<CA+TgmobAm8vB1MNQ1taO1et+HxTN8ULnQvEG+grNDqNQTBjbtA@mail.gmail.com>
<F845AC50-A479-43F3-ADFB-861A4D2AD287@enterprisedb.com>
<5F743835-3399-419C-8324-2D424237E999@enterprisedb.com>
<CAFiTN-syyHc3jZoou51v0SR8z0POoNfktqEO6MaGig4YS8mosA@mail.gmail.com>
<E6F4258A-B661-4F2C-8A35-278C9436930A@enterprisedb.com>
<CAFiTN-s-VX3DLfqv7OOo0zGCTOZ3xVU9UZ6GgzBZdJTPNnYFNg@mail.gmail.com>
<885D2EC9-7D06-4903-A17D-72C184A1E1C5@enterprisedb.com>
<CAFiTN-uYg_PiZ854=5R-D9skratapAHKhaDAG7fhJv-Vn5UhaQ@mail.gmail.com>
<5CC85F28-F210-47AC-8DA8-EC4E82AF3FD5@enterprisedb.com>
<CAFiTN-sP5iFJGERTq4a4OLTCzjZKhLC5P6_k2bxQKXnuvdmzGA@mail.gmail.com>
<CAFiTN-tTg0R5AHvXC1ro9XphroX1wUtc5t91hU_+7DeGTpE5VQ@mail.gmail.com>
<0C0349EB-DD85-4DA8-82C7-DF70AB714592@enterprisedb.com>
<CAFiTN-uD3WSeggU20yU6voGou6BDb-cQcfkE8B1rAzWKZVK-Bw@mail.gmail.com>
<BA6E870F-241C-4DDB-B92D-DB39E679C7D4@enterprisedb.com>
<CA+TgmobQUdfM2Q7WAR606RR2YehCZ5GHRKBc-VWfRj-1ZUKqDA@mail.gmail.com>
<C4A3E5AE-CBB2-4E3F-8BAE-29EE216762C2@enterprisedb.com>
<CAAJ_b96_Xr2a+D0j8xKbOnCzURcVQm0V2DS+T_pfc+wt3BpL8g@mail.gmail.com>
<9DF93B83-FE46-4EC4-A99D-77A60729FD2D@enterprisedb.com>
<CA+TgmoZT3kCOaKq6=m5wHGKh_9XQDbCC1LZXL6w_+NMB-n1+mw@mail.gmail.com>
<CAAJ_b94hsrF4x7UKQj03416fAEXJorUpZ9WhsZZjqgv-4+Wg9A@mail.gmail.com>
> On Aug 16, 2020, at 9:37 PM, Amul Sul <sulamul@gmail.com> wrote:
>
> In addition to this, I found a few more things while reading v13 patch are as
> below:
>
> Patch v13-0001:
>
> -
> +#include "amcheck.h"
>
> Not in correct order.
Fixed.
> +typedef struct BtreeCheckContext
> +{
> + TupleDesc tupdesc;
> + Tuplestorestate *tupstore;
> + bool is_corrupt;
> + bool on_error_stop;
> +} BtreeCheckContext;
>
> Unnecessary spaces/tabs between } and BtreeCheckContext.
This refers to a change in verify_nbtree.c that has been removed. Per discussions with Peter and Robert, I have simply withdrawn that portion of the patch.
> static void bt_index_check_internal(Oid indrelid, bool parentcheck,
> - bool heapallindexed, bool rootdescend);
> + bool heapallindexed, bool rootdescend,
> + BtreeCheckContext * ctx);
>
> Unnecessary space between * and ctx. The same changes needed for other places as
> well.
Same as above. The changes to verify_nbtree.c have been withdrawn.
> ---
>
> Patch v13-0002:
>
> +-- partitioned tables (the parent ones) don't have visibility maps
> +create table test_partitioned (a int, b text default repeat('x', 5000))
> + partition by list (a);
> +-- these should all fail
> +select * from verify_heapam('test_partitioned',
> + on_error_stop := false,
> + skip := NULL,
> + startblock := NULL,
> + endblock := NULL);
> +ERROR: "test_partitioned" is not a table, materialized view, or TOAST table
> +create table test_partition partition of test_partitioned for values in (1);
> +create index test_index on test_partition (a);
>
> Can't we make it work? If the input is partitioned, I think we could
> collect all its leaf partitions and process them one by one. Thoughts?
I was following the example from pg_visibility. I haven't thought about your proposal enough to have much opinion as yet, except that if we do this for pg_amcheck we should do likewise to pg_visibility, for consistency of the user interface.
> + ctx->chunkno++;
>
> Instead of incrementing in check_toast_tuple(), I think incrementing should
> happen at the caller -- just after check_toast_tuple() call.
I agree.
> ---
>
> Patch v13-0003:
>
> + resetPQExpBuffer(query);
> + destroyPQExpBuffer(query);
>
> resetPQExpBuffer() will be unnecessary if the next call is destroyPQExpBuffer().
Thanks. I removed it in cases where destroyPQExpBuffer is obviously the very next call.
> + appendPQExpBuffer(query,
> + "SELECT c.relname, v.blkno, v.offnum, v.lp_off, "
> + "v.lp_flags, v.lp_len, v.attnum, v.chunk, v.msg"
> + "\nFROM verify_heapam(rel := %u, on_error_stop := %s, "
> + "skip := %s, startblock := %s, endblock := %s) v, "
> + "pg_class c"
> + "\nWHERE c.oid = %u",
> + tbloid, stop, skip, settings.startblock,
> + settings.endblock, tbloid
>
> pg_class should be schema-qualified like elsewhere.
Agreed, and changed.
> IIUC, pg_class is meant to
> get relname only, instead, we could use '%u'::pg_catalog.regclass in the target
> list for the relname. Thoughts?
get_table_check_list() creates the list of all tables to be checked, which check_tables() then iterates over, calling check_table() for each one. I think some verification that the table still exists is in order. Using '%u'::pg_catalog.regclass for a table that has since been dropped would pass in the old table Oid and draw an error of the 'ERROR: could not open relation with OID 36311' variety, whereas the current coding will just skip the dropped table.
> Also I think we should skip '\n' from the query string (see appendPQExpBuffer()
> in pg_dump.c)
I'm not sure I understand. pg_dump.c uses "\n" in query strings it passes to appendPQExpBuffer(), in a manner very similar to what this patch does.
> + appendPQExpBuffer(query,
> + "SELECT i.indexrelid"
> + "\nFROM pg_catalog.pg_index i, pg_catalog.pg_class c"
> + "\nWHERE i.indexrelid = c.oid"
> + "\n AND c.relam = %u"
> + "\n AND i.indrelid = %u",
> + BTREE_AM_OID, tbloid);
> +
> + ExecuteSqlStatement("RESET search_path");
> + res = ExecuteSqlQuery(query->data, PGRES_TUPLES_OK);
> + PQclear(ExecuteSqlQueryForSingleRow(ALWAYS_SECURE_SEARCH_PATH_SQL));
>
> I don't think we need the search_path query. The main query doesn't have any
> dependencies on it. Same is in check_indexes(), check_index (),
> expand_table_name_patterns() & get_table_check_list().
> Correct me if I am missing something.
Right.
> + output = PageOutput(lines + 2, NULL);
> + for (lineno = 0; usage_text[lineno]; lineno++)
> + fprintf(output, "%s\n", usage_text[lineno]);
> + fprintf(output, "Report bugs to <%s>.\n", PACKAGE_BUGREPORT);
> + fprintf(output, "%s home page: <%s>\n", PACKAGE_NAME, PACKAGE_URL);
>
> I am not sure why we want PageOutput() if the second argument is always going to
> be NULL? Can't we directly use printf() instead of PageOutput() + fprintf() ?
> e.g. usage() function in pg_basebackup.c.
Done.
Please find attached the next version of the patch. In addition to your review comments (above), I have made changes in response to Peter and Robert's review comments upthread.
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:download all (2 files, .tar.gz)v14-0001-Adding-function-verify_heapam-to-amcheck-module.patch (application/octet-stream, 78.9K) download
From 322ebe44a5e54c6ec3ede10aef7073aa91a1d1fb Mon Sep 17 00:00:00 2001
From: Mark Dilger <mark.dilger@enterprisedb.com>
Date: Wed, 19 Aug 2020 18:55:11 -0700
Subject: [PATCH v14 1/2] Adding function verify_heapam to amcheck module
Adding new function verify_heapam for checking a heap relation and
optionally its associated toast relation, if any.
---
contrib/amcheck/Makefile | 7 +-
contrib/amcheck/amcheck--1.2--1.3.sql | 27 +
contrib/amcheck/amcheck.control | 2 +-
contrib/amcheck/amcheck.h | 5 +
contrib/amcheck/expected/check_heap.out | 146 +++
contrib/amcheck/sql/check_heap.sql | 100 ++
contrib/amcheck/t/001_verify_heapam.pl | 242 ++++
contrib/amcheck/verify_heapam.c | 1432 +++++++++++++++++++++++
doc/src/sgml/amcheck.sgml | 215 ++++
src/backend/access/heap/hio.c | 11 +
src/backend/access/transam/multixact.c | 19 +
src/include/access/multixact.h | 1 +
src/tools/pgindent/typedefs.list | 2 +
13 files changed, 2206 insertions(+), 3 deletions(-)
create mode 100644 contrib/amcheck/amcheck--1.2--1.3.sql
create mode 100644 contrib/amcheck/amcheck.h
create mode 100644 contrib/amcheck/expected/check_heap.out
create mode 100644 contrib/amcheck/sql/check_heap.sql
create mode 100644 contrib/amcheck/t/001_verify_heapam.pl
create mode 100644 contrib/amcheck/verify_heapam.c
diff --git a/contrib/amcheck/Makefile b/contrib/amcheck/Makefileindex a2b1b1036b..b82f221e50 100644--- a/contrib/amcheck/Makefile+++ b/contrib/amcheck/Makefile@@ -3,13 +3,16 @@
MODULE_big = amcheck
OBJS = \
$(WIN32RES) \
+ verify_heapam.o \
verify_nbtree.o
EXTENSION = amcheck
-DATA = amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql+DATA = amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
PGFILEDESC = "amcheck - function for verifying relation integrity"
-REGRESS = check check_btree+REGRESS = check check_btree check_heap++TAP_TESTS = 1
ifdef USE_PGXS
PG_CONFIG = pg_config
diff --git a/contrib/amcheck/amcheck--1.2--1.3.sql b/contrib/amcheck/amcheck--1.2--1.3.sqlnew file mode 100644index 0000000000..aa7c381ccd--- /dev/null+++ b/contrib/amcheck/amcheck--1.2--1.3.sql@@ -0,0 +1,27 @@+/* contrib/amcheck/amcheck--1.2--1.3.sql */++-- complain if script is sourced in psql, rather than via CREATE EXTENSION+\echo Use "ALTER EXTENSION amcheck UPDATE TO '1.3'" to load this file. \quit++--+-- verify_heapam()+--+CREATE FUNCTION verify_heapam(relation regclass,+ on_error_stop boolean default false,+ check_toast boolean default false,+ skip cstring default 'none',+ startblock bigint default null,+ endblock bigint default null,+ blkno OUT bigint,+ offnum OUT integer,+ attnum OUT integer,+ msg OUT text+ )+RETURNS SETOF record+AS 'MODULE_PATHNAME', 'verify_heapam'+LANGUAGE C;++-- Don't want this to be available to public+REVOKE ALL ON FUNCTION+verify_heapam(regclass, boolean, boolean, cstring, bigint, bigint)+FROM PUBLIC;diff --git a/contrib/amcheck/amcheck.control b/contrib/amcheck/amcheck.controlindex c6e310046d..ab50931f75 100644--- a/contrib/amcheck/amcheck.control+++ b/contrib/amcheck/amcheck.control@@ -1,5 +1,5 @@
# amcheck extension
comment = 'functions for verifying relation integrity'
-default_version = '1.2'+default_version = '1.3'
module_pathname = '$libdir/amcheck'
relocatable = true
diff --git a/contrib/amcheck/amcheck.h b/contrib/amcheck/amcheck.hnew file mode 100644index 0000000000..74edfc2f65--- /dev/null+++ b/contrib/amcheck/amcheck.h@@ -0,0 +1,5 @@+#include "postgres.h"++Datum verify_heapam(PG_FUNCTION_ARGS);+Datum bt_index_check(PG_FUNCTION_ARGS);+Datum bt_index_parent_check(PG_FUNCTION_ARGS);diff --git a/contrib/amcheck/expected/check_heap.out b/contrib/amcheck/expected/check_heap.outnew file mode 100644index 0000000000..c494b28198--- /dev/null+++ b/contrib/amcheck/expected/check_heap.out@@ -0,0 +1,146 @@+CREATE TABLE heaptest (a integer, b text);+-- Check that invalid skip option is rejected+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'rope');+ERROR: invalid skip option+HINT: Valid skip options are "all-visible", "all-frozen", and "none".+-- Check that block range is reject for an empty table+SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 0, endblock := 0);+ERROR: starting block is out of bounds for relation with no blocks: 0+-- Check that valid options are not rejected nor corruption reported+-- for an empty table+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'none');+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-frozen');+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-visible');+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++-- Add some data so subsequent tests are not entirely trivial+INSERT INTO heaptest (a, b)+ (SELECT gs, repeat('x', gs)+ FROM generate_series(1,50) gs);+-- Check that an invalid block range is rejected+SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 100000, endblock := 200000);+ERROR: block range is out of bounds for relation with block count 1: 100000 .. 200000+-- Check that valid options are not rejected nor corruption reported+-- for a non-empty table+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'none');+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-frozen');+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-visible');+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 0, endblock := 0);+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++-- Vacuum freeze to change the xids encountered in subsequent tests+VACUUM FREEZE heaptest;+-- Check that valid options are not rejected nor corruption reported+-- for a non-empty frozen table+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'none');+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-frozen');+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-visible');+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 0, endblock := 0);+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++-- Check that partitioned tables (the parent ones) which don't have visibility+-- maps are rejected+create table test_partitioned (a int, b text default repeat('x', 5000))+ partition by list (a);+select * from verify_heapam('test_partitioned',+ on_error_stop := false,+ skip := NULL,+ startblock := NULL,+ endblock := NULL);+ERROR: "test_partitioned" is not a table, materialized view, or TOAST table+-- Check that valid options are not rejected nor corruption reported+-- for an empty partition table (the child one)+create table test_partition partition of test_partitioned for values in (1);+select * from verify_heapam('test_partition',+ on_error_stop := false,+ skip := NULL,+ startblock := NULL,+ endblock := NULL);+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++-- Check that valid options are not rejected nor corruption reported+-- for a non-empty partition table (the child one)+insert into test_partitioned (a) (select 1 from generate_series(1,1000) gs);+select * from verify_heapam('test_partition',+ on_error_stop := false,+ skip := NULL,+ startblock := NULL,+ endblock := NULL);+ blkno | offnum | attnum | msg +-------+--------+--------+-----+(0 rows)++-- Check that indexes are rejected+create index test_index on test_partition (a);+select * from verify_heapam('test_index',+ on_error_stop := false,+ skip := NULL,+ startblock := NULL,+ endblock := NULL);+ERROR: "test_index" is not a table, materialized view, or TOAST table+-- Check that views are rejected+create view test_view as select 1;+select * from verify_heapam('test_view',+ on_error_stop := false,+ skip := NULL,+ startblock := NULL,+ endblock := NULL);+ERROR: "test_view" is not a table, materialized view, or TOAST table+-- Check that sequences are rejected+create sequence test_sequence;+select * from verify_heapam('test_sequence',+ on_error_stop := false,+ skip := NULL,+ startblock := NULL,+ endblock := NULL);+ERROR: "test_sequence" is not a table, materialized view, or TOAST table+-- Check that foreign tables are rejected+create foreign data wrapper dummy;+create server dummy_server foreign data wrapper dummy;+create foreign table test_foreign_table () server dummy_server;+select * from verify_heapam('test_foreign_table',+ on_error_stop := false,+ skip := NULL,+ startblock := NULL,+ endblock := NULL);+ERROR: "test_foreign_table" is not a table, materialized view, or TOAST tablediff --git a/contrib/amcheck/sql/check_heap.sql b/contrib/amcheck/sql/check_heap.sqlnew file mode 100644index 0000000000..30d00571f2--- /dev/null+++ b/contrib/amcheck/sql/check_heap.sql@@ -0,0 +1,100 @@+CREATE TABLE heaptest (a integer, b text);++-- Check that invalid skip option is rejected+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'rope');++-- Check that block range is reject for an empty table+SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 0, endblock := 0);++-- Check that valid options are not rejected nor corruption reported+-- for an empty table+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'none');+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-frozen');+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-visible');++-- Add some data so subsequent tests are not entirely trivial+INSERT INTO heaptest (a, b)+ (SELECT gs, repeat('x', gs)+ FROM generate_series(1,50) gs);++-- Check that an invalid block range is rejected+SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 100000, endblock := 200000);++-- Check that valid options are not rejected nor corruption reported+-- for a non-empty table+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'none');+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-frozen');+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-visible');+SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 0, endblock := 0);++-- Vacuum freeze to change the xids encountered in subsequent tests+VACUUM FREEZE heaptest;++-- Check that valid options are not rejected nor corruption reported+-- for a non-empty frozen table+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'none');+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-frozen');+SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-visible');+SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 0, endblock := 0);++-- Check that partitioned tables (the parent ones) which don't have visibility+-- maps are rejected+create table test_partitioned (a int, b text default repeat('x', 5000))+ partition by list (a);+select * from verify_heapam('test_partitioned',+ on_error_stop := false,+ skip := NULL,+ startblock := NULL,+ endblock := NULL);++-- Check that valid options are not rejected nor corruption reported+-- for an empty partition table (the child one)+create table test_partition partition of test_partitioned for values in (1);+select * from verify_heapam('test_partition',+ on_error_stop := false,+ skip := NULL,+ startblock := NULL,+ endblock := NULL);++-- Check that valid options are not rejected nor corruption reported+-- for a non-empty partition table (the child one)+insert into test_partitioned (a) (select 1 from generate_series(1,1000) gs);+select * from verify_heapam('test_partition',+ on_error_stop := false,+ skip := NULL,+ startblock := NULL,+ endblock := NULL);++-- Check that indexes are rejected+create index test_index on test_partition (a);+select * from verify_heapam('test_index',+ on_error_stop := false,+ skip := NULL,+ startblock := NULL,+ endblock := NULL);++-- Check that views are rejected+create view test_view as select 1;+select * from verify_heapam('test_view',+ on_error_stop := false,+ skip := NULL,+ startblock := NULL,+ endblock := NULL);++-- Check that sequences are rejected+create sequence test_sequence;+select * from verify_heapam('test_sequence',+ on_error_stop := false,+ skip := NULL,+ startblock := NULL,+ endblock := NULL);++-- Check that foreign tables are rejected+create foreign data wrapper dummy;+create server dummy_server foreign data wrapper dummy;+create foreign table test_foreign_table () server dummy_server;+select * from verify_heapam('test_foreign_table',+ on_error_stop := false,+ skip := NULL,+ startblock := NULL,+ endblock := NULL);diff --git a/contrib/amcheck/t/001_verify_heapam.pl b/contrib/amcheck/t/001_verify_heapam.plnew file mode 100644index 0000000000..7da8526198--- /dev/null+++ b/contrib/amcheck/t/001_verify_heapam.pl@@ -0,0 +1,242 @@+use strict;+use warnings;++use PostgresNode;+use TestLib;++use Test::More tests => 125;++my ($node, $result);++#+# Test set-up+#+$node = get_new_node('test');+$node->init;+$node->append_conf('postgresql.conf', 'autovacuum=off');+$node->start;+$node->safe_psql('postgres', q(CREATE EXTENSION amcheck));++#+# Check a table with data loaded but no corruption, freezing, etc.+#+fresh_test_table('test');+check_all_options_uncorrupted('test', 'plain');++#+# Check a corrupt table+#+fresh_test_table('test');+corrupt_first_page('test');+detects_corruption(+ "verify_heapam('test')",+ "plain corrupted table");+detects_corruption(+ "verify_heapam('test', skip := 'all-visible')",+ "plain corrupted table skipping all-visible");+detects_corruption(+ "verify_heapam('test', skip := 'all-frozen')",+ "plain corrupted table skipping all-frozen");+detects_corruption(+ "verify_heapam('test', check_toast := false)",+ "plain corrupted table skipping toast");+detects_corruption(+ "verify_heapam('test', startblock := 0, endblock := 0)",+ "plain corrupted table checking only block zero");++#+# Check a corrupt table with all-frozen data+#+fresh_test_table('test');+$node->safe_psql('postgres', q(VACUUM FREEZE test));+corrupt_first_page('test');+detects_corruption(+ "verify_heapam('test')",+ "all-frozen corrupted table");+detects_no_corruption(+ "verify_heapam('test', skip := 'all-frozen')",+ "all-frozen corrupted table skipping all-frozen");++#+# Check a corrupt table with corrupt page header+#+fresh_test_table('test');+corrupt_first_page_and_header('test');+detects_corruption(+ "verify_heapam('test')",+ "corrupted test table with bad page header");++#+# Check an uncorrupted table with corrupt toast page header+#+fresh_test_table('test');+my $toast = get_toast_for('test');+corrupt_first_page_and_header($toast);+detects_corruption(+ "verify_heapam('test', check_toast := true)",+ "table with corrupted toast page header checking toast");+detects_no_corruption(+ "verify_heapam('test', check_toast := false)",+ "table with corrupted toast page header skipping toast");+detects_corruption(+ "verify_heapam('$toast')",+ "corrupted toast page header");++#+# Check an uncorrupted table with corrupt toast+#+fresh_test_table('test');+$toast = get_toast_for('test');+corrupt_first_page($toast);+detects_corruption(+ "verify_heapam('test', check_toast := true)",+ "table with corrupted toast checking toast");+detects_no_corruption(+ "verify_heapam('test', check_toast := false)",+ "table with corrupted toast skipping toast");+detects_corruption(+ "verify_heapam('$toast')",+ "corrupted toast table");++#+# Check an uncorrupted all-frozen table with corrupt toast+#+fresh_test_table('test');+$node->safe_psql('postgres', q(VACUUM FREEZE test));+$toast = get_toast_for('test');+corrupt_first_page($toast);+detects_corruption(+ "verify_heapam('test', check_toast := true)",+ "all-frozen table with corrupted toast checking toast");+detects_no_corruption(+ "verify_heapam('test', check_toast := false)",+ "all-frozen table with corrupted toast skipping toast");+detects_corruption(+ "verify_heapam('$toast')",+ "corrupted toast table of all-frozen table");++# Returns the filesystem path for the named relation.+sub relation_filepath+{+ my ($relname) = @_;++ my $pgdata = $node->data_dir;+ my $rel = $node->safe_psql('postgres',+ qq(SELECT pg_relation_filepath('$relname')));+ die "path not found for relation $relname" unless defined $rel;+ return "$pgdata/$rel";+}++# Returns the fully qualified name of the toast table for the named relation+sub get_toast_for+{+ my ($relname) = @_;+ $node->safe_psql('postgres', qq(+ SELECT 'pg_toast.' || t.relname+ FROM pg_catalog.pg_class c, pg_catalog.pg_class t+ WHERE c.relname = '$relname'+ AND c.reltoastrelid = t.oid));+}++# (Re)create and populate a test table of the given name.+sub fresh_test_table+{+ my ($relname) = @_;+ $node->safe_psql('postgres', qq(+ DROP TABLE IF EXISTS $relname CASCADE;+ CREATE TABLE $relname (a integer, b text);+ ALTER TABLE $relname SET (autovacuum_enabled=false);+ ALTER TABLE $relname ALTER b SET STORAGE external;+ INSERT INTO $relname (a, b)+ (SELECT gs, repeat('b',gs*10) FROM generate_series(1,1000) gs);+ ));+}++# Stops the test node, corrupts the first page of the named relation, and+# restarts the node.+sub corrupt_first_page_internal+{+ my ($relname, $corrupt_header) = @_;+ my $relpath = relation_filepath($relname);++ $node->stop;+ my $fh;+ open($fh, '+<', $relpath);+ binmode $fh;++ # If we corrupt the header, postgres won't allow the page into the buffer.+ syswrite($fh, '\xFF\xFF\xFF\xFF', 8) if ($corrupt_header);++ # Corrupt at least the line pointers. Exactly what this corrupts will+ # depend on the page, as it may run past the line pointers into the user+ # data. We stop short of writing 2048 bytes (2k), the smallest supported+ # page size, as we don't want to corrupt the next page.+ seek($fh, 32, 0);+ syswrite($fh, '\x77\x77\x77\x77', 500);+ close($fh);+ $node->start;+}++sub corrupt_first_page+{+ corrupt_first_page_internal($_[0], undef);+}++sub corrupt_first_page_and_header+{+ corrupt_first_page_internal($_[0], 1);+}++sub detects_corruption+{+ my ($function, $testname) = @_;++ my $result = $node->safe_psql('postgres',+ qq(SELECT COUNT(*) > 0 FROM $function));+ is($result, 't', $testname);+}++sub detects_no_corruption+{+ my ($function, $testname) = @_;++ my $result = $node->safe_psql('postgres',+ qq(SELECT COUNT(*) = 0 FROM $function));+ is($result, 't', $testname);+}++# Check various options are stable (don't abort) and do not report corruption+# when running verify_heapam on an uncorrupted test table.+#+# The relname *must* be an uncorrupted table, or this will fail.+#+# The prefix is used to identify the test, along with the options,+# and should be unique.+sub check_all_options_uncorrupted+{+ my ($relname, $prefix) = @_;+ for my $stop (qw(NULL true false))+ {+ for my $check_toast (qw(NULL true false))+ {+ for my $skip ("'none'", "'all-frozen'", "'all-visible'")+ {+ for my $startblock (qw(NULL 0))+ {+ for my $endblock (qw(NULL 0))+ {+ my $opts = "on_error_stop := $stop, " .+ "check_toast := $check_toast, " .+ "skip := $skip, " .+ "startblock := $startblock, " .+ "endblock := $endblock";++ detects_no_corruption(+ "verify_heapam('$relname', $opts)",+ "$prefix: $opts");+ }+ }+ }+ }+ }+}diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.cnew file mode 100644index 0000000000..dbdf3d23f4--- /dev/null+++ b/contrib/amcheck/verify_heapam.c@@ -0,0 +1,1432 @@+/*-------------------------------------------------------------------------+ *+ * verify_heapam.c+ * Functions to check postgresql heap relations for corruption+ *+ * Copyright (c) 2016-2020, PostgreSQL Global Development Group+ *+ * contrib/amcheck/verify_heapam.c+ *-------------------------------------------------------------------------+ */+#include "postgres.h"++#include "access/detoast.h"+#include "access/genam.h"+#include "access/heapam.h"+#include "access/heaptoast.h"+#include "access/htup_details.h"+#include "access/multixact.h"+#include "access/toast_internals.h"+#include "access/visibilitymap.h"+#include "access/xact.h"+#include "catalog/pg_am.h"+#include "catalog/pg_type.h"+#include "catalog/storage_xlog.h"+#include "funcapi.h"+#include "miscadmin.h"+#include "storage/bufmgr.h"+#include "storage/procarray.h"+#include "storage/smgr.h"+#include "utils/builtins.h"+#include "utils/fmgroids.h"+#include "utils/lsyscache.h"+#include "utils/rel.h"+#include "utils/snapmgr.h"+#include "utils/syscache.h"++PG_FUNCTION_INFO_V1(verify_heapam);++/* The number of columns in tuples returned by verify_heapam */+#define HEAPCHECK_RELATION_COLS 4++typedef enum XidCommitStatus+{+ XID_TOO_NEW,+ XID_TOO_OLD,+ XID_COMMITTED,+ XID_IN_PROGRESS,+ XID_ABORTED+} XidCommitStatus;++typedef enum SkipPages+{+ SKIP_PAGES_ALL_FROZEN,+ SKIP_PAGES_ALL_VISIBLE,+ SKIP_PAGES_NONE+} SkipPages;++/*+ * Struct holding the running context information during+ * a lifetime of a verify_heapam execution.+ */+typedef struct HeapCheckContext+{+ /*+ * Cached copies of values from ShmemVariableCache and computed values from+ * them.+ */+ FullTransactionId next_fxid; /* ShmemVariableCache->nextXid */+ TransactionId next_xid; /* 32-bit version of next_fxid */+ TransactionId oldest_xid; /* ShmemVariableCache->oldestXid */+ FullTransactionId oldest_fxid; /* 64-bit version of oldest_xid, computed+ * relative to next_fxid */++ /*+ * Cached copy of value from MultiXactState+ */+ MultiXactId next_mxact; /* MultiXactState->nextMXact */+ MultiXactId oldest_mxact; /* MultiXactState->oldestMultiXactId */++ /*+ * Cached copies of the most recently checked xid and its status.+ */+ TransactionId cached_xid;+ XidCommitStatus cached_status;++ /* Values concerning the heap relation being checked */+ Relation rel;+ TransactionId relfrozenxid;+ FullTransactionId relfrozenfxid;+ TransactionId relminmxid;+ Relation toast_rel;+ Relation *toast_indexes;+ Relation valid_toast_index;+ int num_toast_indexes;++ /* Values for iterating over pages in the relation */+ BlockNumber nblocks;+ BlockNumber blkno;+ BufferAccessStrategy bstrategy;+ Buffer buffer;+ Page page;++ /* Values for iterating over tuples within a page */+ OffsetNumber offnum;+ ItemId itemid;+ uint16 lp_len;+ HeapTupleHeader tuphdr;+ int natts;++ /* Values for iterating over attributes within the tuple */+ uint32 offset; /* offset in tuple data */+ AttrNumber attnum;++ /* Values for iterating over toast for the attribute */+ int32 chunkno;+ int32 attrsize;+ int32 endchunk;+ int32 totalchunks;++ /* Whether verify_heapam has yet encountered any corrupt tuples */+ bool is_corrupt;++ /* The descriptor and tuplestore for verify_heapam's result tuples */+ TupleDesc tupdesc;+ Tuplestorestate *tupstore;+} HeapCheckContext;++/* Internal implementation */+static void check_relation_relkind_and_relam(Relation rel);+static void check_tuple(HeapCheckContext *ctx);+static void check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx);++static bool check_tuple_attribute(HeapCheckContext *ctx);+static bool check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx);++static void report_corruption(HeapCheckContext *ctx, char *msg);+static TupleDesc verify_heapam_tupdesc(void);+static bool xid_valid_in_rel(TransactionId xid, HeapCheckContext *ctx);+static FullTransactionId FullTransactionIdFromXidAndCtx(TransactionId xid, const HeapCheckContext *ctx);+static void update_cached_xid_range(HeapCheckContext *ctx);+static void update_cached_mxid_range(HeapCheckContext *ctx);+static XidCommitStatus get_xid_status(TransactionId xid, HeapCheckContext *ctx);++/*+ * Return whether the given FullTransactionId is within our cached valid+ * transaction ID range.+ */+static inline bool+fxid_in_cached_range(FullTransactionId fxid, const HeapCheckContext *ctx)+{+ return (FullTransactionIdPrecedesOrEquals(ctx->oldest_fxid, fxid) &&+ FullTransactionIdPrecedes(fxid, ctx->next_fxid));+}++/*+ * Scan and report corruption in heap pages, optionally reconciling toasted+ * attributes with entries in the associated toast table. Intended to be+ * called from SQL with the following parameters:+ *+ * relation+ * The Oid of the heap relation to be checked.+ *+ * on_error_stop:+ * Whether to stop at the end of the first page for which errors are+ * detected. Note that multiple rows may be returned.+ *+ * check_toast:+ * Whether to check each toasted attribute against the toast table to+ * verify that it can be found there.+ *+ * skip:+ * What kinds of pages in the heap relation should be skipped. Valid+ * options are "all-visible", "all-frozen", and "none".+ *+ * Returns to the SQL caller a set of tuples, each containing the location+ * and a description of a corruption found in the heap.+ *+ * Note that if check_toast is true, it is the caller's responsibility to+ * provide that the toast table and index are not corrupt, and that they+ * do not become corrupt while this function is running.+ */+Datum+verify_heapam(PG_FUNCTION_ARGS)+{+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;+ MemoryContext old_context;+ bool random_access;+ HeapCheckContext ctx;+ Buffer vmbuffer = InvalidBuffer;+ Oid relid;+ bool on_error_stop;+ bool check_toast;+ SkipPages skip_option = SKIP_PAGES_NONE;+ int64 start_block;+ int64 end_block;++ /* Check to see if caller supports us returning a tuplestore */+ if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))+ ereport(ERROR,+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),+ errmsg("set-valued function called in context that cannot accept a set")));+ if (!(rsinfo->allowedModes & SFRM_Materialize))+ ereport(ERROR,+ (errcode(ERRCODE_SYNTAX_ERROR),+ errmsg("materialize mode required, but it is not allowed in this context")));++ /* Check supplied arguments */+ if (PG_ARGISNULL(0))+ ereport(ERROR,+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),+ errmsg("relation cannot be null")));+ relid = PG_GETARG_OID(0);+ on_error_stop = PG_ARGISNULL(1) ? false : PG_GETARG_BOOL(1);+ check_toast = PG_ARGISNULL(2) ? false : PG_GETARG_BOOL(2);+ if (!PG_ARGISNULL(3))+ {+ const char *skip = PG_GETARG_CSTRING(3);++ if (pg_strcasecmp(skip, "all-visible") == 0)+ skip_option = SKIP_PAGES_ALL_VISIBLE;+ else if (pg_strcasecmp(skip, "all-frozen") == 0)+ skip_option = SKIP_PAGES_ALL_FROZEN;+ else if (pg_strcasecmp(skip, "none") == 0)+ skip_option = SKIP_PAGES_NONE;+ else+ ereport(ERROR,+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),+ errmsg("invalid skip option"),+ errhint("Valid skip options are \"all-visible\", \"all-frozen\", and \"none\".")));+ }++ memset(&ctx, 0, sizeof(HeapCheckContext));+ ctx.cached_xid = InvalidTransactionId;++ /* The tupdesc and tuplestore must be created in ecxt_per_query_memory */+ old_context = MemoryContextSwitchTo(rsinfo->econtext->ecxt_per_query_memory);+ random_access = (rsinfo->allowedModes & SFRM_Materialize_Random) != 0;+ ctx.tupdesc = verify_heapam_tupdesc();+ ctx.tupstore = tuplestore_begin_heap(random_access, false, work_mem);+ rsinfo->returnMode = SFRM_Materialize;+ rsinfo->setResult = ctx.tupstore;+ rsinfo->setDesc = ctx.tupdesc;+ MemoryContextSwitchTo(old_context);++ /*+ * Open the relation.+ */+ ctx.rel = relation_open(relid, AccessShareLock);+ check_relation_relkind_and_relam(ctx.rel);+ ctx.nblocks = RelationGetNumberOfBlocks(ctx.rel);++ /* Early exit if the relation is empty */+ if (!ctx.nblocks)+ {+ /*+ * For consistency, we need to enforce that the start_block and+ * end_block are within the valid range if the user specified them.+ * Yet, for an empty table with no blocks, no specified block can be+ * in range.+ */+ if (!PG_ARGISNULL(4))+ ereport(ERROR,+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),+ /*------+ translator: The integer value is the offset number of the+ block. */+ errmsg("starting block is out of bounds for relation with no blocks: " INT64_FORMAT,+ PG_GETARG_INT64(4))));+ if (!PG_ARGISNULL(5))+ ereport(ERROR,+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),+ /*------+ translator: The integer value is the offset number of the+ block. */+ errmsg("ending block is out of bounds for relation with no blocks: " INT64_FORMAT,+ PG_GETARG_INT64(5))));+ relation_close(ctx.rel, AccessShareLock);+ PG_RETURN_NULL();+ }++ ctx.bstrategy = GetAccessStrategy(BAS_BULKREAD);+ ctx.buffer = InvalidBuffer;+ ctx.page = NULL;++ /* If we get this far, we know the relation has at least one block */+ start_block = PG_ARGISNULL(4) ? 0 : PG_GETARG_INT64(4);+ end_block = PG_ARGISNULL(5) ? ((int64) ctx.nblocks) - 1 : PG_GETARG_INT64(5);+ if (start_block < 0 || end_block >= ctx.nblocks || start_block > end_block)+ ereport(ERROR,+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),+ /*------+ translator: The first integer value is the total number of+ blocks in the relation. The second and third integer values+ represent starting and ending block offsets. */+ errmsg("block range is out of bounds for relation with block count %u: " INT64_FORMAT " .. " INT64_FORMAT,+ ctx.nblocks, start_block, end_block)));++ /*+ * Optionally open the toast relation, if any, also protected from+ * concurrent vacuums.+ */+ if (ctx.rel->rd_rel->reltoastrelid && check_toast)+ {+ int offset;++ /* Main relation has associated toast relation */+ ctx.toast_rel = table_open(ctx.rel->rd_rel->reltoastrelid,+ AccessShareLock);+ offset = toast_open_indexes(ctx.toast_rel,+ AccessShareLock,+ &(ctx.toast_indexes),+ &(ctx.num_toast_indexes));+ ctx.valid_toast_index = ctx.toast_indexes[offset];+ }+ else+ {+ /*+ * Main relation has no associated toast relation, or we're+ * intentionally skipping it+ */+ ctx.toast_rel = NULL;+ ctx.toast_indexes = NULL;+ ctx.num_toast_indexes = 0;+ }++ update_cached_xid_range(&ctx);+ update_cached_mxid_range(&ctx);+ ctx.relfrozenxid = ctx.rel->rd_rel->relfrozenxid;+ ctx.relfrozenfxid = FullTransactionIdFromXidAndCtx(ctx.relfrozenxid, &ctx);+ ctx.relminmxid = ctx.rel->rd_rel->relminmxid;++ if (TransactionIdIsNormal(ctx.relfrozenxid))+ ctx.oldest_xid = ctx.relfrozenxid;++ for (ctx.blkno = start_block; ctx.blkno <= end_block; ctx.blkno++)+ {+ int32 mapbits;+ OffsetNumber maxoff;+ PageHeader ph;++ /* Optionally skip over all-frozen or all-visible blocks */+ if (skip_option != SKIP_PAGES_NONE)+ {+ bool all_frozen,+ all_visible;++ mapbits = (int32) visibilitymap_get_status(ctx.rel, ctx.blkno,+ &vmbuffer);+ all_frozen = mapbits & VISIBILITYMAP_ALL_VISIBLE;+ all_visible = mapbits & VISIBILITYMAP_ALL_FROZEN;++ if ((all_frozen && skip_option == SKIP_PAGES_ALL_FROZEN) ||+ (all_visible && skip_option == SKIP_PAGES_ALL_VISIBLE))+ {+ continue;+ }+ }++ /* Read and lock the next page. */+ ctx.buffer = ReadBufferExtended(ctx.rel, MAIN_FORKNUM, ctx.blkno,+ RBM_NORMAL, ctx.bstrategy);+ LockBuffer(ctx.buffer, BUFFER_LOCK_SHARE);+ ctx.page = BufferGetPage(ctx.buffer);+ ph = (PageHeader) ctx.page;++ /* Perform tuple checks */+ maxoff = PageGetMaxOffsetNumber(ctx.page);+ for (ctx.offnum = FirstOffsetNumber; ctx.offnum <= maxoff;+ ctx.offnum = OffsetNumberNext(ctx.offnum))+ {+ ctx.itemid = PageGetItemId(ctx.page, ctx.offnum);++ /* Skip over unused/dead line pointers */+ if (!ItemIdIsUsed(ctx.itemid) || ItemIdIsDead(ctx.itemid))+ continue;++ /*+ * If this line pointer has been redirected, check that it+ * redirects to a valid offset within the line pointer array.+ */+ if (ItemIdIsRedirected(ctx.itemid))+ {+ OffsetNumber rdoffnum = ItemIdGetRedirect(ctx.itemid);+ ItemId rditem;++ if (rdoffnum < FirstOffsetNumber || rdoffnum > maxoff)+ {+ report_corruption(&ctx,+ /* translator: Both %u are offsets. */+ psprintf(_("line pointer redirection to item at offset exceeding maximum: %u vs. %u"),+ (unsigned) rdoffnum,+ (unsigned) maxoff));+ continue;+ }+ rditem = PageGetItemId(ctx.page, rdoffnum);+ if (!ItemIdIsUsed(rditem))+ report_corruption(&ctx,+ /* translator: The %u is an offset. */+ psprintf(_("line pointer redirection to unused item at offset: %u"),+ (unsigned) rdoffnum));+ continue;+ }++ /* Set up context information about this next tuple */+ ctx.lp_len = ItemIdGetLength(ctx.itemid);+ ctx.tuphdr = (HeapTupleHeader) PageGetItem(ctx.page, ctx.itemid);+ ctx.natts = HeapTupleHeaderGetNatts(ctx.tuphdr);++ /* Ok, ready to check this next tuple */+ check_tuple(&ctx);+ }++ /* clean up */+ UnlockReleaseBuffer(ctx.buffer);++ if (on_error_stop && ctx.is_corrupt)+ break;+ }++ if (vmbuffer != InvalidBuffer)+ ReleaseBuffer(vmbuffer);++ /* Close the associated toast table and indexes, if any. */+ if (ctx.toast_indexes)+ toast_close_indexes(ctx.toast_indexes, ctx.num_toast_indexes,+ AccessShareLock);+ if (ctx.toast_rel)+ table_close(ctx.toast_rel, AccessShareLock);++ /* Close the main relation */+ relation_close(ctx.rel, AccessShareLock);++ PG_RETURN_NULL();+}++/*+ * Check that a relation is of a supported relkind.+ */+static void+check_relation_relkind_and_relam(Relation rel)+{+ if (rel->rd_rel->relkind != RELKIND_RELATION &&+ rel->rd_rel->relkind != RELKIND_MATVIEW &&+ rel->rd_rel->relkind != RELKIND_TOASTVALUE)+ ereport(ERROR,+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),+ /* translator: %s is a user supplied object name */+ errmsg("\"%s\" is not a table, materialized view, or TOAST table",+ RelationGetRelationName(rel))));+ if (rel->rd_rel->relam != HEAP_TABLE_AM_OID)+ ereport(ERROR,+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),+ errmsg("only heap AM is supported")));+}++/*+ * Record a single corruption found in the table. The values in ctx should+ * reflect the location of the corruption, and the msg argument should contain+ * a human readable description of the corruption.+ *+ * The msg argument is pfree'd by this function.+ */+static void+report_corruption(HeapCheckContext *ctx, char *msg)+{+ Datum values[HEAPCHECK_RELATION_COLS];+ bool nulls[HEAPCHECK_RELATION_COLS];+ HeapTuple tuple;++ MemSet(values, 0, sizeof(values));+ MemSet(nulls, 0, sizeof(nulls));+ values[0] = Int64GetDatum(ctx->blkno);+ values[1] = Int32GetDatum(ctx->offnum);+ values[2] = Int32GetDatum(ctx->attnum);+ nulls[2] = (ctx->attnum < 0);+ values[3] = CStringGetTextDatum(msg);++ /*+ * In principle, there is nothing to prevent a scan over a large, highly+ * corrupted table from using work_mem worth of memory building up the+ * tuplestore. That's ok, but if we also leak the msg argument memory+ * until the end of the query, we could exceed work_mem by more than a+ * trivial amount. Therefore, free the msg argument each time we are+ * called rather than waiting for our current memory context to be freed.+ */+ pfree(msg);++ tuple = heap_form_tuple(ctx->tupdesc, values, nulls);+ tuplestore_puttuple(ctx->tupstore, tuple);+ ctx->is_corrupt = true;+}++/*+ * Construct the TupleDesc used to report messages about corruptions found+ * while scanning the heap.+ */+static TupleDesc+verify_heapam_tupdesc(void)+{+ TupleDesc tupdesc;+ AttrNumber a = 0;++ tupdesc = CreateTemplateTupleDesc(HEAPCHECK_RELATION_COLS);+ TupleDescInitEntry(tupdesc, ++a, "blkno", INT8OID, -1, 0);+ TupleDescInitEntry(tupdesc, ++a, "offnum", INT4OID, -1, 0);+ TupleDescInitEntry(tupdesc, ++a, "attnum", INT4OID, -1, 0);+ TupleDescInitEntry(tupdesc, ++a, "msg", TEXTOID, -1, 0);+ Assert(a == HEAPCHECK_RELATION_COLS);++ return BlessTupleDesc(tupdesc);+}++/*+ * Return whether a transaction ID is in the cached valid range.+ */+static inline bool+XidInValidRange(TransactionId xid, HeapCheckContext *ctx)+{+ return (TransactionIdPrecedesOrEquals(ctx->oldest_xid, xid) &&+ TransactionIdPrecedesOrEquals(ctx->relfrozenxid, xid) &&+ TransactionIdPrecedes(xid, ctx->next_xid));+}++/*+ * Return wehter a multitransaction ID is in the cached valid range.+ */+static inline bool+MxidInValidRange(MultiXactId mxid, HeapCheckContext *ctx)+{+ return (MultiXactIdPrecedesOrEquals(ctx->relminmxid, mxid) &&+ MultiXactIdPrecedesOrEquals(ctx->oldest_mxact, mxid) &&+ MultiXactIdPrecedes(mxid, ctx->next_mxact));+}++/*+ * Return whether the given transaction ID is (or was recently) valid to appear+ * in the heap being checked.+ *+ * We cache the range of valid transaction IDs. If xid is in that range, we+ * conclude that it is valid, even though concurrent changes to the table might+ * invalidate it under certain corrupt conditions. (For example, if the+ * table contains corrupt all frozen bits, a concurrent vacuum might skip the+ * page(s) containing the xid and then truncate clog and advance the+ * relfrozenxid beyond xid.) Reporting the xid as valid under such conditions+ * seems acceptable, since if we had checked it earlier in our scan it would+ * have truly been valid at that time, and we break no MVCC guarantees by+ * failing to notice the concurrent change in its status.+ */+static bool+xid_valid_in_rel(TransactionId xid, HeapCheckContext *ctx)+{+ /* Quick return for special xids */+ switch (xid)+ {+ case InvalidTransactionId:+ return false;+ case BootstrapTransactionId:+ case FrozenTransactionId:+ return true;+ }++ /* Quick return for xids within cached range */+ if (XidInValidRange(xid, ctx))+ return true;++ /* The latest valid xid may have advanced. Recheck. */+ update_cached_xid_range(ctx);+ if (XidInValidRange(xid, ctx))+ return true;++ /* No good. This xid is invalid. */+ return false;+}++/*+ * Returns whether the given mxid is valid to appear in the heap being+ * checked.+ *+ * This function attempts to return quickly by caching the known valid mxid+ * range in ctx. Callers should already have performed the initial setup of+ * the cache prior to the first call to this function.+ */+static bool+mxid_valid_in_rel(MultiXactId mxid, HeapCheckContext *ctx)+{+ if (MxidInValidRange(mxid, ctx))+ return true;++ /* The range may have advanced. Recheck. */+ update_cached_mxid_range(ctx);+ if (MxidInValidRange(mxid, ctx))+ return true;++ return false;+}++/*+ * Check for tuple header corruption and tuple visibility.+ *+ * Since we do not hold a snapshot, tuple visibility is not a question of+ * whether we should be able to see the tuple relative to any particular+ * snapshot, but rather a question of whether it is safe and reasonable to+ * to check the tuple attributes.+ *+ * Some kinds of tuple header corruption make it unsafe to check the tuple+ * attributes, for example when the tuple is foreshortened and such checks+ * would read beyond the end of the line pointer (and perhaps the page). In+ * such cases, we return false (not visible) after recording appropriate+ * corruption messages.+ *+ * Some other kinds of tuple header corruption confuse the question of where+ * the tuple attributes begin, or how long the nulls bitmap is, etc., making it+ * unreasonable to attempt to check attributes, even if all candidate answers+ * to those questions would not result in reading past the end of the line+ * pointer or page. In such cases, like above, we record corruption messages+ * about the header and then return false.+ *+ * Other kinds of tuple header corruption do not bare on the question of+ * whether the tuple attributes can be checked, so we record corruption+ * messages for them but do not base our visibility determination on them. (In+ * other words, we do not return false merely because we detected them.)+ *+ * For visibility determination not specifically related to corruption, what we+ * want to know is if a tuple is potentially visible to any running+ * transaction. If you are tempted to replace this function's visibility logic+ * with a call to another visibility checking function, keep in mind that this+ * function does not update hint bits, as it seems imprudent to write hint bits+ * (or anything at all) to a table during a corruption check. Nor does this+ * function bother classifying tuple visibility beyond a boolean visible vs.+ * not visible.+ *+ * The caller should already have checked that xmin and xmax are not out of+ * bounds for the relation.+ *+ * Returns whether the tuple is both visible and sufficiently sensible to+ * undergo attribute checks.+ */+static bool+check_tuple_header_and_visibilty(HeapTupleHeader tuphdr, HeapCheckContext *ctx)+{+ uint16 infomask = tuphdr->t_infomask;+ bool header_garbled = false;+ unsigned expected_hoff;;++ if (ctx->tuphdr->t_hoff > ctx->lp_len)+ {+ report_corruption(ctx,+ /*------+ translator: First %u is the offset, second %u is the+ total length. */+ psprintf(_("data begins at offset beyond the tuple length: %u vs. %u"),+ ctx->tuphdr->t_hoff, ctx->lp_len));+ header_garbled = true;+ }+ if ((ctx->tuphdr->t_infomask & HEAP_XMAX_LOCK_ONLY) &&+ (ctx->tuphdr->t_infomask2 & HEAP_KEYS_UPDATED))+ {+ report_corruption(ctx,+ pstrdup(_("updating transaction ID marked incompatibly as keys updated and locked only")));+ header_garbled = true;+ }++ if ((ctx->tuphdr->t_infomask & HEAP_XMAX_COMMITTED) &&+ (ctx->tuphdr->t_infomask & HEAP_XMAX_IS_MULTI))+ {+ report_corruption(ctx,+ pstrdup(_("updating transaction ID marked incompatibly as committed and as a multitransaction ID")));++ /*+ * This condition is clearly wrong, but we do not consider the header+ * garbled, because we don't rely on this property for determining if+ * the tuple is visible or for interpreting other relevant header+ * fields.+ */+ }++ if (infomask & HEAP_HASNULL)+ expected_hoff = MAXALIGN(SizeofHeapTupleHeader + BITMAPLEN(ctx->natts));+ else+ expected_hoff = MAXALIGN(SizeofHeapTupleHeader);+ if (ctx->tuphdr->t_hoff != expected_hoff)+ {+ if ((infomask & HEAP_HASNULL) && ctx->natts == 1)+ report_corruption(ctx,+ /* translator: Both %u represent an offset. */+ psprintf(_("data offset differs from expected: %u vs. %u (1 attribute, has nulls)"),+ ctx->tuphdr->t_hoff, expected_hoff));+ else if ((infomask & HEAP_HASNULL))+ report_corruption(ctx,+ /*------+ translator: First and second %u represent offsets,+ third %u represents the number of attributes. */+ psprintf(_("data offset differs from expected: %u vs. %u (%u attributes, has nulls)"),+ ctx->tuphdr->t_hoff, expected_hoff, ctx->natts));+ else if (ctx->natts == 1)+ report_corruption(ctx,+ /* translator: Both %u represent offsets. */+ psprintf(_("data offset differs from expected: %u vs. %u (1 attribute, no nulls)"),+ ctx->tuphdr->t_hoff, expected_hoff));+ else+ report_corruption(ctx,+ /*------+ translator: First and second %u represent an+ offset, third %u represents the number of+ attributes. */+ psprintf(_("data offset differs from expected: %u vs. %u (%u attributes, no nulls)"),+ ctx->tuphdr->t_hoff, expected_hoff, ctx->natts));+ header_garbled = true;+ }++ if (header_garbled)+ return false; /* checking of this tuple should not continue */++ /*+ * Ok, we can examine the header for tuple visibility purposes, though we+ * still need to be careful about a few remaining types of header+ * corruption. This logic roughly follows that of+ * HeapTupleSatisfiesVacuum. Where possible the comments indicate which+ * HTSV_Result we think that function might return for this tuple.+ */+ if (!HeapTupleHeaderXminCommitted(tuphdr))+ {+ TransactionId raw_xmin = HeapTupleHeaderGetRawXmin(tuphdr);++ if (HeapTupleHeaderXminInvalid(tuphdr))+ return false; /* HEAPTUPLE_DEAD */+ /* Used by pre-9.0 binary upgrades */+ else if (infomask & HEAP_MOVED_OFF ||+ infomask & HEAP_MOVED_IN)+ {+ XidCommitStatus xvac_status;+ TransactionId xvac = HeapTupleHeaderGetXvac(tuphdr);++ if (!TransactionIdIsValid(xvac))+ {+ report_corruption(ctx,+ pstrdup(_("old-style VACUUM FULL transaction ID is invalid")));+ return false; /* corrupt */+ }++ xvac_status = get_xid_status(xvac, ctx);+ if (xvac_status == XID_IN_PROGRESS)+ return true; /* HEAPTUPLE_DELETE_IN_PROGRESS */+ if (xvac_status == XID_TOO_NEW)+ {+ report_corruption(ctx,+ /*------+ translator: %u is a transaction identifier. */+ psprintf(_("old-style VACUUM FULL transaction ID is in the future: %u"),+ xvac));+ return false; /* corrupt */+ }+ if (xvac_status == XID_TOO_OLD)+ {+ report_corruption(ctx,+ /*------+ translator: %u is a transaction identifier. */+ psprintf(_("old-style VACUUM FULL transaction ID precedes freeze threshold: %u"),+ xvac));+ }+ if (!xid_valid_in_rel(xvac, ctx))+ {+ report_corruption(ctx,+ /*------+ translator: %u is a transaction identifier. */+ psprintf(_("old-style VACUUM FULL transaction ID is invalid in this relation: %u"),+ xvac));+ return false; /* corrupt */+ }+ if (xvac_status == XID_COMMITTED)+ return false; /* HEAPTUPLE_DEAD */+ }+ else+ {+ XidCommitStatus raw_xmin_status;++ if (!TransactionIdIsValid(raw_xmin))+ {+ report_corruption(ctx,+ pstrdup(_("inserting transaction ID is invalid")));+ return false;+ }++ raw_xmin_status = get_xid_status(raw_xmin, ctx);+ if (raw_xmin_status == XID_IN_PROGRESS)+ return true; /* insert or delete in progress */+ if (raw_xmin_status != XID_COMMITTED)+ return false; /* HEAPTUPLE_DEAD */+ if (raw_xmin_status == XID_TOO_NEW)+ {+ report_corruption(ctx,+ /*------+ translator: %u is a transaction identifier. */+ psprintf(_("inserting transaction ID is in the future: %u"),+ raw_xmin));+ return false; /* corrupt */+ }+ if (raw_xmin_status == XID_TOO_OLD)+ {+ report_corruption(ctx,+ /*------+ translator: %u is a transaction identifier. */+ psprintf(_("inserting transaction ID precedes freeze threshold: %u"),+ raw_xmin));+ return false; /* corrupt */+ }+ }+ }++ if (!(infomask & HEAP_XMAX_INVALID) && !HEAP_XMAX_IS_LOCKED_ONLY(infomask))+ {+ if (infomask & HEAP_XMAX_IS_MULTI)+ {+ XidCommitStatus xmax_status;+ TransactionId xmax = HeapTupleGetUpdateXid(tuphdr);++ /* not LOCKED_ONLY, so it has to have an xmax */+ if (!TransactionIdIsValid(xmax))+ {+ report_corruption(ctx,+ pstrdup(_("updating transaction ID is invalid")));+ return false; /* corrupt */+ }++ xmax_status = get_xid_status(xmax, ctx);+ if (xmax_status == XID_IN_PROGRESS)+ return true; /* HEAPTUPLE_DELETE_IN_PROGRESS */+ if (xmax_status == XID_COMMITTED)+ return false; /* HEAPTUPLE_RECENTLY_DEAD or HEAPTUPLE_DEAD */+ if (xmax_status == XID_TOO_NEW)+ {+ report_corruption(ctx,+ /*------+ translator: %u is a transaction identifier. */+ psprintf(_("updating transaction ID is in the future: %u"),+ xmax));+ return false; /* corrupt */+ }+ if (xmax_status == XID_TOO_OLD)+ {+ report_corruption(ctx,+ /*------+ translator: %u is a transaction identifier. */+ psprintf(_("updating transaction ID precedes freeze threshold: %u"),+ xmax));+ return false; /* corrupt */+ }++ /* Ok, the tuple is live */+ }+ else if (!(infomask & HEAP_XMAX_COMMITTED))+ return true; /* HEAPTUPLE_DELETE_IN_PROGRESS or+ * HEAPTUPLE_LIVE */+ else+ return false; /* HEAPTUPLE_RECENTLY_DEAD or HEAPTUPLE_DEAD */+ }+ return true; /* not dead */+}++/*+ * Check the current toast tuple against the state tracked in ctx, recording+ * any corruption found in ctx->tupstore.+ *+ * This is not equivalent to running verify_heapam on the toast table itself,+ * and is not hardened against corruption of the toast table. Rather, when+ * validating a toasted attribute in the main table, the sequence of toast+ * tuples that store the toasted value are retrieved and checked in order, with+ * each toast tuple being checked against where we are in the sequence, as well+ * as each toast tuple having its varlena structure sanity checked.+ */+static void+check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx)+{+ int32 curchunk;+ Pointer chunk;+ bool isnull;+ char *chunkdata;+ int32 chunksize;+ int32 expected_size;++ /*+ * Have a chunk, extract the sequence number and the data+ */+ curchunk = DatumGetInt32(fastgetattr(toasttup, 2,+ ctx->toast_rel->rd_att, &isnull));+ if (isnull)+ {+ report_corruption(ctx,+ pstrdup(_("toast chunk sequence number is null")));+ return;+ }+ chunk = DatumGetPointer(fastgetattr(toasttup, 3,+ ctx->toast_rel->rd_att, &isnull));+ if (isnull)+ {+ report_corruption(ctx,+ pstrdup(_("toast chunk data is null")));+ return;+ }+ if (!VARATT_IS_EXTENDED(chunk))+ {+ chunksize = VARSIZE(chunk) - VARHDRSZ;+ chunkdata = VARDATA(chunk);+ }+ else if (VARATT_IS_SHORT(chunk))+ {+ /*+ * could happen due to heap_form_tuple doing its thing+ */+ chunksize = VARSIZE_SHORT(chunk) - VARHDRSZ_SHORT;+ chunkdata = VARDATA_SHORT(chunk);+ }+ else+ {+ /* should never happen */+ uint32 header = ((varattrib_4b *) chunk)->va_4byte.va_header;++ report_corruption(ctx,+ /*------+ translator: %0x represents a bit pattern in+ hexadecimal, %d represents the sequence number. */+ psprintf(_("corrupt extended toast chunk has invalid varlena header: %0x (sequence number %d)"),+ header, curchunk));+ return;+ }++ /*+ * Some checks on the data we've found+ */+ if (curchunk != ctx->chunkno)+ {+ report_corruption(ctx,+ /* translator: Both %u represent sequence numbers. */+ psprintf(_("toast chunk sequence number not the expected sequence number: %u vs. %u"),+ curchunk, ctx->chunkno));+ return;+ }+ if (curchunk > ctx->endchunk)+ {+ report_corruption(ctx,+ /* translator: Both %u represent sequence numbers. */+ psprintf(_("toast chunk sequence number exceeds the end chunk sequence number: %u vs. %u"),+ curchunk, ctx->endchunk));+ return;+ }++ expected_size = curchunk < ctx->totalchunks - 1 ? TOAST_MAX_CHUNK_SIZE+ : ctx->attrsize - ((ctx->totalchunks - 1) * TOAST_MAX_CHUNK_SIZE);+ if (chunksize != expected_size)+ {+ report_corruption(ctx,+ /* translator: Both %u represent a chunk size. */+ psprintf(_("toast chunk size differs from expected size: %u vs. %u"),+ chunksize, expected_size));+ return;+ }+}++/*+ * Check the current attribute as tracked in ctx, recording any corruption+ * found in ctx->tupstore.+ *+ * This function follows the logic performed by heap_deform_tuple(), and in the+ * case of a toasted value, optionally continues along the logic of+ * detoast_external_attr(), checking for any conditions that would result in+ * either of those functions Asserting or crashing the backend. The checks+ * performed by Asserts present in those two functions are also performed here.+ * In cases where those two functions are a bit cavalier in their assumptions+ * about data being correct, we perform additional checks not present in either+ * of those two functions. Where some condition is checked in both of those+ * functions, we perform it here twice, as we parallel the logical flow of+ * those two functions. The presence of duplicate checks seems a reasonable+ * price to pay for keeping this code tightly coupled with the code it+ * protects.+ *+ * Returns true if the tuple attribute is sane enough for processing to+ * continue on to the next attribute, false otherwise.+ */+static bool+check_tuple_attribute(HeapCheckContext *ctx)+{+ struct varatt_external toast_pointer;+ ScanKeyData toastkey;+ SysScanDesc toastscan;+ SnapshotData SnapshotToast;+ HeapTuple toasttup;+ bool found_toasttup;+ Datum attdatum;+ struct varlena *attr;+ char *tp; /* pointer to the tuple data */+ uint16 infomask;+ Form_pg_attribute thisatt;++ infomask = ctx->tuphdr->t_infomask;+ thisatt = TupleDescAttr(RelationGetDescr(ctx->rel), ctx->attnum);++ tp = (char *) ctx->tuphdr + ctx->tuphdr->t_hoff;++ if (ctx->tuphdr->t_hoff + ctx->offset > ctx->lp_len)+ {+ report_corruption(ctx,+ /*------+ translator: First %u represents an offset, second and+ third %u represents a length. */+ psprintf(_("attribute starts at offset beyond total tuple length: %u vs. %u (attribute length %u)"),+ ctx->tuphdr->t_hoff + ctx->offset, ctx->lp_len,+ thisatt->attlen));+ return false;+ }++ /* Skip null values */+ if (infomask & HEAP_HASNULL && att_isnull(ctx->attnum, ctx->tuphdr->t_bits))+ return true;++ /* Skip non-varlena values, but update offset first */+ if (thisatt->attlen != -1)+ {+ ctx->offset = att_align_nominal(ctx->offset, thisatt->attalign);+ ctx->offset = att_addlength_pointer(ctx->offset, thisatt->attlen,+ tp + ctx->offset);+ if (ctx->tuphdr->t_hoff + ctx->offset > ctx->lp_len)+ {+ report_corruption(ctx,+ /*------+ translator: First %u represents an offset, second+ and third %u represents a length. */+ psprintf(_("attribute ends at offset beyond total tuple length: %u vs. %u (attribute length %u)"),+ ctx->tuphdr->t_hoff + ctx->offset,+ ctx->lp_len, thisatt->attlen));+ return false;+ }+ return true;+ }++ /* Ok, we're looking at a varlena attribute. */+ ctx->offset = att_align_pointer(ctx->offset, thisatt->attalign, -1,+ tp + ctx->offset);++ /* Get the (possibly corrupt) varlena datum */+ attdatum = fetchatt(thisatt, tp + ctx->offset);++ /*+ * We have the datum, but we cannot decode it carelessly, as it may still+ * be corrupt.+ */++ /*+ * Check that VARTAG_SIZE won't hit a TrapMacro on a corrupt va_tag before+ * risking a call into att_addlength_pointer+ */+ if (VARATT_IS_EXTERNAL(tp + ctx->offset))+ {+ uint8 va_tag = VARTAG_EXTERNAL(tp + ctx->offset);++ if (va_tag != VARTAG_ONDISK)+ {+ report_corruption(ctx,+ /*------+ translator: %u represents an enumeration value. */+ psprintf(_("toasted attribute has unexpected TOAST tag: %u"),+ va_tag));+ /* We can't know where the next attribute begins */+ return false;+ }+ }++ /* Ok, should be safe now */+ ctx->offset = att_addlength_pointer(ctx->offset, thisatt->attlen,+ tp + ctx->offset);++ if (ctx->tuphdr->t_hoff + ctx->offset > ctx->lp_len)+ {+ report_corruption(ctx,+ /*------+ translator: First %u represents an offset, second and+ third %u represents a length. */+ psprintf(_("attribute ends at offset beyond total tuple length: %u vs. %u (attribute length %u)"),+ ctx->tuphdr->t_hoff + ctx->offset,+ ctx->lp_len, thisatt->attlen));++ return false;+ }++ /*+ * heap_deform_tuple would be done with this attribute at this point,+ * having stored it in values[], and would continue to the next attribute.+ * We go further, because we need to check if the toast datum is corrupt.+ */++ attr = (struct varlena *) DatumGetPointer(attdatum);++ /*+ * Now we follow the logic of detoast_external_attr(), with the same+ * caveats about being paranoid about corruption.+ */++ /* Skip values that are not external */+ if (!VARATT_IS_EXTERNAL(attr))+ return true;++ /* It is external, and we're looking at a page on disk */++ /* The tuple header better claim to contain toasted values */+ if (!(infomask & HEAP_HASEXTERNAL))+ {+ report_corruption(ctx,+ pstrdup(_("attribute is external but tuple header flag HEAP_HASEXTERNAL not set")));+ return true;+ }++ /* The relation better have a toast table */+ if (!ctx->rel->rd_rel->reltoastrelid)+ {+ report_corruption(ctx,+ pstrdup(_("attribute is external but relation has no toast relation")));+ return true;+ }++ /* If we were told to skip toast checking, then we're done. */+ if (ctx->toast_rel == NULL)+ return true;++ /*+ * Must copy attr into toast_pointer for alignment considerations+ */+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);++ ctx->attrsize = toast_pointer.va_extsize;+ ctx->endchunk = (ctx->attrsize - 1) / TOAST_MAX_CHUNK_SIZE;+ ctx->totalchunks = ctx->endchunk + 1;++ /*+ * Setup a scan key to find chunks in toast table with matching va_valueid+ */+ ScanKeyInit(&toastkey,+ (AttrNumber) 1,+ BTEqualStrategyNumber, F_OIDEQ,+ ObjectIdGetDatum(toast_pointer.va_valueid));++ /*+ * Check if any chunks for this toasted object exist in the toast table,+ * accessible via the index.+ */+ init_toast_snapshot(&SnapshotToast);+ toastscan = systable_beginscan_ordered(ctx->toast_rel,+ ctx->valid_toast_index,+ &SnapshotToast, 1,+ &toastkey);+ ctx->chunkno = 0;+ found_toasttup = false;+ while ((toasttup =+ systable_getnext_ordered(toastscan,+ ForwardScanDirection)) != NULL)+ {+ found_toasttup = true;+ check_toast_tuple(toasttup, ctx);+ ctx->chunkno++;+ }+ if (ctx->chunkno != (ctx->endchunk + 1))+ report_corruption(ctx,+ /* translator: Both %u represent a chunk number. */+ psprintf(_("final toast chunk number differs from expected value: %u vs. %u"),+ ctx->chunkno, (ctx->endchunk + 1)));+ if (!found_toasttup)+ report_corruption(ctx,+ pstrdup(_("toasted value missing from toast table")));+ systable_endscan_ordered(toastscan);++ return true;+}++/*+ * Check the current tuple as tracked in ctx, recording any corruption found in+ * ctx->tupstore.+ */+static void+check_tuple(HeapCheckContext *ctx)+{+ TransactionId xmin;+ TransactionId xmax;+ bool fatal = false;+ uint16 infomask = ctx->tuphdr->t_infomask;++ /*+ * If we report corruption before iterating over individual attributes, we+ * need attnum to be reported as NULL. Set that up before any corruption+ * reporting might happen.+ */+ ctx->attnum = -1;++ /*+ * If the line pointer for this tuple does not reserve enough space for a+ * complete tuple header, we dare not read the tuple header.+ */+ if (ctx->lp_len < MAXALIGN(SizeofHeapTupleHeader))+ {+ report_corruption(ctx,+ /* translator: Both %u represent a size. */+ psprintf(_("line pointer length is less than the minimum tuple header size: %u vs. %u"),+ ctx->lp_len, (uint32) MAXALIGN(SizeofHeapTupleHeader)));+ return;+ }++ /* If xmin is normal, it should be within valid range */+ xmin = HeapTupleHeaderGetXmin(ctx->tuphdr);+ if (TransactionIdIsNormal(xmin))+ {+ if (TransactionIdIsNormal(ctx->relfrozenxid) &&+ TransactionIdPrecedes(xmin, ctx->relfrozenxid))+ {+ report_corruption(ctx,+ /* translator: Both %u are transaction IDs. */+ psprintf(_("inserting transaction ID is from before freeze cutoff: %u vs. %u"),+ xmin, ctx->relfrozenxid));+ fatal = true;+ }+ else if (!xid_valid_in_rel(xmin, ctx))+ {+ report_corruption(ctx,+ /* translator: %u is a transaction ID. */+ psprintf(_("inserting transaction ID is in the future: %u"),+ xmin));+ fatal = true;+ }+ }++ /* If xmax is a multixact, it should be within valid range */+ xmax = HeapTupleHeaderGetRawXmax(ctx->tuphdr);+ if ((infomask & HEAP_XMAX_IS_MULTI) && !mxid_valid_in_rel(xmax, ctx))+ {+ if (MultiXactIdPrecedes(xmax, ctx->relminmxid))+ {+ report_corruption(ctx,+ /* translator: Both %u are multitransaction IDs. */+ psprintf(_("multitransaction ID is from before relation cutoff: %u vs. %u"),+ xmax, ctx->relminmxid));+ fatal = true;+ }+ else if (MultiXactIdPrecedes(xmax, ctx->oldest_mxact))+ {+ report_corruption(ctx,+ /* translator: Both %u are multitransaction IDs. */+ psprintf(_("multitransaction ID is from before cutoff: %u vs. %u"),+ xmax, ctx->oldest_mxact));+ fatal = true;+ }+ else if (MultiXactIdPrecedesOrEquals(ctx->next_mxact, xmax))+ {+ report_corruption(ctx,+ /* translator: %u is a multitransaction ID. */+ psprintf(_("multitransaction ID is in the future: %u"),+ xmax));+ fatal = true;+ }+ }++ /* If xmax is normal, it should be within valid range */+ if (TransactionIdIsNormal(xmax))+ {+ if (TransactionIdIsNormal(ctx->relfrozenxid) &&+ TransactionIdPrecedes(xmax, ctx->relfrozenxid))+ {+ report_corruption(ctx,+ /* translator: Both %u are transaction IDs. */+ psprintf(_("updating transaction ID is from before relation cutoff: %u vs. %u"),+ xmax, ctx->relfrozenxid));+ fatal = true;+ }+ else if (!xid_valid_in_rel(xmax, ctx))+ {+ report_corruption(ctx,+ /* translator: %u is a transaction ID. */+ psprintf(_("updating transaction ID is in the future: %u"),+ xmax));+ fatal = true;+ }+ }++ /*+ * Cannot process tuple data if tuple header was corrupt, as the offsets+ * within the page cannot be trusted, leaving too much risk of reading+ * garbage if we continue.+ *+ * We also cannot process the tuple if the xmin or xmax were invalid+ * relative to relfrozenxid or relminmxid, as clog entries for the xids+ * may already be gone.+ */+ if (fatal)+ return;++ /*+ * Check various forms of tuple header corruption. If the header is too+ * corrupt to continue checking, or if the tuple is not visible to anyone,+ * we cannot continue with other checks.+ */+ if (!check_tuple_header_and_visibilty(ctx->tuphdr, ctx))+ return;++ /*+ * The tuple is visible, so it must be compatible with the current version+ * of the relation descriptor. It might have fewer columns than are+ * present in the relation descriptor, but it cannot have more.+ */+ if (RelationGetDescr(ctx->rel)->natts < ctx->natts)+ {+ report_corruption(ctx,+ /* translator: Both %u are a number. */+ psprintf(_("number of attributes exceeds maximum expected for table: %u vs. %u"),+ ctx->natts,+ RelationGetDescr(ctx->rel)->natts));+ return;+ }++ /*+ * Check each attribute unless we hit corruption that confuses what to do+ * next, at which point we abort further attribute checks for this tuple.+ * Note that we don't abort for all types of corruption, only for those+ * types where we don't know how to continue.+ */+ ctx->offset = 0;+ for (ctx->attnum = 0; ctx->attnum < ctx->natts; ctx->attnum++)+ if (!check_tuple_attribute(ctx))+ break; /* cannot continue */+}++/*+ * Convert a TransactionId into a FullTransactionId using our cached values of+ * the valid transaction ID range. It is the caller's responsibility to have+ * already updated the cached values, if necessary.+ */+static FullTransactionId+FullTransactionIdFromXidAndCtx(TransactionId xid, const HeapCheckContext *ctx)+{+ uint32 epoch;++ if (!TransactionIdIsNormal(xid))+ return FullTransactionIdFromEpochAndXid(0, xid);+ epoch = EpochFromFullTransactionId(ctx->next_fxid);+ if (xid > ctx->next_xid)+ epoch--;+ return FullTransactionIdFromEpochAndXid(epoch, xid);+}++/*+ * Update our cached range of valid transaction IDs.+ */+static void+update_cached_xid_range(HeapCheckContext *ctx)+{+ /* Make cached copies */+ LWLockAcquire(XidGenLock, LW_SHARED);+ ctx->next_fxid = ShmemVariableCache->nextXid;+ ctx->oldest_xid = ShmemVariableCache->oldestXid;+ LWLockRelease(XidGenLock);++ /* And compute alternate versions of the same */+ ctx->oldest_fxid = FullTransactionIdFromXidAndCtx(ctx->oldest_xid, ctx);+ ctx->next_xid = XidFromFullTransactionId(ctx->next_fxid);+}++/*+ * Update our cached range of valid multitransaction IDs.+ */+static void+update_cached_mxid_range(HeapCheckContext *ctx)+{+ ReadMultiXactIdRange(&ctx->oldest_mxact, &ctx->next_mxact);+}++/*+ * Return the commit status for a TransactionId. The cached range of+ * valid transaction IDs may be updated as a side effect.+ */+static XidCommitStatus+get_xid_status(TransactionId xid, HeapCheckContext *ctx)+{+ XidCommitStatus result;+ FullTransactionId fxid;+ FullTransactionId clog_horizon;++ Assert(TransactionIdIsValid(xid));++ /* If we just checked this xid, return the cached status */+ if (xid == ctx->cached_xid)+ return ctx->cached_status;++ /* Check if the xid is within bounds */+ fxid = FullTransactionIdFromXidAndCtx(xid, ctx);+ if (!fxid_in_cached_range(fxid, ctx))+ {+ /*+ * We may have been checking against stale values. Update the cached+ * range to be sure, and since we relied on the cached range when we+ * performed the full xid conversion, reconvert.+ */+ update_cached_xid_range(ctx);+ fxid = FullTransactionIdFromXidAndCtx(xid, ctx);++ if (FullTransactionIdPrecedesOrEquals(ctx->next_fxid, fxid))+ {+ ctx->cached_xid = xid;+ ctx->cached_status = XID_TOO_NEW;+ return XID_TOO_NEW;+ }+ if (FullTransactionIdPrecedes(fxid, ctx->oldest_fxid) ||+ FullTransactionIdPrecedes(fxid, ctx->relfrozenfxid))+ {+ ctx->cached_xid = xid;+ ctx->cached_status = XID_TOO_OLD;+ return XID_TOO_OLD;+ }+ }++ result = XID_COMMITTED;+ LWLockAcquire(XactTruncationLock, LW_SHARED);+ clog_horizon = FullTransactionIdFromXidAndCtx(ShmemVariableCache->oldestClogXid, ctx);+ if (FullTransactionIdPrecedesOrEquals(clog_horizon, fxid))+ {+ if (TransactionIdIsCurrentTransactionId(xid))+ result = XID_IN_PROGRESS;+ else if (TransactionIdDidCommit(xid))+ result = XID_COMMITTED;+ else if (TransactionIdDidAbort(xid))+ result = XID_ABORTED;+ else+ result = XID_IN_PROGRESS;+ }+ LWLockRelease(XactTruncationLock);+ ctx->cached_xid = xid;+ ctx->cached_status = result;+ return result;+}diff --git a/doc/src/sgml/amcheck.sgml b/doc/src/sgml/amcheck.sgmlindex a9df2c1a9d..159b1e63bf 100644--- a/doc/src/sgml/amcheck.sgml+++ b/doc/src/sgml/amcheck.sgml@@ -187,6 +187,221 @@ SET client_min_messages = DEBUG1;
</para>
</tip>
+ <variablelist>+ <varlistentry>+ <term>+ <function>+ verify_heapam(relation regclass,+ on_error_stop boolean,+ check_toast boolean,+ skip cstring,+ startblock bigint,+ endblock bigint,+ blkno OUT bigint,+ offnum OUT integer,+ attnum OUT integer,+ msg OUT text)+ returns record+ </function>+ </term>+ <listitem>+ <para>+ Checks a table for structural corruption, where pages in the relation+ contain data that is invalidly formatted, and for logical corruption,+ where pages are structurally valid but inconsistent with the rest of the+ database cluster. Example usage:+<screen>+test=# select * from verify_heapam('mytable', check_toast := true);+ blkno | offnum | attnum | msg+-------+--------+--------+---------------------------------------------------------------------------------------------------+ 0 | 1 | | inserting transaction ID is from before freeze cutoff: 3 vs. 524+ 0 | 2 | | inserting transaction ID is from before freeze cutoff: 4026531839 vs. 524+ 0 | 3 | | updating transaction ID is from before relation cutoff: 4026531839 vs. 524+ 0 | 4 | | data begins at offset beyond the tuple length: 152 vs. 58+ 0 | 4 | | data offset differs from expected: 152 vs. 24 (3 attributes, no nulls)+ 0 | 5 | | data offset differs from expected: 27 vs. 24 (3 attributes, no nulls)+ 0 | 6 | | data offset differs from expected: 16 vs. 24 (3 attributes, no nulls)+ 0 | 7 | | data offset differs from expected: 21 vs. 24 (3 attributes, no nulls)+ 0 | 8 | | number of attributes exceeds maximum expected for table: 2047 vs. 3+ 0 | 9 | | data offset differs from expected: 24 vs. 280 (2047 attributes, has nulls)+ 0 | 10 | | number of attributes exceeds maximum expected for table: 67 vs. 3+ 0 | 11 | 1 | attribute ends at offset beyond total tuple length: 416848000 vs. 58 (attribute length 4294967295)+ 0 | 12 | 2 | final toast chunk number differs from expected value: 0 vs. 6+ 0 | 12 | 2 | toasted value missing from toast table+ 0 | 13 | | updating transaction ID marked incompatibly as keys updated and locked only+ 0 | 14 | | multitransaction ID is from before relation cutoff: 0 vs. 1+++(16 rows)+</screen>+ As this example shows, the Tuple ID (TID) of the corrupt tuple is given+ in the (<literal>blkno</literal>, <literal>offnum</literal>) columns, and+ for corruptions specific to a particular attribute in the tuple, the+ <literal>attnum</literal> field shows which one.+ </para>+ <para>+ Structural corruption can happen due to faulty storage hardware, or+ relation files being overwritten or modified by unrelated software.+ This kind of corruption can also be detected with+ <link linkend="app-initdb-data-checksums"><application>data page+ checksums</application></link>.+ </para>+ <para>+ Relation pages which are correctly formatted, internally consistent, and+ correct relative to their own internal checksums may still contain+ logical corruption. As such, this kind of corruption cannot be detected+ with <application>checksums</application>. Examples include toasted+ values in the main table which lack a corresponding entry in the toast+ table, and tuples in the main table with a Transaction ID that is older+ than the oldest valid Transaction ID in the database or cluster.+ </para>+ <para>+ Multiple causes of logical corruption have been observed in production+ systems, including bugs in the <productname>PostgreSQL</productname>+ server software, faulty and ill-conceived backup and restore tools, and+ user error.+ </para>+ <para>+ Corrupt relations are most concerning in live production environments,+ precisely the same environments where high risk activities are least+ welcome. For this reason, <function>verify_heapam</function> has been+ designed to diagnose corruption without undue risk. It cannot guard+ against all causes of backend crashes, as even executing the calling+ query could be unsafe on a badly corrupted system. Access to <link+ linkend="catalogs-overview">catalog tables</link> are performed and could+ be problematic if the catalogs themselves are corrupted.+ </para>+ <para>+ The design principle adhered to in <function>verify_heapam</function> is+ that, if the rest of the system and server hardware are correct, under+ default options, <function>verify_heapam</function> will not crash the+ server due merely to structural or logical corruption in the target+ table.+ </para>+ <para>+ An experimental option, <literal>check_toast</literal>, exists to+ reconcile the target table against entries in its corresponding toast+ table. This option may change in future, is disabled by default, and is+ known to be slow. It is also unsafe under some conditions. If the+ target relation's corresponding toast table or toast index are corrupt,+ reconciling the target table against toast values may be unsafe. If the+ catalogs, toast table and toast index are uncorrupted, and remain so+ during the check of the target table, reconciling the target table+ against its toast table should be safe.+ </para>+ <para>+ The following optional arguments are recognized:+ </para>+ <variablelist>+ <varlistentry>+ <term>on_error_stop</term>+ <listitem>+ <para>+ If true, corruption checking stops at the end of the first block on+ which any corruptions are found.+ </para>+ <para>+ Defaults to false.+ </para>+ </listitem>+ </varlistentry>+ <varlistentry>+ <term>check_toast</term>+ <listitem>+ <para>+ If this experimental option is true, toasted values are checked gainst+ the corresponding TOAST table.+ </para>+ <para>+ Defaults to false.+ </para>+ </listitem>+ </varlistentry>+ <varlistentry>+ <term>skip</term>+ <listitem>+ <para>+ If not <literal>none</literal>, corruption checking skips blocks that+ are marked as all-visible or all-frozen, as given.+ Valid options are <literal>all-visible</literal>,+ <literal>all-frozen</literal> and <literal>none</literal>.+ </para>+ <para>+ Defaults to <literal>none</literal>.+ </para>+ </listitem>+ </varlistentry>+ <varlistentry>+ <term>startblock</term>+ <listitem>+ <para>+ If specified, corruption checking begins at the specified block,+ skipping all previous blocks. It is an error to specify a+ <literal>startblock</literal> outside the range of blocks in the+ target table.+ </para>+ <para>+ By default, does not skip any blocks.+ </para>+ </listitem>+ </varlistentry>+ <varlistentry>+ <term>endblock</term>+ <listitem>+ <para>+ If specified, corruption checking ends at the specified block,+ skipping all remaining blocks. It is an error to specify an+ <literal>endblock</literal> outside the range of blocks in the target+ table.+ </para>+ <para>+ By default, does not skip any blocks.+ </para>+ </listitem>+ </varlistentry>+ </variablelist>+ <para>+ For each corruption detected, <function>verify_heapam</function> returns+ a row with the following columns:+ </para>+ <variablelist>+ <varlistentry>+ <term>blkno</term>+ <listitem>+ <para>+ The number of the block containing the corrupt page.+ </para>+ </listitem>+ </varlistentry>+ <varlistentry>+ <term>offnum</term>+ <listitem>+ <para>+ The OffsetNumber of the corrupt tuple.+ </para>+ </listitem>+ </varlistentry>+ <varlistentry>+ <term>attnum</term>+ <listitem>+ <para>+ The attribute number of the corrupt column in the tuple, if the+ corruption is specific to a column and not the tuple as a whole.+ </para>+ </listitem>+ </varlistentry>+ <varlistentry>+ <term>msg</term>+ <listitem>+ <para>+ A human readable message describing the corruption in the page.+ </para>+ </listitem>+ </varlistentry>+ </variablelist>+ </listitem>+ </varlistentry>+ </variablelist>
</sect2>
<sect2>
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.cindex aa3f14c019..ca357410a2 100644--- a/src/backend/access/heap/hio.c+++ b/src/backend/access/heap/hio.c@@ -47,6 +47,17 @@ RelationPutHeapTuple(Relation relation,
*/
Assert(!token || HeapTupleHeaderIsSpeculative(tuple->t_data));
+ /*+ * Do not allow tuples with invalid combinations of hint bits to be placed+ * on a page. These combinations are detected as corruption by the+ * contrib/amcheck logic, so if you disable one or both of these+ * assertions, make corresponding changes there.+ */+ Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_LOCK_ONLY) &&+ (tuple->t_data->t_infomask2 & HEAP_KEYS_UPDATED)));+ Assert(!((tuple->t_data->t_infomask & HEAP_XMAX_COMMITTED) &&+ (tuple->t_data->t_infomask & HEAP_XMAX_IS_MULTI)));+
/* Add the tuple to the page */
pageHeader = BufferGetPage(buffer);
diff --git a/src/backend/access/transam/multixact.c b/src/backend/access/transam/multixact.cindex b8bedca04a..5f0c622ad8 100644--- a/src/backend/access/transam/multixact.c+++ b/src/backend/access/transam/multixact.c@@ -735,6 +735,25 @@ ReadNextMultiXactId(void)
return mxid;
}
+/*+ * ReadMultiXactIdRange+ * Get the range of IDs that may still be referenced by a relation.+ */+void+ReadMultiXactIdRange(MultiXactId *oldest, MultiXactId *next)+{+ LWLockAcquire(MultiXactGenLock, LW_SHARED);+ *oldest = MultiXactState->oldestMultiXactId;+ *next = MultiXactState->nextMXact;+ LWLockRelease(MultiXactGenLock);++ if (*oldest < FirstMultiXactId)+ *oldest = FirstMultiXactId;+ if (*next < FirstMultiXactId)+ *next = FirstMultiXactId;+}++
/*
* MultiXactIdCreateFromMembers
* Make a new MultiXactId from the specified set of members
diff --git a/src/include/access/multixact.h b/src/include/access/multixact.hindex 6d729008c6..f67f52057c 100644--- a/src/include/access/multixact.h+++ b/src/include/access/multixact.h@@ -108,6 +108,7 @@ extern MultiXactId MultiXactIdCreateFromMembers(int nmembers,
MultiXactMember *members);
extern MultiXactId ReadNextMultiXactId(void);
+extern void ReadMultiXactIdRange(MultiXactId *oldest, MultiXactId *next);
extern bool MultiXactIdIsRunning(MultiXactId multi, bool isLockOnly);
extern void MultiXactIdSetOldestMember(void);
extern int GetMultiXactIdMembers(MultiXactId multi, MultiXactMember **xids,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.listindex 3d990463ce..c48d453793 100644--- a/src/tools/pgindent/typedefs.list+++ b/src/tools/pgindent/typedefs.list@@ -1019,6 +1019,7 @@ HbaToken
HeadlineJsonState
HeadlineParsedText
HeadlineWordEntry
+HeapCheckContext
HeapScanDesc
HeapTuple
HeapTupleData
@@ -2282,6 +2283,7 @@ SimpleStringList
SimpleStringListCell
SingleBoundSortItem
Size
+SkipPages
SlabBlock
SlabChunk
SlabContext
--
2.21.1 (Apple Git-122.3)
=
v14-0002-Adding-contrib-module-pg_amcheck.patch (application/octet-stream, 82.6K) download
From 6383bd9ec2bf0760b823cca5911e39be4aa043ea Mon Sep 17 00:00:00 2001
From: Mark Dilger <mark.dilger@enterprisedb.com>
Date: Wed, 19 Aug 2020 18:56:14 -0700
Subject: [PATCH v14 2/2] Adding contrib module pg_amcheck
Adding new contrib module pg_amcheck, which is a command line
interface for running amcheck's verifications against tables and
indexes.
---
contrib/Makefile | 1 +
contrib/pg_amcheck/.gitignore | 3 +
contrib/pg_amcheck/Makefile | 28 +
contrib/pg_amcheck/pg_amcheck.c | 1280 ++++++++++++++++++++
contrib/pg_amcheck/pg_amcheck.control | 5 +
contrib/pg_amcheck/t/001_basic.pl | 9 +
contrib/pg_amcheck/t/002_nonesuch.pl | 60 +
contrib/pg_amcheck/t/003_check.pl | 231 ++++
contrib/pg_amcheck/t/004_verify_heapam.pl | 426 +++++++
contrib/pg_amcheck/t/005_opclass_damage.pl | 52 +
doc/src/sgml/contrib.sgml | 1 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/pgamcheck.sgml | 229 ++++
src/tools/pgindent/typedefs.list | 2 +
14 files changed, 2328 insertions(+)
create mode 100644 contrib/pg_amcheck/.gitignore
create mode 100644 contrib/pg_amcheck/Makefile
create mode 100644 contrib/pg_amcheck/pg_amcheck.c
create mode 100644 contrib/pg_amcheck/pg_amcheck.control
create mode 100644 contrib/pg_amcheck/t/001_basic.pl
create mode 100644 contrib/pg_amcheck/t/002_nonesuch.pl
create mode 100644 contrib/pg_amcheck/t/003_check.pl
create mode 100644 contrib/pg_amcheck/t/004_verify_heapam.pl
create mode 100644 contrib/pg_amcheck/t/005_opclass_damage.pl
create mode 100644 doc/src/sgml/pgamcheck.sgml
diff --git a/contrib/Makefile b/contrib/Makefileindex 1846d415b6..ed5589d97b 100644--- a/contrib/Makefile+++ b/contrib/Makefile@@ -29,6 +29,7 @@ SUBDIRS = \
oid2name \
pageinspect \
passwordcheck \
+ pg_amcheck \
pg_buffercache \
pg_freespacemap \
pg_prewarm \
diff --git a/contrib/pg_amcheck/.gitignore b/contrib/pg_amcheck/.gitignorenew file mode 100644index 0000000000..07ad380105--- /dev/null+++ b/contrib/pg_amcheck/.gitignore@@ -0,0 +1,3 @@+/pg_amcheck++/tmp_check/diff --git a/contrib/pg_amcheck/Makefile b/contrib/pg_amcheck/Makefilenew file mode 100644index 0000000000..74554b9e8d--- /dev/null+++ b/contrib/pg_amcheck/Makefile@@ -0,0 +1,28 @@+# contrib/pg_amcheck/Makefile++PGFILEDESC = "pg_amcheck - detects corruption within database relations"+PGAPPICON = win32++PROGRAM = pg_amcheck+OBJS = \+ $(WIN32RES) \+ pg_amcheck.o++REGRESS_OPTS += --load-extension=amcheck --load-extension=pageinspect+EXTRA_INSTALL += contrib/amcheck contrib/pageinspect++TAP_TESTS = 1++PG_CPPFLAGS = -I$(libpq_srcdir)+PG_LIBS_INTERNAL = -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport)++ifdef USE_PGXS+PG_CONFIG = pg_config+PGXS := $(shell $(PG_CONFIG) --pgxs)+include $(PGXS)+else+subdir = contrib/pg_amcheck+top_builddir = ../..+include $(top_builddir)/src/Makefile.global+include $(top_srcdir)/contrib/contrib-global.mk+endifdiff --git a/contrib/pg_amcheck/pg_amcheck.c b/contrib/pg_amcheck/pg_amcheck.cnew file mode 100644index 0000000000..1e725112e3--- /dev/null+++ b/contrib/pg_amcheck/pg_amcheck.c@@ -0,0 +1,1280 @@+/*-------------------------------------------------------------------------+ *+ * pg_amcheck.c+ * Detects corruption within database relations.+ *+ * Copyright (c) 2017-2020, PostgreSQL Global Development Group+ *+ * IDENTIFICATION+ * contrib/pg_amcheck/pg_amcheck.c+ *+ *-------------------------------------------------------------------------+ */+#include "postgres_fe.h"++#include "catalog/pg_am.h"+#include "catalog/pg_class.h"+#include "common/logging.h"+#include "common/username.h"+#include "common/connect.h"+#include "fe_utils/print.h"+#include "fe_utils/simple_list.h"+#include "fe_utils/string_utils.h"+#include "pg_getopt.h"++const char *usage_text[] = {+ "pg_amcheck is the PostgreSQL command line frontend for the amcheck database corruption checker.",+ "",+ "Usage:",+ " pg_amcheck [OPTION]... [DBNAME [USERNAME]]",+ "",+ "General options:",+ " -V, --version output version information, then exit",+ " -?, --help show this help, then exit",+ " -s, --strict-names require include patterns to match at least one entity each",+ " -o, --on-error-stop stop checking at end of first corrupt page",+ "",+ "Schema checking options:",+ " -n, --schema=PATTERN check relations in the specified schema(s) only",+ " -N, --exclude-schema=PATTERN do NOT check relations in the specified schema(s)",+ "",+ "Table checking options:",+ " -t, --table=PATTERN check the specified table(s) only",+ " -T, --exclude-table=PATTERN do NOT check the specified table(s)",+ " -b, --startblock begin checking table(s) at the given starting block number",+ " -e, --endblock check table(s) only up to the given ending block number",+ " -f, --skip-all-frozen do NOT check blocks marked as all frozen",+ " -v, --skip-all-visible do NOT check blocks marked as all visible",+ "",+ "TOAST table checking options:",+ " -z, --check-toast check associated toast tables and toast indexes",+ " -Z, --skip-toast do NOT check associated toast tables and toast indexes",+ " -B, --toast-startblock begin checking toast table(s) at the given starting block",+ " -E, --toast-endblock check toast table(s) only up to the given ending block",+ "",+ "Index checking options:",+ " -x, --check-indexes check btree indexes associated with tables being checked",+ " -X, --skip-indexes do NOT check any btree indexes",+ " -i, --index=PATTERN check the specified index(es) only",+ " -I, --exclude-index=PATTERN do NOT check the specified index(es)",+ " -c, --check-corrupt check indexes even if their associated table is corrupt",+ " -C, --skip-corrupt do NOT check indexes if their associated table is corrupt",+ " -a, --heapallindexed check index tuples against the table tuples",+ " -A, --no-heapallindexed do NOT check index tuples against the table tuples",+ " -r, --rootdescend search from the root page for each index tuple",+ " -R, --no-rootdescend do NOT search from the root page for each index tuple",+ "",+ "Connection options:",+ " -d, --dbname=DBNAME database name to connect to",+ " -h, --host=HOSTNAME database server host or socket directory",+ " -p, --port=PORT database server port",+ " -U, --username=USERNAME database user name",+ " -w, --no-password never prompt for password",+ " -W, --password force password prompt (should happen automatically)",+ "",+ NULL /* sentinel */+};++typedef struct+AmCheckSettings+{+ char *dbname;+ char *host;+ char *port;+ char *username;+} ConnectOptions;++typedef enum trivalue+{+ TRI_DEFAULT,+ TRI_NO,+ TRI_YES+} trivalue;++typedef struct+{+ PGconn *db; /* connection to backend */+ bool notty; /* stdin or stdout is not a tty (as determined+ * on startup) */+ trivalue getPassword; /* prompt for a username and password */+ const char *progname; /* in case you renamed pg_amcheck */+ bool strict_names; /* The specified names/patterns should to+ * match at least one entity */+ bool on_error_stop; /* The checking of each table should stop+ * after the first corrupt page is found. */+ bool skip_frozen; /* Do not check pages marked all frozen */+ bool skip_visible; /* Do not check pages marked all visible */+ bool check_indexes; /* Check btree indexes */+ bool check_toast; /* Check associated toast tables and indexes */+ bool check_corrupt; /* Check indexes even if table is corrupt */+ bool heapallindexed; /* Perform index to table reconciling checks */+ bool rootdescend; /* Perform index rootdescend checks */+ char *startblock; /* Block number where checking begins */+ char *endblock; /* Block number where checking ends, inclusive */+ char *toaststart; /* Block number where toast checking begins */+ char *toastend; /* Block number where toast checking ends,+ * inclusive */+} AmCheckSettings;++static AmCheckSettings settings;++/*+ * Object inclusion/exclusion lists+ *+ * The string lists record the patterns given by command-line switches,+ * which we then convert to lists of Oids of matching objects.+ */+static SimpleStringList schema_include_patterns = {NULL, NULL};+static SimpleOidList schema_include_oids = {NULL, NULL};+static SimpleStringList schema_exclude_patterns = {NULL, NULL};+static SimpleOidList schema_exclude_oids = {NULL, NULL};++static SimpleStringList table_include_patterns = {NULL, NULL};+static SimpleOidList table_include_oids = {NULL, NULL};+static SimpleStringList table_exclude_patterns = {NULL, NULL};+static SimpleOidList table_exclude_oids = {NULL, NULL};++static SimpleStringList index_include_patterns = {NULL, NULL};+static SimpleOidList index_include_oids = {NULL, NULL};+static SimpleStringList index_exclude_patterns = {NULL, NULL};+static SimpleOidList index_exclude_oids = {NULL, NULL};++/*+ * List of tables to be checked, compiled from above lists.+ */+static SimpleOidList checklist = {NULL, NULL};++/*+ * Strings to be constructed once upon first use. These could be made+ * string constants instead, but that would require embedding knowledge+ * of the single character values for each relkind, such as 'm' for+ * materialized views, which we'd rather not embed here.+ */+static char *table_relkind_quals = NULL;+static char *index_relkind_quals = NULL;++/*+ * Functions to get pointers to the two strings, above, after initializing+ * them upon the first call to the function.+ */+static const char *get_table_relkind_quals(void);+static const char *get_index_relkind_quals(void);++/*+ * Functions for running the various corruption checks.+ */+static void check_tables(SimpleOidList *checklist);+static uint64 check_toast(Oid tbloid);+static uint64 check_table(Oid tbloid, const char *startblock,+ const char *endblock, bool on_error_stop,+ bool check_toast);+static uint64 check_indexes(Oid tbloid, const SimpleOidList *include_oids,+ const SimpleOidList *exclude_oids);+static uint64 check_index(const char *idxoid, const char *idxname,+ const char *tblname);++/*+ * Functions implementing standard command line behaviors.+ */+static void parse_cli_options(int argc, char *argv[],+ ConnectOptions *connOpts);+static void usage(void);+static void showVersion(void);+static void NoticeProcessor(void *arg, const char *message);++/*+ * Functions for converting command line options that include or exclude+ * schemas, tables, and indexes by pattern into internally useful lists of+ * Oids for objects that match those patterns.+ */+static void expand_schema_name_patterns(const SimpleStringList *patterns,+ const SimpleOidList *exclude_oids,+ SimpleOidList *oids,+ bool strict_names);+static void expand_relkind_name_patterns(const SimpleStringList *patterns,+ const SimpleOidList *exclude_nsp_oids,+ const SimpleOidList *exclude_oids,+ SimpleOidList *oids,+ bool strict_names,+ const char *missing_errtext,+ const char *relkind_quals);+static void expand_table_name_patterns(const SimpleStringList *patterns,+ const SimpleOidList *exclude_nsp_oids,+ const SimpleOidList *exclude_oids,+ SimpleOidList *oids,+ bool strict_names);+static void expand_index_name_patterns(const SimpleStringList *patterns,+ const SimpleOidList *exclude_nsp_oids,+ const SimpleOidList *exclude_oids,+ SimpleOidList *oids,+ bool strict_names);+static void get_table_check_list(const SimpleOidList *include_nsp,+ const SimpleOidList *exclude_nsp,+ const SimpleOidList *include_tbl,+ const SimpleOidList *exclude_tbl,+ SimpleOidList *checklist);+static PGresult *ExecuteSqlQuery(const char *query, char **error);+static PGresult *ExecuteSqlQueryOrDie(const char *query);++static void append_csv_oids(PQExpBuffer querybuf, const SimpleOidList *oids);+static void apply_filter(PQExpBuffer querybuf, const char *lval,+ const SimpleOidList *oids, bool include);++#define fatal(...) do { pg_log_error(__VA_ARGS__); exit(1); } while(0)++/* Like fatal(), but with a complaint about a particular query. */+static void+die_on_query_failure(const char *query)+{+ pg_log_error("query failed: %s",+ PQerrorMessage(settings.db));+ fatal("query was: %s", query);+}++#define NOPAGER 0+#define EXIT_BADCONN 2++int+main(int argc, char **argv)+{+ ConnectOptions connOpts;+ bool have_password = false;+ char password[100];+ bool new_pass;++ pg_logging_init(argv[0]);+ set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_amcheck"));++ if (argc > 1)+ {+ if ((strcmp(argv[1], "-?") == 0) ||+ (argc == 2 && (strcmp(argv[1], "--help") == 0)))+ {+ usage();+ exit(EXIT_SUCCESS);+ }+ if (strcmp(argv[1], "--version") == 0 || strcmp(argv[1], "-V") == 0)+ {+ showVersion();+ exit(EXIT_SUCCESS);+ }+ }++ memset(&settings, 0, sizeof(settings));+ settings.progname = get_progname(argv[0]);++ settings.db = NULL;+ setDecimalLocale();++ settings.notty = (!isatty(fileno(stdin)) || !isatty(fileno(stdout)));++ settings.getPassword = TRI_DEFAULT;++ /*+ * Default behaviors for user settable options. Note that these default to+ * doing all the safe checks and none of the unsafe ones, on the theory+ * that if a user says "pg_amcheck mydb" without specifying any additional+ * options, we should check everything we know how to check without risking+ * any backend aborts.+ */++ settings.on_error_stop = false;+ settings.skip_frozen = false;+ settings.skip_visible = false;++ /* Index checking options */+ settings.check_indexes = false;+ settings.check_corrupt = false;+ settings.heapallindexed = false;+ settings.rootdescend = false;++ /*+ * Reconciling toasted attributes from the main table against the toast+ * table can crash the backend if the toast table or index are corrupt. We+ * can optionally check the toast table and then the toast index prior to+ * checking the main table, but if the toast table or index are+ * concurrently corrupted after we conclude they are valid, the check of+ * the main table can crash the backend. The oneous is on any caller who+ * enables this option to make certain the environment is sufficiently+ * stable that concurrent corruptions of the toast is not possible.+ */+ settings.check_toast = false;++ parse_cli_options(argc, argv, &connOpts);++ if (settings.getPassword == TRI_YES)+ {+ /*+ * We can't be sure yet of the username that will be used, so don't+ * offer a potentially wrong one. Typical uses of this option are+ * noninteractive anyway.+ */+ simple_prompt("Password: ", password, sizeof(password), false);+ have_password = true;+ }++ /* loop until we have a password if requested by backend */+ do+ {+#define ARRAY_SIZE 8+ const char **keywords = pg_malloc(ARRAY_SIZE * sizeof(*keywords));+ const char **values = pg_malloc(ARRAY_SIZE * sizeof(*values));++ keywords[0] = "host";+ values[0] = connOpts.host;+ keywords[1] = "port";+ values[1] = connOpts.port;+ keywords[2] = "user";+ values[2] = connOpts.username;+ keywords[3] = "password";+ values[3] = have_password ? password : NULL;+ keywords[4] = "dbname"; /* see do_connect() */+ if (connOpts.dbname == NULL)+ {+ if (getenv("PGDATABASE"))+ values[4] = getenv("PGDATABASE");+ else if (getenv("PGUSER"))+ values[4] = getenv("PGUSER");+ else+ values[4] = "postgres";+ }+ else+ values[4] = connOpts.dbname;+ keywords[5] = "fallback_application_name";+ values[5] = settings.progname;+ keywords[6] = "client_encoding";+ values[6] = (settings.notty ||+ getenv("PGCLIENTENCODING")) ? NULL : "auto";+ keywords[7] = NULL;+ values[7] = NULL;++ new_pass = false;+ settings.db = PQconnectdbParams(keywords, values, true);+ if (settings.db == NULL)+ {+ pg_log_error("no connection to server after initial attempt");+ exit(EXIT_BADCONN);+ }++ free(keywords);+ free(values);++ if (PQstatus(settings.db) == CONNECTION_BAD &&+ PQconnectionNeedsPassword(settings.db) &&+ !have_password &&+ settings.getPassword != TRI_NO)+ {+ /*+ * Before closing the old PGconn, extract the user name that was+ * actually connected with.+ */+ const char *realusername = PQuser(settings.db);+ char *password_prompt;++ if (realusername && realusername[0])+ password_prompt = psprintf(_("Password for user %s: "),+ realusername);+ else+ password_prompt = pg_strdup(_("Password: "));+ PQfinish(settings.db);++ simple_prompt(password_prompt, password, sizeof(password), false);+ free(password_prompt);+ have_password = true;+ new_pass = true;+ }+ } while (new_pass);++ if (!settings.db)+ {+ pg_log_error("no connection to server");+ exit(EXIT_BADCONN);+ }++ if (PQstatus(settings.db) == CONNECTION_BAD)+ {+ pg_log_error("could not connect to server: %s",+ PQerrorMessage(settings.db));+ PQfinish(settings.db);+ exit(EXIT_BADCONN);+ }++ /*+ * Expand schema, table, and index exclusion patterns, if any. Note that+ * non-matching exclusion patterns are not an error, even when+ * --strict-names was specified.+ */+ expand_schema_name_patterns(&schema_exclude_patterns, NULL,+ &schema_exclude_oids, false);+ expand_table_name_patterns(&table_exclude_patterns, NULL, NULL,+ &table_exclude_oids, false);+ expand_index_name_patterns(&index_exclude_patterns, NULL, NULL,+ &index_exclude_oids, false);++ /* Expand schema selection patterns into Oid lists */+ if (schema_include_patterns.head != NULL)+ {+ expand_schema_name_patterns(&schema_include_patterns,+ &schema_exclude_oids,+ &schema_include_oids,+ settings.strict_names);+ if (schema_include_oids.head == NULL)+ fatal("no matching schemas were found");+ }++ /* Expand table selection patterns into Oid lists */+ if (table_include_patterns.head != NULL)+ {+ expand_table_name_patterns(&table_include_patterns,+ &schema_exclude_oids,+ &table_exclude_oids,+ &table_include_oids,+ settings.strict_names);+ if (table_include_oids.head == NULL)+ fatal("no matching tables were found");+ }++ /* Expand index selection patterns into Oid lists */+ if (index_include_patterns.head != NULL)+ {+ expand_index_name_patterns(&index_include_patterns,+ &schema_exclude_oids,+ &index_exclude_oids,+ &index_include_oids,+ settings.strict_names);+ if (index_include_oids.head == NULL)+ fatal("no matching indexes were found");+ }++ /*+ * Compile list of all tables to be checked based on namespace and table+ * includes and excludes.+ */+ get_table_check_list(&schema_include_oids, &schema_exclude_oids,+ &table_include_oids, &table_exclude_oids, &checklist);++ PQsetNoticeProcessor(settings.db, NoticeProcessor, NULL);++ /*+ * All information about corrupt indexes are returned via ereport, not as+ * tuples. We want all the details to report if corruption exists.+ */+ PQsetErrorVerbosity(settings.db, PQERRORS_VERBOSE);++ check_tables(&checklist);++ return 0;+}++/*+ * Conditionally add a restriction to a query such that lval must be an Oid in+ * the given list of Oids, except that for a null or empty oids list argument,+ * no filtering is done and we return without having modified the query buffer.+ *+ * The query argument must already have begun the WHERE clause and must be in a+ * state where we can append an AND clause. No checking of this requirement is+ * done here.+ *+ * On return, the query buffer will be extended with an AND clause that filters+ * only those rows where the lval is an Oid present in the given list of oids.+ */+static inline void+include_filter(PQExpBuffer querybuf, const char *lval, const SimpleOidList *oids)+{+ apply_filter(querybuf, lval, oids, true);+}++/*+ * Same as include_filter, above, except that for a non-null, non-empty oids+ * list, the lval is restricted to not be any of the values in the list.+ */+static inline void+exclude_filter(PQExpBuffer querybuf, const char *lval, const SimpleOidList *oids)+{+ apply_filter(querybuf, lval, oids, false);+}++/*+ * Check each table from the given checklist per the user specified options.+ */+static void+check_tables(SimpleOidList *checklist)+{+ const SimpleOidListCell *cell;++ for (cell = checklist->head; cell; cell = cell->next)+ {+ uint64 corruptions = 0;+ bool reconcile_toast;++ /*+ * If we skip checking the toast table, or if during the check we+ * detect any toast table corruption, the main table checks below must+ * not reconcile toasted attributes against the toast table, as such+ * accesses to the toast table might crash the backend. Instead, skip+ * such reconciliations for this table.+ *+ * This protection contains a race condition; the toast table or index+ * could become corrupted concurrently with our checks, but prevention+ * of such concurrent corruption is documented as the caller's+ * reponsibility, so we don't worry about it here.+ */+ reconcile_toast = false;+ if (settings.check_toast)+ {+ if (check_toast(cell->val) == 0)+ reconcile_toast = true;+ }++ corruptions = check_table(cell->val,+ settings.startblock,+ settings.endblock,+ settings.on_error_stop,+ reconcile_toast);++ if (settings.check_indexes)+ {+ bool old_heapallindexed;++ /* Optionally skip the index checks for a corrupt table. */+ if (corruptions && !settings.check_corrupt)+ continue;++ /*+ * The btree checking logic which optionally checks the contents of+ * an index against the corresponding table has not yet been+ * sufficiently hardened against corrupt tables. In particular,+ * when called with heapallindexed true, it segfaults if the file+ * backing the table relation has been erroneously unlinked. In+ * any event, it seems unwise to reconcile an index against its+ * table when we already know the table is corrupt.+ */+ old_heapallindexed = settings.heapallindexed;+ if (corruptions)+ settings.heapallindexed = false;++ corruptions += check_indexes(cell->val,+ &index_include_oids,+ &index_exclude_oids);++ settings.heapallindexed = old_heapallindexed;+ }+ }+}++/*+ * For a given main table relation, returns the associated toast table,+ * or InvalidOid if none exists.+ */+static Oid+get_toast_oid(Oid tbloid)+{+ PQExpBuffer querybuf = createPQExpBuffer();+ PGresult *res;+ char *error = NULL;+ Oid result = InvalidOid;++ appendPQExpBuffer(querybuf,+ "SELECT c.reltoastrelid"+ "\nFROM pg_catalog.pg_class c"+ "\nWHERE c.oid = %u",+ tbloid);+ res = ExecuteSqlQuery(querybuf->data, &error);+ if (PQresultStatus(res) == PGRES_TUPLES_OK && PQntuples(res) > 0)+ result = atooid(PQgetvalue(res, 0, 0));+ else if (error)+ die_on_query_failure(querybuf->data);++ PQclear(res);+ destroyPQExpBuffer(querybuf);++ return result;+}++/*+ * For the given main table relation, checks the associated toast table and+ * index, in any. This should be performed *before* checking the main table+ * relation, as the checks inside verify_heapam assume both the toast table and+ * toast index are usable.+ *+ * Returns the number of corruptions detected.+ */+static uint64+check_toast(Oid tbloid)+{+ Oid toastoid;+ uint64 corruption_cnt = 0;++ if (settings.db == NULL)+ fatal("no connection on entry to check_toast");++ toastoid = get_toast_oid(tbloid);+ if (OidIsValid(toastoid))+ {+ corruption_cnt = check_table(toastoid, settings.toaststart,+ settings.toastend, settings.on_error_stop,+ false);+ /*+ * If the toast table is corrupt, checking the index is not safe.+ * There is a race condition here, as the toast table could be+ * concurrently corrupted, but preventing concurrent corruption is the+ * caller's responsibility, not ours.+ */+ if (corruption_cnt == 0)+ corruption_cnt += check_indexes(toastoid, NULL, NULL);+ }++ return corruption_cnt;+}++/*+ * Checks the given table for corruption, returning the number of corruptions+ * detected and printed to the user.+ */+static uint64+check_table(Oid tbloid, const char *startblock, const char *endblock,+ bool on_error_stop, bool check_toast)+{+ PQExpBuffer querybuf;+ PGresult *res;+ int i;+ char *skip;+ char *toast;+ const char *stop;+ char *error = NULL;+ uint64 corruption_cnt = 0;++ if (settings.db == NULL)+ fatal("no connection on entry to check_table");++ if (startblock == NULL)+ startblock = "NULL";+ if (endblock == NULL)+ endblock = "NULL";+ if (settings.skip_frozen)+ skip = pg_strdup("'all frozen'");+ else if (settings.skip_visible)+ skip = pg_strdup("'all visible'");+ else+ skip = pg_strdup("NULL");+ stop = (on_error_stop) ? "true" : "false";+ toast = (check_toast) ? "true" : "false";++ querybuf = createPQExpBuffer();++ appendPQExpBuffer(querybuf,+ "SELECT c.relname, v.blkno, v.offnum, v.attnum, v.msg"+ "\nFROM public.verify_heapam("+ "\nrelation := %u,"+ "\non_error_stop := %s,"+ "\nskip := %s,"+ "\ncheck_toast := %s,"+ "\nstartblock := %s,"+ "\nendblock := %s) v, "+ "\npg_catalog.pg_class c"+ "\nWHERE c.oid = %u",+ tbloid, stop, skip, toast, startblock, endblock, tbloid);++ res = ExecuteSqlQuery(querybuf->data, &error);+ if (PQresultStatus(res) == PGRES_TUPLES_OK && PQntuples(res) > 0)+ {+ corruption_cnt += PQntuples(res);+ for (i = 0; i < PQntuples(res); i++)+ {+ printf("(relname=%s,blkno=%s,offnum=%s,attnum=%s)\n%s\n",+ PQgetvalue(res, i, 0), /* relname */+ PQgetvalue(res, i, 1), /* blkno */+ PQgetvalue(res, i, 2), /* offnum */+ PQgetvalue(res, i, 3), /* attnum */+ PQgetvalue(res, i, 4)); /* msg */+ }+ }+ else if (error)+ {+ corruption_cnt++;+ printf("%s\n", error);+ pfree(error);+ }++ PQclear(res);+ destroyPQExpBuffer(querybuf);+ return corruption_cnt;+}++static uint64+check_indexes(Oid tbloid, const SimpleOidList *include_oids,+ const SimpleOidList *exclude_oids)+{+ PQExpBuffer querybuf;+ PGresult *res;+ int i;+ char *error = NULL;+ uint64 corruption_cnt = 0;++ if (settings.db == NULL)+ fatal("no connection on entry to check_indexes");++ querybuf = createPQExpBuffer();+ appendPQExpBuffer(querybuf,+ "SELECT i.indexrelid, ci.relname, ct.relname"+ "\nFROM pg_catalog.pg_index i, pg_catalog.pg_class ci, "+ "pg_catalog.pg_class ct"+ "\nWHERE i.indexrelid = ci.oid"+ "\n AND i.indrelid = ct.oid"+ "\n AND ci.relam = %u"+ "\n AND i.indrelid = %u",+ BTREE_AM_OID, tbloid);+ include_filter(querybuf, "i.indexrelid", include_oids);+ exclude_filter(querybuf, "i.indexrelid", exclude_oids);++ res = ExecuteSqlQuery(querybuf->data, &error);+ if (PQresultStatus(res) == PGRES_TUPLES_OK)+ {+ for (i = 0; i < PQntuples(res); i++)+ corruption_cnt += check_index(PQgetvalue(res, i, 0),+ PQgetvalue(res, i, 1),+ PQgetvalue(res, i, 2));+ }+ else if (error)+ {+ corruption_cnt++;+ printf("%s\n", error);+ pfree(error);+ }++ PQclear(res);+ destroyPQExpBuffer(querybuf);++ return corruption_cnt;+}++static uint64+check_index(const char *idxoid, const char *idxname, const char *tblname)+{+ PQExpBuffer querybuf;+ PGresult *res;+ uint64 corruption_cnt = 0;++ if (settings.db == NULL)+ fatal("no connection on entry to check_index");+ if (idxname == NULL)+ fatal("no index name on entry to check_index");+ if (tblname == NULL)+ fatal("no table name on entry to check_index");++ querybuf = createPQExpBuffer();+ appendPQExpBuffer(querybuf,+ "SELECT public.bt_index_parent_check('%s'::regclass, %s, %s)",+ idxoid,+ settings.heapallindexed ? "true" : "false",+ settings.rootdescend ? "true" : "false");+ res = PQexec(settings.db, querybuf->data);+ if (PQresultStatus(res) != PGRES_TUPLES_OK)+ {+ corruption_cnt++;+ printf("index check failed for index %s of table %s:\n",+ idxname, tblname);+ printf("%s", PQerrorMessage(settings.db));+ }++ PQclear(res);+ destroyPQExpBuffer(querybuf);++ return corruption_cnt;+}++static void+parse_cli_options(int argc, char *argv[], ConnectOptions *connOpts)+{+ static struct option long_options[] =+ {+ {"check-corrupt", no_argument, NULL, 'c'},+ {"check-indexes", no_argument, NULL, 'x'},+ {"check-toast", no_argument, NULL, 'z'},+ {"dbname", required_argument, NULL, 'd'},+ {"endblock", required_argument, NULL, 'e'},+ {"exclude-index", required_argument, NULL, 'I'},+ {"exclude-schema", required_argument, NULL, 'N'},+ {"exclude-table", required_argument, NULL, 'T'},+ {"heapallindexed", no_argument, NULL, 'a'},+ {"help", optional_argument, NULL, '?'},+ {"host", required_argument, NULL, 'h'},+ {"index", required_argument, NULL, 'i'},+ {"no-heapallindexed", no_argument, NULL, 'A'},+ {"no-password", no_argument, NULL, 'w'},+ {"no-rootdescend", no_argument, NULL, 'R'},+ {"on-error-stop", no_argument, NULL, 'o'},+ {"password", no_argument, NULL, 'W'},+ {"port", required_argument, NULL, 'p'},+ {"rootdescend", no_argument, NULL, 'r'},+ {"schema", required_argument, NULL, 'n'},+ {"skip-all-frozen", no_argument, NULL, 'f'},+ {"skip-all-visible", no_argument, NULL, 'v'},+ {"skip-corrupt", no_argument, NULL, 'C'},+ {"skip-indexes", no_argument, NULL, 'X'},+ {"skip-toast", no_argument, NULL, 'Z'},+ {"startblock", required_argument, NULL, 'b'},+ {"strict-names", no_argument, NULL, 's'},+ {"table", required_argument, NULL, 't'},+ {"toast-endblock", required_argument, NULL, 'E'},+ {"toast-startblock", required_argument, NULL, 'B'},+ {"username", required_argument, NULL, 'U'},+ {"version", no_argument, NULL, 'V'},+ {NULL, 0, NULL, 0}+ };++ int optindex;+ int c;++ memset(connOpts, 0, sizeof *connOpts);++ while ((c = getopt_long(argc, argv, "aAb:B:cCd:e:E:fh:i:I:n:N:op:rRst:T:U:vVwWxXzZ?1",+ long_options, &optindex)) != -1)+ {+ switch (c)+ {+ case 'a':+ settings.heapallindexed = true;+ break;+ case 'A':+ settings.heapallindexed = false;+ break;+ case 'b':+ settings.startblock = pg_strdup(optarg);+ break;+ case 'B':+ settings.toaststart = pg_strdup(optarg);+ break;+ case 'c':+ settings.check_corrupt = true;+ break;+ case 'C':+ settings.check_corrupt = false;+ break;+ case 'd':+ connOpts->dbname = pg_strdup(optarg);+ break;+ case 'e':+ settings.endblock = pg_strdup(optarg);+ break;+ case 'E':+ settings.toastend = pg_strdup(optarg);+ break;+ case 'f':+ settings.skip_frozen = true;+ break;+ case 'h':+ connOpts->host = pg_strdup(optarg);+ break;+ case 'i':+ simple_string_list_append(&index_include_patterns, optarg);+ break;+ case 'I':+ simple_string_list_append(&index_exclude_patterns, optarg);+ break;+ case 'n': /* include schema(s) */+ simple_string_list_append(&schema_include_patterns, optarg);+ break;+ case 'N': /* exclude schema(s) */+ simple_string_list_append(&schema_exclude_patterns, optarg);+ break;+ case 'o':+ settings.on_error_stop = true;+ break;+ case 'p':+ connOpts->port = pg_strdup(optarg);+ break;+ case 's':+ settings.strict_names = true;+ break;+ case 'r':+ settings.rootdescend = true;+ break;+ case 'R':+ settings.rootdescend = false;+ break;+ case 't': /* include table(s) */+ simple_string_list_append(&table_include_patterns, optarg);+ break;+ case 'T': /* exclude table(s) */+ simple_string_list_append(&table_exclude_patterns, optarg);+ break;+ case 'U':+ connOpts->username = pg_strdup(optarg);+ break;+ case 'v':+ settings.skip_visible = true;+ break;+ case 'V':+ showVersion();+ exit(EXIT_SUCCESS);+ case 'w':+ settings.getPassword = TRI_NO;+ break;+ case 'W':+ settings.getPassword = TRI_YES;+ break;+ case 'x':+ settings.check_indexes = true;+ break;+ case 'X':+ settings.check_indexes = false;+ break;+ case 'z':+ settings.check_toast = true;+ break;+ case 'Z':+ settings.check_toast = false;+ break;+ case '?':+ if (optind <= argc &&+ strcmp(argv[optind - 1], "-?") == 0)+ {+ /* actual help option given */+ usage();+ exit(EXIT_SUCCESS);+ }+ else+ {+ /* getopt error (unknown option or missing argument) */+ goto unknown_option;+ }+ break;+ case 1:+ {+ if (!optarg || strcmp(optarg, "options") == 0)+ usage();+ else+ goto unknown_option;++ exit(EXIT_SUCCESS);+ }+ break;+ default:+ unknown_option:+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),+ settings.progname);+ exit(EXIT_FAILURE);+ break;+ }+ }++ /*+ * if we still have arguments, use it as the database name and username+ */+ while (argc - optind >= 1)+ {+ if (!connOpts->dbname)+ connOpts->dbname = argv[optind];+ else if (!connOpts->username)+ connOpts->username = argv[optind];+ else+ pg_log_warning("extra command-line argument \"%s\" ignored",+ argv[optind]);++ optind++;+ }++}++/*+ * usage+ *+ * print out command line arguments+ */+static void+usage(void)+{+ int lineno;++ for (lineno = 0; usage_text[lineno]; lineno++)+ printf("%s\n", usage_text[lineno]);+ printf("Report bugs to <%s>.\n", PACKAGE_BUGREPORT);+ printf("%s home page: <%s>\n", PACKAGE_NAME, PACKAGE_URL);+}++static void+showVersion(void)+{+ puts("pg_amcheck (PostgreSQL) " PG_VERSION);+}++/*+ * for backend Notice messages (INFO, WARNING, etc)+ */+static void+NoticeProcessor(void *arg, const char *message)+{+ (void) arg; /* not used */+ pg_log_info("%s", message);+}++/*+ * Helper function for apply_filter, below.+ */+static void+append_csv_oids(PQExpBuffer querybuf, const SimpleOidList *oids)+{+ const SimpleOidListCell *cell;+ const char *comma;++ for (comma = "", cell = oids->head; cell; comma = ", ", cell = cell->next)+ appendPQExpBuffer(querybuf, "%s%u", comma, cell->val);+}++/*+ * Internal implementation of include_filter and exclude_filter+ */+static void+apply_filter(PQExpBuffer querybuf, const char *lval, const SimpleOidList *oids,+ bool include)+{+ if (!oids || !oids->head)+ return;+ if (include)+ appendPQExpBuffer(querybuf, "\nAND %s OPERATOR(pg_catalog.=) ANY(array[", lval);+ else+ appendPQExpBuffer(querybuf, "\nAND %s OPERATOR(pg_catalog.!=) ALL(array[", lval);+ append_csv_oids(querybuf, oids);+ appendPQExpBuffer(querybuf, "]::OID[])");+}++/*+ * Find and append to the given Oid list the Oids of all schemas matching the+ * given list of patterns but not included in the given list of excluded Oids.+ */+static void+expand_schema_name_patterns(const SimpleStringList *patterns,+ const SimpleOidList *exclude_nsp,+ SimpleOidList *oids,+ bool strict_names)+{+ PQExpBuffer querybuf;+ PGresult *res;+ SimpleStringListCell *cell;+ int i;++ if (settings.db == NULL)+ fatal("no connection on entry to expand_schema_name_patterns");++ if (patterns->head == NULL)+ return; /* nothing to do */++ querybuf = createPQExpBuffer();++ /*+ * The loop below runs multiple SELECTs might sometimes result in+ * duplicate entries in the Oid list, but we don't care.+ */++ for (cell = patterns->head; cell; cell = cell->next)+ {+ appendPQExpBufferStr(querybuf,+ "SELECT oid FROM pg_catalog.pg_namespace n\n");+ processSQLNamePattern(settings.db, querybuf, cell->val, false,+ false, NULL, "n.nspname", NULL, NULL);+ exclude_filter(querybuf, "n.oid", exclude_nsp);++ res = ExecuteSqlQueryOrDie(querybuf->data);+ if (strict_names && PQntuples(res) == 0)+ fatal("no matching schemas were found for pattern \"%s\"",+ cell->val);++ for (i = 0; i < PQntuples(res); i++)+ {+ simple_oid_list_append(oids, atooid(PQgetvalue(res, i, 0)));+ }++ PQclear(res);+ resetPQExpBuffer(querybuf);+ }++ destroyPQExpBuffer(querybuf);+}++/*+ * Find and append to the given Oid list the Oids of all relations matching the+ * given list of patterns but not included in the given list of excluded Oids+ * nor in one of the given excluded namespaces. The relations are filtered by+ * the given schema_quals. They are further filtered by the given+ * relkind_quals, allowing the caller to restrict the relations to just indexes+ * or tables. The missing_errtext should be a message for use in error+ * messages if no matching relations are found and strict_names was specified.+ */+static void+expand_relkind_name_patterns(const SimpleStringList *patterns,+ const SimpleOidList *exclude_nsp_oids,+ const SimpleOidList *exclude_oids,+ SimpleOidList *oids,+ bool strict_names,+ const char *missing_errtext,+ const char *relkind_quals)+{+ PQExpBuffer querybuf;+ PGresult *res;+ SimpleStringListCell *cell;+ int i;++ if (settings.db == NULL)+ fatal("no connection on entry to expand_relkind_name_patterns");++ if (patterns->head == NULL)+ return; /* nothing to do */++ querybuf = createPQExpBuffer();++ /*+ * this might sometimes result in duplicate entries in the Oid list, but+ * we don't care.+ */++ for (cell = patterns->head; cell; cell = cell->next)+ {+ /*+ * Query must remain ABSOLUTELY devoid of unqualified names. This+ * would be unnecessary given a pg_table_is_visible() variant taking a+ * search_path argument.+ */+ appendPQExpBuffer(querybuf,+ "SELECT c.oid"+ "\nFROM pg_catalog.pg_class c"+ "\n LEFT JOIN pg_catalog.pg_namespace n"+ "\n ON n.oid OPERATOR(pg_catalog.=) c.relnamespace"+ "\nWHERE c.relkind OPERATOR(pg_catalog.=) %s\n",+ relkind_quals);+ exclude_filter(querybuf, "c.oid", exclude_oids);+ exclude_filter(querybuf, "n.oid", exclude_nsp_oids);+ processSQLNamePattern(settings.db, querybuf, cell->val, true,+ false, "n.nspname", "c.relname", NULL, NULL);+ res = ExecuteSqlQueryOrDie(querybuf->data);+ if (strict_names && PQntuples(res) == 0)+ fatal("%s \"%s\"", missing_errtext, cell->val);++ for (i = 0; i < PQntuples(res); i++)+ simple_oid_list_append(oids, atooid(PQgetvalue(res, i, 0)));++ PQclear(res);+ resetPQExpBuffer(querybuf);+ }++ destroyPQExpBuffer(querybuf);+}++/*+ * Find the Oids of all tables matching the given list of patterns,+ * and append them to the given Oid list.+ */+static void+expand_table_name_patterns(const SimpleStringList *patterns, const SimpleOidList *exclude_nsp_oids,+ const SimpleOidList *exclude_oids, SimpleOidList *oids, bool strict_names)+{+ expand_relkind_name_patterns(patterns, exclude_nsp_oids, exclude_oids, oids, strict_names,+ "no matching tables were found for pattern",+ get_table_relkind_quals());+}++/*+ * Find the Oids of all indexes matching the given list of patterns,+ * and append them to the given Oid list.+ */+static void+expand_index_name_patterns(const SimpleStringList *patterns, const SimpleOidList *exclude_nsp_oids,+ const SimpleOidList *exclude_oids, SimpleOidList *oids, bool strict_names)+{+ expand_relkind_name_patterns(patterns, exclude_nsp_oids, exclude_oids, oids, strict_names,+ "no matching indexes were found for pattern",+ get_index_relkind_quals());+}++static void+get_table_check_list(const SimpleOidList *include_nsp, const SimpleOidList *exclude_nsp,+ const SimpleOidList *include_tbl, const SimpleOidList *exclude_tbl,+ SimpleOidList *checklist)+{+ PQExpBuffer querybuf;+ PGresult *res;+ int i;++ if (settings.db == NULL)+ fatal("no connection on entry to get_table_check_list");++ querybuf = createPQExpBuffer();++ appendPQExpBuffer(querybuf,+ "SELECT c.oid"+ "\nFROM pg_catalog.pg_class c, pg_catalog.pg_namespace n"+ "\nWHERE n.oid OPERATOR(pg_catalog.=) c.relnamespace"+ "\n AND c.relkind OPERATOR(pg_catalog.=) %s\n",+ get_table_relkind_quals());+ include_filter(querybuf, "n.oid", include_nsp);+ exclude_filter(querybuf, "n.oid", exclude_nsp);+ include_filter(querybuf, "c.oid", include_tbl);+ exclude_filter(querybuf, "c.oid", exclude_tbl);++ res = ExecuteSqlQueryOrDie(querybuf->data);+ for (i = 0; i < PQntuples(res); i++)+ simple_oid_list_append(checklist, atooid(PQgetvalue(res, i, 0)));++ PQclear(res);+ destroyPQExpBuffer(querybuf);+}++static PGresult *+ExecuteSqlQueryOrDie(const char *query)+{+ PGresult *res;++ res = PQexec(settings.db, query);+ if (PQresultStatus(res) != PGRES_TUPLES_OK)+ die_on_query_failure(query);+ return res;+}++/*+ * Execute the given SQL query. This function should only be used for queries+ * which are not expected to fail under normal circumstances, as failures will+ * result in the printing of error messages, which will look a bit messy when+ * interleaved with corruption reports.+ *+ * On error, use the supplied error_context string and the error string+ * returned from the database connection to print an error message for the+ * user.+ *+ * The error_context argument is pfree'd by us at the end of the call.+ */+static PGresult *+ExecuteSqlQuery(const char *query, char **error)+{+ PGresult *res;++ res = PQexec(settings.db, query);+ if (PQresultStatus(res) != PGRES_TUPLES_OK)+ *error = pstrdup(PQerrorMessage(settings.db));+ return res;+}++/*+ * Return the cached relkind quals string for tables, computing it first if we+ * don't have one cached.+ */+static const char *+get_table_relkind_quals(void)+{+ if (!table_relkind_quals)+ table_relkind_quals = psprintf("ANY(array['%c', '%c', '%c'])",+ RELKIND_RELATION, RELKIND_MATVIEW,+ RELKIND_PARTITIONED_TABLE);+ return table_relkind_quals;+}++/*+ * Return the cached relkind quals string for indexes, computing it first if we+ * don't have one cached.+ */+static const char *+get_index_relkind_quals(void)+{+ if (!index_relkind_quals)+ index_relkind_quals = psprintf("'%c'", RELKIND_INDEX);+ return index_relkind_quals;+}diff --git a/contrib/pg_amcheck/pg_amcheck.control b/contrib/pg_amcheck/pg_amcheck.controlnew file mode 100644index 0000000000..395f368101--- /dev/null+++ b/contrib/pg_amcheck/pg_amcheck.control@@ -0,0 +1,5 @@+# pg_amcheck extension+comment = 'command-line tool for verifying relation integrity'+default_version = '1.3'+module_pathname = '$libdir/pg_amcheck'+relocatable = truediff --git a/contrib/pg_amcheck/t/001_basic.pl b/contrib/pg_amcheck/t/001_basic.plnew file mode 100644index 0000000000..dfa0ae9e06--- /dev/null+++ b/contrib/pg_amcheck/t/001_basic.pl@@ -0,0 +1,9 @@+use strict;+use warnings;++use TestLib;+use Test::More tests => 8;++program_help_ok('pg_amcheck');+program_version_ok('pg_amcheck');+program_options_handling_ok('pg_amcheck');diff --git a/contrib/pg_amcheck/t/002_nonesuch.pl b/contrib/pg_amcheck/t/002_nonesuch.plnew file mode 100644index 0000000000..68be9c6585--- /dev/null+++ b/contrib/pg_amcheck/t/002_nonesuch.pl@@ -0,0 +1,60 @@+use strict;+use warnings;++use PostgresNode;+use TestLib;+use Test::More tests => 14;++# Test set-up+my ($node, $port);+$node = get_new_node('test');+$node->init;+$node->start;+$port = $node->port;++# Load the amcheck extension, upon which pg_amcheck depends+$node->safe_psql('postgres', q(CREATE EXTENSION amcheck));++#########################################+# Test connecting to a non-existent database++command_fails_like(+ [ 'pg_amcheck', '-p', "$port", 'qqq' ],+ qr/\Qpg_amcheck: error: could not connect to server: FATAL: database "qqq" does not exist\E/,+ 'connecting to a non-existent database');++#########################################+# Test connecting with a non-existent user++command_fails_like(+ [ 'pg_amcheck', '-p', "$port", '-U=no_such_user' ],+ qr/\Qpg_amcheck: error: could not connect to server: FATAL: role "=no_such_user" does not exist\E/,+ 'connecting with a non-existent user');++#########################################+# Test checking a non-existent schema, table, and patterns with --strict-names++command_fails_like(+ [ 'pg_amcheck', '-p', "$port", '-n', 'nonexistent' ],+ qr/\Qpg_amcheck: error: no matching schemas were found\E/,+ 'checking a non-existent schema');++command_fails_like(+ [ 'pg_amcheck', '-p', "$port", '-t', 'nonexistent' ],+ qr/\Qpg_amcheck: error: no matching tables were found\E/,+ 'checking a non-existent table');++command_fails_like(+ [ 'pg_amcheck', '-p', "$port", '--strict-names', '-n', 'nonexistent*' ],+ qr/\Qpg_amcheck: error: no matching schemas were found for pattern\E/,+ 'no matching schemas');++command_fails_like(+ [ 'pg_amcheck', '-p', "$port", '--strict-names', '-t', 'nonexistent*' ],+ qr/\Qpg_amcheck: error: no matching tables were found for pattern\E/,+ 'no matching tables');++command_fails_like(+ [ 'pg_amcheck', '-p', "$port", '--strict-names', '-i', 'nonexistent*' ],+ qr/\Qpg_amcheck: error: no matching indexes were found for pattern\E/,+ 'no matching indexes');diff --git a/contrib/pg_amcheck/t/003_check.pl b/contrib/pg_amcheck/t/003_check.plnew file mode 100644index 0000000000..4d8e61d871--- /dev/null+++ b/contrib/pg_amcheck/t/003_check.pl@@ -0,0 +1,231 @@+use strict;+use warnings;++use PostgresNode;+use TestLib;+use Test::More tests => 39;++my ($node, $port);++# Returns the filesystem path for the named relation.+#+# Assumes the test node is running+sub relation_filepath($)+{+ my ($relname) = @_;++ my $pgdata = $node->data_dir;+ my $rel = $node->safe_psql('postgres',+ qq(SELECT pg_relation_filepath('$relname')));+ die "path not found for relation $relname" unless defined $rel;+ return "$pgdata/$rel";+}++# Stops the test node, corrupts the first page of the named relation, and+# restarts the node.+#+# Assumes the node is running.+sub corrupt_first_page($)+{+ my ($relname) = @_;+ my $relpath = relation_filepath($relname);++ $node->stop;+ my $fh;+ open($fh, '+<', $relpath);+ binmode $fh;+ seek($fh, 32, 0);+ syswrite($fh, '\x77\x77\x77\x77', 500);+ close($fh);+ $node->start;+}++# Stops the test node, unlinks the file from the filesystem that backs the+# relation, and restarts the node.+#+# Assumes the test node is running+sub remove_relation_file($)+{+ my ($relname) = @_;+ my $relpath = relation_filepath($relname);++ $node->stop();+ unlink($relpath);+ $node->start;+}++# Test set-up+$node = get_new_node('test');+$node->init;+$node->start;+$port = $node->port;++# Load the amcheck extension, upon which pg_amcheck depends+$node->safe_psql('postgres', q(CREATE EXTENSION amcheck));++# Create schemas and tables for checking pg_amcheck's include+# and exclude schema and table command line options+$node->safe_psql('postgres', q(+-- We'll corrupt all indexes in s1+CREATE SCHEMA s1;+CREATE TABLE s1.t1 (a TEXT);+CREATE TABLE s1.t2 (a TEXT);+CREATE INDEX i1 ON s1.t1(a);+CREATE INDEX i2 ON s1.t2(a);+INSERT INTO s1.t1 (a) (SELECT gs::TEXT FROM generate_series(1,10000) AS gs);+INSERT INTO s1.t2 (a) (SELECT gs::TEXT FROM generate_series(1,10000) AS gs);++-- We'll corrupt all tables in s2+CREATE SCHEMA s2;+CREATE TABLE s2.t1 (a TEXT);+CREATE TABLE s2.t2 (a TEXT);+CREATE INDEX i1 ON s2.t1(a);+CREATE INDEX i2 ON s2.t2(a);+INSERT INTO s2.t1 (a) (SELECT gs::TEXT FROM generate_series(1,10000) AS gs);+INSERT INTO s2.t2 (a) (SELECT gs::TEXT FROM generate_series(1,10000) AS gs);++-- We'll corrupt all tables and indexes in s3+CREATE SCHEMA s3;+CREATE TABLE s3.t1 (a TEXT);+CREATE TABLE s3.t2 (a TEXT);+CREATE INDEX i1 ON s3.t1(a);+CREATE INDEX i2 ON s3.t2(a);+INSERT INTO s3.t1 (a) (SELECT gs::TEXT FROM generate_series(1,10000) AS gs);+INSERT INTO s3.t2 (a) (SELECT gs::TEXT FROM generate_series(1,10000) AS gs);++-- We'll leave everything in s4 uncorrupted+CREATE SCHEMA s4;+CREATE TABLE s4.t1 (a TEXT);+CREATE TABLE s4.t2 (a TEXT);+CREATE INDEX i1 ON s4.t1(a);+CREATE INDEX i2 ON s4.t2(a);+INSERT INTO s4.t1 (a) (SELECT gs::TEXT FROM generate_series(1,10000) AS gs);+INSERT INTO s4.t2 (a) (SELECT gs::TEXT FROM generate_series(1,10000) AS gs);+));++# Corrupt indexes in schema "s1"+remove_relation_file('s1.i1');+corrupt_first_page('s1.i2');++# Corrupt tables in schema "s2"+remove_relation_file('s2.t1');+corrupt_first_page('s2.t2');++# Corrupt tables and indexes in schema "s3"+remove_relation_file('s3.i1');+corrupt_first_page('s3.i2');+remove_relation_file('s3.t1');+corrupt_first_page('s3.t2');++# Leave schema "s4" alone+++# The pg_amcheck command itself should return a success exit status, even+# though tables and indexes are corrupt. An error code returned would mean the+# pg_amcheck command itself failed, for example because a connection to the+# database could not be established.+#+# For these checks, we're ignoring any corruption reported and focusing+# exclusively on the exit code from pg_amcheck.+#+$node->command_ok(+ [ 'pg_amcheck', '-x', '-p', $port, 'postgres' ],+ 'pg_amcheck all schemas and tables');++$node->command_ok(+ [ 'pg_amcheck', '-p', $port, 'postgres' ],+ 'pg_amcheck all schemas, tables and indexes');++$node->command_ok(+ [ 'pg_amcheck', '-p', $port, 'postgres', '-n', 's1' ],+ 'pg_amcheck all objects in schema s1');++$node->command_ok(+ [ 'pg_amcheck', '-p', $port, 'postgres', '-n', 's*', '-t', 't1' ],+ 'pg_amcheck all tables named t1 and their indexes');++$node->command_ok(+ [ 'pg_amcheck', '-x', '-p', $port, 'postgres', '-T', 't1' ],+ 'pg_amcheck all tables not named t1');++$node->command_ok(+ [ 'pg_amcheck', '-x', '-p', $port, 'postgres', '-N', 's1', '-T', 't1' ],+ 'pg_amcheck all tables not named t1 nor in schema s1');++# Scans of indexes in s1 should detect the specific corruption that we created+# above. For missing relation forks, we know what the error message looks+# like. For corrupted index pages, the error might vary depending on how the+# page was formatted on disk, including variations due to alignment differences+# between platforms, so we accept any non-empty error message.+#+$node->command_like(+ [ 'pg_amcheck', '-x', '-p', $port, 'postgres', '-n', 's1', '-i', 'i1' ],+ qr/index "i1" lacks a main relation fork/,+ 'pg_amcheck index s1.i1 reports missing main relation fork');++$node->command_like(+ [ 'pg_amcheck', '-x', '-p', $port, 'postgres', '-n', 's1', '-i', 'i2' ],+ qr/.+/, # Any non-empty error message is acceptable+ 'pg_amcheck index s1.s2 reports index corruption');+++# In schema s3, the tables and indexes are both corrupt. Ordinarily, checking+# of indexes will not be performed for corrupt tables, but the --check-corrupt+# option (-c) forces the indexes to also be checked.+#+$node->command_like(+ [ 'pg_amcheck', '-x', '-c', '-p', $port, 'postgres', '-n', 's3', '-i', 'i1' ],+ qr/index "i1" lacks a main relation fork/,+ 'pg_amcheck index s3.i1 reports missing main relation fork');++$node->command_like(+ [ 'pg_amcheck', '-x', '-c', '-p', $port, 'postgres', '-n', 's3', '-i', 'i2' ],+ qr/.+/, # Any non-empty error message is acceptable+ 'pg_amcheck index s3.s2 reports index corruption');+++# Check that '-x' and '-X' work as expected. Since only index corruption+# (and not table corruption) exists in s1, '-X' should give no errors, and+# '-x' should give errors about index corruption.+#+$node->command_like(+ [ 'pg_amcheck', '-x', '-p', $port, 'postgres', '-n', 's1' ],+ qr/.+/, # Any non-empty error message is acceptable+ 'pg_amcheck over tables and indexes in schema s1 reports corruption');++$node->command_like(+ [ 'pg_amcheck', '-X', '-p', $port, 'postgres', '-n', 's1' ],+ qr/^$/, # Empty+ 'pg_amcheck over only tables in schema s1 reports no corruption');+++# Check that table corruption is reported as expected, with or without+# index checking+#+$node->command_like(+ [ 'pg_amcheck', '-x', '-p', $port, 'postgres', '-n', 's2' ],+ qr/could not open file/,+ 'pg_amcheck over tables in schema s2 reports table corruption');++$node->command_like(+ [ 'pg_amcheck', '-p', $port, 'postgres', '-n', 's2' ],+ qr/could not open file/,+ 'pg_amcheck over tables and indexes in schema s2 reports table corruption');++# Check that no corruption is reported in schema s4+$node->command_like(+ [ 'pg_amcheck', '-p', $port, 'postgres', '-n', 's4' ],+ qr/^$/, # Empty+ 'pg_amcheck over schema s4 reports no corruption');++# Check that no corruption is reported if we exclude corrupt schemas+$node->command_like(+ [ 'pg_amcheck', '-p', $port, 'postgres', '-N', 's1', '-N', 's2', '-N', 's3' ],+ qr/^$/, # Empty+ 'pg_amcheck excluding corrupt schemas reports no corruption');++# Check that no corruption is reported if we exclude corrupt tables+$node->command_like(+ [ 'pg_amcheck', '-p', $port, 'postgres', '-T', 't1', '-T', 't2' ],+ qr/^$/, # Empty+ 'pg_amcheck excluding corrupt tables reports no corruption');diff --git a/contrib/pg_amcheck/t/004_verify_heapam.pl b/contrib/pg_amcheck/t/004_verify_heapam.plnew file mode 100644index 0000000000..da1a48f747--- /dev/null+++ b/contrib/pg_amcheck/t/004_verify_heapam.pl@@ -0,0 +1,426 @@+use strict;+use warnings;++use PostgresNode;+use TestLib;++use Test::More tests => 39;++# This regression test demonstrates that the verify_heapam() function supplied+# with the amcheck contrib module and depended upon by this pg_amcheck contrib+# module correctly identifies specific kinds of corruption within pages. To+# test this, we need a mechanism to create corrupt pages with predictable,+# repeatable corruption. The postgres backend cannot be expected to help us+# with this, as its design is not consistent with the goal of intentionally+# corrupting pages.+#+# Instead, we create a table to corrupt, and with careful consideration of how+# postgresql lays out heap pages, we seek to offsets within the page and+# overwrite deliberately chosen bytes with specific values calculated to+# corrupt the page in expected ways. We then verify that verify_heapam+# reports the corruption, and that it runs without crashing. Note that the+# backend cannot simply be started to run queries against the corrupt table, as+# the backend will crash, at least for some of the corruption types we+# generate.+#+# Autovacuum potentially touching the table in the background makes the exact+# behavior of this test harder to reason about. We turn it off to keep things+# simpler. We use a "belt and suspenders" approach, turning it off for the+# system generally in postgresql.conf, and turning it off specifically for the+# test table.+#+# This test depends on the table being written to the heap file exactly as we+# expect it to be, so we take care to arrange the columns of the table, and+# insert rows of the table, that give predictable sizes and locations within+# the table page.+#+# The HeapTupleHeaderData has 23 bytes of fixed size fields before the variable+# length t_bits[] array. We have exactly 3 columns in the table, so natts = 3,+# t_bits is 1 byte long, and t_hoff = MAXALIGN(23 + 1) = 24.+#+# We're not too fussy about which datatypes we use for the test, but we do care+# about some specific properties. We'd like to test both fixed size and+# varlena types. We'd like some varlena data inline and some toasted. And+# we'd like the layout of the table such that the datums land at predictable+# offsets within the tuple. We choose a structure without padding on all+# supported architectures:+#+# a BIGINT+# b TEXT+# c TEXT+#+# We always insert a 7-ascii character string into field 'b', which with a+# 1-byte varlena header gives an 8 byte inline value. We always insert a long+# text string in field 'c', long enough to force toast storage.+#+#+# We choose to read and write binary copies of our table's tuples, using perl's+# pack() and unpack() functions. Perl uses a packing code system in which:+#+# L = "Unsigned 32-bit Long",+# S = "Unsigned 16-bit Short",+# C = "Unsigned 8-bit Octet",+# c = "signed 8-bit octet",+# q = "signed 64-bit quadword"+#+# Each tuple in our table has a layout as follows:+#+# xx xx xx xx t_xmin: xxxx offset = 0 L+# xx xx xx xx t_xmax: xxxx offset = 4 L+# xx xx xx xx t_field3: xxxx offset = 8 L+# xx xx bi_hi: xx offset = 12 S+# xx xx bi_lo: xx offset = 14 S+# xx xx ip_posid: xx offset = 16 S+# xx xx t_infomask2: xx offset = 18 S+# xx xx t_infomask: xx offset = 20 S+# xx t_hoff: x offset = 22 C+# xx t_bits: x offset = 23 C+# xx xx xx xx xx xx xx xx 'a': xxxxxxxx offset = 24 q+# xx xx xx xx xx xx xx xx 'b': xxxxxxxx offset = 32 Cccccccc+# xx xx xx xx xx xx xx xx 'c': xxxxxxxx offset = 40 SSSS+# xx xx xx xx xx xx xx xx : xxxxxxxx ...continued SSSS+# xx xx : xx ...continued S+#+# We could choose to read and write columns 'b' and 'c' in other ways, but+# it is convenient enough to do it this way. We define packing code+# constants here, where they can be compared easily against the layout.++use constant HEAPTUPLE_PACK_CODE => 'LLLSSSSSCCqCcccccccSSSSSSSSS';+use constant HEAPTUPLE_PACK_LENGTH => 58; # Total size++# Read a tuple of our table from a heap page.+#+# Takes an open filehandle to the heap file, and the offset of the tuple.+#+# Rather than returning the binary data from the file, unpacks the data into a+# perl hash with named fields. These fields exactly match the ones understood+# by write_tuple(), below. Returns a reference to this hash.+#+sub read_tuple ($$)+{+ my ($fh, $offset) = @_;+ my ($buffer, %tup);+ seek($fh, $offset, 0);+ sysread($fh, $buffer, HEAPTUPLE_PACK_LENGTH);++ @_ = unpack(HEAPTUPLE_PACK_CODE, $buffer);+ %tup = (t_xmin => shift,+ t_xmax => shift,+ t_field3 => shift,+ bi_hi => shift,+ bi_lo => shift,+ ip_posid => shift,+ t_infomask2 => shift,+ t_infomask => shift,+ t_hoff => shift,+ t_bits => shift,+ a => shift,+ b_header => shift,+ b_body1 => shift,+ b_body2 => shift,+ b_body3 => shift,+ b_body4 => shift,+ b_body5 => shift,+ b_body6 => shift,+ b_body7 => shift,+ c1 => shift,+ c2 => shift,+ c3 => shift,+ c4 => shift,+ c5 => shift,+ c6 => shift,+ c7 => shift,+ c8 => shift,+ c9 => shift);+ # Stitch together the text for column 'b'+ $tup{b} = join('', map { chr($tup{"b_body$_"}) } (1..7));+ return \%tup;+}++# Write a tuple of our table to a heap page.+#+# Takes an open filehandle to the heap file, the offset of the tuple, and a+# reference to a hash with the tuple values, as returned by read_tuple().+# Writes the tuple fields from the hash into the heap file.+#+# The purpose of this function is to write a tuple back to disk with some+# subset of fields modified. The function does no error checking. Use+# cautiously.+#+sub write_tuple($$$)+{+ my ($fh, $offset, $tup) = @_;+ my $buffer = pack(HEAPTUPLE_PACK_CODE,+ $tup->{t_xmin},+ $tup->{t_xmax},+ $tup->{t_field3},+ $tup->{bi_hi},+ $tup->{bi_lo},+ $tup->{ip_posid},+ $tup->{t_infomask2},+ $tup->{t_infomask},+ $tup->{t_hoff},+ $tup->{t_bits},+ $tup->{a},+ $tup->{b_header},+ $tup->{b_body1},+ $tup->{b_body2},+ $tup->{b_body3},+ $tup->{b_body4},+ $tup->{b_body5},+ $tup->{b_body6},+ $tup->{b_body7},+ $tup->{c1},+ $tup->{c2},+ $tup->{c3},+ $tup->{c4},+ $tup->{c5},+ $tup->{c6},+ $tup->{c7},+ $tup->{c8},+ $tup->{c9});+ seek($fh, $offset, 0);+ syswrite($fh, $buffer, HEAPTUPLE_PACK_LENGTH);+ return;+}++# Set umask so test directories and files are created with default permissions+umask(0077);++# Set up the node. Once we create and corrupt the table,+# autovacuum workers visiting the table could crash the backend.+# Disable autovacuum so that won't happen.+my $node = get_new_node('test');+$node->init;+$node->append_conf('postgresql.conf', 'autovacuum=off');++# Start the node and load the extensions. We depend on both+# amcheck and pageinspect for this test.+$node->start;+my $port = $node->port;+my $pgdata = $node->data_dir;+$node->safe_psql('postgres', "CREATE EXTENSION amcheck");+$node->safe_psql('postgres', "CREATE EXTENSION pageinspect");++# Create the test table with precisely the schema that our+# corruption function expects.+$node->safe_psql(+ 'postgres', qq(+ CREATE TABLE public.test (a BIGINT, b TEXT, c TEXT);+ ALTER TABLE public.test SET (autovacuum_enabled=false);+ ALTER TABLE public.test ALTER COLUMN c SET STORAGE EXTERNAL;+ CREATE INDEX test_idx ON public.test(a, b);+ ));++my $rel = $node->safe_psql('postgres', qq(SELECT pg_relation_filepath('public.test')));+my $relpath = "$pgdata/$rel";++use constant ROWCOUNT => 14;+$node->safe_psql('postgres', qq(+ INSERT INTO public.test (a, b, c)+ VALUES (+ 12345678,+ 'abcdefg',+ repeat('w', 10000)+ );+ VACUUM FREEZE public.test+ )) for (1..ROWCOUNT);++my $relfrozenxid = $node->safe_psql('postgres',+ q(select relfrozenxid from pg_class where relname = 'test'));++# Find where each of the tuples is located on the page.+my @lp_off;+for my $tup (0..ROWCOUNT-1)+{+ push (@lp_off, $node->safe_psql('postgres', qq(+select lp_off from heap_page_items(get_raw_page('test', 'main', 0))+ offset $tup limit 1)));+}++# Check that pg_amcheck runs against the uncorrupted table without error.+$node->command_ok(['pg_amcheck', '-p', $port, 'postgres'],+ 'pg_amcheck test table, prior to corruption');++# Check that pg_amcheck runs against the uncorrupted table and index without error.+$node->command_ok(['pg_amcheck', '-p', $port, 'postgres'],+ 'pg_amcheck test table and index, prior to corruption');++$node->stop;++# Some #define constants from access/htup_details.h for use while corrupting.+use constant HEAP_HASNULL => 0x0001;+use constant HEAP_XMAX_LOCK_ONLY => 0x0080;+use constant HEAP_XMIN_COMMITTED => 0x0100;+use constant HEAP_XMIN_INVALID => 0x0200;+use constant HEAP_XMAX_COMMITTED => 0x0400;+use constant HEAP_XMAX_INVALID => 0x0800;+use constant HEAP_NATTS_MASK => 0x07FF;+use constant HEAP_XMAX_IS_MULTI => 0x1000;+use constant HEAP_KEYS_UPDATED => 0x2000;++# Corrupt the tuples, one type of corruption per tuple. Some types of+# corruption cause verify_heapam to skip to the next tuple without+# performing any remaining checks, so we can't exercise the system properly if+# we focus all our corruption on a single tuple.+#+my $file;+open($file, '+<', $relpath);+binmode $file;++for (my $tupidx = 0; $tupidx < ROWCOUNT; $tupidx++)+{+ my $offset = $lp_off[$tupidx];+ my $tup = read_tuple($file, $offset);++ # Sanity-check that the data appears on the page where we expect.+ if ($tup->{a} ne '12345678' || $tup->{b} ne 'abcdefg')+ {+ fail('Page layout differs from our expectations');+ $node->clean_node;+ exit;+ }++ if ($tupidx == 0)+ {+ # Corruptly set xmin < relfrozenxid+ $tup->{t_xmin} = 3;+ $tup->{t_infomask} &= ~HEAP_XMIN_COMMITTED;+ $tup->{t_infomask} &= ~HEAP_XMIN_INVALID;+ }+ elsif ($tupidx == 1)+ {+ # Corruptly set xmin < relfrozenxid, further back+ $tup->{t_xmin} = 4026531839; # Note circularity of xid comparison+ $tup->{t_infomask} &= ~HEAP_XMIN_COMMITTED;+ $tup->{t_infomask} &= ~HEAP_XMIN_INVALID;+ }+ elsif ($tupidx == 2)+ {+ # Corruptly set xmax < relminmxid;+ $tup->{t_xmax} = 4026531839; # Note circularity of xid comparison+ $tup->{t_infomask} &= ~HEAP_XMAX_INVALID;+ }+ elsif ($tupidx == 3)+ {+ # Corrupt the tuple t_hoff, but keep it aligned properly+ $tup->{t_hoff} += 128;+ }+ elsif ($tupidx == 4)+ {+ # Corrupt the tuple t_hoff, wrong alignment+ $tup->{t_hoff} += 3;+ }+ elsif ($tupidx == 5)+ {+ # Corrupt the tuple t_hoff, underflow but correct alignment+ $tup->{t_hoff} -= 8;+ }+ elsif ($tupidx == 6)+ {+ # Corrupt the tuple t_hoff, underflow and wrong alignment+ $tup->{t_hoff} -= 3;+ }+ elsif ($tupidx == 7)+ {+ # Corrupt the tuple to look like it has lots of attributes, not just 3+ $tup->{t_infomask2} |= HEAP_NATTS_MASK;+ }+ elsif ($tupidx == 8)+ {+ # Corrupt the tuple to look like it has lots of attributes, some of+ # them null. This falsely creates the impression that the t_bits+ # array is longer than just one byte, but t_hoff still says otherwise.+ $tup->{t_infomask} |= HEAP_HASNULL;+ $tup->{t_infomask2} |= HEAP_NATTS_MASK;+ $tup->{t_bits} = 0xAA;+ }+ elsif ($tupidx == 9)+ {+ # Same as above, but this time t_hoff plays along+ $tup->{t_infomask} |= HEAP_HASNULL;+ $tup->{t_infomask2} |= (HEAP_NATTS_MASK & 0x40);+ $tup->{t_bits} = 0xAA;+ $tup->{t_hoff} = 32;+ }+ elsif ($tupidx == 10)+ {+ # Corrupt the bits in column 'b' 1-byte varlena header+ $tup->{b_header} = 0x80;+ }+ elsif ($tupidx == 11)+ {+ # Corrupt the bits in column 'c' toast pointer+ $tup->{c6} = 41;+ $tup->{c7} = 41;+ }+ elsif ($tupidx == 12)+ {+ # Set both HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED+ $tup->{t_infomask} |= HEAP_XMAX_LOCK_ONLY;+ $tup->{t_infomask2} |= HEAP_KEYS_UPDATED;+ }+ elsif ($tupidx == 13)+ {+ # Set both HEAP_XMAX_COMMITTED and HEAP_XMAX_IS_MULTI+ $tup->{t_infomask} |= HEAP_XMAX_COMMITTED;+ $tup->{t_infomask} |= HEAP_XMAX_IS_MULTI;+ }+ write_tuple($file, $offset, $tup);+}+close($file);++# Run verify_heapam on the corrupted file+$node->start;++my $result = $node->safe_psql(+ 'postgres',+ q(SELECT * FROM verify_heapam('test', on_error_stop := false, skip := NULL, check_toast := true, startblock := NULL, endblock := NULL)));+is ($result,+"0|1||inserting transaction ID is from before freeze cutoff: 3 vs. $relfrozenxid+0|2||inserting transaction ID is from before freeze cutoff: 4026531839 vs. $relfrozenxid+0|3||updating transaction ID is from before relation cutoff: 4026531839 vs. $relfrozenxid+0|4||data begins at offset beyond the tuple length: 152 vs. 58+0|4||data offset differs from expected: 152 vs. 24 (3 attributes, no nulls)+0|5||data offset differs from expected: 27 vs. 24 (3 attributes, no nulls)+0|6||data offset differs from expected: 16 vs. 24 (3 attributes, no nulls)+0|7||data offset differs from expected: 21 vs. 24 (3 attributes, no nulls)+0|8||number of attributes exceeds maximum expected for table: 2047 vs. 3+0|9||data offset differs from expected: 24 vs. 280 (2047 attributes, has nulls)+0|10||number of attributes exceeds maximum expected for table: 67 vs. 3+0|11|1|attribute ends at offset beyond total tuple length: 416848000 vs. 58 (attribute length 4294967295)+0|12|2|final toast chunk number differs from expected value: 0 vs. 6+0|12|2|toasted value missing from toast table+0|13||updating transaction ID marked incompatibly as keys updated and locked only+0|14||multitransaction ID is from before relation cutoff: 0 vs. 1",+"Expected verify_heapam output");++# Each table corruption message is returned with a standard header, and we can+# check for those headers to verify that corruption is being reported. We can+# also check for each individual corruption that we would expect to see.+my @corruption_re = (++ # standard header+ qr/relname=test,blkno=\d*,offnum=\d*,attnum=\d*/,++ # individual detected corruptions+ qr/attribute ends at offset beyond total tuple length: \d+ vs. \d+ \(attribute length \d+\)/,+ qr/data begins at offset beyond the tuple length: \d+ vs. \d+/,+ qr/data offset differs from expected: \d+ vs. \d+ \(\d+ attributes, has nulls\)/,+ qr/data offset differs from expected: \d+ vs. \d+ \(\d+ attributes, no nulls\)/,+ qr/final toast chunk number differs from expected value: \d+ vs. \d+/,+ qr/inserting transaction ID is from before freeze cutoff: \d+ vs. \d+/,+ qr/multitransaction ID is from before relation cutoff: \d+ vs. \d+/,+ qr/number of attributes exceeds maximum expected for table: \d+ vs. \d+/,+ qr/toasted value missing from toast table/,+ qr/updating transaction ID is from before relation cutoff: \d+ vs. \d+/,+ qr/updating transaction ID marked incompatibly as keys updated and locked only/,+);++$node->command_like(+ ['pg_amcheck', '--check-toast', '--skip-indexes', '-p', $port, 'postgres'], $_,+ "pg_amcheck reports: $_"+ ) for(@corruption_re);++$node->teardown_node;+$node->clean_node;diff --git a/contrib/pg_amcheck/t/005_opclass_damage.pl b/contrib/pg_amcheck/t/005_opclass_damage.plnew file mode 100644index 0000000000..fdbb1ea402--- /dev/null+++ b/contrib/pg_amcheck/t/005_opclass_damage.pl@@ -0,0 +1,52 @@+# This regression test checks the behavior of the btree validation in the+# presence of breaking sort order changes.+#+use strict;+use warnings;+use PostgresNode;+use TestLib;+use Test::More tests => 6;++my $node = get_new_node('test');+$node->init;+$node->start;++# Create a custom operator class and an index which uses it.+$node->safe_psql('postgres', q(+ CREATE EXTENSION amcheck;++ CREATE FUNCTION int4_asc_cmp (a int4, b int4) RETURNS int LANGUAGE sql AS $$+ SELECT CASE WHEN $1 = $2 THEN 0 WHEN $1 > $2 THEN 1 ELSE -1 END; $$;++ CREATE OPERATOR CLASS int4_fickle_ops FOR TYPE int4 USING btree AS+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),+ OPERATOR 5 > (int4, int4), FUNCTION 1 int4_asc_cmp(int4, int4);++ CREATE TABLE int4tbl (i int4);+ INSERT INTO int4tbl (SELECT * FROM generate_series(1,1000) gs);+ CREATE INDEX fickleidx ON int4tbl USING btree (i int4_fickle_ops);+));++# We have not yet broken the index, so we should get no corruption+$node->command_like(+ [ 'pg_amcheck', '-p', $node->port, 'postgres' ],+ qr/^$/,+ 'pg_amcheck all schemas, tables and indexes reports no corruption');++# Change the operator class to use a function which sorts in a different+# order to corrupt the btree index+$node->safe_psql('postgres', q(+ CREATE FUNCTION int4_desc_cmp (int4, int4) RETURNS int LANGUAGE sql AS $$+ SELECT CASE WHEN $1 = $2 THEN 0 WHEN $1 > $2 THEN -1 ELSE 1 END; $$;+ UPDATE pg_catalog.pg_amproc+ SET amproc = 'int4_desc_cmp'::regproc+ WHERE amproc = 'int4_asc_cmp'::regproc+));++# Index corruption should now be reported+$node->command_like(+ [ 'pg_amcheck', '-x', '-p', $node->port, 'postgres' ],+ qr/item order invariant violated for index "fickleidx"/,+ 'pg_amcheck all schemas, tables and indexes reports fickleidx corruption'+);diff --git a/doc/src/sgml/contrib.sgml b/doc/src/sgml/contrib.sgmlindex 261a559e81..4babcbb39c 100644--- a/doc/src/sgml/contrib.sgml+++ b/doc/src/sgml/contrib.sgml@@ -118,6 +118,7 @@ CREATE EXTENSION <replaceable>module_name</replaceable>;
<ree;
&pageinspect;
&passwordcheck;
+ &pgamcheck;
&pgbuffercache;
&pgcrypto;
&pgfreespacemap;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgmlindex 64b5da0070..2e8588b879 100644--- a/doc/src/sgml/filelist.sgml+++ b/doc/src/sgml/filelist.sgml@@ -131,6 +131,7 @@
<!ENTITY oid2name SYSTEM "oid2name.sgml">
<!ENTITY pageinspect SYSTEM "pageinspect.sgml">
<!ENTITY passwordcheck SYSTEM "passwordcheck.sgml">
+<!ENTITY pgamcheck SYSTEM "pgamcheck.sgml">
<!ENTITY pgbuffercache SYSTEM "pgbuffercache.sgml">
<!ENTITY pgcrypto SYSTEM "pgcrypto.sgml">
<!ENTITY pgfreespacemap SYSTEM "pgfreespacemap.sgml">
diff --git a/doc/src/sgml/pgamcheck.sgml b/doc/src/sgml/pgamcheck.sgmlnew file mode 100644index 0000000000..affe2abf8e--- /dev/null+++ b/doc/src/sgml/pgamcheck.sgml@@ -0,0 +1,229 @@+<!-- doc/src/sgml/pg_amcheck.sgml -->++<sect1 id="pg_amcheck" xreflabel="pg_amcheck">+ <title>pg_amcheck</title>++ <indexterm zone="pg_amcheck">+ <primary>pg_amcheck</primary>+ </indexterm>++ <para>+ The <filename>pg_amcheck</filename> module provides a command line interface+ to the <xref linkend="amcheck"/> corruption checking functionality.+ </para>++ <para>+ <application>pg_amcheck</application> is a regular+ <productname>PostgreSQL</productname> client application. You can perform+ corruption checks from any remote host that has access to the database+ connecting as a user with sufficient privileges to check tables and indexes.+ Currently, this requires superuser privileges.+ </para>++<synopsis>+pg_amcheck [OPTION]... [DBNAME [USERNAME]]+ General options:+ -V, --version output version information, then exit+ -?, --help show this help, then exit+ -s, --strict-names require include patterns to match at least one entity each+ -o, --on-error-stop stop checking at end of first corrupt page++ Schema checking options:+ -n, --schema=PATTERN check relations in the specified schema(s) only+ -N, --exclude-schema=PATTERN do NOT check relations in the specified schema(s)++ Table checking options:+ -t, --table=PATTERN check the specified table(s) only+ -T, --exclude-table=PATTERN do NOT check the specified table(s)+ -b, --startblock begin checking table(s) at the given starting block number+ -e, --endblock check table(s) only up to the given ending block number+ -f, --skip-all-frozen do NOT check blocks marked as all frozen+ -v, --skip-all-visible do NOT check blocks marked as all visible++ TOAST table checking options:+ -z, --check-toast check associated toast tables and toast indexes+ -Z, --skip-toast do NOT check associated toast tables and toast indexes+ -B, --toast-startblock begin checking toast table(s) at the given starting block+ -E, --toast-endblock check toast table(s) only up to the given ending block++ Index checking options:+ -x, --check-indexes check btree indexes associated with tables being checked+ -X, --skip-indexes do NOT check any btree indexes+ -i, --index=PATTERN check the specified index(es) only+ -I, --exclude-index=PATTERN do NOT check the specified index(es)+ -c, --check-corrupt check indexes even if their associated table is corrupt+ -C, --skip-corrupt do NOT check indexes if their associated table is corrupt+ -a, --heapallindexed check index tuples against the table tuples+ -A, --no-heapallindexed do NOT check index tuples against the table tuples+ -r, --rootdescend search from the root page for each index tuple+ -R, --no-rootdescend do NOT search from the root page for each index tuple++ Connection options:+ -d, --dbname=DBNAME database name to connect to+ -h, --host=HOSTNAME database server host or socket directory+ -p, --port=PORT database server port+ -U, --username=USERNAME database user name+ -w, --no-password never prompt for password+ -W, --password force password prompt (should happen automatically)+</synopsis>++ <sect2>+ <title>Options</title>++ <para>+ To specify which database server <application>pg_amcheck</application> should+ contact, use the command line options <option>-h</option> or+ <option>--host</option> and <option>-p</option> or+ <option>port</option>. The default host is the local host+ or whatever your <envar>PGHOST</envar> environment variable specifies.+ Similarly, the default port is indicated by the <envar>PGPORT</envar>+ environment variable or, failing that, by the compiled-in default.+ </para>+ + <para>+ Like any other <productname>PostgreSQL</productname> client application,+ <application>pg_amcheck</application> will by default connect with the+ database user name that is equal to the current operating system user name.+ To override this, either specify the <option>-U</option> option or set the+ environment variable <envar>PGUSER</envar>. Remember that+ <application>pg_amcheck</application> connections are subject to the normal+ client authentication mechanisms (which are described in <xref+ linkend="client-authentication"/>).+ </para>+ + <para>+ To restrict checking of tables and indexes to specific schemas, specify the+ <option>-s</option> or <option>--schema</option> option with a pattern.+ To exclude checking of tables and indexes within specific schemas, specify+ the <option>-N</option> or <option>--exclude-schema</option> option with+ a pattern.+ </para>+ + <para>+ To specify which tables are checked, specify the+ <option>-t</option> or <option>--table</option> option with a pattern.+ To exclude checking of tables, specify the+ <option>-T</option> or <option>--exclude-table</option> option with a+ pattern.+ </para>+ + <para>+ To check indexes associated with checked tables, specify the+ <option>-i</option> or <option>--check-indexes</option> option. Only+ indexes on tables which are being checked will themselves be checked. To+ check all indexes in a database, all tables on which the indexes exist must+ also be checked. This restriction may be relaxed in the future.+ </para>+ + <para>+ To restrict the range of blocks within a table that are checked, specify the+ <option>-b</option> or <option>--startblock</option> and/or+ <option>-e</option> or <option>--endblock</option> options with numeric+ values for the starting and ending block numbers. Although these options+ make the most sense when applied to a single table, if specified along with+ options that select multiple tables, each table check will be restricted to+ the specified blocks. If <option>--startblock</option> is omitted, checking+ begins with the first block. If <option>--endblock</option> is omitted,+ checking continues to the end of the relation.+ </para>+ + <para>+ Some users may wish to periodically check tables without incurring the cost+ of rechecking older table blocks, presumably because those blocks have+ already been checked in the past. There is at present no perfect way to do+ this. Although the <option>--startblock</option> and <option>--endblock</option>+ options can be used to restrict blocks, the user is not expected to have+ perfect knowledge of which blocks have already been checked, and in any+ event, some blocks that were previously checked may have been subject to+ modification since the last check. As an approximation to the desired+ functionality, one can specify the+ <option>-f</option> or <option>--skip-all-frozen</option> option, or+ alternatively the+ <option>-v</option> or <option>--skip-all-visible</option> option to skip+ blocks marked all frozen or all visible, respectively.+ </para>+ </sect2>++ <sect2>+ <title>Example Usage</title>++ <para>+ For table corruption, each detected corruption is reported on two lines, the+ first line shows the location and the second line shows a message describing+ the problem.+ </para>++ <para>+ Checking an entire database which contains one corrupt table, "mytable",+ along with the output:+ </para>++<screen>+% pg_amcheck --check-toast --skip-indexes mydb+(relname=mytable,blkno=0,offnum=1,attnum=)+inserting transaction ID is from before freeze cutoff: 3 vs. 524+(relname=mytable,blkno=0,offnum=2,attnum=)+inserting transaction ID is from before freeze cutoff: 4026531839 vs. 524+(relname=mytable,blkno=0,offnum=3,attnum=)+updating transaction ID is from before relation cutoff: 4026531839 vs. 524+(relname=mytable,blkno=0,offnum=4,attnum=)+data begins at offset beyond the tuple length: 152 vs. 58+(relname=mytable,blkno=0,offnum=4,attnum=)+data offset differs from expected: 152 vs. 24 (3 attributes, no nulls)+(relname=mytable,blkno=0,offnum=5,attnum=)+data offset differs from expected: 27 vs. 24 (3 attributes, no nulls)+(relname=mytable,blkno=0,offnum=6,attnum=)+data offset differs from expected: 16 vs. 24 (3 attributes, no nulls)+(relname=mytable,blkno=0,offnum=7,attnum=)+data offset differs from expected: 21 vs. 24 (3 attributes, no nulls)+(relname=mytable,blkno=0,offnum=8,attnum=)+number of attributes exceeds maximum expected for table: 2047 vs. 3+(relname=mytable,blkno=0,offnum=9,attnum=)+data offset differs from expected: 24 vs. 280 (2047 attributes, has nulls)+(relname=mytable,blkno=0,offnum=10,attnum=)+number of attributes exceeds maximum expected for table: 67 vs. 3+(relname=mytable,blkno=0,offnum=11,attnum=1)+attribute ends at offset beyond total tuple length: 416848000 vs. 58 (attribute length 4294967295)+(relname=mytable,blkno=0,offnum=12,attnum=2)+final toast chunk number differs from expected value: 0 vs. 6+(relname=mytable,blkno=0,offnum=12,attnum=2)+toasted value missing from toast table+(relname=mytable,blkno=0,offnum=13,attnum=)+updating transaction ID marked incompatibly as keys updated and locked only+(relname=mytable,blkno=0,offnum=14,attnum=)+multitransaction ID is from before relation cutoff: 0 vs. 1+</screen>++ <para>+ For index corruption, the output is more free-form, and may span more than+ one line per corruption detected.+ </para>++ <para>+ Checking an entire database which contains one corrupt index,+ "corrupt_index", with corruption in the page header, along with the output:+ </para>++<screen>+% pg_amcheck --check-toast --check-indexes --schema=public --table=table_with_corrupt_index mydb+index check failed for index corrupt_index of table table_with_corrupt_index:+ERROR: XX002: index "corrupt_index" is not a btree+LOCATION: _bt_getmeta, nbtpage.c:152+</screen>++ <para>+ Checking again after rebuilding the index but corrupting the contents,+ along with the output:+ </para>++<screen>+% pg_amcheck --check-toast --check-indexes --schema=public --table=table_with_corrupt_index mydb+index check failed for index corrupt_index of table table_with_corrupt_index:+ERROR: XX002: index tuple size does not equal lp_len in index "corrupt_index"+DETAIL: Index tid=(39,49) tuple size=3373 lp_len=24 page lsn=0/2B548C0.+HINT: This could be a torn page problem.+LOCATION: bt_target_page_check, verify_nbtree.c:1125+</screen>++ </sect2>+</sect1>diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.listindex c48d453793..426da02784 100644--- a/src/tools/pgindent/typedefs.list+++ b/src/tools/pgindent/typedefs.list@@ -102,6 +102,7 @@ AlterUserMappingStmt
AlteredTableInfo
AlternativeSubPlan
AlternativeSubPlanState
+AmCheckSettings
AnalyzeAttrComputeStatsFunc
AnalyzeAttrFetchFunc
AnalyzeForeignTable_function
@@ -402,6 +403,7 @@ ConfigData
ConfigVariable
ConnCacheEntry
ConnCacheKey
+ConnectOptions
ConnStatusType
ConnType
ConnectionStateEnum
--
2.21.1 (Apple Git-122.3)
=
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: mark.dilger@enterprisedb.com, sulamul@gmail.com, robertmhaas@gmail.com, dilipbalaut@gmail.com, pg@bowt.ie, andres@anarazel.de
Subject: Re: new heapcheck contrib module
In-Reply-To: <50F6468F-5A5E-412A-86EC-DF24B549D733@enterprisedb.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox