Date: Thu, 5 Oct 2023 11:45:18 -0500
From: Justin Pryzby <pryzby@telsasoft.com>
To: pgsql-hackers@lists.postgresql.org
Subject: pg16: invalid page/page verification failed
Message-ID: <ZR7oHreoXMN+Dvq2@pryzbyj2023>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Archived-At: <https://www.postgresql.org/message-id/ZR7oHreoXMN%2BDvq2%40pryzbyj2023>
Precedence: bulk

On an instance running pg16.0:

log_time         | 2023-10-05 10:03:00.014-05
backend_type     | autovacuum worker
left             | page verification failed, calculated checksum 5074 but expected 5050
context          | while scanning block 119 of relation "public.postgres_log_2023_10_05_0900"

This is the only error I've seen so far, and for all I know there's a
issue on the storage behind the VM, or a cosmic ray hit.  But I moved
the table out of the way and saved a copy of get_raw_page() in case
someone wants to ask about it.

 public | BROKEN_postgres_log_2023_10_05_0900 | table | postgres | permanent   | heap | 1664 kB

This table is what it sounds like: a partition into which CSV logs are
COPY'ed.  It would've been created around 8am.  There's no special
params set for the table nor for autovacuum.

Although we have a ZFS tablespace, these tables aren't on it, and
full_page_writes=on.  There's no crashes, and the instance has been up
since it was pg_upgraded from v15.4 on sep25.

pg_stat_all_tables indicates that the table was never (successfully)
vacuumed.

This was compiled to RPM on centos7, and might include a few commits
made since v16.0.

postgres=# SELECT * FROM heap_page_item_attrs(get_raw_page(801594131::regclass::text, 119), 801594131);
 lp  | lp_off | lp_flags | lp_len | t_xmin | t_xmax | t_field3 | t_ctid | t_infomask2 | t_infomask | t_hoff | t_bits | t_oid | t_attrs 
   1 |   2304 |        1 |     16 |        |        |          |        |             |            |        |        |       | 
   2 |   8160 |        1 |     16 |        |        |          |        |             |            |        |        |       | 
   3 |   8144 |        1 |     16 |        |        |          |        |             |            |        |        |       | 
...all the same except for lp_off...
 365 |   2352 |        1 |     16 |        |        |          |        |             |            |        |        |       | 
 366 |   2336 |        1 |     16 |        |        |          |        |             |            |        |        |       | 
 367 |   2320 |        1 |     16 |        |        |          |        |             |            |        |        |       | 

postgres=# SELECT FROM (SELECT tuple_data_split(801594131, t_data, t_infomask, t_infomask2, t_bits) a FROM heap_page_items(get_raw_page(801594131::regclass::text, 119))) WHERE a IS NOT NULL;
WARNING:  page verification failed, calculated checksum 5074 but expected 5050
(0 rows)

-- 
Justin