From: Heikki Linnakangas <hlinnaka@iki.fi>
Subject: prion failed with ERROR: missing chunk number 0 for toast value 14334
 in pg_toast_2619
To: pgsql-hackers <pgsql-hackers@postgresql.org>
Cc: Justin Pryzby <pryzby@telsasoft.com>
Message-ID: <f03ea04a-9b77-e371-9ab9-182cb35db1f9@iki.fi>
Date: Mon, 23 Nov 2020 12:10:31 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.4.0
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Precedence: bulk

Hi,

After my commit c532d15ddd to split up copy.c, buildfarm animal "prion" 
failed in pg_upgrade 
(https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prion&dt=2020-11-23%2009%3A23%3A16):

> Upgrade Complete
> ----------------
> Optimizer statistics are not transferred by pg_upgrade so,
> once you start the new server, consider running:
>     /home/ec2-user/bf/root/HEAD/pgsql.build/tmp_install/home/ec2-user/bf/root/HEAD/inst/bin/vacuumdb --all --analyze-in-stages
> 
> Running this script will delete the old cluster's data files:
>     ./delete_old_cluster.sh
> waiting for server to start.... done
> server started
> pg_dump: error: query failed: ERROR:  missing chunk number 0 for toast value 14334 in pg_toast_2619
> pg_dump: error: query was: SELECT
> a.attnum,
> a.attname,
> a.atttypmod,
> a.attstattarget,
> a.attstorage,
> t.typstorage,
> a.attnotnull,
> a.atthasdef,
> a.attisdropped,
> a.attlen,
> a.attalign,
> a.attislocal,
> pg_catalog.format_type(t.oid, a.atttypmod) AS atttypname,
> array_to_string(a.attoptions, ', ') AS attoptions,
> CASE WHEN a.attcollation <> t.typcollation THEN a.attcollation ELSE 0 END AS attcollation,
> pg_catalog.array_to_string(ARRAY(SELECT pg_catalog.quote_ident(option_name) || ' ' || pg_catalog.quote_literal(option_value) FROM pg_catalog.pg_options_to_table(attfdwoptions) ORDER BY option_name), E',
>     ') AS attfdwoptions,
> a.attidentity,
> CASE WHEN a.atthasmissing AND NOT a.attisdropped THEN a.attmissingval ELSE null END AS attmissingval,
> a.attgenerated
> FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t ON a.atttypid = t.oid
> WHERE a.attrelid = '35221'::pg_catalog.oid AND a.attnum > 0::pg_catalog.int2
> ORDER BY a.attnum
> pg_dumpall: error: pg_dump failed on database "regression", exiting
> waiting for server to shut down.... done
> server stopped
> pg_dumpall of post-upgrade database cluster failed
> make: *** [check] Error 1

I don't think this is related to commit c532d15ddd, as there is no COPY 
involved here. I have no clue what might might be going on here, though. 
Any ideas?

Googling around, I found one report that looks somewhat similar: 
https://www.postgresql.org/message-id/20190830142655.GA14011%40telsasoft.com. 
That was with CLUSTER/VACUUM FULL, while pg_upgrade performs ANALYZE.

- Heikki