From: Tom Lane <tgl@sss.pgh.pa.us>
To: Arjun Ranade <ranade@nodalexchange.com>
cc: pgsql-admin@postgresql.org
Subject: Re: Missing Chunk Error when doing a VACUUM FULL operation - DB
 Corruption?
In-reply-to: 
 <CANrrCRy1_mu7A31Tas+-oKWcYK4kpNK8wTFW7VWVq+5yj2uApQ@mail.gmail.com>
References: 
 <CANrrCRwH9ZnW+XtOxozEUPqWuHrfNNqDw9K3H69k513i7-YryQ@mail.gmail.com>
 <13631.1509479942@sss.pgh.pa.us>
 <CANrrCRy-qdDMX=a=xQsC9bbv36kD5biKUK3RgpNxUO9M-VfcTA@mail.gmail.com>
 <23996.1509486974@sss.pgh.pa.us>
 <CANrrCRyd81iKsjpjVpptBG+jzLxHt8rfWk9oj-tRDh7Agfbp8A@mail.gmail.com>
 <29089.1509487959@sss.pgh.pa.us>
 <CANrrCRxcFdXQvae7vrtWttHtXbVmB8yoYEt+2DhfADqm1fvb-g@mail.gmail.com>
 <CANrrCRy1_mu7A31Tas+-oKWcYK4kpNK8wTFW7VWVq+5yj2uApQ@mail.gmail.com>
Comments: In-reply-to Arjun Ranade <ranade@nodalexchange.com>
	message dated "Wed, 01 Nov 2017 21:49:20 -0400"
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <19335.1509589417.1@sss.pgh.pa.us>
Date: Wed, 01 Nov 2017 22:23:37 -0400
Message-ID: <19336.1509589417@sss.pgh.pa.us>
Precedence: bulk
Sender: pgsql-admin-owner@postgresql.org

Arjun Ranade <ranade@nodalexchange.com> writes:
> So we had a maintenance window scheduled today and I restarted the DB.
> When it came back up, it reported the same toast error when vacuum'ing
> pg_statistic.  It also reported the "oldest xmin is far in the past" error.

> I even brought the DB into single user mode and tried to manually vacuum
> pg_statistic but got the same error.  I also tried to TRUNCATE pg_statistic
> in single user mode which failed (see below).

> At this point, I am not sure what to make of the issue.  The DB is still
> complaining about old XMIN even when there are no users connected to it
> after a fresh restart:

Clearly, something is holding back the xmin horizon, and if the behavior
persisted past a DB restart then it surely isn't a regular transaction.

Are you *really* sure there are no prepared transactions?

It now occurs to me that there's one other mechanism that could hold back
the xmin horizon, and that is replication slots.  Looking into the
pg_replication_slots view should show you what's up there.  I'm not
an expert on replication slots, but I do know that they were new as of
9.4.  I am thinking maybe your replication setup has "leaked" a slot
that's not being used but is still preventing global xmin from advancing.

			regards, tom lane


-- 
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin