pg.ddx.io  pgsql-general@postgresql.org mailing list archive  
help / color / mirror / Atom feed
WAL contains references to invalid pages
6+ messages / 4 participants
[nested] [flat]

* WAL contains references to invalid pages
@ 2013-05-16 14:12 JotaComm <jota.comm@gmail.com>
  2013-05-16 14:36 ` Re: WAL contains references to invalid pages Fabrízio de Royes Mello <fabriziomello@gmail.com>
  0 siblings, 1 reply; 6+ messages in thread

From: JotaComm @ 2013-05-16 14:12 UTC (permalink / raw)
  To: pgsql-general

Hello, guys

Yesterday I identified the following messages in my log file (slave):

user=,db= WARNING:  page 6629 of relation base/20449/24818 is uninitialized
user=,db= CONTEXT:  xlog redo vacuum: rel 1663/20449/24818; blk 6631,
lastBlockVacuumed 6626
user=,db= PANIC:  WAL contains references to invalid pages
user=,db= CONTEXT:  xlog redo vacuum: rel 1663/20449/24818; blk 6631,
lastBlockVacuumed 6626
user=,db= LOG:  startup process (PID 26293) was terminated by signal 6:
Aborted
user=,db= LOG:  terminating any other active server processes

Information:

PostgreSQL 9.2.3 (master and slave)

Operational System: CentOS release 6.3 (Final)

The parameter full_page_writes is enabled in both servers.

Analyzing the objects in my cluster (master) I identified the database
[20449] and the relation [24818]. The relation 24818 is an index, so I ran
the command REINDEX to try solving the problem. Immediately after, I tried
to up the slave but I received the same errors.

user=,db= WARNING:  page 6629 of relation base/20449/24818 is uninitialized
user=,db= CONTEXT:  xlog redo vacuum: rel 1663/20449/24818; blk 6631,
lastBlockVacuumed 6626
user=,db= PANIC:  WAL contains references to invalid pages
user=,db= CONTEXT:  xlog redo vacuum: rel 1663/20449/24818; blk 6631,
lastBlockVacuumed 6626
user=,db= LOG:  startup process (PID 26293) was terminated by signal 6:
Aborted
user=,db= LOG:  terminating any other active server processes

As the problem is in the wal file, so the process (above) doesn't work
according my wish.

Any idea?

Thanks a lot.

Regards

-- 
JotaComm
http://jotacomm.wordpress.com

^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: WAL contains references to invalid pages
  2013-05-16 14:12 WAL contains references to invalid pages JotaComm <jota.comm@gmail.com>
@ 2013-05-16 14:36 ` Fabrízio de Royes Mello <fabriziomello@gmail.com>
  2013-05-16 17:44   ` Re: WAL contains references to invalid pages JotaComm <jota.comm@gmail.com>
  0 siblings, 1 reply; 6+ messages in thread

From: Fabrízio de Royes Mello @ 2013-05-16 14:36 UTC (permalink / raw)
  To: JotaComm <jota.comm@gmail.com>; +Cc: pgsql-general

On Thu, May 16, 2013 at 11:12 AM, JotaComm <jota.comm@gmail.com> wrote:

>
> [...]
> Yesterday I identified the following messages in my log file (slave):
>
> user=,db= WARNING:  page 6629 of relation base/20449/24818 is uninitialized
> user=,db= CONTEXT:  xlog redo vacuum: rel 1663/20449/24818; blk 6631,
> lastBlockVacuumed 6626
> user=,db= PANIC:  WAL contains references to invalid pages
> user=,db= CONTEXT:  xlog redo vacuum: rel 1663/20449/24818; blk 6631,
> lastBlockVacuumed 6626
> user=,db= LOG:  startup process (PID 26293) was terminated by signal 6:
> Aborted
> user=,db= LOG:  terminating any other active server processes
>
> Information:
>
> PostgreSQL 9.2.3 (master and slave)
> Operational System: CentOS release 6.3 (Final)
> The parameter full_page_writes is enabled in both servers.
>
> Analyzing the objects in my cluster (master) I identified the database
> [20449] and the relation [24818]. The relation 24818 is an index, so I ran
> the command REINDEX to try solving the problem. Immediately after, I tried
> to up the slave but I received the same errors.
>
> user=,db= WARNING:  page 6629 of relation base/20449/24818 is uninitialized
> user=,db= CONTEXT:  xlog redo vacuum: rel 1663/20449/24818; blk 6631,
> lastBlockVacuumed 6626
> user=,db= PANIC:  WAL contains references to invalid pages
> user=,db= CONTEXT:  xlog redo vacuum: rel 1663/20449/24818; blk 6631,
> lastBlockVacuumed 6626
> user=,db= LOG:  startup process (PID 26293) was terminated by signal 6:
> Aborted
> user=,db= LOG:  terminating any other active server processes
>
> As the problem is in the wal file, so the process (above) doesn't work
> according my wish.
>
> Any idea?
>
>
Hi JotaComm,

IMHO as it is your slave you could just rebuild it.

However if you want to make an attempt to recover you can do:

1) make a physical backup of this cluster
2) in your postgresql.conf set 'zero_damaged_pages = on' [1]
3) start your cluster

I really don't know if it will work, but you can try... :-)


Regards,

[1]
http://www.postgresql.org/docs/9.2/static/runtime-config-developer.html#GUC-ZERO-DAMAGED-PAGES

-- 
Fabrízio de Royes Mello
Consultoria/Coaching PostgreSQL
>> Blog sobre TI: http://fabriziomello.blogspot.com
>> Perfil Linkedin: http://br.linkedin.com/in/fabriziomello
>> Twitter: http://twitter.com/fabriziomello

^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: WAL contains references to invalid pages
  2013-05-16 14:12 WAL contains references to invalid pages JotaComm <jota.comm@gmail.com>
  2013-05-16 14:36 ` Re: WAL contains references to invalid pages Fabrízio de Royes Mello <fabriziomello@gmail.com>
@ 2013-05-16 17:44   ` JotaComm <jota.comm@gmail.com>
  2013-05-21 11:48     ` Re: WAL contains references to invalid pages Adarsh Sharma <eddy.adarsh@gmail.com>
  0 siblings, 1 reply; 6+ messages in thread

From: JotaComm @ 2013-05-16 17:44 UTC (permalink / raw)
  To: Fabrízio Mello <fabriziomello@gmail.com>; +Cc: pgsql-general

Hello, Fabrízio


2013/5/16 Fabrízio de Royes Mello <fabriziomello@gmail.com>

>
> On Thu, May 16, 2013 at 11:12 AM, JotaComm <jota.comm@gmail.com> wrote:
>
>>
>> [...]
>>
>> Yesterday I identified the following messages in my log file (slave):
>>
>> user=,db= WARNING:  page 6629 of relation base/20449/24818 is
>> uninitialized
>> user=,db= CONTEXT:  xlog redo vacuum: rel 1663/20449/24818; blk 6631,
>> lastBlockVacuumed 6626
>> user=,db= PANIC:  WAL contains references to invalid pages
>> user=,db= CONTEXT:  xlog redo vacuum: rel 1663/20449/24818; blk 6631,
>> lastBlockVacuumed 6626
>> user=,db= LOG:  startup process (PID 26293) was terminated by signal 6:
>> Aborted
>> user=,db= LOG:  terminating any other active server processes
>>
>> Information:
>>
>> PostgreSQL 9.2.3 (master and slave)
>> Operational System: CentOS release 6.3 (Final)
>> The parameter full_page_writes is enabled in both servers.
>>
>> Analyzing the objects in my cluster (master) I identified the database
>> [20449] and the relation [24818]. The relation 24818 is an index, so I ran
>> the command REINDEX to try solving the problem. Immediately after, I tried
>> to up the slave but I received the same errors.
>>
>> user=,db= WARNING:  page 6629 of relation base/20449/24818 is
>> uninitialized
>> user=,db= CONTEXT:  xlog redo vacuum: rel 1663/20449/24818; blk 6631,
>> lastBlockVacuumed 6626
>> user=,db= PANIC:  WAL contains references to invalid pages
>> user=,db= CONTEXT:  xlog redo vacuum: rel 1663/20449/24818; blk 6631,
>> lastBlockVacuumed 6626
>> user=,db= LOG:  startup process (PID 26293) was terminated by signal 6:
>> Aborted
>> user=,db= LOG:  terminating any other active server processes
>>
>> As the problem is in the wal file, so the process (above) doesn't work
>> according my wish.
>>
>> Any idea?
>>
>>
> Hi JotaComm,
>
> IMHO as it is your slave you could just rebuild it.
>
> However if you want to make an attempt to recover you can do:
>
> 1) make a physical backup of this cluster
> 2) in your postgresql.conf set 'zero_damaged_pages = on' [1]
> 3) start your cluster
>
> I really don't know if it will work, but you can try... :-)
>

Thanks for your suggestion :)

I tried it and I had the same errors. I believe that will be necessary to
rebuild the cluster, because the problem is in the wal file.


>
> Regards,
>
> [1]
> http://www.postgresql.org/docs/9.2/static/runtime-config-developer.html#GUC-ZERO-DAMAGED-PAGES
>
> --
> Fabrízio de Royes Mello
> Consultoria/Coaching PostgreSQL
> >> Blog sobre TI: http://fabriziomello.blogspot.com
> >> Perfil Linkedin: http://br.linkedin.com/in/fabriziomello
> >> Twitter: http://twitter.com/fabriziomello
>


Regards
-- 
JotaComm
http://jotacomm.wordpress.com

^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: WAL contains references to invalid pages
  2013-05-16 14:12 WAL contains references to invalid pages JotaComm <jota.comm@gmail.com>
  2013-05-16 14:36 ` Re: WAL contains references to invalid pages Fabrízio de Royes Mello <fabriziomello@gmail.com>
  2013-05-16 17:44   ` Re: WAL contains references to invalid pages JotaComm <jota.comm@gmail.com>
@ 2013-05-21 11:48     ` Adarsh Sharma <eddy.adarsh@gmail.com>
  2013-05-24 12:46       ` Re: WAL contains references to invalid pages JotaComm <jota.comm@gmail.com>
  0 siblings, 1 reply; 6+ messages in thread

From: Adarsh Sharma @ 2013-05-21 11:48 UTC (permalink / raw)
  To: JotaComm <jota.comm@gmail.com>; +Cc: Fabrízio Mello <fabriziomello@gmail.com>; pgsql-general

Try to take backups of that table & index only. If succeeded drop and
recreate them. May be it fix your issue.

Thanks


On Thu, May 16, 2013 at 11:14 PM, JotaComm <jota.comm@gmail.com> wrote:

> Hello, Fabrízio
>
>
> 2013/5/16 Fabrízio de Royes Mello <fabriziomello@gmail.com>
>
>>
>> On Thu, May 16, 2013 at 11:12 AM, JotaComm <jota.comm@gmail.com> wrote:
>>
>>>
>>> [...]
>>>
>>> Yesterday I identified the following messages in my log file (slave):
>>>
>>> user=,db= WARNING:  page 6629 of relation base/20449/24818 is
>>> uninitialized
>>> user=,db= CONTEXT:  xlog redo vacuum: rel 1663/20449/24818; blk 6631,
>>> lastBlockVacuumed 6626
>>> user=,db= PANIC:  WAL contains references to invalid pages
>>> user=,db= CONTEXT:  xlog redo vacuum: rel 1663/20449/24818; blk 6631,
>>> lastBlockVacuumed 6626
>>> user=,db= LOG:  startup process (PID 26293) was terminated by signal 6:
>>> Aborted
>>> user=,db= LOG:  terminating any other active server processes
>>>
>>> Information:
>>>
>>> PostgreSQL 9.2.3 (master and slave)
>>> Operational System: CentOS release 6.3 (Final)
>>> The parameter full_page_writes is enabled in both servers.
>>>
>>> Analyzing the objects in my cluster (master) I identified the database
>>> [20449] and the relation [24818]. The relation 24818 is an index, so I ran
>>> the command REINDEX to try solving the problem. Immediately after, I tried
>>> to up the slave but I received the same errors.
>>>
>>> user=,db= WARNING:  page 6629 of relation base/20449/24818 is
>>> uninitialized
>>> user=,db= CONTEXT:  xlog redo vacuum: rel 1663/20449/24818; blk 6631,
>>> lastBlockVacuumed 6626
>>> user=,db= PANIC:  WAL contains references to invalid pages
>>> user=,db= CONTEXT:  xlog redo vacuum: rel 1663/20449/24818; blk 6631,
>>> lastBlockVacuumed 6626
>>> user=,db= LOG:  startup process (PID 26293) was terminated by signal 6:
>>> Aborted
>>> user=,db= LOG:  terminating any other active server processes
>>>
>>> As the problem is in the wal file, so the process (above) doesn't work
>>> according my wish.
>>>
>>> Any idea?
>>>
>>>
>> Hi JotaComm,
>>
>> IMHO as it is your slave you could just rebuild it.
>>
>> However if you want to make an attempt to recover you can do:
>>
>> 1) make a physical backup of this cluster
>> 2) in your postgresql.conf set 'zero_damaged_pages = on' [1]
>> 3) start your cluster
>>
>> I really don't know if it will work, but you can try... :-)
>>
>
> Thanks for your suggestion :)
>
> I tried it and I had the same errors. I believe that will be necessary to
> rebuild the cluster, because the problem is in the wal file.
>
>
>>
>> Regards,
>>
>> [1]
>> http://www.postgresql.org/docs/9.2/static/runtime-config-developer.html#GUC-ZERO-DAMAGED-PAGES
>>
>> --
>> Fabrízio de Royes Mello
>> Consultoria/Coaching PostgreSQL
>> >> Blog sobre TI: http://fabriziomello.blogspot.com
>> >> Perfil Linkedin: http://br.linkedin.com/in/fabriziomello
>> >> Twitter: http://twitter.com/fabriziomello
>>
>
>
> Regards
> --
> JotaComm
> http://jotacomm.wordpress.com
>

^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: WAL contains references to invalid pages
  2013-05-16 14:12 WAL contains references to invalid pages JotaComm <jota.comm@gmail.com>
  2013-05-16 14:36 ` Re: WAL contains references to invalid pages Fabrízio de Royes Mello <fabriziomello@gmail.com>
  2013-05-16 17:44   ` Re: WAL contains references to invalid pages JotaComm <jota.comm@gmail.com>
  2013-05-21 11:48     ` Re: WAL contains references to invalid pages Adarsh Sharma <eddy.adarsh@gmail.com>
@ 2013-05-24 12:46       ` JotaComm <jota.comm@gmail.com>
  0 siblings, 0 replies; 6+ messages in thread

From: JotaComm @ 2013-05-24 12:46 UTC (permalink / raw)
  To: pgsql-general

Hello,
>
>
>
>
> 2013/5/21 Adarsh Sharma <eddy.adarsh@gmail.com>
>
>> Try to take backups of that table & index only. If succeeded drop and
>> recreate them. May be it fix your issue.
>>
>
> On Monday night I made the slave server. Yesterday I was analyzing the log
> files and I found the following messages.
>
> 2013-05-21 15:13:48 BRT [30686]: [25-1] user=,db= WARNING:  page 136714 of
> relation base/79251/79262 is uninitialized
> 2013-05-21 15:13:48 BRT [30686]: [26-1] user=,db= CONTEXT:  xlog redo
> visible: rel 1663/79251/79262; blk 136714
> 2013-05-21 15:13:48 BRT [30686]: [27-1] user=,db= PANIC:  WAL contains
> references to invalid pages
> 2013-05-21 15:13:48 BRT [30686]: [28-1] user=,db= CONTEXT:  xlog redo
> visible: rel 1663/79251/79262; blk 136714
> 2013-05-21 15:13:49 BRT [30684]: [2-1] user=,db= LOG:  startup process
> (PID 30686) was terminated by signal 6: Aborted
> 2013-05-21 15:13:49 BRT [30684]: [3-1] user=,db= LOG:  terminating any
> other active server processes
>
> It's the same problem, but now is in another table.
>
> According the documentation:
> http://www.postgresql.org/docs/9.2/interactive/release-9-2-3.html
>
>
>    -
>
>    Fix multiple problems in detection of when a consistent database state
>    has been reached during WAL replay (Fujii Masao, Heikki Linnakangas, Simon
>    Riggs, Andres Freund)
>    -
>
>    Fix detection of end-of-backup point when no actual redo work is
>    required (Heikki Linnakangas)
>
>    This mistake could result in incorrect "WAL ends before end of online
>    backup" errors.
>
>
> I believe that my problem is described here. What do you think about it?
>
>
>>
>>
>>
>> On Thu, May 16, 2013 at 11:14 PM, JotaComm <jota.comm@gmail.com> wrote:
>>
>>> Hello, Fabrízio
>>>
>>>
>>> 2013/5/16 Fabrízio de Royes Mello <fabriziomello@gmail.com>
>>>
>>>>
>>>> On Thu, May 16, 2013 at 11:12 AM, JotaComm <jota.comm@gmail.com> wrote:
>>>>
>>>>>
>>>>> [...]
>>>>>
>>>>> Yesterday I identified the following messages in my log file (slave):
>>>>>
>>>>> user=,db= WARNING:  page 6629 of relation base/20449/24818 is
>>>>> uninitialized
>>>>> user=,db= CONTEXT:  xlog redo vacuum: rel 1663/20449/24818; blk 6631,
>>>>> lastBlockVacuumed 6626
>>>>> user=,db= PANIC:  WAL contains references to invalid pages
>>>>> user=,db= CONTEXT:  xlog redo vacuum: rel 1663/20449/24818; blk 6631,
>>>>> lastBlockVacuumed 6626
>>>>> user=,db= LOG:  startup process (PID 26293) was terminated by signal
>>>>> 6: Aborted
>>>>> user=,db= LOG:  terminating any other active server processes
>>>>>
>>>>> Information:
>>>>>
>>>>> PostgreSQL 9.2.3 (master and slave)
>>>>> Operational System: CentOS release 6.3 (Final)
>>>>> The parameter full_page_writes is enabled in both servers.
>>>>>
>>>>> Analyzing the objects in my cluster (master) I identified the database
>>>>> [20449] and the relation [24818]. The relation 24818 is an index, so I ran
>>>>> the command REINDEX to try solving the problem. Immediately after, I tried
>>>>> to up the slave but I received the same errors.
>>>>>
>>>>> user=,db= WARNING:  page 6629 of relation base/20449/24818 is
>>>>> uninitialized
>>>>> user=,db= CONTEXT:  xlog redo vacuum: rel 1663/20449/24818; blk 6631,
>>>>> lastBlockVacuumed 6626
>>>>> user=,db= PANIC:  WAL contains references to invalid pages
>>>>> user=,db= CONTEXT:  xlog redo vacuum: rel 1663/20449/24818; blk 6631,
>>>>> lastBlockVacuumed 6626
>>>>> user=,db= LOG:  startup process (PID 26293) was terminated by signal
>>>>> 6: Aborted
>>>>> user=,db= LOG:  terminating any other active server processes
>>>>>
>>>>> As the problem is in the wal file, so the process (above) doesn't work
>>>>> according my wish.
>>>>>
>>>>> Any idea?
>>>>>
>>>>>
>>>> Hi JotaComm,
>>>>
>>>> IMHO as it is your slave you could just rebuild it.
>>>>
>>>> However if you want to make an attempt to recover you can do:
>>>>
>>>> 1) make a physical backup of this cluster
>>>> 2) in your postgresql.conf set 'zero_damaged_pages = on' [1]
>>>> 3) start your cluster
>>>>
>>>> I really don't know if it will work, but you can try... :-)
>>>>
>>>
>>> Thanks for your suggestion :)
>>>
>>>  I tried it and I had the same errors. I believe that will be necessary
>>> to rebuild the cluster, because the problem is in the wal file.
>>>
>>>
>>>>
>>>> Regards,
>>>>
>>>> [1]
>>>> http://www.postgresql.org/docs/9.2/static/runtime-config-developer.html#GUC-ZERO-DAMAGED-PAGES
>>>>
>>>> --
>>>> Fabrízio de Royes Mello
>>>> Consultoria/Coaching PostgreSQL
>>>> >> Blog sobre TI: http://fabriziomello.blogspot.com
>>>> >> Perfil Linkedin: http://br.linkedin.com/in/fabriziomello
>>>> >> Twitter: http://twitter.com/fabriziomello
>>>>
>>>
>>>
>>> Regards
>>> --
>>> JotaComm
>>> http://jotacomm.wordpress.com
>>>
>>
>>
>
> Thanks a lot
>
> Regards
>
> --
> JotaComm
> http://jotacomm.wordpress.com
>

Thank you

Regards

-- 
JotaComm
http://jotacomm.wordpress.com

^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* WAL contains references to invalid pages in hot standby
@ 2022-11-26 16:06 Andrus <kobruleht2@hot.ee>
  0 siblings, 0 replies; 6+ messages in thread

From: Andrus @ 2022-11-26 16:06 UTC (permalink / raw)
  To: pgsql-general

Hi!

WAL playback in hot standby server crashes  with "WAL contains 
references to invalid pages" error :

2022-11-26 17:48:12.889 EET [497] LOG:  restored log file 
"000000010000057900000064" from archive
2022-11-26 17:48:20.897 EET [497] LOG:  restored log file 
"000000010000057900000065" from archive
2022-11-26 17:48:26.564 EET [497] LOG:  restored log file 
"000000010000057900000066" from archive
2022-11-26 17:48:32.019 EET [497] LOG:  consistent recovery state 
reached at 579/66954858
2022-11-26 17:48:32.019 EET [495] LOG:  database system is ready to 
accept read only connections
2022-11-26 17:48:32.019 EET [497] WARNING:  page 11 of relation 
base/54455050/83221012 is uninitialized
2022-11-26 17:48:32.019 EET [497] CONTEXT:  WAL redo at 579/66967DB0 for 
Heap2/VISIBLE: cutoff xid 167913422 flags 0x01
2022-11-26 17:48:32.019 EET [497] PANIC:  WAL contains references to 
invalid pages
2022-11-26 17:48:32.019 EET [497] CONTEXT:  WAL redo at 579/66967DB0 for 
Heap2/VISIBLE: cutoff xid 167913422 flags 0x01
2022-11-26 17:48:34.315 EET [495] LOG:  startup process (PID 497) was 
terminated by signal 6: Aborted
2022-11-26 17:48:34.315 EET [495] LOG:  terminating any other active 
server processes
2022-11-26 17:48:34.330 EET [495] LOG:  database system is shut down

It looks like file 000000010000057900000067 is corrupted. I looked into 
this file and it contains data like other WAL files.

How to fix this error so that Wal playback can continue?

Master server is in Postgres 12 in Debian 10.3

Hot standby in Postgres 12 in WSL Debian 11.5

Can Debian version 10.3/11.5 difference cause this exception?

Andrus.

^ permalink  raw  reply  [nested|flat] 6+ messages in thread


end of thread, other threads:[~2022-11-26 16:06 UTC | newest]

Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2013-05-16 14:12 WAL contains references to invalid pages JotaComm <jota.comm@gmail.com>
2013-05-16 14:36 ` Fabrízio de Royes Mello <fabriziomello@gmail.com>
2013-05-16 17:44   ` JotaComm <jota.comm@gmail.com>
2013-05-21 11:48     ` Adarsh Sharma <eddy.adarsh@gmail.com>
2013-05-24 12:46       ` JotaComm <jota.comm@gmail.com>
2022-11-26 16:06 WAL contains references to invalid pages in hot standby Andrus <kobruleht2@hot.ee>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox