pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedwal exist in slave but getting err requested WAL segment has already been removed
7+ messages / 3 participants
[nested] [flat]
* wal exist in slave but getting err requested WAL segment has already been removed
@ 2018-07-11 13:09 Mariel Cherkassky <mariel.cherkassky@gmail.com>
2018-07-11 13:26 ` Re: wal exist in slave but getting err requested WAL segment has already been removed Achilleas Mantzios <achill@matrix.gatewaynet.com>
0 siblings, 1 reply; 7+ messages in thread
From: Mariel Cherkassky @ 2018-07-11 13:09 UTC (permalink / raw)
To: pgsql-admin@lists.postgresql.org
Hi,
I have in my cluster 3 nodes (1 master version 9.6.3+ 2 slaves version
9.6.3). I configured repmgr (with repmgrd active) v 4.0.4.
Suddenly today after a few good weeks I noticed that there is a lag in one
of the slaves and the error in the log indicated that the slave didnt get
the wal :
could not receive data from WAL stream: ERROR: requested WAL segment
0000000900002E61000000BD has already been removed
However, when I check if the wal was recieveed :
postgres=# select
pg_is_in_recovery(),pg_is_xlog_replay_paused(),pg_last_xlog_receive_location(),pg_last_xlog_replay_location();
pg_is_in_recovery | pg_is_xlog_replay_paused |
pg_last_xlog_receive_location | pg_last_xlog_replay_location
-------------------+--------------------------+-------------------------------+------------------------------
t | f | 2E61/BDF5C000
| 2E61/BDF5B930
(1 row)
and I checked in pg_xlog directory :
ls -l ../pg_xlog/0000000900002E61000000BD
-rw------- 1 postgres postgres 16777216 Jul 11 11:13
../pg_xlog/0000000900002E61000000BD
and the xlog is exist.
Now is my question, why the wal wasnt replayed ?
In my repmgr.conf I dont have any parameters regarding recovery just some
basic things. The recovery.conf file in the data directory :
standby_mode = 'on'
primary_conninfo = 'host=xxxxxxx user=repmgr application_name=''psgsqldb2''
connect_timeout=2'
recovery_target_timeline = 'latest'
any idea ?
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: wal exist in slave but getting err requested WAL segment has already been removed
2018-07-11 13:09 wal exist in slave but getting err requested WAL segment has already been removed Mariel Cherkassky <mariel.cherkassky@gmail.com>
@ 2018-07-11 13:26 ` Achilleas Mantzios <achill@matrix.gatewaynet.com>
2018-07-11 13:32 ` Re: wal exist in slave but getting err requested WAL segment has already been removed Mariel Cherkassky <mariel.cherkassky@gmail.com>
0 siblings, 1 reply; 7+ messages in thread
From: Achilleas Mantzios @ 2018-07-11 13:26 UTC (permalink / raw)
To: pgsql-admin@lists.postgresql.org
On 11/07/2018 16:09, Mariel Cherkassky wrote:
> Hi,
> I have in my cluster 3 nodes (1 master version 9.6.3+ 2 slaves version 9.6.3). I configured repmgr (with repmgrd active) v 4.0.4.
>
> Suddenly today after a few good weeks I noticed that there is a lag in one of the slaves and the error in the log indicated that the slave didnt get the wal :
>
> could not receive data from WAL stream: ERROR: requested WAL segment 0000000900002E61000000BD has already been removed
>
> However, when I check if the wal was recieveed :
> postgres=# select pg_is_in_recovery(),pg_is_xlog_replay_paused(),pg_last_xlog_receive_location(),pg_last_xlog_replay_location();
> pg_is_in_recovery | pg_is_xlog_replay_paused | pg_last_xlog_receive_location | pg_last_xlog_replay_location
> -------------------+--------------------------+-------------------------------+------------------------------
> t | f | 2E61/BDF5C000 | 2E61/BDF5B930
> (1 row)
>
> and I checked in pg_xlog directory :
> ls -l ../pg_xlog/0000000900002E61000000BD
> -rw------- 1 postgres postgres 16777216 Jul 11 11:13 ../pg_xlog/0000000900002E61000000BD
>
> and the xlog is exist.
In which node did you check for the file?
If the file in the primary is still available, try to compare their md5sum .
If you have a working WAL shipping method in place, then add the appropriate line in the recovery.conf of your standby :
restore_command = 'rsync somemachine:/somepath/pitr/%f "%p" '
> Now is my question, why the wal wasnt replayed ?
> In my repmgr.conf I dont have any parameters regarding recovery just some basic things. The recovery.conf file in the data directory :
>
> standby_mode = 'on'
> primary_conninfo = 'host=xxxxxxx user=repmgr application_name=''psgsqldb2'' connect_timeout=2'
> recovery_target_timeline = 'latest'
>
>
> any idea ?
--
Achilleas Mantzios
IT DEV Lead
IT DEPT
Dynacom Tankers Mgmt
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: wal exist in slave but getting err requested WAL segment has already been removed
2018-07-11 13:09 wal exist in slave but getting err requested WAL segment has already been removed Mariel Cherkassky <mariel.cherkassky@gmail.com>
2018-07-11 13:26 ` Re: wal exist in slave but getting err requested WAL segment has already been removed Achilleas Mantzios <achill@matrix.gatewaynet.com>
@ 2018-07-11 13:32 ` Mariel Cherkassky <mariel.cherkassky@gmail.com>
2018-07-11 13:39 ` Re: wal exist in slave but getting err requested WAL segment has already been removed Achilleas Mantzios <achill@matrix.gatewaynet.com>
0 siblings, 1 reply; 7+ messages in thread
From: Mariel Cherkassky @ 2018-07-11 13:32 UTC (permalink / raw)
To: Achilleas Mantzios <achill@matrix.gatewaynet.com>; +Cc: pgsql-admin@lists.postgresql.org
The wal is available on the standby, not on the primary. It is already in
the pg_xlog directory of the slave...
2018-07-11 16:26 GMT+03:00 Achilleas Mantzios <achill@matrix.gatewaynet.com>
:
> On 11/07/2018 16:09, Mariel Cherkassky wrote:
>
> Hi,
> I have in my cluster 3 nodes (1 master version 9.6.3+ 2 slaves version
> 9.6.3). I configured repmgr (with repmgrd active) v 4.0.4.
>
> Suddenly today after a few good weeks I noticed that there is a lag in one
> of the slaves and the error in the log indicated that the slave didnt get
> the wal :
>
> could not receive data from WAL stream: ERROR: requested WAL segment
> 0000000900002E61000000BD has already been removed
>
> However, when I check if the wal was recieveed :
> postgres=# select pg_is_in_recovery(),pg_is_xlog_replay_paused(),pg_last_
> xlog_receive_location(),pg_last_xlog_replay_location();
> pg_is_in_recovery | pg_is_xlog_replay_paused |
> pg_last_xlog_receive_location | pg_last_xlog_replay_location
> -------------------+--------------------------+-------------
> ------------------+------------------------------
> t | f | 2E61/BDF5C000
> | 2E61/BDF5B930
> (1 row)
>
> and I checked in pg_xlog directory :
> ls -l ../pg_xlog/0000000900002E61000000BD
> -rw------- 1 postgres postgres 16777216 Jul 11 11:13 ../pg_xlog/
> 0000000900002E61000000BD
>
> and the xlog is exist.
>
>
> In which node did you check for the file?
> If the file in the primary is still available, try to compare their md5sum
> .
> If you have a working WAL shipping method in place, then add the
> appropriate line in the recovery.conf of your standby :
>
> restore_command = 'rsync somemachine:/somepath/pitr/%f "%p" '
>
>
> Now is my question, why the wal wasnt replayed ?
> In my repmgr.conf I dont have any parameters regarding recovery just some
> basic things. The recovery.conf file in the data directory :
>
> standby_mode = 'on'
> primary_conninfo = 'host=xxxxxxx user=repmgr
> application_name=''psgsqldb2'' connect_timeout=2'
> recovery_target_timeline = 'latest'
>
>
> any idea ?
>
>
> --
> Achilleas Mantzios
> IT DEV Lead
> IT DEPT
> Dynacom Tankers Mgmt
>
>
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: wal exist in slave but getting err requested WAL segment has already been removed
2018-07-11 13:09 wal exist in slave but getting err requested WAL segment has already been removed Mariel Cherkassky <mariel.cherkassky@gmail.com>
2018-07-11 13:26 ` Re: wal exist in slave but getting err requested WAL segment has already been removed Achilleas Mantzios <achill@matrix.gatewaynet.com>
2018-07-11 13:32 ` Re: wal exist in slave but getting err requested WAL segment has already been removed Mariel Cherkassky <mariel.cherkassky@gmail.com>
@ 2018-07-11 13:39 ` Achilleas Mantzios <achill@matrix.gatewaynet.com>
2018-07-11 13:44 ` Re: wal exist in slave but getting err requested WAL segment has already been removed Mariel Cherkassky <mariel.cherkassky@gmail.com>
0 siblings, 1 reply; 7+ messages in thread
From: Achilleas Mantzios @ 2018-07-11 13:39 UTC (permalink / raw)
To: pgsql-admin@lists.postgresql.org
On 11/07/2018 16:32, Mariel Cherkassky wrote:
> The wal is available on the standby, not on the primary. It is already in the pg_xlog directory of the slave...
Ok but apparently this is not complete. Can you see its contents with pg_waldump (or pg_xlogdump) ?
Do you have any backup mechanism in place? Any WAL shipping / archiving mechanism ?
>
> 2018-07-11 16:26 GMT+03:00 Achilleas Mantzios <achill@matrix.gatewaynet.com <mailto:achill@matrix.gatewaynet.com>>:
>
> On 11/07/2018 16:09, Mariel Cherkassky wrote:
>> Hi,
>> I have in my cluster 3 nodes (1 master version 9.6.3+ 2 slaves version 9.6.3). I configured repmgr (with repmgrd active) v 4.0.4.
>>
>> Suddenly today after a few good weeks I noticed that there is a lag in one of the slaves and the error in the log indicated that the slave didnt get the wal :
>>
>> could not receive data from WAL stream: ERROR: requested WAL segment 0000000900002E61000000BD has already been removed
>>
>> However, when I check if the wal was recieveed :
>> postgres=# select pg_is_in_recovery(),pg_is_xlog_replay_paused(),pg_last_xlog_receive_location(),pg_last_xlog_replay_location();
>> pg_is_in_recovery | pg_is_xlog_replay_paused | pg_last_xlog_receive_location | pg_last_xlog_replay_location
>> -------------------+--------------------------+-------------------------------+------------------------------
>> t | f | 2E61/BDF5C000 | 2E61/BDF5B930
>> (1 row)
>>
>> and I checked in pg_xlog directory :
>> ls -l ../pg_xlog/0000000900002E61000000BD
>> -rw------- 1 postgres postgres 16777216 Jul 11 11:13 ../pg_xlog/0000000900002E61000000BD
>>
>> and the xlog is exist.
>
> In which node did you check for the file?
> If the file in the primary is still available, try to compare their md5sum .
> If you have a working WAL shipping method in place, then add the appropriate line in the recovery.conf of your standby :
>
> restore_command = 'rsync somemachine:/somepath/pitr/%f "%p" '
>
>
>> Now is my question, why the wal wasnt replayed ?
>> In my repmgr.conf I dont have any parameters regarding recovery just some basic things. The recovery.conf file in the data directory :
>>
>> standby_mode = 'on'
>> primary_conninfo = 'host=xxxxxxx user=repmgr application_name=''psgsqldb2'' connect_timeout=2'
>> recovery_target_timeline = 'latest'
>>
>>
>> any idea ?
>
>
> --
> Achilleas Mantzios
> IT DEV Lead
> IT DEPT
> Dynacom Tankers Mgmt
>
>
--
Achilleas Mantzios
IT DEV Lead
IT DEPT
Dynacom Tankers Mgmt
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: wal exist in slave but getting err requested WAL segment has already been removed
2018-07-11 13:09 wal exist in slave but getting err requested WAL segment has already been removed Mariel Cherkassky <mariel.cherkassky@gmail.com>
2018-07-11 13:26 ` Re: wal exist in slave but getting err requested WAL segment has already been removed Achilleas Mantzios <achill@matrix.gatewaynet.com>
2018-07-11 13:32 ` Re: wal exist in slave but getting err requested WAL segment has already been removed Mariel Cherkassky <mariel.cherkassky@gmail.com>
2018-07-11 13:39 ` Re: wal exist in slave but getting err requested WAL segment has already been removed Achilleas Mantzios <achill@matrix.gatewaynet.com>
@ 2018-07-11 13:44 ` Mariel Cherkassky <mariel.cherkassky@gmail.com>
2018-07-11 13:50 ` Re: wal exist in slave but getting err requested WAL segment has already been removed Kenneth Marshall <ktm@rice.edu>
0 siblings, 1 reply; 7+ messages in thread
From: Mariel Cherkassky @ 2018-07-11 13:44 UTC (permalink / raw)
To: Achilleas Mantzios <achill@matrix.gatewaynet.com>; +Cc: pgsql-admin@lists.postgresql.org
Yes i can see its content. However in the end of its content I'm getting
the next msg :
pg_xlogdump: FATAL: error in WAL record at 2E61/BDF59950: invalid magic
number 0000 in log segment 0000000000002E61000000BD, offset 16105472
Maybe this is the reason behind it ?
2018-07-11 16:39 GMT+03:00 Achilleas Mantzios <achill@matrix.gatewaynet.com>
:
> On 11/07/2018 16:32, Mariel Cherkassky wrote:
>
> The wal is available on the standby, not on the primary. It is already in
> the pg_xlog directory of the slave...
>
> Ok but apparently this is not complete. Can you see its contents with
> pg_waldump (or pg_xlogdump) ?
> Do you have any backup mechanism in place? Any WAL shipping / archiving
> mechanism ?
>
>
> 2018-07-11 16:26 GMT+03:00 Achilleas Mantzios <
> achill@matrix.gatewaynet.com>:
>
>> On 11/07/2018 16:09, Mariel Cherkassky wrote:
>>
>> Hi,
>> I have in my cluster 3 nodes (1 master version 9.6.3+ 2 slaves version
>> 9.6.3). I configured repmgr (with repmgrd active) v 4.0.4.
>>
>> Suddenly today after a few good weeks I noticed that there is a lag in
>> one of the slaves and the error in the log indicated that the slave didnt
>> get the wal :
>>
>> could not receive data from WAL stream: ERROR: requested WAL segment
>> 0000000900002E61000000BD has already been removed
>>
>> However, when I check if the wal was recieveed :
>> postgres=# select pg_is_in_recovery(),pg_is_xlog
>> _replay_paused(),pg_last_xlog_receive_location(),pg_last_
>> xlog_replay_location();
>> pg_is_in_recovery | pg_is_xlog_replay_paused |
>> pg_last_xlog_receive_location | pg_last_xlog_replay_location
>> -------------------+--------------------------+-------------
>> ------------------+------------------------------
>> t | f | 2E61/BDF5C000
>> | 2E61/BDF5B930
>> (1 row)
>>
>> and I checked in pg_xlog directory :
>> ls -l ../pg_xlog/0000000900002E61000000BD
>> -rw------- 1 postgres postgres 16777216 Jul 11 11:13
>> ../pg_xlog/0000000900002E61000000BD
>>
>> and the xlog is exist.
>>
>>
>> In which node did you check for the file?
>> If the file in the primary is still available, try to compare their
>> md5sum .
>> If you have a working WAL shipping method in place, then add the
>> appropriate line in the recovery.conf of your standby :
>>
>> restore_command = 'rsync somemachine:/somepath/pitr/%f "%p" '
>>
>>
>> Now is my question, why the wal wasnt replayed ?
>> In my repmgr.conf I dont have any parameters regarding recovery just some
>> basic things. The recovery.conf file in the data directory :
>>
>> standby_mode = 'on'
>> primary_conninfo = 'host=xxxxxxx user=repmgr
>> application_name=''psgsqldb2'' connect_timeout=2'
>> recovery_target_timeline = 'latest'
>>
>>
>> any idea ?
>>
>>
>> --
>> Achilleas Mantzios
>> IT DEV Lead
>> IT DEPT
>> Dynacom Tankers Mgmt
>>
>>
>
> --
> Achilleas Mantzios
> IT DEV Lead
> IT DEPT
> Dynacom Tankers Mgmt
>
>
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: wal exist in slave but getting err requested WAL segment has already been removed
2018-07-11 13:09 wal exist in slave but getting err requested WAL segment has already been removed Mariel Cherkassky <mariel.cherkassky@gmail.com>
2018-07-11 13:26 ` Re: wal exist in slave but getting err requested WAL segment has already been removed Achilleas Mantzios <achill@matrix.gatewaynet.com>
2018-07-11 13:32 ` Re: wal exist in slave but getting err requested WAL segment has already been removed Mariel Cherkassky <mariel.cherkassky@gmail.com>
2018-07-11 13:39 ` Re: wal exist in slave but getting err requested WAL segment has already been removed Achilleas Mantzios <achill@matrix.gatewaynet.com>
2018-07-11 13:44 ` Re: wal exist in slave but getting err requested WAL segment has already been removed Mariel Cherkassky <mariel.cherkassky@gmail.com>
@ 2018-07-11 13:50 ` Kenneth Marshall <ktm@rice.edu>
2018-07-11 14:12 ` Re: wal exist in slave but getting err requested WAL segment has already been removed Mariel Cherkassky <mariel.cherkassky@gmail.com>
0 siblings, 1 reply; 7+ messages in thread
From: Kenneth Marshall @ 2018-07-11 13:50 UTC (permalink / raw)
To: Mariel Cherkassky <mariel.cherkassky@gmail.com>; +Cc: Achilleas Mantzios <achill@matrix.gatewaynet.com>; pgsql-admin@lists.postgresql.org
On Wed, Jul 11, 2018 at 04:44:24PM +0300, Mariel Cherkassky wrote:
> Yes i can see its content. However in the end of its content I'm getting
> the next msg :
> pg_xlogdump: FATAL: error in WAL record at 2E61/BDF59950: invalid magic
> number 0000 in log segment 0000000000002E61000000BD, offset 16105472
> Maybe this is the reason behind it ?
>
Hi Mariel,
I do not know if this applies to your case, but 9.6.9 has this
in the release notes:
Fix a corner case where a streaming standby gets stuck at a WAL continuation record (Kyotaro Horiguchi)
Regards,
Ken
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: wal exist in slave but getting err requested WAL segment has already been removed
2018-07-11 13:09 wal exist in slave but getting err requested WAL segment has already been removed Mariel Cherkassky <mariel.cherkassky@gmail.com>
2018-07-11 13:26 ` Re: wal exist in slave but getting err requested WAL segment has already been removed Achilleas Mantzios <achill@matrix.gatewaynet.com>
2018-07-11 13:32 ` Re: wal exist in slave but getting err requested WAL segment has already been removed Mariel Cherkassky <mariel.cherkassky@gmail.com>
2018-07-11 13:39 ` Re: wal exist in slave but getting err requested WAL segment has already been removed Achilleas Mantzios <achill@matrix.gatewaynet.com>
2018-07-11 13:44 ` Re: wal exist in slave but getting err requested WAL segment has already been removed Mariel Cherkassky <mariel.cherkassky@gmail.com>
2018-07-11 13:50 ` Re: wal exist in slave but getting err requested WAL segment has already been removed Kenneth Marshall <ktm@rice.edu>
@ 2018-07-11 14:12 ` Mariel Cherkassky <mariel.cherkassky@gmail.com>
0 siblings, 0 replies; 7+ messages in thread
From: Mariel Cherkassky @ 2018-07-11 14:12 UTC (permalink / raw)
To: Kenneth Marshall <ktm@rice.edu>; +Cc: Achilleas Mantzios <achill@matrix.gatewaynet.com>; pgsql-admin@lists.postgresql.org
How can I get more info regarding this bug ? I would like to be sure that i
faced a real bug.
2018-07-11 16:50 GMT+03:00 Kenneth Marshall <ktm@rice.edu>:
> On Wed, Jul 11, 2018 at 04:44:24PM +0300, Mariel Cherkassky wrote:
> > Yes i can see its content. However in the end of its content I'm getting
> > the next msg :
> > pg_xlogdump: FATAL: error in WAL record at 2E61/BDF59950: invalid magic
> > number 0000 in log segment 0000000000002E61000000BD, offset 16105472
> > Maybe this is the reason behind it ?
> >
>
> Hi Mariel,
>
> I do not know if this applies to your case, but 9.6.9 has this
> in the release notes:
>
> Fix a corner case where a streaming standby gets stuck at a WAL
> continuation record (Kyotaro Horiguchi)
>
> Regards,
> Ken
>
^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2018-07-11 14:12 UTC | newest]
Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2018-07-11 13:09 wal exist in slave but getting err requested WAL segment has already been removed Mariel Cherkassky <mariel.cherkassky@gmail.com>
2018-07-11 13:26 ` Achilleas Mantzios <achill@matrix.gatewaynet.com>
2018-07-11 13:32 ` Mariel Cherkassky <mariel.cherkassky@gmail.com>
2018-07-11 13:39 ` Achilleas Mantzios <achill@matrix.gatewaynet.com>
2018-07-11 13:44 ` Mariel Cherkassky <mariel.cherkassky@gmail.com>
2018-07-11 13:50 ` Kenneth Marshall <ktm@rice.edu>
2018-07-11 14:12 ` Mariel Cherkassky <mariel.cherkassky@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox