pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feed(no subject)
11+ messages / 6 participants
[nested] [flat]
* (no subject)
@ 2014-04-25 17:37 Henry Korszun <henryk302@yahoo.com>
2014-04-25 17:43 ` Re: Scott Whitney <scott@journyx.com>
2014-04-25 17:46 ` Re: Payal Singh <payal@omniti.com>
0 siblings, 2 replies; 11+ messages in thread
From: Henry Korszun @ 2014-04-25 17:37 UTC (permalink / raw)
To: pgsql-admin
I'm using PostgreSQL 9.2.4 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.1.2 20070115 (SUSE Linux), 64-bit.
I set up streaming replication for a read/write primary and a read/only
standby. The replication works fine for a while, and then out of the
blue BOTH machines become read/write, but with no replication from the
original primary to the newly read/write standby.
The only log entry that seems relevant is as follows:
FATAL,57P01,"terminating walreceiver process due to administrator
command",,,,,,,,"ProcessWalRcvInterrupts, walreceiver.c:150",""
Any help/guidance would be appreciated. Thanks in advance.
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re:
2014-04-25 17:37 (no subject) Henry Korszun <henryk302@yahoo.com>
@ 2014-04-25 17:43 ` Scott Whitney <scott@journyx.com>
2014-04-25 17:57 ` Re: Henry Korszun <henryk302@yahoo.com>
1 sibling, 1 reply; 11+ messages in thread
From: Scott Whitney @ 2014-04-25 17:43 UTC (permalink / raw)
To: Henry Korszun <henryk302@yahoo.com>; +Cc: pgsql-admin
Sounds like you might have a "trigger_file" set in your recovery.conf. Do you? That or someone is issuing a pg_ctl promote command.
----- Original Message -----
> I'm using PostgreSQL 9.2.4 on x86_64-unknown-linux-gnu, compiled by gcc (GCC)
> 4.1.2 20070115 (SUSE Linux), 64-bit.
> I set up streaming replication for a read/write primary and a read/only
> standby. The replication works fine for a while, and then out of the blue
> BOTH machines become read/write, but with no replication from the original
> primary to the newly read/write standby.
> The only log entry that seems relevant is as follows:
> FATAL,57P01,"terminating walreceiver process due to administrator
> command",,,,,,,,"ProcessWalRcvInterrupts, walreceiver.c:150",""
> Any help/guidance would be appreciated. Thanks in advance.
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re:
2014-04-25 17:37 (no subject) Henry Korszun <henryk302@yahoo.com>
2014-04-25 17:43 ` Re: Scott Whitney <scott@journyx.com>
@ 2014-04-25 17:57 ` Henry Korszun <henryk302@yahoo.com>
2014-04-25 18:00 ` Re: Payal Singh <payal@omniti.com>
2014-04-25 18:00 ` Re: Scott Whitney <scott@journyx.com>
0 siblings, 2 replies; 11+ messages in thread
From: Henry Korszun @ 2014-04-25 17:57 UTC (permalink / raw)
To: Scott Whitney <swhitney@journyx.com>; +Cc: pgsql-admin
There IS a trigger file, which does appear to have been "touch"ed. But the problem is that a fail-over hasn't really occurred since the original read/write primary continues to be a fully functioning read/write machine. But it's no longer replicating to the erstwhile standby, which has become read/write. Bottom line, I now have 2 read/write machines, but with no replication between them.
On Friday, April 25, 2014 1:43 PM, Scott Whitney <scott@journyx.com> wrote:
Sounds like you might have a "trigger_file" set in your recovery.conf. Do you? That or someone is issuing a pg_ctl promote command.
________________________________
I'm using PostgreSQL 9.2.4 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.1.2 20070115 (SUSE Linux), 64-bit.
>
>I set up streaming replication for a read/write primary and a read/only
standby. The replication works fine for a while, and then out of the
blue BOTH machines become read/write, but with no replication from the
original primary to the newly read/write standby.
>
>The only log entry that seems relevant is as follows:
FATAL,57P01,"terminating walreceiver process due to administrator
command",,,,,,,,"ProcessWalRcvInterrupts, walreceiver.c:150",""
>
>Any help/guidance would be appreciated. Thanks in advance.
>
>
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re:
2014-04-25 17:37 (no subject) Henry Korszun <henryk302@yahoo.com>
2014-04-25 17:43 ` Re: Scott Whitney <scott@journyx.com>
2014-04-25 17:57 ` Re: Henry Korszun <henryk302@yahoo.com>
@ 2014-04-25 18:00 ` Payal Singh <payal@omniti.com>
1 sibling, 0 replies; 11+ messages in thread
From: Payal Singh @ 2014-04-25 18:00 UTC (permalink / raw)
To: Henry Korszun <henryk302@yahoo.com>; +Cc: Scott Whitney <swhitney@journyx.com>; pgsql-admin
Once a trigger file is touched on slave, it makes the slave standalone, but doesn't stop the old primary server automatically. You have to handle that, by either stopping the old primary altogether or pointing a virtual ip to the new slave.
On Fri, Apr 25, 2014 at 10:57:16AM -0700, Henry Korszun wrote:
> There IS a trigger file, which does appear to have been "touch"ed. But the problem is that a fail-over hasn't really occurred since the original read/write primary continues to be a fully functioning read/write machine. But it's no longer replicating to the erstwhile standby, which has become read/write. Bottom line, I now have 2 read/write machines, but with no replication between them.
>
> On Friday, April 25, 2014 1:43 PM, Scott Whitney <scott@journyx.com> wrote:
>
> Sounds like you might have a "trigger_file" set in your recovery.conf. Do you? That or someone is issuing a pg_ctl promote command.
>
> ________________________________
>
> I'm using PostgreSQL 9.2.4 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.1.2 20070115 (SUSE Linux), 64-bit.
> >
> >I set up streaming replication for a read/write primary and a read/only
> standby. The replication works fine for a while, and then out of the
> blue BOTH machines become read/write, but with no replication from the
> original primary to the newly read/write standby.
> >
> >The only log entry that seems relevant is as follows:
> FATAL,57P01,"terminating walreceiver process due to administrator
> command",,,,,,,,"ProcessWalRcvInterrupts, walreceiver.c:150",""
> >
> >Any help/guidance would be appreciated. Thanks in advance.
> >
> >
--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re:
2014-04-25 17:37 (no subject) Henry Korszun <henryk302@yahoo.com>
2014-04-25 17:43 ` Re: Scott Whitney <scott@journyx.com>
2014-04-25 17:57 ` Re: Henry Korszun <henryk302@yahoo.com>
@ 2014-04-25 18:00 ` Scott Whitney <scott@journyx.com>
2014-04-25 18:09 ` Re: Henry Korszun <henryk302@yahoo.com>
1 sibling, 1 reply; 11+ messages in thread
From: Scott Whitney @ 2014-04-25 18:00 UTC (permalink / raw)
To: Henry Korszun <henryk302@yahoo.com>; +Cc: Scott Whitney <swhitney@journyx.com>; pgsql-admin
The slave doesn't "turn off" the master. The trigger file is intended to be touched _when the master is down_.
Since the master never WENT down (or came back up) and the trigger file was touched, the slave got promoted.
You'll need to stop the slave, run your select pg_startbackup(), rsync, etc to get your slave back to slave mode.
----- Original Message -----
> There IS a trigger file, which does appear to have been "touch"ed. But the
> problem is that a fail-over hasn't really occurred since the original
> read/write primary continues to be a fully functioning read/write machine.
> But it's no longer replicating to the erstwhile standby, which has become
> read/write. Bottom line, I now have 2 read/write machines, but with no
> replication between them.
> On Friday, April 25, 2014 1:43 PM, Scott Whitney <scott@journyx.com> wrote:
> Sounds like you might have a "trigger_file" set in your recovery.conf. Do
> you? That or someone is issuing a pg_ctl promote command.
> > I'm using PostgreSQL 9.2.4 on x86_64-unknown-linux-gnu, compiled by gcc
> > (GCC)
> > 4.1.2 20070115 (SUSE Linux), 64-bit.
>
> > I set up streaming replication for a read/write primary and a read/only
> > standby. The replication works fine for a while, and then out of the blue
> > BOTH machines become read/write, but with no replication from the original
> > primary to the newly read/write standby.
>
> > The only log entry that seems relevant is as follows:
> > FATAL,57P01,"terminating walreceiver process due to administrator
> > command",,,,,,,,"ProcessWalRcvInterrupts, walreceiver.c:150",""
>
> > Any help/guidance would be appreciated. Thanks in advance.
>
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re:
2014-04-25 17:37 (no subject) Henry Korszun <henryk302@yahoo.com>
2014-04-25 17:43 ` Re: Scott Whitney <scott@journyx.com>
2014-04-25 17:57 ` Re: Henry Korszun <henryk302@yahoo.com>
2014-04-25 18:00 ` Re: Scott Whitney <scott@journyx.com>
@ 2014-04-25 18:09 ` Henry Korszun <henryk302@yahoo.com>
2014-04-25 18:23 ` Re: Jerry Sievers <gsievers19@comcast.net>
0 siblings, 1 reply; 11+ messages in thread
From: Henry Korszun @ 2014-04-25 18:09 UTC (permalink / raw)
To: Scott Whitney <swhitney@journyx.com>; +Cc: pgsql-admin
I understand what you're saying, but I don't know what's causing the "touch" in the first place. I guess I need to further examine/debug. Thanks for your help.
On Friday, April 25, 2014 2:00 PM, Scott Whitney <scott@journyx.com> wrote:
The slave doesn't "turn off" the master. The trigger file is intended to be touched _when the master is down_.
Since the master never WENT down (or came back up) and the trigger file was touched, the slave got promoted.
You'll need to stop the slave, run your select pg_startbackup(), rsync, etc to get your slave back to slave mode.
________________________________
There IS a trigger file, which does appear to have been "touch"ed. But the problem is that a fail-over hasn't really occurred since the original read/write primary continues to be a fully functioning read/write machine. But it's no longer replicating to the erstwhile standby, which has become read/write. Bottom line, I now have 2 read/write machines, but with no replication between them.
>
>On Friday, April 25, 2014 1:43 PM, Scott Whitney <scott@journyx.com> wrote:
>
>Sounds like you might have a "trigger_file" set in your recovery.conf. Do you? That or someone is issuing a pg_ctl promote command.
>
>
>________________________________
>
>I'm using PostgreSQL 9.2.4 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.1.2 20070115 (SUSE Linux), 64-bit.
>>
>>I set up streaming replication for a read/write primary and a read/only
standby. The replication works fine for a while, and then out of the
blue BOTH machines become read/write, but with no replication from the
original primary to the newly read/write standby.
>>
>>The only log entry that seems relevant is as follows:
FATAL,57P01,"terminating walreceiver process due to administrator
command",,,,,,,,"ProcessWalRcvInterrupts, walreceiver.c:150",""
>>
>>Any help/guidance would be appreciated. Thanks in advance.
>>
>>
>
>
>
>
>
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re:
2014-04-25 17:37 (no subject) Henry Korszun <henryk302@yahoo.com>
2014-04-25 17:43 ` Re: Scott Whitney <scott@journyx.com>
2014-04-25 17:57 ` Re: Henry Korszun <henryk302@yahoo.com>
2014-04-25 18:00 ` Re: Scott Whitney <scott@journyx.com>
2014-04-25 18:09 ` Re: Henry Korszun <henryk302@yahoo.com>
@ 2014-04-25 18:23 ` Jerry Sievers <gsievers19@comcast.net>
2014-04-27 19:49 ` Re: Jim Mercer <jim@reptiles.org>
0 siblings, 1 reply; 11+ messages in thread
From: Jerry Sievers @ 2014-04-25 18:23 UTC (permalink / raw)
To: Henry Korszun <henryk302@yahoo.com>; +Cc: Scott Whitney <swhitney@journyx.com>; pgsql-admin
Henry Korszun <henryk302@yahoo.com> writes:
> I understand what you're saying, but I don't know what's causing the "touch" in the first place. I guess I need to further examine/debug. Thanks for your help.
> On Friday, April 25, 2014 2:00 PM, Scott Whitney <scott@journyx.com> wrote:
> The slave doesn't "turn off" the master. The trigger file is intended to be touched _when the master is down_.
Nor do we.
Possibly your system is running some HA software and it's onlining your
standby due to false-positive.
>
> Since the master never WENT down (or came back up) and the trigger file was touched, the slave got promoted.
>
> You'll need to stop the slave, run your select pg_startbackup(), rsync, etc to get your slave back to slave mode.
>
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> There IS a trigger file, which does appear to have been "touch"ed. But the problem is that a fail-over hasn't really occurred since the original read/write primary
> continues to be a fully functioning read/write machine. But it's no longer replicating to the erstwhile standby, which has become read/write. Bottom line, I now
> have 2 read/write machines, but with no replication between them.
> On Friday, April 25, 2014 1:43 PM, Scott Whitney <scott@journyx.com> wrote:
> Sounds like you might have a "trigger_file" set in your recovery.conf. Do you? That or someone is issuing a pg_ctl promote command.
>
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> I'm using PostgreSQL 9.2.4 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.1.2 20070115 (SUSE Linux), 64-bit.
>
> I set up streaming replication for a read/write primary and a read/only standby. The replication works fine for a while, and then out of the blue BOTH machines
> become read/write, but with no replication from the original primary to the newly read/write standby.
>
> The only log entry that seems relevant is as follows: FATAL,57P01,"terminating walreceiver process due to administrator
> command",,,,,,,,"ProcessWalRcvInterrupts, walreceiver.c:150",""
>
> Any help/guidance would be appreciated. Thanks in advance.
>
--
Jerry Sievers
Postgres DBA/Development Consulting
e: postgres.consulting@comcast.net
p: 312.241.7800
--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re:
2014-04-25 17:37 (no subject) Henry Korszun <henryk302@yahoo.com>
2014-04-25 17:43 ` Re: Scott Whitney <scott@journyx.com>
2014-04-25 17:57 ` Re: Henry Korszun <henryk302@yahoo.com>
2014-04-25 18:00 ` Re: Scott Whitney <scott@journyx.com>
2014-04-25 18:09 ` Re: Henry Korszun <henryk302@yahoo.com>
2014-04-25 18:23 ` Re: Jerry Sievers <gsievers19@comcast.net>
@ 2014-04-27 19:49 ` Jim Mercer <jim@reptiles.org>
2014-04-27 20:26 ` Re: David G Johnston <david.g.johnston@gmail.com>
0 siblings, 1 reply; 11+ messages in thread
From: Jim Mercer @ 2014-04-27 19:49 UTC (permalink / raw)
To: Jerry Sievers <gsievers19@comcast.net>; +Cc: Henry Korszun <henryk302@yahoo.com>; Scott Whitney <swhitney@journyx.com>; pgsql-admin
On Fri, Apr 25, 2014 at 01:23:30PM -0500, Jerry Sievers wrote:
> Henry Korszun <henryk302@yahoo.com> writes:
> > I understand what you're saying, but I don't know what's causing the "touch" in the first place. I guess I need to further examine/debug. Thanks for your help.
it may be a semantic difference, but is recovery_mode dependent on the existence
of the trigger file, or the timestamp.
"touch" in the above context could either be the creation of the file,
or simply updating the timestamp of the file.
i suspect that the recovery is triggered by the mere existence of the file,
while henry might be talking in the 'update timestamp' context.
--jim
--
Jim Mercer Reptilian Research jim@reptiles.org +1 416 410-5633
"He who dies with the most toys is nonetheless dead"
--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re:
2014-04-25 17:37 (no subject) Henry Korszun <henryk302@yahoo.com>
2014-04-25 17:43 ` Re: Scott Whitney <scott@journyx.com>
2014-04-25 17:57 ` Re: Henry Korszun <henryk302@yahoo.com>
2014-04-25 18:00 ` Re: Scott Whitney <scott@journyx.com>
2014-04-25 18:09 ` Re: Henry Korszun <henryk302@yahoo.com>
2014-04-25 18:23 ` Re: Jerry Sievers <gsievers19@comcast.net>
2014-04-27 19:49 ` Re: Jim Mercer <jim@reptiles.org>
@ 2014-04-27 20:26 ` David G Johnston <david.g.johnston@gmail.com>
0 siblings, 0 replies; 11+ messages in thread
From: David G Johnston @ 2014-04-27 20:26 UTC (permalink / raw)
To: pgsql-admin
Jim Mercer wrote
> On Fri, Apr 25, 2014 at 01:23:30PM -0500, Jerry Sievers wrote:
>> Henry Korszun <
> henryk302@
> > writes:
>> > I understand what you're saying, but I don't know what's causing the
>> "touch" in the first place. I guess I need to further examine/debug.
>> Thanks for your help.
>
> it may be a semantic difference, but is recovery_mode dependent on the
> existence
> of the trigger file, or the timestamp.
>
> "touch" in the above context could either be the creation of the file,
> or simply updating the timestamp of the file.
>
> i suspect that the recovery is triggered by the mere existence of the
> file,
> while henry might be talking in the 'update timestamp' context.
I'm seriously doubting the timestamp info matters - the timestamp would
almost always be in the past (specifying a future recovery date doesn't make
sense anyway) and no arbitrary age is reasonable to make the file invalid
and would be extremely confusing if one did.
"Touch" is a shorthand for "a file whose mere existence is all that is
necessary" and by convention implies that what is in the file doesn't matter
(since actually touching a non-existent file creates a new empty file).
If the file already existed (not that timeframes are being defined all that
well here) the system would never have been in recovery mode...
Henry's last comment is that it is not known what process is creating the
empty trigger file in the first place - whether that process uses touch or
some other means to create the file is irrelevant to the issue at hand.
David J.
--
View this message in context: http://postgresql.1045698.n5.nabble.com/no-subject-tp5801536p5801671.html
Sent from the PostgreSQL - admin mailing list archive at Nabble.com.
--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re:
2014-04-25 17:37 (no subject) Henry Korszun <henryk302@yahoo.com>
@ 2014-04-25 17:46 ` Payal Singh <payal@omniti.com>
2014-04-25 17:47 ` Re: Payal Singh <payal@omniti.com>
1 sibling, 1 reply; 11+ messages in thread
From: Payal Singh @ 2014-04-25 17:46 UTC (permalink / raw)
To: Henry Korszun <henryk302@yahoo.com>; +Cc: pgsql-admin
I would check for any automated jobs that touch the trigger file on the
slave
Payal Singh,
Database Administrator,
OmniTI Computer Consulting Inc.
Phone: 240.646.0770 x 253
On Fri, Apr 25, 2014 at 1:37 PM, Henry Korszun <henryk302@yahoo.com> wrote:
> I'm using PostgreSQL 9.2.4 on x86_64-unknown-linux-gnu, compiled by gcc
> (GCC) 4.1.2 20070115 (SUSE Linux), 64-bit.
>
> I set up streaming replication for a read/write primary and a read/only
> standby. The replication works fine for a while, and then out of the blue
> BOTH machines become read/write, but with no replication from the original
> primary to the newly read/write standby.
>
> The only log entry that seems relevant is as follows:
> FATAL,57P01,"terminating walreceiver process due to administrator
> command",,,,,,,,"ProcessWalRcvInterrupts, walreceiver.c:150",""
>
> Any help/guidance would be appreciated. Thanks in advance.
>
>
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re:
2014-04-25 17:37 (no subject) Henry Korszun <henryk302@yahoo.com>
2014-04-25 17:46 ` Re: Payal Singh <payal@omniti.com>
@ 2014-04-25 17:47 ` Payal Singh <payal@omniti.com>
0 siblings, 0 replies; 11+ messages in thread
From: Payal Singh @ 2014-04-25 17:47 UTC (permalink / raw)
To: Henry Korszun <henryk302@yahoo.com>; +Cc: pgsql-admin
Also, if you are using chef/puppet to automate the configurations, maybe
the file recovery.conf is being overwritten or removed.
Payal Singh,
Database Administrator,
OmniTI Computer Consulting Inc.
Phone: 240.646.0770 x 253
On Fri, Apr 25, 2014 at 1:46 PM, Payal Singh <payal@omniti.com> wrote:
> I would check for any automated jobs that touch the trigger file on the
> slave
>
> Payal Singh,
> Database Administrator,
> OmniTI Computer Consulting Inc.
> Phone: 240.646.0770 x 253
>
>
> On Fri, Apr 25, 2014 at 1:37 PM, Henry Korszun <henryk302@yahoo.com>wrote:
>
>> I'm using PostgreSQL 9.2.4 on x86_64-unknown-linux-gnu, compiled by gcc
>> (GCC) 4.1.2 20070115 (SUSE Linux), 64-bit.
>>
>> I set up streaming replication for a read/write primary and a read/only
>> standby. The replication works fine for a while, and then out of the blue
>> BOTH machines become read/write, but with no replication from the original
>> primary to the newly read/write standby.
>>
>> The only log entry that seems relevant is as follows:
>> FATAL,57P01,"terminating walreceiver process due to administrator
>> command",,,,,,,,"ProcessWalRcvInterrupts, walreceiver.c:150",""
>>
>> Any help/guidance would be appreciated. Thanks in advance.
>>
>>
>
^ permalink raw reply [nested|flat] 11+ messages in thread
end of thread, other threads:[~2014-04-27 20:26 UTC | newest]
Thread overview: 11+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2014-04-25 17:37 Henry Korszun <henryk302@yahoo.com>
2014-04-25 17:43 ` Re: Scott Whitney <scott@journyx.com>
2014-04-25 17:57 ` Re: Henry Korszun <henryk302@yahoo.com>
2014-04-25 18:00 ` Re: Payal Singh <payal@omniti.com>
2014-04-25 18:00 ` Re: Scott Whitney <scott@journyx.com>
2014-04-25 18:09 ` Re: Henry Korszun <henryk302@yahoo.com>
2014-04-25 18:23 ` Re: Jerry Sievers <gsievers19@comcast.net>
2014-04-27 19:49 ` Re: Jim Mercer <jim@reptiles.org>
2014-04-27 20:26 ` Re: David G Johnston <david.g.johnston@gmail.com>
2014-04-25 17:46 ` Re: Payal Singh <payal@omniti.com>
2014-04-25 17:47 ` Re: Payal Singh <payal@omniti.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox