pg.ddx.io pgsql-docs@postgresql.org mailing list archive
help / color / mirror / Atom feedpgsql: Make standby server continuously retry restoring the next WAL
51+ messages / 15 participants
[nested] [flat]
Commits from this thread:
2020-12-08 9a2641911a 101c8ae02c 10c601578a 3410a9b570 0921b86ca1 b4a5d54065 c5ba660770 Doc: explain that the string types can't store \0 (ASCII NUL).
* pgsql: Make standby server continuously retry restoring the next WAL
@ 2010-01-27 15:27 Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
0 siblings, 1 reply; 51+ messages in thread
From: heikki@postgresql.org @ 2010-01-27 15:27 UTC (permalink / raw)
To: pgsql-committers@postgresql.org
Log Message:
-----------
Make standby server continuously retry restoring the next WAL segment with
restore_command, if the connection to the primary server is lost. This
ensures that the standby can recover automatically, if the connection is
lost for a long time and standby falls behind so much that the required
WAL segments have been archived and deleted in the master.
This also makes standby_mode useful without streaming replication; the
server will keep retrying restore_command every few seconds until the
trigger file is found. That's the same basic functionality pg_standby
offers, but without the bells and whistles.
To implement that, refactor the ReadRecord/FetchRecord functions. The
FetchRecord() function introduced in the original streaming replication
patch is removed, and all the retry logic is now in a new function called
XLogReadPage(). XLogReadPage() is now responsible for executing
restore_command, launching walreceiver, and waiting for new WAL to arrive
from primary, as required.
This also changes the life cycle of walreceiver. When launched, it now only
tries to connect to the master once, and exits if the connection fails, or
is lost during streaming for any reason. The startup process detects the
death, and re-launches walreceiver if necessary.
Modified Files:
--------------
pgsql/src/backend/access/transam:
xlog.c (r1.361 -> r1.362)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/access/transam/xlog.c?r1=1.361&r2=1.3...)
pgsql/src/backend/postmaster:
postmaster.c (r1.601 -> r1.602)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/postmaster/postmaster.c?r1=1.601&r2=1...)
pgsql/src/backend/replication:
walreceiver.c (r1.1 -> r1.2)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/replication/walreceiver.c?r1=1.1&r2=1...)
walreceiverfuncs.c (r1.2 -> r1.3)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/replication/walreceiverfuncs.c?r1=1.2&...)
pgsql/src/include/replication:
walreceiver.h (r1.4 -> r1.5)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/include/replication/walreceiver.h?r1=1.4&r2=1...)
pgsql/src/include/storage:
pmsignal.h (r1.28 -> r1.29)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/include/storage/pmsignal.h?r1=1.28&r2=1.29)
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
@ 2010-02-10 05:05 ` Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
0 siblings, 1 reply; 51+ messages in thread
From: Fujii Masao @ 2010-02-10 05:05 UTC (permalink / raw)
To: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; +Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
On Thu, Jan 28, 2010 at 12:27 AM, Heikki Linnakangas
<heikki@postgresql.org> wrote:
> Log Message:
> -----------
> Make standby server continuously retry restoring the next WAL segment with
> restore_command, if the connection to the primary server is lost. This
> ensures that the standby can recover automatically, if the connection is
> lost for a long time and standby falls behind so much that the required
> WAL segments have been archived and deleted in the master.
>
> This also makes standby_mode useful without streaming replication; the
> server will keep retrying restore_command every few seconds until the
> trigger file is found. That's the same basic functionality pg_standby
> offers, but without the bells and whistles.
http://archives.postgresql.org/pgsql-hackers/2010-01/msg01520.php
http://archives.postgresql.org/pgsql-hackers/2010-01/msg02589.php
As I pointed out previously, the standby might restore a partially-filled
WAL file that is being archived by the primary, and cause a FATAL error.
And this happened in my box when I was testing the SR.
sby [20088] FATAL: archive file "000000010000000000000087" has
wrong size: 14139392 instead of 16777216
sby [20076] LOG: startup process (PID 20088) exited with exit code 1
sby [20076] LOG: terminating any other active server processes
act [18164] LOG: received immediate shutdown request
If the startup process is in standby mode, I think that it should retry
starting replication instead of emitting an error when it finds a
partially-filled file in the archive. Then if the replication has been
terminated, it has only to restore the archived file again. Thought?
Regards,
--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
@ 2010-02-10 07:32 ` Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-10 09:19 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 13:45 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
0 siblings, 3 replies; 51+ messages in thread
From: Heikki Linnakangas @ 2010-02-10 07:32 UTC (permalink / raw)
To: Fujii Masao <masao.fujii@gmail.com>; +Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Fujii Masao wrote:
> As I pointed out previously, the standby might restore a partially-filled
> WAL file that is being archived by the primary, and cause a FATAL error.
> And this happened in my box when I was testing the SR.
>
> sby [20088] FATAL: archive file "000000010000000000000087" has
> wrong size: 14139392 instead of 16777216
> sby [20076] LOG: startup process (PID 20088) exited with exit code 1
> sby [20076] LOG: terminating any other active server processes
> act [18164] LOG: received immediate shutdown request
>
> If the startup process is in standby mode, I think that it should retry
> starting replication instead of emitting an error when it finds a
> partially-filled file in the archive. Then if the replication has been
> terminated, it has only to restore the archived file again. Thought?
Hmm, so after running restore_command, check the file size and if it's
too short, treat it the same as if restore_command returned non-zero?
And it will be retried on the next iteration. Works for me, though OTOH
it will then fail to complain about a genuinely WAL file that's
truncated for some reason. I guess there's no way around that, even if
you have a script as restore_command that does the file size check, it
will have the same problem.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
@ 2010-02-10 09:19 ` Fujii Masao <masao.fujii@gmail.com>
2 siblings, 0 replies; 51+ messages in thread
From: Fujii Masao @ 2010-02-10 09:19 UTC (permalink / raw)
To: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; +Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
On Wed, Feb 10, 2010 at 4:32 PM, Heikki Linnakangas
<heikki.linnakangas@enterprisedb.com> wrote:
> Hmm, so after running restore_command, check the file size and if it's
> too short, treat it the same as if restore_command returned non-zero?
Yes, only in standby mode case. OTOH I think that normal archive recovery
should treat it as a FATAL error.
> And it will be retried on the next iteration. Works for me, though OTOH
> it will then fail to complain about a genuinely WAL file that's
> truncated for some reason. I guess there's no way around that, even if
> you have a script as restore_command that does the file size check, it
> will have the same problem.
Right. But the server in standby mode also needs to complain about that?
We might be able to read completely such a WAL file that looks truncated
from the primary via SR, or from the archive after a few seconds. So it's
odd for me to give up continuing the standby only by finding the WAL file
whose file size is short. I believe that the warm standby (+ pg_standby)
also is based on that thought.
Regards,
--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
@ 2010-02-10 13:45 ` Aidan Van Dyk <aidan@highrise.ca>
2010-02-10 16:53 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2 siblings, 1 reply; 51+ messages in thread
From: Aidan Van Dyk @ 2010-02-10 13:45 UTC (permalink / raw)
To: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; +Cc: Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
* Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> [100210 02:33]:
> Hmm, so after running restore_command, check the file size and if it's
> too short, treat it the same as if restore_command returned non-zero?
> And it will be retried on the next iteration. Works for me, though OTOH
> it will then fail to complain about a genuinely WAL file that's
> truncated for some reason. I guess there's no way around that, even if
> you have a script as restore_command that does the file size check, it
> will have the same problem.
But isn't this something every current PITR archive already "works
around"... Everybody doing PITR archives already know the importance of
making the *appearance* of the WAL filename in the archive atomic.
Don't docs warn about plain cp not being atomic and allowing "short"
files to appear in the archive...
I'm not sure why "streaming recovery" suddenly changes the requirements...
a.
--
Aidan Van Dyk Create like a god,
aidan@highrise.ca command like a king,
http://www.highrise.ca/ work like a slave.
Attachments:
signature.asc (application/pgp-signature, 188B) download
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-10 13:45 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
@ 2010-02-10 16:53 ` Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
0 siblings, 0 replies; 51+ messages in thread
From: Heikki Linnakangas @ 2010-02-10 16:53 UTC (permalink / raw)
To: Aidan Van Dyk <aidan@highrise.ca>; +Cc: Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
Aidan Van Dyk wrote:
> * Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> [100210 02:33]:
>
>> Hmm, so after running restore_command, check the file size and if it's
>> too short, treat it the same as if restore_command returned non-zero?
>> And it will be retried on the next iteration. Works for me, though OTOH
>> it will then fail to complain about a genuinely WAL file that's
>> truncated for some reason. I guess there's no way around that, even if
>> you have a script as restore_command that does the file size check, it
>> will have the same problem.
>
> But isn't this something every current PITR archive already "works
> around"... Everybody doing PITR archives already know the importance of
> making the *appearance* of the WAL filename in the archive atomic.
Well, pg_standby does defend against that, but you don't use pg_standby
with the built-in standby mode anymore. It would be reasonable to have
the same level of defenses built-in. It's essentially a one-line change,
and saves a lot of trouble and risk of subtle misconfiguration for admins.
> Don't docs warn about plain cp not being atomic and allowing "short"
> files to appear in the archive...
Hmm, I don't see anything about that at quick glance. Besides, normal
PITR doesn't have a problem with that, because it will stop when it
reaches the end of archived WAL anyway.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
@ 2010-02-11 10:37 ` Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2 siblings, 1 reply; 51+ messages in thread
From: Simon Riggs @ 2010-02-11 10:37 UTC (permalink / raw)
To: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; +Cc: Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
On Wed, 2010-02-10 at 09:32 +0200, Heikki Linnakangas wrote:
> Fujii Masao wrote:
> > As I pointed out previously, the standby might restore a partially-filled
> > WAL file that is being archived by the primary, and cause a FATAL error.
> > And this happened in my box when I was testing the SR.
> >
> > sby [20088] FATAL: archive file "000000010000000000000087" has
> > wrong size: 14139392 instead of 16777216
> > sby [20076] LOG: startup process (PID 20088) exited with exit code 1
> > sby [20076] LOG: terminating any other active server processes
> > act [18164] LOG: received immediate shutdown request
> >
> > If the startup process is in standby mode, I think that it should retry
> > starting replication instead of emitting an error when it finds a
> > partially-filled file in the archive. Then if the replication has been
> > terminated, it has only to restore the archived file again. Thought?
>
> Hmm, so after running restore_command, check the file size and if it's
> too short, treat it the same as if restore_command returned non-zero?
> And it will be retried on the next iteration. Works for me, though OTOH
> it will then fail to complain about a genuinely WAL file that's
> truncated for some reason. I guess there's no way around that, even if
> you have a script as restore_command that does the file size check, it
> will have the same problem.
Are we trying to re-invent pg_standby here?
--
Simon Riggs www.2ndQuadrant.com
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
@ 2010-02-11 12:22 ` Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
0 siblings, 1 reply; 51+ messages in thread
From: Heikki Linnakangas @ 2010-02-11 12:22 UTC (permalink / raw)
To: Simon Riggs <simon@2ndQuadrant.com>; +Cc: Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
Simon Riggs wrote:
> On Wed, 2010-02-10 at 09:32 +0200, Heikki Linnakangas wrote:
>> Hmm, so after running restore_command, check the file size and if it's
>> too short, treat it the same as if restore_command returned non-zero?
>> And it will be retried on the next iteration. Works for me, though OTOH
>> it will then fail to complain about a genuinely WAL file that's
>> truncated for some reason. I guess there's no way around that, even if
>> you have a script as restore_command that does the file size check, it
>> will have the same problem.
>
> Are we trying to re-invent pg_standby here?
That's not the goal, but we seem to need some of the same functionality
in the backend now.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
@ 2010-02-11 12:27 ` Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
0 siblings, 1 reply; 51+ messages in thread
From: Simon Riggs @ 2010-02-11 12:27 UTC (permalink / raw)
To: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; +Cc: Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
On Thu, 2010-02-11 at 14:22 +0200, Heikki Linnakangas wrote:
> Simon Riggs wrote:
> > On Wed, 2010-02-10 at 09:32 +0200, Heikki Linnakangas wrote:
> >> Hmm, so after running restore_command, check the file size and if it's
> >> too short, treat it the same as if restore_command returned non-zero?
> >> And it will be retried on the next iteration. Works for me, though OTOH
> >> it will then fail to complain about a genuinely WAL file that's
> >> truncated for some reason. I guess there's no way around that, even if
> >> you have a script as restore_command that does the file size check, it
> >> will have the same problem.
> >
> > Are we trying to re-invent pg_standby here?
>
> That's not the goal, but we seem to need some of the same functionality
> in the backend now.
I think you need to say why...
--
Simon Riggs www.2ndQuadrant.com
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
@ 2010-02-11 12:44 ` Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
0 siblings, 1 reply; 51+ messages in thread
From: Heikki Linnakangas @ 2010-02-11 12:44 UTC (permalink / raw)
To: Simon Riggs <simon@2ndQuadrant.com>; +Cc: Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
Simon Riggs wrote:
> On Thu, 2010-02-11 at 14:22 +0200, Heikki Linnakangas wrote:
>> Simon Riggs wrote:
>>> On Wed, 2010-02-10 at 09:32 +0200, Heikki Linnakangas wrote:
>>>> Hmm, so after running restore_command, check the file size and if it's
>>>> too short, treat it the same as if restore_command returned non-zero?
>>>> And it will be retried on the next iteration. Works for me, though OTOH
>>>> it will then fail to complain about a genuinely WAL file that's
>>>> truncated for some reason. I guess there's no way around that, even if
>>>> you have a script as restore_command that does the file size check, it
>>>> will have the same problem.
>>> Are we trying to re-invent pg_standby here?
>> That's not the goal, but we seem to need some of the same functionality
>> in the backend now.
>
> I think you need to say why...
See the quoted paragraph above. We should check the file size, so that
we will not fail if the WAL file is just being copied into the archive
directory.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
@ 2010-02-11 13:06 ` Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:28 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:41 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Dimitri Fontaine <dfontaine@hi-media.com>
0 siblings, 2 replies; 51+ messages in thread
From: Simon Riggs @ 2010-02-11 13:06 UTC (permalink / raw)
To: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; +Cc: Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
On Thu, 2010-02-11 at 14:44 +0200, Heikki Linnakangas wrote:
> Simon Riggs wrote:
> > On Thu, 2010-02-11 at 14:22 +0200, Heikki Linnakangas wrote:
> >> Simon Riggs wrote:
> >>> On Wed, 2010-02-10 at 09:32 +0200, Heikki Linnakangas wrote:
> >>>> Hmm, so after running restore_command, check the file size and if it's
> >>>> too short, treat it the same as if restore_command returned non-zero?
> >>>> And it will be retried on the next iteration. Works for me, though OTOH
> >>>> it will then fail to complain about a genuinely WAL file that's
> >>>> truncated for some reason. I guess there's no way around that, even if
> >>>> you have a script as restore_command that does the file size check, it
> >>>> will have the same problem.
> >>> Are we trying to re-invent pg_standby here?
> >> That's not the goal, but we seem to need some of the same functionality
> >> in the backend now.
> >
> > I think you need to say why...
>
> See the quoted paragraph above. We should check the file size, so that
> we will not fail if the WAL file is just being copied into the archive
> directory.
We can read, but that's not an explanation. By giving terse answers in
that way you are giving the impression that you don't want discussion on
these points.
If you were running pg_standby as the restore_command then this error
wouldn't happen. So you need to explain why running pg_standby cannot
solve your problem and why we must fix it by replicating code that has
previously existed elsewhere.
--
Simon Riggs www.2ndQuadrant.com
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
@ 2010-02-11 13:28 ` Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:46 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 14:01 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
1 sibling, 2 replies; 51+ messages in thread
From: Heikki Linnakangas @ 2010-02-11 13:28 UTC (permalink / raw)
To: Simon Riggs <simon@2ndQuadrant.com>; +Cc: Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
Simon Riggs wrote:
> If you were running pg_standby as the restore_command then this error
> wouldn't happen. So you need to explain why running pg_standby cannot
> solve your problem and why we must fix it by replicating code that has
> previously existed elsewhere.
pg_standby cannot be used with streaming replication.
I guess you're next question is: why not?
The startup process alternates between streaming, and restoring files
from archive using restore_command. It will progress using streaming as
long as it can, but if the connection is lost, it will try to poll the
archive until the connection is established again. The startup process
expects the restore_command to try to restore the file and fail if it's
not found. If the restore_command goes into sleep, waiting for the file
to arrive, that will defeat the retry logic in the server because the
startup process won't get control again to retry establishing the
connection.
That's the the essence of my proposal here:
http://archives.postgresql.org/message-id/4B50AFB4.4060902@enterprisedb.com
which is what has now been implemented.
To suppport a restore_command that does the sleeping itself, like
pg_standby, would require a major rearchitecting of the retry logic. And
I don't see why that'd desirable anyway. It's easier for the admin to
set up using simple commands like 'cp' or 'scp', than require him/her to
write scripts that handle the sleeping and retry logic.
The real problem we have right now is missing documentation. It's
starting to hurt us more and more every day, as more people start to
test this. As shown by this thread and some other recent posts.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:28 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
@ 2010-02-11 13:46 ` Simon Riggs <simon@2ndQuadrant.com>
1 sibling, 0 replies; 51+ messages in thread
From: Simon Riggs @ 2010-02-11 13:46 UTC (permalink / raw)
To: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; +Cc: Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
On Thu, 2010-02-11 at 15:28 +0200, Heikki Linnakangas wrote:
> Simon Riggs wrote:
> > If you were running pg_standby as the restore_command then this error
> > wouldn't happen. So you need to explain why running pg_standby cannot
> > solve your problem and why we must fix it by replicating code that has
> > previously existed elsewhere.
>
> pg_standby cannot be used with streaming replication.
> I guess you're next question is: why not?
>
> The startup process alternates between streaming, and restoring files
> from archive using restore_command. It will progress using streaming as
> long as it can, but if the connection is lost, it will try to poll the
> archive until the connection is established again. The startup process
> expects the restore_command to try to restore the file and fail if it's
> not found. If the restore_command goes into sleep, waiting for the file
> to arrive, that will defeat the retry logic in the server because the
> startup process won't get control again to retry establishing the
> connection.
Why does the startup process need to regain control? Why not just let it
sit and wait? Have you seen that if someone does use pg_standby or
similar scripts in the restore_command that the server will never regain
control in the way you hope. Would that cause a sporadic hang?
The overall design was previously that the solution implementor was in
charge of the archive and only they knew its characteristics.
It seems strange that we will be forced to explicitly ban people from
using a utility they were previously used to using and is still included
with the distro. Then we implement in the server the very things the
utility did. Only this time the solution implementor will not be in
control.
I would not be against implementing all aspects of pg_standby into the
server. It would make life easier in some ways. I am against
implementing only a *few* of the aspects because that leaves solution
architects in a difficult position to know what to do.
Please lay out some options here for discussion by the community. This
seems like a difficult area and not one to be patched up quickly.
> That's the the essence of my proposal here:
> http://archives.postgresql.org/message-id/4B50AFB4.4060902@enterprisedb.com
> which is what has now been implemented.
>
> To suppport a restore_command that does the sleeping itself, like
> pg_standby, would require a major rearchitecting of the retry logic. And
> I don't see why that'd desirable anyway. It's easier for the admin to
> set up using simple commands like 'cp' or 'scp', than require him/her to
> write scripts that handle the sleeping and retry logic.
>
>
> The real problem we have right now is missing documentation. It's
> starting to hurt us more and more every day, as more people start to
> test this. As shown by this thread and some other recent posts.
--
Simon Riggs www.2ndQuadrant.com
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:28 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
@ 2010-02-11 14:01 ` Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
1 sibling, 1 reply; 51+ messages in thread
From: Aidan Van Dyk @ 2010-02-11 14:01 UTC (permalink / raw)
To: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; +Cc: Simon Riggs <simon@2ndQuadrant.com>; Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
* Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> [100211 08:29]:
> To suppport a restore_command that does the sleeping itself, like
> pg_standby, would require a major rearchitecting of the retry logic. And
> I don't see why that'd desirable anyway. It's easier for the admin to
> set up using simple commands like 'cp' or 'scp', than require him/her to
> write scripts that handle the sleeping and retry logic.
But colour me confused, I'm still not understanding why this is any
different that with normal PITR recovery.
So even with a plain "cp" in your recovery command instead of a
sleep+copy (a la pg_standby, or PITR tools, or all the home-grown
solutions out thery), I'm not seeing how it's going to get "half files".
The only way I can see that is if you're out of disk space in your
recovering pg_xlog.
It's well know in PostgreSQL wal archivne - you don't just "shove" files
into the archive, you make sure they appear there with the right name
atomically. And if the master is only running the archive command on
whole WAL files, I just don't understand this whole short wal problem.
And don't try and tell me your just "poaching" files from a running
cluster's pg_xlog directory, because I'm going to cry...
a.
--
Aidan Van Dyk Create like a god,
aidan@highrise.ca command like a king,
http://www.highrise.ca/ work like a slave.
Attachments:
signature.asc (application/pgp-signature, 188B) download
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:28 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:01 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
@ 2010-02-11 14:17 ` Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:42 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
0 siblings, 1 reply; 51+ messages in thread
From: Heikki Linnakangas @ 2010-02-11 14:17 UTC (permalink / raw)
To: Aidan Van Dyk <aidan@highrise.ca>; +Cc: Simon Riggs <simon@2ndQuadrant.com>; Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
Aidan Van Dyk wrote:
> But colour me confused, I'm still not understanding why this is any
> different that with normal PITR recovery.
>
> So even with a plain "cp" in your recovery command instead of a
> sleep+copy (a la pg_standby, or PITR tools, or all the home-grown
> solutions out thery), I'm not seeing how it's going to get "half files".
If the file is just being copied to the archive when restore_command
('cp', say) is launched, it will copy a half file. That's not a problem
for PITR, because PITR will end at the end of valid WAL anyway, but
returning a half WAL file in standby mode is a problem.
> It's well know in PostgreSQL wal archivne - you don't just "shove" files
> into the archive, you make sure they appear there with the right name
> atomically. And if the master is only running the archive command on
> whole WAL files, I just don't understand this whole short wal problem.
Yeah, if you're careful about that, then this change isn't required. But
pg_standby protects against that, so I think it'd be reasonable to have
the same level of protection built-in. It's not a lot of code.
We could well just document that you should do that, ie. make sure the
file appears in the archive atomically with the right size.
> And don't try and tell me your just "poaching" files from a running
> cluster's pg_xlog directory, because I'm going to cry...
No :-).
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:28 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:01 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
@ 2010-02-11 14:42 ` Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 17:04 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 17:29 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
0 siblings, 2 replies; 51+ messages in thread
From: Aidan Van Dyk @ 2010-02-11 14:42 UTC (permalink / raw)
To: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; +Cc: Simon Riggs <simon@2ndQuadrant.com>; Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
* Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> [100211 09:17]:
> If the file is just being copied to the archive when restore_command
> ('cp', say) is launched, it will copy a half file. That's not a problem
> for PITR, because PITR will end at the end of valid WAL anyway, but
> returning a half WAL file in standby mode is a problem.
But it can be a problem - without the last WAL (or at least enough of
it) the master switched and archived, you have no guarantee of having
being consistent again (I'm thinking specifically of recovering from a
fresh backup)
> Yeah, if you're careful about that, then this change isn't required. But
> pg_standby protects against that, so I think it'd be reasonable to have
> the same level of protection built-in. It's not a lot of code.
This 1 check isn't, but what about the rest of the things pg_standby
does. How much functionality should we bring it? Ideally, "all" of it.
> We could well just document that you should do that, ie. make sure the
> file appears in the archive atomically with the right size.
I have to admit, today was the first time I went and re-read the PITR
docs, and no, the docs don't seem to talk about that... Maybe it was
just plain obvious to me because it (the atomic apperance) is something
unix devloppers have always had to deal with, so it's ingrained in me.
But I'm *sure* that I've seen that bandied around as common knowledge on
the lists, and one of the reasons we alway see warnings about using
rsync instead of plain SCP, etc.
So ya, we should probably mention that somewhere in the docs. Section
24.3.6. Caveats?
a.
--
Aidan Van Dyk Create like a god,
aidan@highrise.ca command like a king,
http://www.highrise.ca/ work like a slave.
Attachments:
signature.asc (application/pgp-signature, 188B) download
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:28 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:01 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:42 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
@ 2010-02-11 17:04 ` Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 17:31 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 18:08 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 2 replies; 51+ messages in thread
From: Heikki Linnakangas @ 2010-02-11 17:04 UTC (permalink / raw)
To: Aidan Van Dyk <aidan@highrise.ca>; +Cc: Simon Riggs <simon@2ndQuadrant.com>; Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
Aidan Van Dyk wrote:
> * Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> [100211 09:17]:
>
>> If the file is just being copied to the archive when restore_command
>> ('cp', say) is launched, it will copy a half file. That's not a problem
>> for PITR, because PITR will end at the end of valid WAL anyway, but
>> returning a half WAL file in standby mode is a problem.
>
> But it can be a problem - without the last WAL (or at least enough of
> it) the master switched and archived, you have no guarantee of having
> being consistent again (I'm thinking specifically of recovering from a
> fresh backup)
You have to wait for the last WAL file required by the backup to be
archived before starting recovery. Otherwise there's no guarantee anyway.
>> We could well just document that you should do that, ie. make sure the
>> file appears in the archive atomically with the right size.
>
> I have to admit, today was the first time I went and re-read the PITR
> docs, and no, the docs don't seem to talk about that... Maybe it was
> just plain obvious to me because it (the atomic apperance) is something
> unix devloppers have always had to deal with, so it's ingrained in me.
> But I'm *sure* that I've seen that bandied around as common knowledge on
> the lists, and one of the reasons we alway see warnings about using
> rsync instead of plain SCP, etc.
>
> So ya, we should probably mention that somewhere in the docs. Section
> 24.3.6. Caveats?
-1. it isn't necessary for PITR. It's a new requirement for
standby_mode='on', unless we add the file size check into the backend. I
think we should add the file size check to the backend instead and save
admins the headache.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:28 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:01 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:42 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 17:04 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
@ 2010-02-11 17:31 ` Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 18:11 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
1 sibling, 1 reply; 51+ messages in thread
From: Aidan Van Dyk @ 2010-02-11 17:31 UTC (permalink / raw)
To: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; +Cc: Simon Riggs <simon@2ndQuadrant.com>; Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
* Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> [100211 12:04]:
> > But it can be a problem - without the last WAL (or at least enough of
> > it) the master switched and archived, you have no guarantee of having
> > being consistent again (I'm thinking specifically of recovering from a
> > fresh backup)
>
> You have to wait for the last WAL file required by the backup to be
> archived before starting recovery. Otherwise there's no guarantee anyway.
Right, but now define "wait for". If you pull only "half" the last WAL
(because you've accepted that you *can* have short WAL files in the
archive), you have the problem with PITR.
Is it "wait for it to be in the archive", or "wait for it to be in the
archive, and be sure that the contents satisfy some criteria".
I've always made my PITR such that "in the archive" (i.e. the first
moment a recovery can see it) implies that it's bit-for-bit identical to
the original (or at least as bit-for-bit I can assume by checking
various hashes I can afford to). I just assumed that was kind of common
practice.
I'm amazed that "partial WAL" files are every available in anyones
archive, for anyone's restore command to actually pull. I find that
scarry, and sure, probably won't regularly be noticed... But man, I'ld
hate the time I need that emergency PITR restore to be the one time when
it needs that WAL, pulls it slightly before the copy has finished (i.e.
the master is pushing the WAL over a WAN to a 2nd site), and have my
restore complete consistently...
a.
--
Aidan Van Dyk Create like a god,
aidan@highrise.ca command like a king,
http://www.highrise.ca/ work like a slave.
Attachments:
signature.asc (application/pgp-signature, 188B) download
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:28 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:01 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:42 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 17:04 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 17:31 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
@ 2010-02-11 18:11 ` Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 18:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Kevin Grittner <Kevin.Grittner@wicourts.gov>
0 siblings, 1 reply; 51+ messages in thread
From: Heikki Linnakangas @ 2010-02-11 18:11 UTC (permalink / raw)
To: Aidan Van Dyk <aidan@highrise.ca>; +Cc: Simon Riggs <simon@2ndQuadrant.com>; Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
Aidan Van Dyk wrote:
> * Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> [100211 12:04]:
>
>>> But it can be a problem - without the last WAL (or at least enough of
>>> it) the master switched and archived, you have no guarantee of having
>>> being consistent again (I'm thinking specifically of recovering from a
>>> fresh backup)
>> You have to wait for the last WAL file required by the backup to be
>> archived before starting recovery. Otherwise there's no guarantee anyway.
>
> Right, but now define "wait for".
As in don't start postmaster until the last WAL file needed by backup
has been fully copied to the archive.
> (because you've accepted that you *can* have short WAL files in the
> archive)
Only momentarily, while the copy is in progress.
> I've always made my PITR such that "in the archive" (i.e. the first
> moment a recovery can see it) implies that it's bit-for-bit identical to
> the original (or at least as bit-for-bit I can assume by checking
> various hashes I can afford to). I just assumed that was kind of common
> practice.
It's certainly good practice, agreed, but hasn't been absolutely required.
> I'm amazed that "partial WAL" files are every available in anyones
> archive, for anyone's restore command to actually pull. I find that
> scarry, and sure, probably won't regularly be noticed... But man, I'ld
> hate the time I need that emergency PITR restore to be the one time when
> it needs that WAL, pulls it slightly before the copy has finished (i.e.
> the master is pushing the WAL over a WAN to a 2nd site), and have my
> restore complete consistently...
It's not as dramatic as you make it sound. We're only talking about the
last WAL file, and only when it's just being copied to the archive. If
you have a archive_command like 'cp', and you look at the archive at the
same millisecond that 'cp' runs, then yes you will see that the latest
WAL file in the archive is only partially copied. It's not a problem for
robustness; if you had looked one millisecond earlier you would not have
seen the file there at all.
Windows 'copy' command preallocates the whole file, which poses a
different problem: if you look at the file while it's being copied, the
file has the right length, but isn't in fact fully copied yet. I think
'rsync' has the same problem. To avoid that issue, you have to use
something like copy+rename to make it atomic. There isn't much we can do
in the server (or in pg_standby) to work around that, because there's no
way to distinguish a file that's being copied from a fully-copied
corrupt file.
We do advise to set up an archive_command that doesn't overwrite
existing files. That together with a partial WAL segment can cause a
problem: if archive_command crashes while it's writing the file, leaving
a partial file in the archive, the subsequent run of archive_command
won't overwrite it and will get stuck trying. However, there's a small
window for that even if you put the file into the archive atomically: if
you crash just after fully copying the file, but before the .done file
is created, upon restart the server will also try to copy the file to
archive, find that it already exists, and fail.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:28 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:01 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:42 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 17:04 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 17:31 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 18:11 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
@ 2010-02-11 18:22 ` Kevin Grittner <Kevin.Grittner@wicourts.gov>
2010-02-11 19:45 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Garick Hamlin <ghamlin@isc.upenn.edu>
0 siblings, 1 reply; 51+ messages in thread
From: Kevin Grittner @ 2010-02-11 18:22 UTC (permalink / raw)
To: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; Aidan Van Dyk <aidan@highrise.ca>; +Cc: Simon Riggs <simon@2ndQuadrant.com>; Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> wrote:
> I think 'rsync' has the same problem.
There is a switch you can use to create the problem under rsync, but
by default rsync copies to a temporary file name and moves the
completed file to the target name.
-Kevin
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:28 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:01 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:42 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 17:04 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 17:31 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 18:11 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 18:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Kevin Grittner <Kevin.Grittner@wicourts.gov>
@ 2010-02-11 19:45 ` Garick Hamlin <ghamlin@isc.upenn.edu>
0 siblings, 0 replies; 51+ messages in thread
From: Garick Hamlin @ 2010-02-11 19:45 UTC (permalink / raw)
To: Kevin Grittner <Kevin.Grittner@wicourts.gov>; +Cc: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; Aidan Van Dyk <aidan@highrise.ca>; Simon Riggs <simon@2ndQuadrant.com>; Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
On Thu, Feb 11, 2010 at 01:22:44PM -0500, Kevin Grittner wrote:
> Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> wrote:
>
> > I think 'rsync' has the same problem.
>
> There is a switch you can use to create the problem under rsync, but
> by default rsync copies to a temporary file name and moves the
> completed file to the target name.
>
> -Kevin
I don't use PITR, So I don't know any of the well understood facts about
PITR with postgres, but my understanding with rsync is ...
It doesn't fsync data before rename, its something like: open / write /
close / rename, which could lead to zero length files on some filesystems.
(are there other anomalies to worry about here?)
Would that be a concern?
Garick
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:28 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:01 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:42 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 17:04 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
@ 2010-02-11 18:08 ` Tom Lane <tgl@sss.pgh.pa.us>
2010-02-11 18:14 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
1 sibling, 1 reply; 51+ messages in thread
From: Tom Lane @ 2010-02-11 18:08 UTC (permalink / raw)
To: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; +Cc: Aidan Van Dyk <aidan@highrise.ca>; Simon Riggs <simon@2ndQuadrant.com>; Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> writes:
> -1. it isn't necessary for PITR. It's a new requirement for
> standby_mode='on', unless we add the file size check into the backend. I
> think we should add the file size check to the backend instead and save
> admins the headache.
I think the file size check needs to be in the backend purely on safety
grounds. Whether we make pg_standby useful for this context is a
different discussion.
regards, tom lane
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:28 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:01 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:42 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 17:04 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 18:08 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Tom Lane <tgl@sss.pgh.pa.us>
@ 2010-02-11 18:14 ` Simon Riggs <simon@2ndQuadrant.com>
2010-02-12 07:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
0 siblings, 1 reply; 51+ messages in thread
From: Simon Riggs @ 2010-02-11 18:14 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; Aidan Van Dyk <aidan@highrise.ca>; Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
On Thu, 2010-02-11 at 13:08 -0500, Tom Lane wrote:
> Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> writes:
> > -1. it isn't necessary for PITR. It's a new requirement for
> > standby_mode='on', unless we add the file size check into the backend. I
> > think we should add the file size check to the backend instead and save
> > admins the headache.
>
> I think the file size check needs to be in the backend purely on safety
> grounds. Whether we make pg_standby useful for this context is a
> different discussion.
Happy with that.
--
Simon Riggs www.2ndQuadrant.com
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:28 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:01 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:42 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 17:04 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 18:08 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Tom Lane <tgl@sss.pgh.pa.us>
2010-02-11 18:14 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
@ 2010-02-12 07:37 ` Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
0 siblings, 0 replies; 51+ messages in thread
From: Heikki Linnakangas @ 2010-02-12 07:37 UTC (permalink / raw)
To: Simon Riggs <simon@2ndQuadrant.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Aidan Van Dyk <aidan@highrise.ca>; Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
Simon Riggs wrote:
> On Thu, 2010-02-11 at 13:08 -0500, Tom Lane wrote:
>> Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> writes:
>>> -1. it isn't necessary for PITR. It's a new requirement for
>>> standby_mode='on', unless we add the file size check into the backend. I
>>> think we should add the file size check to the backend instead and save
>>> admins the headache.
>> I think the file size check needs to be in the backend purely on safety
>> grounds. Whether we make pg_standby useful for this context is a
>> different discussion.
>
> Happy with that.
Committed such a check now.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:28 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:01 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:42 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
@ 2010-02-11 17:29 ` Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 18:11 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-18 11:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
1 sibling, 2 replies; 51+ messages in thread
From: Heikki Linnakangas @ 2010-02-11 17:29 UTC (permalink / raw)
To: Aidan Van Dyk <aidan@highrise.ca>; +Cc: Simon Riggs <simon@2ndQuadrant.com>; Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
Aidan Van Dyk wrote:
> * Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> [100211 09:17]:
>
>> Yeah, if you're careful about that, then this change isn't required. But
>> pg_standby protects against that, so I think it'd be reasonable to have
>> the same level of protection built-in. It's not a lot of code.
>
> This 1 check isn't, but what about the rest of the things pg_standby
> does. How much functionality should we bring it? Ideally, "all" of it.
Well, how about we bite the bullet then and add enough bells and
whistles to the backend that pg_standby really isn't needed anymore, and
remove it from contrib?
Looking at the options to pg_standby, we're not missing much:
> Options:
> -c copies file from archive (default)
> -l links into archive (leaves file in archive)
Obsolete (link mode not supported anymore)
> -d generate lots of debugging output (testing only)
We have DEBUG statements in the server...
> -k NUMFILESTOKEEP if RESTARTWALFILE not used, removes files prior to limit
> (0 keeps all)
This is dangerous, and obsoleted by the RESTARTWALFILE option (%r).
> -r MAXRETRIES max number of times to retry, with progressive wait
> (default=3)
Frankly this seems pretty useless, but it would be easy to implement
> -s SLEEPTIME seconds to wait between file checks (min=1, max=60,
> default=5)
The sleep time in the backend is currently hard-coded at 5 s. Should we
make it configurable?
> -t TRIGGERFILE defines a trigger file to initiate failover (no default)
We have this in the backend already.
> -w MAXWAITTIME max seconds to wait for a file (0=no limit) (default=0)
We don't have a timeout in the backend. Should we? (the timeout in
pg_standby won't work, even if we add the option to use pg_standby with
standby_mode='on' as Simon suggested)
So the only major feature we're missing is the ability to clean up old
files.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:28 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:01 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:42 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 17:29 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
@ 2010-02-11 18:11 ` Simon Riggs <simon@2ndQuadrant.com>
1 sibling, 0 replies; 51+ messages in thread
From: Simon Riggs @ 2010-02-11 18:11 UTC (permalink / raw)
To: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; +Cc: Aidan Van Dyk <aidan@highrise.ca>; Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
On Thu, 2010-02-11 at 19:29 +0200, Heikki Linnakangas wrote:
> Aidan Van Dyk wrote:
> > * Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> [100211 09:17]:
> >
> >> Yeah, if you're careful about that, then this change isn't required. But
> >> pg_standby protects against that, so I think it'd be reasonable to have
> >> the same level of protection built-in. It's not a lot of code.
> >
> > This 1 check isn't, but what about the rest of the things pg_standby
> > does. How much functionality should we bring it? Ideally, "all" of it.
>
> Well, how about we bite the bullet then and add enough bells and
> whistles to the backend that pg_standby really isn't needed anymore, and
> remove it from contrib?
>
> Looking at the options to pg_standby, we're not missing much:
You're attitude to this makes me laugh, and cry.
Removing pg_standby can be done and if you manage it well, I would be
happy. I laugh because it seems like you think I have some ego tied up
in it. It's hardly my finest hour of coding, especially since I've not
had the ability to improve it much before now.
Not missing *much*?! I cry because you're so blase about missing one or
two essential features that took years to put in place. Keeping
pg_standby at least guarantees we can do something about it when we
discover you didn't do a good enough job removing it.
--
Simon Riggs www.2ndQuadrant.com
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:28 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:01 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:42 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 17:29 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
@ 2010-02-18 11:37 ` Fujii Masao <masao.fujii@gmail.com>
1 sibling, 0 replies; 51+ messages in thread
From: Fujii Masao @ 2010-02-18 11:37 UTC (permalink / raw)
To: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; +Cc: Aidan Van Dyk <aidan@highrise.ca>; Simon Riggs <simon@2ndquadrant.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
On Fri, Feb 12, 2010 at 2:29 AM, Heikki Linnakangas
<heikki.linnakangas@enterprisedb.com> wrote:
> So the only major feature we're missing is the ability to clean up old
> files.
I found another missing feature in new file-based log shipping (i.e.,
standby_mode is enabled and 'cp' is used as restore_command).
After the trigger file is found, the startup process with pg_standby
tries to replay all of the WAL files in both pg_xlog and the archive.
So, when the primary fails, if the latest WAL file in pg_xlog of the
primary can be read, we can prevent the data loss by copying it to
pg_xlog of the standby before creating the trigger file.
On the other hand, the startup process with standby mode doesn't
replay the WAL files in pg_xlog after the trigger file is found. So
failover always causes the data loss even if the latest WAL file can
be read from the primary. And if the latest WAL file is copied to the
archive instead, it can be replayed but a PANIC error would happen
because it's not filled.
We should remove this restriction?
Regards,
--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
@ 2010-02-11 13:41 ` Dimitri Fontaine <dfontaine@hi-media.com>
2010-02-11 13:50 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
1 sibling, 1 reply; 51+ messages in thread
From: Dimitri Fontaine @ 2010-02-11 13:41 UTC (permalink / raw)
To: Simon Riggs <simon@2ndQuadrant.com>; +Cc: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
Simon Riggs <simon@2ndQuadrant.com> writes:
> If you were running pg_standby as the restore_command then this error
> wouldn't happen. So you need to explain why running pg_standby cannot
> solve your problem and why we must fix it by replicating code that has
> previously existed elsewhere.
Let me try.
pg_standby will not let the server get back to streaming replication
mode once it's done with driving the replay of all the WAL files
available in the archive, but will have the server sits there waiting
for the next file.
The way we want that is implemented now is to have the server switch
back and forth between replaying from the archive and streaming from the
master. So we want the server to restore from the archive the same way
pg_standby used to, except that if the archive does not contain the next
WAL files, we want to get back to streaming.
And the archive reading will resume at next network glitch.
I think it's the reasonning, I hope it explains what you see happening.
--
dim
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:41 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Dimitri Fontaine <dfontaine@hi-media.com>
@ 2010-02-11 13:50 ` Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:55 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
0 siblings, 1 reply; 51+ messages in thread
From: Simon Riggs @ 2010-02-11 13:50 UTC (permalink / raw)
To: Dimitri Fontaine <dfontaine@hi-media.com>; +Cc: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
On Thu, 2010-02-11 at 14:41 +0100, Dimitri Fontaine wrote:
> Simon Riggs <simon@2ndQuadrant.com> writes:
> > If you were running pg_standby as the restore_command then this error
> > wouldn't happen. So you need to explain why running pg_standby cannot
> > solve your problem and why we must fix it by replicating code that has
> > previously existed elsewhere.
>
> Let me try.
>
> pg_standby will not let the server get back to streaming replication
> mode once it's done with driving the replay of all the WAL files
> available in the archive, but will have the server sits there waiting
> for the next file.
>
> The way we want that is implemented now is to have the server switch
> back and forth between replaying from the archive and streaming from the
> master. So we want the server to restore from the archive the same way
> pg_standby used to, except that if the archive does not contain the next
> WAL files, we want to get back to streaming.
>
> And the archive reading will resume at next network glitch.
>
> I think it's the reasonning, I hope it explains what you see happening.
OK, thanks.
One question then: how do we ensure that the archive does not grow too
big? pg_standby cleans down the archive using %R. That function appears
to not exist anymore.
--
Simon Riggs www.2ndQuadrant.com
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:41 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Dimitri Fontaine <dfontaine@hi-media.com>
2010-02-11 13:50 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
@ 2010-02-11 13:55 ` Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
0 siblings, 1 reply; 51+ messages in thread
From: Heikki Linnakangas @ 2010-02-11 13:55 UTC (permalink / raw)
To: Simon Riggs <simon@2ndQuadrant.com>; +Cc: Dimitri Fontaine <dfontaine@hi-media.com>; Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
Simon Riggs wrote:
> One question then: how do we ensure that the archive does not grow too
> big? pg_standby cleans down the archive using %R. That function appears
> to not exist anymore.
You can still use %R. Of course, plain 'cp' won't know what to do with
it, so a script will then be required. We should probably provide a
sample of that in the docs, or even a ready-made tool similar to
pg_standby but without the waiting.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:41 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Dimitri Fontaine <dfontaine@hi-media.com>
2010-02-11 13:50 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:55 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
@ 2010-02-11 14:17 ` Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 14:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
0 siblings, 1 reply; 51+ messages in thread
From: Simon Riggs @ 2010-02-11 14:17 UTC (permalink / raw)
To: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; +Cc: Dimitri Fontaine <dfontaine@hi-media.com>; Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
On Thu, 2010-02-11 at 15:55 +0200, Heikki Linnakangas wrote:
> Simon Riggs wrote:
> > One question then: how do we ensure that the archive does not grow too
> > big? pg_standby cleans down the archive using %R. That function appears
> > to not exist anymore.
>
> You can still use %R. Of course, plain 'cp' won't know what to do with
> it, so a script will then be required. We should probably provide a
> sample of that in the docs, or even a ready-made tool similar to
> pg_standby but without the waiting.
So we still need a script but it can't be pg_standby? Hmmm, OK...
Might it not be simpler to add a parameter onto pg_standby?
We send %s to tell pg_standby the standby_mode of the server which is
calling it so it can decide how to act in each case.
--
Simon Riggs www.2ndQuadrant.com
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:41 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Dimitri Fontaine <dfontaine@hi-media.com>
2010-02-11 13:50 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:55 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
@ 2010-02-11 14:22 ` Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:38 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 14:55 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Greg Smith <greg@2ndquadrant.com>
2010-02-12 05:38 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
0 siblings, 3 replies; 51+ messages in thread
From: Heikki Linnakangas @ 2010-02-11 14:22 UTC (permalink / raw)
To: Simon Riggs <simon@2ndQuadrant.com>; +Cc: Dimitri Fontaine <dfontaine@hi-media.com>; Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
Simon Riggs wrote:
> Might it not be simpler to add a parameter onto pg_standby?
> We send %s to tell pg_standby the standby_mode of the server which is
> calling it so it can decide how to act in each case.
That would work too, but it doesn't seem any simpler to me. On the contrary.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:41 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Dimitri Fontaine <dfontaine@hi-media.com>
2010-02-11 13:50 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:55 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 14:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
@ 2010-02-11 14:38 ` Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 15:21 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 15:29 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Euler Taveira de Oliveira <euler@timbira.com>
2 siblings, 2 replies; 51+ messages in thread
From: Simon Riggs @ 2010-02-11 14:38 UTC (permalink / raw)
To: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; +Cc: Dimitri Fontaine <dfontaine@hi-media.com>; Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
On Thu, 2010-02-11 at 16:22 +0200, Heikki Linnakangas wrote:
> Simon Riggs wrote:
> > Might it not be simpler to add a parameter onto pg_standby?
> > We send %s to tell pg_standby the standby_mode of the server which is
> > calling it so it can decide how to act in each case.
>
> That would work too, but it doesn't seem any simpler to me. On the contrary.
It would mean that pg_standby would act appropriately according to the
setting of standby_mode. So you wouldn't need multiple examples of use,
it would all just work whatever the setting of standby_mode. Nice simple
entry in the docs.
We've said we need a script and pg_standby is it. Having a second
script, pg_standby2, rather than adding something to the existing tools
just seems confusing and easier to get wrong.
I'm happy to do the patch if you like.
--
Simon Riggs www.2ndQuadrant.com
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:41 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Dimitri Fontaine <dfontaine@hi-media.com>
2010-02-11 13:50 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:55 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 14:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:38 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
@ 2010-02-11 15:21 ` Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
1 sibling, 0 replies; 51+ messages in thread
From: Heikki Linnakangas @ 2010-02-11 15:21 UTC (permalink / raw)
To: Simon Riggs <simon@2ndQuadrant.com>; +Cc: Dimitri Fontaine <dfontaine@hi-media.com>; Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
Simon Riggs wrote:
> On Thu, 2010-02-11 at 16:22 +0200, Heikki Linnakangas wrote:
>> Simon Riggs wrote:
>>> Might it not be simpler to add a parameter onto pg_standby?
>>> We send %s to tell pg_standby the standby_mode of the server which is
>>> calling it so it can decide how to act in each case.
>> That would work too, but it doesn't seem any simpler to me. On the contrary.
>
> It would mean that pg_standby would act appropriately according to the
> setting of standby_mode. So you wouldn't need multiple examples of use,
> it would all just work whatever the setting of standby_mode. Nice simple
> entry in the docs.
>
> We've said we need a script and pg_standby is it. Having a second
> script, pg_standby2, rather than adding something to the existing tools
> just seems confusing and easier to get wrong.
>
> I'm happy to do the patch if you like.
Well, doesn't really seem that useful to me, but I won't object if you
want to do it.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:41 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Dimitri Fontaine <dfontaine@hi-media.com>
2010-02-11 13:50 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:55 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 14:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:38 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
@ 2010-02-11 15:29 ` Euler Taveira de Oliveira <euler@timbira.com>
1 sibling, 0 replies; 51+ messages in thread
From: Euler Taveira de Oliveira @ 2010-02-11 15:29 UTC (permalink / raw)
To: Simon Riggs <simon@2ndQuadrant.com>; +Cc: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; Dimitri Fontaine <dfontaine@hi-media.com>; Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
Simon Riggs escreveu:
> It would mean that pg_standby would act appropriately according to the
> setting of standby_mode. So you wouldn't need multiple examples of use,
> it would all just work whatever the setting of standby_mode. Nice simple
> entry in the docs.
>
+1. I like the %s idea. IMHO fixing pg_standby for SR is a must-fix. I'm
foreseeing a lot of users asking why pg_standby doesn't work on SR mode ...
--
Euler Taveira de Oliveira
http://www.timbira.com/
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:41 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Dimitri Fontaine <dfontaine@hi-media.com>
2010-02-11 13:50 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:55 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 14:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
@ 2010-02-11 14:55 ` Greg Smith <greg@2ndquadrant.com>
2 siblings, 0 replies; 51+ messages in thread
From: Greg Smith @ 2010-02-11 14:55 UTC (permalink / raw)
To: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; +Cc: Simon Riggs <simon@2ndQuadrant.com>; Dimitri Fontaine <dfontaine@hi-media.com>; Fujii Masao <masao.fujii@gmail.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
Heikki Linnakangas wrote:
> Simon Riggs wrote:
>
>> Might it not be simpler to add a parameter onto pg_standby?
>> We send %s to tell pg_standby the standby_mode of the server which is
>> calling it so it can decide how to act in each case.
>>
>
> That would work too, but it doesn't seem any simpler to me. On the contrary.
>
I don't know that the ideas are mutually exclusive. Should the server
internals do a basic check that the files they're about to process seem
sane before processing them? Yes. Should we provide a full-featured
program that knows how far back it can delete archives rather than
expecting people to write their own? Yes. And a modified pg_standby
seems the easiest way to do that, since it already knows how to do the
computations, among other benefits.
I already have a work in progress patch to pg_standby I'm racing to
finish here that cleans up some of the UI warts in the program,
including the scary sounding and avoidable warnings Selena submitted a
patch to improve. I think I'm going to add this feature to it, too,
whether or not it's deemed necessary. I'm really tired of example
setups provided that say "just write a simple script using cp like
<trivial example>" and then supporting those non-production quality
messes in the field. My scripts for this sort of thing have more error
checking in them than functional code. And pg_standby already has a
healthy sense of paranoia built into it.
--
Greg Smith 2ndQuadrant US Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.us
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:41 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Dimitri Fontaine <dfontaine@hi-media.com>
2010-02-11 13:50 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:55 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 14:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
@ 2010-02-12 05:38 ` Fujii Masao <masao.fujii@gmail.com>
2010-02-12 12:51 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2 siblings, 1 reply; 51+ messages in thread
From: Fujii Masao @ 2010-02-12 05:38 UTC (permalink / raw)
To: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; +Cc: Simon Riggs <simon@2ndquadrant.com>; Dimitri Fontaine <dfontaine@hi-media.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
On Thu, Feb 11, 2010 at 11:22 PM, Heikki Linnakangas
<heikki.linnakangas@enterprisedb.com> wrote:
> Simon Riggs wrote:
>> Might it not be simpler to add a parameter onto pg_standby?
>> We send %s to tell pg_standby the standby_mode of the server which is
>> calling it so it can decide how to act in each case.
>
> That would work too, but it doesn't seem any simpler to me. On the contrary.
Agreed.
There could be three kinds of SR configurations. Let's think of them separately.
(1) SR without restore_command
(2) SR with 'cp'
(3) SR with pg_standby
(1) is the straightforward configuration. In this case the standby replays only
the WAL files in pg_xlog directory, and starts SR when it has found the invalid
record or been able to find no more WAL file. Then if SR is terminated for some
reasons, the standby would periodically try to connect to the primary and start
SR again. If you choose this, you don't need to care about the problem discussed
on the thread.
In the (2) case the standby replays the WAL files in not only pg_xlog but also
the archive, and starts SR when it has found the invalid record or been able to
find no more WAL file. If the archive is shared between the primary and the
standby, the standby might restore the partial WAL file being archived (copied)
by the primary. This could happen because 'cp' is not an atomic operation.
Currently when the standby finds the WAL file whose file size is less than 16MB,
it emits the FATAL error. This is the problem that I presented upthread. That is
undesirable behavior, so I proposed to just treat that case the same as if no
more WAL file is found. If so, the standby can start SR instead of emitting the
FATAL error. (2) is useful configuration as described in Heikki's
commig message.
http://archives.postgresql.org/pgsql-committers/2010-01/msg00395.php
(3) was unexpected configuration (at least for me). This would work fine as a
*file-based* log shipping but not SR. Since pg_standby doesn't return when no
more WAL file is found in the archive (i.e., it waits until the next complete
WAL file is available), SR will never start. OTOH, since pg_standby treats the
partial file as "nonexistence", the problem discussed on this thread doesn't
happen.
Questions:
(A) Is my proposal for (2) reasonable? For me, Yes.
(B) Should we allow (3) to work as "streaming replication"? In fact, we should
create the new mode that makes pg_standby return as soon as it doesn't find
a complete WAL file in the archive? I agree with Heikki, i.e., don't think
it's worth doing. Though pg_standby already has the capability to remove the
old WAL files, we would still need the cron job that removes them
periodically
because pg_standby is not executed during SR is running normally.
Regards,
--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:41 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Dimitri Fontaine <dfontaine@hi-media.com>
2010-02-11 13:50 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:55 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 14:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-12 05:38 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
@ 2010-02-12 12:51 ` Simon Riggs <simon@2ndQuadrant.com>
2010-02-12 13:10 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-12 14:34 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
0 siblings, 2 replies; 51+ messages in thread
From: Simon Riggs @ 2010-02-12 12:51 UTC (permalink / raw)
To: Fujii Masao <masao.fujii@gmail.com>; +Cc: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; Dimitri Fontaine <dfontaine@hi-media.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
On Fri, 2010-02-12 at 14:38 +0900, Fujii Masao wrote:
> On Thu, Feb 11, 2010 at 11:22 PM, Heikki Linnakangas
> <heikki.linnakangas@enterprisedb.com> wrote:
> > Simon Riggs wrote:
> >> Might it not be simpler to add a parameter onto pg_standby?
> >> We send %s to tell pg_standby the standby_mode of the server which is
> >> calling it so it can decide how to act in each case.
> >
> > That would work too, but it doesn't seem any simpler to me. On the contrary.
>
> Agreed.
>
> There could be three kinds of SR configurations. Let's think of them separately.
>
> (1) SR without restore_command
> (2) SR with 'cp'
> (3) SR with pg_standby
Thanks for the explanation.
> (1) is the straightforward configuration. In this case the standby replays only
> the WAL files in pg_xlog directory, and starts SR when it has found the invalid
> record or been able to find no more WAL file. Then if SR is terminated for some
> reasons, the standby would periodically try to connect to the primary and start
> SR again. If you choose this, you don't need to care about the problem discussed
> on the thread.
>
> In the (2) case the standby replays the WAL files in not only pg_xlog but also
> the archive, and starts SR when it has found the invalid record or been able to
> find no more WAL file. If the archive is shared between the primary and the
> standby, the standby might restore the partial WAL file being archived (copied)
> by the primary. This could happen because 'cp' is not an atomic operation.
>
> Currently when the standby finds the WAL file whose file size is less than 16MB,
> it emits the FATAL error. This is the problem that I presented upthread. That is
> undesirable behavior, so I proposed to just treat that case the same as if no
> more WAL file is found. If so, the standby can start SR instead of emitting the
> FATAL error. (2) is useful configuration as described in Heikki's
> commig message.
> http://archives.postgresql.org/pgsql-committers/2010-01/msg00395.php
> (3) was unexpected configuration (at least for me). This would work fine as a
> *file-based* log shipping but not SR. Since pg_standby doesn't return when no
> more WAL file is found in the archive (i.e., it waits until the next complete
> WAL file is available), SR will never start. OTOH, since pg_standby treats the
> partial file as "nonexistence", the problem discussed on this thread doesn't
> happen.
When we refer to "pg_standby" we mean any script. pg_standby is just a
reference implementation of a useful script. The discussion shouldn't
really focus on pg_standby, nor should we think of it as a different
approach. My original question was whether we are seeking to remove
pg_standby and, if so, have we implemented all of the technical features
that pg_standby provides? Worryingly the answer seems to be Yes and No.
I don't care if we get rid of pg_standby as long as we retain all the
things it does. *Losing* features is not acceptable.
> Questions:
> (A) Is my proposal for (2) reasonable? For me, Yes.
> (B) Should we allow (3) to work as "streaming replication"? In fact, we should
> create the new mode that makes pg_standby return as soon as it doesn't find
> a complete WAL file in the archive? I agree with Heikki, i.e., don't think
> it's worth doing. Though pg_standby already has the capability to remove the
> old WAL files, we would still need the cron job that removes them
> periodically
> because pg_standby is not executed during SR is running normally.
Yes, I realised that myself overnight and was going to raise this point
with you today.
In 8.4 it is pg_standby that was responsible for clearing down the
archive, which is why I suggested using pg_standby for that again. I
agree that will not work. The important thing is not pg_standby but that
we have a valid mechanism for clearing down the archive.
If (2) is a fully supported replication method then we cannot rely on
the existence of an external cron job to clear down the archive. Most
importantly, that cron job would not know the point up to which to clear
down the archive, the information given to pg_standby by %r.
So I suggest that you have a new action that gets called after every
checkpoint to clear down the archive. It will remove all files from the
archive prior to %r. We can implement that as a sequence of unlink()s
from within the server, or we can just call a script to do it. I prefer
the latter approach. However we do it, we need something initiated by
the server to maintain the archive and stop it from overflowing.
--
Simon Riggs www.2ndQuadrant.com
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:41 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Dimitri Fontaine <dfontaine@hi-media.com>
2010-02-11 13:50 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:55 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 14:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-12 05:38 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-12 12:51 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
@ 2010-02-12 13:10 ` Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-12 15:47 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
1 sibling, 1 reply; 51+ messages in thread
From: Heikki Linnakangas @ 2010-02-12 13:10 UTC (permalink / raw)
To: Simon Riggs <simon@2ndQuadrant.com>; +Cc: Fujii Masao <masao.fujii@gmail.com>; Dimitri Fontaine <dfontaine@hi-media.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
Simon Riggs wrote:
> In 8.4 it is pg_standby that was responsible for clearing down the
> archive, which is why I suggested using pg_standby for that again. I
> agree that will not work. The important thing is not pg_standby but that
> we have a valid mechanism for clearing down the archive.
Good point. When streaming from the master, the standby doesn't call
restore_command, and restore_command doesn't get a chance to clean up
old files.
> So I suggest that you have a new action that gets called after every
> checkpoint to clear down the archive. It will remove all files from the
> archive prior to %r. We can implement that as a sequence of unlink()s
> from within the server, or we can just call a script to do it. I prefer
> the latter approach. However we do it, we need something initiated by
> the server to maintain the archive and stop it from overflowing.
+1
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:41 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Dimitri Fontaine <dfontaine@hi-media.com>
2010-02-11 13:50 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:55 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 14:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-12 05:38 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-12 12:51 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-12 13:10 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
@ 2010-02-12 15:47 ` Fujii Masao <masao.fujii@gmail.com>
2010-02-12 16:10 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
0 siblings, 1 reply; 51+ messages in thread
From: Fujii Masao @ 2010-02-12 15:47 UTC (permalink / raw)
To: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; +Cc: Simon Riggs <simon@2ndquadrant.com>; Dimitri Fontaine <dfontaine@hi-media.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
On Fri, Feb 12, 2010 at 10:10 PM, Heikki Linnakangas
<heikki.linnakangas@enterprisedb.com> wrote:
>> So I suggest that you have a new action that gets called after every
>> checkpoint to clear down the archive. It will remove all files from the
>> archive prior to %r. We can implement that as a sequence of unlink()s
>> from within the server, or we can just call a script to do it. I prefer
>> the latter approach. However we do it, we need something initiated by
>> the server to maintain the archive and stop it from overflowing.
>
> +1
If we leave executing the remove_command to the bgwriter, the restartpoint
might not happen unfortunately for a long time. To prevent that situation, the
archiver should execute the command, I think. Thought?
Regards,
--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:41 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Dimitri Fontaine <dfontaine@hi-media.com>
2010-02-11 13:50 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:55 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 14:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-12 05:38 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-12 12:51 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-12 13:10 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-12 15:47 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
@ 2010-02-12 16:10 ` Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-15 06:29 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
0 siblings, 1 reply; 51+ messages in thread
From: Heikki Linnakangas @ 2010-02-12 16:10 UTC (permalink / raw)
To: Fujii Masao <masao.fujii@gmail.com>; +Cc: Simon Riggs <simon@2ndquadrant.com>; Dimitri Fontaine <dfontaine@hi-media.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
Fujii Masao wrote:
> On Fri, Feb 12, 2010 at 10:10 PM, Heikki Linnakangas
> <heikki.linnakangas@enterprisedb.com> wrote:
>>> So I suggest that you have a new action that gets called after every
>>> checkpoint to clear down the archive. It will remove all files from the
>>> archive prior to %r. We can implement that as a sequence of unlink()s
>>> from within the server, or we can just call a script to do it. I prefer
>>> the latter approach. However we do it, we need something initiated by
>>> the server to maintain the archive and stop it from overflowing.
>> +1
>
> If we leave executing the remove_command to the bgwriter, the restartpoint
> might not happen unfortunately for a long time.
Are you thinking of a scenario where remove_command gets stuck, and
prevents bgwriter from performing restartpoints while it's stuck? You
have trouble if restore_command gets stuck like that as well, so I think
we can require that the remove_command returns in a reasonable period of
time, ie. in a few minutes.
> To prevent that situation, the
> archiver should execute the command, I think. Thought?
The archiver isn't running in standby, so that's not going to work. And
it's not connected to shared memory either, so it doesn't know what the
latest restartpoint is.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:41 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Dimitri Fontaine <dfontaine@hi-media.com>
2010-02-11 13:50 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:55 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 14:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-12 05:38 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-12 12:51 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-12 13:10 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-12 15:47 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-12 16:10 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
@ 2010-02-15 06:29 ` Fujii Masao <masao.fujii@gmail.com>
0 siblings, 0 replies; 51+ messages in thread
From: Fujii Masao @ 2010-02-15 06:29 UTC (permalink / raw)
To: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; +Cc: Simon Riggs <simon@2ndquadrant.com>; Dimitri Fontaine <dfontaine@hi-media.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
On Sat, Feb 13, 2010 at 1:10 AM, Heikki Linnakangas
<heikki.linnakangas@enterprisedb.com> wrote:
> Are you thinking of a scenario where remove_command gets stuck, and
> prevents bgwriter from performing restartpoints while it's stuck?
Yes. If there is the archive in the remote server and the network outage
happens, remove_command might get stuck, I'm afraid.
> You
> have trouble if restore_command gets stuck like that as well, so I think
> we can require that the remove_command returns in a reasonable period of
> time, ie. in a few minutes.
Oh, you are right!
BTW, we need to note that remove_command approach would be useless if one
archive is shared by multiple standbys. One standby might wrongly remove
the archived WAL file that has been still required for another standby.
In this case, we need to have the job script that calculates the archived
WAL files that are required by no standbys, and removes them.
Regards,
--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:41 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Dimitri Fontaine <dfontaine@hi-media.com>
2010-02-11 13:50 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:55 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 14:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-12 05:38 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-12 12:51 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
@ 2010-02-12 14:34 ` Simon Riggs <simon@2ndQuadrant.com>
2010-02-12 17:25 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Dimitri Fontaine <dfontaine@hi-media.com>
1 sibling, 1 reply; 51+ messages in thread
From: Simon Riggs @ 2010-02-12 14:34 UTC (permalink / raw)
To: Fujii Masao <masao.fujii@gmail.com>; +Cc: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; Dimitri Fontaine <dfontaine@hi-media.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
On Fri, 2010-02-12 at 12:54 +0000, Simon Riggs wrote:
> So I suggest that you have a new action that gets called after every
> checkpoint to clear down the archive. It will remove all files from the
> archive prior to %r. We can implement that as a sequence of unlink()s
> from within the server, or we can just call a script to do it. I prefer
> the latter approach. However we do it, we need something initiated by
> the server to maintain the archive and stop it from overflowing.
Attached patch implements pg_standby for use as an
archive_cleanup_command, reusing existing code with new -a option.
e.g.
archive_cleanup_command = 'pg_standby -a -d /mnt/server/archiverdir %r'
Happy to add the archive_cleanup_command into main server as well, if
you like. Won't take long.
--
Simon Riggs www.2ndQuadrant.com
Attachments:
archive_cleanup_from_pg_standby.patch (text/x-patch, 7.2K) download
Index: contrib/pg_standby/pg_standby.c
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/contrib/pg_standby/pg_standby.c,v
retrieving revision 1.27
diff -c -r1.27 pg_standby.c
*** contrib/pg_standby/pg_standby.c 4 Nov 2009 12:51:30 -0000 1.27
--- contrib/pg_standby/pg_standby.c 12 Feb 2010 14:26:42 -0000
***************
*** 53,58 ****
--- 53,59 ----
int keepfiles = 0; /* number of WAL files to keep, 0 keep all */
int maxretries = 3; /* number of retries on restore command */
bool debug = false; /* are we debugging? */
+ bool cleanup_only = false; /* -a option new for 9.0 */
bool need_cleanup = false; /* do we need to remove files from
* archive? */
***************
*** 243,249 ****
/*
* Work out name of prior file from current filename
*/
! if (nextWALFileType == XLOG_DATA)
{
int rc;
DIR *xldir;
--- 244,250 ----
/*
* Work out name of prior file from current filename
*/
! if (cleanup_only || nextWALFileType == XLOG_DATA)
{
int rc;
DIR *xldir;
***************
*** 334,340 ****
* these files from archive. This shouldn't happen, but better safe
* than sorry.
*/
! if (strcmp(restartWALFileName, nextWALFileName) > 0)
return false;
strcpy(exclusiveCleanupFileName, restartWALFileName);
--- 335,341 ----
* these files from archive. This shouldn't happen, but better safe
* than sorry.
*/
! if (!cleanup_only && strcmp(restartWALFileName, nextWALFileName) > 0)
return false;
strcpy(exclusiveCleanupFileName, restartWALFileName);
***************
*** 512,518 ****
static void
usage(void)
{
! printf("%s allows PostgreSQL warm standby servers to be configured.\n\n", progname);
printf("Usage:\n");
printf(" %s [OPTION]... ARCHIVELOCATION NEXTWALFILE XLOGFILEPATH [RESTARTWALFILE]\n", progname);
printf("\n"
--- 513,519 ----
static void
usage(void)
{
! printf("%s allows PostgreSQL standby servers to be configured.\n\n", progname);
printf("Usage:\n");
printf(" %s [OPTION]... ARCHIVELOCATION NEXTWALFILE XLOGFILEPATH [RESTARTWALFILE]\n", progname);
printf("\n"
***************
*** 520,526 ****
--- 521,534 ----
" restore_command = 'pg_standby [OPTION]... ARCHIVELOCATION %%f %%p %%r'\n"
"e.g.\n"
" restore_command = 'pg_standby -l /mnt/server/archiverdir %%f %%p %%r'\n");
+ printf(" %s -a ARCHIVELOCATION RESTARTWALFILE\n", progname);
+ printf("\n"
+ "with main intended use as an archive_cleanup_command in the recovery.conf:\n"
+ " archive_cleanup_command = 'pg_standby -a ARCHIVELOCATION %%r'\n"
+ "e.g.\n"
+ " archive_cleanup_command = 'pg_standby -a -d /mnt/server/archiverdir %%r'\n");
printf("\nOptions:\n");
+ printf(" -a cleans archive only\n");
printf(" -c copies file from archive (default)\n");
printf(" -d generate lots of debugging output (testing only)\n");
printf(" -k NUMFILESTOKEEP if RESTARTWALFILE not used, removes files prior to limit\n"
***************
*** 595,604 ****
(void) signal(SIGQUIT, sigquit_handler);
#endif
! while ((c = getopt(argc, argv, "cdk:lr:s:t:w:")) != -1)
{
switch (c)
{
case 'c': /* Use copy */
restoreCommandType = RESTORE_COMMAND_COPY;
break;
--- 603,615 ----
(void) signal(SIGQUIT, sigquit_handler);
#endif
! while ((c = getopt(argc, argv, "acdk:lr:s:t:w:")) != -1)
{
switch (c)
{
+ case 'a': /* Cleanup only */
+ cleanup_only = true;
+ break;
case 'c': /* Use copy */
restoreCommandType = RESTORE_COMMAND_COPY;
break;
***************
*** 684,734 ****
exit(2);
}
! if (optind < argc)
! {
! nextWALFileName = argv[optind];
! optind++;
! }
! else
{
! fprintf(stderr, "%s: use %%f to specify nextWALFileName\n", progname);
! fprintf(stderr, "Try \"%s --help\" for more information.\n", progname);
! exit(2);
}
if (optind < argc)
{
! xlogFilePath = argv[optind];
optind++;
}
! else
{
! fprintf(stderr, "%s: use %%p to specify xlogFilePath\n", progname);
fprintf(stderr, "Try \"%s --help\" for more information.\n", progname);
exit(2);
}
- if (optind < argc)
- {
- restartWALFileName = argv[optind];
- optind++;
- }
-
CustomizableInitialize();
need_cleanup = SetWALFileNameForCleanup();
if (debug)
{
! fprintf(stderr, "Trigger file : %s\n", triggerPath ? triggerPath : "<not set>");
! fprintf(stderr, "Waiting for WAL file : %s\n", nextWALFileName);
! fprintf(stderr, "WAL file path : %s\n", WALFilePath);
! fprintf(stderr, "Restoring to : %s\n", xlogFilePath);
! fprintf(stderr, "Sleep interval : %d second%s\n",
! sleeptime, (sleeptime > 1 ? "s" : " "));
! fprintf(stderr, "Max wait interval : %d %s\n",
! maxwaittime, (maxwaittime > 0 ? "seconds" : "forever"));
! fprintf(stderr, "Command for restore : %s\n", restoreCommand);
fprintf(stderr, "Keep archive history : ");
if (need_cleanup)
fprintf(stderr, "%s and later\n", exclusiveCleanupFileName);
--- 695,758 ----
exit(2);
}
! if (!cleanup_only)
{
! if (optind < argc)
! {
! nextWALFileName = argv[optind];
! optind++;
! }
! else
! {
! fprintf(stderr, "%s: use %%f to specify nextWALFileName\n", progname);
! fprintf(stderr, "Try \"%s --help\" for more information.\n", progname);
! exit(2);
! }
!
! if (optind < argc)
! {
! xlogFilePath = argv[optind];
! optind++;
! }
! else
! {
! fprintf(stderr, "%s: use %%p to specify xlogFilePath\n", progname);
! fprintf(stderr, "Try \"%s --help\" for more information.\n", progname);
! exit(2);
! }
}
if (optind < argc)
{
! restartWALFileName = argv[optind];
optind++;
}
!
! if (cleanup_only && !restartWALFileName)
{
! fprintf(stderr, "%s: must specify restart WAL filename\n", progname);
fprintf(stderr, "Try \"%s --help\" for more information.\n", progname);
exit(2);
}
CustomizableInitialize();
need_cleanup = SetWALFileNameForCleanup();
if (debug)
{
! if (!cleanup_only)
! {
! fprintf(stderr, "Trigger file : %s\n", triggerPath ? triggerPath : "<not set>");
! fprintf(stderr, "Waiting for WAL file : %s\n", nextWALFileName);
! fprintf(stderr, "WAL file path : %s\n", WALFilePath);
! fprintf(stderr, "Restoring to : %s\n", xlogFilePath);
! fprintf(stderr, "Sleep interval : %d second%s\n",
! sleeptime, (sleeptime > 1 ? "s" : " "));
! fprintf(stderr, "Max wait interval : %d %s\n",
! maxwaittime, (maxwaittime > 0 ? "seconds" : "forever"));
! fprintf(stderr, "Command for restore : %s\n", restoreCommand);
! }
fprintf(stderr, "Keep archive history : ");
if (need_cleanup)
fprintf(stderr, "%s and later\n", exclusiveCleanupFileName);
***************
*** 738,743 ****
--- 762,777 ----
}
/*
+ * Cleanup archive immediately if that's all we need to do.
+ */
+ if (cleanup_only)
+ {
+ if (need_cleanup)
+ CustomizableCleanupPriorWALFiles();
+ exit(0);
+ }
+
+ /*
* Check for initial history file: always the first file to be requested
* It's OK if the file isn't there - all other files need to wait
*/
* Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:41 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Dimitri Fontaine <dfontaine@hi-media.com>
2010-02-11 13:50 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:55 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 14:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-12 05:38 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-12 12:51 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-12 14:34 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
@ 2010-02-12 17:25 ` Dimitri Fontaine <dfontaine@hi-media.com>
2010-02-12 17:47 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Greg Stark <stark@mit.edu>
0 siblings, 1 reply; 51+ messages in thread
From: Dimitri Fontaine @ 2010-02-12 17:25 UTC (permalink / raw)
To: Simon Riggs <simon@2ndQuadrant.com>; +Cc: Fujii Masao <masao.fujii@gmail.com>; Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>
Simon Riggs <simon@2ndQuadrant.com> writes:
> Attached patch implements pg_standby for use as an
> archive_cleanup_command, reusing existing code with new -a option.
>
> Happy to add the archive_cleanup_command into main server as well, if
> you like. Won't take long.
Would it be possible to have the server do the cleanup (and the cp for
that matter) on its own or use a script?
Either have archive_cleanup = on and either an commented out (empty)
archive_cleanup_command and the same for the restore_command, or a
special magic value, like 'postgresql' to activate the internal one.
This way we have both a zero config working default setup and the
flexibility to adapt to any archiving solution our user might already be
using.
A default archive_command would be nice too. Like you give it a
directory name or a rsync path and it does the basic right thing.
I'm not sure my detailed approach is the right one, but the birdview is
to have the simple case really simple to setup, and the complex cases
still possible. Default included archive, restore and cleanup commands
would be awesome.
Oh, and the basic simple case actually is with a remote standby. Without
NFS or the like, no shared mount point for archiving.
Regards,
--
dim
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:41 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Dimitri Fontaine <dfontaine@hi-media.com>
2010-02-11 13:50 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:55 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 14:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-12 05:38 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Fujii Masao <masao.fujii@gmail.com>
2010-02-12 12:51 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-12 14:34 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-12 17:25 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Dimitri Fontaine <dfontaine@hi-media.com>
@ 2010-02-12 17:47 ` Greg Stark <stark@mit.edu>
0 siblings, 0 replies; 51+ messages in thread
From: Greg Stark @ 2010-02-12 17:47 UTC (permalink / raw)
To: Dimitri Fontaine <dfontaine@hi-media.com>; +Cc: Simon Riggs <simon@2ndquadrant.com>; PostgreSQL-development <pgsql-hackers@postgresql.org>; Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; Fujii Masao <masao.fujii@gmail.com>
so I from by like having the server doing the cleanup because it down by
necessarily have the while picture. it down nt know of it is the only
replica reading these log files our if the site policy is to keep them for
disaster recovery purposes.
I like having this as an return val command though. Or alternately now that
we have read-only replicas you could implement this by polling the slaves
and asking them what log position they are up to.
greg
On 12 Feb 2010 17:25, "Dimitri Fontaine" <dfontaine@hi-media.com> wrote:
Simon Riggs <simon@2ndQuadrant.com> writes:
> Attached patch implements pg_standby for use as an
> archive_cleanup_command, reusing existing cod...
> Happy to add the archive_cleanup_command into main server as well, if
> you like. Won't take long....
Would it be possible to have the server do the cleanup (and the cp for
that matter) on its own or use a script?
Either have archive_cleanup = on and either an commented out (empty)
archive_cleanup_command and the same for the restore_command, or a
special magic value, like 'postgresql' to activate the internal one.
This way we have both a zero config working default setup and the
flexibility to adapt to any archiving solution our user might already be
using.
A default archive_command would be nice too. Like you give it a
directory name or a rsync path and it does the basic right thing.
I'm not sure my detailed approach is the right one, but the birdview is
to have the simple case really simple to setup, and the complex cases
still possible. Default included archive, restore and cleanup commands
would be awesome.
Oh, and the basic simple case actually is with a remote standby. Without
NFS or the like, no shared mount point for archiving.
Regards,
--
dim
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subs...
^ permalink raw reply [nested|flat] 51+ messages in thread
* Mention invalid null byte sequence
@ 2020-12-05 21:58 PG Doc comments form <noreply@postgresql.org>
2020-12-07 09:02 ` Re: Mention invalid null byte sequence Laurenz Albe <laurenz.albe@cybertec.at>
0 siblings, 1 reply; 51+ messages in thread
From: PG Doc comments form @ 2020-12-05 21:58 UTC (permalink / raw)
To: pgsql-docs@lists.postgresql.org; +Cc: bugs-postgresql@antipoul.fr
The following documentation comment has been logged on the website:
Page: https://www.postgresql.org/docs/13/datatype-character.html
Description:
I discovered accidentally that PostgreSQL doesn't accept null byte in text
type. It seems that Oracle does (see
https://www.postgresql.org/message-id/de752e01-f36c-821e-9181-cfba78c0fbc8%40propaas.com)
and SQLite does it too.
So it should written in the character type that null byte are not accepted,
it would make like easier to migrate to PostgreSQL :)
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Mention invalid null byte sequence
2020-12-05 21:58 Mention invalid null byte sequence PG Doc comments form <noreply@postgresql.org>
@ 2020-12-07 09:02 ` Laurenz Albe <laurenz.albe@cybertec.at>
2020-12-07 09:57 ` Re: Mention invalid null byte sequence Adrien CLERC <bugs-postgresql@antipoul.fr>
2020-12-07 20:27 ` Re: Mention invalid null byte sequence Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 2 replies; 51+ messages in thread
From: Laurenz Albe @ 2020-12-07 09:02 UTC (permalink / raw)
To: bugs-postgresql@antipoul.fr; pgsql-docs@lists.postgresql.org
On Sat, 2020-12-05 at 21:58 +0000, PG Doc comments form wrote:
> The following documentation comment has been logged on the website:
>
> Page: https://www.postgresql.org/docs/13/datatype-character.html
> Description:
>
> I discovered accidentally that PostgreSQL doesn't accept null byte in text
> type. It seems that Oracle does (see
> https://www.postgresql.org/message-id/de752e01-f36c-821e-9181-cfba78c0fbc8%40propaas.com)
> and SQLite does it too.
>
> So it should written in the character type that null byte are not accepted,
> it would make like easier to migrate to PostgreSQL :)
+1; how about the attached patch?
Yours,
Laurenz Albe
Attachments:
Document-that-NUL-characters-are-not-supported.patch (text/x-patch, 600B) download
diff --git a/doc/src/sgml/charset.sgml b/doc/src/sgml/charset.sgml
index cebc09ef91..7525b5c56e 100644
--- a/doc/src/sgml/charset.sgml
+++ b/doc/src/sgml/charset.sgml
@@ -1023,6 +1023,11 @@ CREATE COLLATION ignore_accents (provider = icu, locale = 'und-u-ks-level1-kc-tr
with most but not all server-side encodings.
</para>
+ <para>
+ Another restriction is that PostgreSQL does not support the NUL character
+ (ASCII 0) in character strings, regardless of the character set used.
+ </para>
+
<sect2 id="multibyte-charset-supported">
<title>Supported Character Sets</title>
* Re: Mention invalid null byte sequence
2020-12-05 21:58 Mention invalid null byte sequence PG Doc comments form <noreply@postgresql.org>
2020-12-07 09:02 ` Re: Mention invalid null byte sequence Laurenz Albe <laurenz.albe@cybertec.at>
@ 2020-12-07 09:57 ` Adrien CLERC <bugs-postgresql@antipoul.fr>
1 sibling, 0 replies; 51+ messages in thread
From: Adrien CLERC @ 2020-12-07 09:57 UTC (permalink / raw)
To: Laurenz Albe <laurenz.albe@cybertec.at>; pgsql-docs@lists.postgresql.org
Le 07/12/2020 à 10:02, Laurenz Albe a écrit :
> On Sat, 2020-12-05 at 21:58 +0000, PG Doc comments form wrote:
>> The following documentation comment has been logged on the website:
>>
>> Page: https://www.postgresql.org/docs/13/datatype-character.html
>> Description:
>>
>> I discovered accidentally that PostgreSQL doesn't accept null byte in text
>> type. It seems that Oracle does (see
>> https://www.postgresql.org/message-id/de752e01-f36c-821e-9181-cfba78c0fbc8%40propaas.com)
>> and SQLite does it too.
>>
>> So it should written in the character type that null byte are not accepted,
>> it would make like easier to migrate to PostgreSQL :)
> +1; how about the attached patch?
That would be a good start indeed.
I don't know the policy for documentation redundancy in PostgreSQL, but
it should be good to mention that also in
https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-STRINGS-ESCAPE
since the basic "SELECT E'la\x00la';" will fail while "SELECT
E'la\x01la';" will not.
And, as a lazy person, I also would like to see it in the general
datatype page, since it's a common behavior.
Anyway, merging the first patch will enable the search for "NUL
character" to return a result, and that will be definitively a nice
improvement!
Have a nice day!
Adrien
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Mention invalid null byte sequence
2020-12-05 21:58 Mention invalid null byte sequence PG Doc comments form <noreply@postgresql.org>
2020-12-07 09:02 ` Re: Mention invalid null byte sequence Laurenz Albe <laurenz.albe@cybertec.at>
@ 2020-12-07 20:27 ` Tom Lane <tgl@sss.pgh.pa.us>
2020-12-08 10:34 ` Re: Mention invalid null byte sequence Laurenz Albe <laurenz.albe@cybertec.at>
1 sibling, 1 reply; 51+ messages in thread
From: Tom Lane @ 2020-12-07 20:27 UTC (permalink / raw)
To: Laurenz Albe <laurenz.albe@cybertec.at>; +Cc: bugs-postgresql@antipoul.fr; pgsql-docs@lists.postgresql.org
Laurenz Albe <laurenz.albe@cybertec.at> writes:
> On Sat, 2020-12-05 at 21:58 +0000, PG Doc comments form wrote:
>> The following documentation comment has been logged on the website:
>> Page: https://www.postgresql.org/docs/13/datatype-character.html
>> Description:
>>
>> So it should written in the character type that null byte are not accepted,
>> it would make like easier to migrate to PostgreSQL :)
> +1; how about the attached patch?
I had thought that this was already documented, but after digging around
I can only find it mentioned in the contexts of saying that literal
strings and quoted identifiers can't contain \0. So yeah, we need to
improve that.
I agree with the submitter that the place one would expect to read about
this is in datatype-character.html. So I'd propose the attached.
Maybe there's reason to repeat the info in charset.sgml, but it seems
like more of a datatype limitation than a character set issue.
regards, tom lane
Attachments:
document-that-you-cant-store-NUL.patch (text/x-diff, 1.3K) download
diff --git a/doc/src/sgml/datatype.sgml b/doc/src/sgml/datatype.sgml
index 5c8a92e250..9eb19a1c61 100644
--- a/doc/src/sgml/datatype.sgml
+++ b/doc/src/sgml/datatype.sgml
@@ -1209,6 +1209,14 @@ SELECT '52093.89'::money::numeric::float8;
regular expressions.
</para>
+ <para>
+ The characters that can be stored in any of these data types are
+ determined by the database character set, which is selected when
+ the database is created. Regardless of the specific character set,
+ the character with code zero (sometimes called NUL) cannot be stored.
+ For more information refer to <xref linkend="multibyte"/>.
+ </para>
+
<para>
The storage requirement for a short string (up to 126 bytes) is 1 byte
plus the actual string, which includes the space padding in the case of
@@ -1246,10 +1254,7 @@ SELECT '52093.89'::money::numeric::float8;
<para>
Refer to <xref linkend="sql-syntax-strings"/> for information about
the syntax of string literals, and to <xref linkend="functions"/>
- for information about available operators and functions. The
- database character set determines the character set used to store
- textual values; for more information on character set support,
- refer to <xref linkend="multibyte"/>.
+ for information about available operators and functions.
</para>
<example>
* Re: Mention invalid null byte sequence
2020-12-05 21:58 Mention invalid null byte sequence PG Doc comments form <noreply@postgresql.org>
2020-12-07 09:02 ` Re: Mention invalid null byte sequence Laurenz Albe <laurenz.albe@cybertec.at>
2020-12-07 20:27 ` Re: Mention invalid null byte sequence Tom Lane <tgl@sss.pgh.pa.us>
@ 2020-12-08 10:34 ` Laurenz Albe <laurenz.albe@cybertec.at>
2020-12-08 17:07 ` Re: Mention invalid null byte sequence Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 51+ messages in thread
From: Laurenz Albe @ 2020-12-08 10:34 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: bugs-postgresql@antipoul.fr; pgsql-docs@lists.postgresql.org
On Mon, 2020-12-07 at 15:27 -0500, Tom Lane wrote:
> Laurenz Albe <laurenz.albe@cybertec.at> writes:
> > On Sat, 2020-12-05 at 21:58 +0000, PG Doc comments form wrote:
> > > The following documentation comment has been logged on the website:
> > > Page: https://www.postgresql.org/docs/13/datatype-character.html
> > > Description:
> > >
> > > So it should written in the character type that null byte are not accepted,
> > > it would make like easier to migrate to PostgreSQL :)
> >
> > +1; how about the attached patch?
>
> I had thought that this was already documented, but after digging around
> I can only find it mentioned in the contexts of saying that literal
> strings and quoted identifiers can't contain \0. So yeah, we need to
> improve that.
>
> I agree with the submitter that the place one would expect to read about
> this is in datatype-character.html. So I'd propose the attached.
> Maybe there's reason to repeat the info in charset.sgml, but it seems
> like more of a datatype limitation than a character set issue.
+1 on your patch.
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Mention invalid null byte sequence
2020-12-05 21:58 Mention invalid null byte sequence PG Doc comments form <noreply@postgresql.org>
2020-12-07 09:02 ` Re: Mention invalid null byte sequence Laurenz Albe <laurenz.albe@cybertec.at>
2020-12-07 20:27 ` Re: Mention invalid null byte sequence Tom Lane <tgl@sss.pgh.pa.us>
2020-12-08 10:34 ` Re: Mention invalid null byte sequence Laurenz Albe <laurenz.albe@cybertec.at>
@ 2020-12-08 17:07 ` Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 51+ messages in thread
From: Tom Lane @ 2020-12-08 17:07 UTC (permalink / raw)
To: Laurenz Albe <laurenz.albe@cybertec.at>; +Cc: bugs-postgresql@antipoul.fr; pgsql-docs@lists.postgresql.org
Laurenz Albe <laurenz.albe@cybertec.at> writes:
> On Mon, 2020-12-07 at 15:27 -0500, Tom Lane wrote:
>> I agree with the submitter that the place one would expect to read about
>> this is in datatype-character.html. So I'd propose the attached.
>> Maybe there's reason to repeat the info in charset.sgml, but it seems
>> like more of a datatype limitation than a character set issue.
> +1 on your patch.
Pushed, thanks for looking it over.
regards, tom lane
^ permalink raw reply [nested|flat] 51+ messages in thread
end of thread, other threads:[~2020-12-08 17:07 UTC | newest]
Thread overview: 51+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2010-01-27 15:27 pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki@postgresql.org>
2010-02-10 05:05 ` Fujii Masao <masao.fujii@gmail.com>
2010-02-10 07:32 ` Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-10 09:19 ` Fujii Masao <masao.fujii@gmail.com>
2010-02-10 13:45 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-10 16:53 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 10:37 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 12:27 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 12:44 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:06 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:28 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 13:46 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 14:01 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:42 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 17:04 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 17:31 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Aidan Van Dyk <aidan@highrise.ca>
2010-02-11 18:11 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 18:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Kevin Grittner <Kevin.Grittner@wicourts.gov>
2010-02-11 19:45 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Garick Hamlin <ghamlin@isc.upenn.edu>
2010-02-11 18:08 ` Tom Lane <tgl@sss.pgh.pa.us>
2010-02-11 18:14 ` Simon Riggs <simon@2ndQuadrant.com>
2010-02-12 07:37 ` Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 17:29 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 18:11 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-18 11:37 ` Fujii Masao <masao.fujii@gmail.com>
2010-02-11 13:41 ` Dimitri Fontaine <dfontaine@hi-media.com>
2010-02-11 13:50 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 13:55 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:17 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 14:22 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 14:38 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Simon Riggs <simon@2ndQuadrant.com>
2010-02-11 15:21 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-11 15:29 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Euler Taveira de Oliveira <euler@timbira.com>
2010-02-11 14:55 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Greg Smith <greg@2ndquadrant.com>
2010-02-12 05:38 ` Fujii Masao <masao.fujii@gmail.com>
2010-02-12 12:51 ` Simon Riggs <simon@2ndQuadrant.com>
2010-02-12 13:10 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-12 15:47 ` Fujii Masao <masao.fujii@gmail.com>
2010-02-12 16:10 ` Re: Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2010-02-15 06:29 ` Fujii Masao <masao.fujii@gmail.com>
2010-02-12 14:34 ` Simon Riggs <simon@2ndQuadrant.com>
2010-02-12 17:25 ` Dimitri Fontaine <dfontaine@hi-media.com>
2010-02-12 17:47 ` Greg Stark <stark@mit.edu>
2020-12-05 21:58 Mention invalid null byte sequence PG Doc comments form <noreply@postgresql.org>
2020-12-07 09:02 ` Re: Mention invalid null byte sequence Laurenz Albe <laurenz.albe@cybertec.at>
2020-12-07 09:57 ` Re: Mention invalid null byte sequence Adrien CLERC <bugs-postgresql@antipoul.fr>
2020-12-07 20:27 ` Re: Mention invalid null byte sequence Tom Lane <tgl@sss.pgh.pa.us>
2020-12-08 10:34 ` Re: Mention invalid null byte sequence Laurenz Albe <laurenz.albe@cybertec.at>
2020-12-08 17:07 ` Re: Mention invalid null byte sequence Tom Lane <tgl@sss.pgh.pa.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox