pg.ddx.io pgsql-hackers-pitr@postgresql.org mailing list archive
help / color / mirror / Atom feedArchival API
5+ messages / 4 participants
[nested] [flat]
* Archival API
@ 2004-02-17 22:41 Simon Riggs <simon@2ndquadrant.com>
0 siblings, 0 replies; 5+ messages in thread
From: Simon Riggs @ 2004-02-17 22:41 UTC (permalink / raw)
To: 'Tom Lane' <tgl@sss.pgh.pa.us>; +Cc: pgsql-hackers-pitr
>Tom Lane [mailto:tgl@sss.pgh.pa.us]
> > Simon Riggs wrote
> > - Write application to archive WAL files to tape, disk, or network
> > Probably need to do first part, but I'm arguing not to do the copy
to
> > tape..
>
> I'd like to somehow see this handled by a user-supplied program or
> script. What we mainly need is to define a good API that lets the
> archiver program understand which WAL segment files to archive when.
>
> > B - Backing up WAL log files
> > -Ordinarily, when old log segment files are no longer needed, they
are
> > recycled (renamed to become the next segments in the numbered
sequence).
> > This means that the data within them must be copied from there to
> > another location
> > AFTER postgres has closed that file
> > BEFORE it is renamed and recycled
>
> My inclination would be to change the backend code so that as soon as
a
> WAL segment is completed, it is flagged as being ready to dump to tape
> (or wherever). Possibly the easiest way to do this is to rename the
> segment file somehow, perhaps "nnn" becomes "nnn.full". Then, after
the
> archiver process has properly dumped the file, reflag it as being
dumped
> (perhaps rename to "nnn.done"). Obviously there are any number of
ways
> we could do this flagging, and depending on an OS rename facility
might
> not be the best.
Yes, that would be the correct time to begin archive.
The way the code is currently written there is a slot in MoveOfflineLogs
which looks to see if XLOG_archive_dir is set before entering a section
which is empty apart from a message. That routine doesn't get called
until we're about to recycle the files, which means we've lost our
window of opportunity to archive them. Making the number of files larger
doesn't effect that being called last.... I'm going to ignore that
"hint" and any patch will include deletion of that code to avoid later
confusion.
The log switch and close occurs during XLogWrite, when it is established
that there is no more room in the current log file for the current
segment.
The file-flagging mechanism only allows a single archiver program to
operate, so I'll structure it as a new function XLogArchiveNotify() so
we can add in extra stuff later to improve/change things. That way we
have a home for the API.
> A segment then can be recycled when it is both (a) older than the
latest
> checkpoint and (b) flagged as dumped. Note that this approach allows
> dumping of a file to start before the first time at which it could be
> recycled. In the event of a crash and restart, WAL replay has to be
> able to find the flagged segments, so the flagging mechanism can't be
> one that would make this impossible.
The number of WAL logs is effectively tunable anyway because it depends
on the number of checkpoint segments, so we can increase that if there
are issues with archival speed v txn rate.
The rename is always safe because the log file names never wrap.
However, I'm loathe to touch the files, in case something crashes
somewhere and we are left with recovery failing because of an
unlocatable file. (To paraphrase one of the existing code comments, only
the truly paranoid survive). A similar way is to have a "buddy" file,
which indicates whether it is full and ready for archival. i.e. when we
close file "nnn" we also write an nearly/empty file called "nnn.full".
That file can then be deleted later BY THE archiver once archival has
finished, allowing it to be recycled by InstallXLogFileSegment(). (Would
require at least 6 more file descriptors, but I'm not sure if that's an
issue).
InstallXLogFileSegment() can check for XLogArchiveBusy() to see whether
it is allowed to reuse or allocate a new one. In initial implementation
this would just test to see whether "nnn.full" still exists. This will
allow a range of behaviour to be catered for, such as long waits while
manual tape mounts are requested by the archiver etc..
So in summary, the API is:
Archiver initialises and waits on notify
Postgresql initialises
...then
Postgresql fills log, switches and close it, then calls
XLogArchiveNotify()
Archiver moves log somewhere safe, then sets state such that...
...sometime later
Postgresql checks XLogArchiveBusy() to see if its safe to recycle file
and discovers the state set by
API is completely unintrusive on current tried and tested operation, and
leaves the archiver(s) free to act as they choose, outside of the
address space of PostgreSQL. That way we don't have to update regession
tests with some destructive non-manual crash tests to show that works.
Clearly, we wouldn't want WAL logs to hang around too long, so we need
an initiation method for the archival process. Otherwise, we'll be
writing "nnn.full" notifications yet without anybody ever deleting them.
Either this could be set at startup with an archive_log_mode parameter
(OK, the names been used before, but if the cap fits, wear it) or
setting a maximum limit to number of archive logs and a few other ideas,
none of which I like.
Hmmmm...any listeners got any ideas here? How do we want this to work?
Anybody want to write a more complex archiver process to act as more
than just a test harness?
Best regards,
Simon Riggs
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Archival API
@ 2004-02-26 10:54 raymond.siebert@mobilcom.de
2004-02-26 21:08 ` Re: Archival API Nicolai Tufar <ntufar@pisem.net>
0 siblings, 1 reply; 5+ messages in thread
From: raymond.siebert@mobilcom.de @ 2004-02-26 10:54 UTC (permalink / raw)
To: simon@2ndquadrant.com; +Cc: pgsql-hackers-pitr
Simon,
>So in summary, the API is:
>
>Archiver initialises and waits on notify
>Postgresql initialises
>...then
>Postgresql fills log, switches and close it, then calls
>XLogArchiveNotify()
>Archiver moves log somewhere safe, then sets state such that...
>...sometime later
>Postgresql checks XLogArchiveBusy() to see if its safe to recycle file
>and discovers the state set by
>
>API is completely unintrusive on current tried and tested operation, and
>leaves the archiver(s) free to act as they choose, outside of the
>address space of PostgreSQL. That way we don't have to update regession
>tests with some destructive non-manual crash tests to show that works.
>
>Clearly, we wouldn't want WAL logs to hang around too long, so we need
>an initiation method for the archival process. Otherwise, we'll be
>writing "nnn.full" notifications yet without anybody ever deleting them.
>Either this could be set at startup with an archive_log_mode parameter
>(OK, the names been used before, but if the cap fits, wear it) or
>setting a maximum limit to number of archive logs and a few other ideas,
>none of which I like.
Another idea is an event handling system in database system.
Sending defined signals to the 'outer world' through an API to a user
supplied program.
This user supplied program can later on handle other events in addition to
backup of WAL logs.
A fixed classification of events and a defined framework in user supplied
program is the way to success.
First event can be 'WAL log archived' starting backup of an archived WAL
log.
After archiver process has created an archived WAL log an event like 'WAL
log archived' calls the user supplied program interpreting the event.
The archiver process must have a naming convention for archived WAL logs -
especially for recovery.
Archiving WAL logs may require an archive destination - also needed for
recovery.
Essential input to user supplied program is the full-pathed archived WAL
log.
A user supplied backup program backups full-pathed archived WAL log to an
whatever Backup Destination (even /dev/null for suicide characters).
After successfull backup the archived WAL log needs to be removed.
All is handled internally in the user supplied program.
Advantage of this idea is that backup of archived WAL logs is
automatically enabled without the manual work of an DB Admin person.
I have to admit I'm not a programer. How much effort to whole source code
my idea will cost I can't calculate.
Any ideas ?
Raymond
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Archival API
2004-02-26 10:54 Re: Archival API raymond.siebert@mobilcom.de
@ 2004-02-26 21:08 ` Nicolai Tufar <ntufar@pisem.net>
0 siblings, 0 replies; 5+ messages in thread
From: Nicolai Tufar @ 2004-02-26 21:08 UTC (permalink / raw)
To: raymond.siebert@mobilcom.de; simon@2ndquadrant.com; +Cc: pgsql-hackers-pitr
-----Original Message-----
From: pgsql-hackers-pitr-owner@postgresql.org
[mailto:pgsql-hackers-pitr-owner@postgresql.org] On Behalf Of
raymond.siebert@mobilcom.de
> Another idea is an event handling system in database system.
> Sending defined signals to the 'outer world' through an API to a user
> supplied program.
> This user supplied program can later on handle other events in
addition to > backup of WAL logs.
> A fixed classification of events and a defined framework in user
supplied > program is the way to success.
>
> [ ... ]
>
This is the method used in DB2. It is called "user exit". You can
specify
a program or a compiled C code to be run before, after or instead of
a specific event.
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Archival API
@ 2004-02-26 21:22 Simon Riggs <simon@2ndquadrant.com>
2004-03-02 14:20 ` Re: Archival API Raymond Siebert <raymond.siebert@mobilcom.de>
0 siblings, 1 reply; 5+ messages in thread
From: Simon Riggs @ 2004-02-26 21:22 UTC (permalink / raw)
To: raymond.siebert@mobilcom.de; +Cc: pgsql-hackers-pitr
>>Raymond Siebert wrote
>>Simon Riggs wrote:
>>So in summary, the API is:
>>Archiver initialises and waits on notify
>>Postgresql initialises
>>...then
>>Postgresql fills log, switches and close it, then calls
>>XLogArchiveNotify()
>>Archiver moves log somewhere safe, then sets state such that...
>>...sometime later
>>Postgresql checks XLogArchiveBusy() to see if its safe to recycle file
>>and discovers the state set by
>>
>>API is completely unintrusive on current tried and tested operation,
and
>>leaves the archiver(s) free to act as they choose, outside of the
>>address space of PostgreSQL. That way we don't have to update
regession
>>tests with some destructive non-manual crash tests to show that works.
>>
>>Clearly, we wouldn't want WAL logs to hang around too long, so we need
>>an initiation method for the archival process. Otherwise, we'll be
>>writing "nnn.full" notifications yet without anybody ever deleting
them.
>>Either this could be set at startup with an archive_log_mode parameter
>>(OK, the names been used before, but if the cap fits, wear it) or
>>setting a maximum limit to number of archive logs and a few other
ideas,
>>none of which I like.
>
>Another idea...
Thank you very much for your input, it is gratefully received.
>A fixed classification of events and a defined framework in user
supplied >program is the way to success.
Yes, I agree. I'm working on a better description of the overall design,
to allow everybody to think it through with me. Should be next week
now...
>Another idea is an event handling system in database system.
>Sending defined signals to the 'outer world' through an API to a user
>supplied program.
>This user supplied program can later on handle other events in addition
to >backup of WAL logs.
That was an approach I considered. One aspect of PITR is that testing is
a very important part of the overall solution. With that in mind, a very
tightly defined, very specific API seems the best way to go. A more
general facility might get mixed up and end up breaking the PITR logic.
External notification is possible (I think, never tried) by using user
defined C language functions. It should be fairly straightforward to use
those to get/put on message queues, send POSIX signals etc.
>The archiver process must have a naming convention for archived WAL
logs - >especially for recovery.
Yes. I would suggest the database name as a prefix to the WAL log file
name. The log file name is a very long string of digits that "never
repeats" according to the manual. I'll cross that bridge when it
happens. That should guarantee uniqueness **within** an organisation,
(assuming there is a unique database naming policy is place) which is
hopefully the limit of archived log files anyway.
>Archiving WAL logs may require an archive destination - also needed for
>recovery.
That would be the user-supplied program's role to define that. The
implementation will clearly require a reference implementation of just
such a program, to allow testing.
>Essential input to user supplied program is the full-pathed archived
WAL >log.
>A user supplied backup program backups full-pathed archived WAL log to
an >whatever Backup Destination (even /dev/null for suicide characters).
>After successfull backup the archived WAL log needs to be removed.
>All is handled internally in the user supplied program.
That's what the basic design proposed, yes.
>Advantage of this idea is that backup of archived WAL logs is
automatically >enabled without the manual work of an DB Admin person.
I think it is important that archival can only be activated by explicit
command from the database administrator. If not, then it could be some
kind of security hole to start sniffing a log without permission. Your
input is good though, because I haven't given security much thought yet
and you have spurred me to consider this further.
I hope I can call on you soon to review the design docs and later to
assist with testing?
More input welcome!
Best Regards, Simon Riggs
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Archival API
2004-02-26 21:22 Re: Archival API Simon Riggs <simon@2ndquadrant.com>
@ 2004-03-02 14:20 ` Raymond Siebert <raymond.siebert@mobilcom.de>
0 siblings, 0 replies; 5+ messages in thread
From: Raymond Siebert @ 2004-03-02 14:20 UTC (permalink / raw)
To: simon@2ndquadrant.com; +Cc: pgsql-hackers-pitr
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
<br>
<br>
Simon Riggs wrote:<br>
<br>
<blockquote cite="mid002601c3fcae$b08c0020$0200000a@LaptopDellXP"
type="cite">
<blockquote type="cite">
<pre wrap="">A fixed classification of events and a defined framework in user
</pre>
</blockquote>
<pre wrap=""><!---->supplied >program is the way to success.
Yes, I agree. I'm working on a better description of the overall design,
to allow everybody to think it through with me. Should be next week
now...
</pre>
</blockquote>
Let's talk in terms of severity from unimportant to critical. That will
be basis for defining 'event handler' in user supplied script.<br>
<blockquote cite="mid002601c3fcae$b08c0020$0200000a@LaptopDellXP"
type="cite">
<pre wrap=""></pre>
<blockquote type="cite">
<pre wrap="">Another idea is an event handling system in database system.
Sending defined signals to the 'outer world' through an API to a user
supplied program.
This user supplied program can later on handle other events in addition
</pre>
</blockquote>
<pre wrap=""><!---->to >backup of WAL logs.
That was an approach I considered. One aspect of PITR is that testing is
a very important part of the overall solution. With that in mind, a very
tightly defined, very specific API seems the best way to go. A more
general facility might get mixed up and end up breaking the PITR logic.
</pre>
</blockquote>
I don't think a general facility would mix up or break something. It's
a question of design.<br>
See Event "WAL log archived" as a very little subset of a large number
of defined events. <br>
<br>
Former Informix Developers (now IBM) have successfully created an
powerful database event handling system which does Backup of
transaction log areas
- as mimimum service.<br>
Fully used it informs about non-critical to critical events via
generated emails - especially for on call people.<br>
<blockquote type="cite">
<pre wrap="">
</pre>
</blockquote>
<blockquote cite="mid002601c3fcae$b08c0020$0200000a@LaptopDellXP"
type="cite">
<pre wrap="">External notification is possible (I think, never tried) by using user
defined C language functions. It should be fairly straightforward to use
those to get/put on message queues, send POSIX signals etc.
</pre>
</blockquote>
As long as the 'user supplied program' is configured in postgresql.conf
the use of system() call can be used to pass defined Arguments.<br>
The design of internal message queues etc. is somehow independent from
that. <br>
It has two aspects: detecting,interpreting and logging internal events
and on the other hand contacting the 'outer world' initiating a defined
action.<br>
<blockquote cite="mid002601c3fcae$b08c0020$0200000a@LaptopDellXP"
type="cite">
<blockquote type="cite">
<pre wrap="">The archiver process must have a naming convention for archived WAL
</pre>
</blockquote>
<pre wrap=""><!---->logs - >especially for recovery.
Yes. I would suggest the database name as a prefix to the WAL log file
name. The log file name is a very long string of digits that "never
repeats" according to the manual. I'll cross that bridge when it
happens. That should guarantee uniqueness **within** an organisation,
(assuming there is a unique database naming policy is place) which is
hopefully the limit of archived log files anyway.
</pre>
</blockquote>
Is the database oid unique even if multiple clusters are active on one
host ?<br>
For identification purposes the most unique naming convention is needed.<br>
<br>
<blockquote cite="mid002601c3fcae$b08c0020$0200000a@LaptopDellXP"
type="cite">
<blockquote type="cite">
<pre wrap="">Archiving WAL logs may require an archive destination - also needed for
recovery.
</pre>
</blockquote>
<pre wrap=""><!---->
That would be the user-supplied program's role to define that. The
implementation will clearly require a reference implementation of just
such a program, to allow testing.
</pre>
</blockquote>
What format is intended for the the user supplied program ? <br>
Can it be anything from interpetors language like SHELL, PERL up to
dynamically linked 32/64bit ELF format ?<br>
It's vital to have a template which can be adapted to your own needs.<br>
<blockquote cite="mid002601c3fcae$b08c0020$0200000a@LaptopDellXP"
type="cite">
<pre wrap=""></pre>
<blockquote type="cite">
<pre wrap="">Advantage of this idea is that backup of archived WAL logs is
</pre>
</blockquote>
<pre wrap=""><!---->automatically >enabled without the manual work of an DB Admin person.
I think it is important that archival can only be activated by explicit
command from the database administrator. If not, then it could be some
kind of security hole to start sniffing a log without permission. Your
input is good though, because I haven't given security much thought yet
and you have spurred me to consider this further.
</pre>
</blockquote>
<br>
In terms of pratically providing 7x24h 365 days a year business that
will bring ugly problems - an DB admin can't always be around.<br>
For my person dealing in this business a Software with sane
intelligence for standard events performing standard actions is a real
benefit.<br>
<br>
DB admin has the choice.<br>
Either an dummy user supplied program is 'contacted' - simply doing
nothing and not telling anyone. This leaves archival initiation to DB
admin.<br>
Or a fully functional user supplied program with archive function to
whatever destination. This leaves archival to database system.<br>
<br>
For security reasons it is recommended to avoid public used areas like
/tmp at any cost.<br>
<blockquote cite="mid002601c3fcae$b08c0020$0200000a@LaptopDellXP"
type="cite">
<pre wrap="">I hope I can call on you soon to review the design docs and later to
assist with testing?
</pre>
</blockquote>
Just contact me.<br>
<br>
Best Regards Raymond <br>
<br>
</body>
</html>
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2004-03-02 14:20 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2004-02-17 22:41 Archival API Simon Riggs <simon@2ndquadrant.com>
2004-02-26 10:54 Re: Archival API raymond.siebert@mobilcom.de
2004-02-26 21:08 ` Nicolai Tufar <ntufar@pisem.net>
2004-02-26 21:22 Re: Archival API Simon Riggs <simon@2ndquadrant.com>
2004-03-02 14:20 ` Raymond Siebert <raymond.siebert@mobilcom.de>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox