X-Original-To: pgsql-patches-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id CA425D1B1C7 for ; Wed, 14 Jul 2004 19:57:18 -0300 (ADT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 68543-06 for ; Wed, 14 Jul 2004 22:57:11 +0000 (GMT) Received: from cmailm3.svr.pol.co.uk (cmailm3.svr.pol.co.uk [195.92.193.19]) by svr1.postgresql.org (Postfix) with ESMTP id 361BCD1B1AB for ; Wed, 14 Jul 2004 19:57:06 -0300 (ADT) Received: from modem-276.monkey.dialup.pol.co.uk ([217.135.209.20] helo=[192.168.0.100]) by cmailm3.svr.pol.co.uk with esmtp (Exim 4.14) id 1Bksg3-0001F6-TJ; Wed, 14 Jul 2004 23:56:40 +0100 Subject: Re: [HACKERS] Point in Time Recovery From: Simon Riggs To: markw@osdl.org Cc: pgman@candle.pha.pa.us, kn@mgnet.de, tgl@sss.pgh.pa.us, pgsql-patches@postgresql.org In-Reply-To: <1089833620.17493.4624.camel@stromboli> References: <200407141555.i6EFtnk22984@mail.osdl.org> <1089833620.17493.4624.camel@stromboli> Content-Type: multipart/mixed; boundary="=-swM3+xD6gWhLkjHvf/5u" Organization: 2nd Quadrant Message-Id: <1089845797.17493.4987.camel@stromboli> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.5 (1.4.5-7) Date: Wed, 14 Jul 2004 23:56:39 +0100 X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.1 tagged_above=0.0 required=5.0 tests=HTML_MESSAGE, RCVD_IN_SORBS X-Spam-Level: X-Archive-Number: 200407/226 X-Sequence-Number: 11896 --=-swM3+xD6gWhLkjHvf/5u Content-Type: text/plain Content-Transfer-Encoding: 7bit On Wed, 2004-07-14 at 20:33, Simon Riggs wrote: > On Wed, 2004-07-14 at 16:55, markw@osdl.org wrote: > > On 14 Jul, Simon Riggs wrote: > > > PITR Patch v5_1 just posted has Point in Time Recovery working.... > > > > > > Still some rough edges....but we really need some testers now to give > > > this a try and let me know what you think. > > > > > > Klaus Naumann and Mark Wong are the only [non-committers] to have tried > > > to run the code (and let me know about it), so please have a look at > > > [PATCHES] and try it out. > > > > > > I just tried applying the v5_1 patch against the cvs tip today and got a > > couple of rejections. I'll copy the patch output here. Let me know if > > you want to see the reject files or anything else: > > > > I'm on it. Sorry 'bout that all - midnight fingers. Latest version, pitr_v5_2.patch... - Updated to cvs tip - Additional tip changes located and patched - Full re-test of both recover to point in time and recover to xid - 2 additional bug fixes - corrected recovery.conf sample - Patch test - Patch manually inspected (pgarch.c, pgarch.h and README identical to previous post) Go for it... Best regards, Simon --=-swM3+xD6gWhLkjHvf/5u Content-Disposition: attachment; filename=pitr_v5_2.patch Content-Type: text/x-patch; name=pitr_v5_2.patch; charset= Content-Transfer-Encoding: 7bit Index: backend/access/nbtree/nbtsort.c =================================================================== RCS file: /projects/cvsroot/pgsql-server/src/backend/access/nbtree/nbtsort.c,v retrieving revision 1.83 diff -c -r1.83 nbtsort.c *** backend/access/nbtree/nbtsort.c 11 Jul 2004 18:01:45 -0000 1.83 --- backend/access/nbtree/nbtsort.c 14 Jul 2004 22:41:19 -0000 *************** *** 67,72 **** --- 67,73 ---- #include "miscadmin.h" #include "storage/smgr.h" #include "utils/tuplesort.h" + #include "access/xlog.h" /* *************** *** 220,235 **** wstate.index = btspool->index; /* ! * We need to log index creation in WAL iff WAL archiving is enabled * AND it's not a temp index. - * - * XXX when WAL archiving is actually supported, this test will likely - * need to change; and the hardwired extern is cruddy anyway ... */ { ! extern char XLOG_archive_dir[]; ! ! wstate.btws_use_wal = XLOG_archive_dir[0] && !wstate.index->rd_istemp; } /* reserve the metapage */ wstate.btws_pages_alloced = BTREE_METAPAGE + 1; --- 221,231 ---- wstate.index = btspool->index; /* ! * We need to log index creation in WAL if WAL archiving is enabled * AND it's not a temp index. */ { ! wstate.btws_use_wal = XLogArchiveMode && !wstate.index->rd_istemp; } /* reserve the metapage */ wstate.btws_pages_alloced = BTREE_METAPAGE + 1; Index: backend/access/transam/xlog.c =================================================================== RCS file: /projects/cvsroot/pgsql-server/src/backend/access/transam/xlog.c,v retrieving revision 1.147 diff -c -r1.147 xlog.c *** backend/access/transam/xlog.c 1 Jul 2004 00:49:50 -0000 1.147 --- backend/access/transam/xlog.c 14 Jul 2004 22:41:24 -0000 *************** *** 36,47 **** #include "storage/proc.h" #include "storage/sinval.h" #include "storage/spin.h" #include "utils/builtins.h" #include "utils/guc.h" #include "utils/relcache.h" #include "miscadmin.h" - /* * This chunk of hackery attempts to determine which file sync methods * are available on the current platform, and to choose an appropriate --- 36,47 ---- #include "storage/proc.h" #include "storage/sinval.h" #include "storage/spin.h" + #include "storage/pmsignal.h" #include "utils/builtins.h" #include "utils/guc.h" #include "utils/relcache.h" #include "miscadmin.h" /* * This chunk of hackery attempts to determine which file sync methods * are available on the current platform, and to choose an appropriate *************** *** 85,96 **** /* User-settable parameters */ int CheckPointSegments = 3; int XLOGbuffers = 8; char *XLOG_sync_method = NULL; const char XLOG_sync_method_default[] = DEFAULT_SYNC_METHOD_STR; - char XLOG_archive_dir[MAXPGPATH]; /* null string means - * delete 'em */ #ifdef WAL_DEBUG bool XLOG_DEBUG = false; --- 85,98 ---- /* User-settable parameters */ + bool XLogArchiveMode = false; + bool XLogArchiveDEBUG = false; + char *XLogArchiveDest; + char *XLogArchiveProgram; int CheckPointSegments = 3; int XLOGbuffers = 8; char *XLOG_sync_method = NULL; const char XLOG_sync_method_default[] = DEFAULT_SYNC_METHOD_STR; #ifdef WAL_DEBUG bool XLOG_DEBUG = false; *************** *** 127,132 **** --- 129,157 ---- /* Are we doing recovery by reading XLOG? */ bool InRecovery = false; + bool InArchiveRecovery = false; + bool restoreFromArchive = false; + bool InRecoveryCleanup = false; + + FILE *archiveStatusLogFD; + + static void readRecoveryCommandFile(void); + + static char recoveryCommandFile[MAXPGPATH]; + + static char recoveryRestoreProgram[MAXPGPATH]; + static char recoveryTargetTimeStr[MAXPGPATH]; + static char recoveryTargetXidStr[MAXPGPATH]; + + static bool recoveryTarget = true; + static bool recoveryTargetExact = false; + static bool recoveryTargetInclusive = true; + + static bool recoveryDebugLog = false; + static bool recoveryUseLocalTime = true; + + static TransactionId recoveryTargetXid; + static time_t recoveryTargetTime; /* * MyLastRecPtr points to the start of the last XLOG record inserted by the *************** *** 393,398 **** --- 418,424 ---- /* File path names */ static char XLogDir[MAXPGPATH]; + static char archiveStatusDir[MAXPGPATH]; static char ControlFilePath[MAXPGPATH]; /* *************** *** 434,439 **** --- 460,468 ---- static bool InRedo = false; + static void XLogArchiveNotify(uint32 log, uint32 seg); + static bool XLogArchiveDone(char xlog[MAXPGPATH]); + static void XLogArchiveCleanup(char xlog[32]); static bool AdvanceXLInsertBuffer(void); static bool WasteXLInsertBuffer(void); *************** *** 444,449 **** --- 473,479 ---- bool find_free, int max_advance, bool use_lock); static int XLogFileOpen(uint32 log, uint32 seg, bool econt); + static void RestoreRecoveryXlog(char *path, uint32 log, uint32 seg); static void PreallocXlogFiles(XLogRecPtr endptr); static void MoveOfflineLogs(uint32 log, uint32 seg, XLogRecPtr endptr); static XLogRecord *ReadRecord(XLogRecPtr *RecPtr, int emode, char *buffer); *************** *** 455,464 **** static void ReadControlFile(void); static char *str_time(time_t tnow); static void issue_xlog_fsync(void); - #ifdef WAL_DEBUG static void xlog_outrec(char *buf, XLogRecord *record); - #endif - /* * Insert an XLOG record having the specified RMID and info bytes, --- 485,491 ---- *************** *** 912,917 **** --- 939,1076 ---- } /* + * XLogArchiveNotify + * + * Writes an archive notification file to the archiveStatusDir + * + * The name of the notification file is the message that will be picked up + * by the archiver, e.g. we write archiveStatusDir/00000001000000C6.ready + * and the archiver then knows to archive XLogDir/00000001000000C6, + * then when complete, rename it to archiveStatusDir/00000001000000C6.done + * + * Called only when in XLogArchiveMode by one backend process + * or by Archive Restore in various situations + */ + static void + XLogArchiveNotify(uint32 log, uint32 seg) + { + char archiveStatusLog[32]; + char archiveStatusLogPath[MAXPGPATH]; + + /* insert an otherwise empty file called .ready */ + sprintf(archiveStatusLog, "%08X%08X.ready", log, seg); + snprintf(archiveStatusLogPath, MAXPGPATH, "%s/%s", archiveStatusDir, archiveStatusLog); + archiveStatusLogFD = NULL; + archiveStatusLogFD = AllocateFile(archiveStatusLogPath, "w"); + if (archiveStatusLogFD == NULL) { + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not write archive_status file \"%s\": %m", + archiveStatusLogPath))); + return; + } + FreeFile(archiveStatusLogFD); + + /* the existence of this file is the message to the archiver to identify + * which files require archiving + * + * if this file is written OK, we then signal the ARCHIVER to do its thang + */ + + if (XLogArchiveDEBUG) { + sprintf(archiveStatusLog, "%08X%08X", log, seg); + ereport(LOG, + (errmsg("transaction log file \"%s\" is now ready to be archived", archiveStatusLog))); + } + + /* + * don't send a signal if we know that the archiver isn't there (yet) + * - the archiver will see the archive_status file as soon as it starts + */ + if (!InArchiveRecovery) + SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER); + + return; + } + + /* + * XLogArchiveDone + * + * Searches for an archive notification file in archiveStatusDir + * + * Reads archiveStatusDir looking for a specific filename. If that filename ends with .done + * then we know that the filename refers to an xlog in XLogDir that is safe to + * recycle. If the filename ends .ready then thats OK, else we have an error. + * + * Called only when in XLogArchiveMode by bgwriter (when performing checkpoint) + * + * XXX code is rehacked from an earlier version, so needs streamlining + */ + static bool + XLogArchiveDone(char xlog[32]) + { + char archiveStatusLogPath[MAXPGPATH]; + struct stat tmpStatBuf; + + /* If .done exists then return true + */ + snprintf(archiveStatusLogPath, MAXPGPATH, "%s/%s.done", archiveStatusDir, xlog); + if (stat(archiveStatusLogPath, &tmpStatBuf) == 0) { + return true; + } + else + { + /* + * else if .ready exists then return false and issue WARNING + * ...this indicates archiver is either not working at all or + * if it is, then its just way too slow or incorrectly configured + */ + snprintf(archiveStatusLogPath, MAXPGPATH, "%s/%s.ready", archiveStatusDir, xlog); + if (stat(archiveStatusLogPath, &tmpStatBuf) == 0) { + ereport(WARNING, + (errcode(ERRCODE_WARNING), + errmsg("archiving of log file \"%s\" has not yet started", + archiveStatusLogPath), + errhint("you should check whether your archive_program" + "\"%s\" is still functioning correctly", XLogArchiveProgram))); + return false; + } + else + { + /* else issue a WARNING.... a notification file should exist, except when you + * have just enabled archive_mode + * assume that error is simply a missing file... + */ + ereport(WARNING, + (errcode_for_file_access(), + errmsg("archive_status file \"%s\" is missing: %m", + archiveStatusLogPath), + errhint("if you have just enabled archive_mode you should:" + "take a full physical backup, then remove \"%s\"",xlog))); + return false; + } + } + } + + /* + * XLogArchiveCleanup + * + * Cleanup an archive notification file for a particular xlog in XLogDir + * + * Called only when in XLogArchiveMode by bgwriter (when performing checkpoint) + * + */ + static void + XLogArchiveCleanup(char xlog[32]) + { + char archiveStatusLogPath[MAXPGPATH]; + + snprintf(archiveStatusLogPath, MAXPGPATH, "%s/%s.done", archiveStatusDir, xlog); + unlink(archiveStatusLogPath); + + } + + /* * Advance the Insert state to the next buffer page, writing out the next * buffer if it still contains unwritten data. * *************** *** 1260,1265 **** --- 1419,1430 ---- { issue_xlog_fsync(); LogwrtResult.Flush = LogwrtResult.Write; /* end of current page */ + + /* + * write archive_status file to notify that xlog is ready to archive + */ + if (XLogArchiveMode) + XLogArchiveNotify(openLogId, openLogSeg); } if (ispartialpage) *************** *** 1619,1625 **** bool use_lock) { char path[MAXPGPATH]; ! struct stat stat_buf; XLogFileName(path, log, seg); --- 1784,1790 ---- bool use_lock) { char path[MAXPGPATH]; ! struct stat tmpStatBuf; XLogFileName(path, log, seg); *************** *** 1637,1643 **** else { /* Find a free slot to put it in */ ! while (stat(path, &stat_buf) == 0) { if (--max_advance < 0) { --- 1802,1808 ---- else { /* Find a free slot to put it in */ ! while (stat(path, &tmpStatBuf) == 0) { if (--max_advance < 0) { *************** *** 1686,1692 **** char path[MAXPGPATH]; int fd; ! XLogFileName(path, log, seg); fd = BasicOpenFile(path, O_RDWR | PG_BINARY | XLOG_SYNC_BIT, S_IRUSR | S_IWUSR); --- 1851,1859 ---- char path[MAXPGPATH]; int fd; ! XLogFileName(path, log, seg); ! if (restoreFromArchive) ! RestoreRecoveryXlog(path, log, seg); fd = BasicOpenFile(path, O_RDWR | PG_BINARY | XLOG_SYNC_BIT, S_IRUSR | S_IWUSR); *************** *** 1706,1715 **** --- 1873,2104 ---- path, log, seg))); } + /* + * if we switched back to local xlogs after having been + * restoring from archive, we need to make sure that the + * local files dont get removed by end-of-recovery checkpoint + * in case we need to re-run the recovery + * + * we want to copy these away as soon as possible, so set + * the archive_status flag to .ready for them + * in case admin isn't cautious enough to have done this anyway + */ + if (InArchiveRecovery && !restoreFromArchive) { + snprintf(path, MAXPGPATH, "%s/%08X%08X.ready", archiveStatusDir, log, seg); + archiveStatusLogFD = NULL; + archiveStatusLogFD = AllocateFile(path, "w"); + if (archiveStatusLogFD == NULL) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not write archive_status file \"%s\" ", + path))); + FreeFile(archiveStatusLogFD); + } + return (fd); } /* + * Get next logfile segment to allow recovery + * + */ + static void + RestoreRecoveryXlog(char *path, uint32 log, uint32 seg) + { + char tmpXlog[32]; + char restoreXlog[32]; + char tmppath[MAXPGPATH]; + char xlogRestoreCmd[MAXPGPATH]; + char recoveryXlog[MAXPGPATH]; + char lastrecoXlog[MAXPGPATH]; + int rc; + struct stat tmpStatBuf; + uint32 prevlog, prevseg; + + /* + * If a RecoveryFile exists, then we know we are in media recovery + * in which case we choose to recover files from archive, even + * though a file of that name may already exist in XLogDir + * + * By doing this, we do not effect crash recovery code path + * when we are not in archive_mode + * + * We take the archived file because, at the point we took backup, + * the current xlog will most probably be only partially full, + * so we MUST refer to the full version of this file and + * NOT the version of the file that exists with the backup. + * + * We could try to optimize this slightly by checking the local + * copy lastchange timestamp against the archived copy, + * but we have no API to do this, nor can we guarantee that the + * lastchange timestamp was preserved correctly when we copied + * to archive. Our aim is robustness, so we elect not to do this. + * + * Try to copy full xlog from archive to pg_xlog, if it is available + * If that succeeds, we pass the RecoveryXlog filepath back for opening + * If that fails, then we try to read a local file if one exists. + * This allows us to cater for situations where the current xlog + * is still available locally and hasn't yet made it to archive. + * This could happen if: + * - we decide to recover database to undo user data changes + * - we have XLogDir on a different disk and the main DataDir drive + * fails, leaving us with just the XLogDir + * + * Notice that we don't actually overwrite any files when we copy back + * from archive because the recoveryRestoreProgram may inadvertently + * restore inappropriate xlogs, or they may be corrupt, so we may + * have to fallback to the segments remaining in current XLogDir later. + * The copy-from-archive xlog is always the same, ensuring that we + * don't run out of disk space on long recoveries. + * + * simon@2ndquadrant.com + */ + + snprintf(recoveryXlog, MAXPGPATH, "%s/RECOVERYXLOG", XLogDir); + snprintf(lastrecoXlog, MAXPGPATH, "%s/LASTRECOXLOG", XLogDir); + + if (stat(recoveryXlog, &tmpStatBuf) == 0) { + /* + * save a copy of the last xlog, before we try to restore + * if the restore fails, we will need it to become current xlog + */ + rc = rename(recoveryXlog, lastrecoXlog); + if (rc !=0) + ereport(WARNING, + (errcode_for_file_access(), + errmsg("rename failed \"%s\" \"%s\":%m", + recoveryXlog, lastrecoXlog))); + /* + * if it fails, ignore it - we'll create one soon... + */ + } + + /* + * Copy xlog from archive_dest to XLogDir + */ + sprintf(restoreXlog, "%08X%08X", log, seg); + snprintf(xlogRestoreCmd, MAXPGPATH, recoveryRestoreProgram, + XLogArchiveDest, restoreXlog, recoveryXlog); + if (XLogArchiveDEBUG) + ereport(LOG, + (errmsg("restore_program=\"%s\"", + xlogRestoreCmd))); + + rc = system(xlogRestoreCmd); + if (rc!=0) { + /* + * remember, we rollforward UNTIL the restore fails + * so failure here is just part of the process... + * that makes it difficult to determine whether the restore + * failed because there isn't an archive to restore, or + * because the administrator has specified the restore + * program incorrectly... + * we could try to restore the testfile that the archiver writes + * when it starts up, but the absence of that file isn't + * very reliable evidence that the restore itself is broken, + * so just trust that the administrator has it correctly, + * XXX enhance that later + */ + ereport(LOG, + (errmsg("could not restore \"%s\" from archive", + restoreXlog))); + /* + * if an archived file is not available, there might just be + * a partially full version of this file still in XLogDir + * so return this as the filename to open. + * In many recovery scenarios we expect this to fail also... + * ...note that once we return to local mode, we could + * find more than one xlog there that were not in the archive, + * if for example, the archiver had stopped working prior + * to the event which led to recovery + */ + snprintf(recoveryXlog, MAXPGPATH, "%s/%s", XLogDir, restoreXlog); + restoreFromArchive = false; + if (stat(recoveryXlog, &tmpStatBuf) == 0) { + ereport(LOG, + (errmsg("recovery will continue using local copy of \"%s\"", + restoreXlog))); + } + /* + * if this file isn't available, then we need to setup the previous + * restored xlog to be the last and current xlog, if it exists + * remember: we've been restoring from recoverXlog, which isn't + * named the same as the normal xlog chain... + * also remember to output a corresponding archive_status of .done + */ + else if ((stat(lastrecoXlog, &tmpStatBuf) == 0) && log==0 && seg > 0) { + prevlog = log; + prevseg = seg; + PrevLogSeg(prevlog, prevseg); + XLogFileName(tmppath, prevlog, prevseg); + if (XLogArchiveDEBUG) + ereport(LOG, + (errmsg("moving last restored xlog to \"%s\"", tmppath))); + rc = rename(lastrecoXlog, tmppath); + if (rc!=0) { + ereport(LOG, + (errcode_for_file_access(), + errmsg("rename failed \"%s\" \"%s\":%m", + lastrecoXlog, tmppath))); + ereport(PANIC, + (errcode_for_file_access(), + errmsg("could not open file \"%s\" (log file %u, segment %u): %m", + tmpXlog, log, seg))); + } + + /* + * write out an archive_status file for previous xlog + * to allow xlog to be recycled when recovered database + * is all up and working again + * ...looks wrong, but checkpointer is smart enough + * not to archive the current xlog! + */ + sprintf(tmpXlog, "%08X%08X", prevlog, prevseg); + snprintf(tmppath, MAXPGPATH, "%s/%s.done", archiveStatusDir, tmpXlog); + archiveStatusLogFD = NULL; + archiveStatusLogFD = AllocateFile(tmppath, "w"); + if (archiveStatusLogFD == NULL) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not write archive_status file \"%s\" ", + tmppath))); + FreeFile(archiveStatusLogFD); + } + /* + * there is NO else here...we just return the filename + * knowing that it isn't there...which then throws the usual error, + * will end with a clear message as to why...but not a problem + */ + } + else { + /* restore success */ + /* + * if backup restored an xlog, yet we didnt use the local copy + * because we used the xlog version of that name from the + * archive instead, we need to write out an archive_status for + * it to show it can be recycled later + */ + XLogFileName(tmppath, log, seg); + if (stat(tmppath, &tmpStatBuf) == 0) { + sprintf(tmpXlog, "%08X%08X", log, seg); + snprintf(tmppath, MAXPGPATH, "%s/%s.done", archiveStatusDir, tmpXlog); + archiveStatusLogFD = NULL; + archiveStatusLogFD = AllocateFile(tmppath, "w"); + if (archiveStatusLogFD == NULL) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not write archive_status file \"%s\": %m", + tmppath))); + FreeFile(archiveStatusLogFD); + } + ereport(LOG, + (errmsg("restored log file \"%s\" from archive", restoreXlog))); + } + strcpy(path, recoveryXlog); + return; + } + + /* * Preallocate log files beyond the specified log endpoint, according to * the XLOGfile user parameter. */ *************** *** 1747,1752 **** --- 2136,2142 ---- struct dirent *xlde; char lastoff[32]; char path[MAXPGPATH]; + bool recycle=false; XLByteToPrevSeg(endptr, endlogId, endlogSeg); *************** *** 1762,1786 **** errno = 0; while ((xlde = readdir(xldir)) != NULL) { if (strlen(xlde->d_name) == 16 && strspn(xlde->d_name, "0123456789ABCDEF") == 16 && strcmp(xlde->d_name, lastoff) <= 0) { snprintf(path, MAXPGPATH, "%s/%s", XLogDir, xlde->d_name); ! if (XLOG_archive_dir[0]) ! { ! ereport(LOG, ! (errmsg("archiving transaction log file \"%s\"", ! xlde->d_name))); ! elog(WARNING, "archiving log files is not implemented"); ! } ! else { /* * Before deleting the file, see if it can be recycled as * a future log segment. We allow recycling segments up ! * to XLOGfileslop segments beyond the current XLOG ! * location. */ if (InstallXLogFileSegment(endlogId, endlogSeg, path, true, XLOGfileslop, --- 2152,2194 ---- errno = 0; while ((xlde = readdir(xldir)) != NULL) { + /* if correct length and alphanumeric makeup of file looks correct + * use the alphanumeric sorting property of the filenames to decide + * which ones are earlier than the lastoff transaction log + * ...maybe should read lastwrite datetime of lastoff, then check that + * only files last written earlier than this are removed/recycled + */ if (strlen(xlde->d_name) == 16 && strspn(xlde->d_name, "0123456789ABCDEF") == 16 && strcmp(xlde->d_name, lastoff) <= 0) { snprintf(path, MAXPGPATH, "%s/%s", XLogDir, xlde->d_name); ! if (XLogArchiveMode) { ! if (InRecoveryCleanup) ! /* ! * this allows recycling of transaction logs ! * during the shutdown checkpoint at end of recovery ! * - we may have restored logs that were not used ! * in the recovery sequence, and so will not have ! * had an archive_status file written for them. ! * - end-of-recovery doesn't clean up ALL xlogs, ! * which is why we also write archive_status files ! * as well as doing this ! */ ! recycle=true; ! else ! recycle=XLogArchiveDone(xlde->d_name); ! } ! else ! recycle=false; ! ! if ( recycle ) { /* * Before deleting the file, see if it can be recycled as * a future log segment. We allow recycling segments up ! * until there are XLOGfileslop segments beyond the ! * current XLOG location, otherwise they are removed. */ if (InstallXLogFileSegment(endlogId, endlogSeg, path, true, XLOGfileslop, *************** *** 1794,1803 **** { /* No need for any more future segments... */ ereport(LOG, ! (errmsg("removing transaction log file \"%s\"", xlde->d_name))); unlink(path); } } } errno = 0; --- 2202,2212 ---- { /* No need for any more future segments... */ ereport(LOG, ! (errmsg("too many transaction log files, removing \"%s\"", xlde->d_name))); unlink(path); } + XLogArchiveCleanup(xlde->d_name); } } errno = 0; *************** *** 2255,2260 **** --- 2664,2670 ---- { /* Init XLOG file paths */ snprintf(XLogDir, MAXPGPATH, "%s/pg_xlog", DataDir); + snprintf(archiveStatusDir, MAXPGPATH, "%s/archive_status", XLogDir); snprintf(ControlFilePath, MAXPGPATH, "%s/global/pg_control", DataDir); } *************** *** 2772,2777 **** --- 3182,3345 ---- } /* + * read in restore command from recovery.conf + * + * XXX longer term intention is to expand this to + * cater for additional parameters and controls + * possibly using a bison grammar to control it + */ + static void + readRecoveryCommandFile(void) + { + FILE *fd; + char *tok1 = NULL; + char *tok2 = NULL; + char cmdline[MAXPGPATH]; + bool syntaxError = false; + struct tm tm; + + fd = AllocateFile(recoveryCommandFile, "r"); + if (fd == NULL) { + ereport(FATAL, + (errcode_for_file_access(), + errmsg("could not open recovery command file \"%s\"",recoveryCommandFile))); + return; + } + + ereport(LOG, + (errmsg("recovery command file found..."))); + + /* + * expecting |restore_program = Qcommand stringQ| + * e.g. |restore_program = 'cp %s/%s %s'| + * where | denote the beginning and end of the string + */ + while (!syntaxError && fgets(cmdline, MAXPGPATH, fd) != NULL) { + + /* ignore # comments ... */ + if(strncmp(cmdline, "#", 1) == 0) continue; + + tok1 = strtok(cmdline, "'"); + tok2 = strtok(NULL, "'"); + + if (tok1 != NULL && tok2 != NULL) { + tok1 = strtok(cmdline, " ="); + if (tok1 == NULL) { + syntaxError = true; + continue; + } + if (strcmp(tok1,"restore_program") == 0) { + strcpy(recoveryRestoreProgram, tok2); + ereport(LOG, + (errmsg("restore_program = %s", recoveryRestoreProgram))); + recoveryTarget = false; + } + else if (strcmp(tok1,"recovery_target_xid") == 0) { + /* + * if recovery_target_xid specified, then this overrides + * recovery_target_time, if it is also set + */ + strcpy(recoveryTargetXidStr, tok2); + recoveryTargetXid = (TransactionId) strtoul(tok2, NULL, 10); + if (errno == EINVAL || errno == ERANGE ) { + ereport(FATAL, + (errmsg("recovery_target_xid is not a valid xid \"%s\"", + recoveryTargetTimeStr))); + } + ereport(LOG,(errmsg("recovery_target_xid = %s", recoveryTargetXidStr))); + recoveryTarget = true; + recoveryTargetExact = true; + } + else if (strcmp(tok1,"recovery_target_time") == 0) { + if (recoveryTargetExact) + continue; + strcpy(recoveryTargetTimeStr, tok2); + recoveryTarget = true; + recoveryTargetExact = false; + /* + * convert the time string given + * by the user to the time_t format. + * The input string format has to be like + * '2004-07-13-16:09:07'. + */ + if (strptime(recoveryTargetTimeStr, + "%Y-%m-%d-%H:%M:%S", &tm) == NULL ) { + ereport(FATAL, + (errmsg("recovery_target_time has a format error \"%s\"", + recoveryTargetTimeStr), + errhint("correct format is YYYY-MM-DD-hh:mm:ss"))); + } + recoveryTargetTime = mktime(&tm); + if (recoveryTargetTime < 0) { + ereport(FATAL, + (errmsg("recovery_target_time is an invalid time \"%s\"", + recoveryTargetTimeStr), + errhint("correct format is YYYY-MM-DD-hh:mm:ss"))); + } + ereport(LOG,(errmsg("recovery_target_time = %s", + recoveryTargetTimeStr))); + } + else if (strcmp(tok1,"recovery_target_inclusive") == 0) { + /* + * does nothing if a recovery_target is not also set + */ + if (strcmp(tok2, "true") == 0) + recoveryTargetInclusive = true; + else + recoveryTargetInclusive = false; + if (recoveryTargetInclusive) + strcpy(tok2, "true"); + else + strcpy(tok2, "false"); + ereport(LOG,(errmsg("recovery_target_inclusive = %s", tok2))); + } + else if (strcmp(tok1,"recovery_debug_log") == 0) { + if (strcmp(tok2, "true") == 0) + recoveryDebugLog = true; + else + recoveryDebugLog = false; + if (recoveryDebugLog) + strcpy(tok2, "true"); + else + strcpy(tok2, "false"); + ereport(LOG,(errmsg("recovery_debug_log = %s", tok2))); + } + else if (strcmp(tok1,"use_local_time") == 0) { + if (strcmp(tok2, "true") == 0) + recoveryUseLocalTime = true; + else + recoveryUseLocalTime = false; + if (recoveryUseLocalTime) + strcpy(tok2, "true"); + else + strcpy(tok2, "false"); + ereport(LOG,(errmsg("use_local_time = %s", tok2))); + } + else { + ereport(LOG, + (errmsg("unrecognized parameter \"%s\"", tok1))); + syntaxError = true; + } + } + else { + ereport(LOG, + (errmsg("incorrectly formatted parameter \"%s\"", cmdline), + errhint("parameter = 'value'"))); + syntaxError = true; + } + } + + FreeFile(fd); + + if (syntaxError) { + ereport(FATAL, + (errmsg("fatal error in \"%s\"", recoveryCommandFile))); + } + + return; + } + + /* * This must be called ONCE during postmaster or standalone-backend startup */ void *************** *** 2787,2792 **** --- 3355,3361 ---- XLogRecord *record; char *buffer; uint32 freespace; + struct stat tmpStatBuf; /* Use malloc() to ensure record buffer is MAXALIGNED */ buffer = (char *) malloc(_INTL_MAXLOGRECSZ); *************** *** 2833,2838 **** --- 3402,3437 ---- pg_usleep(60000000L); #endif + /* + * Check now for recovery.conf + * + * if this file exists, it demonstrates the intention of the administrator + * to recover this database using archived xlogs + * + * we do this now because the first xlog is about to be opened for the + * first time. We've read the checkpoint pointer from the control file + * and we are about to use that to open the xlog it points to, and + * will begin rollforward recovery from that point + */ + snprintf(recoveryCommandFile, MAXPGPATH, "%s/recovery.conf", DataDir); + if (stat(recoveryCommandFile, &tmpStatBuf) == 0) { + + readRecoveryCommandFile(); + /* + * clearly indicate our state + */ + InArchiveRecovery = true; + /* + * set initial state for checking transaction logs + * this may change if the archive runs dry while still InArchiveRecovery + */ + restoreFromArchive = true; + + ereport(LOG, + (errmsg("starting archive recovery"))); + + } + /* * Get the last valid checkpoint record. If the latest one according * to pg_control is broken, try the next-to-last one. *************** *** 2863,2874 **** LastRec = RecPtr = checkPointLoc; memcpy(&checkPoint, XLogRecGetData(record), sizeof(CheckPoint)); wasShutdown = (record->xl_info == XLOG_CHECKPOINT_SHUTDOWN); ! ereport(LOG, (errmsg("redo record is at %X/%X; undo record is at %X/%X; shutdown %s", checkPoint.redo.xlogid, checkPoint.redo.xrecoff, checkPoint.undo.xlogid, checkPoint.undo.xrecoff, ! wasShutdown ? "TRUE" : "FALSE"))); ereport(LOG, (errmsg("next transaction ID: %u; next OID: %u", checkPoint.nextXid, checkPoint.nextOid))); --- 3462,3481 ---- LastRec = RecPtr = checkPointLoc; memcpy(&checkPoint, XLogRecGetData(record), sizeof(CheckPoint)); wasShutdown = (record->xl_info == XLOG_CHECKPOINT_SHUTDOWN); ! /* ! * we report the state of the control_file, not the checkpoint, why? ! * wasShutdown refers to whether the last checkpoint was a ! * shutdown checkpoint, NOT whether the database was shutdown ! * correctly according to control file. This distinction is only ! * important InArchiveRecovery, since otherwise we could ! * report that the database was shutdown, when the control file disagrees ! */ ereport(LOG, (errmsg("redo record is at %X/%X; undo record is at %X/%X; shutdown %s", checkPoint.redo.xlogid, checkPoint.redo.xrecoff, checkPoint.undo.xlogid, checkPoint.undo.xrecoff, ! (ControlFile->state == DB_SHUTDOWNED) ? "TRUE" : "FALSE"))); ! ereport(LOG, (errmsg("next transaction ID: %u; next OID: %u", checkPoint.nextXid, checkPoint.nextOid))); *************** *** 2915,2921 **** /* REDO */ if (InRecovery) { ! int rmid; ereport(LOG, (errmsg("database system was not properly shut down; " --- 3522,3537 ---- /* REDO */ if (InRecovery) { ! int rmid; ! char recoveryLogPath[MAXPGPATH]; ! bool recoveryContinue = true; ! bool recoveryApply = true; ! int recoveryLogFD = -1; ! char *recbuf = NULL; ! uint8 record_info; ! xl_xact_commit *recordXactCommitData; ! xl_xact_abort *recordXactAbortData; ! time_t recordXtime; ereport(LOG, (errmsg("database system was not properly shut down; " *************** *** 2948,2953 **** --- 3564,3589 ---- ereport(LOG, (errmsg("redo starts at %X/%X", ReadRecPtr.xlogid, ReadRecPtr.xrecoff))); + #ifdef WAL_DEBUG + if (XLOG_DEBUG) + recoveryDebugLog = true; + #endif + if (XLogArchiveDEBUG) + recoveryDebugLog = true; + + if (recoveryDebugLog) { + recbuf = (char *) malloc(BLCKSZ); + snprintf(recoveryLogPath, MAXPGPATH, "%s/recovery.log", DataDir); + unlink(recoveryLogPath); + recoveryLogFD = BasicOpenFile(recoveryLogPath, O_RDWR | O_CREAT | O_EXCL, + S_IRUSR | S_IWUSR); + if (recoveryLogFD < 0) + recoveryDebugLog = false; + } + + /* + * main redo apply loop + */ do { /* nextXid must be beyond record's xid */ *************** *** 2958,2990 **** TransactionIdAdvance(ShmemVariableCache->nextXid); } ! #ifdef WAL_DEBUG ! if (XLOG_DEBUG) { ! char buf[8192]; ! ! sprintf(buf, "REDO @ %X/%X; LSN %X/%X: ", ReadRecPtr.xlogid, ReadRecPtr.xrecoff, EndRecPtr.xlogid, EndRecPtr.xrecoff); ! xlog_outrec(buf, record); ! strcat(buf, " - "); ! RmgrTable[record->xl_rmid].rm_desc(buf, record->xl_info, XLogRecGetData(record)); ! elog(LOG, "%s", buf); } - #endif if (record->xl_info & XLR_BKP_BLOCK_MASK) RestoreBkpBlocks(record, EndRecPtr); - RmgrTable[record->xl_rmid].rm_redo(EndRecPtr, record); - record = ReadRecord(NULL, LOG, buffer); - } while (record != NULL); ereport(LOG, (errmsg("redo done at %X/%X", ReadRecPtr.xlogid, ReadRecPtr.xrecoff))); LastRec = ReadRecPtr; InRedo = false; } else ereport(LOG, --- 3594,3729 ---- TransactionIdAdvance(ShmemVariableCache->nextXid); } ! if (recoveryDebugLog && record->xl_rmid == 1) { ! sprintf(recbuf, "\nREDO %X/%X;LSN %X/%X: ", ReadRecPtr.xlogid, ReadRecPtr.xrecoff, EndRecPtr.xlogid, EndRecPtr.xrecoff); ! sprintf(recbuf + strlen(recbuf), "rm %u;info %u ", ! record->xl_rmid, record->xl_info); ! xlog_outrec(recbuf, record); ! strcat(recbuf, "--"); ! RmgrTable[record->xl_rmid].rm_desc(recbuf, record->xl_info, XLogRecGetData(record)); ! write(recoveryLogFD, recbuf, strlen(recbuf)); } if (record->xl_info & XLR_BKP_BLOCK_MASK) RestoreBkpBlocks(record, EndRecPtr); + /* + * Have we reached our recovery target? + * check whether this is the last record we should apply - + * we can only stop at transaction end record types, + * COMMIT or ABORT, which are only ever associated + * with the XLOG resource manager (1) + */ + if (recoveryTarget == true && + record->xl_rmid == 1) { + + record_info = record->xl_info & ~XLR_INFO_MASK; + + if (record_info == XLOG_XACT_COMMIT || + record_info == XLOG_XACT_ABORT) { + + if (recoveryTargetExact) { + /* + * there can be only one transaction end record + * with this exact transactionid + * + * when testing for an xid, we MUST test for + * equality only, since transactions are numbered + * in the order they start, not the order they + * complete. A higher numbered xid will complete + * before you about 50% of the time... + */ + if (record->xl_xid == recoveryTargetXid) { + recoveryContinue = false; + recoveryApply = recoveryTargetInclusive; + } + if (!recoveryContinue) { + ereport(LOG, + (errmsg("rollforward stopping at commit/abort:" + " transactionId %u", + record->xl_xid))); + } + } + else { + /* + * there can be many transactions that + * share the same commit time, so + * we stop after the last one, if we are + * inclusive, or stop at the first one + * if we are exclusive + * + * timestamp is first part of record body + */ + if (record_info == XLOG_XACT_COMMIT) { + recordXactCommitData = + (xl_xact_commit *) XLogRecGetData(record); + recordXtime = recordXactCommitData->xtime; + } + else { + recordXactAbortData = + (xl_xact_abort *) XLogRecGetData(record); + recordXtime = recordXactAbortData->xtime; + } + + if (recoveryTargetInclusive) { + if ((unsigned int)(recordXtime) > + (unsigned int)(recoveryTargetTime)) + recoveryContinue = false; + /* + * because we have already just included the + * records that match (==), and we now want + * to stop just beyond + */ + recoveryApply = false; + } + else { + if ((unsigned int)(recordXtime) >= + (unsigned int)(recoveryTargetTime)) + recoveryContinue = false; + recoveryApply = false; + } + if (!recoveryContinue) { + struct tm *tm = localtime(&recordXtime); + ereport(LOG, + (errmsg("rollforward stopping at commit/abort:" + " time %04u-%02u-%02u %02u:%02u:%02u", + tm->tm_year + 1900, tm->tm_mon + 1, tm->tm_mday, + tm->tm_hour, tm->tm_min, tm->tm_sec))); + } + } + } + } + + if (recoveryContinue || recoveryApply) { + RmgrTable[record->xl_rmid].rm_redo(EndRecPtr, record); + record = ReadRecord(NULL, LOG, buffer); + } + + } while (record != NULL && recoveryContinue); + /* + * end of main redo apply + */ + + if (recoveryLogFD >= 0) { + close(recoveryLogFD); + if (recoveryDebugLog) + free(recbuf); + } ereport(LOG, (errmsg("redo done at %X/%X", ReadRecPtr.xlogid, ReadRecPtr.xrecoff))); LastRec = ReadRecPtr; InRedo = false; + if (InArchiveRecovery) { + InRecoveryCleanup = true; + InArchiveRecovery = false; + /* LastRec.xlogid = LastRec.xlogid + TIMELINE_INCREMENT */ + } + Assert(restoreFromArchive == false); } else ereport(LOG, *************** *** 3106,3118 **** * Note that we write a shutdown checkpoint. This is correct since * the records following it will use SUI one more than what is * shown in the checkpoint's ThisStartUpID. ! * * In case we had to use the secondary checkpoint, make sure that it * will still be shown as the secondary checkpoint after this * CreateCheckPoint operation; we don't want the broken primary ! * checkpoint to become prevCheckPoint... */ ! ControlFile->checkPoint = checkPointLoc; CreateCheckPoint(true, true); /* --- 3845,3864 ---- * Note that we write a shutdown checkpoint. This is correct since * the records following it will use SUI one more than what is * shown in the checkpoint's ThisStartUpID. ! */ ! ControlFile->checkPoint = checkPointLoc; ! /* * In case we had to use the secondary checkpoint, make sure that it * will still be shown as the secondary checkpoint after this * CreateCheckPoint operation; we don't want the broken primary ! * checkpoint to become prevCheckPoint...unless we have just ! * performed an archive recovery, in which case reset the ! * secondary checkpoint as well...to avoid going back over ! * old ground */ ! if (InRecoveryCleanup) ! ControlFile->prevCheckPoint = checkPointLoc; ! CreateCheckPoint(true, true); /* *************** *** 3149,3154 **** --- 3895,3906 ---- * Okay, we're officially UP. */ InRecovery = false; + if (InRecoveryCleanup) { + unlink(recoveryCommandFile); + InRecoveryCleanup = false; + ereport(LOG, + (errmsg("archive recovery complete"))); + } ControlFile->state = DB_IN_PRODUCTION; ControlFile->time = time(NULL); *************** *** 3706,3719 **** strcat(buf, "UNKNOWN"); } - #ifdef WAL_DEBUG static void xlog_outrec(char *buf, XLogRecord *record) { int bkpb; int i; ! sprintf(buf + strlen(buf), "prev %X/%X; xprev %X/%X; xid %u", record->xl_prev.xlogid, record->xl_prev.xrecoff, record->xl_xact_prev.xlogid, record->xl_xact_prev.xrecoff, record->xl_xid); --- 4458,4470 ---- strcat(buf, "UNKNOWN"); } static void xlog_outrec(char *buf, XLogRecord *record) { int bkpb; int i; ! sprintf(buf + strlen(buf), "prv %X/%X;xprv %X/%X;xid %u", record->xl_prev.xlogid, record->xl_prev.xrecoff, record->xl_xact_prev.xlogid, record->xl_xact_prev.xrecoff, record->xl_xid); *************** *** 3731,3738 **** sprintf(buf + strlen(buf), ": %s", RmgrTable[record->xl_rmid].rm_name); } - #endif /* WAL_DEBUG */ - /* * GUC support --- 4482,4487 ---- Index: backend/commands/tablecmds.c =================================================================== RCS file: /projects/cvsroot/pgsql-server/src/backend/commands/tablecmds.c,v retrieving revision 1.119 diff -c -r1.119 tablecmds.c *** backend/commands/tablecmds.c 11 Jul 2004 23:13:53 -0000 1.119 --- backend/commands/tablecmds.c 14 Jul 2004 22:41:46 -0000 *************** *** 5398,5413 **** FlushRelationBuffers(rel, 0); /* ! * We need to log the copied data in WAL iff WAL archiving is enabled * AND it's not a temp rel. - * - * XXX when WAL archiving is actually supported, this test will likely - * need to change; and the hardwired extern is cruddy anyway ... */ { ! extern char XLOG_archive_dir[]; ! ! use_wal = XLOG_archive_dir[0] && !rel->rd_istemp; } nblocks = RelationGetNumberOfBlocks(rel); --- 5398,5408 ---- FlushRelationBuffers(rel, 0); /* ! * We need to log the copied data in WAL if WAL archiving is enabled * AND it's not a temp rel. */ { ! use_wal = XLogArchiveMode && !rel->rd_istemp; } nblocks = RelationGetNumberOfBlocks(rel); Index: backend/postmaster/Makefile =================================================================== RCS file: /projects/cvsroot/pgsql-server/src/backend/postmaster/Makefile,v retrieving revision 1.15 diff -c -r1.15 Makefile *** backend/postmaster/Makefile 29 May 2004 22:48:19 -0000 1.15 --- backend/postmaster/Makefile 14 Jul 2004 22:41:46 -0000 *************** *** 12,18 **** top_builddir = ../../.. include $(top_builddir)/src/Makefile.global ! OBJS = postmaster.o bgwriter.o pgstat.o all: SUBSYS.o --- 12,18 ---- top_builddir = ../../.. include $(top_builddir)/src/Makefile.global ! OBJS = postmaster.o bgwriter.o pgstat.o pgarch.o all: SUBSYS.o Index: backend/postmaster/postmaster.c =================================================================== RCS file: /projects/cvsroot/pgsql-server/src/backend/postmaster/postmaster.c,v retrieving revision 1.411 diff -c -r1.411 postmaster.c *** backend/postmaster/postmaster.c 12 Jul 2004 19:14:56 -0000 1.411 --- backend/postmaster/postmaster.c 14 Jul 2004 22:41:48 -0000 *************** *** 117,123 **** #include "utils/ps_status.h" #include "bootstrap/bootstrap.h" #include "pgstat.h" ! /* * List of active backends (or child processes anyway; we don't actually --- 117,123 ---- #include "utils/ps_status.h" #include "bootstrap/bootstrap.h" #include "pgstat.h" ! #include "pgarch.h" /* * List of active backends (or child processes anyway; we don't actually *************** *** 198,203 **** --- 198,204 ---- /* PIDs of special child processes; 0 when not running */ static pid_t StartupPID = 0, BgWriterPID = 0, + PgArchPID = 0, PgStatPID = 0; /* Startup/shutdown state */ *************** *** 1217,1222 **** --- 1218,1228 ---- kill(BgWriterPID, SIGUSR2); } + /* If we have lost the archiver, try to start a new one */ + if (XLogArchiveMode && PgArchPID == 0 && + StartupPID == 0 && !FatalError && Shutdown == NoShutdown) + PgArchPID = pgarch_start(); + /* If we have lost the stats collector, try to start a new one */ if (PgStatPID == 0 && StartupPID == 0 && !FatalError && Shutdown == NoShutdown) *************** *** 1821,1826 **** --- 1827,1835 ---- /* Tell pgstat to shut down too; nothing left for it to do */ if (PgStatPID != 0) kill(PgStatPID, SIGQUIT); + /* Tell pgarch to shut down too; nothing left for it to do */ + if (PgArchPID != 0) + kill(PgArchPID, SIGQUIT); break; case SIGINT: *************** *** 1865,1870 **** --- 1874,1882 ---- /* Tell pgstat to shut down too; nothing left for it to do */ if (PgStatPID != 0) kill(PgStatPID, SIGQUIT); + /* Tell pgarch to shut down too; nothing left for it to do */ + if (PgArchPID != 0) + kill(PgArchPID, SIGQUIT); break; case SIGQUIT: *************** *** 1882,1887 **** --- 1894,1901 ---- kill(BgWriterPID, SIGQUIT); if (PgStatPID != 0) kill(PgStatPID, SIGQUIT); + if (PgArchPID != 0) + kill(PgArchPID, SIGQUIT); if (DLGetHead(BackendList)) SignalChildren(SIGQUIT); ExitPostmaster(0); *************** *** 1971,1978 **** */ if (Shutdown > NoShutdown && BgWriterPID != 0) kill(BgWriterPID, SIGUSR2); ! else if (PgStatPID == 0 && Shutdown == NoShutdown) ! PgStatPID = pgstat_start(); continue; } --- 1985,1996 ---- */ if (Shutdown > NoShutdown && BgWriterPID != 0) kill(BgWriterPID, SIGUSR2); ! else if (Shutdown == NoShutdown) { ! if (PgStatPID == 0) ! PgStatPID = pgstat_start(); ! if (PgArchPID == 0) ! PgArchPID = pgarch_start(); ! } continue; } *************** *** 2021,2026 **** --- 2039,2060 ---- } /* + * Was it the archiver? If so, just try to start a new + * one; no need to force reset of the rest of the system. (If fail, + * we'll try again in future cycles of the main loop.) + */ + if (PgArchPID != 0 && pid == PgArchPID) + { + PgArchPID = 0; + if (exitstatus != 0) + LogChildExit(LOG, gettext("archiver process"), + pid, exitstatus); + if (StartupPID == 0 && !FatalError && Shutdown == NoShutdown) + PgArchPID = pgarch_start(); + continue; + } + + /* * Else do standard backend child cleanup. */ CleanupProc(pid, exitstatus); *************** *** 2202,2207 **** --- 2236,2252 ---- kill(PgStatPID, SIGQUIT); } + /* Force a power-cycle of the pgarch processes too */ + /* (Shouldn't be necessary, but just for luck) */ + if (PgArchPID != 0 && !FatalError) + { + ereport(DEBUG2, + (errmsg_internal("sending %s to process %d", + "SIGQUIT", + (int) PgArchPID))); + kill(PgArchPID, SIGQUIT); + } + FatalError = true; } *************** *** 2951,2956 **** --- 2996,3013 ---- if (Shutdown <= SmartShutdown) SignalChildren(SIGUSR1); } + + if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER)) + { + /* + * Send SIGUSR1 to ARCHIVER process, to wake it up and begin + * archiving next transaction log file. Backend should only + * send if in XLogArchiveMode... + */ + if (XLogArchiveMode && Shutdown == NoShutdown) { + kill(PgArchPID,SIGUSR1); + } + } PG_SETMASK(&UnBlockSig); Index: backend/utils/misc/guc.c =================================================================== RCS file: /projects/cvsroot/pgsql-server/src/backend/utils/misc/guc.c,v retrieving revision 1.219 diff -c -r1.219 guc.c *** backend/utils/misc/guc.c 12 Jul 2004 02:22:51 -0000 1.219 --- backend/utils/misc/guc.c 14 Jul 2004 22:41:52 -0000 *************** *** 342,347 **** --- 342,363 ---- static struct config_bool ConfigureNamesBool[] = { + { + {"archive_mode", PGC_POSTMASTER, WAL_SETTINGS, + gettext_noop("Enable archiving of full transaction log files to a specified archival destination."), + NULL + }, + &XLogArchiveMode, + false, NULL, NULL + }, + { + {"archive_debug", PGC_SIGHUP, WAL_SETTINGS, + gettext_noop("Provide debug output for archive activities."), + NULL + }, + &XLogArchiveDEBUG, + false, NULL, NULL + }, { {"enable_seqscan", PGC_USERSET, QUERY_TUNING_METHOD, gettext_noop("Enables the planner's use of sequential-scan plans."), *************** *** 1371,1376 **** --- 1387,1410 ---- static struct config_string ConfigureNamesString[] = { + { + {"archive_dest", PGC_POSTMASTER, WAL_SETTINGS, + gettext_noop("Specifies where to archive WAL logs."), + gettext_noop("A directory or specific location for archiving transation log files from PostgreSQL") + }, + &XLogArchiveDest, + "", NULL, NULL + }, + + { + {"archive_program", PGC_POSTMASTER, WAL_SETTINGS, + gettext_noop("Archive program"), + gettext_noop("The external program that will be called to execute the archival process") + }, + &XLogArchiveProgram, + "", NULL, NULL + }, + { {"client_encoding", PGC_USERSET, CLIENT_CONN_LOCALE, gettext_noop("Sets the client's character set encoding."), Index: backend/utils/misc/postgresql.conf.sample =================================================================== RCS file: /projects/cvsroot/pgsql-server/src/backend/utils/misc/postgresql.conf.sample,v retrieving revision 1.115 diff -c -r1.115 postgresql.conf.sample *** backend/utils/misc/postgresql.conf.sample 11 Jul 2004 21:48:25 -0000 1.115 --- backend/utils/misc/postgresql.conf.sample 14 Jul 2004 22:41:52 -0000 *************** *** 113,118 **** --- 113,129 ---- #--------------------------------------------------------------------------- + # ARCHIVING + #--------------------------------------------------------------------------- + + # - Settings - + + #archive_mode = true # enables archiving of full txn log files + #archive_dest = '/tmp' # specifies destination of archive files + #archive_program = 'cp %s %s' # external archiving program command line + + + #--------------------------------------------------------------------------- # QUERY TUNING #--------------------------------------------------------------------------- Index: bin/initdb/initdb.c =================================================================== RCS file: /projects/cvsroot/pgsql-server/src/bin/initdb/initdb.c,v retrieving revision 1.43 diff -c -r1.43 initdb.c *** bin/initdb/initdb.c 14 Jul 2004 17:55:10 -0000 1.43 --- bin/initdb/initdb.c 14 Jul 2004 22:41:55 -0000 *************** *** 2023,2029 **** char *pgdenv; /* PGDATA value got from sent to * environment */ char *subdirs[] = ! {"global", "pg_xlog", "pg_clog", "pg_subtrans", "base", "base/1", "pg_tblspc"}; progname = get_progname(argv[0]); set_pglocale_pgservice(argv[0], "initdb"); --- 2023,2030 ---- char *pgdenv; /* PGDATA value got from sent to * environment */ char *subdirs[] = ! ! {"global", "pg_xlog", "pg_xlog/archive_status", "pg_clog", "pg_subtrans", "base", "base/1", "pg_tblspc"}; progname = get_progname(argv[0]); set_pglocale_pgservice(argv[0], "initdb"); Index: include/access/xlog.h =================================================================== RCS file: /projects/cvsroot/pgsql-server/src/include/access/xlog.h,v retrieving revision 1.52 diff -c -r1.52 xlog.h *** include/access/xlog.h 1 Jul 2004 00:51:38 -0000 1.52 --- include/access/xlog.h 14 Jul 2004 22:41:56 -0000 *************** *** 210,215 **** --- 210,219 ---- extern int XLOGbuffers; extern char *XLOG_sync_method; extern const char XLOG_sync_method_default[]; + extern bool XLogArchiveMode; + extern bool XLogArchiveDEBUG; + extern char *XLogArchiveDest; + extern char *XLogArchiveProgram; #ifdef WAL_DEBUG extern bool XLOG_DEBUG; Index: include/storage/pmsignal.h =================================================================== RCS file: /projects/cvsroot/pgsql-server/src/include/storage/pmsignal.h,v retrieving revision 1.8 diff -c -r1.8 pmsignal.h *** include/storage/pmsignal.h 29 May 2004 22:48:23 -0000 1.8 --- include/storage/pmsignal.h 14 Jul 2004 22:41:56 -0000 *************** *** 24,29 **** --- 24,30 ---- { PMSIGNAL_PASSWORD_CHANGE, /* pg_pwd file has changed */ PMSIGNAL_WAKEN_CHILDREN, /* send a SIGUSR1 signal to all backends */ + PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to ARCHIVER */ NUM_PMSIGNALS /* Must be last value of enum! */ } PMSignalReason; --=-swM3+xD6gWhLkjHvf/5u Content-Disposition: attachment; filename=recovery.conf.sample Content-Type: text/plain; name=recovery.conf.sample; charset= Content-Transfer-Encoding: 7bit # ------------------------------- # PostgreSQL recovery config file # ------------------------------- # # Edit this file to provide the parameters that PostgreSQL # needs to perform an archive recovery of a database # # This file consists of lines of the form: # # name = value # # (The ' = ' is NOT optional.) # # Comments are introduced with '#' at beginning of a line. # # The complete list of option names and # allowed values can be found in the PostgreSQL documentation. The # commented-out settings shown in this file show optional values. # # These options cannot be specified in any other way. There are # no command line switches, to ensure that when recovery completes # we do not accidentally re-enter archive recovery. # # This file is read on postmaster startup. # #--------------------------------------------------------------------------- # REQUIRED PARAMETERS #--------------------------------------------------------------------------- # # restore program # # specifies the program that is called to copy log files # back from archive for use # command string must specify 3 parameters, using %s # parameter1 will be substituted with archive_dest # parameter2 will be substituted with the single log file # that will be restored (one file per execution) # parameter3 will be substituted with the full path of # the file to which the restore program will restore # # restore_program = 'cp %s/%s %s' # # #--------------------------------------------------------------------------- # RECOVERY TARGET #--------------------------------------------------------------------------- # # By default, recovery will rollforward to the end of logs # If you want to stop rollforward before that point, you # MUST set a recovery target. # # You may set a recovery target either by transactionId, or # by timestamp. Recovery may either include or exclude the # records with the recovery target value. # # #recovery_target_time = '2004-07-14-22:39:00' #use_local_time = 'true' # recovery_target_xid = '11000' # # true or false recovery_target_inclusive = 'true' # #--------------------------------------------------------------------------- # OPTIONAL PARAMETERS #--------------------------------------------------------------------------- # recovery_debug_log = 'true' # #--------------------------------------------------------------------------- --=-swM3+xD6gWhLkjHvf/5u Content-Disposition: attachment; filename=pgarch.c Content-Type: text/x-c; name=pgarch.c; charset= Content-Transfer-Encoding: 7bit /* ---------- * pgarch.c * * PostgreSQL transaction log archiver * * All functions relating to archiver are included here * * - All functions executed by archiver process * * - archiver is forked from postmaster, and the two * processes then communicate using signals. All functions * executed by postmaster are included in this file. * * Simon Riggs simon@2ndquadrant.com * * ---------- */ #include "postgres.h" #include #include #include #include #include #include #include #include #include "pgarch.h" #include "storage/fd.h" #include "miscadmin.h" #include "access/xlog.h" #include "libpq/pqsignal.h" #include "storage/ipc.h" #include "postmaster/postmaster.h" #include "storage/pg_shmem.h" #include "storage/pmsignal.h" #include "utils/ps_status.h" /* ---------- * GUC parameters * ---------- */ bool XLogArchiveMode; bool XLogArchiveDEBUG; char *XLogArchiveDest; char *XLogArchiveProgram; /* ---------- * Other global variables * ---------- */ bool pgarch_is_running = false; /* ---------- * Local data * ---------- */ static char XLogDir[MAXPGPATH]; static char XLogArchiveStatusDir[MAXPGPATH]; static time_t last_pgarch_start_time; static time_t last_pgarch_archivercopy_time; static time_t curtime; static bool archiving_in_progress = false; /* ---------- * Local function forward declarations * ---------- */ NON_EXEC_STATIC void pgarch_Main(int argc, char *argv[]); static void pgarch_exit(SIGNAL_ARGS); static void pgarch_waken(SIGNAL_ARGS); static void pgarch_MainWaitLoop(void); static void pgarch_ArchiverCopyLoop(void); static bool pgarch_archiveXlog(char *xlog, bool testmode); static bool pgarch_readyXlog(char *xlog); static void pgarch_archiveDone(char *xlog); /* ------------------------------------------------------------ * Public functions called from postmaster follow * ------------------------------------------------------------ */ /* ---------- * pgarch_start() - * * Called from postmaster at startup * or after the previous archiver died * * Note: if fail, we will be called again from the postmaster main loop. * ---------- */ int pgarch_start(void) { time_t curtime; pid_t pgArchPid; /* * Do nothing if no archiver needed */ if (!XLogArchiveMode) return 0; /* * Do nothing if too soon since last archiver start. This is a * safety valve to protect against continuous respawn attempts if the * archiver is dying immediately at launch. Note that since we will * be re-called from the postmaster main loop, we will get another * chance later. */ curtime = time(NULL); if ((unsigned int) (curtime - last_pgarch_start_time) < (unsigned int) PGARCH_RESTART_INTERVAL) { return 0; } last_pgarch_start_time = curtime; fflush(stdout); fflush(stderr); #ifdef __BEOS__ /* Specific beos actions before backend startup */ beos_before_backend_startup(); #endif #ifdef EXEC_BACKEND switch ((pgArchPid = (int) pgarch_forkexec(STAT_PROC_BUFFER))) #else switch ((pgArchPid = (int) fork())) #endif { case -1: #ifdef __BEOS__ /* Specific beos actions */ beos_backend_startup_failed(); #endif ereport(LOG, (errmsg("could not fork archiver"))); exit(1); #ifndef EXEC_BACKEND case 0: /* in postmaster child ... */ #ifdef __BEOS__ /* Specific beos actions after backend startup */ beos_backend_startup(); #endif /* Close the postmaster's sockets */ ClosePostmasterPorts(); /* Drop our connection to postmaster's shared memory, as well */ PGSharedMemoryDetach(); pgarch_Main(0, NULL); break; #endif default: return (int) pgArchPid; } /* shouldn't get here */ return 0; } /* ------------------------------------------------------------ * Local functions called by archiver follow * ------------------------------------------------------------ */ /* ---------- * pgarch_Main() - * * * The argc/argv parameters are valid only in EXEC_BACKEND case. * ---------- */ NON_EXEC_STATIC void pgarch_Main(int argc, char *argv[]) { char testxlog[32] = "0000TEST0000TEST"; char testpath[MAXPGPATH]; FILE *rlogFD; IsUnderPostmaster = true; /* we are a postmaster subprocess now */ MyProcPid = getpid(); /* reset MyProcPid */ /* Lose the postmaster's on-exit routines */ on_exit_reset(); /* Init XLOG file paths */ snprintf(XLogDir, MAXPGPATH, "%s/pg_xlog", DataDir); snprintf(XLogArchiveStatusDir, MAXPGPATH, "%s/archive_status", XLogDir); /* * Test whether archive_program and archive_dest have been set * correctly in postgresql.conf. This test does *not* look at * archive_status, since it is a direct test of archival */ snprintf(testpath, MAXPGPATH, "%s/%s", XLogDir, testxlog); unlink(testpath); rlogFD = AllocateFile(testpath, "w"); if (rlogFD == NULL) { ereport(ERROR, (errcode_for_file_access(), errmsg("could not write test file to pg_xlog \"%s\": %m", testpath))); exit(1); } FreeFile(rlogFD); if (!pgarch_archiveXlog(testxlog, true)) { ereport(ERROR, (errmsg("your archive_program failed to archive a test file"), errhint("your archive_program is not specified correctly " "or the archive system is currently unavailable. " "test your program again before restarting"))); exit(1); } unlink(testpath); /* * Ignore all signals usually bound to some action in the postmaster, */ pqsignal(SIGHUP, SIG_IGN); pqsignal(SIGTERM, SIG_IGN); pqsignal(SIGINT, SIG_IGN); pqsignal(SIGQUIT, pgarch_exit); pqsignal(SIGALRM, SIG_IGN); pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, pgarch_waken); pqsignal(SIGUSR2, SIG_IGN); pqsignal(SIGCHLD, SIG_IGN); pqsignal(SIGTTIN, SIG_DFL); pqsignal(SIGTTOU, SIG_DFL); pqsignal(SIGCONT, SIG_DFL); pqsignal(SIGWINCH, SIG_DFL); PG_SETMASK(&UnBlockSig); #ifdef EXEC_BACKEND pgstat_parseArgs(argc,argv); #endif /* * Identify via ps */ init_ps_display("archiver process", "", ""); set_ps_display(""); ereport(LOG, (errmsg("archiver started"))); /* * When first started, check for outstanding archive files * which may be present if we did a (normal) fast shutdown * or if archiver died with some form of error */ pgarch_ArchiverCopyLoop(); pgarch_MainWaitLoop(); exit(0); } static void pgarch_exit(SIGNAL_ARGS) { exit(0); } #ifdef NOT_USED static bool shutdowncalled = false; /* * Experiment later with having archiver complete its work on smart shutdown... * */ static void pgarch_shutdown(SIGNAL_ARGS) { shutdowncalled = true; /* * we do not block signals here, to allow a later, * upgraded (fast/immediate) request for shutdown to * override this behaviour */ if (archiving_in_progress) { /* complete archiving, then exit */ return; } else { /* This is a Smart or Fast Shutdown, * so try archiving one last time. This is consistent * with behaviour of a fast shutdown, in that we still * write a shutdown checkpoint and try to recycle the * log files - so before we do this, we try to archive * away the last few .ready xlogs * * XXX archive partial xlog file, even though not full * though we'd have to remember which one was next to * handle that correctly on restore/recover */ pgarch_ArchiverCopyLoop(); return; } } #endif static void pgarch_waken(SIGNAL_ARGS) { pgarch_ArchiverCopyLoop(); return; } /* ---------- * pgarch_MainWaitLoop() - * * Main wait loop for archiver * ---------- */ static void pgarch_MainWaitLoop(void) { /* * There shouldn't be anything for the archiver to do except * to wait, so we could use pause(3) here... * ...however, the archiver exists to protect our data, so * she wakes up occaisionally to allow herself to be proactive. * This shouldn't be required, but our data is important * and this won't hurt to be cautious */ do { /* * Sleep for a while, hoping to be interrupted by signal * if no signal, then check anyway....just to be sure */ sleep(PGARCH_AUTOWAKE_INTERVAL); curtime = time(NULL); if ((unsigned int) (curtime - last_pgarch_archivercopy_time) >= (unsigned int) PGARCH_AUTOWAKE_INTERVAL) { pgarch_ArchiverCopyLoop(); } last_pgarch_archivercopy_time = curtime; } while (PostmasterIsAlive(true)); return; } /* ---------- * pgarch_ArchiverCopyLoop() - * * Archives all outstanding xlogs then exits * ---------- */ static void pgarch_ArchiverCopyLoop(void) { char xlog[32]; int try = 1; /* * We continue to trap for all signals, except for the * one that brought us here in the first place. We * loop through all transaction log files that require * archiving, so no need to be interrupted to continue * that task */ pqsignal(SIGUSR1, SIG_IGN); archiving_in_progress = true; /* * loop through all xlogs with archive_status of .ready * then archive them...mostly we expect this to be a single * file, though not just a simple loop because we may add new * files onto the list of those that need archiving while we * are still copying earlier archives */ while (pgarch_readyXlog(xlog) && try <= NUM_ARCHIVE_RETRIES) { if (pgarch_archiveXlog(xlog, false)) { pgarch_archiveDone(xlog); } else { ereport(WARNING, (errmsg("transaction log file \"%s\" could not be archived", xlog), errhint("archive_dest \"%s\" may not have enough disk space " "or the archive system is currently unavailable", XLogArchiveDest))); } /* if we have copied one file, we do not wait: immediately loop back round and check to see if another is there. Hopefully, we're quick enough....so we fall out and sleep again */ try++; } archiving_in_progress = false; pqsignal(SIGUSR1, pgarch_waken); return; } /* * pgarch_archiveXlog * * Invokes system(3) to copy one archive file to XLogArchiveDest * We assume xlog is a correct filename and that both * XLogArchiveProgram and XLogArchiveDest are set correctly */ static bool pgarch_archiveXlog(char *xlog, bool testmode) { char xlogarchcmd[MAXPGPATH]; char xlogpath[MAXPGPATH]; int rc; snprintf(xlogpath, MAXPGPATH, "%s/%s", XLogDir,xlog); /* * set the string for the program and its parameters * XLogArchiveProgram should contain 2 positional parameters * xlog must be a full path to xlog */ snprintf(xlogarchcmd, MAXPGPATH, XLogArchiveProgram, xlogpath, XLogArchiveDest); if (XLogArchiveDEBUG) ereport(LOG, (errmsg("archive command=\"%s\"", xlogarchcmd))); rc = system(xlogarchcmd); if (rc != 0 ) { ereport(LOG, (errmsg("archive command=\"%s\",return code=%i", xlogarchcmd, rc))); return false; } if (!testmode) ereport(LOG, (errmsg("archived transaction log file \"%s\"", xlog))); return true; } /* * XLogArchiveXlogs * * Return name of the oldest xlog file that has not yet been archived. * No notification is set that file archiving is now in progress, [so * this would need to be extended if multiple concurrent archival * tasks were created]. If a failure occurs, we would completely * re-copy the file at the next available opportunity. * * It is important that we return the oldest, so that we archive xlogs * in order that they were written, for two reasons: * 1) to maintain the sequential chain of xlogs required for recovery * 2) because the oldest ones will sooner become candidates for * recycling at time of checkpoint * */ static bool pgarch_readyXlog(char *xlog) { /* * open XLogArchive directory and read through list of * rlogs that have the .ready suffix, looking for earliest file. * It is possible to optimise this code, though only a single * file is expected on the vast majority of calls, so.... */ char newxlog[32]; char emptystr[32] = "\0"; DIR *rldir; struct dirent *rlde; bool firstfile; rldir = AllocateDir(XLogArchiveStatusDir); if (rldir == NULL) ereport(ERROR, (errcode_for_file_access(), errmsg("could not open archive_status directory \"%s\": %m", XLogArchiveStatusDir))); firstfile = true; while ((rlde = readdir(rldir)) != NULL) { if (strlen(rlde->d_name) == 22 && strspn(rlde->d_name, "0123456789ABCDEF") == 16 && strcmp(rlde->d_name + 16, ".ready") == 0) { if (firstfile) { strcpy(newxlog, rlde->d_name); firstfile = false; } else { if (strcmp(rlde->d_name, newxlog) <= 0) strcpy(newxlog, rlde->d_name); } } } FreeDir(rldir); if (!firstfile) { strcpy(xlog, emptystr); strncat(xlog, newxlog, 16); return true; } else return false; } /* * pgarch_archiveDone * * Write notification that an xlog has now been successfully archived */ static void pgarch_archiveDone(char *xlog) { char rlogready[MAXPGPATH]; char rlogdone[MAXPGPATH]; int rc; snprintf(rlogready, MAXPGPATH, "%s/%s.ready", XLogArchiveStatusDir, xlog); snprintf(rlogdone, MAXPGPATH, "%s/%s.done", XLogArchiveStatusDir, xlog); rc = rename(rlogready, rlogdone); if (rc < 0) { ereport(WARNING,(errcode_for_file_access(), errmsg("could not update archive_status for \"%s\"", rlogready))); return; } return; } --=-swM3+xD6gWhLkjHvf/5u Content-Disposition: attachment; filename=pgarch.h Content-Type: text/x-c-header; name=pgarch.h; charset= Content-Transfer-Encoding: 7bit /* ---------- * pgarch.h * * Definitions for the PostgreSQL archiver daemon. * * ---------- */ /* ---------- * Timer definitions. * ---------- */ #define PGARCH_AUTOWAKE_INTERVAL 600 /* How often to wake and poll */ #define PGARCH_RESTART_INTERVAL 60 /* How often to attempt to restart */ /* a failed statistics collector; in seconds. */ #define NUM_ARCHIVE_RETRIES 3 /* ---------- * Functions called from postmaster * ---------- */ extern int pgarch_start(void); --=-swM3+xD6gWhLkjHvf/5u Content-Disposition: attachment; filename=README Content-Type: text/html; name=README; charset= Content-Transfer-Encoding: 7bit PITR Archive Recovery, 8 July 2004

What's in this patch?

- All code from previous patches: working now against CVS tip as of now
- Partially complete code for PITR, though not enabled as yet.

This is a full, working patch for discussion and testing, while I complete the additional code.

Archive Recovery Overview

The rule is: when you startup with archive_mode on, take a backup.

This patch provides an additional mode for transactional log recovery.
The current mode provides Crash Recovery, allowing the database to
restart and to read the last few online transaction logs to recover. The
additional mode is known as Archive Recovery, enabled in postgresql.conf
using archive_mode=true.

The administrator also specifies and archive location and a program to
perform the copying (discussed later). As database writes occur and
transaction logs fill they are copied to the archive location by a new
sub-process of the postmaster, called the archiver process.

If the database crashes when in archive_mode, it will recover just as
before. If a major system crash, or other reason to restore occurs, then
the new Archive Recovery mode can be invoked by:
- restoring the full physical backup
- creating DataDir/recovery.conf
- restart PostgreSQL

PostgreSQL will recover using almost the same code path as before,
except: all transaction logs will be restored for use directly from
archive in a stream that prevents disk space from overflowing. Extended
recovery is possible, over many transaction logs - far more than was
previously possible.

The essential aspect to recovery is maintaining a solid backup/log
chain. This must consist of:
i) a full backup of every file in PostgreSQL - miss none!
ii) archived transaction logs from just before backup taken to whenever
you want to recover to...

If you "break the chain" i.e. the two sets of files don't match, or
you've deleted a transaction log or lost a tape - then you will only be
able to recover up to the point the chain broke. The longer the chain,
the easier it will be to break, so backup regularly and work out a
robust archiving policy.

Extensibility
Archive Recovery allows the administrator to specify a program or script
to use when both archiving and restoring transaction log files. As a
result, the administrator can choose to store transaction log files on
any drive, any other system or integrating with any Backup Archive
Recovery (BAR) software.

The Archive Program is provided with 3 parameters, which can be placed
anywhere within the supplied command string. Each parameter is
represented by a "%s" character string.
- The first parameter is replaced with the full path of the transaction
log to be archived.
- The second parameter is replaced with the value of the archive_dest
parameter
e.g. with these settings
archive_mode = true
archive_dest = '/mount/disk2/pgarchive'
archive_program = 'cp %s %s'
with transaction log 00000000000003A4 to archive
with DataDir of web123

would execute the following command
'cp /usr/local/pgsql/web123/00000000000003A4 /disk2/pgarchive'

To enter Archive Recovery the administrator must create a recovery.conf
file. [Currently an empty file will do, though intended as the one place
where all recovery options and parameters would be set]

The Restore Program is provided with 2 parameters
- The first parameter is replaced with the value of the archive_dest
parameter
- The second parameter is replaced with the value of the transaction log
which recovery is requesting should now be restored
- The third parameter is replaced with the full path of the filename
which should be the target of the single file recovery operation.
[In this patch, XlogArchRestoreProgram is hardcoded to "cp %s/%s %s",
pending short discussion on how to specify this...]

Possibilities
Using the current features, it is possible to implement an Automated
Standby Database. This is an active-passive High Availability option. In
this mode, the main server sends archived log files to a second, standby
server. The standby server is set up to be in "permanent recovery", by
using a RestoreProgram that waits for each file to be shipped to it. The
standby system receives each file, then recovers up to that point - so
the standby system is always a few seconds from completing its startup
should it be required.

Patch Status & Current Caveats
- A number of recovery scenarios have been tested and the patch is
believed to be stable and ready for others to begin commentary and
testing...please understand that there are many scenarios that work and
many that do not...these last are not bugs

- Recovery to a specific point in time is not yet implemented. The
administrator has the following recovery options:
--can recover the system fully, all the way to the end of logs
--stop recovery by withholding log files at an appropriate place,
thereby forcing the termination of recovery

How it Works

archive_debug is a hidden postgresql.conf setting which can be used to
show more debug output for the archive recovery facilities.
...archive_debug = true

[This currently generates an additional log file called recovery.log,
showing the record headers and types of commit/abort xlog records.]

Each PostgreSQL user has a corresponding backend process performing work
for them. These backends write any transaction log data to the log as
each operation occurs, then marks transactions as either committed or
aborted. The log is split into log segment files. When each log file
fills a notification file is created in archive_status, and at the same
time a signal is sent to the archiver (via the postmaster) to begin
archiving.

The archiver executes the administrator's program or script using a
system(3) call, archiving each file one at a time.

The notification file provides a number of features. First, its
simplicity ensures that we don't need a shared memory link between each
backend and the archiver, which would then be prone to failure - which
is exactly not what we want. Second, the archive_status file is written
whether or not the postmaster is still up, so will still function
correctly even when the worst has happened. Third, as a persistent
record, it will allow archiving to restart at the same point later on
(even after a backup/restore), allowing it to recover gracefully from
administrator's immediate shutdown requests as well as postmaster or
archive system failures.

When recovery completes, archive_status files are written to ensure
cleanup of all transaction logs leaves the database in a fit state for
production. The recovery.conf file is removed to ensure that a
subsequent crash doesn't accidentally begin recovery from archives...

The archiver process starts only when recovery has fully completed.

How to Fail

All of the following ways to fail have already occurred in testing...

1. if you do all 3 of these, you will not have a backup/log chain:
a) use the 'cp' command, or any other command that can write to either a
file or a directory
b) use archive_dest to specify a directory
c) forget to create the directory, which the command in a) then
interprets as a file and then all archived logs overwrite each other and
break the chain. You will be unable to rollforward AT ALL.

2. if you do all 5 of these, you will not rollforward very far, if any.
a) restore the database backup
b) forget to specify a recovery.conf file
c) startup and let database recover using only the xlogs that were
present when the backup was taken
d) then immediately after recovery run a job which causes xlog to be
written, and then a file to fill
e) use an archive command that allows overwriting of previously archived
files
This sequence causes the database to recover successfully, but not using
your full archive chain. The current redo pointer points half way
through your backup/log chain, so when the file is archived, it
overwrites a file in the middle of the chain, thus breaking the chain.
There isn't an option on cp that will prevent this...

3. turn archive_mode on and off...
Turning archive_mode off means that when an xlog fills, it won't EVER
get archived. This will break the backup/log chain. If you turn
archive_mode back on, and an xlog filled, then it will not be picked up
as having filled and you will be missing a link in the chain. Sometimes
you'll get away with it and then you'll think it will work all the time.
The rule is: when you startup with archive_mode on, take a backup.

4. Forget that xlogs are the same size, whether they are full or
not...you can't tell (yet) by looking at one whether it has a record
within it that will break the chain

How to succeed
- Test your recovery procedures before you need them
- When you're in a recovery situation backup everything you can lay your
hands on, to make sure you have a known position to return to while you
ATTEMPT recovery - you may make a mistake and need to retry
- be cool - we all make mistakes, just don't let errors multiply

- disable listen_addresses in postgresql.conf when you recover, to give
yourself some breathing space to check things, when you finally do get
the database ready
- recover with archive_debug = true to give me/others a chance to debug
any problems or answer any questions you may have

The rule is: when you startup with archive_mode on, take a backup.

When enabling archive_mode for the first time, if you have run
previously without enabling archive_mode, then when you will inevitably
get messages saying "cannot find archive_status file". After you have
taken a backup, you can manually create files in the archive_status
directory of <xlog>.done, which will then allow the bgwriter to clean
them up when it next takes a checkpoint.

Additional Work

- recovery.conf needs some minor work to specify recover options. It is
envisaged that this would be a short bison grammar, very very similar to
postgresql.conf

- It seems possible to easily work around the requirement to take the
backup while the database is open. It would be a good option to have
both hot and cold backup options...

- full documentation will also follow shortly

Credits

This work is the final push in a long series of patches and discussions
about how to achieve archive recovery with PostgreSQL. The work of
J.R.Nield and Patrick MacDonald has provided the detailed underpinnings
for this recent work, which in turn rests upon Vadim Mikheev's original
work on WAL and MVCC. Tom Lane has provided considerable technical
assistance and quality review, whilst Bruce Momjian has provided many of
the ideas and smoothed the way for much of the work. Thanks, all.

Simon Riggs, simon@2ndquadrant.com

--=-swM3+xD6gWhLkjHvf/5u--