pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedPITR COPY Failure (was Point in Time Recovery)
19+ messages / 3 participants
[nested] [flat]
* PITR COPY Failure (was Point in Time Recovery)
@ 2004-07-18 05:50 Mark Kirkwood <markir@coretech.co.nz>
0 siblings, 3 replies; 19+ messages in thread
From: Mark Kirkwood @ 2004-07-18 05:50 UTC (permalink / raw)
To: Simon Riggs <simon@2ndquadrant.com>; +Cc: pgsql-hackers@postgresql.org
I decided to produce a nice simple example, so that anyone could
hopefully replicate what I am seeing.
The scenario is the same as before (the 11 steps), but the CREATE TABLE
and COPY step has been reduced to:
CREATE TABLE test0 (filler VARCHAR(120));
COPY test0 FROM '/data0/dump/test0.dat' USING DELIMITERS ',';
Now the file 'test0.dat' consists of (128293) identical lines, each of
109 'a' charactors (plus end of line)
A script to run the whole business can be found here :
http://homepages.paradise.net.nz/markir/download/pitr-bug.tar.gz
(It will need a bit of editing for things like location of Pg, PGDATA,
and you will need to make your own data file)
The main points of interest are:
- anything <=128392 rows in test0.dat results in 1 archived log, and the
recovery succeeds
- anything >=128393 rows in test0.dat results in 2 or more archived
logs, and recovery fails on the second log (and gives the zero length
redo at 0/1FFFFE0 message).
Let me know if I can do any more legwork on this (I am considering
re-compiling with WAL_DEBUG now that example is simpler)
regards
Mark
Simon Riggs wrote:
>On Thu, 2004-07-15 at 10:47, Mark Kirkwood wrote:
>
>
>>I tried what I thought was a straightforward scenario, and seem to have
>>broken it :-(
>>
>>Here is the little tale
>>
>>1) initdb
>>2) set archive_mode and archive_dest in postgresql.conf
>>3) startup
>>4) create database called 'test'
>>5) connect to 'test' and type 'checkpoint'
>>6) backup PGDATA using 'tar -zcvf'
>>7) create tables in 'test' and add data using COPY (exactly 2 logs worth)
>>8) shutdown and remove PGDATA
>>9) recover using 'tar -zxvf'
>>10) copy recovery.conf into PGDATA
>>11) startup
>>
>>This is what I get :
>>
>>LOG: database system was interrupted at 2004-07-15 21:24:04 NZST
>>LOG: recovery command file found...
>>LOG: restore_program = cp %s/%s %s
>>LOG: recovery_target_inclusive = true
>>LOG: recovery_debug_log = true
>>LOG: starting archive recovery
>>LOG: restored log file "0000000000000000" from archive
>>LOG: checkpoint record is at 0/A48054
>>LOG: redo record is at 0/A48054; undo record is at 0/0; shutdown FALSE
>>LOG: next transaction ID: 496; next OID: 25419
>>LOG: database system was not properly shut down; automatic recovery in
>>progress
>>LOG: redo starts at 0/A48094
>>LOG: restored log file "0000000000000001" from archive
>>LOG: record with zero length at 0/1FFFFE0
>>LOG: redo done at 0/1FFFF30
>>LOG: restored log file "0000000000000001" from archive
>>LOG: restored log file "0000000000000001" from archive
>>PANIC: concurrent transaction log activity while database system is
>>shutting down
>>LOG: startup process (PID 13492) was terminated by signal 6
>>LOG: aborting startup due to startup process failure
>>
>>The concurrent access is a bit of a puzzle, as this is my home machine
>>(i.e. I am *sure* noone else is connected!)
>>
>>
>>
>
>I can see what is wrong now, but you'll have to help me on details your
>end...
>
>The log shows that xlog 1 was restored from archive. It contains a zero
>length record, which indicates that it isn't yet full (or thats what the
>existing recovery code assumes it means). Which also indicates that it
>should never have been archived in the first place, and should not
>therefore be a candidate for a restore from archive.
>
>The double message "restored log file" can only occur after you've
>retrieved a partially full file from archive - which as I say, shouldn't
>be there.
>
>Other messages are essentially spurious in those circumstances.
>
>Either:
>- somehow the files have been mixed up in the archive directory, which
>is possible if the filing discipline is not strict - various ways,
>unfortunately I would guess this to be the most likely, somehow
>- the file that has been restored has been damaged in some way
>- the archiver has archived a file too early (very unlikely, IMHO -
>thats the most robust bit of the code)
>- some aspect of the code has written a zero length record to WAL (which
>is supposed to not be possible, but we musn't discount an error in
>recent committed work)
>
>- there may also be an effect going on with checkpoints that I don't
>understand...spurious checkpoint warning messages have already been
>observed and reported,
>
>Best regards, Simon Riggs
>
>
>
>
>
>
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: PITR COPY Failure (was Point in Time Recovery)
@ 2004-07-18 06:15 Tom Lane <tgl@sss.pgh.pa.us>
parent: Mark Kirkwood <markir@coretech.co.nz>
2 siblings, 0 replies; 19+ messages in thread
From: Tom Lane @ 2004-07-18 06:15 UTC (permalink / raw)
To: Mark Kirkwood <markir@coretech.co.nz>; +Cc: Simon Riggs <simon@2ndquadrant.com>; pgsql-hackers@postgresql.org
Mark Kirkwood <markir@coretech.co.nz> writes:
> - anything >=128393 rows in test0.dat results in 2 or more archived
> logs, and recovery fails on the second log (and gives the zero length
> redo at 0/1FFFFE0 message).
Zero length record is not an error, it's the normal way of detecting
end-of-log.
regards, tom lane
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: PITR COPY Failure (was Point in Time Recovery)
@ 2004-07-18 06:48 Mark Kirkwood <markir@coretech.co.nz>
parent: Mark Kirkwood <markir@coretech.co.nz>
2 siblings, 1 reply; 19+ messages in thread
From: Mark Kirkwood @ 2004-07-18 06:48 UTC (permalink / raw)
To: Mark Kirkwood <markir@coretech.co.nz>; +Cc: Simon Riggs <simon@2ndquadrant.com>; pgsql-hackers@postgresql.org
There are some silly bugs in the script:
- forgot to export PGDATA and PATH after changing them
- forgot to mention the need to edit test.sql (COPY line needs path to
dump file)
Apologies - I will submit a fixed version a little later
regards
Mark
Mark Kirkwood wrote:
>
> A script to run the whole business can be found here :
>
> http://homepages.paradise.net.nz/markir/download/pitr-bug.tar.gz
>
> (It will need a bit of editing for things like location of Pg, PGDATA,
> and you will need to make your own data file)
>
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: PITR COPY Failure (was Point in Time Recovery)
@ 2004-07-18 08:05 Mark Kirkwood <markir@coretech.co.nz>
parent: Mark Kirkwood <markir@coretech.co.nz>
0 siblings, 0 replies; 19+ messages in thread
From: Mark Kirkwood @ 2004-07-18 08:05 UTC (permalink / raw)
To: ; +Cc: Simon Riggs <simon@2ndquadrant.com>; pgsql-hackers@postgresql.org
fixed.
Mark Kirkwood wrote:
> There are some silly bugs in the script:
>
> - forgot to export PGDATA and PATH after changing them
> - forgot to mention the need to edit test.sql (COPY line needs path to
> dump file)
>
> Apologies - I will submit a fixed version a little later
>
> regards
>
> Mark
>
> Mark Kirkwood wrote:
>
>>
>> A script to run the whole business can be found here :
>>
>> http://homepages.paradise.net.nz/markir/download/pitr-bug.tar.gz
>>
>> (It will need a bit of editing for things like location of Pg,
>> PGDATA, and you will need to make your own data file)
>>
>
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: PITR COPY Failure (was Point in Time Recovery)
@ 2004-07-20 00:48 Mark Kirkwood <markir@coretech.co.nz>
parent: Mark Kirkwood <markir@coretech.co.nz>
2 siblings, 1 reply; 19+ messages in thread
From: Mark Kirkwood @ 2004-07-20 00:48 UTC (permalink / raw)
To: Mark Kirkwood <markir@coretech.co.nz>; +Cc: Simon Riggs <simon@2ndquadrant.com>; pgsql-hackers@postgresql.org
I have been doing some re-testing with CVS HEAD from about 1 hour ago
using the simplified example posted previously.
It is quite interesting:
i) create the table as:
CREATE TABLE test0 (filler TEXT);
and COPY 100 000 rows on length 109, then recovery succeeds.
ii) create the table as:
CREATE TABLE test0 (filler VARCHAR(120));
and COPY as above, then recovery *fails* with the the signal 6 error below.
LOG: database system was not properly shut down; automatic recovery in
progress
LOG: redo starts at 0/A4807C
LOG: record with zero length at 0/FFFFE0
LOG: redo done at 0/FFFF30
LOG: restored log file "0000000000000000" from archive
LOG: archive recovery complete
PANIC: concurrent transaction log activity while database system is
shutting down
LOG: startup process (PID 17546) was terminated by signal 6
LOG: aborting startup due to startup process failure
(I am pretty sure both TEXT and VARCHAR(120) failed using the original
patch)
Any suggestions for the best way to dig a bit deeper?
regards
Mark
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: PITR COPY Failure (was Point in Time Recovery)
@ 2004-07-20 04:14 Tom Lane <tgl@sss.pgh.pa.us>
parent: Mark Kirkwood <markir@coretech.co.nz>
0 siblings, 3 replies; 19+ messages in thread
From: Tom Lane @ 2004-07-20 04:14 UTC (permalink / raw)
To: Mark Kirkwood <markir@coretech.co.nz>; +Cc: Simon Riggs <simon@2ndquadrant.com>; pgsql-hackers@postgresql.org
Mark Kirkwood <markir@coretech.co.nz> writes:
> I have been doing some re-testing with CVS HEAD from about 1 hour ago
> using the simplified example posted previously.
> It is quite interesting:
The problem seems to be that the computation of checkPoint.redo at
xlog.c lines 4162-4169 (all line numbers are per CVS tip) is not
allowing for the possibility that XLogInsert will decide it doesn't
want to split the checkpoint record across XLOG files, and will then
insert a WASTED_SPACE record to avoid that (see comment and following
code at lines 758-795). This wouldn't really matter except that there
is a safety crosscheck at line 4268 that tries to detect unexpected
insertions of other records during a shutdown checkpoint.
I think the code in CreateCheckPoint was correct when it was written,
because we only recently changed XLogInsert to not split records
across files. But it's got a boundary-case bug now, which your test
scenario is able to exercise by making the recovery run try to write
a shutdown checkpoint exactly at the end of a WAL file segment.
The quick and dirty solution would be to dike out the safety check at
4268ff. I don't much care for that, but am too tired right now to work
out a better answer. I'm not real sure whether it's better to adjust
the computation of checkPoint.redo or to smarten the safety check
... but one or the other needs to allow for file-end padding, or maybe
we could hack some update of the state in WasteXLInsertBuffer(). (But
at some point you have to say "this is more trouble than it's worth",
so maybe we'll end up taking out the safety check.)
In any case this isn't a fundamental bug, just an insufficiently
smart safety check. But thanks for finding it! As is, the code has
a nonzero probability of failure in the field :-( and I don't know
how we'd have tracked it down without a reproducible test case.
regards, tom lane
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: PITR COPY Failure (was Point in Time Recovery)
@ 2004-07-20 09:04 Simon Riggs <simon@2ndquadrant.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
2 siblings, 0 replies; 19+ messages in thread
From: Simon Riggs @ 2004-07-20 09:04 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Mark Kirkwood <markir@coretech.co.nz>; pgsql-hackers@postgresql.org
On Tue, 2004-07-20 at 05:14, Tom Lane wrote:
> Mark Kirkwood <markir@coretech.co.nz> writes:
> > I have been doing some re-testing with CVS HEAD from about 1 hour ago
> > using the simplified example posted previously.
>
> > It is quite interesting:
> The problem seems to be that the computation of checkPoint.redo at
> xlog.c lines 4162-4169 (all line numbers are per CVS tip) is not
> allowing for the possibility that XLogInsert will decide it doesn't
> want to split the checkpoint record across XLOG files, and will then
> insert a WASTED_SPACE record to avoid that (see comment and following
> code at lines 758-795). This wouldn't really matter except that there
> is a safety crosscheck at line 4268 that tries to detect unexpected
> insertions of other records during a shutdown checkpoint.
>
> I think the code in CreateCheckPoint was correct when it was written,
> because we only recently changed XLogInsert to not split records
> across files. But it's got a boundary-case bug now, which your test
> scenario is able to exercise by making the recovery run try to write
> a shutdown checkpoint exactly at the end of a WAL file segment.
>
Thanks for locating that, I was suspicious of that piece of code, but it
would have taken me longer than this to locate it exactly.
It was clear (to me) that it had to be of this nature, since I've done a
fair amount of recovery testing and not hit anything like that.
> The quick and dirty solution would be to dike out the safety check at
> 4268ff. I don't much care for that, but am too tired right now to work
> out a better answer. I'm not real sure whether it's better to adjust
> the computation of checkPoint.redo or to smarten the safety check
> ... but one or the other needs to allow for file-end padding, or maybe
> we could hack some update of the state in WasteXLInsertBuffer(). (But
> at some point you have to say "this is more trouble than it's worth",
> so maybe we'll end up taking out the safety check.)
>
I'll take a look
> In any case this isn't a fundamental bug, just an insufficiently
> smart safety check. But thanks for finding it! As is, the code has
> a nonzero probability of failure in the field :-( and I don't know
> how we'd have tracked it down without a reproducible test case.
All code has a non-zero probability of failure in the field, its just
they don't tell you that usually. The main thing here is that we write
everything we need to write to the logs in the first place.
If that is true, then the code can always be adjusted or the logs dumped
and re-spliced to recover data.
Definitely: Thanks Mark! Reproducibility is key.
Best regards, Simon Riggs
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: PITR COPY Failure (was Point in Time Recovery)
@ 2004-07-20 09:45 Mark Kirkwood <markir@coretech.co.nz>
parent: Tom Lane <tgl@sss.pgh.pa.us>
2 siblings, 0 replies; 19+ messages in thread
From: Mark Kirkwood @ 2004-07-20 09:45 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; Simon Riggs <simon@2ndquadrant.com>; +Cc: pgsql-hackers@postgresql.org
Great that it's not fundamental - and hopefully with this discovery, the
probability you mentioned is being squashed towards zero a bit more :-)
Don't let this early bug detract from what is really a superb piece of work!
regards
Mark
Tom Lane wrote:
>In any case this isn't a fundamental bug, just an insufficiently
>smart safety check. But thanks for finding it! As is, the code has
>a nonzero probability of failure in the field :-( and I don't know
>how we'd have tracked it down without a reproducible test case.
>
>
>
>
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: PITR COPY Failure (was Point in Time Recovery)
@ 2004-07-20 11:57 Simon Riggs <simon@2ndquadrant.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
2 siblings, 1 reply; 19+ messages in thread
From: Simon Riggs @ 2004-07-20 11:57 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Mark Kirkwood <markir@coretech.co.nz>; pgsql-hackers@postgresql.org
On Tue, 2004-07-20 at 05:14, Tom Lane wrote:
> Mark Kirkwood <markir@coretech.co.nz> writes:
> > I have been doing some re-testing with CVS HEAD from about 1 hour ago
> > using the simplified example posted previously.
>
> > It is quite interesting:
>
> The problem seems to be that the computation of checkPoint.redo at
> xlog.c lines 4162-4169 (all line numbers are per CVS tip) is not
> allowing for the possibility that XLogInsert will decide it doesn't
> want to split the checkpoint record across XLOG files, and will then
> insert a WASTED_SPACE record to avoid that (see comment and following
> code at lines 758-795). This wouldn't really matter except that there
> is a safety crosscheck at line 4268 that tries to detect unexpected
> insertions of other records during a shutdown checkpoint.
>
> I think the code in CreateCheckPoint was correct when it was written,
> because we only recently changed XLogInsert to not split records
> across files. But it's got a boundary-case bug now, which your test
> scenario is able to exercise by making the recovery run try to write
> a shutdown checkpoint exactly at the end of a WAL file segment.
>
> The quick and dirty solution would be to dike out the safety check at
> 4268ff.
Well, taking out the safety check isn't the answer.
The check produces the last error message "concurrent transaction...",
but it isn't the cause of the mismatch in the first place.
If you take out that check, we still fail because the wasted space at
the end is causing a "record with zero length" error.
> I'm not real sure whether it's better to adjust
> the computation of checkPoint.redo or to smarten the safety check
> ... but one or the other needs to allow for file-end padding, or maybe
> we could hack some update of the state in WasteXLInsertBuffer(). (But
> at some point you have to say "this is more trouble than it's worth",
> so maybe we'll end up taking out the safety check.)
...I'm looking at other options now.
Best Regards, Simon Riggs
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: PITR COPY Failure (was Point in Time Recovery)
@ 2004-07-20 12:51 Tom Lane <tgl@sss.pgh.pa.us>
parent: Simon Riggs <simon@2ndquadrant.com>
0 siblings, 1 reply; 19+ messages in thread
From: Tom Lane @ 2004-07-20 12:51 UTC (permalink / raw)
To: Simon Riggs <simon@2ndquadrant.com>; +Cc: Mark Kirkwood <markir@coretech.co.nz>; pgsql-hackers@postgresql.org
Simon Riggs <simon@2ndquadrant.com> writes:
>> The quick and dirty solution would be to dike out the safety check at
>> 4268ff.
> If you take out that check, we still fail because the wasted space at
> the end is causing a "record with zero length" error.
Ugh. I'm beginning to think we ought to revert the patch that added the
don't-split-across-files logic to XLogInsert; that seems to have broken
more assumptions than I realized. That was added here:
2004-02-11 17:55 tgl
* src/: backend/access/transam/xact.c,
backend/access/transam/xlog.c, backend/access/transam/xlogutils.c,
backend/storage/smgr/md.c, backend/storage/smgr/smgr.c,
bin/pg_controldata/pg_controldata.c,
bin/pg_resetxlog/pg_resetxlog.c, include/access/xact.h,
include/access/xlog.h, include/access/xlogutils.h,
include/pg_config_manual.h, include/catalog/pg_control.h,
include/storage/smgr.h: Commit the reasonably uncontroversial parts
of J.R. Nield's PITR patch, to wit: Add a header record to each WAL
segment file so that it can be reliably identified. Avoid
splitting WAL records across segment files (this is not strictly
necessary, but makes it simpler to incorporate the header records).
Make WAL entries for file creation, deletion, and truncation (as
foreseen but never implemented by Vadim). Also, add support for
making XLOG_SEG_SIZE configurable at compile time, similarly to
BLCKSZ. Fix a couple bugs I introduced in WAL replay during recent
smgr API changes. initdb is forced due to changes in pg_control
contents.
There are other ways to do this, for example we could treat the WAL page
headers as variable-size, and stick the file labeling info into the
first page's header instead of making it be a separate record. The
separate-record way makes it easier to incorporate future additions to
the file labeling info, but I don't really think it's critical to allow
for that.
regards, tom lane
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: PITR COPY Failure (was Point in Time Recovery)
@ 2004-07-20 13:11 Simon Riggs <simon@2ndquadrant.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 2 replies; 19+ messages in thread
From: Simon Riggs @ 2004-07-20 13:11 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Mark Kirkwood <markir@coretech.co.nz>; pgsql-hackers@postgresql.org
On Tue, 2004-07-20 at 13:51, Tom Lane wrote:
> Simon Riggs <simon@2ndquadrant.com> writes:
> >> The quick and dirty solution would be to dike out the safety check at
> >> 4268ff.
>
> > If you take out that check, we still fail because the wasted space at
> > the end is causing a "record with zero length" error.
>
> Ugh. I'm beginning to think we ought to revert the patch that added the
> don't-split-across-files logic to XLogInsert; that seems to have broken
> more assumptions than I realized. That was added here:
>
> 2004-02-11 17:55 tgl
>
> * src/: backend/access/transam/xact.c,
> backend/access/transam/xlog.c, backend/access/transam/xlogutils.c,
> backend/storage/smgr/md.c, backend/storage/smgr/smgr.c,
> bin/pg_controldata/pg_controldata.c,
> bin/pg_resetxlog/pg_resetxlog.c, include/access/xact.h,
> include/access/xlog.h, include/access/xlogutils.h,
> include/pg_config_manual.h, include/catalog/pg_control.h,
> include/storage/smgr.h: Commit the reasonably uncontroversial parts
> of J.R. Nield's PITR patch, to wit: Add a header record to each WAL
> segment file so that it can be reliably identified. Avoid
> splitting WAL records across segment files (this is not strictly
> necessary, but makes it simpler to incorporate the header records).
> Make WAL entries for file creation, deletion, and truncation (as
> foreseen but never implemented by Vadim). Also, add support for
> making XLOG_SEG_SIZE configurable at compile time, similarly to
> BLCKSZ. Fix a couple bugs I introduced in WAL replay during recent
> smgr API changes. initdb is forced due to changes in pg_control
> contents.
>
> There are other ways to do this, for example we could treat the WAL page
> headers as variable-size, and stick the file labeling info into the
> first page's header instead of making it be a separate record. The
> separate-record way makes it easier to incorporate future additions to
> the file labeling info, but I don't really think it's critical to allow
> for that.
>
I think I've fixed it now...but wait 20
The problem was that a zero length XLOG_WASTED_SPACE record just fell
out of ReadRecord when it shouldn't have. By giving it a helping hand it
makes it through with pointers correctly set, and everything else was
already thought of in the earlier patch, so xlog_redo etc happens.
I'll update again in a few minutes....no point us both looking at this.
Best regards, Simon Riggs
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: PITR COPY Failure (was Point in Time Recovery)
@ 2004-07-20 14:00 Tom Lane <tgl@sss.pgh.pa.us>
parent: Simon Riggs <simon@2ndquadrant.com>
1 sibling, 1 reply; 19+ messages in thread
From: Tom Lane @ 2004-07-20 14:00 UTC (permalink / raw)
To: Simon Riggs <simon@2ndquadrant.com>; +Cc: Mark Kirkwood <markir@coretech.co.nz>; pgsql-hackers@postgresql.org
Simon Riggs <simon@2ndquadrant.com> writes:
> On Tue, 2004-07-20 at 13:51, Tom Lane wrote:
>> Ugh. I'm beginning to think we ought to revert the patch that added the
>> don't-split-across-files logic to XLogInsert; that seems to have broken
>> more assumptions than I realized.
> The problem was that a zero length XLOG_WASTED_SPACE record just fell
> out of ReadRecord when it shouldn't have. By giving it a helping hand it
> makes it through with pointers correctly set, and everything else was
> already thought of in the earlier patch, so xlog_redo etc happens.
Yeah, but the WASTED_SPACE/FILE_HEADER stuff is already pretty ugly, and
adding two more warts to the code to support it is sticking in my craw.
I'm thinking it would be cleaner to treat the extra labeling information
as an extension of the WAL page header.
regards, tom lane
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: PITR COPY Failure (was Point in Time Recovery)
@ 2004-07-20 14:19 Simon Riggs <simon@2ndquadrant.com>
parent: Simon Riggs <simon@2ndquadrant.com>
1 sibling, 2 replies; 19+ messages in thread
From: Simon Riggs @ 2004-07-20 14:19 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Mark Kirkwood <markir@coretech.co.nz>; pgsql-hackers@postgresql.org
On Tue, 2004-07-20 at 14:11, Simon Riggs wrote:
> On Tue, 2004-07-20 at 13:51, Tom Lane wrote:
> > Simon Riggs <simon@2ndquadrant.com> writes:
> > >> The quick and dirty solution would be to dike out the safety check at
> > >> 4268ff.
> >
> > > If you take out that check, we still fail because the wasted space at
> > > the end is causing a "record with zero length" error.
> >
> > Ugh. I'm beginning to think we ought to revert the patch that added the
> > don't-split-across-files logic to XLogInsert; that seems to have broken
> > more assumptions than I realized. That was added here:
> >
> > 2004-02-11 17:55 tgl
> >
> > * src/: backend/access/transam/xact.c,
> > backend/access/transam/xlog.c, backend/access/transam/xlogutils.c,
> > backend/storage/smgr/md.c, backend/storage/smgr/smgr.c,
> > bin/pg_controldata/pg_controldata.c,
> > bin/pg_resetxlog/pg_resetxlog.c, include/access/xact.h,
> > include/access/xlog.h, include/access/xlogutils.h,
> > include/pg_config_manual.h, include/catalog/pg_control.h,
> > include/storage/smgr.h: Commit the reasonably uncontroversial parts
> > of J.R. Nield's PITR patch, to wit: Add a header record to each WAL
> > segment file so that it can be reliably identified. Avoid
> > splitting WAL records across segment files (this is not strictly
> > necessary, but makes it simpler to incorporate the header records).
> > Make WAL entries for file creation, deletion, and truncation (as
> > foreseen but never implemented by Vadim). Also, add support for
> > making XLOG_SEG_SIZE configurable at compile time, similarly to
> > BLCKSZ. Fix a couple bugs I introduced in WAL replay during recent
> > smgr API changes. initdb is forced due to changes in pg_control
> > contents.
> >
> > There are other ways to do this, for example we could treat the WAL page
> > headers as variable-size, and stick the file labeling info into the
> > first page's header instead of making it be a separate record. The
> > separate-record way makes it easier to incorporate future additions to
> > the file labeling info, but I don't really think it's critical to allow
> > for that.
> >
>
> I think I've fixed it now...but wait 20
>
> The problem was that a zero length XLOG_WASTED_SPACE record just fell
> out of ReadRecord when it shouldn't have. By giving it a helping hand it
> makes it through with pointers correctly set, and everything else was
> already thought of in the earlier patch, so xlog_redo etc happens.
>
> I'll update again in a few minutes....no point us both looking at this.
>
This was a very confusing test...Here's what I think happened:
Mark discovered a numerological coincidence that meant that the
XLOG_WASTED_SPACE record was zero length at the end of EACH file he was
writing to, as long as there was just that one writer. So no matter how
many records were inserted, each xlog file had a zero length
XLOG_WASTED_SPACE record at its end.
ReadRecord failed on seeing a zero length record, i.e. when it got to
the FIRST of the XLOG_WASTED_SPACE records. Thats why the test fails no
matter how many records you give it, as long as it was more than enough
to write into a second xlog segment file.
By telling ReadRecord that XLOG_WASTED_SPACE records of zero length are
in fact *OK*, it continues happily. (Thats just a partial fix, see
later)
The test works, but gives what looks like strange results: the test
blows away the data directory completely, so the then-current xlog dies
too. That contained the commit for the large COPY, so even though the
recovery now works, the table has zero rows in it. (When things die
you're still likely to lose *some* data).
Anyway, so then we put the "concurrent transaction" test back in and the
test passes because we have now set the pointers correctly.
After all that, I think the wasted space idea is still sensible. You
musn't have a continuation record across files, otherwise we'll end up
with half a commit one-day, which would break ACID.
I'm happy that we have the explicit test in XLogInsert for zero-length
records. Somebody will one-day write a resource manager with zero length
records when they didn't mean to and we need to catch that at write
time, not at recovery time like Mark has done. The WasteXLInsertBuffer
was the only part of the code that *can* write a zero-length record, so
we will *not* see another recurrence of this situation --at recovery
time--.
Though further concerns along this theme are:
- what happens when the space at the end of a file is so small we can't
even write a zero-length XLOG_WASTED_SPACE record? Hopefully, you're
gonna say "damn your eyes...couldnt you see that, its already there".
- if the space at the end of a file was just zeros, then the "concurrent
transaction test" would still fail....we probably need to enhance this
to treat a few zeros at end of file AS IF it was an XLOG_WASTED_SPACE
record an continue. (That scenario would happen if we were doing a
recovery that included a local un-archived xlog that was very close to
being full - probably more likely to occur in crash recovery than
archive recovery)
The included patch doesn't attempt to address those issues, yet.
Best regards, Simon Riggs
Attachments:
zerolength.patch (text/x-patch, 1.4K) download
Index: backend/access/transam/xlog.c
===================================================================
RCS file: /projects/cvsroot/pgsql-server/src/backend/access/transam/xlog.c,v
retrieving revision 1.149
diff -d -c -r1.149 xlog.c
*** backend/access/transam/xlog.c 19 Jul 2004 14:34:39 -0000 1.149
--- backend/access/transam/xlog.c 20 Jul 2004 14:03:12 -0000
***************
*** 2370,2376 ****
* Currently, xl_len == 0 must be bad data, but that might not be true
* forever. See note in XLogInsert.
*/
! if (record->xl_len == 0)
{
ereport(emode,
(errmsg("record with zero length at %X/%X",
--- 2370,2380 ----
* Currently, xl_len == 0 must be bad data, but that might not be true
* forever. See note in XLogInsert.
*/
! if (record->xl_len == 0 && record->xl_info != XLOG_WASTED_SPACE)
{
ereport(emode,
(errmsg("record with zero length at %X/%X",
***************
*** 2489,2495 ****
}
/* Record does not cross a page boundary */
! if (!RecordIsValid(record, *RecPtr, emode))
goto next_record_is_invalid;
if (BLCKSZ - SizeOfXLogRecord >= RecPtr->xrecoff % BLCKSZ +
MAXALIGN(total_len))
--- 2493,2499 ----
}
/* Record does not cross a page boundary */
! if (!RecordIsValid(record, *RecPtr, emode) && record->xl_info != XLOG_WASTED_SPACE)
goto next_record_is_invalid;
if (BLCKSZ - SizeOfXLogRecord >= RecPtr->xrecoff % BLCKSZ +
MAXALIGN(total_len))
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: PITR COPY Failure (was Point in Time Recovery)
@ 2004-07-20 14:24 Simon Riggs <simon@2ndquadrant.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 19+ messages in thread
From: Simon Riggs @ 2004-07-20 14:24 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Mark Kirkwood <markir@coretech.co.nz>; pgsql-hackers@postgresql.org
On Tue, 2004-07-20 at 15:00, Tom Lane wrote:
> Simon Riggs <simon@2ndquadrant.com> writes:
> > On Tue, 2004-07-20 at 13:51, Tom Lane wrote:
> >> Ugh. I'm beginning to think we ought to revert the patch that added the
> >> don't-split-across-files logic to XLogInsert; that seems to have broken
> >> more assumptions than I realized.
>
> > The problem was that a zero length XLOG_WASTED_SPACE record just fell
> > out of ReadRecord when it shouldn't have. By giving it a helping hand it
> > makes it through with pointers correctly set, and everything else was
> > already thought of in the earlier patch, so xlog_redo etc happens.
>
> Yeah, but the WASTED_SPACE/FILE_HEADER stuff is already pretty ugly, and
> adding two more warts to the code to support it is sticking in my craw.
> I'm thinking it would be cleaner to treat the extra labeling information
> as an extension of the WAL page header.
Sounds like a better solution than scrabbling around at the end of file
with too many edge cases to test properly
...over to you then...
Best Regards, Simon Riggs
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: PITR COPY Failure (was Point in Time Recovery)
@ 2004-07-21 02:27 Mark Kirkwood <markir@coretech.co.nz>
parent: Simon Riggs <simon@2ndquadrant.com>
1 sibling, 0 replies; 19+ messages in thread
From: Mark Kirkwood @ 2004-07-21 02:27 UTC (permalink / raw)
To: Simon Riggs <simon@2ndquadrant.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers@postgresql.org
FYI - I can confirm that the patch fixes main issue.
Simon Riggs wrote:
>
>This was a very confusing test...Here's what I think happened:
>.....
>The included patch doesn't attempt to address those issues, yet.
>
>Best regards, Simon Riggs
>
>
>
>
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: PITR COPY Failure (was Point in Time Recovery)
@ 2004-07-21 02:39 Mark Kirkwood <markir@coretech.co.nz>
parent: Simon Riggs <simon@2ndquadrant.com>
1 sibling, 0 replies; 19+ messages in thread
From: Mark Kirkwood @ 2004-07-21 02:39 UTC (permalink / raw)
To: Simon Riggs <simon@2ndquadrant.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers@postgresql.org
This is presumably a standard feature of any PITR design - if the
failure event destroys the current transaction log, then you can only
recover transactions that committed in the last *archived* log.
regards
Mark
Simon Riggs wrote:
>
>The test works, but gives what looks like strange results: the test
>blows away the data directory completely, so the then-current xlog dies
>too. That contained the commit for the large COPY, so even though the
>recovery now works, the table has zero rows in it. (When things die
>you're still likely to lose *some* data).
>
>
>
>
>
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: PITR COPY Failure (was Point in Time Recovery)
@ 2004-07-21 22:43 Tom Lane <tgl@sss.pgh.pa.us>
parent: Simon Riggs <simon@2ndquadrant.com>
0 siblings, 1 reply; 19+ messages in thread
From: Tom Lane @ 2004-07-21 22:43 UTC (permalink / raw)
To: Simon Riggs <simon@2ndquadrant.com>; +Cc: Mark Kirkwood <markir@coretech.co.nz>; pgsql-hackers@postgresql.org
Simon Riggs <simon@2ndquadrant.com> writes:
> On Tue, 2004-07-20 at 15:00, Tom Lane wrote:
>> Yeah, but the WASTED_SPACE/FILE_HEADER stuff is already pretty ugly, and
>> adding two more warts to the code to support it is sticking in my craw.
>> I'm thinking it would be cleaner to treat the extra labeling information
>> as an extension of the WAL page header.
> Sounds like a better solution than scrabbling around at the end of file
> with too many edge cases to test properly
This is done in CVS tip. Mark, could you retest to verify it's fixed?
regards, tom lane
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: PITR COPY Failure (was Point in Time Recovery)
@ 2004-07-22 00:39 Mark Kirkwood <markir@coretech.co.nz>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 19+ messages in thread
From: Mark Kirkwood @ 2004-07-22 00:39 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Simon Riggs <simon@2ndquadrant.com>; pgsql-hackers@postgresql.org
Looks good to me. Log file numbering scheme seems to have changed - is
that part of the fix too?.
Tom Lane wrote:
>
>This is done in CVS tip. Mark, could you retest to verify it's fixed?
>
> regards, tom lane
>
>
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: PITR COPY Failure (was Point in Time Recovery)
@ 2004-07-22 00:43 Tom Lane <tgl@sss.pgh.pa.us>
parent: Mark Kirkwood <markir@coretech.co.nz>
0 siblings, 0 replies; 19+ messages in thread
From: Tom Lane @ 2004-07-22 00:43 UTC (permalink / raw)
To: Mark Kirkwood <markir@coretech.co.nz>; +Cc: Simon Riggs <simon@2ndquadrant.com>; pgsql-hackers@postgresql.org
Mark Kirkwood <markir@coretech.co.nz> writes:
> Looks good to me. Log file numbering scheme seems to have changed - is
> that part of the fix too?.
That's for timelines ... it's not directly related but I thought I
should put in both changes at once to avoid forcing an extra initdb.
regards, tom lane
^ permalink raw reply [nested|flat] 19+ messages in thread
end of thread, other threads:[~2004-07-22 00:43 UTC | newest]
Thread overview: 19+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2004-07-18 05:50 PITR COPY Failure (was Point in Time Recovery) Mark Kirkwood <markir@coretech.co.nz>
2004-07-18 06:15 ` Tom Lane <tgl@sss.pgh.pa.us>
2004-07-18 06:48 ` Mark Kirkwood <markir@coretech.co.nz>
2004-07-18 08:05 ` Mark Kirkwood <markir@coretech.co.nz>
2004-07-20 00:48 ` Mark Kirkwood <markir@coretech.co.nz>
2004-07-20 04:14 ` Tom Lane <tgl@sss.pgh.pa.us>
2004-07-20 09:04 ` Simon Riggs <simon@2ndquadrant.com>
2004-07-20 09:45 ` Mark Kirkwood <markir@coretech.co.nz>
2004-07-20 11:57 ` Simon Riggs <simon@2ndquadrant.com>
2004-07-20 12:51 ` Tom Lane <tgl@sss.pgh.pa.us>
2004-07-20 13:11 ` Simon Riggs <simon@2ndquadrant.com>
2004-07-20 14:00 ` Tom Lane <tgl@sss.pgh.pa.us>
2004-07-20 14:24 ` Simon Riggs <simon@2ndquadrant.com>
2004-07-21 22:43 ` Tom Lane <tgl@sss.pgh.pa.us>
2004-07-22 00:39 ` Mark Kirkwood <markir@coretech.co.nz>
2004-07-22 00:43 ` Tom Lane <tgl@sss.pgh.pa.us>
2004-07-20 14:19 ` Simon Riggs <simon@2ndquadrant.com>
2004-07-21 02:27 ` Mark Kirkwood <markir@coretech.co.nz>
2004-07-21 02:39 ` Mark Kirkwood <markir@coretech.co.nz>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox