pg.ddx.io pgsql-docs@postgresql.org mailing list archive
help / color / mirror / Atom feedpg_total_relation_size() and CHECKPOINT
24+ messages / 6 participants
[nested] [flat]
* pg_total_relation_size() and CHECKPOINT
@ 2008-03-13 19:49 Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-13 20:20 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 24+ messages in thread
From: Zubkovsky, Sergey @ 2008-03-13 19:49 UTC (permalink / raw)
To: pgsql-docs
Hello,
I've detected that a result value of pg_total_relation_size() for an
actively updated table might be significantly differ from a result that
is returned after explicit CHECKPOINT command execution.
select version();
"PostgreSQL 8.2.6 on i686-pc-mingw32, compiled by GCC gcc.exe (GCC)
3.4.2 (mingw-special)"
I understand the reasons of such behavior: cache buffers must be flushed
in order to be sure that pg_total_relation_size() result will be like we
expect.
Is this right?
If so I think it would be very useful to add a description of this
peculiarity to the PG documentation.
Thanks in advance,
Zubkovsky Sergey
^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: pg_total_relation_size() and CHECKPOINT
2008-03-13 19:49 pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
@ 2008-03-13 20:20 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 15:23 ` Re: pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
0 siblings, 1 reply; 24+ messages in thread
From: Tom Lane @ 2008-03-13 20:20 UTC (permalink / raw)
To: Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>; +Cc: pgsql-docs
"Zubkovsky, Sergey" <Sergey.Zubkovsky@transas.com> writes:
> I've detected that a result value of pg_total_relation_size() for an
> actively updated table might be significantly differ from a result that
> is returned after explicit CHECKPOINT command execution.
Uh, can you show a specific example of what you mean?
> I understand the reasons of such behavior: cache buffers must be flushed
> in order to be sure that pg_total_relation_size() result will be like we
> expect.
I wouldn't think so. The space for a page is allocated immediately when
needed --- its *contents* might not be up to date, but that shouldn't
affect pg_total_relation_size.
regards, tom lane
^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: pg_total_relation_size() and CHECKPOINT
2008-03-13 19:49 pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-13 20:20 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-03-14 15:23 ` Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 16:19 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 24+ messages in thread
From: Zubkovsky, Sergey @ 2008-03-14 15:23 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-docs
Hi,
Here is my example.
We are creating 2 tables:
create table t1 ( a int, b varchar( 30 ) );
create table t1_arh ( c text );
and filling 't1':
insert into t1 select generate_series(1, 100000 ), generate_series(1,
100000 );
The "arch_table_sp" user-function will be used for extracting data from
't1' and archiving it to 't1_arh'.
CREATE OR REPLACE FUNCTION "arch_table_sp" ( tblName name, arcTblName
name )
RETURNS void
AS $BODY$
DECLARE
fn text;
chunk CONSTANT bigint := 512*1024;
off bigint := 0;
rdBytes bigint;
buf text;
BEGIN
SELECT setting INTO STRICT fn FROM pg_settings WHERE name =
'data_directory';
fn := fn || '/tbldata.txt';
PERFORM pg_file_unlink( fn );
EXECUTE 'COPY ( SELECT * FROM ' || quote_ident( tblName ) || ' ) TO
' || quote_literal( fn );
EXECUTE '
CREATE OR REPLACE FUNCTION "__InsertChunk__sp" ( data text )
RETURNS void AS
$_$
INSERT INTO ' || quote_ident( arcTblName ) || ' ( c ) VALUES ( $1 );
$_$
LANGUAGE sql;';
LOOP
buf := pg_file_read( fn, off, chunk );
rdBytes := length( buf );
IF ( rdBytes > 0 ) THEN
PERFORM "__InsertChunk__sp"( buf );
off := off + rdBytes;
END IF;
EXIT WHEN ( rdBytes <> chunk );
END LOOP;
PERFORM pg_file_unlink( fn );
END;
$BODY$ LANGUAGE plpgsql;
Now we are executing the following statements in one transaction:
select "arch_table_sp"( 't1', 't1_arh' );
select pg_total_relation_size( 't1_arh' );
The result is 417792 (in the general case it may be another value, for
example, I received 303104, 573440 and etc).
If we are executing these statements in separate transactions with a
couple of seconds between them than we have received another value:
truncate table t1_arh;
select "arch_table_sp"( 't1', 't1_arh' );
select pg_total_relation_size( 't1_arh' );
The result is 688128!
With explicit CHECKPOINT we will have one more value:
truncate table t1_arh;
select "arch_table_sp"( 't1', 't1_arh' );
CHECKPOINT;
select pg_total_relation_size( 't1_arh' );
The result is 696320!
It would be interesting why we have such results...
It's obviously that CHECKPOINT is not a good decision.
Can you suggest some other approach instead of explicit CHECKPOINT?
Sorry for my English.
I hope this example is quite clear.
Thanks in advance,
Zubkovsky Sergey
-----Original Message-----
From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
Sent: Thursday, March 13, 2008 11:21 PM
To: Zubkovsky, Sergey
Cc: pgsql-docs@postgresql.org
Subject: Re: [DOCS] pg_total_relation_size() and CHECKPOINT
"Zubkovsky, Sergey" <Sergey.Zubkovsky@transas.com> writes:
> I've detected that a result value of pg_total_relation_size() for an
> actively updated table might be significantly differ from a result
that
> is returned after explicit CHECKPOINT command execution.
Uh, can you show a specific example of what you mean?
> I understand the reasons of such behavior: cache buffers must be
flushed
> in order to be sure that pg_total_relation_size() result will be like
we
> expect.
I wouldn't think so. The space for a page is allocated immediately when
needed --- its *contents* might not be up to date, but that shouldn't
affect pg_total_relation_size.
regards, tom lane
^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: pg_total_relation_size() and CHECKPOINT
2008-03-13 19:49 pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-13 20:20 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 15:23 ` Re: pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
@ 2008-03-14 16:19 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 16:33 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
0 siblings, 1 reply; 24+ messages in thread
From: Tom Lane @ 2008-03-14 16:19 UTC (permalink / raw)
To: Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>; +Cc: pgsql-docs; pgsql-hackers@postgresql.org
[ moved to -hackers --- see original thread here
http://archives.postgresql.org/pgsql-docs/2008-03/msg00039.php
]
"Zubkovsky, Sergey" <Sergey.Zubkovsky@transas.com> writes:
> Here is my example.
Hmm ... on my Fedora machine I get the same result (704512) in
all these cases, which is what I'd expect. (The exact value
could vary across platforms, of course.)
You said you were using the MinGW build --- maybe MinGW's version
of stat(2) isn't trustworthy?
regards, tom lane
^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: [DOCS] pg_total_relation_size() and CHECKPOINT
2008-03-13 19:49 pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-13 20:20 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 15:23 ` Re: pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 16:19 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-03-14 16:33 ` Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 17:21 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 24+ messages in thread
From: Zubkovsky, Sergey @ 2008-03-14 16:33 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-hackers@postgresql.org
The previous results were received on PG 8.3 version:
"PostgreSQL 8.3.0, compiled by Visual C++ build 1400"
-----Original Message-----
From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
Sent: Friday, March 14, 2008 7:19 PM
To: Zubkovsky, Sergey
Cc: pgsql-docs@postgresql.org; pgsql-hackers@postgresql.org
Subject: Re: [DOCS] pg_total_relation_size() and CHECKPOINT
[ moved to -hackers --- see original thread here
http://archives.postgresql.org/pgsql-docs/2008-03/msg00039.php
]
"Zubkovsky, Sergey" <Sergey.Zubkovsky@transas.com> writes:
> Here is my example.
Hmm ... on my Fedora machine I get the same result (704512) in
all these cases, which is what I'd expect. (The exact value
could vary across platforms, of course.)
You said you were using the MinGW build --- maybe MinGW's version
of stat(2) isn't trustworthy?
regards, tom lane
^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: [DOCS] pg_total_relation_size() and CHECKPOINT
2008-03-13 19:49 pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-13 20:20 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 15:23 ` Re: pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 16:19 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 16:33 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
@ 2008-03-14 17:21 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 18:30 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Gregory Stark <stark@enterprisedb.com>
0 siblings, 1 reply; 24+ messages in thread
From: Tom Lane @ 2008-03-14 17:21 UTC (permalink / raw)
To: Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>; +Cc: pgsql-hackers@postgresql.org
"Zubkovsky, Sergey" <Sergey.Zubkovsky@transas.com> writes:
> The previous results were received on PG 8.3 version:
> "PostgreSQL 8.3.0, compiled by Visual C++ build 1400"
Hmm. I find the whole thing fairly worrisome, because what it suggests
is that Windows isn't actually allocating file space during smgrextend,
which would mean that we'd be prone to running out of disk space at
unfortunate times --- like during a checkpoint, after we've already
promised the client the data is committed.
Can any Windows hackers look into this and find out what's really
happening?
regards, tom lane
^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: [DOCS] pg_total_relation_size() and CHECKPOINT
2008-03-13 19:49 pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-13 20:20 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 15:23 ` Re: pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 16:19 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 16:33 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 17:21 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-03-14 18:30 ` Gregory Stark <stark@enterprisedb.com>
2008-03-14 20:02 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 24+ messages in thread
From: Gregory Stark @ 2008-03-14 18:30 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Zubkovsky\, Sergey <Sergey.Zubkovsky@transas.com>; pgsql-hackers@postgresql.org
"Tom Lane" <tgl@sss.pgh.pa.us> writes:
> "Zubkovsky, Sergey" <Sergey.Zubkovsky@transas.com> writes:
>> The previous results were received on PG 8.3 version:
>> "PostgreSQL 8.3.0, compiled by Visual C++ build 1400"
>
> Hmm. I find the whole thing fairly worrisome, because what it suggests
> is that Windows isn't actually allocating file space during smgrextend,
> which would mean that we'd be prone to running out of disk space at
> unfortunate times --- like during a checkpoint, after we've already
> promised the client the data is committed.
Surely we can't lose after the fsync? Losing at commit rather than at the time
of insert might still be poor, but how could we lose after we've promised the
data is committed?
--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's Slony Replication support!
^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: [DOCS] pg_total_relation_size() and CHECKPOINT
2008-03-13 19:49 pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-13 20:20 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 15:23 ` Re: pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 16:19 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 16:33 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 17:21 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 18:30 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Gregory Stark <stark@enterprisedb.com>
@ 2008-03-14 20:02 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-03-26 13:22 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Alvaro Herrera <alvherre@commandprompt.com>
0 siblings, 1 reply; 24+ messages in thread
From: Tom Lane @ 2008-03-14 20:02 UTC (permalink / raw)
To: Gregory Stark <stark@enterprisedb.com>; +Cc: Zubkovsky\, Sergey <Sergey.Zubkovsky@transas.com>; pgsql-hackers@postgresql.org
Gregory Stark <stark@enterprisedb.com> writes:
> "Tom Lane" <tgl@sss.pgh.pa.us> writes:
>> Hmm. I find the whole thing fairly worrisome, because what it suggests
>> is that Windows isn't actually allocating file space during smgrextend,
>> which would mean that we'd be prone to running out of disk space at
>> unfortunate times --- like during a checkpoint, after we've already
>> promised the client the data is committed.
> Surely we can't lose after the fsync? Losing at commit rather than at
> the time of insert might still be poor, but how could we lose after
> we've promised the data is committed?
What I'm afraid of is write() returning ENOSPC for a write to a disk
block we thought we had allocated previously. If such a situation is
persistent we'd be unable to flush dirty data from shared buffers and
thus never be able to complete a checkpoint. We'd never *get* to the
fsync, so whether the data is safe after fsync is moot.
The way it is supposed to work is that ENOSPC ought to happen during
smgrextend, that is before we've put any data into a shared buffer
corresponding to a new page of the file. With that, we will never be
able to commit a transaction that requires disk space we don't have.
The real question here is whether Windows' stat() is telling the truth
about how much filesystem space has actually been allocated to a file.
It seems entirely possible that it's not; but if it is, then I think we
have a problem.
regards, tom lane
^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: [DOCS] pg_total_relation_size() and CHECKPOINT
2008-03-13 19:49 pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-13 20:20 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 15:23 ` Re: pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 16:19 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 16:33 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 17:21 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 18:30 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Gregory Stark <stark@enterprisedb.com>
2008-03-14 20:02 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-03-26 13:22 ` Alvaro Herrera <alvherre@commandprompt.com>
2008-03-26 13:56 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
0 siblings, 1 reply; 24+ messages in thread
From: Alvaro Herrera @ 2008-03-26 13:22 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Gregory Stark <stark@enterprisedb.com>; Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>; pgsql-hackers@postgresql.org, Magnus Hagander <magnus@hagander.net>
Tom Lane wrote:
> The real question here is whether Windows' stat() is telling the truth
> about how much filesystem space has actually been allocated to a file.
> It seems entirely possible that it's not; but if it is, then I think we
> have a problem.
Has this been examined by a Windows hacker?
--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.
^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: [DOCS] pg_total_relation_size() and CHECKPOINT
2008-03-13 19:49 pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-13 20:20 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 15:23 ` Re: pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 16:19 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 16:33 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 17:21 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 18:30 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Gregory Stark <stark@enterprisedb.com>
2008-03-14 20:02 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-26 13:22 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Alvaro Herrera <alvherre@commandprompt.com>
@ 2008-03-26 13:56 ` Andrew Dunstan <andrew@dunslane.net>
2008-03-26 14:44 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-26 15:41 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 2 replies; 24+ messages in thread
From: Andrew Dunstan @ 2008-03-26 13:56 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@commandprompt.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Gregory Stark <stark@enterprisedb.com>; Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>; pgsql-hackers@postgresql.org, Magnus Hagander <magnus@hagander.net>
Alvaro Herrera wrote:
> Tom Lane wrote:
>
>
>> The real question here is whether Windows' stat() is telling the truth
>> about how much filesystem space has actually been allocated to a file.
>> It seems entirely possible that it's not; but if it is, then I think we
>> have a problem.
>>
>
> Has this been examined by a Windows hacker?
>
>
If someone can suggest a test program I'll be happy to run it.
cheers
andrew
^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: [DOCS] pg_total_relation_size() and CHECKPOINT
2008-03-13 19:49 pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-13 20:20 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 15:23 ` Re: pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 16:19 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 16:33 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 17:21 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 18:30 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Gregory Stark <stark@enterprisedb.com>
2008-03-14 20:02 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-26 13:22 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Alvaro Herrera <alvherre@commandprompt.com>
2008-03-26 13:56 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
@ 2008-03-26 14:44 ` Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
1 sibling, 0 replies; 24+ messages in thread
From: Zubkovsky, Sergey @ 2008-03-26 14:44 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; Alvaro Herrera <alvherre@commandprompt.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Gregory Stark <stark@enterprisedb.com>; pgsql-hackers@postgresql.org; Magnus Hagander <magnus@hagander.net>
Can anybody tell me how filesystem space is allocated and point me to
the sources if it's possible?
I have some experience with programming for Windows and I'll try to
investigate this problem.
-----Original Message-----
From: Andrew Dunstan [mailto:andrew@dunslane.net]
Sent: Wednesday, March 26, 2008 4:56 PM
To: Alvaro Herrera
Cc: Tom Lane; Gregory Stark; Zubkovsky, Sergey;
pgsql-hackers@postgresql.org; Magnus Hagander
Subject: Re: [HACKERS] [DOCS] pg_total_relation_size() and CHECKPOINT
Alvaro Herrera wrote:
> Tom Lane wrote:
>
>
>> The real question here is whether Windows' stat() is telling the
truth
>> about how much filesystem space has actually been allocated to a
file.
>> It seems entirely possible that it's not; but if it is, then I think
we
>> have a problem.
>>
>
> Has this been examined by a Windows hacker?
>
>
If someone can suggest a test program I'll be happy to run it.
cheers
andrew
^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: [DOCS] pg_total_relation_size() and CHECKPOINT
2008-03-13 19:49 pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-13 20:20 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 15:23 ` Re: pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 16:19 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 16:33 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 17:21 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 18:30 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Gregory Stark <stark@enterprisedb.com>
2008-03-14 20:02 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-26 13:22 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Alvaro Herrera <alvherre@commandprompt.com>
2008-03-26 13:56 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
@ 2008-03-26 15:41 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 00:03 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
1 sibling, 1 reply; 24+ messages in thread
From: Tom Lane @ 2008-03-26 15:41 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: Alvaro Herrera <alvherre@commandprompt.com>; Gregory Stark <stark@enterprisedb.com>; Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>; pgsql-hackers@postgresql.org, Magnus Hagander <magnus@hagander.net>
Andrew Dunstan <andrew@dunslane.net> writes:
> Alvaro Herrera wrote:
>> Tom Lane wrote:
>>> The real question here is whether Windows' stat() is telling the truth
>>> about how much filesystem space has actually been allocated to a file.
>>> It seems entirely possible that it's not; but if it is, then I think we
>>> have a problem.
>> Has this been examined by a Windows hacker?
> If someone can suggest a test program I'll be happy to run it.
One thing that would be good is just to see who else can reproduce
the original observation:
http://archives.postgresql.org/pgsql-docs/2008-03/msg00041.php
It might occur only on some versions of Windows, for instance.
regards, tom lane
^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: [DOCS] pg_total_relation_size() and CHECKPOINT
2008-03-13 19:49 pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-13 20:20 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 15:23 ` Re: pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 16:19 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 16:33 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 17:21 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 18:30 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Gregory Stark <stark@enterprisedb.com>
2008-03-14 20:02 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-26 13:22 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Alvaro Herrera <alvherre@commandprompt.com>
2008-03-26 13:56 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-26 15:41 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-03-27 00:03 ` Andrew Dunstan <andrew@dunslane.net>
2008-03-27 01:08 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 24+ messages in thread
From: Andrew Dunstan @ 2008-03-27 00:03 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Alvaro Herrera <alvherre@commandprompt.com>; Gregory Stark <stark@enterprisedb.com>; Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>; pgsql-hackers@postgresql.org, Magnus Hagander <magnus@hagander.net>
Tom Lane wrote:
> Andrew Dunstan <andrew@dunslane.net> writes:
>
>> Alvaro Herrera wrote:
>>
>>> Tom Lane wrote:
>>>
>>>> The real question here is whether Windows' stat() is telling the truth
>>>> about how much filesystem space has actually been allocated to a file.
>>>> It seems entirely possible that it's not; but if it is, then I think we
>>>> have a problem.
>>>>
>
>
>>> Has this been examined by a Windows hacker?
>>>
>
>
>> If someone can suggest a test program I'll be happy to run it.
>>
>
> One thing that would be good is just to see who else can reproduce
> the original observation:
> http://archives.postgresql.org/pgsql-docs/2008-03/msg00041.php
>
> It might occur only on some versions of Windows, for instance.
>
>
>
I have reproduced it in XP-Pro/SP2 running in a VMWare machine on an FC6
host.
cheers
andrew
^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: [DOCS] pg_total_relation_size() and CHECKPOINT
2008-03-13 19:49 pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-13 20:20 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 15:23 ` Re: pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 16:19 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 16:33 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 17:21 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 18:30 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Gregory Stark <stark@enterprisedb.com>
2008-03-14 20:02 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-26 13:22 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Alvaro Herrera <alvherre@commandprompt.com>
2008-03-26 13:56 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-26 15:41 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 00:03 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
@ 2008-03-27 01:08 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 02:37 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-27 03:25 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
0 siblings, 2 replies; 24+ messages in thread
From: Tom Lane @ 2008-03-27 01:08 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: Alvaro Herrera <alvherre@commandprompt.com>; Gregory Stark <stark@enterprisedb.com>; Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>; pgsql-hackers@postgresql.org, Magnus Hagander <magnus@hagander.net>
Andrew Dunstan <andrew@dunslane.net> writes:
> Tom Lane wrote:
>>> The real question here is whether Windows' stat() is telling the truth
>>> about how much filesystem space has actually been allocated to a file.
>>
>> One thing that would be good is just to see who else can reproduce
>> the original observation:
>> http://archives.postgresql.org/pgsql-docs/2008-03/msg00041.php
> I have reproduced it in XP-Pro/SP2 running in a VMWare machine on an FC6
> host.
OK, so the next question is do we really have an issue, or is this just
an observational artifact? What I'd try is deliberately running the
machine out of disk space with a long series of inserts, and then see
whether subsequent checkpoint attempts fail due to ENOSPC errors while
trying to write out dirty buffers.
To avoid conflating this effect with anything else, it'd be best if you
could put the DB on its own small partition, and *not* put pg_xlog
there.
regards, tom lane
^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: [DOCS] pg_total_relation_size() and CHECKPOINT
2008-03-13 19:49 pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-13 20:20 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 15:23 ` Re: pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 16:19 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 16:33 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 17:21 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 18:30 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Gregory Stark <stark@enterprisedb.com>
2008-03-14 20:02 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-26 13:22 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Alvaro Herrera <alvherre@commandprompt.com>
2008-03-26 13:56 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-26 15:41 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 00:03 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-27 01:08 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-03-27 02:37 ` Andrew Dunstan <andrew@dunslane.net>
2008-03-27 11:14 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Alvaro Herrera <alvherre@commandprompt.com>
1 sibling, 1 reply; 24+ messages in thread
From: Andrew Dunstan @ 2008-03-27 02:37 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Alvaro Herrera <alvherre@commandprompt.com>; Gregory Stark <stark@enterprisedb.com>; Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>; pgsql-hackers@postgresql.org, Magnus Hagander <magnus@hagander.net>
Tom Lane wrote:
> Andrew Dunstan <andrew@dunslane.net> writes:
>
>> Tom Lane wrote:
>>
>>>> The real question here is whether Windows' stat() is telling the truth
>>>> about how much filesystem space has actually been allocated to a file.
>>>>
>>> One thing that would be good is just to see who else can reproduce
>>> the original observation:
>>> http://archives.postgresql.org/pgsql-docs/2008-03/msg00041.php
>>>
>
>
>> I have reproduced it in XP-Pro/SP2 running in a VMWare machine on an FC6
>> host.
>>
>
> OK, so the next question is do we really have an issue, or is this just
> an observational artifact? What I'd try is deliberately running the
> machine out of disk space with a long series of inserts, and then see
> whether subsequent checkpoint attempts fail due to ENOSPC errors while
> trying to write out dirty buffers.
>
> To avoid conflating this effect with anything else, it'd be best if you
> could put the DB on its own small partition, and *not* put pg_xlog
> there.
>
>
>
I'm working on this (thank goodness for junctions). Maybe we shopuld
look at providing a config setting for pg_xlog.
cheers
andrew
^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: [DOCS] pg_total_relation_size() and CHECKPOINT
2008-03-13 19:49 pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-13 20:20 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 15:23 ` Re: pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 16:19 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 16:33 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 17:21 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 18:30 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Gregory Stark <stark@enterprisedb.com>
2008-03-14 20:02 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-26 13:22 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Alvaro Herrera <alvherre@commandprompt.com>
2008-03-26 13:56 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-26 15:41 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 00:03 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-27 01:08 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 02:37 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
@ 2008-03-27 11:14 ` Alvaro Herrera <alvherre@commandprompt.com>
2008-03-27 13:54 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 24+ messages in thread
From: Alvaro Herrera @ 2008-03-27 11:14 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Gregory Stark <stark@enterprisedb.com>; Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>; pgsql-hackers@postgresql.org, Magnus Hagander <magnus@hagander.net>
Andrew Dunstan wrote:
> I'm working on this (thank goodness for junctions). Maybe we shopuld
> look at providing a config setting for pg_xlog.
I hope you mean an initdb switch -- otherwise it is way too easy to
misuse.
--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support
^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: [DOCS] pg_total_relation_size() and CHECKPOINT
2008-03-13 19:49 pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-13 20:20 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 15:23 ` Re: pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 16:19 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 16:33 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 17:21 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 18:30 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Gregory Stark <stark@enterprisedb.com>
2008-03-14 20:02 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-26 13:22 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Alvaro Herrera <alvherre@commandprompt.com>
2008-03-26 13:56 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-26 15:41 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 00:03 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-27 01:08 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 02:37 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-27 11:14 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Alvaro Herrera <alvherre@commandprompt.com>
@ 2008-03-27 13:54 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 14:29 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
0 siblings, 1 reply; 24+ messages in thread
From: Tom Lane @ 2008-03-27 13:54 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@commandprompt.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Gregory Stark <stark@enterprisedb.com>; Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>; pgsql-hackers@postgresql.org, Magnus Hagander <magnus@hagander.net>
Alvaro Herrera <alvherre@commandprompt.com> writes:
> Andrew Dunstan wrote:
>> I'm working on this (thank goodness for junctions). Maybe we shopuld
>> look at providing a config setting for pg_xlog.
> I hope you mean an initdb switch -- otherwise it is way too easy to
> misuse.
There's one already ..
regards, tom lane
^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: [DOCS] pg_total_relation_size() and CHECKPOINT
2008-03-13 19:49 pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-13 20:20 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 15:23 ` Re: pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 16:19 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 16:33 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 17:21 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 18:30 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Gregory Stark <stark@enterprisedb.com>
2008-03-14 20:02 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-26 13:22 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Alvaro Herrera <alvherre@commandprompt.com>
2008-03-26 13:56 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-26 15:41 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 00:03 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-27 01:08 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 02:37 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-27 11:14 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Alvaro Herrera <alvherre@commandprompt.com>
2008-03-27 13:54 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-03-27 14:29 ` Andrew Dunstan <andrew@dunslane.net>
0 siblings, 0 replies; 24+ messages in thread
From: Andrew Dunstan @ 2008-03-27 14:29 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Alvaro Herrera <alvherre@commandprompt.com>; Gregory Stark <stark@enterprisedb.com>; Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>; pgsql-hackers@postgresql.org, Magnus Hagander <magnus@hagander.net>
Tom Lane wrote:
> Alvaro Herrera <alvherre@commandprompt.com> writes:
>
>> Andrew Dunstan wrote:
>>
>>> I'm working on this (thank goodness for junctions). Maybe we shopuld
>>> look at providing a config setting for pg_xlog.
>>>
>
>
>> I hope you mean an initdb switch -- otherwise it is way too easy to
>> misuse.
>>
>
> There's one already ..
>
>
>
heh, the things that creep up on you when you're not looking ...
cheers
andrew
^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: [DOCS] pg_total_relation_size() and CHECKPOINT
2008-03-13 19:49 pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-13 20:20 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 15:23 ` Re: pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 16:19 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 16:33 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 17:21 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 18:30 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Gregory Stark <stark@enterprisedb.com>
2008-03-14 20:02 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-26 13:22 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Alvaro Herrera <alvherre@commandprompt.com>
2008-03-26 13:56 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-26 15:41 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 00:03 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-27 01:08 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-03-27 03:25 ` Andrew Dunstan <andrew@dunslane.net>
2008-03-27 04:13 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 24+ messages in thread
From: Andrew Dunstan @ 2008-03-27 03:25 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Alvaro Herrera <alvherre@commandprompt.com>; Gregory Stark <stark@enterprisedb.com>; Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>; pgsql-hackers@postgresql.org, Magnus Hagander <magnus@hagander.net>
Tom Lane wrote:
> Andrew Dunstan <andrew@dunslane.net> writes:
>
>> Tom Lane wrote:
>>
>>>> The real question here is whether Windows' stat() is telling the truth
>>>> about how much filesystem space has actually been allocated to a file.
>>>>
>>> One thing that would be good is just to see who else can reproduce
>>> the original observation:
>>> http://archives.postgresql.org/pgsql-docs/2008-03/msg00041.php
>>>
>
>
>> I have reproduced it in XP-Pro/SP2 running in a VMWare machine on an FC6
>> host.
>>
>
> OK, so the next question is do we really have an issue, or is this just
> an observational artifact? What I'd try is deliberately running the
> machine out of disk space with a long series of inserts, and then see
> whether subsequent checkpoint attempts fail due to ENOSPC errors while
> trying to write out dirty buffers.
>
> To avoid conflating this effect with anything else, it'd be best if you
> could put the DB on its own small partition, and *not* put pg_xlog
> there.
>
>
>
OK, a very large insert failed as expected. Checkpoint succeeded. Then
vacuum recovered the space.
I suspect that the size reported by stat() is a little delayed here, but
the file system is keeping proper track of it, so the lseek that tries
to extend the file fails at the right spot.
cheers
andrew
^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: [DOCS] pg_total_relation_size() and CHECKPOINT
2008-03-13 19:49 pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-13 20:20 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 15:23 ` Re: pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 16:19 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 16:33 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 17:21 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 18:30 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Gregory Stark <stark@enterprisedb.com>
2008-03-14 20:02 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-26 13:22 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Alvaro Herrera <alvherre@commandprompt.com>
2008-03-26 13:56 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-26 15:41 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 00:03 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-27 01:08 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 03:25 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
@ 2008-03-27 04:13 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 07:33 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Magnus Hagander <magnus@hagander.net>
2008-03-27 12:30 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
0 siblings, 2 replies; 24+ messages in thread
From: Tom Lane @ 2008-03-27 04:13 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: Alvaro Herrera <alvherre@commandprompt.com>; Gregory Stark <stark@enterprisedb.com>; Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>; pgsql-hackers@postgresql.org, Magnus Hagander <magnus@hagander.net>
Andrew Dunstan <andrew@dunslane.net> writes:
> I suspect that the size reported by stat() is a little delayed here, but
> the file system is keeping proper track of it, so the lseek that tries
> to extend the file fails at the right spot.
Hmm. If it really works that way, one would hope Microsoft would've
documented that someplace. Can anyone find a statement that Windows'
stat() is not current?
regards, tom lane
^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: [DOCS] pg_total_relation_size() and CHECKPOINT
2008-03-13 19:49 pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-13 20:20 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 15:23 ` Re: pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 16:19 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 16:33 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 17:21 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 18:30 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Gregory Stark <stark@enterprisedb.com>
2008-03-14 20:02 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-26 13:22 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Alvaro Herrera <alvherre@commandprompt.com>
2008-03-26 13:56 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-26 15:41 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 00:03 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-27 01:08 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 03:25 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-27 04:13 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-03-27 07:33 ` Magnus Hagander <magnus@hagander.net>
1 sibling, 0 replies; 24+ messages in thread
From: Magnus Hagander @ 2008-03-27 07:33 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Alvaro Herrera <alvherre@commandprompt.com>; Gregory Stark <stark@enterprisedb.com>; Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>; pgsql-hackers@postgresql.org
On Thu, 27 Mar 2008 00:13:42 -0400
Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Andrew Dunstan <andrew@dunslane.net> writes:
> > I suspect that the size reported by stat() is a little delayed
> > here, but the file system is keeping proper track of it, so the
> > lseek that tries to extend the file fails at the right spot.
>
> Hmm. If it really works that way, one would hope Microsoft would've
> documented that someplace. Can anyone find a statement that Windows'
> stat() is not current?
I'm not in a position to test it myself now (doing training, and then
I'll be off to pg-east...), but it'd be interesting to see if it acts
the same way with GetFileSize(), or if it's just stat()...
/Magnus
^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: [DOCS] pg_total_relation_size() and CHECKPOINT
2008-03-13 19:49 pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-13 20:20 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 15:23 ` Re: pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 16:19 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 16:33 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 17:21 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 18:30 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Gregory Stark <stark@enterprisedb.com>
2008-03-14 20:02 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-26 13:22 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Alvaro Herrera <alvherre@commandprompt.com>
2008-03-26 13:56 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-26 15:41 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 00:03 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-27 01:08 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 03:25 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-27 04:13 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-03-27 12:30 ` Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-27 12:54 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
1 sibling, 1 reply; 24+ messages in thread
From: Zubkovsky, Sergey @ 2008-03-27 12:30 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; Andrew Dunstan <andrew@dunslane.net>; +Cc: Alvaro Herrera <alvherre@commandprompt.com>; Gregory Stark <stark@enterprisedb.com>; pgsql-hackers@postgresql.org; Magnus Hagander <magnus@hagander.net>
Maybe this helps:
"It is not an error to set a file pointer to a position beyond the end
of the file. The size of the file does not increase until you call the
SetEndOfFile, WriteFile, or WriteFileEx function. A write operation
increases the size of the file to the file pointer position plus the
size of the buffer written, which results in the intervening bytes
uninitialized."
http://msdn2.microsoft.com/en-us/library/aa365541(VS.85).aspx
According to Windows' lseek implementation (attached) SetEndOfFile()
isn't called for this case.
Thanks,
Sergey Zubkovsky
-----Original Message-----
From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
Sent: Thursday, March 27, 2008 7:14 AM
To: Andrew Dunstan
Cc: Alvaro Herrera; Gregory Stark; Zubkovsky, Sergey;
pgsql-hackers@postgresql.org; Magnus Hagander
Subject: Re: [HACKERS] [DOCS] pg_total_relation_size() and CHECKPOINT
Andrew Dunstan <andrew@dunslane.net> writes:
> I suspect that the size reported by stat() is a little delayed here,
but
> the file system is keeping proper track of it, so the lseek that tries
> to extend the file fails at the right spot.
Hmm. If it really works that way, one would hope Microsoft would've
documented that someplace. Can anyone find a statement that Windows'
stat() is not current?
regards, tom lane
Attachments:
lseek.c (application/octet-stream, 3.9K) download
^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: [DOCS] pg_total_relation_size() and CHECKPOINT
2008-03-13 19:49 pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-13 20:20 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 15:23 ` Re: pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 16:19 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 16:33 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 17:21 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 18:30 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Gregory Stark <stark@enterprisedb.com>
2008-03-14 20:02 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-26 13:22 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Alvaro Herrera <alvherre@commandprompt.com>
2008-03-26 13:56 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-26 15:41 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 00:03 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-27 01:08 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 03:25 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-27 04:13 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 12:30 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
@ 2008-03-27 12:54 ` Andrew Dunstan <andrew@dunslane.net>
2008-03-28 15:43 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
0 siblings, 1 reply; 24+ messages in thread
From: Andrew Dunstan @ 2008-03-27 12:54 UTC (permalink / raw)
To: Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Alvaro Herrera <alvherre@commandprompt.com>; Gregory Stark <stark@enterprisedb.com>; pgsql-hackers@postgresql.org, Magnus Hagander <magnus@hagander.net>
Zubkovsky, Sergey wrote:
> Maybe this helps:
>
> "It is not an error to set a file pointer to a position beyond the end
> of the file. The size of the file does not increase until you call the
> SetEndOfFile, WriteFile, or WriteFileEx function. A write operation
> increases the size of the file to the file pointer position plus the
> size of the buffer written, which results in the intervening bytes
> uninitialized."
>
> http://msdn2.microsoft.com/en-us/library/aa365541(VS.85).aspx
>
> According to Windows' lseek implementation (attached) SetEndOfFile()
> isn't called for this case.
>
>
>
Yes, but we immediately follow the lseek bye a write(). See
src/backend/storage/smgr/md.c:mdextend() .
cheers
andrew
^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: [DOCS] pg_total_relation_size() and CHECKPOINT
2008-03-13 19:49 pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-13 20:20 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 15:23 ` Re: pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 16:19 ` Re: pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 16:33 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 17:21 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 18:30 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Gregory Stark <stark@enterprisedb.com>
2008-03-14 20:02 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-26 13:22 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Alvaro Herrera <alvherre@commandprompt.com>
2008-03-26 13:56 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-26 15:41 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 00:03 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-27 01:08 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 03:25 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
2008-03-27 04:13 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 12:30 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-27 12:54 ` Re: [DOCS] pg_total_relation_size() and CHECKPOINT Andrew Dunstan <andrew@dunslane.net>
@ 2008-03-28 15:43 ` Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
0 siblings, 0 replies; 24+ messages in thread
From: Zubkovsky, Sergey @ 2008-03-28 15:43 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Alvaro Herrera <alvherre@commandprompt.com>; Gregory Stark <stark@enterprisedb.com>; pgsql-hackers@postgresql.org; Magnus Hagander <magnus@hagander.net>
It seems I've found the cause and the workaround of the problem.
MSVC's stat() is implemented by using FindNextFile().
MSDN contains the following suspicious paragraph аbout FindNextFile():
"In rare cases, file attribute information on NTFS file systems may not be current at the time you call this function. To obtain the current NTFS file system file attributes, call GetFileInformationByHandle."
Since we generally cannot open an examined file, we need another way.
In the prepared custom build of PG 8.3.1 the native MSVC's stat() was rewrote by adding GetFileAttributesEx() to correct stat's st_size value.
I had seen that a result of MSVC's stat() and a result of GetFileAttributesEx() may be differ by the file size values at least.
The most important thing is the test in the original post
( http://archives.postgresql.org/pgsql-docs/2008-03/msg00041.php )
doesn't reproduce any inconsistence now.
All work fine.
This was tested on my WinXP SP2 platform but I suppose it will work on any NT-based OS.
Thanks,
Sergey Zubkovsky
-----Original Message-----
From: Andrew Dunstan [mailto:andrew@dunslane.net]
Sent: Thursday, March 27, 2008 3:54 PM
To: Zubkovsky, Sergey
Cc: Tom Lane; Alvaro Herrera; Gregory Stark; pgsql-hackers@postgresql.org; Magnus Hagander
Subject: Re: [HACKERS] [DOCS] pg_total_relation_size() and CHECKPOINT
Zubkovsky, Sergey wrote:
> Maybe this helps:
>
> "It is not an error to set a file pointer to a position beyond the end
> of the file. The size of the file does not increase until you call the
> SetEndOfFile, WriteFile, or WriteFileEx function. A write operation
> increases the size of the file to the file pointer position plus the
> size of the buffer written, which results in the intervening bytes
> uninitialized."
>
> http://msdn2.microsoft.com/en-us/library/aa365541(VS.85).aspx
>
> According to Windows' lseek implementation (attached) SetEndOfFile()
> isn't called for this case.
>
>
>
Yes, but we immediately follow the lseek bye a write(). See
src/backend/storage/smgr/md.c:mdextend() .
cheers
andrew
^ permalink raw reply [nested|flat] 24+ messages in thread
end of thread, other threads:[~2008-03-28 15:43 UTC | newest]
Thread overview: 24+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2008-03-13 19:49 pg_total_relation_size() and CHECKPOINT Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-13 20:20 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 15:23 ` Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 16:19 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 16:33 ` Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-14 17:21 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-03-14 18:30 ` Gregory Stark <stark@enterprisedb.com>
2008-03-14 20:02 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-03-26 13:22 ` Alvaro Herrera <alvherre@commandprompt.com>
2008-03-26 13:56 ` Andrew Dunstan <andrew@dunslane.net>
2008-03-26 14:44 ` Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-26 15:41 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 00:03 ` Andrew Dunstan <andrew@dunslane.net>
2008-03-27 01:08 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 02:37 ` Andrew Dunstan <andrew@dunslane.net>
2008-03-27 11:14 ` Alvaro Herrera <alvherre@commandprompt.com>
2008-03-27 13:54 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 14:29 ` Andrew Dunstan <andrew@dunslane.net>
2008-03-27 03:25 ` Andrew Dunstan <andrew@dunslane.net>
2008-03-27 04:13 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-03-27 07:33 ` Magnus Hagander <magnus@hagander.net>
2008-03-27 12:30 ` Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
2008-03-27 12:54 ` Andrew Dunstan <andrew@dunslane.net>
2008-03-28 15:43 ` Zubkovsky, Sergey <Sergey.Zubkovsky@transas.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox