pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedRe: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
40+ messages / 8 participants
[nested] [flat]
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
@ 2010-02-02 17:36 Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 40+ messages in thread
From: Robert Haas @ 2010-02-02 17:36 UTC (permalink / raw)
To: Greg Stark <gsstark@mit.edu>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Andres Freund <andres@anarazel.de>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Fri, Jan 29, 2010 at 1:56 PM, Greg Stark <gsstark@mit.edu> wrote:
> On Tue, Jan 19, 2010 at 3:25 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> That function *seriously* needs documentation, in particular the fact
>> that it's a no-op on machines without the right kernel call. The name
>> you've chosen is very bad for those semantics. I'd pick something
>> else myself. Maybe "pg_start_data_flush" or something like that?
>>
>
> I would like to make one token argument in favour of the name I
> picked. If it doesn't convince I'll change it since we can always
> revisit the API down the road.
>
> I envision having two function calls, pg_fsync_start() and
> pg_fsync_finish(). The latter will wait until the data synced in the
> first call is actually synced. The fall-back if there's no
> implementation of this would be for fsync_start() to be a noop (or
> something unreliable like posix_fadvise) and fsync_finish() to just be
> a regular fsync.
>
> I think we can accomplish this with sync_file_range() but I need to
> read up on how it actually works a bit more. In this case it doesn't
> make a difference since when we call fsync_finish() it's going to be
> for the entire file and nothing else will have been writing to these
> files. But for wal writing and checkpointing it might have very
> different performance characteristics.
>
> The big objection to this is that then we don't really have an api for
> FADV_DONT_NEED which is more about cache policy than about syncing to
> disk. So for example a sequential scan might want to indicate that it
> isn't planning on reading the buffers it's churning through but
> doesn't want to force them to be written sooner than otherwise and is
> never going to call fsync_finish().
I took a look at this patch today and I agree with Tom that
pg_fsync_start() is a very confusing name. I don't know what the
right name is, but this doesn't fsync so I don't think it shuld have
fsync in the name. Maybe something like pg_advise_abandon() or
pg_abandon_cache(). The current name is really wishful thinking:
you're hoping that it will make the kernel start the fsync, but it
might not. I think pg_start_data_flush() is similarly optimistic.
...Robert
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
@ 2010-02-02 17:43 ` Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 40+ messages in thread
From: Andres Freund @ 2010-02-02 17:43 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Greg Stark <gsstark@mit.edu>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tuesday 02 February 2010 18:36:12 Robert Haas wrote:
> On Fri, Jan 29, 2010 at 1:56 PM, Greg Stark <gsstark@mit.edu> wrote:
> > On Tue, Jan 19, 2010 at 3:25 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >> That function *seriously* needs documentation, in particular the fact
> >> that it's a no-op on machines without the right kernel call. The name
> >> you've chosen is very bad for those semantics. I'd pick something
> >> else myself. Maybe "pg_start_data_flush" or something like that?
> >
> > I would like to make one token argument in favour of the name I
> > picked. If it doesn't convince I'll change it since we can always
> > revisit the API down the road.
> >
> > I envision having two function calls, pg_fsync_start() and
> > pg_fsync_finish(). The latter will wait until the data synced in the
> > first call is actually synced. The fall-back if there's no
> > implementation of this would be for fsync_start() to be a noop (or
> > something unreliable like posix_fadvise) and fsync_finish() to just be
> > a regular fsync.
> >
> > I think we can accomplish this with sync_file_range() but I need to
> > read up on how it actually works a bit more. In this case it doesn't
> > make a difference since when we call fsync_finish() it's going to be
> > for the entire file and nothing else will have been writing to these
> > files. But for wal writing and checkpointing it might have very
> > different performance characteristics.
> >
> > The big objection to this is that then we don't really have an api for
> > FADV_DONT_NEED which is more about cache policy than about syncing to
> > disk. So for example a sequential scan might want to indicate that it
> > isn't planning on reading the buffers it's churning through but
> > doesn't want to force them to be written sooner than otherwise and is
> > never going to call fsync_finish().
>
> I took a look at this patch today and I agree with Tom that
> pg_fsync_start() is a very confusing name. I don't know what the
> right name is, but this doesn't fsync so I don't think it shuld have
> fsync in the name. Maybe something like pg_advise_abandon() or
> pg_abandon_cache(). The current name is really wishful thinking:
> you're hoping that it will make the kernel start the fsync, but it
> might not. I think pg_start_data_flush() is similarly optimistic.
What about: pg_fsync_prepare(). That gives the reason why were doing that and
doesnt promise that it is actually doing an fsync.
I dislike really having "cache" in the name, because the primary aim is not to
discard the cache...
Andres
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
@ 2010-02-02 17:50 ` Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
0 siblings, 1 reply; 40+ messages in thread
From: Tom Lane @ 2010-02-02 17:50 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Robert Haas <robertmhaas@gmail.com>; Greg Stark <gsstark@mit.edu>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
Andres Freund <andres@anarazel.de> writes:
> On Tuesday 02 February 2010 18:36:12 Robert Haas wrote:
>> I took a look at this patch today and I agree with Tom that
>> pg_fsync_start() is a very confusing name. I don't know what the
>> right name is, but this doesn't fsync so I don't think it shuld have
>> fsync in the name. Maybe something like pg_advise_abandon() or
>> pg_abandon_cache(). The current name is really wishful thinking:
>> you're hoping that it will make the kernel start the fsync, but it
>> might not. I think pg_start_data_flush() is similarly optimistic.
> What about: pg_fsync_prepare().
prepare_for_fsync()?
regards, tom lane
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
@ 2010-02-02 18:14 ` Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 40+ messages in thread
From: Robert Haas @ 2010-02-02 18:14 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andres Freund <andres@anarazel.de>; Greg Stark <gsstark@mit.edu>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tue, Feb 2, 2010 at 12:50 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Andres Freund <andres@anarazel.de> writes:
>> On Tuesday 02 February 2010 18:36:12 Robert Haas wrote:
>>> I took a look at this patch today and I agree with Tom that
>>> pg_fsync_start() is a very confusing name. I don't know what the
>>> right name is, but this doesn't fsync so I don't think it shuld have
>>> fsync in the name. Maybe something like pg_advise_abandon() or
>>> pg_abandon_cache(). The current name is really wishful thinking:
>>> you're hoping that it will make the kernel start the fsync, but it
>>> might not. I think pg_start_data_flush() is similarly optimistic.
>
>> What about: pg_fsync_prepare().
>
> prepare_for_fsync()?
It still seems mis-descriptive to me. Couldn't the same routine be
used simply to abandon undirtied data that we no longer care about
caching?
...Robert
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
@ 2010-02-02 18:34 ` Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
0 siblings, 1 reply; 40+ messages in thread
From: Andres Freund @ 2010-02-02 18:34 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Greg Stark <gsstark@mit.edu>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tuesday 02 February 2010 19:14:40 Robert Haas wrote:
> On Tue, Feb 2, 2010 at 12:50 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > Andres Freund <andres@anarazel.de> writes:
> >> On Tuesday 02 February 2010 18:36:12 Robert Haas wrote:
> >>> I took a look at this patch today and I agree with Tom that
> >>> pg_fsync_start() is a very confusing name. I don't know what the
> >>> right name is, but this doesn't fsync so I don't think it shuld have
> >>> fsync in the name. Maybe something like pg_advise_abandon() or
> >>> pg_abandon_cache(). The current name is really wishful thinking:
> >>> you're hoping that it will make the kernel start the fsync, but it
> >>> might not. I think pg_start_data_flush() is similarly optimistic.
> >>
> >> What about: pg_fsync_prepare().
> >
> > prepare_for_fsync()?
>
> It still seems mis-descriptive to me. Couldn't the same routine be
> used simply to abandon undirtied data that we no longer care about
> caching?
For now it could - but it very well might be converted to sync_file_range or
similar, which would have different "sideeffects".
As the potential code duplication is rather small I would prefer to describe
the prime effect not the sideeffects...
Andres
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
@ 2010-02-02 19:06 ` Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:08 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 2 replies; 40+ messages in thread
From: Robert Haas @ 2010-02-02 19:06 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Greg Stark <gsstark@mit.edu>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tue, Feb 2, 2010 at 1:34 PM, Andres Freund <andres@anarazel.de> wrote:
> For now it could - but it very well might be converted to sync_file_range or
> similar, which would have different "sideeffects".
>
> As the potential code duplication is rather small I would prefer to describe
> the prime effect not the sideeffects...
Hmm, in that case, I think the problem is that this function has no
comment explaining its intended charter.
...Robert
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
@ 2010-02-02 19:08 ` Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 40+ messages in thread
From: Andres Freund @ 2010-02-02 19:08 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Greg Stark <gsstark@mit.edu>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tuesday 02 February 2010 20:06:32 Robert Haas wrote:
> On Tue, Feb 2, 2010 at 1:34 PM, Andres Freund <andres@anarazel.de> wrote:
> > For now it could - but it very well might be converted to sync_file_range
> > or similar, which would have different "sideeffects".
> >
> > As the potential code duplication is rather small I would prefer to
> > describe the prime effect not the sideeffects...
>
> Hmm, in that case, I think the problem is that this function has no
> comment explaining its intended charter.
I agree there. Greg, do you want to update the patch with some comments or
shall I?
Andres
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
@ 2010-02-02 19:33 ` Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
1 sibling, 1 reply; 40+ messages in thread
From: Tom Lane @ 2010-02-02 19:33 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Greg Stark <gsstark@mit.edu>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
Robert Haas <robertmhaas@gmail.com> writes:
> Hmm, in that case, I think the problem is that this function has no
> comment explaining its intended charter.
That's certainly a big problem, but a comment won't fix the fact that
the name is misleading. We need both a comment and a name change.
regards, tom lane
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
@ 2010-02-02 19:45 ` Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
0 siblings, 1 reply; 40+ messages in thread
From: Robert Haas @ 2010-02-02 19:45 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andres Freund <andres@anarazel.de>; Greg Stark <gsstark@mit.edu>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tue, Feb 2, 2010 at 2:33 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> Hmm, in that case, I think the problem is that this function has no
>> comment explaining its intended charter.
>
> That's certainly a big problem, but a comment won't fix the fact that
> the name is misleading. We need both a comment and a name change.
I think you're probably right, but it's not clear what the new name
should be until we have a comment explaining what the function is
responsible for.
...Robert
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
@ 2010-02-03 11:53 ` Greg Stark <gsstark@mit.edu>
2010-02-03 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
0 siblings, 2 replies; 40+ messages in thread
From: Greg Stark @ 2010-02-03 11:53 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Andres Freund <andres@anarazel.de>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tue, Feb 2, 2010 at 7:45 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> I think you're probably right, but it's not clear what the new name
> should be until we have a comment explaining what the function is
> responsible for.
So I wrote some comments but wasn't going to repost the patch with the
unchanged name without explanation... But I think you're right though
I was looking at it the other way around. I want to have an API for a
two-stage sync and of course if I do that I'll comment it to explain
that clearly.
The gist of the comments was that the function is preparing to fsync
to initiate the i/o early and allow the later fsync to fast -- but
also at the same time have the beneficial side-effect of avoiding
cache poisoning. It's not clear that the two are necessarily linked
though. Perhaps we need two separate apis, though it'll be hard to
keep them separate on all platforms.
--
greg
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
@ 2010-02-03 12:03 ` Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 40+ messages in thread
From: Andres Freund @ 2010-02-03 12:03 UTC (permalink / raw)
To: Greg Stark <gsstark@mit.edu>; +Cc: Robert Haas <robertmhaas@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On 02/03/10 12:53, Greg Stark wrote:
> On Tue, Feb 2, 2010 at 7:45 PM, Robert Haas<robertmhaas@gmail.com> wrote:
>> I think you're probably right, but it's not clear what the new name
>> should be until we have a comment explaining what the function is
>> responsible for.
>
> So I wrote some comments but wasn't going to repost the patch with the
> unchanged name without explanation... But I think you're right though
> I was looking at it the other way around. I want to have an API for a
> two-stage sync and of course if I do that I'll comment it to explain
> that clearly.
>
> The gist of the comments was that the function is preparing to fsync
> to initiate the i/o early and allow the later fsync to fast -- but
> also at the same time have the beneficial side-effect of avoiding
> cache poisoning. It's not clear that the two are necessarily linked
> though. Perhaps we need two separate apis, though it'll be hard to
> keep them separate on all platforms.
I vote for two seperate apis - sure, there will be some unfortunate
overlap for most unixoid platforms but its sure better possibly to allow
adding more platforms later at a centralized place than having to
analyze every place where the api is used.
Andres
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
@ 2010-02-03 13:42 ` Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 40+ messages in thread
From: Robert Haas @ 2010-02-03 13:42 UTC (permalink / raw)
To: Greg Stark <gsstark@mit.edu>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Andres Freund <andres@anarazel.de>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Wed, Feb 3, 2010 at 6:53 AM, Greg Stark <gsstark@mit.edu> wrote:
> On Tue, Feb 2, 2010 at 7:45 PM, Robert Haas <robertmhaas@gmail.com> wrote:
>> I think you're probably right, but it's not clear what the new name
>> should be until we have a comment explaining what the function is
>> responsible for.
>
> So I wrote some comments but wasn't going to repost the patch with the
> unchanged name without explanation... But I think you're right though
> I was looking at it the other way around. I want to have an API for a
> two-stage sync and of course if I do that I'll comment it to explain
> that clearly.
>
> The gist of the comments was that the function is preparing to fsync
> to initiate the i/o early and allow the later fsync to fast -- but
> also at the same time have the beneficial side-effect of avoiding
> cache poisoning. It's not clear that the two are necessarily linked
> though. Perhaps we need two separate apis, though it'll be hard to
> keep them separate on all platforms.
Well, maybe we should start with a discussion of what kernel calls
you're aware of on different platforms and then we could try to put an
API around it. I mean, right now all you've got is
POSIX_FADV_DONTNEED, so given just that I feel like the API could
simply be pg_dontneed() or something. It's hard to design a general
framework based on one example.
...Robert
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
@ 2010-02-03 14:19 ` Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
0 siblings, 1 reply; 40+ messages in thread
From: Andres Freund @ 2010-02-03 14:19 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Greg Stark <gsstark@mit.edu>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On 02/03/10 14:42, Robert Haas wrote:
> On Wed, Feb 3, 2010 at 6:53 AM, Greg Stark<gsstark@mit.edu> wrote:
>> On Tue, Feb 2, 2010 at 7:45 PM, Robert Haas<robertmhaas@gmail.com> wrote:
>>> I think you're probably right, but it's not clear what the new name
>>> should be until we have a comment explaining what the function is
>>> responsible for.
>>
>> So I wrote some comments but wasn't going to repost the patch with the
>> unchanged name without explanation... But I think you're right though
>> I was looking at it the other way around. I want to have an API for a
>> two-stage sync and of course if I do that I'll comment it to explain
>> that clearly.
>>
>> The gist of the comments was that the function is preparing to fsync
>> to initiate the i/o early and allow the later fsync to fast -- but
>> also at the same time have the beneficial side-effect of avoiding
>> cache poisoning. It's not clear that the two are necessarily linked
>> though. Perhaps we need two separate apis, though it'll be hard to
>> keep them separate on all platforms.
>
> Well, maybe we should start with a discussion of what kernel calls
> you're aware of on different platforms and then we could try to put an
> API around it.
In linux there is sync_file_range. On newer Posixish systems one can
emulate that with mmap() and msync() (in batches obviously).
No idea about windows.
Andres
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
@ 2010-02-06 05:03 ` Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 40+ messages in thread
From: Greg Smith @ 2010-02-06 05:03 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Robert Haas <robertmhaas@gmail.com>; Greg Stark <gsstark@mit.edu>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
Andres Freund wrote:
> On 02/03/10 14:42, Robert Haas wrote:
>> Well, maybe we should start with a discussion of what kernel calls
>> you're aware of on different platforms and then we could try to put an
>> API around it.
> In linux there is sync_file_range. On newer Posixish systems one can
> emulate that with mmap() and msync() (in batches obviously).
>
> No idea about windows.
There's a series of parameters you can pass into CreateFile:
http://msdn.microsoft.com/en-us/library/aa363858(VS.85).aspx
A lot of these are already mapped inside of src/port/open.c in a pretty
straightforward way from the POSIX-oriented interface:
O_RDWR,O_WRONLY -> GENERIC_WRITE, GENERIC_READ
O_RANDOM -> FILE_FLAG_RANDOM_ACCESS
O_SEQUENTIAL -> FILE_FLAG_SEQUENTIAL_SCAN
O_SHORT_LIVED -> FILE_ATTRIBUTE_TEMPORARY
O_TEMPORARY -> FILE_FLAG_DELETE_ON_CLOSE
O_DIRECT -> FILE_FLAG_NO_BUFFERING
O_DSYNC -> FILE_FLAG_WRITE_THROUGH
You have to read the whole "Caching Behavior" section to see exactly how
all of those interact, and even then notes like
http://support.microsoft.com/kb/99794 are needed to follow the fine
points of things like FILE_FLAG_NO_BUFFERING vs. FILE_FLAG_WRITE_THROUGH.
So anything that's setting those POSIX open flags better than before is
getting the benefit of that improvement on Windows, too. But that's not
quite the same as the changes using fadvise to provide better targeted
cache control hints.
I'm getting the impression that doing much better on Windows might fall
into the same sort of category as Solaris, where the primary interface
for this sort of thing is to use an AIO implementation instead:
http://msdn.microsoft.com/en-us/library/aa365683(VS.85).aspx
The effective_io_concurrency feature had proof of concept test programs
that worked using AIO, but actually following through on that
implementation would require a major restructuring of how the database
interacts with the OS in terms of reads and writes of blocks. It looks
to me like doing something similar to sync_file_range on Windows would
be similarly difficult.
--
Greg Smith 2ndQuadrant US Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.us
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
@ 2010-02-06 12:03 ` Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
0 siblings, 1 reply; 40+ messages in thread
From: Andres Freund @ 2010-02-06 12:03 UTC (permalink / raw)
To: Greg Smith <greg@2ndquadrant.com>; +Cc: Robert Haas <robertmhaas@gmail.com>; Greg Stark <gsstark@mit.edu>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Saturday 06 February 2010 06:03:30 Greg Smith wrote:
> Andres Freund wrote:
> > On 02/03/10 14:42, Robert Haas wrote:
> >> Well, maybe we should start with a discussion of what kernel calls
> >> you're aware of on different platforms and then we could try to put an
> >> API around it.
> >
> > In linux there is sync_file_range. On newer Posixish systems one can
> > emulate that with mmap() and msync() (in batches obviously).
> >
> > No idea about windows.
> The effective_io_concurrency feature had proof of concept test programs
> that worked using AIO, but actually following through on that
> implementation would require a major restructuring of how the database
> interacts with the OS in terms of reads and writes of blocks. It looks
> to me like doing something similar to sync_file_range on Windows would
> be similarly difficult.
Looking a bit arround it seems one could achieve something approximediately
similar to pg_prepare_fsync() by using
CreateFileMapping && MapViewOfFile && FlushViewOfFile
If I understand it correctly that will flush, but not wait. Unfortunately you
cant event make it wait, so its not possible to implement sync_file_range or
similar fully.
Andres
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
@ 2010-02-07 05:13 ` Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
0 siblings, 1 reply; 40+ messages in thread
From: Robert Haas @ 2010-02-07 05:13 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Greg Smith <greg@2ndquadrant.com>; Greg Stark <gsstark@mit.edu>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Sat, Feb 6, 2010 at 7:03 AM, Andres Freund <andres@anarazel.de> wrote:
> On Saturday 06 February 2010 06:03:30 Greg Smith wrote:
>> Andres Freund wrote:
>> > On 02/03/10 14:42, Robert Haas wrote:
>> >> Well, maybe we should start with a discussion of what kernel calls
>> >> you're aware of on different platforms and then we could try to put an
>> >> API around it.
>> >
>> > In linux there is sync_file_range. On newer Posixish systems one can
>> > emulate that with mmap() and msync() (in batches obviously).
>> >
>> > No idea about windows.
>> The effective_io_concurrency feature had proof of concept test programs
>> that worked using AIO, but actually following through on that
>> implementation would require a major restructuring of how the database
>> interacts with the OS in terms of reads and writes of blocks. It looks
>> to me like doing something similar to sync_file_range on Windows would
>> be similarly difficult.
> Looking a bit arround it seems one could achieve something approximediately
> similar to pg_prepare_fsync() by using
> CreateFileMapping && MapViewOfFile && FlushViewOfFile
>
> If I understand it correctly that will flush, but not wait. Unfortunately you
> cant event make it wait, so its not possible to implement sync_file_range or
> similar fully.
Well it seems that what we're trying to implement is more like
it_would_be_nice_if_you_would_start_syncing_this_file_range_but_its_ok_if_you_dont(),
so maybe that would work.
Anyway, is there something that we can agree on and get committed here
for 9.0, or should we postpone this to 9.1? It seems simple enough
that we ought to be able to get it done, but we're running out of time
and we don't seem to have a clear vision here yet...
...Robert
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
@ 2010-02-07 09:23 ` Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 40+ messages in thread
From: Greg Smith @ 2010-02-07 09:23 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Greg Stark <gsstark@mit.edu>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
Robert Haas wrote:
> Well it seems that what we're trying to implement is more like
> it_would_be_nice_if_you_would_start_syncing_this_file_range_but_its_ok_if_you_dont(),
> so maybe that would work.
>
> Anyway, is there something that we can agree on and get committed here
> for 9.0, or should we postpone this to 9.1? It seems simple enough
> that we ought to be able to get it done, but we're running out of time
> and we don't seem to have a clear vision here yet...
>
This is turning into yet another one of those situations where something
simple and useful is being killed by trying to generalize it way more
than it needs to be, given its current goals and its lack of external
interfaces. There's no catversion bump or API breakage to hinder future
refactoring if this isn't optimally designed internally from day one.
The feature is valuable and there seems at least one spot where it may
be resolving the possibility of a subtle OS interaction bug by being
more thorough in the way that it writes and syncs. The main contention
seems to be over naming and completely optional additional abstraction.
I consider the whole "let's make this cover every type of complicated
sync on every platform" goal interesting and worthwhile, but it's
completely optional for this release. The stuff being fretted over now
is ultimately an internal interface that can be refactored at will in
later releases with no user impact.
If the goal here could be shifted back to finding the minimal level of
abstraction that doesn't seem completely wrong, then updating the
function names and comments to match that more closely, this could
return to committable. That's all I thought was left to do when I moved
it to "ready for committer", and as far as I've seen this expanded scope
of discussion has just moved backwards from that point.
--
Greg Smith 2ndQuadrant Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.com
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
@ 2010-02-07 16:24 ` Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
0 siblings, 1 reply; 40+ messages in thread
From: Tom Lane @ 2010-02-07 16:24 UTC (permalink / raw)
To: Greg Smith <greg@2ndquadrant.com>; +Cc: Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Greg Stark <gsstark@mit.edu>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
Greg Smith <greg@2ndquadrant.com> writes:
> This is turning into yet another one of those situations where something
> simple and useful is being killed by trying to generalize it way more
> than it needs to be, given its current goals and its lack of external
> interfaces. There's no catversion bump or API breakage to hinder future
> refactoring if this isn't optimally designed internally from day one.
I agree that it's too late in the cycle for any major redesign of the
patch. But is it too much to ask to use a less confusing name for the
function?
regards, tom lane
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
@ 2010-02-07 18:23 ` Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 40+ messages in thread
From: Robert Haas @ 2010-02-07 18:23 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Greg Smith <greg@2ndquadrant.com>; Andres Freund <andres@anarazel.de>; Greg Stark <gsstark@mit.edu>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Sun, Feb 7, 2010 at 11:24 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Greg Smith <greg@2ndquadrant.com> writes:
>> This is turning into yet another one of those situations where something
>> simple and useful is being killed by trying to generalize it way more
>> than it needs to be, given its current goals and its lack of external
>> interfaces. There's no catversion bump or API breakage to hinder future
>> refactoring if this isn't optimally designed internally from day one.
>
> I agree that it's too late in the cycle for any major redesign of the
> patch. But is it too much to ask to use a less confusing name for the
> function?
+1. Let's just rename the thing, add some comments, and call it good.
...Robert
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
@ 2010-02-07 18:27 ` Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 40+ messages in thread
From: Andres Freund @ 2010-02-07 18:27 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Greg Smith <greg@2ndquadrant.com>; Greg Stark <gsstark@mit.edu>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Sunday 07 February 2010 19:23:10 Robert Haas wrote:
> On Sun, Feb 7, 2010 at 11:24 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > Greg Smith <greg@2ndquadrant.com> writes:
> >> This is turning into yet another one of those situations where something
> >> simple and useful is being killed by trying to generalize it way more
> >> than it needs to be, given its current goals and its lack of external
> >> interfaces. There's no catversion bump or API breakage to hinder future
> >> refactoring if this isn't optimally designed internally from day one.
> >
> > I agree that it's too late in the cycle for any major redesign of the
> > patch. But is it too much to ask to use a less confusing name for the
> > function?
>
> +1. Let's just rename the thing, add some comments, and call it good.
Will post a updated patch in the next hours unless somebody beats me too it.
Andres
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
@ 2010-02-08 01:31 ` Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
0 siblings, 1 reply; 40+ messages in thread
From: Andres Freund @ 2010-02-08 01:31 UTC (permalink / raw)
To: pgsql-hackers@postgresql.org; +Cc: Robert Haas <robertmhaas@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; Greg Smith <greg@2ndquadrant.com>; Greg Stark <gsstark@mit.edu>; pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Sunday 07 February 2010 19:27:02 Andres Freund wrote:
> On Sunday 07 February 2010 19:23:10 Robert Haas wrote:
> > On Sun, Feb 7, 2010 at 11:24 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > > Greg Smith <greg@2ndquadrant.com> writes:
> > >> This is turning into yet another one of those situations where
> > >> something simple and useful is being killed by trying to generalize
> > >> it way more than it needs to be, given its current goals and its lack
> > >> of external interfaces. There's no catversion bump or API breakage
> > >> to hinder future refactoring if this isn't optimally designed
> > >> internally from day one.
> > >
> > > I agree that it's too late in the cycle for any major redesign of the
> > > patch. But is it too much to ask to use a less confusing name for the
> > > function?
> >
> > +1. Let's just rename the thing, add some comments, and call it good.
>
> Will post a updated patch in the next hours unless somebody beats me too
> it.
Here we go.
I left the name at my suggestion pg_fsync_prepare instead of Tom's
prepare_for_fsync because it seemed more consistend with the naming in the
rest of the file. Obviously feel free to adjust.
I personally think the fsync on the directory should be added to the stable
branches - other opinions?
If wanted I can prepare patches for that.
Andres
Attachments:
faster_createdb_v3.patch (text/x-patch, 5.8K) download
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 7ffa2eb..bc5753a 100644
*** a/src/backend/storage/file/fd.c
--- b/src/backend/storage/file/fd.c
*************** pg_fdatasync(int fd)
*** 320,325 ****
--- 320,361 ----
}
/*
+ * pg_fsync_prepare --- try to make a later fsync on the same file faster
+ *
+ * A call to this function does not guarantee anything!
+ *
+ * The idea is to tell the kernel to write out its cache so that a
+ * fsync later on has less to write out synchronously. This allows
+ * that write requests get reordered more freely.
+ *
+ * In the current implementation this has the additional effect of
+ * dropping the cache in that region and thus can be used to avoid
+ * cache poisoning. This may or may not be wanted.
+ *
+ * XXX: Ideally this API would use sync_file_range (or similar on
+ * platforms other than linux) and a seperate one for cache
+ * control. We are not there yet.
+ *
+ * Look at the thread below 200912282354.51892.andres@anarazel.de in
+ * pgsql-hackers for a longer discussion.
+ */
+ int
+ pg_fsync_prepare(int fd, off_t offset, off_t amount)
+ {
+ if (enableFsync)
+ {
+ #if defined(USE_POSIX_FADVISE) && defined(POSIX_FADV_DONTNEED)
+ return posix_fadvise(fd, offset, amount, POSIX_FADV_DONTNEED);
+ #else
+ return 0;
+ #endif
+ }
+ else
+ return 0;
+ }
+
+
+ /*
* InitFileAccess --- initialize this module during backend startup
*
* This is called during either normal or standalone backend start.
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index 21cb024..b1a4b49 100644
*** a/src/include/storage/fd.h
--- b/src/include/storage/fd.h
*************** extern int pg_fsync_no_writethrough(int
*** 99,104 ****
--- 99,106 ----
extern int pg_fsync_writethrough(int fd);
extern int pg_fdatasync(int fd);
+ extern int prepare_for_fsync(int fd, off_t offset, off_t amount);
+
/* Filename components for OpenTemporaryFile */
#define PG_TEMP_FILES_DIR "pgsql_tmp"
#define PG_TEMP_FILE_PREFIX "pgsql_tmp"
diff --git a/src/port/copydir.c b/src/port/copydir.c
index 72fbf36..eef3cfb 100644
*** a/src/port/copydir.c
--- b/src/port/copydir.c
***************
*** 37,42 ****
--- 37,43 ----
static void copy_file(char *fromfile, char *tofile);
+ static void fsync_fname(char *fname);
/*
*************** copydir(char *fromdir, char *todir, bool
*** 64,69 ****
--- 65,73 ----
(errcode_for_file_access(),
errmsg("could not open directory \"%s\": %m", fromdir)));
+ /*
+ * Copy all the files
+ */
while ((xlde = ReadDir(xldir, fromdir)) != NULL)
{
struct stat fst;
*************** copydir(char *fromdir, char *todir, bool
*** 89,96 ****
--- 93,127 ----
else if (S_ISREG(fst.st_mode))
copy_file(fromfile, tofile);
}
+ FreeDir(xldir);
+
+ /*
+ * Be paranoid here and fsync all files to ensure we catch problems.
+ */
+ xldir = AllocateDir(fromdir);
+ if (xldir == NULL)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open directory \"%s\": %m", fromdir)));
+
+ while ((xlde = ReadDir(xldir, fromdir)) != NULL)
+ {
+ if (strcmp(xlde->d_name, ".") == 0 ||
+ strcmp(xlde->d_name, "..") == 0)
+ continue;
+ snprintf(tofile, MAXPGPATH, "%s/%s", todir, xlde->d_name);
+ fsync_fname(tofile);
+ }
FreeDir(xldir);
+
+ /* It's important to fsync the destination directory itself as
+ * individual file fsyncs don't guarantee that the directory entry
+ * for the file is synced. Recent versions of ext4 have made the
+ * window much wider but it's been true for ext3 and other
+ * filesyetems in the past
+ */
+ fsync_fname(todir);
}
/*
*************** copy_file(char *fromfile, char *tofile)
*** 103,108 ****
--- 134,140 ----
int srcfd;
int dstfd;
int nbytes;
+ off_t offset;
/* Use palloc to ensure we get a maxaligned buffer */
#define COPY_BUF_SIZE (8 * BLCKSZ)
*************** copy_file(char *fromfile, char *tofile)
*** 128,134 ****
/*
* Do the data copying.
*/
! for (;;)
{
nbytes = read(srcfd, buffer, COPY_BUF_SIZE);
if (nbytes < 0)
--- 160,166 ----
/*
* Do the data copying.
*/
! for (offset=0; ; offset+=nbytes)
{
nbytes = read(srcfd, buffer, COPY_BUF_SIZE);
if (nbytes < 0)
*************** copy_file(char *fromfile, char *tofile)
*** 147,161 ****
(errcode_for_file_access(),
errmsg("could not write to file \"%s\": %m", tofile)));
}
- }
! /*
! * Be paranoid here to ensure we catch problems.
! */
! if (pg_fsync(dstfd) != 0)
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not fsync file \"%s\": %m", tofile)));
if (close(dstfd))
ereport(ERROR,
--- 179,197 ----
(errcode_for_file_access(),
errmsg("could not write to file \"%s\": %m", tofile)));
}
! /*
! * Start flushing the dirty memory early.
! *
! * Empirical results seem to show a big speed benefit on
! * recent Linux kernels.
! * As a side effect this avoids cache pollution by throwing
! * away the cache of the new file.
! * XXX: It might be more sensible to do that on the origin
! * than the source.
! */
! pg_fsync_prepare(dstfd, offset, nbytes);
! }
if (close(dstfd))
ereport(ERROR,
*************** copy_file(char *fromfile, char *tofile)
*** 166,168 ****
--- 202,226 ----
pfree(buffer);
}
+
+ /*
+ * fsync a file
+ */
+ static void
+ fsync_fname(char *fname)
+ {
+ int fd = BasicOpenFile(fname,
+ O_RDONLY | PG_BINARY,
+ S_IRUSR | S_IWUSR);
+
+ if (fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m", fname)));
+
+ if (pg_fsync(fd) != 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not fsync file \"%s\": %m", fname)));
+ close(fd);
+ }
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
@ 2010-02-08 03:09 ` Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
0 siblings, 1 reply; 40+ messages in thread
From: Alvaro Herrera @ 2010-02-08 03:09 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Robert Haas <robertmhaas@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; Greg Smith <greg@2ndquadrant.com>; Greg Stark <gsstark@mit.edu>; pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
Andres Freund escribió:
> I personally think the fsync on the directory should be added to the stable
> branches - other opinions?
> If wanted I can prepare patches for that.
Yeah, it seems there are two patches here -- one is the addition of
fsync_fname() and the other is the fsync_prepare stuff.
--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
@ 2010-02-08 04:53 ` Robert Haas <robertmhaas@gmail.com>
2010-02-08 07:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-11 02:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
0 siblings, 3 replies; 40+ messages in thread
From: Robert Haas @ 2010-02-08 04:53 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@commandprompt.com>; +Cc: Andres Freund <andres@anarazel.de>; pgsql-hackers@postgresql.org, Tom Lane <tgl@sss.pgh.pa.us>; Greg Smith <greg@2ndquadrant.com>; Greg Stark <gsstark@mit.edu>; pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Sun, Feb 7, 2010 at 10:09 PM, Alvaro Herrera
<alvherre@commandprompt.com> wrote:
> Andres Freund escribió:
>> I personally think the fsync on the directory should be added to the stable
>> branches - other opinions?
>> If wanted I can prepare patches for that.
>
> Yeah, it seems there are two patches here -- one is the addition of
> fsync_fname() and the other is the fsync_prepare stuff.
Andres, you want to take a crack at splitting this up?
...Robert
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
@ 2010-02-08 07:13 ` Andres Freund <andres@anarazel.de>
2 siblings, 0 replies; 40+ messages in thread
From: Andres Freund @ 2010-02-08 07:13 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Alvaro Herrera <alvherre@commandprompt.com>; pgsql-hackers@postgresql.org, Tom Lane <tgl@sss.pgh.pa.us>; Greg Smith <greg@2ndquadrant.com>; Greg Stark <gsstark@mit.edu>; pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Monday 08 February 2010 05:53:23 Robert Haas wrote:
> On Sun, Feb 7, 2010 at 10:09 PM, Alvaro Herrera
>
> <alvherre@commandprompt.com> wrote:
> > Andres Freund escribió:
> >> I personally think the fsync on the directory should be added to the
> >> stable branches - other opinions?
> >> If wanted I can prepare patches for that.
> >
> > Yeah, it seems there are two patches here -- one is the addition of
> > fsync_fname() and the other is the fsync_prepare stuff.
>
> Andres, you want to take a crack at splitting this up?
Will do. Later today or tomorrow morning.
Andres
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
@ 2010-02-08 18:34 ` Greg Stark <gsstark@mit.edu>
2010-02-08 19:29 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2 siblings, 1 reply; 40+ messages in thread
From: Greg Stark @ 2010-02-08 18:34 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Alvaro Herrera <alvherre@commandprompt.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers@postgresql.org, Tom Lane <tgl@sss.pgh.pa.us>; Greg Smith <greg@2ndquadrant.com>; pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Mon, Feb 8, 2010 at 4:53 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Sun, Feb 7, 2010 at 10:09 PM, Alvaro Herrera
>> Yeah, it seems there are two patches here -- one is the addition of
>> fsync_fname() and the other is the fsync_prepare stuff.
Sorry, I'm just catching up on my mail from FOSDEM this past weekend.
I had come to the same conclusion as Greg that I might as well just
commit it with Tom's "pg_flush_data()" name and we can decide later if
and when we have pg_fsync_start()/pg_fsync_finish() whether it's worth
keeping two apis or not.
So I was just going to commit it like that but I discovered last week
that I don't have cvs write access set up yet. I'll commit it as soon
as I generate a new ssh key and Dave installs it, etc. I intentionally
picked a small simple patch that nobody was waiting on because I knew
there was a risk of delays like this and the paperwork. I'm nearly
there.
--
greg
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-08 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
@ 2010-02-08 19:29 ` Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 40+ messages in thread
From: Andres Freund @ 2010-02-08 19:29 UTC (permalink / raw)
To: Greg Stark <gsstark@mit.edu>; +Cc: Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@commandprompt.com>; pgsql-hackers@postgresql.org, Tom Lane <tgl@sss.pgh.pa.us>; Greg Smith <greg@2ndquadrant.com>; pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Monday 08 February 2010 19:34:01 Greg Stark wrote:
> On Mon, Feb 8, 2010 at 4:53 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> > On Sun, Feb 7, 2010 at 10:09 PM, Alvaro Herrera
> >
> >> Yeah, it seems there are two patches here -- one is the addition of
> >> fsync_fname() and the other is the fsync_prepare stuff.
>
> Sorry, I'm just catching up on my mail from FOSDEM this past weekend.
>
> I had come to the same conclusion as Greg that I might as well just
> commit it with Tom's "pg_flush_data()" name and we can decide later if
> and when we have pg_fsync_start()/pg_fsync_finish() whether it's worth
> keeping two apis or not.
>
> So I was just going to commit it like that but I discovered last week
> that I don't have cvs write access set up yet. I'll commit it as soon
> as I generate a new ssh key and Dave installs it, etc. I intentionally
> picked a small simple patch that nobody was waiting on because I knew
> there was a risk of delays like this and the paperwork. I'm nearly
> there.
Do you still want me to split the patches into two or do you want to do it
yourself?
One in multiple versions for the directory fsync and another one for 9.0?
Andres
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
@ 2010-02-11 02:27 ` Andres Freund <andres@anarazel.de>
2010-02-12 15:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2 siblings, 1 reply; 40+ messages in thread
From: Andres Freund @ 2010-02-11 02:27 UTC (permalink / raw)
To: pgsql-hackers@postgresql.org; +Cc: Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@commandprompt.com>; Tom Lane <tgl@sss.pgh.pa.us>; Greg Smith <greg@2ndquadrant.com>; Greg Stark <gsstark@mit.edu>; pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Monday 08 February 2010 05:53:23 Robert Haas wrote:
> On Sun, Feb 7, 2010 at 10:09 PM, Alvaro Herrera
>
> <alvherre@commandprompt.com> wrote:
> > Andres Freund escribió:
> >> I personally think the fsync on the directory should be added to the
> >> stable branches - other opinions?
> >> If wanted I can prepare patches for that.
> >
> > Yeah, it seems there are two patches here -- one is the addition of
> > fsync_fname() and the other is the fsync_prepare stuff.
>
> Andres, you want to take a crack at splitting this up?
I hope I didnt duplicate Gregs work, but I didnt hear back from him, so...
Everything <8.1 is hopeless because cp is used there... I didnt see it worth
to replace that. The patch applies cleanly for 8.1 to 8.4 and survives the
regression tests
Given pg's heavy commit model I didnt see a point to split the patch for 9.0
as well...
Andres
Attachments:
directory-fsync-8.1-to-8.4.patch (text/x-patch, 1.1K) download
diff --git a/src/port/copydir.c b/src/port/copydir.c
index 72fbf36..b057ffa 100644
*** a/src/port/copydir.c
--- b/src/port/copydir.c
*************** copydir(char *fromdir, char *todir, bool
*** 50,55 ****
--- 50,56 ----
{
DIR *xldir;
struct dirent *xlde;
+ int dirfd;
char fromfile[MAXPGPATH];
char tofile[MAXPGPATH];
*************** copydir(char *fromdir, char *todir, bool
*** 91,96 ****
--- 92,116 ----
}
FreeDir(xldir);
+
+ /*
+ * fsync the directory to make sure data has reached the
+ * disk. While needed by most filesystems, the window got bigger
+ * with newer ones like ext4.
+ */
+ dirfd = BasicOpenFile(todir,
+ O_RDONLY | PG_BINARY,
+ S_IRUSR | S_IWUSR);
+ if(dirfd == -1)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open directory for fsync \"%s\": %m", todir)));
+
+ if(pg_fsync(dirfd) == -1)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not fsync directory \"%s\": %m", todir)));
+ close(dirfd);
}
/*
* Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-11 02:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
@ 2010-02-12 15:49 ` Robert Haas <robertmhaas@gmail.com>
2010-02-14 14:03 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
0 siblings, 1 reply; 40+ messages in thread
From: Robert Haas @ 2010-02-12 15:49 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Greg Stark <gsstark@mit.edu>
On Wed, Feb 10, 2010 at 9:27 PM, Andres Freund <andres@anarazel.de> wrote:
> On Monday 08 February 2010 05:53:23 Robert Haas wrote:
>> On Sun, Feb 7, 2010 at 10:09 PM, Alvaro Herrera
>>
>> <alvherre@commandprompt.com> wrote:
>> > Andres Freund escribió:
>> >> I personally think the fsync on the directory should be added to the
>> >> stable branches - other opinions?
>> >> If wanted I can prepare patches for that.
>> >
>> > Yeah, it seems there are two patches here -- one is the addition of
>> > fsync_fname() and the other is the fsync_prepare stuff.
>>
>> Andres, you want to take a crack at splitting this up?
> I hope I didnt duplicate Gregs work, but I didnt hear back from him, so...
>
> Everything <8.1 is hopeless because cp is used there... I didnt see it worth
> to replace that. The patch applies cleanly for 8.1 to 8.4 and survives the
> regression tests
>
> Given pg's heavy commit model I didnt see a point to split the patch for 9.0
> as well...
I'd probably argue for committing this patch to both HEAD and the
back-branches, and doing a second commit with the remaining stuff for
HEAD only, but I don't care very much.
Greg Stark, have you managed to get your access issues sorted out? If
you like, I can do the actual commit on this one.
...Robert
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-11 02:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-12 15:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
@ 2010-02-14 14:03 ` Greg Stark <gsstark@mit.edu>
2010-02-14 15:31 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
0 siblings, 1 reply; 40+ messages in thread
From: Greg Stark @ 2010-02-14 14:03 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; pgsql-hackers@postgresql.org
On Fri, Feb 12, 2010 at 3:49 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> Greg Stark, have you managed to get your access issues sorted out? If
Yep, will look at this today.
--
greg
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-11 02:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-12 15:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-14 14:03 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
@ 2010-02-14 15:31 ` Greg Stark <gsstark@mit.edu>
2010-02-14 17:11 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-14 20:57 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
0 siblings, 2 replies; 40+ messages in thread
From: Greg Stark @ 2010-02-14 15:31 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; pgsql-hackers@postgresql.org
On Sun, Feb 14, 2010 at 2:03 PM, Greg Stark <gsstark@mit.edu> wrote:
> On Fri, Feb 12, 2010 at 3:49 PM, Robert Haas <robertmhaas@gmail.com> wrote:
>> Greg Stark, have you managed to get your access issues sorted out? If
>
> Yep, will look at this today.
So I think we have a bigger problem than just copydir.c. It seems to
me we should be fsyncing the table space data directories on every
checkpoint. Otherwise any newly created relations or removed relations
could disappear even though the data in them was fsynced. I'm thinking
I should add an _mdfd_opentblspc(reln) call which returns a file
descriptor for the tablespace and have mdsync() use that to sync the
directory whenever it fsyncs a relation. It would be nice to remember
which tablespaces have been fsynced and only fsync them once though,
that would need another hash table just for tablespaces.
We probably also need to fsync the pg_xlog directory every time we
create or rename an xlog segment.
Are there any other places we do directory operations which we need to
be permanent?
--
greg
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-11 02:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-12 15:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-14 14:03 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 15:31 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
@ 2010-02-14 17:11 ` Tom Lane <tgl@sss.pgh.pa.us>
2010-02-14 17:27 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 40+ messages in thread
From: Tom Lane @ 2010-02-14 17:11 UTC (permalink / raw)
To: Greg Stark <gsstark@mit.edu>; +Cc: Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers@postgresql.org
Greg Stark <gsstark@mit.edu> writes:
> So I think we have a bigger problem than just copydir.c. It seems to
> me we should be fsyncing the table space data directories on every
> checkpoint.
Is there any evidence that anyone anywhere has ever lost data because
of a lack of directory fsyncs? I sure don't recall any bug reports
that seem to match that theory.
It seems to me that we're talking about a huge hit in both code
complexity and performance to deal with a problem that doesn't actually
occur in the field; and which furthermore is trivially solved on any
modern filesystem by choosing the right filesystem options. Why don't
we just document those options, instead?
regards, tom lane
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-11 02:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-12 15:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-14 14:03 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 15:31 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 17:11 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
@ 2010-02-14 17:27 ` Andres Freund <andres@anarazel.de>
2010-02-14 17:37 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 40+ messages in thread
From: Andres Freund @ 2010-02-14 17:27 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Greg Stark <gsstark@mit.edu>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers@postgresql.org
On Sunday 14 February 2010 18:11:39 Tom Lane wrote:
> Greg Stark <gsstark@mit.edu> writes:
> > So I think we have a bigger problem than just copydir.c. It seems to
> > me we should be fsyncing the table space data directories on every
> > checkpoint.
>
> Is there any evidence that anyone anywhere has ever lost data because
> of a lack of directory fsyncs? I sure don't recall any bug reports
> that seem to match that theory.
I have actually seen the issue during create database at least. In a
virtualized hw though...
~1GB template database, lots and lots of small tables, the crash occured maybe
a minute after CREATE DB, filesystem was xfs, kernel 2.6.30.y.
> It seems to me that we're talking about a huge hit in both code
> complexity and performance to deal with a problem that doesn't actually
> occur in the field; and which furthermore is trivially solved on any
> modern filesystem by choosing the right filesystem options. Why don't
> we just document those options, instead?
Which options would that be? I am not aware that there any for any of the
recent linux filesystems.
Well, except "sync" that is, but that sure would be more of a performance hit
than fsyncing the directory...
Andres
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-11 02:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-12 15:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-14 14:03 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 15:31 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 17:11 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-14 17:27 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
@ 2010-02-14 17:37 ` Tom Lane <tgl@sss.pgh.pa.us>
2010-02-14 20:24 ` Re: Re: Faster CREATE DATABASE by delaying fsync Florian Weimer <fw@deneb.enyo.de>
0 siblings, 1 reply; 40+ messages in thread
From: Tom Lane @ 2010-02-14 17:37 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Greg Stark <gsstark@mit.edu>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers@postgresql.org
Andres Freund <andres@anarazel.de> writes:
> On Sunday 14 February 2010 18:11:39 Tom Lane wrote:
>> It seems to me that we're talking about a huge hit in both code
>> complexity and performance to deal with a problem that doesn't actually
>> occur in the field; and which furthermore is trivially solved on any
>> modern filesystem by choosing the right filesystem options. Why don't
>> we just document those options, instead?
> Which options would that be? I am not aware that there any for any of the
> recent linux filesystems.
Shouldn't journaling of metadata be sufficient?
regards, tom lane
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-11 02:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-12 15:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-14 14:03 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 15:31 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 17:11 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-14 17:27 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-14 17:37 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
@ 2010-02-14 20:24 ` Florian Weimer <fw@deneb.enyo.de>
2010-02-14 20:41 ` Re: Re: Faster CREATE DATABASE by delaying fsync Mark Mielke <mark@mark.mielke.cc>
0 siblings, 1 reply; 40+ messages in thread
From: Florian Weimer @ 2010-02-14 20:24 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andres Freund <andres@anarazel.de>; Greg Stark <gsstark@mit.edu>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers@postgresql.org
* Tom Lane:
>> Which options would that be? I am not aware that there any for any of the
>> recent linux filesystems.
>
> Shouldn't journaling of metadata be sufficient?
You also need to enforce ordering between the directory update and the
file update. The file metadata is flushed with fsync(), but the
directory isn't. On some systems, all directory operations are
synchronous, but not on Linux.
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-11 02:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-12 15:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-14 14:03 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 15:31 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 17:11 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-14 17:27 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-14 17:37 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-14 20:24 ` Re: Re: Faster CREATE DATABASE by delaying fsync Florian Weimer <fw@deneb.enyo.de>
@ 2010-02-14 20:41 ` Mark Mielke <mark@mark.mielke.cc>
2010-02-14 20:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 40+ messages in thread
From: Mark Mielke @ 2010-02-14 20:41 UTC (permalink / raw)
To: Florian Weimer <fw@deneb.enyo.de>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Andres Freund <andres@anarazel.de>; Greg Stark <gsstark@mit.edu>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers@postgresql.org
On 02/14/2010 03:24 PM, Florian Weimer wrote:
> * Tom Lane:
>
>>> Which options would that be? I am not aware that there any for any of the
>>> recent linux filesystems.
>>>
>> Shouldn't journaling of metadata be sufficient?
>>
> You also need to enforce ordering between the directory update and the
> file update. The file metadata is flushed with fsync(), but the
> directory isn't. On some systems, all directory operations are
> synchronous, but not on Linux.
>
dirsync
All directory updates within the filesystem should be
done syn-
chronously. This affects the following system calls:
creat,
link, unlink, symlink, mkdir, rmdir, mknod and rename.
The widely reported problems, though, did not tend to be a problem with
directory changes written too late - but directory changes being written
too early. That is, the directory change is written to disk, but the
file content is not. This is likely because of the "ordered journal"
mode widely used in ext3/ext4 where metadata changes are journalled, but
file pages are not journalled. Therefore, it is important for some
operations, that the file pages are pushed to disk using fsync(file),
before the metadata changes are journalled.
In theory there is some open hole where directory updates need to be
synchronized with file updates, as POSIX doesn't enforce this ordering,
and we can't trust that all file systems implicitly order things
correctly, but in practice, I don't see this sort of problem happening.
If you are concerned, enable dirsync.
Cheers,
mark
--
Mark Mielke<mark@mielke.cc>
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-11 02:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-12 15:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-14 14:03 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 15:31 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 17:11 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-14 17:27 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-14 17:37 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-14 20:24 ` Re: Re: Faster CREATE DATABASE by delaying fsync Florian Weimer <fw@deneb.enyo.de>
2010-02-14 20:41 ` Re: Re: Faster CREATE DATABASE by delaying fsync Mark Mielke <mark@mark.mielke.cc>
@ 2010-02-14 20:49 ` Andres Freund <andres@anarazel.de>
2010-02-15 00:08 ` Re: Re: Faster CREATE DATABASE by delaying fsync Mark Mielke <mark@mark.mielke.cc>
0 siblings, 1 reply; 40+ messages in thread
From: Andres Freund @ 2010-02-14 20:49 UTC (permalink / raw)
To: pgsql-hackers@postgresql.org; +Cc: Mark Mielke <mark@mark.mielke.cc>; Florian Weimer <fw@deneb.enyo.de>; Tom Lane <tgl@sss.pgh.pa.us>; Greg Stark <gsstark@mit.edu>; Robert Haas <robertmhaas@gmail.com>
On Sunday 14 February 2010 21:41:02 Mark Mielke wrote:
> On 02/14/2010 03:24 PM, Florian Weimer wrote:
> > * Tom Lane:
> >>> Which options would that be? I am not aware that there any for any of
> >>> the recent linux filesystems.
> >>
> >> Shouldn't journaling of metadata be sufficient?
> >
> > You also need to enforce ordering between the directory update and the
> > file update. The file metadata is flushed with fsync(), but the
> > directory isn't. On some systems, all directory operations are
> > synchronous, but not on Linux.
>
> dirsync
> All directory updates within the filesystem should be
> done syn-
> chronously. This affects the following system calls:
> creat,
> link, unlink, symlink, mkdir, rmdir, mknod and rename.
>
> The widely reported problems, though, did not tend to be a problem with
> directory changes written too late - but directory changes being written
> too early. That is, the directory change is written to disk, but the
> file content is not. This is likely because of the "ordered journal"
> mode widely used in ext3/ext4 where metadata changes are journalled, but
> file pages are not journalled. Therefore, it is important for some
> operations, that the file pages are pushed to disk using fsync(file),
> before the metadata changes are journalled.
Well, but thats not a problem with pg as it fsyncs the file contents.
> In theory there is some open hole where directory updates need to be
> synchronized with file updates, as POSIX doesn't enforce this ordering,
> and we can't trust that all file systems implicitly order things
> correctly, but in practice, I don't see this sort of problem happening.
I can try to reproduce it if you want...
> If you are concerned, enable dirsync.
If the filesystem already behaves that way a fsync on it should be fairly
cheap. If it doesnt behave that way doing it is correct...
Besides there is no reason to fsync the directory before the checkpoint, so
dirsync would require a higher cost than doing it correctly.
Andres
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-11 02:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-12 15:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-14 14:03 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 15:31 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 17:11 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-14 17:27 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-14 17:37 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-14 20:24 ` Re: Re: Faster CREATE DATABASE by delaying fsync Florian Weimer <fw@deneb.enyo.de>
2010-02-14 20:41 ` Re: Re: Faster CREATE DATABASE by delaying fsync Mark Mielke <mark@mark.mielke.cc>
2010-02-14 20:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync Andres Freund <andres@anarazel.de>
@ 2010-02-15 00:08 ` Mark Mielke <mark@mark.mielke.cc>
0 siblings, 0 replies; 40+ messages in thread
From: Mark Mielke @ 2010-02-15 00:08 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Florian Weimer <fw@deneb.enyo.de>; Tom Lane <tgl@sss.pgh.pa.us>; Greg Stark <gsstark@mit.edu>; Robert Haas <robertmhaas@gmail.com>
On 02/14/2010 03:49 PM, Andres Freund wrote:
> On Sunday 14 February 2010 21:41:02 Mark Mielke wrote:
>
>> The widely reported problems, though, did not tend to be a problem with
>> directory changes written too late - but directory changes being written
>> too early. That is, the directory change is written to disk, but the
>> file content is not. This is likely because of the "ordered journal"
>> mode widely used in ext3/ext4 where metadata changes are journalled, but
>> file pages are not journalled. Therefore, it is important for some
>> operations, that the file pages are pushed to disk using fsync(file),
>> before the metadata changes are journalled.
>>
> Well, but thats not a problem with pg as it fsyncs the file contents.
>
Exactly. Not a problem.
>> If you are concerned, enable dirsync.
>>
> If the filesystem already behaves that way a fsync on it should be fairly
> cheap. If it doesnt behave that way doing it is correct...
>
Well, I disagree, as the whole point of this thread is that fsync() is
*not* cheap. :-)
> Besides there is no reason to fsync the directory before the checkpoint, so
> dirsync would require a higher cost than doing it correctly.
>
Using "ordered" metadata journaling has approximately the same effect.
Provided that the data is fsync()'d before the metadata is required,
either the metadata is recorded in the journal, in which case the data
is accessible, or the metadata is NOT recorded in the journal, in which
case, the files will appear missing. The races that theoretically exist
would be in situations where the data of one file references a separate
file that does not yet exist.
You said you would try and reproduce - are you going to try and
reproduce on ext3/ext4 with ordered journalling enabled? I think
reproducing outside of a case such as CREATE DATABASE would be
difficult. It would have to be something like:
open(O_CREAT)/write()/fsync()/close() of new data file, where data
gets written, but directory data is not yet written out to journal
open()/.../write()/fsync()/close() of existing file to point to new
data file, but directory data is still not yet written out to journal
crash
In this case, "dirsync" should be effective at closing this hole.
As for cost? Well, most PostgreSQL data is stored within file content,
not directory metadata. I think "dirsync" might slow down some
operations like CREATE DATABASE or "rm -fr", but I would not expect it
to effect day-to-day performance of the database under real load. Many
operating systems enable the equivalent of "dirsync" by default. I
believe Solaris does this, for example, and other than slowing down "rm
-fr", I don't recall any real complaints about the cost of "dirsync".
After writing the above, I'm seriously considering adding "dirsync" to
my /db mounts that hold PostgreSQL and MySQL data.
Cheers,
mark
--
Mark Mielke<mark@mielke.cc>
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-11 02:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-12 15:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-14 14:03 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 15:31 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
@ 2010-02-14 20:57 ` Robert Haas <robertmhaas@gmail.com>
2010-02-14 21:43 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-14 23:33 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
1 sibling, 2 replies; 40+ messages in thread
From: Robert Haas @ 2010-02-14 20:57 UTC (permalink / raw)
To: Greg Stark <gsstark@mit.edu>; +Cc: Andres Freund <andres@anarazel.de>; pgsql-hackers@postgresql.org
On Sun, Feb 14, 2010 at 10:31 AM, Greg Stark <gsstark@mit.edu> wrote:
> On Sun, Feb 14, 2010 at 2:03 PM, Greg Stark <gsstark@mit.edu> wrote:
>> On Fri, Feb 12, 2010 at 3:49 PM, Robert Haas <robertmhaas@gmail.com> wrote:
>>> Greg Stark, have you managed to get your access issues sorted out? If
>>
>> Yep, will look at this today.
>
> So I think we have a bigger problem than just copydir.c. It seems to
> me we should be fsyncing the table space data directories on every
> checkpoint. Otherwise any newly created relations or removed relations
> could disappear even though the data in them was fsynced. I'm thinking
> I should add an _mdfd_opentblspc(reln) call which returns a file
> descriptor for the tablespace and have mdsync() use that to sync the
> directory whenever it fsyncs a relation. It would be nice to remember
> which tablespaces have been fsynced and only fsync them once though,
> that would need another hash table just for tablespaces.
>
> We probably also need to fsync the pg_xlog directory every time we
> create or rename an xlog segment.
>
> Are there any other places we do directory operations which we need to
> be permanent?
I agree with Tom that we need to see some actual reproducible test
cases where this is an issue before we go too crazy with it. In
theory what you're talking about could also happen when extending a
relation, if we extend into a new file; but I think we need to
convince ourselves that it really happens before we make any more
changes.
On a pragmatic note, if this does turn out to be a problem, it's a
bug: and we can and do fix bugs whenever we discover them. But the
other part of this patch - to speed up createdb - is a feature - and
we are very rapidly running out of time for 9.0 features. So I'd like
to vote for getting the feature part of this committed (assuming it's
in good shape, of course) and we can continue to investigate the other
issues but without quite as much urgency.
...Robert
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-11 02:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-12 15:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-14 14:03 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 15:31 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 20:57 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
@ 2010-02-14 21:43 ` Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 40+ messages in thread
From: Andres Freund @ 2010-02-14 21:43 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Greg Stark <gsstark@mit.edu>; pgsql-hackers@postgresql.org
On Sunday 14 February 2010 21:57:08 Robert Haas wrote:
> On Sun, Feb 14, 2010 at 10:31 AM, Greg Stark <gsstark@mit.edu> wrote:
> > On Sun, Feb 14, 2010 at 2:03 PM, Greg Stark <gsstark@mit.edu> wrote:
> >> On Fri, Feb 12, 2010 at 3:49 PM, Robert Haas <robertmhaas@gmail.com>
wrote:
> >>> Greg Stark, have you managed to get your access issues sorted out? If
> >>
> >> Yep, will look at this today.
> >
> > So I think we have a bigger problem than just copydir.c. It seems to
> > me we should be fsyncing the table space data directories on every
> > checkpoint. Otherwise any newly created relations or removed relations
> > could disappear even though the data in them was fsynced. I'm thinking
> > I should add an _mdfd_opentblspc(reln) call which returns a file
> > descriptor for the tablespace and have mdsync() use that to sync the
> > directory whenever it fsyncs a relation. It would be nice to remember
> > which tablespaces have been fsynced and only fsync them once though,
> > that would need another hash table just for tablespaces.
> >
> > We probably also need to fsync the pg_xlog directory every time we
> > create or rename an xlog segment.
> >
> > Are there any other places we do directory operations which we need to
> > be permanent?
>
> I agree with Tom that we need to see some actual reproducible test
> cases where this is an issue before we go too crazy with it. In
> theory what you're talking about could also happen when extending a
> relation, if we extend into a new file; but I think we need to
> convince ourselves that it really happens before we make any more
> changes.
Ok, will try to reproduce.
> On a pragmatic note, if this does turn out to be a problem, it's a
> bug: and we can and do fix bugs whenever we discover them. But the
> other part of this patch - to speed up createdb - is a feature - and
> we are very rapidly running out of time for 9.0 features. So I'd like
> to vote for getting the feature part of this committed (assuming it's
> in good shape, of course) and we can continue to investigate the other
> issues but without quite as much urgency.
Sound sensible.
Andres
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-11 02:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-12 15:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-14 14:03 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 15:31 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 20:57 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
@ 2010-02-14 23:33 ` Greg Stark <gsstark@mit.edu>
1 sibling, 0 replies; 40+ messages in thread
From: Greg Stark @ 2010-02-14 23:33 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; pgsql-hackers@postgresql.org
On Sun, Feb 14, 2010 at 8:57 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> On a pragmatic note, if this does turn out to be a problem, it's a
> bug: and we can and do fix bugs whenever we discover them. But the
> other part of this patch - to speed up createdb - is a feature - and
> we are very rapidly running out of time for 9.0 features. So I'd like
> to vote for getting the feature part of this committed (assuming it's
> in good shape, of course) and we can continue to investigate the other
> issues but without quite as much urgency.
No problem, I already committed the part that overlaps so I can commit
the rest now. I just want to take extra care given how much wine I've
already had tonight...
Incidentally, sorry Andres, I forgot to credit you in the first commit.
--
greg
^ permalink raw reply [nested|flat] 40+ messages in thread
end of thread, other threads:[~2010-02-15 00:08 UTC | newest]
Thread overview: 40+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2010-02-02 17:36 Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:08 ` Andres Freund <andres@anarazel.de>
2010-02-02 19:33 ` Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Greg Stark <gsstark@mit.edu>
2010-02-03 12:03 ` Andres Freund <andres@anarazel.de>
2010-02-03 13:42 ` Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Robert Haas <robertmhaas@gmail.com>
2010-02-08 07:13 ` Andres Freund <andres@anarazel.de>
2010-02-08 18:34 ` Greg Stark <gsstark@mit.edu>
2010-02-08 19:29 ` Andres Freund <andres@anarazel.de>
2010-02-11 02:27 ` Andres Freund <andres@anarazel.de>
2010-02-12 15:49 ` Robert Haas <robertmhaas@gmail.com>
2010-02-14 14:03 ` Greg Stark <gsstark@mit.edu>
2010-02-14 15:31 ` Greg Stark <gsstark@mit.edu>
2010-02-14 17:11 ` Tom Lane <tgl@sss.pgh.pa.us>
2010-02-14 17:27 ` Andres Freund <andres@anarazel.de>
2010-02-14 17:37 ` Tom Lane <tgl@sss.pgh.pa.us>
2010-02-14 20:24 ` Re: Re: Faster CREATE DATABASE by delaying fsync Florian Weimer <fw@deneb.enyo.de>
2010-02-14 20:41 ` Re: Re: Faster CREATE DATABASE by delaying fsync Mark Mielke <mark@mark.mielke.cc>
2010-02-14 20:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync Andres Freund <andres@anarazel.de>
2010-02-15 00:08 ` Re: Re: Faster CREATE DATABASE by delaying fsync Mark Mielke <mark@mark.mielke.cc>
2010-02-14 20:57 ` Robert Haas <robertmhaas@gmail.com>
2010-02-14 21:43 ` Andres Freund <andres@anarazel.de>
2010-02-14 23:33 ` Greg Stark <gsstark@mit.edu>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox