pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedFaster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
70+ messages / 10 participants
[nested] [flat]
* Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
@ 2009-12-28 22:54 Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 23:06 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2009-12-29 00:27 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
0 siblings, 3 replies; 70+ messages in thread
From: Andres Freund @ 2009-12-28 22:54 UTC (permalink / raw)
To: pgsql-performance@postgresql.org; pgsql-hackers; +Cc: Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Saturday 12 December 2009 21:38:41 Andres Freund wrote:
> On Saturday 12 December 2009 21:36:27 Michael Clemmons wrote:
> > If ppl think its worth it I'll create a ticket
> Thanks, no need. I will post a patch tomorrow or so.
Well. It was a long day...
Anyway.
In this patch I delay the fsync done in copy_file and simply do a second pass
over the directory in copy_dir and fsync everything in that pass.
Including the directory - which was not done before and actually might be
necessary in some cases.
I added a posix_fadvise(..., FADV_DONTNEED) to make it more likely that the
copied file reaches storage before the fsync. Without the speed benefits were
quite a bit smaller and essentially random (which seems sensible).
This speeds up CREATE DATABASE from ~9 seconds to something around 0.8s on my
laptop. Still slower than with fsync off (~0.25) but quite a worthy
improvement.
The benefits are obviously bigger if the template database includes anything
added.
Andres
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
@ 2009-12-28 22:59 ` Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2 siblings, 1 reply; 70+ messages in thread
From: Andres Freund @ 2009-12-28 22:59 UTC (permalink / raw)
To: pgsql-hackers; +Cc: pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Monday 28 December 2009 23:54:51 Andres Freund wrote:
> On Saturday 12 December 2009 21:38:41 Andres Freund wrote:
> > On Saturday 12 December 2009 21:36:27 Michael Clemmons wrote:
> > > If ppl think its worth it I'll create a ticket
> >
> > Thanks, no need. I will post a patch tomorrow or so.
>
> Well. It was a long day...
>
> Anyway.
> In this patch I delay the fsync done in copy_file and simply do a second
> pass over the directory in copy_dir and fsync everything in that pass.
> Including the directory - which was not done before and actually might be
> necessary in some cases.
> I added a posix_fadvise(..., FADV_DONTNEED) to make it more likely that the
> copied file reaches storage before the fsync. Without the speed benefits
> were quite a bit smaller and essentially random (which seems sensible).
>
> This speeds up CREATE DATABASE from ~9 seconds to something around 0.8s on
> my laptop. Still slower than with fsync off (~0.25) but quite a worthy
> improvement.
>
> The benefits are obviously bigger if the template database includes
> anything added.
Obviously the patch would be helpfull.
Andres
Attachments:
0001-Delay-fsyncing-files-during-copying-in-CREATE-DATABA.patch (text/x-patch, 3.2K) download
From bd80748883d1328a71607a447677b0bfb1f54ab0 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Mon, 28 Dec 2009 23:43:57 +0100
Subject: [PATCH] Delay fsyncing files during copying in CREATE DATABASE - this
dramatically speeds up CREATE DATABASE on non battery backed
rotational storage.
Additionally fsync() the directory to ensure all metadata reaches
storage.
---
src/port/copydir.c | 58 +++++++++++++++++++++++++++++++++++++++++++++------
1 files changed, 51 insertions(+), 7 deletions(-)
diff --git a/src/port/copydir.c b/src/port/copydir.c
index a70477e..cde3dc7 100644
*** a/src/port/copydir.c
--- b/src/port/copydir.c
***************
*** 37,42 ****
--- 37,43 ----
static void copy_file(char *fromfile, char *tofile);
+ static void fsync_fname(char *fname);
/*
*************** copydir(char *fromdir, char *todir, bool
*** 64,69 ****
--- 65,73 ----
(errcode_for_file_access(),
errmsg("could not open directory \"%s\": %m", fromdir)));
+ /*
+ * Copy all the files
+ */
while ((xlde = ReadDir(xldir, fromdir)) != NULL)
{
struct stat fst;
*************** copydir(char *fromdir, char *todir, bool
*** 89,96 ****
else if (S_ISREG(fst.st_mode))
copy_file(fromfile, tofile);
}
-
FreeDir(xldir);
}
/*
--- 93,120 ----
else if (S_ISREG(fst.st_mode))
copy_file(fromfile, tofile);
}
FreeDir(xldir);
+
+ /*
+ * Be paranoid here and fsync all files to ensure we catch problems.
+ */
+ xldir = AllocateDir(fromdir);
+ if (xldir == NULL)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open directory \"%s\": %m", fromdir)));
+
+ while ((xlde = ReadDir(xldir, fromdir)) != NULL)
+ {
+ struct stat fst;
+
+ if (strcmp(xlde->d_name, ".") == 0 ||
+ strcmp(xlde->d_name, "..") == 0)
+ continue;
+
+ snprintf(tofile, MAXPGPATH, "%s/%s", todir, xlde->d_name);
+ fsync_fname(tofile);
+ }
}
/*
*************** copy_file(char *fromfile, char *tofile)
*** 150,162 ****
}
/*
! * Be paranoid here to ensure we catch problems.
*/
! if (pg_fsync(dstfd) != 0)
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not fsync file \"%s\": %m", tofile)));
!
if (close(dstfd))
ereport(ERROR,
(errcode_for_file_access(),
--- 174,185 ----
}
/*
! * We tell the kernel here to write the data back in order to make
! * the later fsync cheaper.
*/
! #if defined(USE_POSIX_FADVISE) && defined(POSIX_FADV_DONTNEED)
! posix_fadvise(dstfd, 0, 0, POSIX_FADV_DONTNEED);
! #endif
if (close(dstfd))
ereport(ERROR,
(errcode_for_file_access(),
*************** copy_file(char *fromfile, char *tofile)
*** 166,168 ****
--- 189,212 ----
pfree(buffer);
}
+
+ /*
+ * fsync a file
+ */
+ static void
+ fsync_fname(char *fname)
+ {
+ int fd = BasicOpenFile(fname, O_RDWR| PG_BINARY,
+ S_IRUSR | S_IWUSR);
+
+ if (fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create file \"%s\": %m", fname)));
+
+ if (pg_fsync(fd) != 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not fsync file \"%s\": %m", fname)));
+ close(fd);
+ }
--
1.6.5.12.gd65df24
* Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
@ 2009-12-29 18:30 ` Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
0 siblings, 1 reply; 70+ messages in thread
From: Andres Freund @ 2009-12-29 18:30 UTC (permalink / raw)
To: pgsql-hackers; +Cc: pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Monday 28 December 2009 23:59:43 Andres Freund wrote:
> On Monday 28 December 2009 23:54:51 Andres Freund wrote:
> > On Saturday 12 December 2009 21:38:41 Andres Freund wrote:
> > > On Saturday 12 December 2009 21:36:27 Michael Clemmons wrote:
> > > > If ppl think its worth it I'll create a ticket
> > >
> > > Thanks, no need. I will post a patch tomorrow or so.
> >
> > Well. It was a long day...
> >
> > Anyway.
> > In this patch I delay the fsync done in copy_file and simply do a second
> > pass over the directory in copy_dir and fsync everything in that pass.
> > Including the directory - which was not done before and actually might be
> > necessary in some cases.
> > I added a posix_fadvise(..., FADV_DONTNEED) to make it more likely that
> > the copied file reaches storage before the fsync. Without the speed
> > benefits were quite a bit smaller and essentially random (which seems
> > sensible).
> >
> > This speeds up CREATE DATABASE from ~9 seconds to something around 0.8s
> > on my laptop. Still slower than with fsync off (~0.25) but quite a
> > worthy improvement.
> >
> > The benefits are obviously bigger if the template database includes
> > anything added.
>
> Obviously the patch would be helpfull.
And it should also be helpfull not to have annoying oversights in there. A
FreeDir(xldir); is missing at the end of copydir().
Andres
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
@ 2010-01-18 16:35 ` Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-20 04:01 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
0 siblings, 2 replies; 70+ messages in thread
From: Greg Stark @ 2010-01-18 16:35 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
Looking at this patch for the commitfest I have a few questions.
1) You said you added an fsync of the new directory -- where is that I
don't see it anywhere.
2) Why does the second pass to do the fsyncs read through fromdir to
find all the filenames. I find that odd and counterintuitive. It would
be much more natural to just loop through the files in the new
directory. But I suppose it serves as an added paranoia check that the
files are in fact still there and we're not fsyncing any files we
didn't just copy. I think it should still work, we should have an
exclusive lock on the template database so there really ought to be no
differences between the directory trees.
3) It would be tempting to do the posix_fadvise on each chunk as we
copy it. That way we avoid poisoning the filesystem cache even as far
as a 1G file. This might actually be quite significant if we're built
without the 1G file chunk size. I'm a bit concerned that the code will
be a big more complex having to depend on a good off_t definition
though. Do we only use >1GB files on systems where off_t is capable of
handling >2^32 without gymnastics?
--
greg
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
@ 2010-01-19 14:52 ` Greg Stark <gsstark@mit.edu>
2010-01-19 14:57 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-20 04:02 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
1 sibling, 4 replies; 70+ messages in thread
From: Greg Stark @ 2010-01-19 14:52 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Mon, Jan 18, 2010 at 4:35 PM, Greg Stark <gsstark@mit.edu> wrote:
> Looking at this patch for the commitfest I have a few questions.
So I've touched this patch up a bit:
1) moved the posix_fadvise call to a new fd.c function
pg_fsync_start(fd,offset,nbytes) which initiates an fsync without
waiting on it. Currently it's only implemented with
posix_fadvise(DONT_NEED) but I want to look into using sync_file_range
in the future -- it looks like this call might be good enough for our
checkpoints.
2) advised each 64k chunk as we write it which should avoid poisoning
the cache if you do a large create database on an active system.
3) added the promised but afaict missing fsync of the directory -- i
think we should actually backpatch this.
Barring any objections shall I commit it like this?
--
greg
--
greg
Attachments:
create-database-speedup-using-pfa.diff (text/x-patch, 4.7K) download
*** a/src/backend/storage/file/fd.c
--- b/src/backend/storage/file/fd.c
***************
*** 319,324 **** pg_fdatasync(int fd)
--- 319,340 ----
return 0;
}
+ int
+ pg_fsync_start(int fd, off_t offset, off_t amount)
+ {
+ if (enableFsync)
+ {
+ #if defined(USE_POSIX_FADVISE) && defined(POSIX_FADV_DONTNEED)
+ return posix_fadvise(fd, offset, amount, POSIX_FADV_DONTNEED);
+ #else
+ return 0;
+ #endif
+ }
+ else
+ return 0;
+ }
+
+
/*
* InitFileAccess --- initialize this module during backend startup
*
*** a/src/include/storage/fd.h
--- b/src/include/storage/fd.h
***************
*** 98,103 **** extern int pg_fsync(int fd);
--- 98,104 ----
extern int pg_fsync_no_writethrough(int fd);
extern int pg_fsync_writethrough(int fd);
extern int pg_fdatasync(int fd);
+ extern int pg_fsync_start(int fd, off_t offset, off_t amount);
/* Filename components for OpenTemporaryFile */
#define PG_TEMP_FILES_DIR "pgsql_tmp"
*** a/src/port/copydir.c
--- b/src/port/copydir.c
***************
*** 37,42 ****
--- 37,43 ----
static void copy_file(char *fromfile, char *tofile);
+ static void fsync_fname(char *fname);
/*
***************
*** 64,69 **** copydir(char *fromdir, char *todir, bool recurse)
--- 65,73 ----
(errcode_for_file_access(),
errmsg("could not open directory \"%s\": %m", fromdir)));
+ /*
+ * Copy all the files
+ */
while ((xlde = ReadDir(xldir, fromdir)) != NULL)
{
struct stat fst;
***************
*** 89,96 **** copydir(char *fromdir, char *todir, bool recurse)
--- 93,127 ----
else if (S_ISREG(fst.st_mode))
copy_file(fromfile, tofile);
}
+ FreeDir(xldir);
+
+ /*
+ * Be paranoid here and fsync all files to ensure we catch problems.
+ */
+ xldir = AllocateDir(fromdir);
+ if (xldir == NULL)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open directory \"%s\": %m", fromdir)));
+
+ while ((xlde = ReadDir(xldir, fromdir)) != NULL)
+ {
+ if (strcmp(xlde->d_name, ".") == 0 ||
+ strcmp(xlde->d_name, "..") == 0)
+ continue;
+ snprintf(tofile, MAXPGPATH, "%s/%s", todir, xlde->d_name);
+ fsync_fname(tofile);
+ }
FreeDir(xldir);
+
+ /* It's important to fsync the destination directory itself as
+ * individual file fsyncs don't guarantee that the directory entry
+ * for the file is synced. Recent versions of ext4 have made the
+ * window much wider but it's been true for ext3 and other
+ * filesyetems in the past
+ */
+ fsync_fname(todir);
}
/*
***************
*** 103,108 **** copy_file(char *fromfile, char *tofile)
--- 134,140 ----
int srcfd;
int dstfd;
int nbytes;
+ off_t offset;
/* Use palloc to ensure we get a maxaligned buffer */
#define COPY_BUF_SIZE (8 * BLCKSZ)
***************
*** 128,134 **** copy_file(char *fromfile, char *tofile)
/*
* Do the data copying.
*/
! for (;;)
{
nbytes = read(srcfd, buffer, COPY_BUF_SIZE);
if (nbytes < 0)
--- 160,166 ----
/*
* Do the data copying.
*/
! for (offset=0; ; offset+=nbytes)
{
nbytes = read(srcfd, buffer, COPY_BUF_SIZE);
if (nbytes < 0)
***************
*** 147,161 **** copy_file(char *fromfile, char *tofile)
(errcode_for_file_access(),
errmsg("could not write to file \"%s\": %m", tofile)));
}
- }
! /*
! * Be paranoid here to ensure we catch problems.
! */
! if (pg_fsync(dstfd) != 0)
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not fsync file \"%s\": %m", tofile)));
if (close(dstfd))
ereport(ERROR,
--- 179,195 ----
(errcode_for_file_access(),
errmsg("could not write to file \"%s\": %m", tofile)));
}
! /*
! * Avoid poisoning the cache with the entire database just
! * like we do for large sequential scans. Actually that's a
! * white lie, the real motivation for this is that it might
! * urge the kernel to flush these buffers when convenient
! * before we get to the later fsyncs. Empirical results seem
! * to show a big speed benefit on recent Linux kernels.
! */
! pg_fsync_start(dstfd, offset, nbytes);
! }
if (close(dstfd))
ereport(ERROR,
***************
*** 166,168 **** copy_file(char *fromfile, char *tofile)
--- 200,224 ----
pfree(buffer);
}
+
+ /*
+ * fsync a file
+ */
+ static void
+ fsync_fname(char *fname)
+ {
+ int fd = BasicOpenFile(fname,
+ O_RDONLY | PG_BINARY,
+ S_IRUSR | S_IWUSR);
+
+ if (fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m", fname)));
+
+ if (pg_fsync(fd) != 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not fsync file \"%s\": %m", fname)));
+ close(fd);
+ }
* Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
@ 2010-01-19 14:57 ` Greg Stark <gsstark@mit.edu>
2010-01-20 04:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-20 05:21 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-01-27 07:21 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
3 siblings, 3 replies; 70+ messages in thread
From: Greg Stark @ 2010-01-19 14:57 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tue, Jan 19, 2010 at 2:52 PM, Greg Stark <gsstark@mit.edu> wrote:
> Barring any objections shall I commit it like this?
Actually before we get there could someone who demonstrated the
speedup verify that this patch still gets that same speedup?
--
greg
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:57 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
@ 2010-01-20 04:13 ` Andres Freund <andres@anarazel.de>
2 siblings, 0 replies; 70+ messages in thread
From: Andres Freund @ 2010-01-20 04:13 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Greg Stark <gsstark@mit.edu>; pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tuesday 19 January 2010 15:57:14 Greg Stark wrote:
> On Tue, Jan 19, 2010 at 2:52 PM, Greg Stark <gsstark@mit.edu> wrote:
> > Barring any objections shall I commit it like this?
>
> Actually before we get there could someone who demonstrated the
> speedup verify that this patch still gets that same speedup?
At least on the three machines I tested last time the result is still in the
same ballpark.
Andres
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:57 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
@ 2010-01-20 05:21 ` Greg Smith <greg@2ndquadrant.com>
2 siblings, 0 replies; 70+ messages in thread
From: Greg Smith @ 2010-01-20 05:21 UTC (permalink / raw)
To: Greg Stark <gsstark@mit.edu>; +Cc: Andres Freund <andres@anarazel.de>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
Greg Stark wrote:
> On Tue, Jan 19, 2010 at 2:52 PM, Greg Stark <gsstark@mit.edu> wrote:
>
>> Barring any objections shall I commit it like this?
>>
>
> Actually before we get there could someone who demonstrated the
> speedup verify that this patch still gets that same speedup?
>
I think the final version of this patch could use at least one more
performance checking report that it does something useful. We got a lot
of data from Andres, but do we know that the improvements here hold for
others too? I can take a look at it later this week, I have some
interest in this area.
--
Greg Smith 2ndQuadrant Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.com
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:57 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
@ 2010-01-27 07:21 ` Greg Smith <greg@2ndquadrant.com>
2 siblings, 0 replies; 70+ messages in thread
From: Greg Smith @ 2010-01-27 07:21 UTC (permalink / raw)
To: Greg Stark <gsstark@mit.edu>; +Cc: Andres Freund <andres@anarazel.de>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
Greg Stark wrote:
> Actually before we get there could someone who demonstrated the
> speedup verify that this patch still gets that same speedup?
>
Let's step back a second and get to the bottom of why some people are
seeing this and others aren't. The original report here suggested this
was an ext4 issue. As I pointed out recently on the performance list,
the reason for that is likely that the working write-barrier support for
ext4 means it's passing through the fsync to "lying" hard drives via a
proper cache flush, which didn't happen on your typical ext3 install.
Given that, I'd expect I could see the same issue with ext3 given a
drive with its write cache turned off, so that the theory I started
trying to prove before seeing the patch operate.
What I did was create a little test program that created 5 databases and
then dropped them:
\timing
create database a;
create database b;
create database c;
create database d;
create database e;
drop database a;
drop database b;
drop database c;
drop database d;
drop database e;
(All of the drop times were very close by the way; around 100ms, nothing
particularly interesting there)
If I have my system's boot drive (attached to the motherboard, not on
the caching controller) in its regular, lying mode with write cache on,
the creates take the following times:
Time: 713.982 ms Time: 659.890 ms Time: 590.842 ms Time: 675.506 ms
Time: 645.521 ms
A second run gives similar results; seems quite repeatable for every
test I ran so I'll just show one run of each.
If I then turn off the write-cache on the drive:
$ sudo hdparm -W 0 /dev/sdb
And repeat, these times show up instead:
Time: 6781.205 ms Time: 6805.271 ms Time: 6947.037 ms Time: 6938.644
ms Time: 7346.838 ms
So there's the problem case reproduced, right on regular old ext3 and
Ubuntu Jaunty: around 7 seconds to create a database, not real impressive.
Applying the last patch you attached, with the cache on, I see this:
Time: 396.105 ms Time: 389.984 ms Time: 469.800 ms Time: 386.043 ms
Time: 441.269 ms
And if I then turn the write cache off, back to slow times, but much better:
Time: 2162.687 ms Time: 2174.057 ms Time: 2215.785 ms Time: 2174.100
ms Time: 2190.811 ms
That makes the average times I'm seeing on my server:
HEAD Cached: 657 ms Uncached: 6964 ms
Patched Cached: 417 ms Uncached: 2183 ms
Modest speedup even with a caching drive, and a huge speedup in the case
when you have one with slow fsync. Looks to me that if you address
Tom's concern about documentation and function naming, comitting this
patch will certainly deliver as promised on the performance side. Maybe
2 seconds is still too long for some people, but it's at least a whole
lot better.
--
Greg Smith 2ndQuadrant Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.co
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
@ 2010-01-19 15:03 ` Andres Freund <andres@anarazel.de>
3 siblings, 0 replies; 70+ messages in thread
From: Andres Freund @ 2010-01-19 15:03 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Greg Stark <gsstark@mit.edu>; pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tuesday 19 January 2010 15:52:25 Greg Stark wrote:
> On Mon, Jan 18, 2010 at 4:35 PM, Greg Stark <gsstark@mit.edu> wrote:
> > Looking at this patch for the commitfest I have a few questions.
>
> So I've touched this patch up a bit:
>
> 1) moved the posix_fadvise call to a new fd.c function
> pg_fsync_start(fd,offset,nbytes) which initiates an fsync without
> waiting on it. Currently it's only implemented with
> posix_fadvise(DONT_NEED) but I want to look into using sync_file_range
> in the future -- it looks like this call might be good enough for our
> checkpoints.
>
> 2) advised each 64k chunk as we write it which should avoid poisoning
> the cache if you do a large create database on an active system.
>
> 3) added the promised but afaict missing fsync of the directory -- i
> think we should actually backpatch this.
Yes, that was a bit stupid from me - I added the fsync for directories which
get recursed into (by not checking if its a file) but not for the uppermost
level.
So all directories should get fsynced right now but the topmost one.
I will review the patch later when I finally will have some time off again...
~4h.
Thanks!
Andres
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
@ 2010-01-19 15:25 ` Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
3 siblings, 1 reply; 70+ messages in thread
From: Tom Lane @ 2010-01-19 15:25 UTC (permalink / raw)
To: Greg Stark <gsstark@mit.edu>; +Cc: Andres Freund <andres@anarazel.de>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
Greg Stark <gsstark@mit.edu> writes:
> 1) moved the posix_fadvise call to a new fd.c function
> pg_fsync_start(fd,offset,nbytes) which initiates an fsync without
> waiting on it. Currently it's only implemented with
> posix_fadvise(DONT_NEED) but I want to look into using sync_file_range
> in the future -- it looks like this call might be good enough for our
> checkpoints.
That function *seriously* needs documentation, in particular the fact
that it's a no-op on machines without the right kernel call. The name
you've chosen is very bad for those semantics. I'd pick something
else myself. Maybe "pg_start_data_flush" or something like that?
Other than that quibble it seems basically sane.
regards, tom lane
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
@ 2010-01-29 18:56 ` Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
0 siblings, 1 reply; 70+ messages in thread
From: Greg Stark @ 2010-01-29 18:56 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andres Freund <andres@anarazel.de>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tue, Jan 19, 2010 at 3:25 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> That function *seriously* needs documentation, in particular the fact
> that it's a no-op on machines without the right kernel call. The name
> you've chosen is very bad for those semantics. I'd pick something
> else myself. Maybe "pg_start_data_flush" or something like that?
>
I would like to make one token argument in favour of the name I
picked. If it doesn't convince I'll change it since we can always
revisit the API down the road.
I envision having two function calls, pg_fsync_start() and
pg_fsync_finish(). The latter will wait until the data synced in the
first call is actually synced. The fall-back if there's no
implementation of this would be for fsync_start() to be a noop (or
something unreliable like posix_fadvise) and fsync_finish() to just be
a regular fsync.
I think we can accomplish this with sync_file_range() but I need to
read up on how it actually works a bit more. In this case it doesn't
make a difference since when we call fsync_finish() it's going to be
for the entire file and nothing else will have been writing to these
files. But for wal writing and checkpointing it might have very
different performance characteristics.
The big objection to this is that then we don't really have an api for
FADV_DONT_NEED which is more about cache policy than about syncing to
disk. So for example a sequential scan might want to indicate that it
isn't planning on reading the buffers it's churning through but
doesn't want to force them to be written sooner than otherwise and is
never going to call fsync_finish().
--
greg
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
@ 2010-02-02 17:36 ` Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 70+ messages in thread
From: Robert Haas @ 2010-02-02 17:36 UTC (permalink / raw)
To: Greg Stark <gsstark@mit.edu>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Andres Freund <andres@anarazel.de>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Fri, Jan 29, 2010 at 1:56 PM, Greg Stark <gsstark@mit.edu> wrote:
> On Tue, Jan 19, 2010 at 3:25 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> That function *seriously* needs documentation, in particular the fact
>> that it's a no-op on machines without the right kernel call. The name
>> you've chosen is very bad for those semantics. I'd pick something
>> else myself. Maybe "pg_start_data_flush" or something like that?
>>
>
> I would like to make one token argument in favour of the name I
> picked. If it doesn't convince I'll change it since we can always
> revisit the API down the road.
>
> I envision having two function calls, pg_fsync_start() and
> pg_fsync_finish(). The latter will wait until the data synced in the
> first call is actually synced. The fall-back if there's no
> implementation of this would be for fsync_start() to be a noop (or
> something unreliable like posix_fadvise) and fsync_finish() to just be
> a regular fsync.
>
> I think we can accomplish this with sync_file_range() but I need to
> read up on how it actually works a bit more. In this case it doesn't
> make a difference since when we call fsync_finish() it's going to be
> for the entire file and nothing else will have been writing to these
> files. But for wal writing and checkpointing it might have very
> different performance characteristics.
>
> The big objection to this is that then we don't really have an api for
> FADV_DONT_NEED which is more about cache policy than about syncing to
> disk. So for example a sequential scan might want to indicate that it
> isn't planning on reading the buffers it's churning through but
> doesn't want to force them to be written sooner than otherwise and is
> never going to call fsync_finish().
I took a look at this patch today and I agree with Tom that
pg_fsync_start() is a very confusing name. I don't know what the
right name is, but this doesn't fsync so I don't think it shuld have
fsync in the name. Maybe something like pg_advise_abandon() or
pg_abandon_cache(). The current name is really wishful thinking:
you're hoping that it will make the kernel start the fsync, but it
might not. I think pg_start_data_flush() is similarly optimistic.
...Robert
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
@ 2010-02-02 17:43 ` Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 70+ messages in thread
From: Andres Freund @ 2010-02-02 17:43 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Greg Stark <gsstark@mit.edu>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tuesday 02 February 2010 18:36:12 Robert Haas wrote:
> On Fri, Jan 29, 2010 at 1:56 PM, Greg Stark <gsstark@mit.edu> wrote:
> > On Tue, Jan 19, 2010 at 3:25 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >> That function *seriously* needs documentation, in particular the fact
> >> that it's a no-op on machines without the right kernel call. The name
> >> you've chosen is very bad for those semantics. I'd pick something
> >> else myself. Maybe "pg_start_data_flush" or something like that?
> >
> > I would like to make one token argument in favour of the name I
> > picked. If it doesn't convince I'll change it since we can always
> > revisit the API down the road.
> >
> > I envision having two function calls, pg_fsync_start() and
> > pg_fsync_finish(). The latter will wait until the data synced in the
> > first call is actually synced. The fall-back if there's no
> > implementation of this would be for fsync_start() to be a noop (or
> > something unreliable like posix_fadvise) and fsync_finish() to just be
> > a regular fsync.
> >
> > I think we can accomplish this with sync_file_range() but I need to
> > read up on how it actually works a bit more. In this case it doesn't
> > make a difference since when we call fsync_finish() it's going to be
> > for the entire file and nothing else will have been writing to these
> > files. But for wal writing and checkpointing it might have very
> > different performance characteristics.
> >
> > The big objection to this is that then we don't really have an api for
> > FADV_DONT_NEED which is more about cache policy than about syncing to
> > disk. So for example a sequential scan might want to indicate that it
> > isn't planning on reading the buffers it's churning through but
> > doesn't want to force them to be written sooner than otherwise and is
> > never going to call fsync_finish().
>
> I took a look at this patch today and I agree with Tom that
> pg_fsync_start() is a very confusing name. I don't know what the
> right name is, but this doesn't fsync so I don't think it shuld have
> fsync in the name. Maybe something like pg_advise_abandon() or
> pg_abandon_cache(). The current name is really wishful thinking:
> you're hoping that it will make the kernel start the fsync, but it
> might not. I think pg_start_data_flush() is similarly optimistic.
What about: pg_fsync_prepare(). That gives the reason why were doing that and
doesnt promise that it is actually doing an fsync.
I dislike really having "cache" in the name, because the primary aim is not to
discard the cache...
Andres
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
@ 2010-02-02 17:50 ` Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
0 siblings, 1 reply; 70+ messages in thread
From: Tom Lane @ 2010-02-02 17:50 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Robert Haas <robertmhaas@gmail.com>; Greg Stark <gsstark@mit.edu>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
Andres Freund <andres@anarazel.de> writes:
> On Tuesday 02 February 2010 18:36:12 Robert Haas wrote:
>> I took a look at this patch today and I agree with Tom that
>> pg_fsync_start() is a very confusing name. I don't know what the
>> right name is, but this doesn't fsync so I don't think it shuld have
>> fsync in the name. Maybe something like pg_advise_abandon() or
>> pg_abandon_cache(). The current name is really wishful thinking:
>> you're hoping that it will make the kernel start the fsync, but it
>> might not. I think pg_start_data_flush() is similarly optimistic.
> What about: pg_fsync_prepare().
prepare_for_fsync()?
regards, tom lane
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
@ 2010-02-02 18:14 ` Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 70+ messages in thread
From: Robert Haas @ 2010-02-02 18:14 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andres Freund <andres@anarazel.de>; Greg Stark <gsstark@mit.edu>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tue, Feb 2, 2010 at 12:50 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Andres Freund <andres@anarazel.de> writes:
>> On Tuesday 02 February 2010 18:36:12 Robert Haas wrote:
>>> I took a look at this patch today and I agree with Tom that
>>> pg_fsync_start() is a very confusing name. I don't know what the
>>> right name is, but this doesn't fsync so I don't think it shuld have
>>> fsync in the name. Maybe something like pg_advise_abandon() or
>>> pg_abandon_cache(). The current name is really wishful thinking:
>>> you're hoping that it will make the kernel start the fsync, but it
>>> might not. I think pg_start_data_flush() is similarly optimistic.
>
>> What about: pg_fsync_prepare().
>
> prepare_for_fsync()?
It still seems mis-descriptive to me. Couldn't the same routine be
used simply to abandon undirtied data that we no longer care about
caching?
...Robert
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
@ 2010-02-02 18:34 ` Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
0 siblings, 1 reply; 70+ messages in thread
From: Andres Freund @ 2010-02-02 18:34 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Greg Stark <gsstark@mit.edu>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tuesday 02 February 2010 19:14:40 Robert Haas wrote:
> On Tue, Feb 2, 2010 at 12:50 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > Andres Freund <andres@anarazel.de> writes:
> >> On Tuesday 02 February 2010 18:36:12 Robert Haas wrote:
> >>> I took a look at this patch today and I agree with Tom that
> >>> pg_fsync_start() is a very confusing name. I don't know what the
> >>> right name is, but this doesn't fsync so I don't think it shuld have
> >>> fsync in the name. Maybe something like pg_advise_abandon() or
> >>> pg_abandon_cache(). The current name is really wishful thinking:
> >>> you're hoping that it will make the kernel start the fsync, but it
> >>> might not. I think pg_start_data_flush() is similarly optimistic.
> >>
> >> What about: pg_fsync_prepare().
> >
> > prepare_for_fsync()?
>
> It still seems mis-descriptive to me. Couldn't the same routine be
> used simply to abandon undirtied data that we no longer care about
> caching?
For now it could - but it very well might be converted to sync_file_range or
similar, which would have different "sideeffects".
As the potential code duplication is rather small I would prefer to describe
the prime effect not the sideeffects...
Andres
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
@ 2010-02-02 19:06 ` Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:08 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 2 replies; 70+ messages in thread
From: Robert Haas @ 2010-02-02 19:06 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Greg Stark <gsstark@mit.edu>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tue, Feb 2, 2010 at 1:34 PM, Andres Freund <andres@anarazel.de> wrote:
> For now it could - but it very well might be converted to sync_file_range or
> similar, which would have different "sideeffects".
>
> As the potential code duplication is rather small I would prefer to describe
> the prime effect not the sideeffects...
Hmm, in that case, I think the problem is that this function has no
comment explaining its intended charter.
...Robert
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
@ 2010-02-02 19:08 ` Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 70+ messages in thread
From: Andres Freund @ 2010-02-02 19:08 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Greg Stark <gsstark@mit.edu>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tuesday 02 February 2010 20:06:32 Robert Haas wrote:
> On Tue, Feb 2, 2010 at 1:34 PM, Andres Freund <andres@anarazel.de> wrote:
> > For now it could - but it very well might be converted to sync_file_range
> > or similar, which would have different "sideeffects".
> >
> > As the potential code duplication is rather small I would prefer to
> > describe the prime effect not the sideeffects...
>
> Hmm, in that case, I think the problem is that this function has no
> comment explaining its intended charter.
I agree there. Greg, do you want to update the patch with some comments or
shall I?
Andres
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
@ 2010-02-02 19:33 ` Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
1 sibling, 1 reply; 70+ messages in thread
From: Tom Lane @ 2010-02-02 19:33 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Greg Stark <gsstark@mit.edu>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
Robert Haas <robertmhaas@gmail.com> writes:
> Hmm, in that case, I think the problem is that this function has no
> comment explaining its intended charter.
That's certainly a big problem, but a comment won't fix the fact that
the name is misleading. We need both a comment and a name change.
regards, tom lane
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
@ 2010-02-02 19:45 ` Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
0 siblings, 1 reply; 70+ messages in thread
From: Robert Haas @ 2010-02-02 19:45 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andres Freund <andres@anarazel.de>; Greg Stark <gsstark@mit.edu>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tue, Feb 2, 2010 at 2:33 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> Hmm, in that case, I think the problem is that this function has no
>> comment explaining its intended charter.
>
> That's certainly a big problem, but a comment won't fix the fact that
> the name is misleading. We need both a comment and a name change.
I think you're probably right, but it's not clear what the new name
should be until we have a comment explaining what the function is
responsible for.
...Robert
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
@ 2010-02-03 11:53 ` Greg Stark <gsstark@mit.edu>
2010-02-03 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
0 siblings, 2 replies; 70+ messages in thread
From: Greg Stark @ 2010-02-03 11:53 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Andres Freund <andres@anarazel.de>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tue, Feb 2, 2010 at 7:45 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> I think you're probably right, but it's not clear what the new name
> should be until we have a comment explaining what the function is
> responsible for.
So I wrote some comments but wasn't going to repost the patch with the
unchanged name without explanation... But I think you're right though
I was looking at it the other way around. I want to have an API for a
two-stage sync and of course if I do that I'll comment it to explain
that clearly.
The gist of the comments was that the function is preparing to fsync
to initiate the i/o early and allow the later fsync to fast -- but
also at the same time have the beneficial side-effect of avoiding
cache poisoning. It's not clear that the two are necessarily linked
though. Perhaps we need two separate apis, though it'll be hard to
keep them separate on all platforms.
--
greg
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
@ 2010-02-03 12:03 ` Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 70+ messages in thread
From: Andres Freund @ 2010-02-03 12:03 UTC (permalink / raw)
To: Greg Stark <gsstark@mit.edu>; +Cc: Robert Haas <robertmhaas@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On 02/03/10 12:53, Greg Stark wrote:
> On Tue, Feb 2, 2010 at 7:45 PM, Robert Haas<robertmhaas@gmail.com> wrote:
>> I think you're probably right, but it's not clear what the new name
>> should be until we have a comment explaining what the function is
>> responsible for.
>
> So I wrote some comments but wasn't going to repost the patch with the
> unchanged name without explanation... But I think you're right though
> I was looking at it the other way around. I want to have an API for a
> two-stage sync and of course if I do that I'll comment it to explain
> that clearly.
>
> The gist of the comments was that the function is preparing to fsync
> to initiate the i/o early and allow the later fsync to fast -- but
> also at the same time have the beneficial side-effect of avoiding
> cache poisoning. It's not clear that the two are necessarily linked
> though. Perhaps we need two separate apis, though it'll be hard to
> keep them separate on all platforms.
I vote for two seperate apis - sure, there will be some unfortunate
overlap for most unixoid platforms but its sure better possibly to allow
adding more platforms later at a centralized place than having to
analyze every place where the api is used.
Andres
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
@ 2010-02-03 13:42 ` Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 70+ messages in thread
From: Robert Haas @ 2010-02-03 13:42 UTC (permalink / raw)
To: Greg Stark <gsstark@mit.edu>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Andres Freund <andres@anarazel.de>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Wed, Feb 3, 2010 at 6:53 AM, Greg Stark <gsstark@mit.edu> wrote:
> On Tue, Feb 2, 2010 at 7:45 PM, Robert Haas <robertmhaas@gmail.com> wrote:
>> I think you're probably right, but it's not clear what the new name
>> should be until we have a comment explaining what the function is
>> responsible for.
>
> So I wrote some comments but wasn't going to repost the patch with the
> unchanged name without explanation... But I think you're right though
> I was looking at it the other way around. I want to have an API for a
> two-stage sync and of course if I do that I'll comment it to explain
> that clearly.
>
> The gist of the comments was that the function is preparing to fsync
> to initiate the i/o early and allow the later fsync to fast -- but
> also at the same time have the beneficial side-effect of avoiding
> cache poisoning. It's not clear that the two are necessarily linked
> though. Perhaps we need two separate apis, though it'll be hard to
> keep them separate on all platforms.
Well, maybe we should start with a discussion of what kernel calls
you're aware of on different platforms and then we could try to put an
API around it. I mean, right now all you've got is
POSIX_FADV_DONTNEED, so given just that I feel like the API could
simply be pg_dontneed() or something. It's hard to design a general
framework based on one example.
...Robert
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
@ 2010-02-03 14:19 ` Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
0 siblings, 1 reply; 70+ messages in thread
From: Andres Freund @ 2010-02-03 14:19 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Greg Stark <gsstark@mit.edu>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On 02/03/10 14:42, Robert Haas wrote:
> On Wed, Feb 3, 2010 at 6:53 AM, Greg Stark<gsstark@mit.edu> wrote:
>> On Tue, Feb 2, 2010 at 7:45 PM, Robert Haas<robertmhaas@gmail.com> wrote:
>>> I think you're probably right, but it's not clear what the new name
>>> should be until we have a comment explaining what the function is
>>> responsible for.
>>
>> So I wrote some comments but wasn't going to repost the patch with the
>> unchanged name without explanation... But I think you're right though
>> I was looking at it the other way around. I want to have an API for a
>> two-stage sync and of course if I do that I'll comment it to explain
>> that clearly.
>>
>> The gist of the comments was that the function is preparing to fsync
>> to initiate the i/o early and allow the later fsync to fast -- but
>> also at the same time have the beneficial side-effect of avoiding
>> cache poisoning. It's not clear that the two are necessarily linked
>> though. Perhaps we need two separate apis, though it'll be hard to
>> keep them separate on all platforms.
>
> Well, maybe we should start with a discussion of what kernel calls
> you're aware of on different platforms and then we could try to put an
> API around it.
In linux there is sync_file_range. On newer Posixish systems one can
emulate that with mmap() and msync() (in batches obviously).
No idea about windows.
Andres
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
@ 2010-02-06 05:03 ` Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 70+ messages in thread
From: Greg Smith @ 2010-02-06 05:03 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Robert Haas <robertmhaas@gmail.com>; Greg Stark <gsstark@mit.edu>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
Andres Freund wrote:
> On 02/03/10 14:42, Robert Haas wrote:
>> Well, maybe we should start with a discussion of what kernel calls
>> you're aware of on different platforms and then we could try to put an
>> API around it.
> In linux there is sync_file_range. On newer Posixish systems one can
> emulate that with mmap() and msync() (in batches obviously).
>
> No idea about windows.
There's a series of parameters you can pass into CreateFile:
http://msdn.microsoft.com/en-us/library/aa363858(VS.85).aspx
A lot of these are already mapped inside of src/port/open.c in a pretty
straightforward way from the POSIX-oriented interface:
O_RDWR,O_WRONLY -> GENERIC_WRITE, GENERIC_READ
O_RANDOM -> FILE_FLAG_RANDOM_ACCESS
O_SEQUENTIAL -> FILE_FLAG_SEQUENTIAL_SCAN
O_SHORT_LIVED -> FILE_ATTRIBUTE_TEMPORARY
O_TEMPORARY -> FILE_FLAG_DELETE_ON_CLOSE
O_DIRECT -> FILE_FLAG_NO_BUFFERING
O_DSYNC -> FILE_FLAG_WRITE_THROUGH
You have to read the whole "Caching Behavior" section to see exactly how
all of those interact, and even then notes like
http://support.microsoft.com/kb/99794 are needed to follow the fine
points of things like FILE_FLAG_NO_BUFFERING vs. FILE_FLAG_WRITE_THROUGH.
So anything that's setting those POSIX open flags better than before is
getting the benefit of that improvement on Windows, too. But that's not
quite the same as the changes using fadvise to provide better targeted
cache control hints.
I'm getting the impression that doing much better on Windows might fall
into the same sort of category as Solaris, where the primary interface
for this sort of thing is to use an AIO implementation instead:
http://msdn.microsoft.com/en-us/library/aa365683(VS.85).aspx
The effective_io_concurrency feature had proof of concept test programs
that worked using AIO, but actually following through on that
implementation would require a major restructuring of how the database
interacts with the OS in terms of reads and writes of blocks. It looks
to me like doing something similar to sync_file_range on Windows would
be similarly difficult.
--
Greg Smith 2ndQuadrant US Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.us
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
@ 2010-02-06 12:03 ` Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
0 siblings, 1 reply; 70+ messages in thread
From: Andres Freund @ 2010-02-06 12:03 UTC (permalink / raw)
To: Greg Smith <greg@2ndquadrant.com>; +Cc: Robert Haas <robertmhaas@gmail.com>; Greg Stark <gsstark@mit.edu>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Saturday 06 February 2010 06:03:30 Greg Smith wrote:
> Andres Freund wrote:
> > On 02/03/10 14:42, Robert Haas wrote:
> >> Well, maybe we should start with a discussion of what kernel calls
> >> you're aware of on different platforms and then we could try to put an
> >> API around it.
> >
> > In linux there is sync_file_range. On newer Posixish systems one can
> > emulate that with mmap() and msync() (in batches obviously).
> >
> > No idea about windows.
> The effective_io_concurrency feature had proof of concept test programs
> that worked using AIO, but actually following through on that
> implementation would require a major restructuring of how the database
> interacts with the OS in terms of reads and writes of blocks. It looks
> to me like doing something similar to sync_file_range on Windows would
> be similarly difficult.
Looking a bit arround it seems one could achieve something approximediately
similar to pg_prepare_fsync() by using
CreateFileMapping && MapViewOfFile && FlushViewOfFile
If I understand it correctly that will flush, but not wait. Unfortunately you
cant event make it wait, so its not possible to implement sync_file_range or
similar fully.
Andres
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
@ 2010-02-07 05:13 ` Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
0 siblings, 1 reply; 70+ messages in thread
From: Robert Haas @ 2010-02-07 05:13 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Greg Smith <greg@2ndquadrant.com>; Greg Stark <gsstark@mit.edu>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Sat, Feb 6, 2010 at 7:03 AM, Andres Freund <andres@anarazel.de> wrote:
> On Saturday 06 February 2010 06:03:30 Greg Smith wrote:
>> Andres Freund wrote:
>> > On 02/03/10 14:42, Robert Haas wrote:
>> >> Well, maybe we should start with a discussion of what kernel calls
>> >> you're aware of on different platforms and then we could try to put an
>> >> API around it.
>> >
>> > In linux there is sync_file_range. On newer Posixish systems one can
>> > emulate that with mmap() and msync() (in batches obviously).
>> >
>> > No idea about windows.
>> The effective_io_concurrency feature had proof of concept test programs
>> that worked using AIO, but actually following through on that
>> implementation would require a major restructuring of how the database
>> interacts with the OS in terms of reads and writes of blocks. It looks
>> to me like doing something similar to sync_file_range on Windows would
>> be similarly difficult.
> Looking a bit arround it seems one could achieve something approximediately
> similar to pg_prepare_fsync() by using
> CreateFileMapping && MapViewOfFile && FlushViewOfFile
>
> If I understand it correctly that will flush, but not wait. Unfortunately you
> cant event make it wait, so its not possible to implement sync_file_range or
> similar fully.
Well it seems that what we're trying to implement is more like
it_would_be_nice_if_you_would_start_syncing_this_file_range_but_its_ok_if_you_dont(),
so maybe that would work.
Anyway, is there something that we can agree on and get committed here
for 9.0, or should we postpone this to 9.1? It seems simple enough
that we ought to be able to get it done, but we're running out of time
and we don't seem to have a clear vision here yet...
...Robert
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
@ 2010-02-07 09:23 ` Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 70+ messages in thread
From: Greg Smith @ 2010-02-07 09:23 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Greg Stark <gsstark@mit.edu>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
Robert Haas wrote:
> Well it seems that what we're trying to implement is more like
> it_would_be_nice_if_you_would_start_syncing_this_file_range_but_its_ok_if_you_dont(),
> so maybe that would work.
>
> Anyway, is there something that we can agree on and get committed here
> for 9.0, or should we postpone this to 9.1? It seems simple enough
> that we ought to be able to get it done, but we're running out of time
> and we don't seem to have a clear vision here yet...
>
This is turning into yet another one of those situations where something
simple and useful is being killed by trying to generalize it way more
than it needs to be, given its current goals and its lack of external
interfaces. There's no catversion bump or API breakage to hinder future
refactoring if this isn't optimally designed internally from day one.
The feature is valuable and there seems at least one spot where it may
be resolving the possibility of a subtle OS interaction bug by being
more thorough in the way that it writes and syncs. The main contention
seems to be over naming and completely optional additional abstraction.
I consider the whole "let's make this cover every type of complicated
sync on every platform" goal interesting and worthwhile, but it's
completely optional for this release. The stuff being fretted over now
is ultimately an internal interface that can be refactored at will in
later releases with no user impact.
If the goal here could be shifted back to finding the minimal level of
abstraction that doesn't seem completely wrong, then updating the
function names and comments to match that more closely, this could
return to committable. That's all I thought was left to do when I moved
it to "ready for committer", and as far as I've seen this expanded scope
of discussion has just moved backwards from that point.
--
Greg Smith 2ndQuadrant Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.com
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
@ 2010-02-07 16:24 ` Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
0 siblings, 1 reply; 70+ messages in thread
From: Tom Lane @ 2010-02-07 16:24 UTC (permalink / raw)
To: Greg Smith <greg@2ndquadrant.com>; +Cc: Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Greg Stark <gsstark@mit.edu>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
Greg Smith <greg@2ndquadrant.com> writes:
> This is turning into yet another one of those situations where something
> simple and useful is being killed by trying to generalize it way more
> than it needs to be, given its current goals and its lack of external
> interfaces. There's no catversion bump or API breakage to hinder future
> refactoring if this isn't optimally designed internally from day one.
I agree that it's too late in the cycle for any major redesign of the
patch. But is it too much to ask to use a less confusing name for the
function?
regards, tom lane
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
@ 2010-02-07 18:23 ` Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 70+ messages in thread
From: Robert Haas @ 2010-02-07 18:23 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Greg Smith <greg@2ndquadrant.com>; Andres Freund <andres@anarazel.de>; Greg Stark <gsstark@mit.edu>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Sun, Feb 7, 2010 at 11:24 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Greg Smith <greg@2ndquadrant.com> writes:
>> This is turning into yet another one of those situations where something
>> simple and useful is being killed by trying to generalize it way more
>> than it needs to be, given its current goals and its lack of external
>> interfaces. There's no catversion bump or API breakage to hinder future
>> refactoring if this isn't optimally designed internally from day one.
>
> I agree that it's too late in the cycle for any major redesign of the
> patch. But is it too much to ask to use a less confusing name for the
> function?
+1. Let's just rename the thing, add some comments, and call it good.
...Robert
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
@ 2010-02-07 18:27 ` Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 70+ messages in thread
From: Andres Freund @ 2010-02-07 18:27 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Greg Smith <greg@2ndquadrant.com>; Greg Stark <gsstark@mit.edu>; pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Sunday 07 February 2010 19:23:10 Robert Haas wrote:
> On Sun, Feb 7, 2010 at 11:24 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > Greg Smith <greg@2ndquadrant.com> writes:
> >> This is turning into yet another one of those situations where something
> >> simple and useful is being killed by trying to generalize it way more
> >> than it needs to be, given its current goals and its lack of external
> >> interfaces. There's no catversion bump or API breakage to hinder future
> >> refactoring if this isn't optimally designed internally from day one.
> >
> > I agree that it's too late in the cycle for any major redesign of the
> > patch. But is it too much to ask to use a less confusing name for the
> > function?
>
> +1. Let's just rename the thing, add some comments, and call it good.
Will post a updated patch in the next hours unless somebody beats me too it.
Andres
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
@ 2010-02-08 01:31 ` Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
0 siblings, 1 reply; 70+ messages in thread
From: Andres Freund @ 2010-02-08 01:31 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Robert Haas <robertmhaas@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; Greg Smith <greg@2ndquadrant.com>; Greg Stark <gsstark@mit.edu>; pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Sunday 07 February 2010 19:27:02 Andres Freund wrote:
> On Sunday 07 February 2010 19:23:10 Robert Haas wrote:
> > On Sun, Feb 7, 2010 at 11:24 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > > Greg Smith <greg@2ndquadrant.com> writes:
> > >> This is turning into yet another one of those situations where
> > >> something simple and useful is being killed by trying to generalize
> > >> it way more than it needs to be, given its current goals and its lack
> > >> of external interfaces. There's no catversion bump or API breakage
> > >> to hinder future refactoring if this isn't optimally designed
> > >> internally from day one.
> > >
> > > I agree that it's too late in the cycle for any major redesign of the
> > > patch. But is it too much to ask to use a less confusing name for the
> > > function?
> >
> > +1. Let's just rename the thing, add some comments, and call it good.
>
> Will post a updated patch in the next hours unless somebody beats me too
> it.
Here we go.
I left the name at my suggestion pg_fsync_prepare instead of Tom's
prepare_for_fsync because it seemed more consistend with the naming in the
rest of the file. Obviously feel free to adjust.
I personally think the fsync on the directory should be added to the stable
branches - other opinions?
If wanted I can prepare patches for that.
Andres
Attachments:
faster_createdb_v3.patch (text/x-patch, 5.8K) download
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 7ffa2eb..bc5753a 100644
*** a/src/backend/storage/file/fd.c
--- b/src/backend/storage/file/fd.c
*************** pg_fdatasync(int fd)
*** 320,325 ****
--- 320,361 ----
}
/*
+ * pg_fsync_prepare --- try to make a later fsync on the same file faster
+ *
+ * A call to this function does not guarantee anything!
+ *
+ * The idea is to tell the kernel to write out its cache so that a
+ * fsync later on has less to write out synchronously. This allows
+ * that write requests get reordered more freely.
+ *
+ * In the current implementation this has the additional effect of
+ * dropping the cache in that region and thus can be used to avoid
+ * cache poisoning. This may or may not be wanted.
+ *
+ * XXX: Ideally this API would use sync_file_range (or similar on
+ * platforms other than linux) and a seperate one for cache
+ * control. We are not there yet.
+ *
+ * Look at the thread below 200912282354.51892.andres@anarazel.de in
+ * pgsql-hackers for a longer discussion.
+ */
+ int
+ pg_fsync_prepare(int fd, off_t offset, off_t amount)
+ {
+ if (enableFsync)
+ {
+ #if defined(USE_POSIX_FADVISE) && defined(POSIX_FADV_DONTNEED)
+ return posix_fadvise(fd, offset, amount, POSIX_FADV_DONTNEED);
+ #else
+ return 0;
+ #endif
+ }
+ else
+ return 0;
+ }
+
+
+ /*
* InitFileAccess --- initialize this module during backend startup
*
* This is called during either normal or standalone backend start.
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index 21cb024..b1a4b49 100644
*** a/src/include/storage/fd.h
--- b/src/include/storage/fd.h
*************** extern int pg_fsync_no_writethrough(int
*** 99,104 ****
--- 99,106 ----
extern int pg_fsync_writethrough(int fd);
extern int pg_fdatasync(int fd);
+ extern int prepare_for_fsync(int fd, off_t offset, off_t amount);
+
/* Filename components for OpenTemporaryFile */
#define PG_TEMP_FILES_DIR "pgsql_tmp"
#define PG_TEMP_FILE_PREFIX "pgsql_tmp"
diff --git a/src/port/copydir.c b/src/port/copydir.c
index 72fbf36..eef3cfb 100644
*** a/src/port/copydir.c
--- b/src/port/copydir.c
***************
*** 37,42 ****
--- 37,43 ----
static void copy_file(char *fromfile, char *tofile);
+ static void fsync_fname(char *fname);
/*
*************** copydir(char *fromdir, char *todir, bool
*** 64,69 ****
--- 65,73 ----
(errcode_for_file_access(),
errmsg("could not open directory \"%s\": %m", fromdir)));
+ /*
+ * Copy all the files
+ */
while ((xlde = ReadDir(xldir, fromdir)) != NULL)
{
struct stat fst;
*************** copydir(char *fromdir, char *todir, bool
*** 89,96 ****
--- 93,127 ----
else if (S_ISREG(fst.st_mode))
copy_file(fromfile, tofile);
}
+ FreeDir(xldir);
+
+ /*
+ * Be paranoid here and fsync all files to ensure we catch problems.
+ */
+ xldir = AllocateDir(fromdir);
+ if (xldir == NULL)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open directory \"%s\": %m", fromdir)));
+
+ while ((xlde = ReadDir(xldir, fromdir)) != NULL)
+ {
+ if (strcmp(xlde->d_name, ".") == 0 ||
+ strcmp(xlde->d_name, "..") == 0)
+ continue;
+ snprintf(tofile, MAXPGPATH, "%s/%s", todir, xlde->d_name);
+ fsync_fname(tofile);
+ }
FreeDir(xldir);
+
+ /* It's important to fsync the destination directory itself as
+ * individual file fsyncs don't guarantee that the directory entry
+ * for the file is synced. Recent versions of ext4 have made the
+ * window much wider but it's been true for ext3 and other
+ * filesyetems in the past
+ */
+ fsync_fname(todir);
}
/*
*************** copy_file(char *fromfile, char *tofile)
*** 103,108 ****
--- 134,140 ----
int srcfd;
int dstfd;
int nbytes;
+ off_t offset;
/* Use palloc to ensure we get a maxaligned buffer */
#define COPY_BUF_SIZE (8 * BLCKSZ)
*************** copy_file(char *fromfile, char *tofile)
*** 128,134 ****
/*
* Do the data copying.
*/
! for (;;)
{
nbytes = read(srcfd, buffer, COPY_BUF_SIZE);
if (nbytes < 0)
--- 160,166 ----
/*
* Do the data copying.
*/
! for (offset=0; ; offset+=nbytes)
{
nbytes = read(srcfd, buffer, COPY_BUF_SIZE);
if (nbytes < 0)
*************** copy_file(char *fromfile, char *tofile)
*** 147,161 ****
(errcode_for_file_access(),
errmsg("could not write to file \"%s\": %m", tofile)));
}
- }
! /*
! * Be paranoid here to ensure we catch problems.
! */
! if (pg_fsync(dstfd) != 0)
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not fsync file \"%s\": %m", tofile)));
if (close(dstfd))
ereport(ERROR,
--- 179,197 ----
(errcode_for_file_access(),
errmsg("could not write to file \"%s\": %m", tofile)));
}
! /*
! * Start flushing the dirty memory early.
! *
! * Empirical results seem to show a big speed benefit on
! * recent Linux kernels.
! * As a side effect this avoids cache pollution by throwing
! * away the cache of the new file.
! * XXX: It might be more sensible to do that on the origin
! * than the source.
! */
! pg_fsync_prepare(dstfd, offset, nbytes);
! }
if (close(dstfd))
ereport(ERROR,
*************** copy_file(char *fromfile, char *tofile)
*** 166,168 ****
--- 202,226 ----
pfree(buffer);
}
+
+ /*
+ * fsync a file
+ */
+ static void
+ fsync_fname(char *fname)
+ {
+ int fd = BasicOpenFile(fname,
+ O_RDONLY | PG_BINARY,
+ S_IRUSR | S_IWUSR);
+
+ if (fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m", fname)));
+
+ if (pg_fsync(fd) != 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not fsync file \"%s\": %m", fname)));
+ close(fd);
+ }
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
@ 2010-02-08 03:09 ` Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
0 siblings, 1 reply; 70+ messages in thread
From: Alvaro Herrera @ 2010-02-08 03:09 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Robert Haas <robertmhaas@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; Greg Smith <greg@2ndquadrant.com>; Greg Stark <gsstark@mit.edu>; pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
Andres Freund escribió:
> I personally think the fsync on the directory should be added to the stable
> branches - other opinions?
> If wanted I can prepare patches for that.
Yeah, it seems there are two patches here -- one is the addition of
fsync_fname() and the other is the fsync_prepare stuff.
--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
@ 2010-02-08 04:53 ` Robert Haas <robertmhaas@gmail.com>
2010-02-08 07:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-11 02:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
0 siblings, 3 replies; 70+ messages in thread
From: Robert Haas @ 2010-02-08 04:53 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@commandprompt.com>; +Cc: Andres Freund <andres@anarazel.de>; pgsql-hackers@postgresql.org, Tom Lane <tgl@sss.pgh.pa.us>; Greg Smith <greg@2ndquadrant.com>; Greg Stark <gsstark@mit.edu>; pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Sun, Feb 7, 2010 at 10:09 PM, Alvaro Herrera
<alvherre@commandprompt.com> wrote:
> Andres Freund escribió:
>> I personally think the fsync on the directory should be added to the stable
>> branches - other opinions?
>> If wanted I can prepare patches for that.
>
> Yeah, it seems there are two patches here -- one is the addition of
> fsync_fname() and the other is the fsync_prepare stuff.
Andres, you want to take a crack at splitting this up?
...Robert
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
@ 2010-02-08 07:13 ` Andres Freund <andres@anarazel.de>
2 siblings, 0 replies; 70+ messages in thread
From: Andres Freund @ 2010-02-08 07:13 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Alvaro Herrera <alvherre@commandprompt.com>; pgsql-hackers@postgresql.org, Tom Lane <tgl@sss.pgh.pa.us>; Greg Smith <greg@2ndquadrant.com>; Greg Stark <gsstark@mit.edu>; pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Monday 08 February 2010 05:53:23 Robert Haas wrote:
> On Sun, Feb 7, 2010 at 10:09 PM, Alvaro Herrera
>
> <alvherre@commandprompt.com> wrote:
> > Andres Freund escribió:
> >> I personally think the fsync on the directory should be added to the
> >> stable branches - other opinions?
> >> If wanted I can prepare patches for that.
> >
> > Yeah, it seems there are two patches here -- one is the addition of
> > fsync_fname() and the other is the fsync_prepare stuff.
>
> Andres, you want to take a crack at splitting this up?
Will do. Later today or tomorrow morning.
Andres
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
@ 2010-02-08 18:34 ` Greg Stark <gsstark@mit.edu>
2010-02-08 19:29 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2 siblings, 1 reply; 70+ messages in thread
From: Greg Stark @ 2010-02-08 18:34 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Alvaro Herrera <alvherre@commandprompt.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers@postgresql.org, Tom Lane <tgl@sss.pgh.pa.us>; Greg Smith <greg@2ndquadrant.com>; pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Mon, Feb 8, 2010 at 4:53 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Sun, Feb 7, 2010 at 10:09 PM, Alvaro Herrera
>> Yeah, it seems there are two patches here -- one is the addition of
>> fsync_fname() and the other is the fsync_prepare stuff.
Sorry, I'm just catching up on my mail from FOSDEM this past weekend.
I had come to the same conclusion as Greg that I might as well just
commit it with Tom's "pg_flush_data()" name and we can decide later if
and when we have pg_fsync_start()/pg_fsync_finish() whether it's worth
keeping two apis or not.
So I was just going to commit it like that but I discovered last week
that I don't have cvs write access set up yet. I'll commit it as soon
as I generate a new ssh key and Dave installs it, etc. I intentionally
picked a small simple patch that nobody was waiting on because I knew
there was a risk of delays like this and the paperwork. I'm nearly
there.
--
greg
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-08 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
@ 2010-02-08 19:29 ` Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 70+ messages in thread
From: Andres Freund @ 2010-02-08 19:29 UTC (permalink / raw)
To: Greg Stark <gsstark@mit.edu>; +Cc: Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@commandprompt.com>; pgsql-hackers@postgresql.org, Tom Lane <tgl@sss.pgh.pa.us>; Greg Smith <greg@2ndquadrant.com>; pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Monday 08 February 2010 19:34:01 Greg Stark wrote:
> On Mon, Feb 8, 2010 at 4:53 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> > On Sun, Feb 7, 2010 at 10:09 PM, Alvaro Herrera
> >
> >> Yeah, it seems there are two patches here -- one is the addition of
> >> fsync_fname() and the other is the fsync_prepare stuff.
>
> Sorry, I'm just catching up on my mail from FOSDEM this past weekend.
>
> I had come to the same conclusion as Greg that I might as well just
> commit it with Tom's "pg_flush_data()" name and we can decide later if
> and when we have pg_fsync_start()/pg_fsync_finish() whether it's worth
> keeping two apis or not.
>
> So I was just going to commit it like that but I discovered last week
> that I don't have cvs write access set up yet. I'll commit it as soon
> as I generate a new ssh key and Dave installs it, etc. I intentionally
> picked a small simple patch that nobody was waiting on because I knew
> there was a risk of delays like this and the paperwork. I'm nearly
> there.
Do you still want me to split the patches into two or do you want to do it
yourself?
One in multiple versions for the directory fsync and another one for 9.0?
Andres
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
@ 2010-02-11 02:27 ` Andres Freund <andres@anarazel.de>
2010-02-12 15:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2 siblings, 1 reply; 70+ messages in thread
From: Andres Freund @ 2010-02-11 02:27 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@commandprompt.com>; Tom Lane <tgl@sss.pgh.pa.us>; Greg Smith <greg@2ndquadrant.com>; Greg Stark <gsstark@mit.edu>; pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Monday 08 February 2010 05:53:23 Robert Haas wrote:
> On Sun, Feb 7, 2010 at 10:09 PM, Alvaro Herrera
>
> <alvherre@commandprompt.com> wrote:
> > Andres Freund escribió:
> >> I personally think the fsync on the directory should be added to the
> >> stable branches - other opinions?
> >> If wanted I can prepare patches for that.
> >
> > Yeah, it seems there are two patches here -- one is the addition of
> > fsync_fname() and the other is the fsync_prepare stuff.
>
> Andres, you want to take a crack at splitting this up?
I hope I didnt duplicate Gregs work, but I didnt hear back from him, so...
Everything <8.1 is hopeless because cp is used there... I didnt see it worth
to replace that. The patch applies cleanly for 8.1 to 8.4 and survives the
regression tests
Given pg's heavy commit model I didnt see a point to split the patch for 9.0
as well...
Andres
Attachments:
directory-fsync-8.1-to-8.4.patch (text/x-patch, 1.1K) download
diff --git a/src/port/copydir.c b/src/port/copydir.c
index 72fbf36..b057ffa 100644
*** a/src/port/copydir.c
--- b/src/port/copydir.c
*************** copydir(char *fromdir, char *todir, bool
*** 50,55 ****
--- 50,56 ----
{
DIR *xldir;
struct dirent *xlde;
+ int dirfd;
char fromfile[MAXPGPATH];
char tofile[MAXPGPATH];
*************** copydir(char *fromdir, char *todir, bool
*** 91,96 ****
--- 92,116 ----
}
FreeDir(xldir);
+
+ /*
+ * fsync the directory to make sure data has reached the
+ * disk. While needed by most filesystems, the window got bigger
+ * with newer ones like ext4.
+ */
+ dirfd = BasicOpenFile(todir,
+ O_RDONLY | PG_BINARY,
+ S_IRUSR | S_IWUSR);
+ if(dirfd == -1)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open directory for fsync \"%s\": %m", todir)));
+
+ if(pg_fsync(dirfd) == -1)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not fsync directory \"%s\": %m", todir)));
+ close(dirfd);
}
/*
* Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-11 02:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
@ 2010-02-12 15:49 ` Robert Haas <robertmhaas@gmail.com>
2010-02-14 14:03 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
0 siblings, 1 reply; 70+ messages in thread
From: Robert Haas @ 2010-02-12 15:49 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Greg Stark <gsstark@mit.edu>
On Wed, Feb 10, 2010 at 9:27 PM, Andres Freund <andres@anarazel.de> wrote:
> On Monday 08 February 2010 05:53:23 Robert Haas wrote:
>> On Sun, Feb 7, 2010 at 10:09 PM, Alvaro Herrera
>>
>> <alvherre@commandprompt.com> wrote:
>> > Andres Freund escribió:
>> >> I personally think the fsync on the directory should be added to the
>> >> stable branches - other opinions?
>> >> If wanted I can prepare patches for that.
>> >
>> > Yeah, it seems there are two patches here -- one is the addition of
>> > fsync_fname() and the other is the fsync_prepare stuff.
>>
>> Andres, you want to take a crack at splitting this up?
> I hope I didnt duplicate Gregs work, but I didnt hear back from him, so...
>
> Everything <8.1 is hopeless because cp is used there... I didnt see it worth
> to replace that. The patch applies cleanly for 8.1 to 8.4 and survives the
> regression tests
>
> Given pg's heavy commit model I didnt see a point to split the patch for 9.0
> as well...
I'd probably argue for committing this patch to both HEAD and the
back-branches, and doing a second commit with the remaining stuff for
HEAD only, but I don't care very much.
Greg Stark, have you managed to get your access issues sorted out? If
you like, I can do the actual commit on this one.
...Robert
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-11 02:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-12 15:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
@ 2010-02-14 14:03 ` Greg Stark <gsstark@mit.edu>
2010-02-14 15:31 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
0 siblings, 1 reply; 70+ messages in thread
From: Greg Stark @ 2010-02-14 14:03 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; pgsql-hackers
On Fri, Feb 12, 2010 at 3:49 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> Greg Stark, have you managed to get your access issues sorted out? If
Yep, will look at this today.
--
greg
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-11 02:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-12 15:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-14 14:03 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
@ 2010-02-14 15:31 ` Greg Stark <gsstark@mit.edu>
2010-02-14 17:11 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-14 20:57 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
0 siblings, 2 replies; 70+ messages in thread
From: Greg Stark @ 2010-02-14 15:31 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; pgsql-hackers
On Sun, Feb 14, 2010 at 2:03 PM, Greg Stark <gsstark@mit.edu> wrote:
> On Fri, Feb 12, 2010 at 3:49 PM, Robert Haas <robertmhaas@gmail.com> wrote:
>> Greg Stark, have you managed to get your access issues sorted out? If
>
> Yep, will look at this today.
So I think we have a bigger problem than just copydir.c. It seems to
me we should be fsyncing the table space data directories on every
checkpoint. Otherwise any newly created relations or removed relations
could disappear even though the data in them was fsynced. I'm thinking
I should add an _mdfd_opentblspc(reln) call which returns a file
descriptor for the tablespace and have mdsync() use that to sync the
directory whenever it fsyncs a relation. It would be nice to remember
which tablespaces have been fsynced and only fsync them once though,
that would need another hash table just for tablespaces.
We probably also need to fsync the pg_xlog directory every time we
create or rename an xlog segment.
Are there any other places we do directory operations which we need to
be permanent?
--
greg
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-11 02:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-12 15:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-14 14:03 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 15:31 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
@ 2010-02-14 17:11 ` Tom Lane <tgl@sss.pgh.pa.us>
2010-02-14 17:27 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 70+ messages in thread
From: Tom Lane @ 2010-02-14 17:11 UTC (permalink / raw)
To: Greg Stark <gsstark@mit.edu>; +Cc: Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers
Greg Stark <gsstark@mit.edu> writes:
> So I think we have a bigger problem than just copydir.c. It seems to
> me we should be fsyncing the table space data directories on every
> checkpoint.
Is there any evidence that anyone anywhere has ever lost data because
of a lack of directory fsyncs? I sure don't recall any bug reports
that seem to match that theory.
It seems to me that we're talking about a huge hit in both code
complexity and performance to deal with a problem that doesn't actually
occur in the field; and which furthermore is trivially solved on any
modern filesystem by choosing the right filesystem options. Why don't
we just document those options, instead?
regards, tom lane
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-11 02:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-12 15:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-14 14:03 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 15:31 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 17:11 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
@ 2010-02-14 17:27 ` Andres Freund <andres@anarazel.de>
2010-02-14 17:37 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 70+ messages in thread
From: Andres Freund @ 2010-02-14 17:27 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Greg Stark <gsstark@mit.edu>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
On Sunday 14 February 2010 18:11:39 Tom Lane wrote:
> Greg Stark <gsstark@mit.edu> writes:
> > So I think we have a bigger problem than just copydir.c. It seems to
> > me we should be fsyncing the table space data directories on every
> > checkpoint.
>
> Is there any evidence that anyone anywhere has ever lost data because
> of a lack of directory fsyncs? I sure don't recall any bug reports
> that seem to match that theory.
I have actually seen the issue during create database at least. In a
virtualized hw though...
~1GB template database, lots and lots of small tables, the crash occured maybe
a minute after CREATE DB, filesystem was xfs, kernel 2.6.30.y.
> It seems to me that we're talking about a huge hit in both code
> complexity and performance to deal with a problem that doesn't actually
> occur in the field; and which furthermore is trivially solved on any
> modern filesystem by choosing the right filesystem options. Why don't
> we just document those options, instead?
Which options would that be? I am not aware that there any for any of the
recent linux filesystems.
Well, except "sync" that is, but that sure would be more of a performance hit
than fsyncing the directory...
Andres
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-11 02:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-12 15:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-14 14:03 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 15:31 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 17:11 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-14 17:27 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
@ 2010-02-14 17:37 ` Tom Lane <tgl@sss.pgh.pa.us>
2010-02-14 20:24 ` Re: Re: Faster CREATE DATABASE by delaying fsync Florian Weimer <fw@deneb.enyo.de>
0 siblings, 1 reply; 70+ messages in thread
From: Tom Lane @ 2010-02-14 17:37 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Greg Stark <gsstark@mit.edu>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
Andres Freund <andres@anarazel.de> writes:
> On Sunday 14 February 2010 18:11:39 Tom Lane wrote:
>> It seems to me that we're talking about a huge hit in both code
>> complexity and performance to deal with a problem that doesn't actually
>> occur in the field; and which furthermore is trivially solved on any
>> modern filesystem by choosing the right filesystem options. Why don't
>> we just document those options, instead?
> Which options would that be? I am not aware that there any for any of the
> recent linux filesystems.
Shouldn't journaling of metadata be sufficient?
regards, tom lane
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-11 02:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-12 15:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-14 14:03 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 15:31 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 17:11 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-14 17:27 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-14 17:37 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
@ 2010-02-14 20:24 ` Florian Weimer <fw@deneb.enyo.de>
2010-02-14 20:41 ` Re: Re: Faster CREATE DATABASE by delaying fsync Mark Mielke <mark@mark.mielke.cc>
0 siblings, 1 reply; 70+ messages in thread
From: Florian Weimer @ 2010-02-14 20:24 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andres Freund <andres@anarazel.de>; Greg Stark <gsstark@mit.edu>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
* Tom Lane:
>> Which options would that be? I am not aware that there any for any of the
>> recent linux filesystems.
>
> Shouldn't journaling of metadata be sufficient?
You also need to enforce ordering between the directory update and the
file update. The file metadata is flushed with fsync(), but the
directory isn't. On some systems, all directory operations are
synchronous, but not on Linux.
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-11 02:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-12 15:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-14 14:03 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 15:31 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 17:11 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-14 17:27 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-14 17:37 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-14 20:24 ` Re: Re: Faster CREATE DATABASE by delaying fsync Florian Weimer <fw@deneb.enyo.de>
@ 2010-02-14 20:41 ` Mark Mielke <mark@mark.mielke.cc>
2010-02-14 20:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 70+ messages in thread
From: Mark Mielke @ 2010-02-14 20:41 UTC (permalink / raw)
To: Florian Weimer <fw@deneb.enyo.de>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Andres Freund <andres@anarazel.de>; Greg Stark <gsstark@mit.edu>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
On 02/14/2010 03:24 PM, Florian Weimer wrote:
> * Tom Lane:
>
>>> Which options would that be? I am not aware that there any for any of the
>>> recent linux filesystems.
>>>
>> Shouldn't journaling of metadata be sufficient?
>>
> You also need to enforce ordering between the directory update and the
> file update. The file metadata is flushed with fsync(), but the
> directory isn't. On some systems, all directory operations are
> synchronous, but not on Linux.
>
dirsync
All directory updates within the filesystem should be
done syn-
chronously. This affects the following system calls:
creat,
link, unlink, symlink, mkdir, rmdir, mknod and rename.
The widely reported problems, though, did not tend to be a problem with
directory changes written too late - but directory changes being written
too early. That is, the directory change is written to disk, but the
file content is not. This is likely because of the "ordered journal"
mode widely used in ext3/ext4 where metadata changes are journalled, but
file pages are not journalled. Therefore, it is important for some
operations, that the file pages are pushed to disk using fsync(file),
before the metadata changes are journalled.
In theory there is some open hole where directory updates need to be
synchronized with file updates, as POSIX doesn't enforce this ordering,
and we can't trust that all file systems implicitly order things
correctly, but in practice, I don't see this sort of problem happening.
If you are concerned, enable dirsync.
Cheers,
mark
--
Mark Mielke<mark@mielke.cc>
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-11 02:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-12 15:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-14 14:03 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 15:31 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 17:11 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-14 17:27 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-14 17:37 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-14 20:24 ` Re: Re: Faster CREATE DATABASE by delaying fsync Florian Weimer <fw@deneb.enyo.de>
2010-02-14 20:41 ` Re: Re: Faster CREATE DATABASE by delaying fsync Mark Mielke <mark@mark.mielke.cc>
@ 2010-02-14 20:49 ` Andres Freund <andres@anarazel.de>
2010-02-15 00:08 ` Re: Re: Faster CREATE DATABASE by delaying fsync Mark Mielke <mark@mark.mielke.cc>
0 siblings, 1 reply; 70+ messages in thread
From: Andres Freund @ 2010-02-14 20:49 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Mark Mielke <mark@mark.mielke.cc>; Florian Weimer <fw@deneb.enyo.de>; Tom Lane <tgl@sss.pgh.pa.us>; Greg Stark <gsstark@mit.edu>; Robert Haas <robertmhaas@gmail.com>
On Sunday 14 February 2010 21:41:02 Mark Mielke wrote:
> On 02/14/2010 03:24 PM, Florian Weimer wrote:
> > * Tom Lane:
> >>> Which options would that be? I am not aware that there any for any of
> >>> the recent linux filesystems.
> >>
> >> Shouldn't journaling of metadata be sufficient?
> >
> > You also need to enforce ordering between the directory update and the
> > file update. The file metadata is flushed with fsync(), but the
> > directory isn't. On some systems, all directory operations are
> > synchronous, but not on Linux.
>
> dirsync
> All directory updates within the filesystem should be
> done syn-
> chronously. This affects the following system calls:
> creat,
> link, unlink, symlink, mkdir, rmdir, mknod and rename.
>
> The widely reported problems, though, did not tend to be a problem with
> directory changes written too late - but directory changes being written
> too early. That is, the directory change is written to disk, but the
> file content is not. This is likely because of the "ordered journal"
> mode widely used in ext3/ext4 where metadata changes are journalled, but
> file pages are not journalled. Therefore, it is important for some
> operations, that the file pages are pushed to disk using fsync(file),
> before the metadata changes are journalled.
Well, but thats not a problem with pg as it fsyncs the file contents.
> In theory there is some open hole where directory updates need to be
> synchronized with file updates, as POSIX doesn't enforce this ordering,
> and we can't trust that all file systems implicitly order things
> correctly, but in practice, I don't see this sort of problem happening.
I can try to reproduce it if you want...
> If you are concerned, enable dirsync.
If the filesystem already behaves that way a fsync on it should be fairly
cheap. If it doesnt behave that way doing it is correct...
Besides there is no reason to fsync the directory before the checkpoint, so
dirsync would require a higher cost than doing it correctly.
Andres
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-11 02:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-12 15:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-14 14:03 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 15:31 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 17:11 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-14 17:27 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-14 17:37 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-14 20:24 ` Re: Re: Faster CREATE DATABASE by delaying fsync Florian Weimer <fw@deneb.enyo.de>
2010-02-14 20:41 ` Re: Re: Faster CREATE DATABASE by delaying fsync Mark Mielke <mark@mark.mielke.cc>
2010-02-14 20:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync Andres Freund <andres@anarazel.de>
@ 2010-02-15 00:08 ` Mark Mielke <mark@mark.mielke.cc>
0 siblings, 0 replies; 70+ messages in thread
From: Mark Mielke @ 2010-02-15 00:08 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Florian Weimer <fw@deneb.enyo.de>; Tom Lane <tgl@sss.pgh.pa.us>; Greg Stark <gsstark@mit.edu>; Robert Haas <robertmhaas@gmail.com>
On 02/14/2010 03:49 PM, Andres Freund wrote:
> On Sunday 14 February 2010 21:41:02 Mark Mielke wrote:
>
>> The widely reported problems, though, did not tend to be a problem with
>> directory changes written too late - but directory changes being written
>> too early. That is, the directory change is written to disk, but the
>> file content is not. This is likely because of the "ordered journal"
>> mode widely used in ext3/ext4 where metadata changes are journalled, but
>> file pages are not journalled. Therefore, it is important for some
>> operations, that the file pages are pushed to disk using fsync(file),
>> before the metadata changes are journalled.
>>
> Well, but thats not a problem with pg as it fsyncs the file contents.
>
Exactly. Not a problem.
>> If you are concerned, enable dirsync.
>>
> If the filesystem already behaves that way a fsync on it should be fairly
> cheap. If it doesnt behave that way doing it is correct...
>
Well, I disagree, as the whole point of this thread is that fsync() is
*not* cheap. :-)
> Besides there is no reason to fsync the directory before the checkpoint, so
> dirsync would require a higher cost than doing it correctly.
>
Using "ordered" metadata journaling has approximately the same effect.
Provided that the data is fsync()'d before the metadata is required,
either the metadata is recorded in the journal, in which case the data
is accessible, or the metadata is NOT recorded in the journal, in which
case, the files will appear missing. The races that theoretically exist
would be in situations where the data of one file references a separate
file that does not yet exist.
You said you would try and reproduce - are you going to try and
reproduce on ext3/ext4 with ordered journalling enabled? I think
reproducing outside of a case such as CREATE DATABASE would be
difficult. It would have to be something like:
open(O_CREAT)/write()/fsync()/close() of new data file, where data
gets written, but directory data is not yet written out to journal
open()/.../write()/fsync()/close() of existing file to point to new
data file, but directory data is still not yet written out to journal
crash
In this case, "dirsync" should be effective at closing this hole.
As for cost? Well, most PostgreSQL data is stored within file content,
not directory metadata. I think "dirsync" might slow down some
operations like CREATE DATABASE or "rm -fr", but I would not expect it
to effect day-to-day performance of the database under real load. Many
operating systems enable the equivalent of "dirsync" by default. I
believe Solaris does this, for example, and other than slowing down "rm
-fr", I don't recall any real complaints about the cost of "dirsync".
After writing the above, I'm seriously considering adding "dirsync" to
my /db mounts that hold PostgreSQL and MySQL data.
Cheers,
mark
--
Mark Mielke<mark@mielke.cc>
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-11 02:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-12 15:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-14 14:03 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 15:31 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
@ 2010-02-14 20:57 ` Robert Haas <robertmhaas@gmail.com>
2010-02-14 21:43 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-14 23:33 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
1 sibling, 2 replies; 70+ messages in thread
From: Robert Haas @ 2010-02-14 20:57 UTC (permalink / raw)
To: Greg Stark <gsstark@mit.edu>; +Cc: Andres Freund <andres@anarazel.de>; pgsql-hackers
On Sun, Feb 14, 2010 at 10:31 AM, Greg Stark <gsstark@mit.edu> wrote:
> On Sun, Feb 14, 2010 at 2:03 PM, Greg Stark <gsstark@mit.edu> wrote:
>> On Fri, Feb 12, 2010 at 3:49 PM, Robert Haas <robertmhaas@gmail.com> wrote:
>>> Greg Stark, have you managed to get your access issues sorted out? If
>>
>> Yep, will look at this today.
>
> So I think we have a bigger problem than just copydir.c. It seems to
> me we should be fsyncing the table space data directories on every
> checkpoint. Otherwise any newly created relations or removed relations
> could disappear even though the data in them was fsynced. I'm thinking
> I should add an _mdfd_opentblspc(reln) call which returns a file
> descriptor for the tablespace and have mdsync() use that to sync the
> directory whenever it fsyncs a relation. It would be nice to remember
> which tablespaces have been fsynced and only fsync them once though,
> that would need another hash table just for tablespaces.
>
> We probably also need to fsync the pg_xlog directory every time we
> create or rename an xlog segment.
>
> Are there any other places we do directory operations which we need to
> be permanent?
I agree with Tom that we need to see some actual reproducible test
cases where this is an issue before we go too crazy with it. In
theory what you're talking about could also happen when extending a
relation, if we extend into a new file; but I think we need to
convince ourselves that it really happens before we make any more
changes.
On a pragmatic note, if this does turn out to be a problem, it's a
bug: and we can and do fix bugs whenever we discover them. But the
other part of this patch - to speed up createdb - is a feature - and
we are very rapidly running out of time for 9.0 features. So I'd like
to vote for getting the feature part of this committed (assuming it's
in good shape, of course) and we can continue to investigate the other
issues but without quite as much urgency.
...Robert
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-11 02:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-12 15:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-14 14:03 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 15:31 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 20:57 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
@ 2010-02-14 21:43 ` Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 70+ messages in thread
From: Andres Freund @ 2010-02-14 21:43 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Greg Stark <gsstark@mit.edu>; pgsql-hackers
On Sunday 14 February 2010 21:57:08 Robert Haas wrote:
> On Sun, Feb 14, 2010 at 10:31 AM, Greg Stark <gsstark@mit.edu> wrote:
> > On Sun, Feb 14, 2010 at 2:03 PM, Greg Stark <gsstark@mit.edu> wrote:
> >> On Fri, Feb 12, 2010 at 3:49 PM, Robert Haas <robertmhaas@gmail.com>
wrote:
> >>> Greg Stark, have you managed to get your access issues sorted out? If
> >>
> >> Yep, will look at this today.
> >
> > So I think we have a bigger problem than just copydir.c. It seems to
> > me we should be fsyncing the table space data directories on every
> > checkpoint. Otherwise any newly created relations or removed relations
> > could disappear even though the data in them was fsynced. I'm thinking
> > I should add an _mdfd_opentblspc(reln) call which returns a file
> > descriptor for the tablespace and have mdsync() use that to sync the
> > directory whenever it fsyncs a relation. It would be nice to remember
> > which tablespaces have been fsynced and only fsync them once though,
> > that would need another hash table just for tablespaces.
> >
> > We probably also need to fsync the pg_xlog directory every time we
> > create or rename an xlog segment.
> >
> > Are there any other places we do directory operations which we need to
> > be permanent?
>
> I agree with Tom that we need to see some actual reproducible test
> cases where this is an issue before we go too crazy with it. In
> theory what you're talking about could also happen when extending a
> relation, if we extend into a new file; but I think we need to
> convince ourselves that it really happens before we make any more
> changes.
Ok, will try to reproduce.
> On a pragmatic note, if this does turn out to be a problem, it's a
> bug: and we can and do fix bugs whenever we discover them. But the
> other part of this patch - to speed up createdb - is a feature - and
> we are very rapidly running out of time for 9.0 features. So I'd like
> to vote for getting the feature part of this committed (assuming it's
> in good shape, of course) and we can continue to investigate the other
> issues but without quite as much urgency.
Sound sensible.
Andres
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 15:25 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:33 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-03 13:42 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-11 02:27 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-02-12 15:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
2010-02-14 14:03 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 15:31 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-02-14 20:57 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Robert Haas <robertmhaas@gmail.com>
@ 2010-02-14 23:33 ` Greg Stark <gsstark@mit.edu>
1 sibling, 0 replies; 70+ messages in thread
From: Greg Stark @ 2010-02-14 23:33 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; pgsql-hackers
On Sun, Feb 14, 2010 at 8:57 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> On a pragmatic note, if this does turn out to be a problem, it's a
> bug: and we can and do fix bugs whenever we discover them. But the
> other part of this patch - to speed up createdb - is a feature - and
> we are very rapidly running out of time for 9.0 features. So I'd like
> to vote for getting the feature part of this committed (assuming it's
> in good shape, of course) and we can continue to investigate the other
> issues but without quite as much urgency.
No problem, I already committed the part that overlaps so I can commit
the rest now. I just want to take extra care given how much wine I've
already had tonight...
Incidentally, sorry Andres, I forgot to credit you in the first commit.
--
greg
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
@ 2010-01-20 04:02 ` Andres Freund <andres@anarazel.de>
3 siblings, 0 replies; 70+ messages in thread
From: Andres Freund @ 2010-01-20 04:02 UTC (permalink / raw)
To: Greg Stark <gsstark@mit.edu>; +Cc: pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
Hi Greg,
On Tuesday 19 January 2010 15:52:25 Greg Stark wrote:
> On Mon, Jan 18, 2010 at 4:35 PM, Greg Stark <gsstark@mit.edu> wrote:
> > Looking at this patch for the commitfest I have a few questions.
>
> So I've touched this patch up a bit:
>
> 1) moved the posix_fadvise call to a new fd.c function
> pg_fsync_start(fd,offset,nbytes) which initiates an fsync without
> waiting on it. Currently it's only implemented with
> posix_fadvise(DONT_NEED) but I want to look into using sync_file_range
> in the future -- it looks like this call might be good enough for our
> checkpoints.
Why exactly should that depend on fsync? Sure, thats where most of the pain
comes from now but avoiding that cache poisoning wouldnt hurt otherwise as
well.
I would rather have it called pg_flush_cache_range or such...
> 2) advised each 64k chunk as we write it which should avoid poisoning
> the cache if you do a large create database on an active system.
>
> 3) added the promised but afaict missing fsync of the directory -- i
> think we should actually backpatch this.
I think as well. You need it during recursing as well though (where I had
added it) and not only for the final directory.
> Barring any objections shall I commit it like this?
Other than the two things above it looks fine to me.
Thanks,
Andres
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
@ 2010-01-20 04:01 ` Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 70+ messages in thread
From: Andres Freund @ 2010-01-20 04:01 UTC (permalink / raw)
To: pgsql-performance@postgresql.org; +Cc: Greg Stark <gsstark@mit.edu>; pgsql-hackers@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
Hi Greg,
On Monday 18 January 2010 17:35:59 Greg Stark wrote:
> 2) Why does the second pass to do the fsyncs read through fromdir to
> find all the filenames. I find that odd and counterintuitive. It would
> be much more natural to just loop through the files in the new
> directory. But I suppose it serves as an added paranoia check that the
> files are in fact still there and we're not fsyncing any files we
> didn't just copy. I think it should still work, we should have an
> exclusive lock on the template database so there really ought to be no
> differences between the directory trees.
If it weren't safe we would already have a big problem....
Andres
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
@ 2009-12-28 23:06 ` Tom Lane <tgl@sss.pgh.pa.us>
2009-12-28 23:20 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 23:31 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2 siblings, 2 replies; 70+ messages in thread
From: Tom Lane @ 2009-12-28 23:06 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-performance@postgresql.org, pgsql-hackers@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
Andres Freund <andres@anarazel.de> writes:
> This speeds up CREATE DATABASE from ~9 seconds to something around 0.8s on my
> laptop. Still slower than with fsync off (~0.25) but quite a worthy
> improvement.
I can't help wondering whether that's real or some kind of
platform-specific artifact. I get numbers more like 3.5s (fsync off)
vs 4.5s (fsync on) on a machine where I believe the disks aren't lying
about write-complete. It makes sense that an fsync at the end would be
a little bit faster, because it would give the kernel some additional
freedom in scheduling the required I/O, but it isn't cutting the total
I/O required at all. So I find it really hard to believe a 10x speedup.
regards, tom lane
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 23:06 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
@ 2009-12-28 23:20 ` Andres Freund <andres@anarazel.de>
2009-12-29 00:46 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
1 sibling, 1 reply; 70+ messages in thread
From: Andres Freund @ 2009-12-28 23:20 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tuesday 29 December 2009 00:06:28 Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > This speeds up CREATE DATABASE from ~9 seconds to something around 0.8s
> > on my laptop. Still slower than with fsync off (~0.25) but quite a
> > worthy improvement.
> I can't help wondering whether that's real or some kind of
> platform-specific artifact. I get numbers more like 3.5s (fsync off)
> vs 4.5s (fsync on) on a machine where I believe the disks aren't lying
> about write-complete. It makes sense that an fsync at the end would be
> a little bit faster, because it would give the kernel some additional
> freedom in scheduling the required I/O, but it isn't cutting the total
> I/O required at all. So I find it really hard to believe a 10x speedup.
Well, a template database is about 5.5MB big here - that shouldnt take too
long when written near-sequentially?
As I said the real benefit only occurred after adding posix_fadvise(..,
FADV_DONTNEED) which is somewhat plausible, because i.e. the directory entries
don't need to get scheduled for every file and because the kernel can reorder a
whole directory nearly sequentially. Without the advice it the kernel doesn't
know in time that it should write that data back and it wont do it for 5
seconds by default on linux or such...
I looked at the strace output - it looks sensible timewise to me. If youre
interested I can give you output of that.
Andres
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 23:06 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2009-12-28 23:20 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
@ 2009-12-29 00:46 ` Greg Smith <greg@2ndquadrant.com>
2009-12-29 02:05 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 70+ messages in thread
From: Greg Smith @ 2009-12-29 00:46 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Tom Lane <tgl@sss.pgh.pa.us>; pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
Andres Freund wrote:
> As I said the real benefit only occurred after adding posix_fadvise(..,
> FADV_DONTNEED) which is somewhat plausible, because i.e. the directory entries
> don't need to get scheduled for every file and because the kernel can reorder a
> whole directory nearly sequentially. Without the advice it the kernel doesn't
> know in time that it should write that data back and it wont do it for 5
> seconds by default on linux or such...
>
I know they just fiddled with the logic in the last release, but for
most of the Linux kernels out there now pdflush wakes up every 5 seconds
by default. But typically it only worries about writing things that
have been in the queue for 30 seconds or more until you've filled quite
a bit of memory, so that's also an interesting number. I tried to
document the main tunables here and describe how they fit together at
http://www.westnet.com/~gsmith/content/linux-pdflush.htm
It would be interesting to graph the "Dirty" and "Writeback" figures in
/proc/meminfo over time with and without this patch in place. That
should make it obvious what the kernel is doing differently in the two
cases.
--
Greg Smith 2ndQuadrant Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.com
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 23:06 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2009-12-28 23:20 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 00:46 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
@ 2009-12-29 02:05 ` Andres Freund <andres@anarazel.de>
2009-12-29 02:53 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Michael Clemmons <glassresistor@gmail.com>
2009-12-29 10:48 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
0 siblings, 2 replies; 70+ messages in thread
From: Andres Freund @ 2009-12-29 02:05 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Greg Smith <greg@2ndquadrant.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tuesday 29 December 2009 01:46:21 Greg Smith wrote:
> Andres Freund wrote:
> > As I said the real benefit only occurred after adding posix_fadvise(..,
> > FADV_DONTNEED) which is somewhat plausible, because i.e. the directory
> > entries don't need to get scheduled for every file and because the kernel
> > can reorder a whole directory nearly sequentially. Without the advice it
> > the kernel doesn't know in time that it should write that data back and
> > it wont do it for 5 seconds by default on linux or such...
> It would be interesting to graph the "Dirty" and "Writeback" figures in
> /proc/meminfo over time with and without this patch in place. That
> should make it obvious what the kernel is doing differently in the two
> cases.
I did some analysis using blktrace (usefull tool btw) and the results show that
the io pattern is *significantly* different.
For one with the direct fsyncing nearly no hardware queuing is used and for
another nearly no requests are merged on software side.
Short stats:
OLD:
Total (8,0):
Reads Queued: 2, 8KiB Writes Queued: 7854, 29672KiB
Read Dispatches: 2, 8KiB Write Dispatches: 1926, 29672KiB
Reads Requeued: 0 Writes Requeued: 0
Reads Completed: 2, 8KiB Writes Completed: 2362, 29672KiB
Read Merges: 0, 0KiB Write Merges: 5492, 21968KiB
PC Reads Queued: 0, 0KiB PC Writes Queued: 0, 0KiB
PC Read Disp.: 436, 0KiB PC Write Disp.: 0, 0KiB
PC Reads Req.: 0 PC Writes Req.: 0
PC Reads Compl.: 0 PC Writes Compl.: 2362
IO unplugs: 2395 Timer unplugs: 557
New:
Total (8,0):
Reads Queued: 0, 0KiB Writes Queued: 1716, 5960KiB
Read Dispatches: 0, 0KiB Write Dispatches: 324, 5960KiB
Reads Requeued: 0 Writes Requeued: 0
Reads Completed: 0, 0KiB Writes Completed: 550, 5960KiB
Read Merges: 0, 0KiB Write Merges: 1166, 4664KiB
PC Reads Queued: 0, 0KiB PC Writes Queued: 0, 0KiB
PC Read Disp.: 226, 0KiB PC Write Disp.: 0, 0KiB
PC Reads Req.: 0 PC Writes Req.: 0
PC Reads Compl.: 0 PC Writes Compl.: 550
IO unplugs: 503 Timer unplugs: 30
Andres
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 23:06 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2009-12-28 23:20 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 00:46 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2009-12-29 02:05 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
@ 2009-12-29 02:53 ` Michael Clemmons <glassresistor@gmail.com>
2009-12-29 02:55 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 70+ messages in thread
From: Michael Clemmons @ 2009-12-29 02:53 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Greg Smith <greg@2ndquadrant.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-performance@postgresql.org, Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
Andres,
Great job. Looking through the emails and thinking about why this works I
think this patch should significantly speedup 8.4 on most any file
system(obviously some more than others) unless the system has significantly
reduced memory or a slow single core. On a Celeron with 256 memory I suspect
it'll crash out or just hit the swap and be a worse bottleneck. Anyone
have something like this to test on?
-Michael
On Mon, Dec 28, 2009 at 9:05 PM, Andres Freund <andres@anarazel.de> wrote:
> On Tuesday 29 December 2009 01:46:21 Greg Smith wrote:
> > Andres Freund wrote:
> > > As I said the real benefit only occurred after adding posix_fadvise(..,
> > > FADV_DONTNEED) which is somewhat plausible, because i.e. the directory
> > > entries don't need to get scheduled for every file and because the
> kernel
> > > can reorder a whole directory nearly sequentially. Without the advice
> it
> > > the kernel doesn't know in time that it should write that data back and
> > > it wont do it for 5 seconds by default on linux or such...
> > It would be interesting to graph the "Dirty" and "Writeback" figures in
> > /proc/meminfo over time with and without this patch in place. That
> > should make it obvious what the kernel is doing differently in the two
> > cases.
> I did some analysis using blktrace (usefull tool btw) and the results show
> that
> the io pattern is *significantly* different.
>
> For one with the direct fsyncing nearly no hardware queuing is used and for
> another nearly no requests are merged on software side.
>
> Short stats:
>
> OLD:
>
> Total (8,0):
> Reads Queued: 2, 8KiB Writes Queued: 7854,
> 29672KiB
> Read Dispatches: 2, 8KiB Write Dispatches: 1926,
> 29672KiB
> Reads Requeued: 0 Writes Requeued: 0
> Reads Completed: 2, 8KiB Writes Completed: 2362,
> 29672KiB
> Read Merges: 0, 0KiB Write Merges: 5492,
> 21968KiB
> PC Reads Queued: 0, 0KiB PC Writes Queued: 0,
> 0KiB
> PC Read Disp.: 436, 0KiB PC Write Disp.: 0,
> 0KiB
> PC Reads Req.: 0 PC Writes Req.: 0
> PC Reads Compl.: 0 PC Writes Compl.: 2362
> IO unplugs: 2395 Timer unplugs: 557
>
>
> New:
>
> Total (8,0):
> Reads Queued: 0, 0KiB Writes Queued: 1716,
> 5960KiB
> Read Dispatches: 0, 0KiB Write Dispatches: 324,
> 5960KiB
> Reads Requeued: 0 Writes Requeued: 0
> Reads Completed: 0, 0KiB Writes Completed: 550,
> 5960KiB
> Read Merges: 0, 0KiB Write Merges: 1166,
> 4664KiB
> PC Reads Queued: 0, 0KiB PC Writes Queued: 0,
> 0KiB
> PC Read Disp.: 226, 0KiB PC Write Disp.: 0,
> 0KiB
> PC Reads Req.: 0 PC Writes Req.: 0
> PC Reads Compl.: 0 PC Writes Compl.: 550
> IO unplugs: 503 Timer unplugs: 30
>
>
> Andres
>
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 23:06 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2009-12-28 23:20 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 00:46 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2009-12-29 02:05 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 02:53 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Michael Clemmons <glassresistor@gmail.com>
@ 2009-12-29 02:55 ` Andres Freund <andres@anarazel.de>
2009-12-29 03:04 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Michael Clemmons <glassresistor@gmail.com>
0 siblings, 1 reply; 70+ messages in thread
From: Andres Freund @ 2009-12-29 02:55 UTC (permalink / raw)
To: Michael Clemmons <glassresistor@gmail.com>; +Cc: pgsql-hackers@postgresql.org, Greg Smith <greg@2ndquadrant.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-performance@postgresql.org, Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tuesday 29 December 2009 03:53:12 Michael Clemmons wrote:
> Andres,
> Great job. Looking through the emails and thinking about why this works I
> think this patch should significantly speedup 8.4 on most any file
> system(obviously some more than others) unless the system has significantly
> reduced memory or a slow single core. On a Celeron with 256 memory I
> suspect it'll crash out or just hit the swap and be a worse bottleneck.
> Anyone have something like this to test on?
Why should it crash? The kernel should just block on writing and write out the
dirty memory before continuing?
Pg is not caching anything here...
Andres
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 23:06 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2009-12-28 23:20 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 00:46 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2009-12-29 02:05 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 02:53 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Michael Clemmons <glassresistor@gmail.com>
2009-12-29 02:55 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
@ 2009-12-29 03:04 ` Michael Clemmons <glassresistor@gmail.com>
2009-12-29 03:11 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 70+ messages in thread
From: Michael Clemmons @ 2009-12-29 03:04 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Greg Smith <greg@2ndquadrant.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-performance@postgresql.org, Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
Maybe not crash out but in this situation.
N=0
while(N>=0):
CREATE DATABASE new_db_N;
Since the fsync is the part which takes the memory and time but is happening
in the background want the fsyncs pile up in the background faster than can
be run filling up the memory and stack.
This is very likely a mistake on my part about how postgres/processes
actually works.
-Michael
On Mon, Dec 28, 2009 at 9:55 PM, Andres Freund <andres@anarazel.de> wrote:
> On Tuesday 29 December 2009 03:53:12 Michael Clemmons wrote:
> > Andres,
> > Great job. Looking through the emails and thinking about why this works
> I
> > think this patch should significantly speedup 8.4 on most any file
> > system(obviously some more than others) unless the system has
> significantly
> > reduced memory or a slow single core. On a Celeron with 256 memory I
> > suspect it'll crash out or just hit the swap and be a worse bottleneck.
> > Anyone have something like this to test on?
> Why should it crash? The kernel should just block on writing and write out
> the
> dirty memory before continuing?
> Pg is not caching anything here...
>
> Andres
>
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 23:06 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2009-12-28 23:20 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 00:46 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2009-12-29 02:05 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 02:53 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Michael Clemmons <glassresistor@gmail.com>
2009-12-29 02:55 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 03:04 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Michael Clemmons <glassresistor@gmail.com>
@ 2009-12-29 03:11 ` Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 70+ messages in thread
From: Andres Freund @ 2009-12-29 03:11 UTC (permalink / raw)
To: Michael Clemmons <glassresistor@gmail.com>; +Cc: pgsql-hackers@postgresql.org, Greg Smith <greg@2ndquadrant.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-performance@postgresql.org, Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tuesday 29 December 2009 04:04:06 Michael Clemmons wrote:
> Maybe not crash out but in this situation.
> N=0
> while(N>=0):
> CREATE DATABASE new_db_N;
> Since the fsync is the part which takes the memory and time but is
> happening in the background want the fsyncs pile up in the background
> faster than can be run filling up the memory and stack.
> This is very likely a mistake on my part about how postgres/processes
The difference should not be visible outside the "CREATE DATABASE ..." at all.
Currently the process simplifiedly works like:
------------
for file in source directory:
copy_file(source/file, target/file);
fsync(target/file);
------------
I changed it to:
-------------
for file in source directory:
copy_file(source/file, target/file);
/*please dear kernel, write this out, but dont block*/
posix_fadvise(target/file, FADV_DONTNEED);
for file in source directory:
fsync(target/file);
-------------
If at any point in time there is not enough cache available to cache anything
copy_file() will just have to wait for the kernel to write out the data.
fsync() does not use memory itself.
Andres
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 23:06 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2009-12-28 23:20 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 00:46 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2009-12-29 02:05 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
@ 2009-12-29 10:48 ` Greg Stark <gsstark@mit.edu>
2009-12-29 11:13 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 70+ messages in thread
From: Greg Stark @ 2009-12-29 10:48 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Greg Smith <greg@2ndquadrant.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tue, Dec 29, 2009 at 2:05 AM, Andres Freund <andres@anarazel.de> wrote:
> Reads Completed: 2, 8KiB Writes Completed: 2362, 29672KiB
> New:
> Reads Completed: 0, 0KiB Writes Completed: 550, 5960KiB
It looks like the new method is only doing 1/6th as much i/o. Do you
know what's going on there?
--
greg
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 23:06 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
2009-12-28 23:20 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 00:46 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2009-12-29 02:05 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 10:48 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
@ 2009-12-29 11:13 ` Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 70+ messages in thread
From: Andres Freund @ 2009-12-29 11:13 UTC (permalink / raw)
To: Greg Stark <gsstark@mit.edu>; +Cc: pgsql-hackers@postgresql.org, Greg Smith <greg@2ndquadrant.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tuesday 29 December 2009 11:48:10 Greg Stark wrote:
> On Tue, Dec 29, 2009 at 2:05 AM, Andres Freund <andres@anarazel.de> wrote:
> > Reads Completed: 2, 8KiB Writes Completed: 2362,
> > 29672KiB New:
> > Reads Completed: 0, 0KiB Writes Completed: 550,
> > 5960KiB
>
> It looks like the new method is only doing 1/6th as much i/o. Do you
> know what's going on there?
While I was surprised by the amount of difference I am not surprised at all
that there is a significant one - currently the fsync will write out a whole
bunch of useless stuff every time its called (all metadata, directory structure
and so on)
This is reproducible...
6MB sounds sensible for the operation btw - the template database is around
5MB.
Will try to analyze later what exactly causes the additional io.
Andres
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 23:06 ` Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Tom Lane <tgl@sss.pgh.pa.us>
@ 2009-12-28 23:31 ` Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 70+ messages in thread
From: Andres Freund @ 2009-12-28 23:31 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-performance@postgresql.org, pgsql-hackers@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tuesday 29 December 2009 00:06:28 Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > This speeds up CREATE DATABASE from ~9 seconds to something around 0.8s
> > on my laptop. Still slower than with fsync off (~0.25) but quite a
> > worthy improvement.
>
> I can't help wondering whether that's real or some kind of
> platform-specific artifact. I get numbers more like 3.5s (fsync off)
> vs 4.5s (fsync on) on a machine where I believe the disks aren't lying
> about write-complete. It makes sense that an fsync at the end would be
> a little bit faster, because it would give the kernel some additional
> freedom in scheduling the required I/O, but it isn't cutting the total
> I/O required at all. So I find it really hard to believe a 10x speedup.
I only comfortably have access to two smaller machines without BBU from here
(being in the Hacker Jeopardy at the ccc congress ;-)) and both show this
behaviour. I guess its somewhat filesystem dependent.
Andres
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
@ 2009-12-29 00:27 ` Greg Stark <gsstark@mit.edu>
2009-12-29 00:29 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 00:30 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) david@lang.hm
2 siblings, 2 replies; 70+ messages in thread
From: Greg Stark @ 2009-12-29 00:27 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-performance@postgresql.org, pgsql-hackers@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Mon, Dec 28, 2009 at 10:54 PM, Andres Freund <andres@anarazel.de> wrote:
> fsync everything in that pass.
> Including the directory - which was not done before and actually might be
> necessary in some cases.
Er. Yes. At least on ext4 this is pretty important. I wish it weren't,
but it doesn't look like we're going to convince the ext4 developers
they're crazy any day soon and it would really suck for a database
created from a template to have files in it go missin.
--
greg
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 00:27 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
@ 2009-12-29 00:29 ` Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 70+ messages in thread
From: Andres Freund @ 2009-12-29 00:29 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Greg Stark <gsstark@mit.edu>; pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tuesday 29 December 2009 01:27:29 Greg Stark wrote:
> On Mon, Dec 28, 2009 at 10:54 PM, Andres Freund <andres@anarazel.de> wrote:
> > fsync everything in that pass.
> > Including the directory - which was not done before and actually might be
> > necessary in some cases.
>
> Er. Yes. At least on ext4 this is pretty important. I wish it weren't,
> but it doesn't look like we're going to convince the ext4 developers
> they're crazy any day soon and it would really suck for a database
> created from a template to have files in it go missin.
Actually it was necessary on ext3 as well - the window to hit the problem just
was much smaller, wasnt it?
Actually that part should possibly get backported.
Andres
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 00:27 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
@ 2009-12-29 00:30 ` david@lang.hm
2009-12-29 00:43 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 70+ messages in thread
From: david@lang.hm @ 2009-12-29 00:30 UTC (permalink / raw)
To: Greg Stark <gsstark@mit.edu>; +Cc: Andres Freund <andres@anarazel.de>; pgsql-performance@postgresql.org, pgsql-hackers@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tue, 29 Dec 2009, Greg Stark wrote:
> On Mon, Dec 28, 2009 at 10:54 PM, Andres Freund <andres@anarazel.de> wrote:
>> fsync everything in that pass.
>> Including the directory - which was not done before and actually might be
>> necessary in some cases.
>
> Er. Yes. At least on ext4 this is pretty important. I wish it weren't,
> but it doesn't look like we're going to convince the ext4 developers
> they're crazy any day soon and it would really suck for a database
> created from a template to have files in it go missin.
actually, as I understand it you need to do this on all filesystems except
ext3, and on ext3 fsync is horribly slow because it writes out
_everything_ that's pending, not just stuff related to the file you do the
fsync on.
David Lang
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 00:27 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2009-12-29 00:30 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) david@lang.hm
@ 2009-12-29 00:43 ` Andres Freund <andres@anarazel.de>
2009-12-29 00:46 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) david@lang.hm
0 siblings, 1 reply; 70+ messages in thread
From: Andres Freund @ 2009-12-29 00:43 UTC (permalink / raw)
To: david@lang.hm; +Cc: Greg Stark <gsstark@mit.edu>; pgsql-performance@postgresql.org, pgsql-hackers@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tuesday 29 December 2009 01:30:17 david@lang.hm wrote:
> On Tue, 29 Dec 2009, Greg Stark wrote:
> > On Mon, Dec 28, 2009 at 10:54 PM, Andres Freund <andres@anarazel.de>
wrote:
> >> fsync everything in that pass.
> >> Including the directory - which was not done before and actually might
> >> be necessary in some cases.
> >
> > Er. Yes. At least on ext4 this is pretty important. I wish it weren't,
> > but it doesn't look like we're going to convince the ext4 developers
> > they're crazy any day soon and it would really suck for a database
> > created from a template to have files in it go missin.
>
> actually, as I understand it you need to do this on all filesystems except
> ext3, and on ext3 fsync is horribly slow because it writes out
> _everything_ that's pending, not just stuff related to the file you do the
> fsync on.
I dont think its all filesystems (ext2 should not be affected...), but generally
youre right. At least jfs, xfs are affected as well.
Its btw not necessarily nearly-safe and slow on ext3 as well (data=writeback).
Andres
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 00:27 ` Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Stark <gsstark@mit.edu>
2009-12-29 00:30 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) david@lang.hm
2009-12-29 00:43 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
@ 2009-12-29 00:46 ` david@lang.hm
0 siblings, 0 replies; 70+ messages in thread
From: david@lang.hm @ 2009-12-29 00:46 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Greg Stark <gsstark@mit.edu>; pgsql-performance@postgresql.org, pgsql-hackers@postgresql.org, Michael Clemmons <glassresistor@gmail.com>; Hannu Krosing <hannu@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>
On Tue, 29 Dec 2009, Andres Freund wrote:
> On Tuesday 29 December 2009 01:30:17 david@lang.hm wrote:
>> On Tue, 29 Dec 2009, Greg Stark wrote:
>>> On Mon, Dec 28, 2009 at 10:54 PM, Andres Freund <andres@anarazel.de>
> wrote:
>>>> fsync everything in that pass.
>>>> Including the directory - which was not done before and actually might
>>>> be necessary in some cases.
>>>
>>> Er. Yes. At least on ext4 this is pretty important. I wish it weren't,
>>> but it doesn't look like we're going to convince the ext4 developers
>>> they're crazy any day soon and it would really suck for a database
>>> created from a template to have files in it go missin.
>>
>> actually, as I understand it you need to do this on all filesystems except
>> ext3, and on ext3 fsync is horribly slow because it writes out
>> _everything_ that's pending, not just stuff related to the file you do the
>> fsync on.
> I dont think its all filesystems (ext2 should not be affected...), but generally
> youre right. At least jfs, xfs are affected as well.
ext2 definantly needs the fsync on the directory as well as the file
(well, if the file metadata like size, change)
> Its btw not necessarily nearly-safe and slow on ext3 as well (data=writeback).
no, then it's just unsafe and slow ;-)
David Lang
^ permalink raw reply [nested|flat] 70+ messages in thread
end of thread, other threads:[~2010-02-15 00:08 UTC | newest]
Thread overview: 70+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2009-12-28 22:54 Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-28 22:59 ` Andres Freund <andres@anarazel.de>
2009-12-29 18:30 ` Andres Freund <andres@anarazel.de>
2010-01-18 16:35 ` Greg Stark <gsstark@mit.edu>
2010-01-19 14:52 ` Greg Stark <gsstark@mit.edu>
2010-01-19 14:57 ` Greg Stark <gsstark@mit.edu>
2010-01-20 04:13 ` Andres Freund <andres@anarazel.de>
2010-01-20 05:21 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-01-27 07:21 ` Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Greg Smith <greg@2ndquadrant.com>
2010-01-19 15:03 ` Andres Freund <andres@anarazel.de>
2010-01-19 15:25 ` Tom Lane <tgl@sss.pgh.pa.us>
2010-01-29 18:56 ` Greg Stark <gsstark@mit.edu>
2010-02-02 17:36 ` Robert Haas <robertmhaas@gmail.com>
2010-02-02 17:43 ` Andres Freund <andres@anarazel.de>
2010-02-02 17:50 ` Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 18:14 ` Robert Haas <robertmhaas@gmail.com>
2010-02-02 18:34 ` Andres Freund <andres@anarazel.de>
2010-02-02 19:06 ` Robert Haas <robertmhaas@gmail.com>
2010-02-02 19:08 ` Andres Freund <andres@anarazel.de>
2010-02-02 19:33 ` Tom Lane <tgl@sss.pgh.pa.us>
2010-02-02 19:45 ` Robert Haas <robertmhaas@gmail.com>
2010-02-03 11:53 ` Greg Stark <gsstark@mit.edu>
2010-02-03 12:03 ` Andres Freund <andres@anarazel.de>
2010-02-03 13:42 ` Robert Haas <robertmhaas@gmail.com>
2010-02-03 14:19 ` Andres Freund <andres@anarazel.de>
2010-02-06 05:03 ` Greg Smith <greg@2ndquadrant.com>
2010-02-06 12:03 ` Andres Freund <andres@anarazel.de>
2010-02-07 05:13 ` Robert Haas <robertmhaas@gmail.com>
2010-02-07 09:23 ` Greg Smith <greg@2ndquadrant.com>
2010-02-07 16:24 ` Tom Lane <tgl@sss.pgh.pa.us>
2010-02-07 18:23 ` Robert Haas <robertmhaas@gmail.com>
2010-02-07 18:27 ` Andres Freund <andres@anarazel.de>
2010-02-08 01:31 ` Andres Freund <andres@anarazel.de>
2010-02-08 03:09 ` Alvaro Herrera <alvherre@commandprompt.com>
2010-02-08 04:53 ` Robert Haas <robertmhaas@gmail.com>
2010-02-08 07:13 ` Andres Freund <andres@anarazel.de>
2010-02-08 18:34 ` Greg Stark <gsstark@mit.edu>
2010-02-08 19:29 ` Andres Freund <andres@anarazel.de>
2010-02-11 02:27 ` Andres Freund <andres@anarazel.de>
2010-02-12 15:49 ` Robert Haas <robertmhaas@gmail.com>
2010-02-14 14:03 ` Greg Stark <gsstark@mit.edu>
2010-02-14 15:31 ` Greg Stark <gsstark@mit.edu>
2010-02-14 17:11 ` Tom Lane <tgl@sss.pgh.pa.us>
2010-02-14 17:27 ` Andres Freund <andres@anarazel.de>
2010-02-14 17:37 ` Tom Lane <tgl@sss.pgh.pa.us>
2010-02-14 20:24 ` Re: Re: Faster CREATE DATABASE by delaying fsync Florian Weimer <fw@deneb.enyo.de>
2010-02-14 20:41 ` Re: Re: Faster CREATE DATABASE by delaying fsync Mark Mielke <mark@mark.mielke.cc>
2010-02-14 20:49 ` Re: Re: Faster CREATE DATABASE by delaying fsync Andres Freund <andres@anarazel.de>
2010-02-15 00:08 ` Re: Re: Faster CREATE DATABASE by delaying fsync Mark Mielke <mark@mark.mielke.cc>
2010-02-14 20:57 ` Robert Haas <robertmhaas@gmail.com>
2010-02-14 21:43 ` Andres Freund <andres@anarazel.de>
2010-02-14 23:33 ` Greg Stark <gsstark@mit.edu>
2010-01-20 04:02 ` Andres Freund <andres@anarazel.de>
2010-01-20 04:01 ` Andres Freund <andres@anarazel.de>
2009-12-28 23:06 ` Tom Lane <tgl@sss.pgh.pa.us>
2009-12-28 23:20 ` Andres Freund <andres@anarazel.de>
2009-12-29 00:46 ` Greg Smith <greg@2ndquadrant.com>
2009-12-29 02:05 ` Andres Freund <andres@anarazel.de>
2009-12-29 02:53 ` Michael Clemmons <glassresistor@gmail.com>
2009-12-29 02:55 ` Andres Freund <andres@anarazel.de>
2009-12-29 03:04 ` Michael Clemmons <glassresistor@gmail.com>
2009-12-29 03:11 ` Andres Freund <andres@anarazel.de>
2009-12-29 10:48 ` Greg Stark <gsstark@mit.edu>
2009-12-29 11:13 ` Andres Freund <andres@anarazel.de>
2009-12-28 23:31 ` Andres Freund <andres@anarazel.de>
2009-12-29 00:27 ` Greg Stark <gsstark@mit.edu>
2009-12-29 00:29 ` Andres Freund <andres@anarazel.de>
2009-12-29 00:30 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) david@lang.hm
2009-12-29 00:43 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) Andres Freund <andres@anarazel.de>
2009-12-29 00:46 ` Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb) david@lang.hm
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox