From: Greg Smith <greg@2ndquadrant.com>
To: Andres Freund <andres@anarazel.de>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: Greg Stark <gsstark@mit.edu>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, Michael Clemmons <glassresistor@gmail.com>
Cc: Hannu Krosing <hannu@2ndquadrant.com>
Cc: Scott Marlowe <scott.marlowe@gmail.com>
Subject: Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
Date: Sat, 06 Feb 2010 00:03:30 -0500
Message-ID: <4B6CF822.9010608@2ndquadrant.com> (raw)
In-Reply-To: <4B698605.50506@anarazel.de>
References: <4427a97a0912101235x3ade510blcc00d5ce9883c6b6@mail.gmail.com>
<23321.1265133015@sss.pgh.pa.us>
<603c8f071002021014h4b489f44s9014377e1b38dbc7@mail.gmail.com>
<201002021934.18444.andres@anarazel.de>
<603c8f071002021106o5eef7b40r1ed21c9bd05251ec@mail.gmail.com>
<29129.1265139210@sss.pgh.pa.us>
<603c8f071002021145x1939c9e7hb2eedc8ae4dfe23e@mail.gmail.com>
<407d949e1002030353g522f1afev782a97fd755c5926@mail.gmail.com>
<603c8f071002030542r779a4f47k78355c62b0b7853@mail.gmail.com>
<4B698605.50506@anarazel.de>
Andres Freund wrote:
> On 02/03/10 14:42, Robert Haas wrote:
>> Well, maybe we should start with a discussion of what kernel calls
>> you're aware of on different platforms and then we could try to put an
>> API around it.
> In linux there is sync_file_range. On newer Posixish systems one can
> emulate that with mmap() and msync() (in batches obviously).
>
> No idea about windows.
There's a series of parameters you can pass into CreateFile:
http://msdn.microsoft.com/en-us/library/aa363858(VS.85).aspx
A lot of these are already mapped inside of src/port/open.c in a pretty
straightforward way from the POSIX-oriented interface:
O_RDWR,O_WRONLY -> GENERIC_WRITE, GENERIC_READ
O_RANDOM -> FILE_FLAG_RANDOM_ACCESS
O_SEQUENTIAL -> FILE_FLAG_SEQUENTIAL_SCAN
O_SHORT_LIVED -> FILE_ATTRIBUTE_TEMPORARY
O_TEMPORARY -> FILE_FLAG_DELETE_ON_CLOSE
O_DIRECT -> FILE_FLAG_NO_BUFFERING
O_DSYNC -> FILE_FLAG_WRITE_THROUGH
You have to read the whole "Caching Behavior" section to see exactly how
all of those interact, and even then notes like
http://support.microsoft.com/kb/99794 are needed to follow the fine
points of things like FILE_FLAG_NO_BUFFERING vs. FILE_FLAG_WRITE_THROUGH.
So anything that's setting those POSIX open flags better than before is
getting the benefit of that improvement on Windows, too. But that's not
quite the same as the changes using fadvise to provide better targeted
cache control hints.
I'm getting the impression that doing much better on Windows might fall
into the same sort of category as Solaris, where the primary interface
for this sort of thing is to use an AIO implementation instead:
http://msdn.microsoft.com/en-us/library/aa365683(VS.85).aspx
The effective_io_concurrency feature had proof of concept test programs
that worked using AIO, but actually following through on that
implementation would require a major restructuring of how the database
interacts with the OS in terms of reads and writes of blocks. It looks
to me like doing something similar to sync_file_range on Windows would
be similarly difficult.
--
Greg Smith 2ndQuadrant US Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.us
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: greg@2ndquadrant.com, andres@anarazel.de, robertmhaas@gmail.com, gsstark@mit.edu, tgl@sss.pgh.pa.us, glassresistor@gmail.com, hannu@2ndquadrant.com, scott.marlowe@gmail.com
Subject: Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
In-Reply-To: <4B6CF822.9010608@2ndquadrant.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox