From: Andres Freund <andres@anarazel.de>
To: Jan Gunnar Dyrset <jan.g.dyrset@cybernetica.no>
Cc: pgsql-performance@postgresql.org <pgsql-performance@postgresql.org>
Subject: Re: PostgreSQL disk fragmentation causes performance problems on Windows
Date: Wed, 29 Apr 2015 10:06:39 +0200
Message-ID: <20150429080639.GC14535@awork2.anarazel.de> (raw)
In-Reply-To: <AMXPR05MB120CFF5F9A4825A531E2018A1ED0@AMXPR05MB120.eurprd05.prod.outlook.com>
References: <AMXPR05MB120CFF5F9A4825A531E2018A1ED0@AMXPR05MB120.eurprd05.prod.outlook.com>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-performance>
Hi,
On 2015-04-23 19:47:06 +0000, Jan Gunnar Dyrset wrote:
> I am using PostgreSQL to log data in my application. A number of rows
> are added periodically, but there are no updates or deletes. There are
> several applications that log to different databases.
>
> This causes terrible disk fragmentation which again causes performance
> degradation when retrieving data from the databases. The table files
> are getting more than 50000 fragments over time (max table size about
> 1 GB).
>
> The problem seems to be that PostgreSQL grows the database with only
> the room it need for the new data each time it is added. Because
> several applications are adding data to different databases, the
> additions are never contiguous.
Which OS and filesystem is this done on? Because many halfway modern
systems, like e.g ext4 and xfs, implement this in the background as
'delayed allocation'.
Is it possible that e.g. you're checkpointing very frequently - which
includes fsyncing dirty files - and that that causes delayed allocation
not to work? How often did you checkpoint?
How did you measure the fragmentation? Using filefrag? If so, could you
perhaps send its output?
> I think that preallocating lumps of a given, configurable size, say 4
> MB, for the tables would remove this problem. The max number of
> fragments on a 1 GB file would then be 250, which is no problem. Is
> this possible to configure in PostgreSQL? If not, how difficult is it
> to implement in the database?
It's not impossible, but there are complexities because a) extension
happens under a sometimes contended lock, and doing more there will have
possible negative scalability implications. we need to restructure the
logging first to make that more realistic. b) postgres also tries to
truncate files, and we need to make sure that happens only in the right
cirumstances.
Greetings,
Andres Freund
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: andres@anarazel.de, jan.g.dyrset@cybernetica.no
Subject: Re: PostgreSQL disk fragmentation causes performance problems on Windows
In-Reply-To: <20150429080639.GC14535@awork2.anarazel.de>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox