From: Merlin Moncure <mmoncure@gmail.com>
To: Jay Manni <JManni@fireeye.com>
Cc: pgsql-performance@postgresql.org <pgsql-performance@postgresql.org>
Subject: Re: High Frequency Inserts to Postgres Database vs Writing to a File
Date: Wed, 4 Nov 2009 08:59:25 -0500
Message-ID: <b42b73150911040559w660d1ad2w38a014fbc34e877f@mail.gmail.com> (raw)
In-Reply-To: <60B0F2124D07B942988329B5B7CA393D01E5B93FF7@mail2.FireEye.com>
References: <95652FF3B9EF3045AC4AE8FE5AF8671EED65CF6B@mail2.FireEye.com>
<60B0F2124D07B942988329B5B7CA393D01E5B93FF7@mail2.FireEye.com>
n Tue, Nov 3, 2009 at 10:12 PM, Jay Manni <JManni@fireeye.com> wrote:
> Hi:
>
> I have an application wherein a process needs to read data from a stream and
> store the records for further analysis and reporting. The data in the stream
> is in the form of variable length records with clearly defined fields – so
> it can be stored in a database or in a file. The only caveat is that the
> rate of records coming in the stream could be several 1000 records a second.
Postgres doing this is going to depend on primarily two things:
*) Your hardware
*) The mechanism you use to insert the data into the database
Postgres can handle multiple 1000 insert/sec but your hardware most
likely can't handle multiple 1000 transaction/sec if fsync is on. You
definitely want to batch the insert into the database somehow, so that
something accumulates the data (could be a simple file), and flushes
it in to the database. The 'flush' ideally should use copy but
multiple row insert is ok too. Try to avoid inserting one row at a
time even if in a transaction.
If you are bulk inserting 1000+ records/sec all day long, make sure
you have provisioned enough storage for this (that's 86M records/day),
and you should immediately start thinking about partitioning and
rotating the log table (if you log to the database, partition/rotate
is basically already baked in anyways).
The effects on other users of the database are really hard to predict
-- it's going to depend on how much resources you have (cpu and
especially disk) to direct towards the loading and how the database is
being used. I expect it shouldn't be too bad unless your dataase is
already i/o loaded. The good news is testing this is relatively easy
you can simulate a load test and just run it during typical use and
see how it affects other users. Standard o/s tools (iostat, top), and
database log with min_duration_statement are going to be a big help
here. If you start seeing big leaps in iowait corresponding with
unexpectedly lagging queries in your app , you probably should think
about scrapping the idea.
merlin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: mmoncure@gmail.com, JManni@fireeye.com
Subject: Re: High Frequency Inserts to Postgres Database vs Writing to a File
In-Reply-To: <b42b73150911040559w660d1ad2w38a014fbc34e877f@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox