pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedimproving write performance for logging application
13+ messages / 6 participants
[nested] [flat]
* improving write performance for logging application
@ 2006-01-03 23:44 Steve Eckmann <eckmann@computer.org>
2006-01-04 00:00 ` Re: improving write performance for logging application Tom Lane <tgl@sss.pgh.pa.us>
2006-01-04 00:06 ` Re: improving write performance for logging application Steinar H. Gunderson <sgunderson@bigfoot.com>
2006-01-04 13:54 ` Re: improving write performance for logging application Ian Westmacott <ianw@intellivid.com>
0 siblings, 3 replies; 13+ messages in thread
From: Steve Eckmann @ 2006-01-03 23:44 UTC (permalink / raw)
To: pgsql-performance
I have questions about how to improve the write performance of PostgreSQL for logging data from a real-time simulation. We found that MySQL 4.1.3 could log about 1480 objects/second using MyISAM tables or about 1225 objects/second using InnoDB tables, but PostgreSQL 8.0.3 could log only about 540 objects/second. (test system: quad-Itanium2, 8GB memory, SCSI RAID, GigE connection from simulation server, nothing running except system processes and database system under test)
We also found that we could improve MySQL performance significantly using MySQL's "INSERT" command extension allowing multiple value-list tuples in a single command; the rate for MyISAM tables improved to about 2600 objects/second. PostgreSQL doesn't support that language extension. Using the COPY command instead of INSERT might help, but since rows are being generated on the fly, I don't see how to use COPY without running a separate process that reads rows from the application and uses COPY to write to the database. The application currently has two processes: the simulation and a data collector that reads events from the sim (queued in shared memory) and writes them as rows to the database, buffering as needed to avoid lost data during periods of high activity. To use COPY I think we would have to split our data collector into two processes communicating via a pipe.
Query performance is not an issue: we found that when suitable indexes are added PostgreSQL is fast enough on the kinds of queries our users make. The crux is writing rows to the database fast enough to keep up with the simulation.
Are there general guidelines for tuning the PostgreSQL server for this kind of application? The suggestions I've found include disabling fsync (done), increasing the value of wal_buffers, and moving the WAL to a different disk, but these aren't likely to produce the 3x improvement that we need. On the client side I've found only two suggestions: disable autocommit and use COPY instead of INSERT. I think I've effectively disabled autocommit by batching up to several hundred INSERT commands in each PQexec() call, and it isn’t clear that COPY is worth the effort in our application.
Thanks.
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: improving write performance for logging application
2006-01-03 23:44 improving write performance for logging application Steve Eckmann <eckmann@computer.org>
@ 2006-01-04 00:00 ` Tom Lane <tgl@sss.pgh.pa.us>
2006-01-04 03:23 ` Re: improving write performance for logging application dlang <dlang@invendra.net>
2006-01-04 14:00 ` Re: improving write performance for logging application Steve Eckmann <eckmann@computer.org>
2 siblings, 2 replies; 13+ messages in thread
From: Tom Lane @ 2006-01-04 00:00 UTC (permalink / raw)
To: Steve Eckmann <eckmann@computer.org>; +Cc: pgsql-performance
Steve Eckmann <eckmann@computer.org> writes:
> We also found that we could improve MySQL performance significantly
> using MySQL's "INSERT" command extension allowing multiple value-list
> tuples in a single command; the rate for MyISAM tables improved to
> about 2600 objects/second. PostgreSQL doesn't support that language
> extension. Using the COPY command instead of INSERT might help, but
> since rows are being generated on the fly, I don't see how to use COPY
> without running a separate process that reads rows from the
> application and uses COPY to write to the database.
Can you conveniently alter your application to batch INSERT commands
into transactions? Ie
BEGIN;
INSERT ...;
... maybe 100 or so inserts ...
COMMIT;
BEGIN;
... lather, rinse, repeat ...
This cuts down the transactional overhead quite a bit. A downside is
that you lose multiple rows if any INSERT fails, but then the same would
be true of multiple VALUES lists per INSERT.
regards, tom lane
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: improving write performance for logging application
2006-01-03 23:44 improving write performance for logging application Steve Eckmann <eckmann@computer.org>
2006-01-04 00:00 ` Re: improving write performance for logging application Tom Lane <tgl@sss.pgh.pa.us>
@ 2006-01-04 03:23 ` dlang <dlang@invendra.net>
2006-01-04 14:13 ` Re: improving write performance for logging application Steve Eckmann <eckmann@computer.org>
1 sibling, 1 reply; 13+ messages in thread
From: dlang @ 2006-01-04 03:23 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Steve Eckmann <eckmann@computer.org>; pgsql-performance
On Tue, 3 Jan 2006, Tom Lane wrote:
> Steve Eckmann <eckmann@computer.org> writes:
> > We also found that we could improve MySQL performance significantly
> > using MySQL's "INSERT" command extension allowing multiple value-list
> > tuples in a single command; the rate for MyISAM tables improved to
> > about 2600 objects/second. PostgreSQL doesn't support that language
> > extension. Using the COPY command instead of INSERT might help, but
> > since rows are being generated on the fly, I don't see how to use COPY
> > without running a separate process that reads rows from the
> > application and uses COPY to write to the database.
>
> Can you conveniently alter your application to batch INSERT commands
> into transactions? Ie
>
> BEGIN;
> INSERT ...;
> ... maybe 100 or so inserts ...
> COMMIT;
> BEGIN;
> ... lather, rinse, repeat ...
>
> This cuts down the transactional overhead quite a bit. A downside is
> that you lose multiple rows if any INSERT fails, but then the same would
> be true of multiple VALUES lists per INSERT.
Steve, you mentioned that you data collector buffers the data before
sending it to the database, modify it so that each time it goes to send
things to the database you send all the data that's in the buffer as a
single transaction.
I am working on useing postgres to deal with log data and wrote a simple
perl script that read in the log files a line at a time, and then wrote
them 1000 at a time to the database. On a dual Opteron 240 box with 2G of
ram 1x 15krpm SCSI drive (and a untuned postgress install with the compile
time defaults) I was getting 5000-8000 lines/sec (I think this was with
fsync disabled, but I don't remember for sure). and postgres was
complaining that it was overrunning it's log sizes (which limits the speed
as it then has to pause to flush the logs)
the key thing is to send multiple lines with one transaction as tom shows
above.
David Lang
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: improving write performance for logging application
2006-01-03 23:44 improving write performance for logging application Steve Eckmann <eckmann@computer.org>
2006-01-04 00:00 ` Re: improving write performance for logging application Tom Lane <tgl@sss.pgh.pa.us>
2006-01-04 03:23 ` Re: improving write performance for logging application dlang <dlang@invendra.net>
@ 2006-01-04 14:13 ` Steve Eckmann <eckmann@computer.org>
0 siblings, 0 replies; 13+ messages in thread
From: Steve Eckmann @ 2006-01-04 14:13 UTC (permalink / raw)
To: dlang <dlang@invendra.net>; +Cc: pgsql-performance
dlang wrote:
>On Tue, 3 Jan 2006, Tom Lane wrote:
>
>
>
>>Steve Eckmann <eckmann@computer.org> writes:
>>
>>
>>>We also found that we could improve MySQL performance significantly
>>>using MySQL's "INSERT" command extension allowing multiple value-list
>>>tuples in a single command; the rate for MyISAM tables improved to
>>>about 2600 objects/second. PostgreSQL doesn't support that language
>>>extension. Using the COPY command instead of INSERT might help, but
>>>since rows are being generated on the fly, I don't see how to use COPY
>>>without running a separate process that reads rows from the
>>>application and uses COPY to write to the database.
>>>
>>>
>>Can you conveniently alter your application to batch INSERT commands
>>into transactions? Ie
>>
>> BEGIN;
>> INSERT ...;
>> ... maybe 100 or so inserts ...
>> COMMIT;
>> BEGIN;
>> ... lather, rinse, repeat ...
>>
>>This cuts down the transactional overhead quite a bit. A downside is
>>that you lose multiple rows if any INSERT fails, but then the same would
>>be true of multiple VALUES lists per INSERT.
>>
>>
>
>Steve, you mentioned that you data collector buffers the data before
>sending it to the database, modify it so that each time it goes to send
>things to the database you send all the data that's in the buffer as a
>single transaction.
>
>I am working on useing postgres to deal with log data and wrote a simple
>perl script that read in the log files a line at a time, and then wrote
>them 1000 at a time to the database. On a dual Opteron 240 box with 2G of
>ram 1x 15krpm SCSI drive (and a untuned postgress install with the compile
>time defaults) I was getting 5000-8000 lines/sec (I think this was with
>fsync disabled, but I don't remember for sure). and postgres was
>complaining that it was overrunning it's log sizes (which limits the speed
>as it then has to pause to flush the logs)
>
>the key thing is to send multiple lines with one transaction as tom shows
>above.
>
>David Lang
>
Thanks, David. I will look more carefully at how to batch multiple rows
per PQexec() call. Regards, Steve.
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: improving write performance for logging application
2006-01-03 23:44 improving write performance for logging application Steve Eckmann <eckmann@computer.org>
2006-01-04 00:00 ` Re: improving write performance for logging application Tom Lane <tgl@sss.pgh.pa.us>
@ 2006-01-04 14:00 ` Steve Eckmann <eckmann@computer.org>
2006-01-04 15:39 ` Re: improving write performance for logging application Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 13+ messages in thread
From: Steve Eckmann @ 2006-01-04 14:00 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-performance
Tom Lane wrote:
>Steve Eckmann <eckmann@computer.org> writes:
>
>
>>We also found that we could improve MySQL performance significantly
>>using MySQL's "INSERT" command extension allowing multiple value-list
>>tuples in a single command; the rate for MyISAM tables improved to
>>about 2600 objects/second. PostgreSQL doesn't support that language
>>extension. Using the COPY command instead of INSERT might help, but
>>since rows are being generated on the fly, I don't see how to use COPY
>>without running a separate process that reads rows from the
>>application and uses COPY to write to the database.
>>
>>
>
>Can you conveniently alter your application to batch INSERT commands
>into transactions? Ie
>
> BEGIN;
> INSERT ...;
> ... maybe 100 or so inserts ...
> COMMIT;
> BEGIN;
> ... lather, rinse, repeat ...
>
>This cuts down the transactional overhead quite a bit. A downside is
>that you lose multiple rows if any INSERT fails, but then the same would
>be true of multiple VALUES lists per INSERT.
>
> regards, tom lane
>
>
Thanks for the suggestion, Tom. Yes, I think I could do that. But I
thought what I was doing now was effectively the same, because the
PostgreSQL 8.0.0 Documentation says (section 27.3.1): "It is allowed to
include multiple SQL commands (separated by semicolons) in the command
string. Multiple queries sent in a single PQexec call are processed in a
single transaction...." Our simulation application has nearly 400 event
types, each of which is a C++ class for which we have a corresponding
database table. So every thousand events or so I issue one PQexec() call
for each event type that has unlogged instances, sending INSERT commands
for all instances. For example,
PQexec(dbConn, "INSERT INTO FlyingObjectState VALUES (...); INSERT
INTO FlyingObjectState VALUES (...); ...");
My thought was that this would be a good compromise between minimizing
transactions (one per event class per buffering interval instead of one
per event) and minimizing disk seeking (since all queries in a single
transaction insert rows into the same table). Am I overlooking something
here? One thing I haven't tried is increasing the buffering interval
from 1000 events to, say, 10,000. It turns out that 1000 is a good
number for Versant, the object database system we're replacing, and for
MySQL, so I assumed it would be a good number for PostgreSQL, too.
Regards, Steve
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: improving write performance for logging application
2006-01-03 23:44 improving write performance for logging application Steve Eckmann <eckmann@computer.org>
2006-01-04 00:00 ` Re: improving write performance for logging application Tom Lane <tgl@sss.pgh.pa.us>
2006-01-04 14:00 ` Re: improving write performance for logging application Steve Eckmann <eckmann@computer.org>
@ 2006-01-04 15:39 ` Tom Lane <tgl@sss.pgh.pa.us>
2006-01-05 00:16 ` Re: improving write performance for logging application Steve Eckmann <eckmann@computer.org>
0 siblings, 1 reply; 13+ messages in thread
From: Tom Lane @ 2006-01-04 15:39 UTC (permalink / raw)
To: Steve Eckmann <eckmann@computer.org>; +Cc: pgsql-performance
Steve Eckmann <eckmann@computer.org> writes:
> Thanks for the suggestion, Tom. Yes, I think I could do that. But I
> thought what I was doing now was effectively the same, because the
> PostgreSQL 8.0.0 Documentation says (section 27.3.1): "It is allowed to
> include multiple SQL commands (separated by semicolons) in the command
> string. Multiple queries sent in a single PQexec call are processed in a
> single transaction...." Our simulation application has nearly 400 event
> types, each of which is a C++ class for which we have a corresponding
> database table. So every thousand events or so I issue one PQexec() call
> for each event type that has unlogged instances, sending INSERT commands
> for all instances. For example,
> PQexec(dbConn, "INSERT INTO FlyingObjectState VALUES (...); INSERT
> INTO FlyingObjectState VALUES (...); ...");
Hmm. I'm not sure if that's a good idea or not. You're causing the
server to take 1000 times the normal amount of memory to hold the
command parsetrees, and if there are any O(N^2) behaviors in parsing
you could be getting hurt badly by that. (I'd like to think there are
not, but would definitely not swear to it.) OTOH you're reducing the
number of network round trips which is a good thing. Have you actually
measured to see what effect this approach has? It might be worth
building a test server with profiling enabled to see if the use of such
long command strings creates any hot spots in the profile.
regards, tom lane
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: improving write performance for logging application
2006-01-03 23:44 improving write performance for logging application Steve Eckmann <eckmann@computer.org>
2006-01-04 00:00 ` Re: improving write performance for logging application Tom Lane <tgl@sss.pgh.pa.us>
2006-01-04 14:00 ` Re: improving write performance for logging application Steve Eckmann <eckmann@computer.org>
2006-01-04 15:39 ` Re: improving write performance for logging application Tom Lane <tgl@sss.pgh.pa.us>
@ 2006-01-05 00:16 ` Steve Eckmann <eckmann@computer.org>
0 siblings, 0 replies; 13+ messages in thread
From: Steve Eckmann @ 2006-01-05 00:16 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-performance
Tom Lane wrote:
>Steve Eckmann <eckmann@computer.org> writes:
>
>
>> <>Thanks for the suggestion, Tom. Yes, I think I could do that. But I
>> thought what I was doing now was effectively the same, because the
>> PostgreSQL 8.0.0 Documentation says (section 27.3.1): "It is allowed to
>> include multiple SQL commands (separated by semicolons) in the command
>> string. Multiple queries sent in a single PQexec call are processed in a
>> single transaction...." Our simulation application has nearly 400 event
>> types, each of which is a C++ class for which we have a corresponding
>> database table. So every thousand events or so I issue one PQexec() call
>> for each event type that has unlogged instances, sending INSERT commands
>> for all instances. For example,
>
>> PQexec(dbConn, "INSERT INTO FlyingObjectState VALUES (...); INSERT
>>INTO FlyingObjectState VALUES (...); ...");
>>
>>
>
>Hmm. I'm not sure if that's a good idea or not. You're causing the
>server to take 1000 times the normal amount of memory to hold the
>command parsetrees, and if there are any O(N^2) behaviors in parsing
>you could be getting hurt badly by that. (I'd like to think there are
>not, but would definitely not swear to it.) OTOH you're reducing the
>number of network round trips which is a good thing. Have you actually
>measured to see what effect this approach has? It might be worth
>building a test server with profiling enabled to see if the use of such
>long command strings creates any hot spots in the profile.
>
> regards, tom lane
>
>
No, I haven't measured it. I will compare this approach with others that
have been suggested. Thanks. -steve
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: improving write performance for logging application
2006-01-03 23:44 improving write performance for logging application Steve Eckmann <eckmann@computer.org>
@ 2006-01-04 00:06 ` Steinar H. Gunderson <sgunderson@bigfoot.com>
2006-01-04 14:08 ` Re: improving write performance for logging application Steve Eckmann <eckmann@computer.org>
2 siblings, 1 reply; 13+ messages in thread
From: Steinar H. Gunderson @ 2006-01-04 00:06 UTC (permalink / raw)
To: Steve Eckmann <eckmann@computer.org>; +Cc: pgsql-performance
On Tue, Jan 03, 2006 at 04:44:28PM -0700, Steve Eckmann wrote:
> Are there general guidelines for tuning the PostgreSQL server for this kind
> of application? The suggestions I've found include disabling fsync (done),
Are you sure you really want this? The results could be catastrophic in case
of a crash.
> On the client side I've found only two suggestions: disable autocommit and
> use COPY instead of INSERT. I think I've effectively disabled autocommit by
> batching up to several hundred INSERT commands in each PQexec() call, and
> it isn’t clear that COPY is worth the effort in our application.
I'm a bit confused here: How can you batch multiple INSERTs into large
statements for MySQL, but not batch multiple INSERTs into COPY statements for
PostgreSQL?
Anyhow, putting it all inside one transaction (or a few) is likely to help
quite a lot, but of course less when you have fsync=false. Bunding multiple
statements in each PQexec() call won't really give you that; you'll have to
tell the database so explicitly.
/* Steinar */
--
Homepage: http://www.sesse.net/
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: improving write performance for logging application
2006-01-03 23:44 improving write performance for logging application Steve Eckmann <eckmann@computer.org>
2006-01-04 00:06 ` Re: improving write performance for logging application Steinar H. Gunderson <sgunderson@bigfoot.com>
@ 2006-01-04 14:08 ` Steve Eckmann <eckmann@computer.org>
2006-01-04 15:40 ` Re: improving write performance for logging application Kelly Burkhart <kelly@kkcsm.net>
0 siblings, 1 reply; 13+ messages in thread
From: Steve Eckmann @ 2006-01-04 14:08 UTC (permalink / raw)
To: Steinar H. Gunderson <sgunderson@bigfoot.com>; +Cc: pgsql-performance
Steinar H. Gunderson wrote:
>On Tue, Jan 03, 2006 at 04:44:28PM -0700, Steve Eckmann wrote:
>
>
>>Are there general guidelines for tuning the PostgreSQL server for this kind
>>of application? The suggestions I've found include disabling fsync (done),
>>
>>
>
>Are you sure you really want this? The results could be catastrophic in case
>of a crash.
>
>
>
>>On the client side I've found only two suggestions: disable autocommit and
>>use COPY instead of INSERT. I think I've effectively disabled autocommit by
>>batching up to several hundred INSERT commands in each PQexec() call, and
>>it isn’t clear that COPY is worth the effort in our application.
>>
>>
>
>I'm a bit confused here: How can you batch multiple INSERTs into large
>statements for MySQL, but not batch multiple INSERTs into COPY statements for
>PostgreSQL?
>
>Anyhow, putting it all inside one transaction (or a few) is likely to help
>quite a lot, but of course less when you have fsync=false. Bunding multiple
>statements in each PQexec() call won't really give you that; you'll have to
>tell the database so explicitly.
>
>/* Steinar */
>
>
Thanks, Steinar. I don't think we would really run with fsync off, but I
need to document the performance tradeoffs. You're right that my
explanation was confusing; probably because I'm confused about how to
use COPY! I could batch multiple INSERTS using COPY statements, I just
don't see how to do it without adding another process to read from
STDIN, since the application that is currently the database client is
constructing rows on the fly. I would need to get those rows into some
process's STDIN stream or into a server-side file before COPY could be
used, right?
You're comment about bundling multiple statements in each PQexec() call
seems to disagree with a statement in 27.3.1 that I interpret as saying
each PQexec() call corresponds to a single transaction. Are you sure my
interpretation is wrong?
Regards, Steve
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: improving write performance for logging application
2006-01-03 23:44 improving write performance for logging application Steve Eckmann <eckmann@computer.org>
2006-01-04 00:06 ` Re: improving write performance for logging application Steinar H. Gunderson <sgunderson@bigfoot.com>
2006-01-04 14:08 ` Re: improving write performance for logging application Steve Eckmann <eckmann@computer.org>
@ 2006-01-04 15:40 ` Kelly Burkhart <kelly@kkcsm.net>
2006-01-05 00:19 ` Re: improving write performance for logging application Steve Eckmann <eckmann@computer.org>
0 siblings, 1 reply; 13+ messages in thread
From: Kelly Burkhart @ 2006-01-04 15:40 UTC (permalink / raw)
To: Steve Eckmann <eckmann@computer.org>; +Cc: pgsql-performance
On 1/4/06, Steve Eckmann <eckmann@computer.org> wrote:
>
> Thanks, Steinar. I don't think we would really run with fsync off, but I
> need to document the performance tradeoffs. You're right that my explanation
> was confusing; probably because I'm confused about how to use COPY! I could
> batch multiple INSERTS using COPY statements, I just don't see how to do it
> without adding another process to read from STDIN, since the application
> that is currently the database client is constructing rows on the fly. I
> would need to get those rows into some process's STDIN stream or into a
> server-side file before COPY could be used, right?
Steve,
You can use copy without resorting to another process. See the libpq
documentation for 'Functions Associated with the copy Command". We do
something like this:
char *mbuf;
// allocate space and fill mbuf with appropriately formatted data somehow
PQexec( conn, "begin" );
PQexec( conn, "copy mytable from stdin" );
PQputCopyData( conn, mbuf, strlen(mbuf) );
PQputCopyEnd( conn, NULL );
PQexec( conn, "commit" );
-K
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: improving write performance for logging application
2006-01-03 23:44 improving write performance for logging application Steve Eckmann <eckmann@computer.org>
2006-01-04 00:06 ` Re: improving write performance for logging application Steinar H. Gunderson <sgunderson@bigfoot.com>
2006-01-04 14:08 ` Re: improving write performance for logging application Steve Eckmann <eckmann@computer.org>
2006-01-04 15:40 ` Re: improving write performance for logging application Kelly Burkhart <kelly@kkcsm.net>
@ 2006-01-05 00:19 ` Steve Eckmann <eckmann@computer.org>
0 siblings, 0 replies; 13+ messages in thread
From: Steve Eckmann @ 2006-01-05 00:19 UTC (permalink / raw)
To: Kelly Burkhart <kelly@kkcsm.net>; +Cc: pgsql-performance
Kelly Burkhart wrote:
> On 1/4/06, Steve Eckmann <eckmann@computer.org
> <mailto:eckmann@computer.org>> wrote:
>
> Thanks, Steinar. I don't think we would really run with fsync off,
> but I need to document the performance tradeoffs. You're right
> that my explanation was confusing; probably because I'm confused
> about how to use COPY! I could batch multiple INSERTS using COPY
> statements, I just don't see how to do it without adding another
> process to read from STDIN, since the application that is
> currently the database client is constructing rows on the fly. I
> would need to get those rows into some process's STDIN stream or
> into a server-side file before COPY could be used, right?
>
>
> Steve,
>
> You can use copy without resorting to another process. See the libpq
> documentation for 'Functions Associated with the copy Command". We do
> something like this:
>
> char *mbuf;
>
> // allocate space and fill mbuf with appropriately formatted data somehow
>
> PQexec( conn, "begin" );
> PQexec( conn, "copy mytable from stdin" );
> PQputCopyData( conn, mbuf, strlen(mbuf) );
> PQputCopyEnd( conn, NULL );
> PQexec( conn, "commit" );
>
> -K
Thanks for the concrete example, Kelly. I had read the relevant libpq
doc but didn't put the pieces together.
Regards, Steve
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: improving write performance for logging application
2006-01-03 23:44 improving write performance for logging application Steve Eckmann <eckmann@computer.org>
@ 2006-01-04 13:54 ` Ian Westmacott <ianw@intellivid.com>
2006-01-04 14:16 ` Re: improving write performance for logging application Steve Eckmann <eckmann@computer.org>
2 siblings, 1 reply; 13+ messages in thread
From: Ian Westmacott @ 2006-01-04 13:54 UTC (permalink / raw)
To: Steve Eckmann <eckmann@computer.org>; +Cc: pgsql-performance
We have a similar application thats doing upwards of 2B inserts
per day. We have spent a lot of time optimizing this, and found the
following to be most beneficial:
1) use COPY (BINARY if possible)
2) don't use triggers or foreign keys
3) put WAL and tables on different spindles (channels if possible)
4) put as much as you can in each COPY, and put as many COPYs as
you can in a single transaction.
5) watch out for XID wraparound
6) tune checkpoint* and bgwriter* parameters for your I/O system
On Tue, 2006-01-03 at 16:44 -0700, Steve Eckmann wrote:
> I have questions about how to improve the write performance of PostgreSQL for logging data from a real-time simulation. We found that MySQL 4.1.3 could log about 1480 objects/second using MyISAM tables or about 1225 objects/second using InnoDB tables, but PostgreSQL 8.0.3 could log only about 540 objects/second. (test system: quad-Itanium2, 8GB memory, SCSI RAID, GigE connection from simulation server, nothing running except system processes and database system under test)
>
> We also found that we could improve MySQL performance significantly using MySQL's "INSERT" command extension allowing multiple value-list tuples in a single command; the rate for MyISAM tables improved to about 2600 objects/second. PostgreSQL doesn't support that language extension. Using the COPY command instead of INSERT might help, but since rows are being generated on the fly, I don't see how to use COPY without running a separate process that reads rows from the application and uses COPY to write to the database. The application currently has two processes: the simulation and a data collector that reads events from the sim (queued in shared memory) and writes them as rows to the database, buffering as needed to avoid lost data during periods of high activity. To use COPY I think we would have to split our data collector into two processes communicating via a pipe.
>
> Query performance is not an issue: we found that when suitable indexes are added PostgreSQL is fast enough on the kinds of queries our users make. The crux is writing rows to the database fast enough to keep up with the simulation.
>
> Are there general guidelines for tuning the PostgreSQL server for this kind of application? The suggestions I've found include disabling fsync (done), increasing the value of wal_buffers, and moving the WAL to a different disk, but these aren't likely to produce the 3x improvement that we need. On the client side I've found only two suggestions: disable autocommit and use COPY instead of INSERT. I think I've effectively disabled autocommit by batching up to several hundred INSERT commands in each PQexec() call, and it isn’t clear that COPY is worth the effort in our application.
>
> Thanks.
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: Don't 'kill -9' the postmaster
--
Ian Westmacott <ianw@intellivid.com>
Intellivid Corp.
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: improving write performance for logging application
2006-01-03 23:44 improving write performance for logging application Steve Eckmann <eckmann@computer.org>
2006-01-04 13:54 ` Re: improving write performance for logging application Ian Westmacott <ianw@intellivid.com>
@ 2006-01-04 14:16 ` Steve Eckmann <eckmann@computer.org>
0 siblings, 0 replies; 13+ messages in thread
From: Steve Eckmann @ 2006-01-04 14:16 UTC (permalink / raw)
To: Ian Westmacott <ianw@intellivid.com>; +Cc: pgsql-performance
Ian Westmacott wrote:
>We have a similar application thats doing upwards of 2B inserts
>per day. We have spent a lot of time optimizing this, and found the
>following to be most beneficial:
>
>1) use COPY (BINARY if possible)
>2) don't use triggers or foreign keys
>3) put WAL and tables on different spindles (channels if possible)
>4) put as much as you can in each COPY, and put as many COPYs as
> you can in a single transaction.
>5) watch out for XID wraparound
>6) tune checkpoint* and bgwriter* parameters for your I/O system
>
Thanks, Ian. I will look at how to implement your suggestions.
Regards, Steve
^ permalink raw reply [nested|flat] 13+ messages in thread
end of thread, other threads:[~2006-01-05 00:19 UTC | newest]
Thread overview: 13+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2006-01-03 23:44 improving write performance for logging application Steve Eckmann <eckmann@computer.org>
2006-01-04 00:00 ` Tom Lane <tgl@sss.pgh.pa.us>
2006-01-04 03:23 ` dlang <dlang@invendra.net>
2006-01-04 14:13 ` Steve Eckmann <eckmann@computer.org>
2006-01-04 14:00 ` Steve Eckmann <eckmann@computer.org>
2006-01-04 15:39 ` Tom Lane <tgl@sss.pgh.pa.us>
2006-01-05 00:16 ` Steve Eckmann <eckmann@computer.org>
2006-01-04 00:06 ` Steinar H. Gunderson <sgunderson@bigfoot.com>
2006-01-04 14:08 ` Steve Eckmann <eckmann@computer.org>
2006-01-04 15:40 ` Kelly Burkhart <kelly@kkcsm.net>
2006-01-05 00:19 ` Steve Eckmann <eckmann@computer.org>
2006-01-04 13:54 ` Ian Westmacott <ianw@intellivid.com>
2006-01-04 14:16 ` Steve Eckmann <eckmann@computer.org>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox