pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  
help / color / mirror / Atom feed
Slow copy with little CPU/disk usage
6+ messages / 4 participants
[nested] [flat]

* Slow copy with little CPU/disk usage
@ 2005-04-20 02:00 Jim C. Nasby <decibel@decibel.org>
  2005-04-20 03:05 ` Re: Slow copy with little CPU/disk usage Tom Lane <tgl@sss.pgh.pa.us>
  2005-04-20 04:34 ` Re: Slow copy with little CPU/disk usage Greg Stark <gsstark@mit.edu>
  0 siblings, 2 replies; 6+ messages in thread

From: Jim C. Nasby @ 2005-04-20 02:00 UTC (permalink / raw)
  To: pgsql-performance

A friend of mine has an application where he's copying in 4000 rows at a
time into a table that has about 4M rows. Each row is 40-50 bytes. This
is taking 25 seconds on a dual PIII-1GHz with 1G of RAM and a 2 disk
SATA mirror, running FBSD 4.10-stable. There's one index on the table.

What's really odd is that neither the CPU or the disk are being
hammered. The box appears to be pretty idle; the postgresql proces is
using 4-5% CPU.

I seem to recall others running into this before, but I can't remember
what the issue was and I can't find it in the archives.

This is version 8.0, btw.
-- 
Jim C. Nasby, Database Consultant               decibel@decibel.org 
Give your computer some brain candy! www.distributed.net Team #1828

Windows: "Where do you want to go today?"
Linux: "Where do you want to go tomorrow?"
FreeBSD: "Are you guys coming, or what?"



^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: Slow copy with little CPU/disk usage
  2005-04-20 02:00 Slow copy with little CPU/disk usage Jim C. Nasby <decibel@decibel.org>
@ 2005-04-20 03:05 ` Tom Lane <tgl@sss.pgh.pa.us>
  2005-04-20 04:37   ` Re: Slow copy with little CPU/disk usage Mischa Sandberg <mischa.sandberg@telus.net>
  1 sibling, 1 reply; 6+ messages in thread

From: Tom Lane @ 2005-04-20 03:05 UTC (permalink / raw)
  To: Jim C. Nasby <decibel@decibel.org>; +Cc: pgsql-performance

"Jim C. Nasby" <decibel@decibel.org> writes:
> A friend of mine has an application where he's copying in 4000 rows at a
> time into a table that has about 4M rows. Each row is 40-50 bytes. This
> is taking 25 seconds on a dual PIII-1GHz with 1G of RAM and a 2 disk
> SATA mirror, running FBSD 4.10-stable. There's one index on the table.

If there's no hidden costs such as foreign key checks, that does seem
pretty dang slow.

> What's really odd is that neither the CPU or the disk are being
> hammered. The box appears to be pretty idle; the postgresql proces is
> using 4-5% CPU.

It's very hard to believe that *neither* disk nor CPU is maxed.
Can we see a reproducible test case, please?

			regards, tom lane



^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: Slow copy with little CPU/disk usage
  2005-04-20 02:00 Slow copy with little CPU/disk usage Jim C. Nasby <decibel@decibel.org>
  2005-04-20 03:05 ` Re: Slow copy with little CPU/disk usage Tom Lane <tgl@sss.pgh.pa.us>
@ 2005-04-20 04:37   ` Mischa Sandberg <mischa.sandberg@telus.net>
  2005-04-20 22:22     ` Re: Slow copy with little CPU/disk usage Jim C. Nasby <decibel@decibel.org>
  0 siblings, 1 reply; 6+ messages in thread

From: Mischa Sandberg @ 2005-04-20 04:37 UTC (permalink / raw)
  To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Jim C. Nasby <decibel@decibel.org>; pgsql-performance

Quoting Tom Lane <tgl@sss.pgh.pa.us>: 
 
> "Jim C. Nasby" <decibel@decibel.org> writes: 
> > A friend of mine has an application where he's copying in 4000 rows at a 
> > time into a table that has about 4M rows. Each row is 40-50 bytes. This 
> > is taking 25 seconds on a dual PIII-1GHz with 1G of RAM and a 2 disk 
> > SATA mirror, running FBSD 4.10-stable. There's one index on the table. 
>  
> If there's no hidden costs such as foreign key checks, that does seem 
> pretty dang slow. 
>  
> > What's really odd is that neither the CPU or the disk are being 
> > hammered. The box appears to be pretty idle; the postgresql proces is 
> > using 4-5% CPU. 
--  
This sounds EXACTLY like my problem, if you make the box to a Xeon 2.4GHz, 2GB 
RAM ... with two SCSI drives (xlog and base); loading 10K rows of about 200 
bytes each; takes about 20 secs at the best, and much longer at the worst. By 
any chance does your friend have several client machines/processes trying to 
mass-load rows at the same time? Or at least some other processes updating 
that table in a bulkish way? What I get is low diskio, low cpu, even low 
context-switches ... and I'm betting he should take a look at pg_locks. For my 
own problem, I gather that an exclusive lock is necessary while updating 
indexes and heap, and the multiple processes doing the update can make that 
pathological. 
 
Anyway, have your friend check pg_locks. 
 
 
"Dreams come true, not free." -- S.Sondheim, ITW 




^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: Slow copy with little CPU/disk usage
  2005-04-20 02:00 Slow copy with little CPU/disk usage Jim C. Nasby <decibel@decibel.org>
  2005-04-20 03:05 ` Re: Slow copy with little CPU/disk usage Tom Lane <tgl@sss.pgh.pa.us>
  2005-04-20 04:37   ` Re: Slow copy with little CPU/disk usage Mischa Sandberg <mischa.sandberg@telus.net>
@ 2005-04-20 22:22     ` Jim C. Nasby <decibel@decibel.org>
  0 siblings, 0 replies; 6+ messages in thread

From: Jim C. Nasby @ 2005-04-20 22:22 UTC (permalink / raw)
  To: Mischa Sandberg <mischa.sandberg@telus.net>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; pgsql-performance

No, this is a single process. And there's known issues with context
storms on Xeons, so that might be what you're seeing.

On Tue, Apr 19, 2005 at 09:37:21PM -0700, Mischa Sandberg wrote:
> Quoting Tom Lane <tgl@sss.pgh.pa.us>: 
>  
> > "Jim C. Nasby" <decibel@decibel.org> writes: 
> > > A friend of mine has an application where he's copying in 4000 rows at a 
> > > time into a table that has about 4M rows. Each row is 40-50 bytes. This 
> > > is taking 25 seconds on a dual PIII-1GHz with 1G of RAM and a 2 disk 
> > > SATA mirror, running FBSD 4.10-stable. There's one index on the table. 
> >  
> > If there's no hidden costs such as foreign key checks, that does seem 
> > pretty dang slow. 
> >  
> > > What's really odd is that neither the CPU or the disk are being 
> > > hammered. The box appears to be pretty idle; the postgresql proces is 
> > > using 4-5% CPU. 
> --  
> This sounds EXACTLY like my problem, if you make the box to a Xeon 2.4GHz, 2GB 
> RAM ... with two SCSI drives (xlog and base); loading 10K rows of about 200 
> bytes each; takes about 20 secs at the best, and much longer at the worst. By 
> any chance does your friend have several client machines/processes trying to 
> mass-load rows at the same time? Or at least some other processes updating 
> that table in a bulkish way? What I get is low diskio, low cpu, even low 
> context-switches ... and I'm betting he should take a look at pg_locks. For my 
> own problem, I gather that an exclusive lock is necessary while updating 
> indexes and heap, and the multiple processes doing the update can make that 
> pathological. 
>  
> Anyway, have your friend check pg_locks. 
>  
>  
> "Dreams come true, not free." -- S.Sondheim, ITW 
> 

-- 
Jim C. Nasby, Database Consultant               decibel@decibel.org 
Give your computer some brain candy! www.distributed.net Team #1828

Windows: "Where do you want to go today?"
Linux: "Where do you want to go tomorrow?"
FreeBSD: "Are you guys coming, or what?"



^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: Slow copy with little CPU/disk usage
  2005-04-20 02:00 Slow copy with little CPU/disk usage Jim C. Nasby <decibel@decibel.org>
@ 2005-04-20 04:34 ` Greg Stark <gsstark@mit.edu>
  2005-04-20 22:23   ` Re: Slow copy with little CPU/disk usage Jim C. Nasby <decibel@decibel.org>
  1 sibling, 1 reply; 6+ messages in thread

From: Greg Stark @ 2005-04-20 04:34 UTC (permalink / raw)
  To: Jim C. Nasby <decibel@decibel.org>; +Cc: pgsql-performance

"Jim C. Nasby" <decibel@decibel.org> writes:

> What's really odd is that neither the CPU or the disk are being
> hammered. The box appears to be pretty idle; the postgresql proces is
> using 4-5% CPU.

Is he committing every row? In that case you would see fairly low i/o
bandwidth usage because most of the time is being spent seeking and waiting
for rotational latency.

-- 
greg




^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: Slow copy with little CPU/disk usage
  2005-04-20 02:00 Slow copy with little CPU/disk usage Jim C. Nasby <decibel@decibel.org>
  2005-04-20 04:34 ` Re: Slow copy with little CPU/disk usage Greg Stark <gsstark@mit.edu>
@ 2005-04-20 22:23   ` Jim C. Nasby <decibel@decibel.org>
  0 siblings, 0 replies; 6+ messages in thread

From: Jim C. Nasby @ 2005-04-20 22:23 UTC (permalink / raw)
  To: Greg Stark <gsstark@mit.edu>; +Cc: pgsql-performance

No, he's using either COPY or \COPY.

On Wed, Apr 20, 2005 at 12:34:27AM -0400, Greg Stark wrote:
> "Jim C. Nasby" <decibel@decibel.org> writes:
> 
> > What's really odd is that neither the CPU or the disk are being
> > hammered. The box appears to be pretty idle; the postgresql proces is
> > using 4-5% CPU.
> 
> Is he committing every row? In that case you would see fairly low i/o
> bandwidth usage because most of the time is being spent seeking and waiting
> for rotational latency.
> 
> -- 
> greg
> 

-- 
Jim C. Nasby, Database Consultant               decibel@decibel.org 
Give your computer some brain candy! www.distributed.net Team #1828

Windows: "Where do you want to go today?"
Linux: "Where do you want to go tomorrow?"
FreeBSD: "Are you guys coming, or what?"



^ permalink  raw  reply  [nested|flat] 6+ messages in thread


end of thread, other threads:[~2005-04-20 22:23 UTC | newest]

Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2005-04-20 02:00 Slow copy with little CPU/disk usage Jim C. Nasby <decibel@decibel.org>
2005-04-20 03:05 ` Tom Lane <tgl@sss.pgh.pa.us>
2005-04-20 04:37   ` Mischa Sandberg <mischa.sandberg@telus.net>
2005-04-20 22:22     ` Jim C. Nasby <decibel@decibel.org>
2005-04-20 04:34 ` Greg Stark <gsstark@mit.edu>
2005-04-20 22:23   ` Jim C. Nasby <decibel@decibel.org>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox