Date: Sat, 1 Oct 2005 18:19:41 +0200
From: Martijn van Oosterhout <kleptog@svana.org>
To: Ron Peacetree <rjpeace@earthlink.net>
Cc: pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org
Subject: Re: [HACKERS] A Better External Sort?
Message-ID: <20051001161936.GE13830@svana.org>
Reply-To: Martijn van Oosterhout <kleptog@svana.org>
References: 
 <25552174.1128176560482.JavaMail.root@elwamui-polski.atl.sa.earthlink.net>
Mime-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="N1GIdlSm9i+YlY4t"
Content-Disposition: inline
In-Reply-To: 
 <25552174.1128176560482.JavaMail.root@elwamui-polski.atl.sa.earthlink.net>
User-Agent: Mutt/1.3.28i


--N1GIdlSm9i+YlY4t
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Sat, Oct 01, 2005 at 10:22:40AM -0400, Ron Peacetree wrote:
> Assuming we get the abyssmal physical IO performance fixed...
> (because until we do, _nothing_ is going to help us as much)

I'm still not convinced this is the major problem. For example, in my
totally unscientific tests on an oldish machine I have here:

Direct filesystem copy to /dev/null
21MB/s    10% user 50% system  (dual cpu, so the system is using a whole CP=
U)

COPY TO /dev/null WITH binary
13MB/s    55% user 45% system  (ergo, CPU bound)

COPY TO /dev/null
4.4MB/s   60% user 40% system

\copy to /dev/null in psql
6.5MB/s   60% user 40% system

This machine is a bit strange setup, not sure why fs copy is so slow.
As to why \copy is faster than COPY, I have no idea, but it is
repeatable. And actually turning the tuples into a printable format is
the most expensive. But it does point out that the whole process is
probably CPU bound more than anything else.

So, I don't think physical I/O is the problem. It's something further
up the call tree. I wouldn't be surprised at all it it had to do with
the creation and destruction of tuples. The cost of comparing tuples
should not be underestimated.
--=20
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.

--N1GIdlSm9i+YlY4t
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQFDPrcXIB7bNG8LQkwRAulpAJ9Qk9yKddvygb0qb5BofJy6dD4yTwCfRTXG
GuqPRZ9u7pNqZet021m5h34=
=0vBS
-----END PGP SIGNATURE-----

--N1GIdlSm9i+YlY4t--