Date: Thu, 6 Oct 2005 18:57:30 +0200
From: Martijn van Oosterhout <kleptog@svana.org>
To: Ron Peacetree <rjpeace@earthlink.net>
Cc: "Jonah H. Harris" <jonah.harris@gmail.com>,
	pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org
Subject: Re: [PERFORM] A Better External Sort?
Message-ID: <20051006165725.GF10127@svana.org>
Reply-To: Martijn van Oosterhout <kleptog@svana.org>
References: 
 <138872.1128556455518.JavaMail.root@elwamui-milano.atl.sa.earthlink.net>
Mime-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="gdTfX7fkYsEEjebm"
Content-Disposition: inline
In-Reply-To: 
 <138872.1128556455518.JavaMail.root@elwamui-milano.atl.sa.earthlink.net>
User-Agent: Mutt/1.3.28i


--gdTfX7fkYsEEjebm
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Wed, Oct 05, 2005 at 07:54:15PM -0400, Ron Peacetree wrote:
> I asked some questions about physical layout and format translation
> overhead being possibly suboptimal that seemed to be agreed to, but
> specifics as to where we are taking the hit don't seem to have been
> made explicit yet.

This hit is easy to see and terribly hard to do anything about at the
same time. Any single row in a table stores its values but the offsets
arn't constant. If a field is NULL, it is skipped. If a field is
variable length, you have to look at the length before you can jump
over to the next value.

If you have no NULLs and no variable length fields, then you can
optimise access. This is already done and it's hard to see how you
could improve it further. To cut costs, many places use
heap_deform_tuple and similar routines so that the costs are reduced,
but they're still there.

Upping the data transfer rate from disk is a worthy goal, just some
people beleive it is of lower priority than improving CPU usage.

> We were also told that evidently we are CPU bound far before one
> would naively expect to be based on the performance specifications
> of the components involved.

As someone pointed out, calls to the C library are not counted
seperately, making it harder to see if we're overcalling some of them.
Pinpointing the performance bottleneck is hard work.

> Now I've asked for the quickest path to detailed understanding of the
> pg IO subsystem.  The goal being to get more up to speed on its
> coding details.  Certainly not to annoy you or anyone else.

Well, the work is all in storage/smgr and storage/file. It's not
terribly complicated, it just sometimes takes a while to understand
*why* it is done this way.

Indeed, one of the things on my list is to remove all the lseeks in
favour of pread. Halving the number of kernel calls has got to be worth
something right? Portability is an issue ofcourse...

But it's been a productive thread, absolutly. Progress has been made...

Have a nice day,
--=20
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.

--gdTfX7fkYsEEjebm
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQFDRVd0IB7bNG8LQkwRApCLAJ0ezBpzEpD8ubpKGjEjhhcPq5Oo6gCdGtZc
q22gs+81mm52zQpPhUScJHA=
=EN7M
-----END PGP SIGNATURE-----

--gdTfX7fkYsEEjebm--