Date: Fri, 23 Sep 2005 20:32:46 +0200
From: Martijn van Oosterhout <kleptog@svana.org>
To: Pailloncy Jean-Gerard <jg@rilk.com>
Cc: Pg Hackers <pgsql-hackers@postgresql.org>,
	Tom Lane <tgl@sss.pgh.pa.us>
Subject: Re: Releasing memory during External sorting?
Message-ID: <20050923183241.GA12774@svana.org>
Reply-To: Martijn van Oosterhout <kleptog@svana.org>
References: <1127468233.4145.178.camel@localhost.localdomain>
	<3465.1127484579@sss.pgh.pa.us>
	<AC85B14C-FFB0-455B-88E3-6997B46269BE@rilk.com>
Mime-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="RnlQjJ0d97Da+TV1"
Content-Disposition: inline
In-Reply-To: <AC85B14C-FFB0-455B-88E3-6997B46269BE@rilk.com>
User-Agent: Mutt/1.3.28i


--RnlQjJ0d97Da+TV1
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Fri, Sep 23, 2005 at 06:39:35PM +0200, Pailloncy Jean-Gerard wrote:
> >On most platforms it's quite unlikely that any memory would actually
> >get released back to the OS before transaction end, because the
> >memory blocks belonging to the tuplesort context will be intermixed
> >with blocks belonging to other contexts.  So I think this is pretty
> >pointless. (If you can't afford to have the sort using all of
> >sort_mem, you've set sort_mem too large, anyway.)

> On OpenBSD 3.8 malloc use mmap(2) and no more sbrk.
> So, as soon as the bloc is free, it returns to the OS.
> Access to the freed pointer crashs immediatly.

Interesting point. Glibc also uses mmap() but only for allocations
greater than a few K, otherwise it's a waste of space.

I guess you would have to look into the postgresql allocator to see if
it doesn't divide the mmap()ed space up between multiple contexts.
Large allocations certainly appear to be passed off to malloc() but I
don't think execSort allocates all it's space in one go, it just counts
the space allocated by palloc().

So, unless someone goes and adds changes the tuplesort code to allocate
big blocks and use them only for tuples, I think you're going to run
into issues with data interleaved, meaning not much to give back to the
OS...
--=20
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.

--RnlQjJ0d97Da+TV1
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQFDNEpIIB7bNG8LQkwRAl/1AJ0ZfpUlNjKI/zz12vPJbzdetjpjOQCggwWU
UykRmsJH02B6+Ro79zXIO+Y=
=f+1O
-----END PGP SIGNATURE-----

--RnlQjJ0d97Da+TV1--