X-Original-To: pgsql-interfaces-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id 5EB88D9F76
	for <pgsql-interfaces-postgresql.org@localhost.postgresql.org>;
	Wed,  9 Nov 2005 14:05:20 -0400 (AST)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 15762-08
	for <pgsql-interfaces-postgresql.org@localhost.postgresql.org>;
	Wed,  9 Nov 2005 18:05:17 +0000 (GMT)
X-Greylist: from auto-whitelisted by SQLgrey-
Received: from svr2.postgresql.org (svr2.postgresql.org [65.19.161.25])
	by svr1.postgresql.org (Postfix) with ESMTP id 1986ED9CF1
	for <pgsql-interfaces@postgresql.org>;
	Wed,  9 Nov 2005 14:05:15 -0400 (AST)
Received: from tigger.fuhr.org (tigger.fuhr.org [63.214.45.158])
	by svr2.postgresql.org (Postfix) with ESMTP id 27C73F1031
	for <pgsql-interfaces@postgresql.org>;
	Wed,  9 Nov 2005 18:05:14 +0000 (GMT)
Received: from winnie.fuhr.org (winnie.fuhr.org [10.1.0.1])
	by tigger.fuhr.org (8.13.3/8.13.3) with ESMTP id jA9I55Of034438
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
	Wed, 9 Nov 2005 11:05:08 -0700 (MST)
	(envelope-from mfuhr@winnie.fuhr.org)
Received: from winnie.fuhr.org (localhost [127.0.0.1])
	by winnie.fuhr.org (8.13.4/8.13.4) with ESMTP id jA9I55bq041982;
	Wed, 9 Nov 2005 11:05:05 -0700 (MST)
	(envelope-from mfuhr@winnie.fuhr.org)
Received: (from mfuhr@localhost)
	by winnie.fuhr.org (8.13.4/8.13.4/Submit) id jA9I556o041981;
	Wed, 9 Nov 2005 11:05:05 -0700 (MST) (envelope-from mfuhr)
Date: Wed, 9 Nov 2005 11:05:05 -0700
From: Michael Fuhr <mike@fuhr.org>
To: Jon Clements <jon.clements@trgstrata.co.uk>
Cc: "pgsql-interfaces@postgresql.org" <pgsql-interfaces@postgresql.org>
Subject: Re: PLPythonU & Out of Memory - Importing Query
Message-ID: <20051109180505.GA41806@winnie.fuhr.org>
References: <4c05f1b439ce248766e7cd5375a83ddc@readgroup.co.uk>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <4c05f1b439ce248766e7cd5375a83ddc@readgroup.co.uk>
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, score=0.006 required=5 tests=[AWL=0.006]
X-Spam-Score: 0.006
X-Spam-Level: 
X-Archive-Number: 200511/10
X-Sequence-Number: 5806

On Tue, Nov 08, 2005 at 02:21:01PM +0000, Jon Clements wrote:
> I am currently experimenting using plpythonu with postgresql 8.0
> for Win32. It's basically a quick script that imports data from CSV
> files, but does some quite complicated data lookups and selections.
> The area in which I'm somewhat confunded is memory usage. The process
> successfully runs, but keeps climbing in memory usage relentlessly,
> successfully importing about 200k records, before the memory usage
> of postgres soars to 2gb and of course, shortly after that, grinds
> to a halt with a "Out of Memory" error.
[ ... ]
> # Setup external CSV data source
> # For each record, that meets certain critera, execute insert...
> for rec in dsource: plpy.execute(myplan, [Value1, Value2] )

What exactly is dsource?  If it were a simple list that was built
in its entirety prior to the "for" loop then I'd expect that to
cause memory problems; however, since you're getting a partial data
load I'm wondering if dsource is some other kind of object that
implements an iterator.  Have you run the same code (minus the
database parts) in a standalone Python program to see if it has a
memory leak?

-- 
Michael Fuhr