From: Michael Fuhr <mike@fuhr.org>
To: Jon Clements <jon.clements@trgstrata.co.uk>
Cc: pgsql-interfaces@postgresql.org <pgsql-interfaces@postgresql.org>
Subject: Re: PLPythonU & Out of Memory - Importing Query
Date: Wed, 9 Nov 2005 11:05:05 -0700
Message-ID: <20051109180505.GA41806@winnie.fuhr.org> (raw)
In-Reply-To: <4c05f1b439ce248766e7cd5375a83ddc@readgroup.co.uk>
References: <4c05f1b439ce248766e7cd5375a83ddc@readgroup.co.uk>
On Tue, Nov 08, 2005 at 02:21:01PM +0000, Jon Clements wrote:
> I am currently experimenting using plpythonu with postgresql 8.0
> for Win32. It's basically a quick script that imports data from CSV
> files, but does some quite complicated data lookups and selections.
> The area in which I'm somewhat confunded is memory usage. The process
> successfully runs, but keeps climbing in memory usage relentlessly,
> successfully importing about 200k records, before the memory usage
> of postgres soars to 2gb and of course, shortly after that, grinds
> to a halt with a "Out of Memory" error.
[ ... ]
> # Setup external CSV data source
> # For each record, that meets certain critera, execute insert...
> for rec in dsource: plpy.execute(myplan, [Value1, Value2] )
What exactly is dsource? If it were a simple list that was built
in its entirety prior to the "for" loop then I'd expect that to
cause memory problems; however, since you're getting a partial data
load I'm wondering if dsource is some other kind of object that
implements an iterator. Have you run the same code (minus the
database parts) in a standalone Python program to see if it has a
memory leak?
--
Michael Fuhr
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-interfaces@postgresql.org
Cc: mike@fuhr.org, jon.clements@trgstrata.co.uk
Subject: Re: PLPythonU & Out of Memory - Importing Query
In-Reply-To: <20051109180505.GA41806@winnie.fuhr.org>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox