From: Tom Lane <tgl@sss.pgh.pa.us>
To: Chris Browne <cbbrowne@acm.org>
Cc: pgsql-performance@postgresql.org
Subject: Re: Very long SQL strings
Date: Thu, 21 Jun 2007 16:59:18 -0400
Message-ID: <5574.1182459558@sss.pgh.pa.us> (raw)
In-Reply-To: <608xad6q6f.fsf@dba2.int.libertyrms.com>
References: <357fa7590706211133u2ae43d09id1653ca63d7a7838@mail.gmail.com>
<2239.1182451554@sss.pgh.pa.us>
<608xad6q6f.fsf@dba2.int.libertyrms.com>
Chris Browne <cbbrowne@acm.org> writes:
> I once ran into the situation where Slony-I generated a query that
> made the parser blow out (some sort of memory problem / running out of
> stack space somewhere thing); it was just short of 640K long, and so
> we figured that evidently it was wrong to conclude that "640K ought to
> be enough for anybody."
> Neil Conway was an observer; he was speculating that, with some
> (possibly nontrivial) change to the parser, we should have been able
> to cope with it.
> The query consisted mostly of a NOT IN clause where the list had some
> atrocious number of entries in it (all integers).
FWIW, we do seem to have improved that as of 8.2. Assuming your entries
were 6-or-so-digit integers, that would have been on the order of 80K
entries, and we can manage it --- not amazingly fast, but it doesn't
blow out the stack anymore.
> (Aside: I wound up writing a "query compressor" (now in 1.2) which
> would read that list and, if it was at all large, try to squeeze any
> sets of consecutive integers into sets of "NOT BETWEEN" clauses.
> Usually, the lists, of XIDs, were more or less consecutive, and
> frequently, in the cases where the query got to MBs in size, there
> would be sets of hundreds or even thousands of consecutive integers
> such that we'd be left with a tiny query after this...)
Probably still a win.
regards, tom lane
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: tgl@sss.pgh.pa.us, cbbrowne@acm.org
Subject: Re: Very long SQL strings
In-Reply-To: <5574.1182459558@sss.pgh.pa.us>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox