pg.ddx.io pgsql-interfaces@postgresql.org mailing list archive
help / color / mirror / Atom feed8K query limit in 6.5?
8+ messages / 4 participants
[nested] [flat]
* 8K query limit in 6.5?
@ 1999-07-13 03:38 Matthew Hagerty <matthew@venux.net>
0 siblings, 2 replies; 8+ messages in thread
From: Matthew Hagerty @ 1999-07-13 03:38 UTC (permalink / raw)
To: pgsql-interfaces
Greetings,
Was the 8K query limit increased in 6.5?
Thanks,
Matthew
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: [INTERFACES] 8K query limit in 6.5?
@ 1999-07-13 13:59 Tom Lane <tgl@sss.pgh.pa.us>
parent: Matthew Hagerty <matthew@venux.net>
1 sibling, 0 replies; 8+ messages in thread
From: Tom Lane @ 1999-07-13 13:59 UTC (permalink / raw)
To: Matthew Hagerty <matthew@venux.net>; +Cc: pgsql-interfaces
Matthew Hagerty <matthew@venux.net> writes:
> Was the 8K query limit increased in 6.5?
Nope.
I tried to stir up some enthusiasm for fixing it in 6.6, but didn't
have much success --- too soon to say whether it will happen for 6.6
or be postponed again.
regards, tom lane
^ permalink raw reply [nested|flat] 8+ messages in thread
* RE: [INTERFACES] 8K query limit in 6.5?
@ 1999-07-13 15:11 Ansley, Michael <Michael.Ansley@intec.co.za>
0 siblings, 1 reply; 8+ messages in thread
From: Ansley, Michael @ 1999-07-13 15:11 UTC (permalink / raw)
To: pgsql-interfaces
Why are these limits even an issue? Surely I should be able to adjust
certain constants in the code (and this would include constants to take care
of the tuple length) in order to adjust the performance of the system. I
realise that life is not that simple, but perhaps it could be made simpler
than it is now. At the moment, some of the constants used are pretty
useless, because they can't really be adjusted.
Can't we come up with a way to make this more amenable to adjustment?
Thoughts...
MikeA
>> -----Original Message-----
>> From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
>> Sent: Tuesday, July 13, 1999 3:59 PM
>> To: Matthew Hagerty
>> Cc: pgsql-interfaces@postgreSQL.org
>> Subject: Re: [INTERFACES] 8K query limit in 6.5?
>>
>>
>> Matthew Hagerty <matthew@venux.net> writes:
>> > Was the 8K query limit increased in 6.5?
>>
>> Nope.
>>
>> I tried to stir up some enthusiasm for fixing it in 6.6, but didn't
>> have much success --- too soon to say whether it will happen for 6.6
>> or be postponed again.
>>
>> regards, tom lane
>>
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: [INTERFACES] 8K query limit in 6.5?
@ 1999-07-13 15:46 Tom Lane <tgl@sss.pgh.pa.us>
parent: Ansley, Michael <Michael.Ansley@intec.co.za>
0 siblings, 0 replies; 8+ messages in thread
From: Tom Lane @ 1999-07-13 15:46 UTC (permalink / raw)
To: Ansley, Michael <Michael.Ansley@intec.co.za>; +Cc: pgsql-interfaces
"Ansley, Michael" <Michael.Ansley@intec.co.za> writes:
> Why are these limits even an issue? Surely I should be able to adjust
> certain constants in the code (and this would include constants to take care
> of the tuple length) in order to adjust the performance of the system. I
> realise that life is not that simple, but perhaps it could be made simpler
> than it is now. At the moment, some of the constants used are pretty
> useless, because they can't really be adjusted.
> Can't we come up with a way to make this more amenable to adjustment?
If it bothers you, step right up and fix it ...
Where I was actually hoping to end up is with no fixed limit at all,
ie make all buffers for text be dynamically resizable to handle the
longest query presented in a given session. But it'd be a useful
intermediate step to find all the places that are dependent on max query
length and drive them all from a single config.h constant. All that
code would eventually be replaced, if we do dynamic sizing, but doing
this would still be helpful as a way of identifying everything that
needs to change. Just finding all the places that depend on max query
length is not as trivial as you might think.
regards, tom lane
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: [INTERFACES] 8K query limit in 6.5?
@ 1999-07-13 16:02 Matthew Hagerty <matthew@venux.net>
0 siblings, 1 reply; 8+ messages in thread
From: Matthew Hagerty @ 1999-07-13 16:02 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-interfaces
stir, stir, stir... please make this bigger! Or tell me how and I'll do it. :)
Is it really such a big deal to increase the size of a query that pg can
handle? I'm talking about the size of a query submitted to pg, from what I
understand there is no limit on the record (tuple) size that can be
returned from pg. Is my understanding correct?
Thanks,
Matthew
At 09:59 AM 7/13/99 -0400, Tom Lane wrote:
>Matthew Hagerty <matthew@venux.net> writes:
>> Was the 8K query limit increased in 6.5?
>
>Nope.
>
>I tried to stir up some enthusiasm for fixing it in 6.6, but didn't
>have much success --- too soon to say whether it will happen for 6.6
>or be postponed again.
>
> regards, tom lane
>
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: [INTERFACES] 8K query limit in 6.5?
@ 1999-07-13 16:42 Tom Lane <tgl@sss.pgh.pa.us>
parent: Matthew Hagerty <matthew@venux.net>
0 siblings, 0 replies; 8+ messages in thread
From: Tom Lane @ 1999-07-13 16:42 UTC (permalink / raw)
To: Matthew Hagerty <matthew@venux.net>; +Cc: pgsql-interfaces
Matthew Hagerty <matthew@venux.net> writes:
> ... please make this bigger! Or tell me how and I'll do it. :)
"Find all the places that need fixing, and fix 'em." :-)
The devil is in the details ... and there are a lot of details.
You could probably get some useful functionality just by fixing
frontend libpq's output buffer and the query input buffer in
backend/tcop/postgres.c. The trouble comes in cleaning up all the
details, like fixed-size buffers in COPY, pg_dump, etc etc etc etc.
Michael Ansley seems to be interested in working on this; I'd
suggest getting together with him if you want to help.
> Is it really such a big deal to increase the size of a query that pg can
> handle? I'm talking about the size of a query submitted to pg, from what I
> understand there is no limit on the record (tuple) size that can be
> returned from pg. Is my understanding correct?
There is a (quite unrelated) limit on the size of a tuple that can be
stored on disk, but AFAIK you can construct an arbitrarily large tuple
as the result of computations specified in a SELECT and it will be
passed back to your frontend application without trouble; as long as
you don't run out of memory on either side, that is.
Where we want to end up, of course, is no limit on the size of a disk
tuple either, but the code changes for that are almost entirely
unrelated to the ones that deal with the textual length of a query.
It's best tackled as two independent projects.
regards, tom lane
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: [INTERFACES] 8K query limit in 6.5?
@ 1999-07-13 17:08 Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 8+ messages in thread
From: Tom Lane @ 1999-07-13 17:08 UTC (permalink / raw)
To: Matthew Hagerty <matthew@venux.net>; +Cc: pgsql-interfaces
Matthew Hagerty <matthew@venux.net> writes:
> Could you give me a few hints on where to start? I have never cracked into
> the pg code. Also, would there be problems with the parser/optimizer
> handling unlimited size queries? I guess my input size relates more to
> having space for large records, etc. but with a bigger input available
> people could write bigger "killer" queries that may choke the parser. Is
> this a problem/concern?
Right, you're beginning to see the size of the project --- it's not just
a matter of changing the size of the query buffer, but of finding all
the downstream places that assume they will never see a query more
complex than X. About the only bright spot is that those limits can
be cured one at a time; you don't need a Big Bang kind of fix. Still,
the job is not done till they are all fixed.
src/interfaces/libpq is the first thing to look at --- it has a buffer
into which the outgoing query must be stuffed. Right now it allocates
an 8K buffer and refuses to deal with anything that doesn't fit. Better
would be to start with maybe a 1K allocation (which would be plenty
for many many apps) and realloc the buffer whenever a larger query is
presented. The input-side buffer is already realloced bigger at need,
so that part is not such a problem. I think the only other trouble spot
in libpq (at least in its core code) is that the error-message buffer is
also fixed size. Long queries could produce long errors, so that buffer
must also become resizable. Eventually; it needn't be your first fix.
Next do the same sort of thing with src/backend/tcop/postgres.c's query
input buffer.
After that you can start stress-testing the backend's parser to see what
problems exist there. I recall someone mentioned a fixed-size buffer in
the lexer, which is produced by flex, so you'll have to read up on flex
to find out how to control that. I have no idea what other limits might
exist in the parser.
The backend error-message handler has fixed size buffers, IIRC, and
after that you can start worrying about the effects of arbitrarily
complex queries on EXPLAIN, pg_dump, the other client apps, the
other client libraries besides libpq, etc etc yadda yadda.
You'll be past a milestone when you can remove the header-file
#defines for MAX_QUERY_BUFFER and friends, but that's a long way
down the pike.
Good luck. This tape will self-destruct in five seconds...
regards, tom lane
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: [INTERFACES] 8K query limit in 6.5?
@ 1999-07-14 12:19 Jeff MacDonald <jeff@hub.org>
parent: Matthew Hagerty <matthew@venux.net>
1 sibling, 0 replies; 8+ messages in thread
From: Jeff MacDonald @ 1999-07-14 12:19 UTC (permalink / raw)
To: Matthew Hagerty <matthew@venux.net>; +Cc: pgsql-interfaces
you can fix this by changing
#define BLCKSZ 8192
in include/config.h
On Mon, 12 Jul 1999, Matthew Hagerty wrote:
> Greetings,
>
> Was the 8K query limit increased in 6.5?
>
> Thanks,
> Matthew
>
======================================================
Jeff MacDonald
jeff@hub.org webpage: http://hub.org/~jeff
jeff@pgsql.com irc: bignose on EFnet
======================================================
^ permalink raw reply [nested|flat] 8+ messages in thread
end of thread, other threads:[~1999-07-14 12:19 UTC | newest]
Thread overview: 8+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
1999-07-13 03:38 8K query limit in 6.5? Matthew Hagerty <matthew@venux.net>
1999-07-13 13:59 ` Tom Lane <tgl@sss.pgh.pa.us>
1999-07-14 12:19 ` Jeff MacDonald <jeff@hub.org>
1999-07-13 15:11 RE: [INTERFACES] 8K query limit in 6.5? Ansley, Michael <Michael.Ansley@intec.co.za>
1999-07-13 15:46 ` Tom Lane <tgl@sss.pgh.pa.us>
1999-07-13 16:02 Re: [INTERFACES] 8K query limit in 6.5? Matthew Hagerty <matthew@venux.net>
1999-07-13 16:42 ` Tom Lane <tgl@sss.pgh.pa.us>
1999-07-13 17:08 Re: [INTERFACES] 8K query limit in 6.5? Tom Lane <tgl@sss.pgh.pa.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox