pg.ddx.io pgsql-interfaces@postgresql.org mailing list archive
help / color / mirror / Atom feedRe: [INTERFACES] Re: [HACKERS] changes in 6.4
29+ messages / 11 participants
[nested] [flat]
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1998-07-15 19:52 Hannu Krosing <hannu@trust.ee>
0 siblings, 4 replies; 29+ messages in thread
From: Hannu Krosing @ 1998-07-15 19:52 UTC (permalink / raw)
To: Bruce Momjian <maillist@candle.pha.pa.us>; +Cc: pgsql-interfaces; hackers@postgreSQL.org
Bruce Momjian wrote:
>
>
> I was afraid I was going to insult someone by saying what I did.
>
> I MEANT that there are no features being added that a non-postgresql
> user would be interested in. subselects was one feature that
> non-postgresql users would understand. Most of our stuff now is
> cleanup/extension of 6.3 features, many of which would be uninteresting
> to potential users.
Not requiring the column to sort on in target list ia also quite
important.
As are the (still elementary) constraints, still elementary becuse
there is no way to use functions or "is null" in check constraint,
and constraints can be used only when defining tables, not in
"alter table" construct.
> The days where every release fixed server crashes, or added a feature
> that users were 'screaming for' may be a thing of the past.
Is anyone working on fixing the exploding optimisations for many OR-s,
at least the canonic case used by access?
My impression is that this has fallen somewhere between
insightdist and Vadim.
> We are nearing a maturity stage, where we can focus on performance,
> documenation, features, and cleanup. The days when we have a 'major'
> feature may be fewer, because we have added 'most' of the major features
> people have been asking for.
Expect them asking more soon ;)
I'm sure that soon being just basic ANSI SQL compliant is not enough;
people will want (in no particular order ;):
* ANSI CLI,
* updatable cursors,
* foreign key constraints,
* distributed databases,
* row level locking,
* better inheritance,
* domains,
* isolation levels,
* PL/SQL,
* better optimisation for special cases,
* temporary tables (both global and session level),
* more SQL3 constructs,
* unlisten command, maybe an argument to listen command,
* better support for installing your own access methods,
* separating the methods typinput/typoutput (native binary)
and typreceive/typsend (wire binary), they are in pg_type
* implementing a new fe/be protocol that is easier to implement
(does not mix zero terminated, and count-prefixed chunks),
preferrably modelled after X-Window protocol.
* getting rid of the 8k limitations, both in fe/be protocol and
in disk storage.
I know that some of these things are being worked on, but I've lost
track which are expected for 6.4, which for 6.5 and which I should
not expect before 8.0 ;)
> Our user base is growing, and the number of sophisticated developers is
> growing too, so we are getting major patches to improve lots of existing
> functionality.
Yep. Great future is awaiting PostgreSQL.
I'm really looking forward to a time when I can find some time to
contribute some actual code.
Hannu
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1998-07-15 20:23 Bruce Momjian <maillist@candle.pha.pa.us>
parent: Hannu Krosing <hannu@trust.ee>
3 siblings, 0 replies; 29+ messages in thread
From: Bruce Momjian @ 1998-07-15 20:23 UTC (permalink / raw)
To: hannu@trust.ee; +Cc: pgsql-interfaces; hackers@postgreSQL.org
> Bruce Momjian wrote:
> >
> >
> > I was afraid I was going to insult someone by saying what I did.
> >
> > I MEANT that there are no features being added that a non-postgresql
> > user would be interested in. subselects was one feature that
> > non-postgresql users would understand. Most of our stuff now is
> > cleanup/extension of 6.3 features, many of which would be uninteresting
> > to potential users.
>
> Not requiring the column to sort on in target list ia also quite
> important.
>
> As are the (still elementary) constraints, still elementary becuse
> there is no way to use functions or "is null" in check constraint,
> and constraints can be used only when defining tables, not in
> "alter table" construct.
>
> > The days where every release fixed server crashes, or added a feature
> > that users were 'screaming for' may be a thing of the past.
>
> Is anyone working on fixing the exploding optimisations for many OR-s,
> at least the canonic case used by access?
>
> My impression is that this has fallen somewhere between
> insightdist and Vadim.
>
> > We are nearing a maturity stage, where we can focus on performance,
> > documenation, features, and cleanup. The days when we have a 'major'
> > feature may be fewer, because we have added 'most' of the major features
> > people have been asking for.
>
> Expect them asking more soon ;)
>
> I'm sure that soon being just basic ANSI SQL compliant is not enough;
> people will want (in no particular order ;):
> * ANSI CLI,
> * updatable cursors,
> * foreign key constraints,
> * distributed databases,
> * row level locking,
> * better inheritance,
> * domains,
> * isolation levels,
> * PL/SQL,
> * better optimisation for special cases,
> * temporary tables (both global and session level),
> * more SQL3 constructs,
> * unlisten command, maybe an argument to listen command,
> * better support for installing your own access methods,
> * separating the methods typinput/typoutput (native binary)
> and typreceive/typsend (wire binary), they are in pg_type
> * implementing a new fe/be protocol that is easier to implement
> (does not mix zero terminated, and count-prefixed chunks),
> preferrably modelled after X-Window protocol.
> * getting rid of the 8k limitations, both in fe/be protocol and
> in disk storage.
>
> I know that some of these things are being worked on, but I've lost
> track which are expected for 6.4, which for 6.5 and which I should
> not expect before 8.0 ;)
>
> > Our user base is growing, and the number of sophisticated developers is
> > growing too, so we are getting major patches to improve lots of existing
> > functionality.
>
> Yep. Great future is awaiting PostgreSQL.
>
> I'm really looking forward to a time when I can find some time to
> contribute some actual code.
>
> Hannu
>
Hard to argue with this. There are more MAJOR things that I had
forgotten.
Still, I will say that the things we are working on now are more
'extras', than the stuff we were working on a year ago, which were
'usablility' issues.
--
Bruce Momjian | 830 Blythe Avenue
maillist@candle.pha.pa.us | Drexel Hill, Pennsylvania 19026
+ If your life is a hard drive, | (610) 353-9879(w)
+ Christ can be your backup. | (610) 853-3000(h)
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1998-07-15 20:25 Vadim Mikheev <vadim@krs.ru>
parent: Hannu Krosing <hannu@trust.ee>
3 siblings, 2 replies; 29+ messages in thread
From: Vadim Mikheev @ 1998-07-15 20:25 UTC (permalink / raw)
To: Hannu Krosing <hannu@trust.ee>; +Cc: Bruce Momjian <maillist@candle.pha.pa.us>; pgsql-interfaces; hackers@postgreSQL.org
Hannu Krosing wrote:
>
> Not requiring the column to sort on in target list ia also quite
> important.
I'm not sure but isn't this already in 6.4-current ?
>
> As are the (still elementary) constraints, still elementary becuse
> there is no way to use functions or "is null" in check constraint,
ispas=> create table t (x int, check (x is null or x = 5));
CREATE
ispas=> insert into t values (1);
ERROR: ExecAppend: rejected due to CHECK constraint $1
ispas=> insert into t values (null);
INSERT 168212 1
ispas=> insert into t values (5);
INSERT 168213 1
And I'm sure that functions are supported too. This is 6.3.2
> and constraints can be used only when defining tables, not in
> "alter table" construct.
I hadn't time to do this when implementing and have no plans
to do this. In "near" future :)
>
> > The days where every release fixed server crashes, or added a feature
> > that users were 'screaming for' may be a thing of the past.
>
> Is anyone working on fixing the exploding optimisations for many OR-s,
> at least the canonic case used by access?
>
> My impression is that this has fallen somewhere between
> insightdist and Vadim.
I'm not working...
Vadim
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1998-07-15 20:39 Bruce Momjian <maillist@candle.pha.pa.us>
parent: Vadim Mikheev <vadim@krs.ru>
1 sibling, 1 reply; 29+ messages in thread
From: Bruce Momjian @ 1998-07-15 20:39 UTC (permalink / raw)
To: vadim@krs.ru; +Cc: hannu@trust.ee; pgsql-interfaces; hackers@postgreSQL.org
> > > The days where every release fixed server crashes, or added a feature
> > > that users were 'screaming for' may be a thing of the past.
> >
> > Is anyone working on fixing the exploding optimisations for many OR-s,
> > at least the canonic case used by access?
> >
> > My impression is that this has fallen somewhere between
> > insightdist and Vadim.
>
> I'm not working...
>
Not sure anyone has an idea how to fix this.
--
Bruce Momjian | 830 Blythe Avenue
maillist@candle.pha.pa.us | Drexel Hill, Pennsylvania 19026
+ If your life is a hard drive, | (610) 353-9879(w)
+ Christ can be your backup. | (610) 853-3000(h)
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1998-07-15 20:40 David Hartwig <daveh@insightdist.com>
parent: Hannu Krosing <hannu@trust.ee>
3 siblings, 1 reply; 29+ messages in thread
From: David Hartwig @ 1998-07-15 20:40 UTC (permalink / raw)
To: Hannu Krosing <hannu@trust.ee>; +Cc: Bruce Momjian <maillist@candle.pha.pa.us>; pgsql-interfaces; hackers@postgreSQL.org
Hannu Krosing wrote:
> Bruce Momjian wrote:
> >
> >
> > I was afraid I was going to insult someone by saying what I did.
> >
> > I MEANT that there are no features being added that a non-postgresql
> > user would be interested in. subselects was one feature that
> > non-postgresql users would understand. Most of our stuff now is
> > cleanup/extension of 6.3 features, many of which would be uninteresting
> > to potential users.
>
> Not requiring the column to sort on in target list ia also quite
> important.
>
Along these lines - I heard someone grumbling a while back about not being
able to use a function in the ORDER/GROUP BY clauses. (i.e. SELECT bar FROM
foo ORDER BY LCASE(alpha);) I believe it is on the TODO list. Bruce, I
will claim this item unless someone else already has.
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1998-07-15 21:02 Bruce Momjian <maillist@candle.pha.pa.us>
parent: David Hartwig <daveh@insightdist.com>
0 siblings, 1 reply; 29+ messages in thread
From: Bruce Momjian @ 1998-07-15 21:02 UTC (permalink / raw)
To: daveh@insightdist.com; +Cc: hannu@trust.ee; pgsql-interfaces; hackers@postgreSQL.org
> Along these lines - I heard someone grumbling a while back about not being
> able to use a function in the ORDER/GROUP BY clauses. (i.e. SELECT bar FROM
> foo ORDER BY LCASE(alpha);) I believe it is on the TODO list. Bruce, I
> will claim this item unless someone else already has.
>
>
Done.
--
Bruce Momjian | 830 Blythe Avenue
maillist@candle.pha.pa.us | Drexel Hill, Pennsylvania 19026
+ If your life is a hard drive, | (610) 353-9879(w)
+ Christ can be your backup. | (610) 853-3000(h)
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1998-07-15 22:16 David Hartwig <daveh@insightdist.com>
parent: Hannu Krosing <hannu@trust.ee>
3 siblings, 3 replies; 29+ messages in thread
From: David Hartwig @ 1998-07-15 22:16 UTC (permalink / raw)
To: Hannu Krosing <hannu@trust.ee>; +Cc: Bruce Momjian <maillist@candle.pha.pa.us>; pgsql-interfaces; hackers@postgreSQL.org
Hannu Krosing wrote:
> > The days where every release fixed server crashes, or added a feature
> > that users were 'screaming for' may be a thing of the past.
>
> Is anyone working on fixing the exploding optimisations for many OR-s,
> at least the canonic case used by access?
>
> My impression is that this has fallen somewhere between
> insightdist and Vadim.
This is really big for the ODBCers. (And I suspect for JDBCers too.) Many
desktop libraries and end-user tools depend on this "record set" strategy to
operate effectively.
I have put together a workable hack that runs just before cnfify(). The
option is activated through the SET command. Once activated, it identifies
queries with this particular multi-OR pattern generated by these RECORD SET
strategies. Qualified query trees are rewritten as multiple UNIONs. (One
for each OR grouping).
The results are profound. Queries that used to scan tables because of the
ORs, now make use of any indexes. Thus, the size of the table has virtually
no effect on performance. Furthermore, queries that used to crash the
backend, now run in under a second.
Currently the down sides are:
1. If there is no usable index, performance is significantly worse. The
patch does not check to make sure that there is a usable index. I could use
some pointers on this.
2. Small tables are actually a bit slower than without the patch.
3. Not very elegant. I am looking for a more generalized solution.
I have lots of ideas, but I would need to know the backend much better before
attempting any of them. My favorite idea is before cnfify(), to factor the
OR terms and pull out the constants into a virtual (temporary) table spaces.
Then rewrite the query as a join. The optimizer will (should) treat the new
query accordingly. This assumes that an efficient factoring algorithm exists
and that temporary tables can exist in the heap.
Illustration:
SELECT ... FROM tab WHERE
(var1 = const1 AND var2 = const2) OR
(var1 = const3 AND var2 = const4) OR
(var1 = const5 AND var2 = const6)
SELECT ... FROM tab, tmp WHERE
(var1 = var_x AND var2 = var_y)
tmp
var_x | var_y
--------------
const1|const2
const3|const4
const5|const6
Comments?
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1998-07-15 23:39 Bruce Momjian <maillist@candle.pha.pa.us>
parent: David Hartwig <daveh@insightdist.com>
2 siblings, 2 replies; 29+ messages in thread
From: Bruce Momjian @ 1998-07-15 23:39 UTC (permalink / raw)
To: daveh@insightdist.com; +Cc: hannu@trust.ee; pgsql-interfaces; hackers@postgreSQL.org
> The results are profound. Queries that used to scan tables because of the
> ORs, now make use of any indexes. Thus, the size of the table has virtually
> no effect on performance. Furthermore, queries that used to crash the
> backend, now run in under a second.
>
> Currently the down sides are:
> 1. If there is no usable index, performance is significantly worse. The
> patch does not check to make sure that there is a usable index. I could use
> some pointers on this.
>
> 2. Small tables are actually a bit slower than without the patch.
>
> 3. Not very elegant. I am looking for a more generalized solution.
> I have lots of ideas, but I would need to know the backend much better before
> attempting any of them. My favorite idea is before cnfify(), to factor the
> OR terms and pull out the constants into a virtual (temporary) table spaces.
> Then rewrite the query as a join. The optimizer will (should) treat the new
> query accordingly. This assumes that an efficient factoring algorithm exists
> and that temporary tables can exist in the heap.
OK, I have an idea. Just today, we allow:
select *
from tab1
where val in (
select x from tab2
union
select y from tab3
)
How about if instead of doing:
select * from tab1 where val = 3
union
select * from tab1 where val = 4
...
you change it to:
select * from tab1 where val in (
select 3
union
select 4
)
This may be a big win. You aren't running the same query over and over
again, with the same joins, and just a different constant.
Let me know.
If it fails for some reason, it is possible my subselect union code has
a bug, so let me know.
--
Bruce Momjian | 830 Blythe Avenue
maillist@candle.pha.pa.us | Drexel Hill, Pennsylvania 19026
+ If your life is a hard drive, | (610) 353-9879(w)
+ Christ can be your backup. | (610) 853-3000(h)
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1998-07-16 01:22 Bruce Momjian <maillist@candle.pha.pa.us>
parent: Bruce Momjian <maillist@candle.pha.pa.us>
1 sibling, 0 replies; 29+ messages in thread
From: Bruce Momjian @ 1998-07-16 01:22 UTC (permalink / raw)
To: maillist@candle.pha.pa.us; +Cc: daveh@insightdist.com; hannu@trust.ee; pgsql-interfaces; hackers@postgreSQL.org
> > The results are profound. Queries that used to scan tables because of the
> OK, I have an idea. Just today, we allow:
>
> select *
> from tab1
> where val in (
> select x from tab2
> union
> select y from tab3
> )
>
> How about if instead of doing:
>
> select * from tab1 where val = 3
> union
> select * from tab1 where val = 4
> ...
>
> you change it to:
>
> select * from tab1 where val in (
> select 3
> union
> select 4
> )
OK, I just ran some test, and it does not look good:
---------------------------------------------------------------------------
son_db=> explain select mmatter from matter where mmatter = 'A01-001';
NOTICE: QUERY PLAN:
Index Scan using i_matt2 on matter (cost=2.05 size=1 width=12)
EXPLAIN
son_db=> explain select mmatter from matter where mmatter in (select 'A01-001');
NOTICE: QUERY PLAN:
Seq Scan on matter (cost=512.20 size=1001 width=12)
SubPlan
-> Result (cost=0.00 size=0 width=0)
EXPLAIN
---------------------------------------------------------------------------
Turns out indexes are not used in outer queries of subselects. Not sure
why. Vadim?
--
Bruce Momjian | 830 Blythe Avenue
maillist@candle.pha.pa.us | Drexel Hill, Pennsylvania 19026
+ If your life is a hard drive, | (610) 353-9879(w)
+ Christ can be your backup. | (610) 853-3000(h)
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1998-07-16 07:45 Maarten Boekhold <maartenb@dutepp0.et.tudelft.nl>
parent: Bruce Momjian <maillist@candle.pha.pa.us>
0 siblings, 0 replies; 29+ messages in thread
From: Maarten Boekhold @ 1998-07-16 07:45 UTC (permalink / raw)
To: hackers@postgreSQL.org; +Cc: pgsql-interfaces
On Wed, 15 Jul 1998, Bruce Momjian wrote:
> > > > The days where every release fixed server crashes, or added a feature
> > > > that users were 'screaming for' may be a thing of the past.
> > >
> > > Is anyone working on fixing the exploding optimisations for many OR-s,
> > > at least the canonic case used by access?
> > >
> > > My impression is that this has fallen somewhere between
> > > insightdist and Vadim.
> >
> > I'm not working...
> >
>
> Not sure anyone has an idea how to fix this.
What? How to get Vadim back to work? ;)
Maarten
_____________________________________________________________________________
| TU Delft, The Netherlands, Faculty of Information Technology and Systems |
| Department of Electrical Engineering |
| Computer Architecture and Digital Technique section |
| M.Boekhold@et.tudelft.nl |
-----------------------------------------------------------------------------
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1998-07-16 08:30 Hannu Krosing <hannu@trust.ee>
parent: Vadim Mikheev <vadim@krs.ru>
1 sibling, 1 reply; 29+ messages in thread
From: Hannu Krosing @ 1998-07-16 08:30 UTC (permalink / raw)
To: Vadim Mikheev <vadim@krs.ru>; +Cc: pgsql-interfaces; hackers@postgreSQL.org
Vadim Mikheev wrote:
>
> Hannu Krosing wrote:
> >
> > Not requiring the column to sort on in target list ia also quite
> > important.
>
> I'm not sure but isn't this already in 6.4-current ?
>
> >
> > As are the (still elementary) constraints, still elementary becuse
> > there is no way to use functions or "is null" in check constraint,
>
> ispas=> create table t (x int, check (x is null or x = 5));
> CREATE
> ispas=> insert into t values (1);
> ERROR: ExecAppend: rejected due to CHECK constraint $1
> ispas=> insert into t values (null);
> INSERT 168212 1
> ispas=> insert into t values (5);
> INSERT 168213 1
>
> And I'm sure that functions are supported too. This is 6.3.2
Sorry, i tried the wrong syntax (without IS ) ;(
but functions still dont work:
hannu=> create table test1 (a text, b text,
hannu-> check (trim(a) <> '' or trim(b) <> ''));
ERROR: parser: parse error at or near "trim"
If I use a non-existing function, I get a different answer
hannu=> create table test1 (a text, b text,
hannu-> check (strip(a) <> '' or strip(b) <> ''));
ERROR: function strip(text) does not exist
So it cant't be just "parser" error
> > and constraints can be used only when defining tables, not in
> > "alter table" construct.
>
> I hadn't time to do this when implementing and have no plans
> to do this. In "near" future :)
>
> >
> > > The days where every release fixed server crashes, or added a feature
> > > that users were 'screaming for' may be a thing of the past.
> >
> > Is anyone working on fixing the exploding optimisations for many OR-s,
> > at least the canonic case used by access?
> >
> > My impression is that this has fallen somewhere between
> > insightdist and Vadim.
>
> I'm not working...
Are you after some general solution, or are you first implementing
the 'rewrite to union' way ?
Hannu
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1998-07-16 08:41 Hannu Krosing <hannu@trust.ee>
parent: Bruce Momjian <maillist@candle.pha.pa.us>
1 sibling, 1 reply; 29+ messages in thread
From: Hannu Krosing @ 1998-07-16 08:41 UTC (permalink / raw)
To: Bruce Momjian <maillist@candle.pha.pa.us>; +Cc: David Hartwig <daveh@insightdist.com>; pgsql-interfaces; hackers@postgreSQL.org
Bruce Momjian wrote:
>
> > The results are profound. Queries that used to scan tables because of the
>
> How about if instead of doing:
>
> select * from tab1 where val = 3
> union
> select * from tab1 where val = 4
> ...
>
> you change it to:
>
> select * from tab1 where val in (
> select 3
> union
> select 4
> )
>
the explosion happens for ORs of multiple ANDs that get rewritten to:
select * from tabl wehere val1=1 and val2=1 and val3=1
union
select * from tabl wehere val1=1 and val2=1 and val3=2
union
...
And there is no way of doing (at least presently):
select * from table where (val1,val2,val3) in (select 1,1,1 union select
1,1,2);
Hannu
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1998-07-16 08:51 Vadim Mikheev <vadim@krs.ru>
parent: Hannu Krosing <hannu@trust.ee>
0 siblings, 1 reply; 29+ messages in thread
From: Vadim Mikheev @ 1998-07-16 08:51 UTC (permalink / raw)
To: Hannu Krosing <hannu@trust.ee>; +Cc: pgsql-interfaces; hackers@postgreSQL.org
Hannu Krosing wrote:
>
> but functions still dont work:
>
> hannu=> create table test1 (a text, b text,
> hannu-> check (trim(a) <> '' or trim(b) <> ''));
> ERROR: parser: parse error at or near "trim"
TRIM is keyword, not a function...
We have to copy some lines in gram.y
Real functions are working...
Vadim
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1998-07-16 10:35 Aleksey Dashevsky <postgres@luckynet.co.il>
parent: Bruce Momjian <maillist@candle.pha.pa.us>
0 siblings, 2 replies; 29+ messages in thread
From: Aleksey Dashevsky @ 1998-07-16 10:35 UTC (permalink / raw)
To: Bruce Momjian <maillist@candle.pha.pa.us>; +Cc: pgsql-interfaces; hackers@postgreSQL.org
hi, guys!
It seems to me that two or three weeks ago there were some messages about
porting libpq for Win32 platform. I think it is very imporant feature and
it should be mentioned with no doubts in all reviews about PostgreSQL
'cause it moved PostgreSQL far beyond any other free DB engeens in the
world of Windowz
Al.
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1998-07-16 10:56 Hannu Krosing <hannu@trust.ee>
parent: Vadim Mikheev <vadim@krs.ru>
0 siblings, 2 replies; 29+ messages in thread
From: Hannu Krosing @ 1998-07-16 10:56 UTC (permalink / raw)
To: Vadim Mikheev <vadim@krs.ru>; +Cc: pgsql-interfaces; hackers@postgreSQL.org
Vadim Mikheev wrote:
>
> Hannu Krosing wrote:
> >
> > but functions still dont work:
> >
> > hannu=> create table test1 (a text, b text,
> > hannu-> check (trim(a) <> '' or trim(b) <> ''));
> > ERROR: parser: parse error at or near "trim"
>
> TRIM is keyword, not a function...
> We have to copy some lines in gram.y
Wow! is this standard ?
I found the function trim by doing 'select oprname from pg_oper'
and tested it as follows:
hannu=> select trim(' x ');
btrim
-----
x
(1 row)
why is the column called btrim ?
some rewrite magic in parser ?
If it must stay a keyword, then perhaps we should remove the proc ?
> Real functions are working...
yep! Thanks:
create table test2(a text,b text, check (btrim(a) <> '' or btrim(b) <>
''));
does work ;)
Hannu
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1998-07-16 13:36 Thomas G. Lockhart <lockhart@alumni.caltech.edu>
parent: Hannu Krosing <hannu@trust.ee>
1 sibling, 0 replies; 29+ messages in thread
From: Thomas G. Lockhart @ 1998-07-16 13:36 UTC (permalink / raw)
To: Hannu Krosing <hannu@trust.ee>; +Cc: Vadim Mikheev <vadim@krs.ru>; hackers@postgreSQL.org
> > TRIM is keyword, not a function...
> > We have to copy some lines in gram.y
> Wow! is this standard ?
> I found the function trim by doing 'select oprname from pg_oper'
> and tested it as follows:
>
> hannu=> select trim(' x ');
> btrim
> -----
> x
> (1 row)
> why is the column called btrim ?
> some rewrite magic in parser ?
> If it must stay a keyword, then perhaps we should remove the proc ?
Uh, yes, I think you are right. Here's why:
The SQL92 syntax for the trim() function is as follows:
TRIM([LEADING|TRAILING|BOTH] [char FROM] string)
This syntax is _not_ the clean "function(arg1,arg2,...)" syntax that the
parser could handle without change, so I had to make TRIM a keyword in
the parser and explicitly decode the possible argument phrases.
To implement all possibilities, I transform the function in the parser
to the functions btrim(), rtrim(), and ltrim() implemented earlier by
Edmund Mergl as the "Oracle compatibility functions".
I'll add TRIM() and the other goofy pseudo-functions to the CHECK
syntax, and take the trim(arg1) declaration out of pg_proc since it can
never get executed.
Oh, btw we allow trimming strings from strings, not just trimming chars
from strings :)
- Tom
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1998-07-16 13:49 Thomas G. Lockhart <lockhart@alumni.caltech.edu>
parent: Hannu Krosing <hannu@trust.ee>
0 siblings, 1 reply; 29+ messages in thread
From: Thomas G. Lockhart @ 1998-07-16 13:49 UTC (permalink / raw)
To: Hannu Krosing <hannu@trust.ee>; +Cc: Bruce Momjian <maillist@candle.pha.pa.us>; David Hartwig <daveh@insightdist.com>; hackers@postgreSQL.org
> And there is no way of doing (at least presently):
>
> select * from table where (val1,val2,val3)
> in (select 1,1,1 union select 1,1,2);
I'll look at that...
- Tom
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1998-07-16 15:33 Bruce Momjian <maillist@candle.pha.pa.us>
parent: Aleksey Dashevsky <postgres@luckynet.co.il>
1 sibling, 0 replies; 29+ messages in thread
From: Bruce Momjian @ 1998-07-16 15:33 UTC (permalink / raw)
To: postgres@luckynet.co.il; +Cc: pgsql-interfaces; hackers@postgresql.org
> hi, guys!
>
>
> It seems to me that two or three weeks ago there were some messages about
> porting libpq for Win32 platform. I think it is very imporant feature and
> it should be mentioned with no doubts in all reviews about PostgreSQL
> 'cause it moved PostgreSQL far beyond any other free DB engeens in the
Already done in the current snapshot on ftp.postgresql.org.
--
Bruce Momjian | 830 Blythe Avenue
maillist@candle.pha.pa.us | Drexel Hill, Pennsylvania 19026
+ If your life is a hard drive, | (610) 353-9879(w)
+ Christ can be your backup. | (610) 853-3000(h)
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1998-07-16 17:27 Peter T Mount <peter@retep.org.uk>
parent: Aleksey Dashevsky <postgres@luckynet.co.il>
1 sibling, 0 replies; 29+ messages in thread
From: Peter T Mount @ 1998-07-16 17:27 UTC (permalink / raw)
To: Aleksey Dashevsky <postgres@luckynet.co.il>; +Cc: Bruce Momjian <maillist@candle.pha.pa.us>; pgsql-interfaces; hackers@postgresql.org
On Thu, 16 Jul 1998, Aleksey Dashevsky wrote:
> It seems to me that two or three weeks ago there were some messages about
> porting libpq for Win32 platform. I think it is very imporant feature and
> it should be mentioned with no doubts in all reviews about PostgreSQL
> 'cause it moved PostgreSQL far beyond any other free DB engeens in the
> world of Windowz
I'd thought that the ODBC driver would have more of an impact with Win32
than porting libpq, especially with existing applications.
--
Peter T Mount peter@retep.org.uk or petermount@earthling.net
Main Homepage: http://www.retep.org.uk
************ Someday I may rebuild this signature completely ;-) ************
Work Homepage: http://www.maidstone.gov.uk Work EMail: peter@maidstone.gov.uk
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1998-07-16 17:30 Peter T Mount <peter@retep.org.uk>
parent: David Hartwig <daveh@insightdist.com>
2 siblings, 0 replies; 29+ messages in thread
From: Peter T Mount @ 1998-07-16 17:30 UTC (permalink / raw)
To: David Hartwig <daveh@insightdist.com>; +Cc: Hannu Krosing <hannu@trust.ee>; Bruce Momjian <maillist@candle.pha.pa.us>; pgsql-interfaces; hackers@postgresql.org
On Wed, 15 Jul 1998, David Hartwig wrote:
> This is really big for the ODBCers. (And I suspect for JDBCers too.) Many
> desktop libraries and end-user tools depend on this "record set" strategy to
> operate effectively.
Although I haven't seen what they produce, it is possible that JBuilder
and others do have this affect with JDBC.
However, not all JDBC applications have this problem. Infact the majority
I've seen only produce much simpler queries.
--
Peter T Mount peter@retep.org.uk or petermount@earthling.net
Main Homepage: http://www.retep.org.uk
************ Someday I may rebuild this signature completely ;-) ************
Work Homepage: http://www.maidstone.gov.uk Work EMail: peter@maidstone.gov.uk
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1998-07-16 18:29 David Gould <dg@illustra.com>
parent: Hannu Krosing <hannu@trust.ee>
1 sibling, 1 reply; 29+ messages in thread
From: dg@illustra.com @ 1998-07-16 18:29 UTC (permalink / raw)
To: hannu@trust.ee; +Cc: vadim@krs.ru; pgsql-interfaces; hackers@postgreSQL.org
> > > hannu-> check (trim(a) <> '' or trim(b) <> ''));
> > > ERROR: parser: parse error at or near "trim"
> >
> > TRIM is keyword, not a function...
> > We have to copy some lines in gram.y
I think that having trim as a keyword is a problem. The primary virtue of
postgres is that everything is either a function or a type and as such is
definable by the user and resolved at runtime. Making a keyword out of a
function spoils that capability.
-dg
David Gould dg@illustra.com 510.628.3783 or 510.305.9468
Informix Software (No, really) 300 Lakeside Drive Oakland, CA 94612
- If simplicity worked, the world would be overrun with insects. -
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1998-07-16 18:37 Bruce Momjian <maillist@candle.pha.pa.us>
parent: David Gould <dg@illustra.com>
0 siblings, 0 replies; 29+ messages in thread
From: Bruce Momjian @ 1998-07-16 18:37 UTC (permalink / raw)
To: dg@illustra.com; +Cc: hannu@trust.ee; vadim@krs.ru; pgsql-interfaces; hackers@postgreSQL.org
> > > > hannu-> check (trim(a) <> '' or trim(b) <> ''));
> > > > ERROR: parser: parse error at or near "trim"
> > >
> > > TRIM is keyword, not a function...
> > > We have to copy some lines in gram.y
>
> I think that having trim as a keyword is a problem. The primary virtue of
> postgres is that everything is either a function or a type and as such is
> definable by the user and resolved at runtime. Making a keyword out of a
> function spoils that capability.
Problem was that SQL standard syntax (or Oracle) did not allow it to be
a function.
--
Bruce Momjian | 830 Blythe Avenue
maillist@candle.pha.pa.us | Drexel Hill, Pennsylvania 19026
+ If your life is a hard drive, | (610) 353-9879(w)
+ Christ can be your backup. | (610) 853-3000(h)
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1998-07-17 07:18 Hannu Krosing <hannu@trust.ee>
parent: Thomas G. Lockhart <lockhart@alumni.caltech.edu>
0 siblings, 0 replies; 29+ messages in thread
From: Hannu Krosing @ 1998-07-17 07:18 UTC (permalink / raw)
To: Thomas G. Lockhart <lockhart@alumni.caltech.edu>; +Cc: Bruce Momjian <maillist@candle.pha.pa.us>; David Hartwig <daveh@insightdist.com>; hackers@postgreSQL.org
Thomas G. Lockhart wrote:
>
> > And there is no way of doing (at least presently):
> >
> > select * from table where (val1,val2,val3)
> > in (select 1,1,1 union select 1,1,2);
>
> I'll look at that...
Could it be a good idea to have the syntax (at least for constants),
changed to (or at least allowed ;) to the following:
select * from table
where (val1,val2,val3)
in ( (1,1,3), (1,1,2), (1,1,1) );
Which brings us to another issue:
Should (val1,val2,val3) be just some construct that gets rewritten to
"something else" in parser, or should it create an instance of
anonymus row type ?
Allowing anonymus row type creation on the fly would allow us many nice
things, for example a way to create new types of aggregate functions,
like
FOR_MAX((price,date)), so that we could do the following in only one
pass
SELECT
FOR_MAX((price,sales_datetime)) as last_price,
MAX(sales_datetime) as last_sale,
WEEK(sales_datetime) week_nr
GROUP BY
week_nr
;
This would get the prices and dates of each weeks last sale, and is
much hairier to do using just standard sql.
Hannu
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1998-08-22 03:53 Bruce Momjian <maillist@candle.pha.pa.us>
parent: David Hartwig <daveh@insightdist.com>
2 siblings, 1 reply; 29+ messages in thread
From: Bruce Momjian @ 1998-08-22 03:53 UTC (permalink / raw)
To: daveh@insightdist.com; +Cc: hannu@trust.ee; pgsql-interfaces; hackers@postgreSQL.org
>
> Hannu Krosing wrote:
>
> > > The days where every release fixed server crashes, or added a feature
> > > that users were 'screaming for' may be a thing of the past.
> >
> > Is anyone working on fixing the exploding optimisations for many OR-s,
> > at least the canonic case used by access?
> >
> > My impression is that this has fallen somewhere between
> > insightdist and Vadim.
>
> This is really big for the ODBCers. (And I suspect for JDBCers too.) Many
> desktop libraries and end-user tools depend on this "record set" strategy to
> operate effectively.
>
> I have put together a workable hack that runs just before cnfify(). The
> option is activated through the SET command. Once activated, it identifies
> queries with this particular multi-OR pattern generated by these RECORD SET
> strategies. Qualified query trees are rewritten as multiple UNIONs. (One
> for each OR grouping).
>
> The results are profound. Queries that used to scan tables because of the
> ORs, now make use of any indexes. Thus, the size of the table has virtually
> no effect on performance. Furthermore, queries that used to crash the
> backend, now run in under a second.
>
> Currently the down sides are:
> 1. If there is no usable index, performance is significantly worse. The
> patch does not check to make sure that there is a usable index. I could use
> some pointers on this.
>
> 2. Small tables are actually a bit slower than without the patch.
>
> 3. Not very elegant. I am looking for a more generalized solution.
> I have lots of ideas, but I would need to know the backend much better before
> attempting any of them. My favorite idea is before cnfify(), to factor the
> OR terms and pull out the constants into a virtual (temporary) table spaces.
> Then rewrite the query as a join. The optimizer will (should) treat the new
> query accordingly. This assumes that an efficient factoring algorithm exists
> and that temporary tables can exist in the heap.
>
> Illustration:
> SELECT ... FROM tab WHERE
> (var1 = const1 AND var2 = const2) OR
> (var1 = const3 AND var2 = const4) OR
> (var1 = const5 AND var2 = const6)
>
> SELECT ... FROM tab, tmp WHERE
> (var1 = var_x AND var2 = var_y)
>
> tmp
> var_x | var_y
> --------------
> const1|const2
> const3|const4
> const5|const6
David, where are we on this? I know we have OR's using indexes. Do we
still need to look this as a fix, or are we OK. I have not gotten far
enough in the optimizer to know how to fix the cnf'ify problem.
--
Bruce Momjian | 830 Blythe Avenue
maillist@candle.pha.pa.us | Drexel Hill, Pennsylvania 19026
+ If your life is a hard drive, | (610) 353-9879(w)
+ Christ can be your backup. | (610) 853-3000(h)
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1998-08-23 23:55 David Hartwig <daybee@bellatlantic.net>
parent: Bruce Momjian <maillist@candle.pha.pa.us>
0 siblings, 2 replies; 29+ messages in thread
From: David Hartwig @ 1998-08-23 23:55 UTC (permalink / raw)
To: Bruce Momjian <maillist@candle.pha.pa.us>; +Cc: hannu@trust.ee; pgsql-interfaces; hackers@postgreSQL.org
Bruce Momjian wrote:
> >
> > Hannu Krosing wrote:
> >
> > > > The days where every release fixed server crashes, or added a feature
> > > > that users were 'screaming for' may be a thing of the past.
> > >
> > > Is anyone working on fixing the exploding optimisations for many OR-s,
> > > at least the canonic case used by access?
> > >
> > > My impression is that this has fallen somewhere between
> > > insightdist and Vadim.
> >
> > This is really big for the ODBCers. (And I suspect for JDBCers too.) Many
> > desktop libraries and end-user tools depend on this "record set" strategy to
> > operate effectively.
> >
> > I have put together a workable hack that runs just before cnfify(). The
> > option is activated through the SET command. Once activated, it identifies
> > queries with this particular multi-OR pattern generated by these RECORD SET
> > strategies. Qualified query trees are rewritten as multiple UNIONs. (One
> > for each OR grouping).
> >
> > The results are profound. Queries that used to scan tables because of the
> > ORs, now make use of any indexes. Thus, the size of the table has virtually
> > no effect on performance. Furthermore, queries that used to crash the
> > backend, now run in under a second.
> >
> > Currently the down sides are:
> > 1. If there is no usable index, performance is significantly worse. The
> > patch does not check to make sure that there is a usable index. I could use
> > some pointers on this.
> >
> > 2. Small tables are actually a bit slower than without the patch.
> >
> > 3. Not very elegant. I am looking for a more generalized solution.
> > I have lots of ideas, but I would need to know the backend much better before
> > attempting any of them. My favorite idea is before cnfify(), to factor the
> > OR terms and pull out the constants into a virtual (temporary) table spaces.
> > Then rewrite the query as a join. The optimizer will (should) treat the new
> > query accordingly. This assumes that an efficient factoring algorithm exists
> > and that temporary tables can exist in the heap.
> >
> > Illustration:
> > SELECT ... FROM tab WHERE
> > (var1 = const1 AND var2 = const2) OR
> > (var1 = const3 AND var2 = const4) OR
> > (var1 = const5 AND var2 = const6)
> >
> > SELECT ... FROM tab, tmp WHERE
> > (var1 = var_x AND var2 = var_y)
> >
> > tmp
> > var_x | var_y
> > --------------
> > const1|const2
> > const3|const4
> > const5|const6
>
> David, where are we on this? I know we have OR's using indexes. Do we
> still need to look this as a fix, or are we OK. I have not gotten far
> enough in the optimizer to know how to fix the
Bruce,
If the question is, have I come up with a solution for the cnf'ify problem: No
If the question is, is it still important: Very much yes.
It is essential for many RAD tools using remote data objects which make use of key
sets. Your recent optimization of the OR list goes a long way, but inevitably
users are confronted with multi-part keys.
When I look at the problem my head spins. I do not have the experience (yet?)
with the backend to be mucking around in the optimizer. As I see it, cnf'ify is
doing just what it is supposed to do. Boundless boolean logic.
I think hope may lay though, in identifying each AND'ed group associated with a key
and tagging it as a special sub-root node which cnf'ify does not penetrate. This
node would be allowed to pass to the later stages of the optimizer where it will be
used to plan index scans. Easy for me to say.
In the meantime, I still have the patch that I described in prior email. It has
worked well for us. Let me restate that. We could not survive without it!
However, I do not feel that is a sufficiently functional approach that should be
incorporated as a final solution. I will submit the patch if you, (anyone) does
not come up with a better solution. It is coded to be activated by a SET KSQO to
minimize its reach.
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1998-08-29 03:44 Bruce Momjian <maillist@candle.pha.pa.us>
parent: David Hartwig <daybee@bellatlantic.net>
1 sibling, 2 replies; 29+ messages in thread
From: Bruce Momjian @ 1998-08-29 03:44 UTC (permalink / raw)
To: daybee@bellatlantic.net; +Cc: hannu@trust.ee; pgsql-interfaces; hackers@postgreSQL.org
>
>
> Bruce Momjian wrote:
>
> > >
> > > Hannu Krosing wrote:
> > >
> > > > > The days where every release fixed server crashes, or added a feature
> > > > > that users were 'screaming for' may be a thing of the past.
> > > >
> > > > Is anyone working on fixing the exploding optimisations for many OR-s,
> > > > at least the canonic case used by access?
> > > >
> > > > My impression is that this has fallen somewhere between
> > > > insightdist and Vadim.
> > >
> > > This is really big for the ODBCers. (And I suspect for JDBCers too.) Many
> > > desktop libraries and end-user tools depend on this "record set" strategy to
> > > operate effectively.
> > >
> > > I have put together a workable hack that runs just before cnfify(). The
> > > option is activated through the SET command. Once activated, it identifies
> > > queries with this particular multi-OR pattern generated by these RECORD SET
> > > strategies. Qualified query trees are rewritten as multiple UNIONs. (One
> > > for each OR grouping).
> > >
> > > The results are profound. Queries that used to scan tables because of the
> > > ORs, now make use of any indexes. Thus, the size of the table has virtually
> > > no effect on performance. Furthermore, queries that used to crash the
> > > backend, now run in under a second.
> > >
> > > Currently the down sides are:
> > > 1. If there is no usable index, performance is significantly worse. The
> > > patch does not check to make sure that there is a usable index. I could use
> > > some pointers on this.
> > >
> > > 2. Small tables are actually a bit slower than without the patch.
> > >
> > > 3. Not very elegant. I am looking for a more generalized solution.
> > > I have lots of ideas, but I would need to know the backend much better before
> > > attempting any of them. My favorite idea is before cnfify(), to factor the
> > > OR terms and pull out the constants into a virtual (temporary) table spaces.
> > > Then rewrite the query as a join. The optimizer will (should) treat the new
> > > query accordingly. This assumes that an efficient factoring algorithm exists
> > > and that temporary tables can exist in the heap.
> > >
> > > Illustration:
> > > SELECT ... FROM tab WHERE
> > > (var1 = const1 AND var2 = const2) OR
> > > (var1 = const3 AND var2 = const4) OR
> > > (var1 = const5 AND var2 = const6)
> > >
> > > SELECT ... FROM tab, tmp WHERE
> > > (var1 = var_x AND var2 = var_y)
> > >
> > > tmp
> > > var_x | var_y
> > > --------------
> > > const1|const2
> > > const3|const4
> > > const5|const6
> >
> > David, where are we on this? I know we have OR's using indexes. Do we
> > still need to look this as a fix, or are we OK. I have not gotten far
> > enough in the optimizer to know how to fix the
>
> Bruce,
>
> If the question is, have I come up with a solution for the cnf'ify problem: No
>
> If the question is, is it still important: Very much yes.
>
> It is essential for many RAD tools using remote data objects which make use of key
> sets. Your recent optimization of the OR list goes a long way, but inevitably
> users are confronted with multi-part keys.
>
> When I look at the problem my head spins. I do not have the experience (yet?)
> with the backend to be mucking around in the optimizer. As I see it, cnf'ify is
> doing just what it is supposed to do. Boundless boolean logic.
>
> I think hope may lay though, in identifying each AND'ed group associated with a key
> and tagging it as a special sub-root node which cnf'ify does not penetrate. This
> node would be allowed to pass to the later stages of the optimizer where it will be
> used to plan index scans. Easy for me to say.
>
> In the meantime, I still have the patch that I described in prior email. It has
> worked well for us. Let me restate that. We could not survive without it!
> However, I do not feel that is a sufficiently functional approach that should be
> incorporated as a final solution. I will submit the patch if you, (anyone) does
> not come up with a better solution. It is coded to be activated by a SET KSQO to
> minimize its reach.
>
>
OK, let me try this one.
Why is the system cnf'ifying the query. Because it wants to have a
list of qualifications that are AND'ed, so it can just pick the most
restrictive/cheapest, and evaluate that one first.
If you have:
(a=b and c=d) or e=1
In this case, without cnf'ify, it has to evaluate both of them, because
if one is false, you can't be sure another would be true. In the
cnf'ify case,
(a=b or e=1) and (c=d or e=1)
In this case, it can choose either, and act on just one, if a row fails
to meet it, it can stop and not evaluate it using the other restriction.
The fact is that it is only going to use fancy join/index in one of the
two cases, so it tries to pick the best one, and does a brute-force
qualification test on the remaining item if the first one tried is true.
The problem is of course large where clauses can exponentially expand
this. What it really trying to do is to pick a cheapest restriction,
but the memory explosion and query failure are serious problems.
The issue is that it thinks it is doing something to help things, while
it is actually hurting things.
In the ODBC case of:
(x=3 and y=4) or
(x=3 and y=5) or
(x=3 and y=6) or ...
it clearly is not going to gain anything by choosing any CHEAPEST path,
because they are all the same in terms of cost, and the use by ODBC
clients is hurting reliability.
I am inclined to agree with David's solution of breaking apart the query
into separate UNION queries in certain cases. It seems to be the most
logical solution, because the cnf'ify code is working counter to its
purpose in these cases.
Now, the question is how/where to implement this. I see your idea of
making the OR a join to a temp table that holds all the constants.
Another idea would be to do actual UNION queries:
SELECT * FROM tab
WHERE (x=3 and y=4)
UNION
SELECT * FROM tab
WHERE (x=3 and y=5)
UNION
SELECT * FROM tab
WHERE (x=3 and y=6) ...
This would work well for tables with indexes, but for a sequential scan,
you are doing a sequential scan for each UNION. Another idea is
subselects. Also, you have to make sure you return the proper rows,
keeping duplicates where they are in the base table, but not returning
them when the meet more than one qualification.
SELECT * FROM tab
WHERE (x,y) IN (SELECT 3, 4
UNION
SELECT 3, 5
UNION
SELECT 3, 6)
I believe we actually support this. This is not going to use an index
on tab, so it may be slow if x and y are indexed.
Another more bizarre solution is:
SELECT * FROM tab
WHERE (x,y) = (SELECT 3, 4) OR
(x,y) = (SELECT 3, 5) OR
(x,y) = (SELECT 3, 6)
Again, I think we do this too. I don't think cnf'ify does anything with
this. I also believe "=" uses indexes on subselects, while IN does not
because IN could return lots of rows, and an index is slower than a
non-index join on lots of rows. Of course, now that we index OR's.
Let me ask another question. If I do:
SELECT * FROM tab WHERE x=3 OR x=4
it works, and uses indexes. Why can't the optimizer just not cnf'ify
things sometimes, and just do:
SELECT * FROM tab
WHERE (x=3 AND y=4) OR
(x=3 AND y=5) OR
(x=3 AND y=6)
Why can it handle x=3 OR x=4, but not the more complicated case above,
without trying to be too smart? If x,y is a multi-key index, it could
use that quite easily. If not, it can do a sequentail scan and run the
tests.
Another issue. To the optimizer, x=3 and x=y are totally different. In
x=3, it is a column compared to a constant, while in x=y, it is a join.
That makes a huge difference.
In the case of (a=b and c=d) or e=1, you pick the best path and do the
a=b join, and throw in the e=1 entries. You can't easily do both joins,
because you also need the e=1 stuff.
I wounder what would happen if we prevent cnf'ifying of cases where the
OR represent only column = constant restrictions.
I meant to really go through the optimizer this month, but other backend
items took my time.
Can someone run some tests on disabling the cnf'ify calls. It is my
understanding that with the non-cnf-ify'ed query, it can't choose an
optimial path, and starts to do either straight index matches,
sequential scans, or cartesian products where it joins every row to
every other row looking for a match.
Let's say we turn off cnf-ify just for non-join queries. Does that
help?
I am not sure of the ramifications of telling the optimizer it no longer
has a variety of paths to choose for evaluating the query.
--
Bruce Momjian | 830 Blythe Avenue
maillist@candle.pha.pa.us | Drexel Hill, Pennsylvania 19026
+ If your life is a hard drive, | (610) 353-9879(w)
+ Christ can be your backup. | (610) 853-3000(h)
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1998-08-30 15:40 David Hartwig <daybee@bellatlantic.net>
parent: Bruce Momjian <maillist@candle.pha.pa.us>
1 sibling, 0 replies; 29+ messages in thread
From: David Hartwig @ 1998-08-30 15:40 UTC (permalink / raw)
To: Bruce Momjian <maillist@candle.pha.pa.us>; +Cc: hannu@trust.ee; pgsql-interfaces; hackers@postgreSQL.org
Bruce Momjian wrote:
> OK, let me try this one.
>
> Why is the system cnf'ifying the query. Because it wants to have a
> list of qualifications that are AND'ed, so it can just pick the most
> restrictive/cheapest, and evaluate that one first.
>
> If you have:
>
> (a=b and c=d) or e=1
>
> In this case, without cnf'ify, it has to evaluate both of them, because
> if one is false, you can't be sure another would be true. In the
> cnf'ify case,
>
> (a=b or e=1) and (c=d or e=1)
>
> In this case, it can choose either, and act on just one, if a row fails
> to meet it, it can stop and not evaluate it using the other restriction.
>
> The fact is that it is only going to use fancy join/index in one of the
> two cases, so it tries to pick the best one, and does a brute-force
> qualification test on the remaining item if the first one tried is true.
>
> The problem is of course large where clauses can exponentially expand
> this. What it really trying to do is to pick a cheapest restriction,
> but the memory explosion and query failure are serious problems.
>
> The issue is that it thinks it is doing something to help things, while
> it is actually hurting things.
>
> In the ODBC case of:
>
> (x=3 and y=4) or
> (x=3 and y=5) or
> (x=3 and y=6) or ...
>
> it clearly is not going to gain anything by choosing any CHEAPEST path,
> because they are all the same in terms of cost, and the use by ODBC
> clients is hurting reliability.
>
> I am inclined to agree with David's solution of breaking apart the query
> into separate UNION queries in certain cases. It seems to be the most
> logical solution, because the cnf'ify code is working counter to its
> purpose in these cases.
>
> Now, the question is how/where to implement this. I see your idea of
> making the OR a join to a temp table that holds all the constants.
> Another idea would be to do actual UNION queries:
>
> SELECT * FROM tab
> WHERE (x=3 and y=4)
> UNION
> SELECT * FROM tab
> WHERE (x=3 and y=5)
> UNION
> SELECT * FROM tab
> WHERE (x=3 and y=6) ...
>
> This would work well for tables with indexes, but for a sequential scan,
> you are doing a sequential scan for each UNION.
Practically speaking, the lack of an index concern, may not be justified. The reason
these queries are being generated, with this shape, is because remote data objects on the
client side are being told that a primary key exists on these tables. The object is told
about these keys in one of two ways.
1. It queries the database for the primary key of the table. The ODBC driver serviced
this request by querying for the attributes used in {table_name}_pkey.
2. The user manually specifies the primary key. In this case an actual index may not
exist. (i.e. MS Access asks the user for this information if a primary key is not found
in a table)
The second case is the only one that would cause a problem. Fortunately, the solution is
simple. Add a primary key index!
My only concern is to be able to accurately identify a query with the proper signature
before rewriting it as a UNION. To what degree should this inspection be taken?
BTW, I would not do the rewrite on OR's without AND's since you have fixed the OR's use
of the index.
There is one other potential issue. My experience with using arrays in tables and UNIONS
creates problems. There are missing array comparison operators which are used by the
implied DISTINCT.
> Another idea is
> subselects. Also, you have to make sure you return the proper rows,
> keeping duplicates where they are in the base table, but not returning
> them when the meet more than one qualification.
>
> SELECT * FROM tab
> WHERE (x,y) IN (SELECT 3, 4
> UNION
> SELECT 3, 5
> UNION
> SELECT 3, 6)
>
> I believe we actually support this. This is not going to use an index
> on tab, so it may be slow if x and y are indexed.
>
> Another more bizarre solution is:
>
> SELECT * FROM tab
> WHERE (x,y) = (SELECT 3, 4) OR
> (x,y) = (SELECT 3, 5) OR
> (x,y) = (SELECT 3, 6)
>
> Again, I think we do this too. I don't think cnf'ify does anything with
> this. I also believe "=" uses indexes on subselects, while IN does not
> because IN could return lots of rows, and an index is slower than a
> non-index join on lots of rows. Of course, now that we index OR's.
>
> Let me ask another question. If I do:
>
> SELECT * FROM tab WHERE x=3 OR x=4
>
> it works, and uses indexes. Why can't the optimizer just not cnf'ify
> things sometimes, and just do:
>
> SELECT * FROM tab
> WHERE (x=3 AND y=4) OR
> (x=3 AND y=5) OR
> (x=3 AND y=6)
>
> Why can it handle x=3 OR x=4, but not the more complicated case above,
> without trying to be too smart? If x,y is a multi-key index, it could
> use that quite easily. If not, it can do a sequentail scan and run the
> tests.
>
> Another issue. To the optimizer, x=3 and x=y are totally different. In
> x=3, it is a column compared to a constant, while in x=y, it is a join.
> That makes a huge difference.
>
> In the case of (a=b and c=d) or e=1, you pick the best path and do the
> a=b join, and throw in the e=1 entries. You can't easily do both joins,
> because you also need the e=1 stuff.
>
> I wounder what would happen if we prevent cnf'ifying of cases where the
> OR represent only column = constant restrictions.
>
> I meant to really go through the optimizer this month, but other backend
> items took my time.
>
> Can someone run some tests on disabling the cnf'ify calls. It is my
> understanding that with the non-cnf-ify'ed query, it can't choose an
> optimial path, and starts to do either straight index matches,
> sequential scans, or cartesian products where it joins every row to
> every other row looking for a match.
>
> Let's say we turn off cnf-ify just for non-join queries. Does that
> help?
>
> I am not sure of the ramifications of telling the optimizer it no longer
> has a variety of paths to choose for evaluating the query.
I did not try this earlier because I thought it was too good to be true. I was right.
I tried commenting out the normalize() function in the cnfify(). The EXPLAIN showed a
sequential scan and the resulting tuple set was empty. Time will not allow me to dig
into this further this weekend.
Unless you come up with a better solution, I am going to submit my patch on Monday to
make the Sept. 1st deadline. It includes a SET switch to activate the rewrite so as not
to cause problems outside the ODBC users. We can either improve, it or yank it, by the
Oct. 1st deadline.
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1998-08-31 06:53 Sbragion Denis <infotecn@tin.it>
parent: Bruce Momjian <maillist@candle.pha.pa.us>
1 sibling, 0 replies; 29+ messages in thread
From: Sbragion Denis @ 1998-08-31 06:53 UTC (permalink / raw)
To: David Hartwig <daybee@bellatlantic.net>; Bruce Momjian <maillist@candle.pha.pa.us>; +Cc: hannu@trust.ee; pgsql-interfaces; hackers@postgreSQL.org
Hello,
At 11.40 30/08/98 -0400, David Hartwig wrote:
>> Why is the system cnf'ifying the query. Because it wants to have a
>> list of qualifications that are AND'ed, so it can just pick the most
>> restrictive/cheapest, and evaluate that one first.
Just a small question about all this optimizations stuff. I'm not a
database expert but I think we are talking about a NP-complete problem.
Could'nt we convert this optimization problem into another NP one that is
known to have a good solution ? For example for the traveling salesman
problem there's an alghoritm that provide a solution that's never more than
two times the optimal one an provides results that are *really* near the
optimal one most of the times. The simplex alghoritm may be another
example. I think that this kind of alghoritm would be better than a
collection ot tricks for special cases, and this tricks could be used
anyway when special cases are detected. Furthermore I also know that exists
a free program I used in the past that provides this kind of optimizations
for chip design. I don't remember the exact name of the program but I
remember it came from Berkeley university. Of course may be I'm totally
missing the point.
Hope it helps !
Bye!
Dr. Sbragion Denis
InfoTecna
Tel, Fax: +39 39 2324054
URL: http://space.tin.it/internet/dsbragio
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: [INTERFACES] Re: [HACKERS] changes in 6.4
@ 1999-09-18 20:10 Bruce Momjian <maillist@candle.pha.pa.us>
parent: David Hartwig <daybee@bellatlantic.net>
1 sibling, 0 replies; 29+ messages in thread
From: Bruce Momjian @ 1999-09-18 20:10 UTC (permalink / raw)
To: David Hartwig <daybee@bellatlantic.net>; +Cc: hannu@trust.ee; pgsql-interfaces; hackers@postgreSQL.org
This is an old message, but still relivant. I belive 6.6 will have much
better OR memory usage.
>
>
> Bruce Momjian wrote:
>
> > >
> > > Hannu Krosing wrote:
> > >
> > > > > The days where every release fixed server crashes, or added a feature
> > > > > that users were 'screaming for' may be a thing of the past.
> > > >
> > > > Is anyone working on fixing the exploding optimisations for many OR-s,
> > > > at least the canonic case used by access?
> > > >
> > > > My impression is that this has fallen somewhere between
> > > > insightdist and Vadim.
> > >
> > > This is really big for the ODBCers. (And I suspect for JDBCers too.) Many
> > > desktop libraries and end-user tools depend on this "record set" strategy to
> > > operate effectively.
> > >
> > > I have put together a workable hack that runs just before cnfify(). The
> > > option is activated through the SET command. Once activated, it identifies
> > > queries with this particular multi-OR pattern generated by these RECORD SET
> > > strategies. Qualified query trees are rewritten as multiple UNIONs. (One
> > > for each OR grouping).
> > >
> > > The results are profound. Queries that used to scan tables because of the
> > > ORs, now make use of any indexes. Thus, the size of the table has virtually
> > > no effect on performance. Furthermore, queries that used to crash the
> > > backend, now run in under a second.
> > >
> > > Currently the down sides are:
> > > 1. If there is no usable index, performance is significantly worse. The
> > > patch does not check to make sure that there is a usable index. I could use
> > > some pointers on this.
> > >
> > > 2. Small tables are actually a bit slower than without the patch.
> > >
> > > 3. Not very elegant. I am looking for a more generalized solution.
> > > I have lots of ideas, but I would need to know the backend much better before
> > > attempting any of them. My favorite idea is before cnfify(), to factor the
> > > OR terms and pull out the constants into a virtual (temporary) table spaces.
> > > Then rewrite the query as a join. The optimizer will (should) treat the new
> > > query accordingly. This assumes that an efficient factoring algorithm exists
> > > and that temporary tables can exist in the heap.
> > >
> > > Illustration:
> > > SELECT ... FROM tab WHERE
> > > (var1 = const1 AND var2 = const2) OR
> > > (var1 = const3 AND var2 = const4) OR
> > > (var1 = const5 AND var2 = const6)
> > >
> > > SELECT ... FROM tab, tmp WHERE
> > > (var1 = var_x AND var2 = var_y)
> > >
> > > tmp
> > > var_x | var_y
> > > --------------
> > > const1|const2
> > > const3|const4
> > > const5|const6
> >
> > David, where are we on this? I know we have OR's using indexes. Do we
> > still need to look this as a fix, or are we OK. I have not gotten far
> > enough in the optimizer to know how to fix the
>
> Bruce,
>
> If the question is, have I come up with a solution for the cnf'ify problem: No
>
> If the question is, is it still important: Very much yes.
>
> It is essential for many RAD tools using remote data objects which make use of key
> sets. Your recent optimization of the OR list goes a long way, but inevitably
> users are confronted with multi-part keys.
>
> When I look at the problem my head spins. I do not have the experience (yet?)
> with the backend to be mucking around in the optimizer. As I see it, cnf'ify is
> doing just what it is supposed to do. Boundless boolean logic.
>
> I think hope may lay though, in identifying each AND'ed group associated with a key
> and tagging it as a special sub-root node which cnf'ify does not penetrate. This
> node would be allowed to pass to the later stages of the optimizer where it will be
> used to plan index scans. Easy for me to say.
>
> In the meantime, I still have the patch that I described in prior email. It has
> worked well for us. Let me restate that. We could not survive without it!
> However, I do not feel that is a sufficiently functional approach that should be
> incorporated as a final solution. I will submit the patch if you, (anyone) does
> not come up with a better solution. It is coded to be activated by a SET KSQO to
> minimize its reach.
>
>
--
Bruce Momjian | http://www.op.net/~candle
maillist@candle.pha.pa.us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026
^ permalink raw reply [nested|flat] 29+ messages in thread
end of thread, other threads:[~1999-09-18 20:10 UTC | newest]
Thread overview: 29+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
1998-07-15 19:52 Re: [INTERFACES] Re: [HACKERS] changes in 6.4 Hannu Krosing <hannu@trust.ee>
1998-07-15 20:23 ` Bruce Momjian <maillist@candle.pha.pa.us>
1998-07-15 20:25 ` Vadim Mikheev <vadim@krs.ru>
1998-07-15 20:39 ` Bruce Momjian <maillist@candle.pha.pa.us>
1998-07-16 07:45 ` Maarten Boekhold <maartenb@dutepp0.et.tudelft.nl>
1998-07-16 08:30 ` Hannu Krosing <hannu@trust.ee>
1998-07-16 08:51 ` Vadim Mikheev <vadim@krs.ru>
1998-07-16 10:56 ` Hannu Krosing <hannu@trust.ee>
1998-07-16 13:36 ` Thomas G. Lockhart <lockhart@alumni.caltech.edu>
1998-07-16 18:29 ` David Gould <dg@illustra.com>
1998-07-16 18:37 ` Bruce Momjian <maillist@candle.pha.pa.us>
1998-07-15 20:40 ` David Hartwig <daveh@insightdist.com>
1998-07-15 21:02 ` Bruce Momjian <maillist@candle.pha.pa.us>
1998-07-16 10:35 ` Aleksey Dashevsky <postgres@luckynet.co.il>
1998-07-16 15:33 ` Bruce Momjian <maillist@candle.pha.pa.us>
1998-07-16 17:27 ` Peter T Mount <peter@retep.org.uk>
1998-07-15 22:16 ` David Hartwig <daveh@insightdist.com>
1998-07-15 23:39 ` Bruce Momjian <maillist@candle.pha.pa.us>
1998-07-16 01:22 ` Bruce Momjian <maillist@candle.pha.pa.us>
1998-07-16 08:41 ` Hannu Krosing <hannu@trust.ee>
1998-07-16 13:49 ` Thomas G. Lockhart <lockhart@alumni.caltech.edu>
1998-07-17 07:18 ` Hannu Krosing <hannu@trust.ee>
1998-07-16 17:30 ` Peter T Mount <peter@retep.org.uk>
1998-08-22 03:53 ` Bruce Momjian <maillist@candle.pha.pa.us>
1998-08-23 23:55 ` David Hartwig <daybee@bellatlantic.net>
1998-08-29 03:44 ` Bruce Momjian <maillist@candle.pha.pa.us>
1998-08-30 15:40 ` David Hartwig <daybee@bellatlantic.net>
1998-08-31 06:53 ` Sbragion Denis <infotecn@tin.it>
1999-09-18 20:10 ` Bruce Momjian <maillist@candle.pha.pa.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox