Subject: big dumps from rewrites

From: [email-masked] (Postgres System User)
Date: Fri, 2 Apr 93 00:26:22 -0800

I certainly don't know if the problem I found is common.  However, I
have discovered at least one way to cause the 70meg dumps discussed
recently.  If rewrite rules fall into a circular logic, postgres will
happily keep applying the same rules to themselves until it dies.  I
had one instance such a follows:

define rewrite rule r1 is
on append to c1
do [ append c2 ... ] 

define rewrite rule r2 is
on replace to c1
do [ replace c2 ... ]

define rewrite rule r3 is
on append to c2 where such and such1
do [ replace c1 ... ]

Where such and such1 is a qualifier that should control the feedback.
However, in rewrite rules the qualifiers are really flaky and don't
seem to be tested before the rule is rewritten into the query.  The
same scheme above does work on instance level rules.  However, you
can't seem to get a much rule down under instance rules as you can
with rewrite rules before pg says sorry to big!  It may however also
be true that the scheme works with instance rules because I increment
the command counter after firing each rule so that the results of all
predecessors are visible to each successive rule as it triggers.  Not
stock I know but more useful in my environment.

The deadly loop above occurs in RewriteHandler.c(QueryRewrite).  It is
actually quite spectacular to watch it get pregnant an explode.
Talk about robust. I had to watch, my poor 486 stopped everything else
while was happening.  A little short on cpu cycles perhaps ;-}

On perhaps a more productive note I found what I believe to be an
error in prs2append.c(prs2Append).  

In prs2append.c(prs2Append) about line 211 is the following:

    if (status == PRS2_STATUS_TUPLE_CHANGED) {
	if (prs2GetNumberOfLocks(newLocks) == 0) {
	    prs2FreeLocks(newLocks);
	    return(PRS2_STATUS_TUPLE_CHANGED);
	} else {
!/* XXX nmm005 	    
! *          HeapTupleSetRuleLock(*returnedTupleP, InvalidBuffer,newLocks); 
! *
! * In attributeValueMakeNewTuple() (actually heap_modifytuple()), the 
! * rulelock is bcopy'd in from 'tuple' which was created outside 
! * of this memory context and gets pfreed upstream!
! * HeapTupleSetRuleLock() pfrees the old lock in *returnedTupleP,
! * that's twice resulting in an error caught in portalmem.c:
! * NOTICE:Mar 30 12:38:45:PortalHeapMemoryFree: 0x8198424 not in alloc set!
! * This of course issues a SIGHUP and aborts whatever was going on.
! * There also seems to be a memory leak somewhere! I thought those
! * were gone?!# in early v3.
! */
+	    (*returnedTupleP)->t_locktype = MEM_RULE_LOCK;
+	    (*returnedTupleP)->t_lock.l_lock = newLocks;
	    return(PRS2_STATUS_TUPLE_CHANGED);
	}

I doubt if this is the best way to fix it but at the moment it was the
most expedient.  It really should be in rac.c with the rest of the
stuff I guess.  

nmm

PS. If someone comes up with a pregnancy fix for the 70meg dumps,
please send a copy.  I must either fix it or rewrite the app. ugh!
If there is a more appropriate place to discuss this kind of techo-babble,
please advise.  I don't wish to annoy the uninterested or look really 
dumb when I'm wrong!  Are the bug fixes at postgres.berkeley.edu ever
made available anymore on an as found basis?
Thanks again for the bandwidth.

Neil M. McQuaig      344 Millicent Way,  Shreveport, LA  71106
VOICE: (318)868-5611 UUCP: mcquaig!nmm (318)861-1051 or uunet!mcquaig!nmm

---

From: [email-masked] (Paul M. Aoki)
Date: Fri, 02 Apr 93 01:16:58 -0800

[email-masked] (Postgres System User) writes:
> happily keep applying the same rules to themselves until it dies.

I have stack traces that go >300 calls deep before the process implodes.

The guy (male ;-) who wrote the code says it's documented somewhere 
that you aren't supposed to use circular rules (like the example
Carole provided to us, or your examples).  Which is probably as 
official an answer as you'll hear until someone decides to start 
throwing programmers at it ...

> It is actually quite spectacular to watch it get pregnant an explode.

This is a really, really disturbing analogy.  You would probably fit 
in well in our research group :-)

> while was happening.  A little short on cpu cycles perhaps ;-}

That's ok.  It brings a DECstation5000 to its knees, too.  Not to 
mention a SparcStation2.  Soon, maybe a DECstation3000. ;-)

The spectre of infinite looping at 200MHz is truly frightening :-)

> dumb when I'm wrong!

I don't think you've been wrong yet.  bug-postgres@postgres just goes
to a few postgres hackers, though.  post_questions@postgres has a slightly
wider distribution, but neither goes outside of UCB.

> Are the bug fixes at postgres.berkeley.edu ever

You'll have to ask the new management.
--
  Paul M. Aoki  |  CS Div., Dept. of EECS, UCB  |  [email-masked]
                |  Berkeley, CA 94720           |  ...!uunet!ucbvax!aoki

---

From: [email-masked] (Jeffrey Goh)
Date: Sat, 03 Apr 1993 12:04:29 +0800

Sigh, the rewrite rule system was _never_ intended to handle circular
rules of the form you have described.  The reason is simply that
rewrite rules can only detect circularity regarding tables and
qualifiers, not indivdual tuples of the table.  That is, we know that if
a retrieve to table emp triggers another retrieve to table emp
then there is a chance that this rule will be self recursive.
However, there is also a chance that the rule will _not_ be self
recursive.  This would be the case if the qualifiers of the
action and the trigger are mutually exclusive.  For example:

	on retrieve to emp where emp.name = "jeff"
	do replace ( current.salary = e1.salary ) where
		e1.name = "mike"

is a trivial example of a rule that would not be self recursive
even though the action actually becomes another retrieve to the
same table.

However, doing logical evaluation of the qualifiers to determine what are
mutually exclusive qualifiers and what are _not_ would mean
having to write a rule solver that muddles around with solving
the qualifiers before even looking at the tuples.  

The alternative would have been to disallow rules that _might_
be self recursive.  I simply decided that this would be too restrictive
and instead put disclaimers that the user better damn well know
not to write self recursive (or mutually recursive) rules. 
The idea then was that rules that were not logically mutually
exclusive should then be defined using the tuple level system.

No flames please,
- jeff