Subject: Re: Automatic compat checking? (was 7.4 compatibility question)

From: Christoph Haller <[email-masked]>
Date: Thu, 23 Oct 2003 11:01:16 METDST

> 
> On Wednesday 22 October 2003 07:37, Neil Conway wrote:
> > The second audience is the people who are really interested in exactly
> > what has changed between the new release of PostgreSQL and the previous
> > release series. It is important that we make it easy for an admin
> > planning a PostgreSQL upgrade at a fairly large site to be able to see
> > what changes in PostgreSQL have been made, and what changes will be
> > necessary in their own applications. 
> 
> Something I was pondering the other day was whether a pg_compat_chk utility 
> would be practical/desirable. You run it against your existing database / 
> schema dump and it prints a set of warnings:
> 
> Old version = 7.2.1
> New version = 7.4.0
> 
> Warning: schema support introduced (v7.3)
>   all objects will be placed in the default schema
> Failure: DEFAULT 'now' not supported (v7.4)
>   table1.column2
>   table2.column3
> Notice: timestamp now holds milliseconds by default (v7.3)
>   tableX.whatever
> 
> My main concern would be that a 90% solution might be worse than nothing at 
> all.
> Incidentally, this is not idle speculation, but something I might well have 
> time to stick in gborg during the 7.5 devt cycle.
> 
> -- 
>   Richard Huxton
>   Archonet Ltd
> 
A pg_compat_chk utility sounds great. 
No idea, if this is practical, but it's desirable - at least to me. 

Regards, Christoph 

PS I'm surprised no one else replied.

---

From: Christopher Kings-Lynne <[email-masked]>
Date: Wed, 22 Oct 2003 10:21:30 +0800

Hi,

Now that DEFAULT 'now' will not work in PostgreSQL 7.4,  will DEFAULT 
'infinity' and DEFAULT '-infinity' and the like stop working as well? 
What is the workaround for those defaults?

Chris

---

From: Tom Lane <[email-masked]>
Date: Wed, 22 Oct 2003 00:03:58 -0400

> 'infinity' and DEFAULT '-infinity' and the like stop working as well? 

No, because those values don't change over time.  The issue here is when
exactly does the default value get evaluated...

			regards, tom lane

---

From: Christopher Kings-Lynne <[email-masked]>
Date: Wed, 22 Oct 2003 12:18:59 +0800

> exactly does the default value get evaluated...

Ah, of course - but what about 'yesterday' and 'tomorrow' - these should 
also be mentioned in the release notes.

Chris

---

From: Tom Lane <[email-masked]>
Date: Wed, 22 Oct 2003 00:21:34 -0400

> also be mentioned in the release notes.

Good point ... not that I think anyone is actually using such a default
in the field, but the behavior did change ...

			regards, tom lane

---

From: Bruce Momjian <[email-masked]>
Date: Wed, 22 Oct 2003 00:31:33 -0400 (EDT)

> in the field, but the behavior did change ...

It would be pretty strange to use those as a default --- I am not
inclined to mention it in the release notes --- we don't mention every
change, only significant ones.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  [email-masked]               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

---

From: Christopher Kings-Lynne <[email-masked]>
Date: Wed, 22 Oct 2003 12:41:39 +0800

> change, only significant ones.

Personally, I think that's a fairly silly policy!  How does it hurt us 
to mention it and you just know that someone, somewhere, is doing it...

Chris

---

From: Tom Lane <[email-masked]>
Date: Wed, 22 Oct 2003 00:42:41 -0400

> change, only significant ones.

A moment's further thought reveals 'today' as another potentially broken
default value, which seems more likely to be used in practice than 
'yesterday' or 'tomorrow'.  I'm too beat to go digging for other legal
inputs, but there may be some...

			regards, tom lane

---

From: Neil Conway <[email-masked]>
Date: Wed, 22 Oct 2003 00:42:44 -0400

On Wed, 2003-10-22 at 00:41, Christopher Kings-Lynne wrote:
> Personally, I think that's a fairly silly policy!

I agree -- documenting possible areas of incompatibility is important,
and I would prefer that we err on the side of mentioning too much,
rather than too little.

-Neil

---

From: Christopher Kings-Lynne <[email-masked]>
Date: Wed, 22 Oct 2003 12:50:58 +0800

> inputs, but there may be some...

Ah, I didn't mention that one because I thought it was obvious and had 
already been mentioned :P

Chris

---

From: Bruce Momjian <[email-masked]>
Date: Wed, 22 Oct 2003 01:00:32 -0400 (EDT)

> to mention it and you just know that someone, somewhere, is doing it...

The release notes are already 550 lines --- adding the mention of
something almost no one uses doesn't make sense --- if we did that, the
list might be 1000 lines and more unreadable.  Same logic goes for the
list of changes --- I only mention user-visible changes, in most cases.

---

From: Bruce Momjian <[email-masked]>
Date: Wed, 22 Oct 2003 01:08:04 -0400 (EDT)

> and I would prefer that we err on the side of mentioning too much,

Docs updated to include 'today':

     <listitem><para> <literal>'now'</literal> will no longer work as a
        column default; <function>now()</> or
        <function>CURRENT_TIMESTAMP</> should be used instead</para></listitem>
     <listitem><para> <literal>'today'</literal> will no longer work as
        a column default; <function>CURRENT_DATE</>
        should be used instead</para></listitem>

As far as yesterday/tomorrow, I think anyone using that will realize
that if 'today' doesn't work, those will not either.  Sure, I like to be
complete too, but at a certain point it becomes overload and people
can't process it.  Part of the reason the release notes are read is
because they are _readable_, or as readable was we can make +300
changes.

Do you think I include every user-visible change in the release notes? 
It would be 2-3x longer, and probably not more useful.

---

From: Neil Conway <[email-masked]>
Date: Wed, 22 Oct 2003 02:37:57 -0400

> Part of the reason the release notes are read is
> because they are _readable_

On the contrary, I think we could do a lot to make the release notes
more readable, while at the same time providing more information to
users.

We're really serving two different audiences with the release notes.
Some people are just interested in the highlights of the release, so
that they are up to date on the latest new PostgreSQL functionality. For
these people, I think the existing "highlights of the release" section
at the top of the release notes is sufficient, although it could
probably be made a less terse.

The second audience is the people who are really interested in exactly
what has changed between the new release of PostgreSQL and the previous
release series. It is important that we make it easy for an admin
planning a PostgreSQL upgrade at a fairly large site to be able to see
what changes in PostgreSQL have been made, and what changes will be
necessary in their own applications. This audience is served fairly well
by the present release notes, but I think we could do better. The
backward-incompatibility section could definitely be improved, and the
whole process of summarizing the CVS logs after the fact is sure to lose
information. Furthermore, many of the release note entries are so brief
that it's difficult, even for someone familiar with PostgreSQL, to tell
exactly what they mean. Sometimes the entry doesn't even bother to
specify exactly what has been changed. Using complete sentences, where
warranted, and describing complex or important changes with a full
paragraph of text would be a good idea, IMHO. I've appended a few
examples of less-than-optimal entries to this mail.

So I think we could make the release notes more useful if we provided a
bit more detail in each entry, and documented changes more extensively.
We could also make better use of SGML, for example by adding <xref>s to
the release notes where applicable. I think we also need to *really*
maintain the release notes incrementally during 7.5 development, rather
than having Bruce summarize the CVS logs at the end. IMHO, every patch
that makes a significant change should update the release notes, when
the patch is applied.

Anyway, those are my thoughts on how to improve the release notes. I've
been meaning to get this off my chest for a while, so thanks for the
chance :-)

Comments are welcome.

-Neil

Here are a few examples of less-than-optimal entries I noticed while
browsing the release notes recently:

This HISTORY entry from 7.4 doesn't really tell anyone anything:

        * Multiple pg_dump fixes, including tar format and large objects

Or take this entry -- I can basically decipher what it means, but it
takes a fair degree of difficulty:

     * Disallow literal carriage return as a data value,
       backslash-carriage-return and \r are still allowed (Bruce)

Or this entry, which once again conveys little useful information, to me
at least:

     * Improve \d display (Christopher)

---

From: Richard Huxton <[email-masked]>
Date: Wed, 22 Oct 2003 10:30:52 +0100

Old version = 7.2.1
New version = 7.4.0

-- 
  Richard Huxton
  Archonet Ltd

---

From: Bruce Momjian <[email-masked]>
Date: Wed, 22 Oct 2003 13:12:20 -0400 (EDT)

>         * Multiple pg_dump fixes, including tar format and large objects

I agree it would be nice to use more xrefs in the release notes.

What happens when you go into more detail is that it comes even more
confusing --- in most cases we fixed some obscure flag combination or
something, and explaining that is even harder than just saying we fixed
some rare pg_dump failure cases.  The people who reported the problem
have already gotten their fix, and in most cases, they are the only ones
to have ever tried that combination --- documenting that doesn't really
add much, at least for me and I would assume 99% of our users.  We could
go for 99.9% of our users, but that makes things harder to filter for
the other 99%.

Please do a test --- do a cvs log of the bin/pg_dump directory and see
if I missed anything significant in the release notes.

If you start listing obscure changes, the significant ones don't really
stand out anymore.  You could add a 'obscure changes' section, but I
think once you create the list and read it, it will look pretty obscure.

>        backslash-carriage-return and \r are still allowed (Bruce)

OK, please improve the wording.

>      * Improve \d display (Christopher)

There were a huge number of \d display improvements --- do \d in 7.4 and
you will see them --- I don't see much value in saying, "Rules now
display as ...", or "defaults now have more parens".

I have always encouraged people to improve the existing notes.  I don't
pretend to use the perfect wording --- please send in improvements.

As far as incrementally updating the release notes --- lots of our work
is incremental, meaning we fix X, then add Y, and Z, and the resulting
change is one release note entry (psql \d display improvements, for
example).  Documenting them separately just leads to a mess of entries
that we have to consolidate at the end anyway.

---

From: Tom Lane <[email-masked]>
Date: Wed, 22 Oct 2003 13:26:00 -0400

> that we have to consolidate at the end anyway.

We do already have a practice of adding notes about significant changes
to release.sgml as they are made.  That's relatively new though, and I
dunno if it's helped Bruce prepare the finished release notes or not.
Bruce, did you use those notes at all this time?

			regards, tom lane

---

From: Bruce Momjian <[email-masked]>
Date: Wed, 22 Oct 2003 13:29:21 -0400 (EDT)

> Bruce, did you use those notes at all this time?

I checked them to make sure they were already included in my CVS notes,
and I might have used some of its wording if it was better than mine.

---

From: Michael Brusser <[email-masked]>
Date: Wed, 22 Oct 2003 13:33:34 -0400

We integrate PostgreSQL with our product, which we ship to the end user.
We do read the release notes, they are important to us.
They don't have to be excruciatingly long, they can't be
ridiculously short and cryptic.
You have to find the golden spot in the middle. Just treat us
the way you want to be treated + some extra allowance for ignorance.

Mike.

---

From: Bruce Momjian <[email-masked]>
Date: Wed, 22 Oct 2003 14:04:58 -0400 (EDT)

> the way you want to be treated + some extra allowance for ignorance.

The big question is whether the current release notes hit he right
balanced.  Do they for you?

---

From: Neil Conway <[email-masked]>
Date: Wed, 22 Oct 2003 14:26:56 -0400

> dunno if it's helped Bruce prepare the finished release notes or not.

Right, we also did a pretty bad job of incrementally updating
release.sgml during the development cycle: only a small portion of all
the changes that we made actually got added to it. I think it would be a
good idea to try to be better at this during the 7.5 cycle. When 7.5
development begins, we should divide release.sgml into the relevant
sections (e.g. "libpq", "contrib", "performance", "highlights of the
release", etc.), and then keep it updated whenever a relevant change is
made.

IMHO that would be bother easier to maintain (see we need to write CVS
changelog entries anyway, and Bruce wouldn't need to spend as long
summarizing the changes at the end of the dev cycle), as well as
producing a better quality release notes -- but since Bruce is the one
actually doing the work, I'm content to leave this in his hands.

-Neil

---

From: Bruce Momjian <[email-masked]>
Date: Wed, 22 Oct 2003 14:31:37 -0400 (EDT)

> actually doing the work, I'm content to leave this in his hands.

I find it easier to generate a list from raw CVS than to merge +500
entries into a consistent whole, and I never expect the maintained list
to be 100% accurate, so I have to do the CVS thing anyway.

---

From: Michael Brusser <[email-masked]>
Date: Thu, 23 Oct 2003 07:02:26 -0400

> -----Original Message-----
> From: Bruce Momjian 
...
> balanced.  Do they for you?

The last time I read the notes was when we upgraded to 7.3.4.
I'll pick up couple entries from Release Notes and the HISTORY file
(which we always read) for examples.

PostgreSQL now supports the ALTER TABLE ... DROP COLUMN functionality. 
 => this is entirely sufficient. Detailed info can be found in the docs.

Optimizer improvements
 => this tells me nothing. I suppose this could be a minor internal code
tweak, which does not affect me. On the other hand this could be a major
breakthrough, so now I can run some stupid query which would take
a week to complete in the previous release. How do I know? 

Fix to_ascii() buffer overruns
 => I don't think I need any more details here

Work around buggy strxfrm() present in some Solaris releases
 => if we did not suffer from this (big thanks for fixing!) I would've
never guessed how it may manifest itself and affect the database,
even though this alone could be a strong reason for upgrade.

If you think this would take too much space and bloat the document,
then maybe the best solution is to have a reference number:
 Bug# 123 : Work around ...
Then I could go to some http://postgres../bugtrack enter this number
and learn more.

Mike.

---

From: Bruce Momjian <[email-masked]>
Date: Thu, 23 Oct 2003 12:33:34 -0400 (EDT)

>  => this is entirely sufficient. Detailed info can be found in the docs.

Good.

> a week to complete in the previous release. How do I know? 

Yes, this is always very hard to explain. The optimizer itself is
complex, and uses complex terms like merge join and key pruning.  It is
hard to explain what queries will be affected, though the basic issue is
that the optimizer will choose a better plan more frequently.

> even though this alone could be a strong reason for upgrade.

We don't actually explain enough in the release notes for people to
determine if they should do _minor_ upgrades --- bottom line is that
minor upgrades only require a stop/install/restart postmaster, so we
assume everyone will do that, and in this case, if you are running
Solaris, that is enough of a reason alone --- whether that particular
bug affects you or not.

> Then I could go to some http://postgres../bugtrack enter this number

Yes, that would be nice.

So, it sounds like we are already pretty close to ideal for you.

---

From: Greg Stark <[email-masked]>
Date: 24 Oct 2003 00:06:35 -0400

> that the optimizer will choose a better plan more frequently.

One thing that might be worth mentioning is that "WHERE foo IN (subquery)"
type queries are much improved. That's a one of the more common complaints
about 7.3 and previous and it's one that fairly easy to recognize.

-- 
greg

---

From: Bruce Momjian <[email-masked]>
Date: Fri, 24 Oct 2003 00:09:17 -0400 (EDT)

> about 7.3 and previous and it's one that fairly easy to recognize.

That is right at the top of the release notes:

   Performance
          IN/NOT IN subqueries are now much more efficient

---

From: Peter Eisentraut <[email-masked]>
Date: Fri, 24 Oct 2003 15:06:13 +0200 (CEST)

> that makes a significant change should update the release notes, when

I've been pushing this agenda for a few releases now, but some people have
been, er, boycotting it.  I think, too, that release notes *must* be
written incrementally at the same time that the feature change is made.
This is the only way we can get accurate and complete release notes, and
the descriptions could even include some context, some motivations, etc.
We have release cycles of 10 months, and there is no way we can make
sensible release notes by gathering individual commit messages over that
period of time.  Heck, ECPG has a full Informix compatibility mode and
there is no mention of that anywhere, because there was no commit "Add
Informix mode."

I suggest we just do it like the documentation:  If you don't document it,
it doesn't exist.  If you don't write a line for the release notes, it
doesn't exist either.

-- 
Peter Eisentraut   [email-masked]

---

From: Peter Eisentraut <[email-masked]>
Date: Fri, 24 Oct 2003 15:11:25 +0200 (CEST)

Peter Eisentraut writes:

> Heck, ECPG has a full Informix compatibility mode and there is no
> mention of that anywhere, because there was no commit "Add Informix
> mode."

Sorry, inconsistent spelling tripped me up on this one.  But the
theoretical point stands.

---

From: Alvaro Herrera <[email-masked]>
Date: Fri, 24 Oct 2003 10:25:42 -0300

> > mention of that anywhere, because there was no commit "Add Informix

I still wonder what "Informix compatibility mode" means.  Is it that you
can take an embedded-C program from Informix and compile it with ECPG?

-- 
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"Doing what he did amounts to sticking his fingers under the hood of the
implementation; if he gets his fingers burnt, it's his problem."  (Tom Lane)

---

From: Bruce Momjian <[email-masked]>
Date: Fri, 24 Oct 2003 10:08:33 -0400 (EDT)

> been, er, boycotting it.  I think, too, that release notes *must* be

If they _must_ be done the way you suggest, why have we been able to
generate reliable release notes all these years?

Basically, I think release notes are more efficiently written in batch
mode, meaning all at once --- sure, we could do it incrementally, but it
is more work to fiddle with it in pieces.

I want people to focus on reliable commit messages and I can handle the
release notes part.  The one advantage of incremental is that folks can
see what we have added so far, but it doesn't seem worth the extra work.

---

From: Bruce Momjian <[email-masked]>
Date: Fri, 24 Oct 2003 10:10:33 -0400 (EDT)

> can take an embedded-C program from Informix and compile it with ECPG?

Yes, it is for compiling/running informix-style ecpg programs with
PostgreSQL.

---

From: Peter Eisentraut <[email-masked]>
Date: Fri, 24 Oct 2003 17:07:21 +0200 (CEST)

> generate reliable release notes all these years?

With all respect for your work and your enthusiasm for this approach, but
personally, I have absolutely no confidence that the release notes are
complete, accurate, or reliable.  That is my deeply founded motivation for
trying to institute changes.  But I might be wrong and my fears might have
no merit.

---

From: Bruce Momjian <[email-masked]>
Date: Fri, 24 Oct 2003 11:23:56 -0400 (EDT)

> trying to institute changes.  But I might be wrong and my fears might have

Sure, I fear missing things too.  One of my goals is to make sure I
understand all the CVS commit messages that appear, so I know later I
can make a release note about it.  Right now, we have a simple process
that you can move through step by step and make sure you get everything.
With a piecemeal approach, I think we are much more likely to miss
something, or have things get confused by merging a CVS log version and
a piecemeal version.

Looking at the past, I don't remember us missing anything --- I do
remember missing a few items that should have been listed in the
compatibilities section, but that is more of a problem of us not
reviewing the release notes thoroughly.

---

From: Tatsuo Ishii <[email-masked]>
Date: Sat, 25 Oct 2003 09:44:03 +0900 (JST)

> it doesn't exist.  If you don't write a line for the release notes, it

I tend to agree it. For every release I and my colleague have been
working on creating detailed release notes (of course in Japanese),
otherwise we cannot tell people what are changed, added or fixed since
there is little info in the official release note. This is painful
since we have to dig into the mail archives and cvs commit messages to
look for what each item of the official release note actually
means. These work take at least 2 to 3 weeks with several people
involved. The hardest part is what are fixed. The only useful
information seems to be the cvs commit messages, however typical
messages are something like "see recent discussions in the mail
archive for more details". This is not very helpful at least for
me. Once I proposed that we add a sequence number to each mail and the
commit messages point to the number. This way we could easily trace
what are the bug report and what are the actual intention for the
fix. For some reason noboy was interested in. Maybe this is due to
"coulture gap"... (In Japan giving a sequence number to each mail in
mailing lists is quite common).
--
Tatsuo Ishii

---

From: Bruce Momjian <[email-masked]>
Date: Sun, 26 Oct 2003 00:40:20 -0400 (EDT)

> mailing lists is quite common).

OK, if Tatsuo and SRA are having problems, I have to address it.  I can
supply a more detailed list to Tatsuo/SRA, or I can beef up the release
notes to contain more information.  Seems some in the community would
like to have this detail so I might as well do it and have it in the
official docs.  One idea would be to add a section at the bottom of the
release notes that goes into detail on changes listed in the release
notes above --- that way, people can still skim the 300-line release
notes, and if they want detailed information about the optimizer changes
or subtle pg_dump fixes, that will be at the bottom.

How does that sound?  I can start on this for 7.4 next week.  It
basically means going through the CVS logs again and pulling out
additional details.

---

From: Tatsuo Ishii <[email-masked]>
Date: Sun, 26 Oct 2003 13:55:22 +0900 (JST)

> basically means going through the CVS logs again and pulling out

Sounds good. However this kind of information could become huge and I
am afraid it does not suite well in the official docs in the source
tree. I think putiing it in somewhere in a web site (maybe
http://developer.postgresql.org/?) might be more appropreate.
What do you think?
--
Tatsuo Ishii

---

From: Bruce Momjian <[email-masked]>
Date: Sun, 26 Oct 2003 01:08:21 -0400 (EDT)

> http://developer.postgresql.org/?) might be more appropreate.

Yes, I had thought about that --- I put something about migrating to 7.3
on a web page and put the URL in the release notes, and the URL kept
becoming invalid as they changed web configurations.  I am afraid we
have to keep this in CVS so we don't lose it over time.

It could be huge.  Current release notes for 7.4 is around 500 lines.  I
think I could do this in another 500 lines.

---

From: Andrew Dunstan <[email-masked]>
Date: Sun, 26 Oct 2003 06:56:44 -0500

>think I could do this in another 500 lines.

This is starting to sound awfully like a conventional bug tracking 
system but without the support infrastructure. Why not go the whole 
distance? Then your release notes could give the numbers and headings of 
bugs fixed  (including features added) and those who want more details 
could look at the bug reports. The process could be a whole lot less 
painful than it appears to be right now, I suspect.

cheers

andrew

---

From: Bruce Momjian <[email-masked]>
Date: Sun, 26 Oct 2003 08:44:53 -0500 (EST)

> painful than it appears to be right now, I suspect.

Bug tracking systems have the same limitation as incremental release
notes --- youi have to do a lot of piecemeal work to get complete output
at the end, rather than doing it more efficiently in one batch.

Most people working on PostgreSQL are volunteers, and one of my primary
jobs is to make it easy for them --- if it takes me a week to get the
release notes together --- so be it --- I am making it easier for
others.

---

From: Andrew Dunstan <[email-masked]>
Date: Sun, 26 Oct 2003 11:43:07 -0500

>release notes together --- so be it --- I am making it easier for

You do a fine job and I know it is appreciated.  I'd hate to think what 
would happen if you got run over by a bus.

It's a bit of a matter of taste - I think bug tracking systems give 
projects better support than mailing lists, but maybe that's just me.

cheers

andrew

---

From: Bruce Momjian <[email-masked]>
Date: Sun, 26 Oct 2003 21:39:44 -0500 (EST)

> projects better support than mailing lists, but maybe that's just me.

I think our goal is to get every known problem on the TODO list so
people can scan it quickly.  It is also easier to have it all
centralized and distilled.  However, it is hard to get more detail on
the bug, so we have TODO.detail.

Bug tracking systems are an extra level of abstraction for projects in
that it categorizes discussion.  However, that abstraction level also
slows things down --- hard to say if it is a win or not.

Ideally, I would like to do all the unpleasant work and making improving
PostgreSQL easy for others.

---

From: Bruce Momjian <[email-masked]>
Date: Wed, 29 Oct 2003 15:44:59 -0500 (EST)

I have added my first release note detail item.  I used <footnote> to
add a description to the first release note item.  You can see it
rendered here:

	http://candle.pha.pa.us/main/writings/pgsql/sgml/release.html#RELEASE-DEVEL

I plan to go through the release notes and add more.  If we decide on a
different format, I can change this.

---------------------------------------------------------------------------

---

From: Bruce Momjian <[email-masked]>
Date: Wed, 29 Oct 2003 15:44:59 -0500 (EST)

---

From: Peter Eisentraut <[email-masked]>
Date: Thu, 30 Oct 2003 18:34:20 +0100 (CET)

> release notes together --- so be it --- I am making it easier for

That is not the scalable community approach that has been successful in
other areas of development.  You might as well say, "Just tell me all the
features you need and I'll implement them."  Now *that* would make it easy
for other people.

Once upon a time we thought that documentation wasn't important or that is
was hindering people to get involved.  I think that has largely been
disproven and we have been very successful with the "document the code
when you write it" approach.  The same approach can be used for the
release notes.

---

From: Peter Eisentraut <[email-masked]>
Date: Thu, 30 Oct 2003 18:35:40 +0100 (CET)

Bruce Momjian writes:

> add a description to the first release note item.

Please don't use footnotes.  They make things really hard to read.  There
are plenty of other mechanisms to organize information.

---

From: Bruce Momjian <[email-masked]>
Date: Thu, 30 Oct 2003 13:16:06 -0500 (EST)

> are plenty of other mechanisms to organize information.

OK, please suggest what I should use.

---

From: Bruce Momjian <[email-masked]>
Date: Thu, 30 Oct 2003 13:22:34 -0500 (EST)

> features you need and I'll implement them."  Now *that* would make it easy

If I can do some cleanup work that enables five other people to work an
hour, my time is well spent.  While we like people to indent their code,
(and sometimes we indent it after submission) we go through an indent it
all at once before the release.  I see the release notes as more of a
housekeeping service, more efficiently done in one big batch.

---

From: Bruce Momjian <[email-masked]>
Date: Thu, 30 Oct 2003 23:59:05 -0500 (EST)

OK, I have committed changes to release.sgml so most complex entries
have a paragraph describing the change.  You can see the result at:

	http://candle.pha.pa.us/main/writings/pgsql/sgml/release.html#RELEASE-7-4

I need people to check this and help me with the items marked 'bjm'.  I
am confused about the proper text for those sections.

---

From: Tom Lane <[email-masked]>
Date: Fri, 31 Oct 2003 00:43:48 -0500

Bruce Momjian <[email-masked]> writes:
> I need people to check this and help me with the items marked 'bjm'.

Okay, a few comments ...

<listitem><para> IN/NOT IN subqueries are now much more efficient</para>
  <para>
   In previous releases, IN/NOT IN subqueries were joined to the
   upper query by sequentially scanning the subquery looking for
   a join.  The 7.4 code uses the same sophisticated techniques
   used by ordinary joins and so is much faster, and is now faster
   than EXISTS subqueries.
  </para>
</listitem>

This might be overstatement.  How about "... is much faster.  An IN
will now usually be as fast as or faster than an equivalent EXISTS
subquery; this reverses the conventional wisdom that applied to previous
Postgres releases."

<listitem><para> Improved GROUP BY processing by using hash buckets</para>
  <para>
   In previous releases, GROUP BY totals were accumulated by
   sequentially scanning the list of groups looking for a match;
   the 7.4 code places GROUP BY values in hash buckets so the
   proper match can be found much quicker.  This is particularly
   significant in speeding up queries that have a large
   number of distinct GROUP BY values.
  </para>
</listitem>

This is backwards.  I suggest "In previous releases, GROUP BY required
sorting the input data to bring group members together.  7.4 can do it
that way, or can accumulate data into per-group hash buckets in-memory.
The hash technique avoids a sort and so can be much faster, if the
number of distinct GROUP BY values is not too large to fit in memory."

<listitem><para> ANSI joins are now better optimized</para>
   <para>
    Prior releases evaluated ANSI join syntax only in the order
    specified by the query;  7.4 allows full optimization of
    queries using ANSI join syntax, meaning the optimizer considers
    all possible join orderings and chooses the most efficient.
   </para>
</listitem>

This is correct only for inner joins.  Outer joins still follow the
syntax-implied ordering.  Not sure what the best rewording is.

<listitem><para> Full support for IPv6 connections and IPv6 address
data types</para>
   <para>
    Prior releases allowed only IPv6 connections and IP data types only
    supported IPv4 addresses. This release adds full IPv6 support in
    both of these areas.
   </para>
</listitem>

Surely "allowed only IPv4 connections".

<listitem><para> New protocol improves connection speed/reliability,
and adds error codes, status information, a binary protocol, error
reporting verbosity, and cleaner startup packets.</para>
</listitem>

I dunno anything about improving connection speed/reliability.  How
about "New client-to-server protocol adds error codes, more status
information, better support for binary data transmission, parameter
values separated from SQL commands, prepared statements available at the
protocol level, clean recovery from COPY failures, and cleaner startup
packets.  The older protocol is still supported by both servers and
clients."

<listitem><para>Align shared buffers on 32-byte boundary for copy speed improvement (Manfred Spraul)</para>
   <para>
    Certain CPU's perform faster data copies when addresses are 32-bit
    aligned.
   </para>
   </listitem>

bit -> byte

<listitem><para>Fix subquery aggregates of upper query columns to match SQL spec. (Tom)</para>
   <para>
    bjm
   </para>
   </listitem>

Try:

Fix aggregates in subqueries to match SQL spec

The SQL spec says that an aggregate function appearing within a nested
subquery belongs to the outer query if its argument contains only
outer-query variables.  Prior PG releases did not handle this fine point
correctly.

<listitem><para>Add option to prevent auto-addition of tables referenced in query (Nigel J.
  Andrews)  </para>
     <para>
      By default, tables mentioned in the query are automatically added
      to the FROM clause if they are not already there. This option
      disabled that behavior.
     </para>
   </listitem>

I'd suggest "... not already there.  This is compatible with
historical Postgres behavior but is contrary to the SQL spec.
This option allows selecting spec-compatible behavior."

<listitem><para>Multiple pggla_dump fixes, including tar format and large objects</para></listitem>

"pggla_dump"?

<listitem><para>Syntax errors now reported as 'syntax error' rather than 'parse error' (Tom)</para></listitem>

Is it worth giving this its own bullet point?  It's far down in the
noise compared to all the other message rewordings.

			regards, tom lane

---

From: Joe Conway <[email-masked]>
Date: Thu, 30 Oct 2003 21:54:10 -0800

> Allow polymorphic SQL functions (Joe)
 > bjm ??

What isn't clear about this? Should/can we refer to related sections of 
the manual?
http://developer.postgresql.org/docs/postgres/xfunc-sql.html#AEN28722
http://developer.postgresql.org/docs/postgres/extend-type-system.html#EXTEND-TYPES-POLYMORPHIC

 > Allow user defined aggregates to use polymorphic functions (Joe)
 > bjm ??

Same question. From this url:
http://developer.postgresql.org/docs/postgres/xaggr.html
see this paragraph:

  Aggregate functions may use polymorphic state transition functions or 
final functions, so that the same functions can be used to implement 
multiple aggregates. See Section 33.2.1  for an explanation of 
polymorphic functions. Going a step further, the aggregate function 
itself may be specified with a polymorphic base type and state type, 
allowing a single aggregate definition to serve for multiple input data 
types. Here is an example of a polymorphic aggregate:

CREATE AGGREGATE array_accum (
     sfunc = array_append,
     basetype = anyelement,
     stype = anyarray,
     initcond = '{}'
);

Joe

---

From: Kurt Roeckx <[email-masked]>
Date: Fri, 31 Oct 2003 18:39:47 +0100

> have a paragraph describing the change.  You can see the result at:

     * Full support for IPv6 connections and IPv6 address data types
       Prior releases allowed only IPv6 connections and IP data types
       only supported IPv4 addresses. This release adds full IPv6 support
       in both of these areas.

You probably want to say "Prior releases allowed only IPv4
connections".

Kurt

---

From: Bruce Momjian <[email-masked]>
Date: Fri, 31 Oct 2003 14:26:38 -0500 (EST)

> subquery; this reverses the conventional wisdom that applied to previous

Done.

> number of distinct GROUP BY values is not too large to fit in memory."

Done.

> Surely "allowed only IPv4 connections".

Yep, fixed.

> packets.  The older protocol is still supported by both servers and

Updated with your text.

I thought connections were faster because we passed fewer packets on
startup, and I thought you measured a speed improvement in connection
startup time.  Am I remembering wrong?

>     Certain CPU's perform faster data copies when addresses are 32-bit

Fixed.

> outer-query variables.  Prior PG releases did not handle this fine point

Updated.

> This option allows selecting spec-compatible behavior."

Updated.

> <listitem><para>Multiple pggla_dump fixes, including tar format and large objects</para></listitem>

Sure, you know, pggla_dump. :-)   Fixed.

> noise compared to all the other message rewordings.

I added it to the compatbility section at the top.

---

From: Bruce Momjian <[email-masked]>
Date: Fri, 31 Oct 2003 14:43:10 -0500 (EST)

Fixed.

---

From: Tom Lane <[email-masked]>
Date: Fri, 31 Oct 2003 14:47:54 -0500

> startup time.  Am I remembering wrong?

You're right, connection startup should be a little faster.  "Connection
speed" implies a more general performance improvement, though, and I
don't think we have a basis for claiming that.  If you want to say
faster startup, go ahead.

			regards, tom lane

---

From: Bruce Momjian <[email-masked]>
Date: Fri, 31 Oct 2003 14:49:00 -0500 (EST)

> CREATE AGGREGATE array_accum (

What had me really confused was the first release item:

How does an SQL function query the data types passed to it?  Once I saw
that I thought I didn't underestand what polymorphic functions were.

Right now we have:
	
	<listitem><para>Allow user defined aggregates to use polymorphic functions (Joe)</para>
	   <para>
	    bjm ??
	   </para>
	   </listitem>
	<listitem><para>Allow polymorphic user defined aggregates  (Joe)</para></listitem>

These seem like duplicates.  Are polymorphic functions currently most
useful for aggregates?  Why would someone want polymorphic aggregates? 
That is what I was hoping for.

---

From: Bruce Momjian <[email-masked]>
Date: Fri, 31 Oct 2003 14:51:12 -0500 (EST)

Updated.

---

From: elein <[email-masked]>
Date: Fri, 31 Oct 2003 12:55:12 -0800

I included some text about the information schema in
this issue of general bits.  I also did some documentation
of comparison of the changes in the postgresql.conf.

Anyone who wants to grab parts of those items in that
issue has my permission. I don't have time to re-edit
for the release note format.  But maybe there is some
clarification text you can use.

http://cookie.varlena.com:8080/varlena/GeneralBits/48.php

elein

On Thu, Oct 30, 2003 at 11:59:05PM -0500, Bruce Momjian wrote:
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to [email-masked]

---

From: Joe Conway <[email-masked]>
Date: Fri, 31 Oct 2003 14:10:26 -0800

Bruce Momjian wrote:
> How does an SQL function query the data types passed to it?  Once I
> saw that I thought I didn't underestand what polymorphic functions
> were.

It doesn't need to. For example:

CREATE OR REPLACE FUNCTION makearray(anyelement, anyelement) returns
anyarray as 'select ARRAY[$1, $2]' language sql;

regression=# select makearray(1,2);
  makearray
-----------
  {1,2}
(1 row)

regression=# select makearray('a'::text,'b');
  makearray
-----------
  {a,b}
(1 row)

> <listitem><para>Allow user defined aggregates to use polymorphic
> functions (Joe)</para> <listitem><para>Allow polymorphic user defined
> aggregates  (Joe)</para></listitem>
> 
> These seem like duplicates.

They aren't. The first says you could create an aggregate with defined
base and state datatypes, but where the state/final functions might be
polymorphic. The second says that the base and state types might be
polymorphic.

> Are polymorphic functions currently most useful for aggregates?  Why
> would someone want polymorphic aggregates? That is what I was hoping
> for.

Well, one example is a calculation aggregate (let's say median) which
you might want to use for any numeric data type. Or an array 
accumulator, e.g.

CREATE AGGREGATE myagg1
(
   BASETYPE = float8,
   SFUNC = array_append,
   STYPE = float8[],
   INITCOND = '{}'
);
CREATE AGGREGATE

CREATE AGGREGATE myagg1p
(
   BASETYPE = anyelement,
   SFUNC = array_append,
   STYPE = anyarray,
   INITCOND = '{}'
);
CREATE AGGREGATE

Joe

---

From: Renê Salomão <[email-masked]>
Date: Wed, 5 Nov 2003 10:50:46 -0200

UNSUBSCRIBE

---

From: Bruce Momjian <[email-masked]>
Date: Mon, 10 Nov 2003 17:26:18 -0500 (EST)

OK, release notes updated to:

<listitem><para>Allow polymorphic PL/pgSQL functions (Tom, Joe)</para></listitem>
<listitem><para>Allow polymorphic SQL functions (Joe)</para>
   <para>
    Allow functions to accept arbitrary data types for input, and return arbitrary types.
   </para>
   </listitem>

Joe Conway wrote:
> TIP 9: the planner will ignore your desire to choose an index scan if your
>       joining column's datatypes do not match
>

---

From: Renê Salomão <[email-masked]>
Date: Wed, 17 Mar 2004 13:54:03 -0300

unsubscribe pgsql-docs "Rene" <[email-masked]>