Subject: Annotated release notes

From: Bruce Momjian <[email-masked]>
Date: Thu, 30 Oct 2003 23:59:05 -0500 (EST)

OK, I have committed changes to release.sgml so most complex entries
have a paragraph describing the change.  You can see the result at:

	http://candle.pha.pa.us/main/writings/pgsql/sgml/release.html#RELEASE-7-4

I need people to check this and help me with the items marked 'bjm'.  I
am confused about the proper text for those sections.

---------------------------------------------------------------------------

Tatsuo Ishii wrote:
> > Tatsuo Ishii wrote:
> > > > I've been pushing this agenda for a few releases now, but some people have
> > > > been, er, boycotting it.  I think, too, that release notes *must* be
> > > > written incrementally at the same time that the feature change is made.
> > > > This is the only way we can get accurate and complete release notes, and
> > > > the descriptions could even include some context, some motivations, etc.
> > > > We have release cycles of 10 months, and there is no way we can make
> > > > sensible release notes by gathering individual commit messages over that
> > > > period of time.  Heck, ECPG has a full Informix compatibility mode and
> > > > there is no mention of that anywhere, because there was no commit "Add
> > > > Informix mode."
> > > > 
> > > > I suggest we just do it like the documentation:  If you don't document it,
> > > > it doesn't exist.  If you don't write a line for the release notes, it
> > > > doesn't exist either.
> > > 
> > > I tend to agree it. For every release I and my colleague have been
> > > working on creating detailed release notes (of course in Japanese),
> > > otherwise we cannot tell people what are changed, added or fixed since
> > > there is little info in the official release note. This is painful
> > > since we have to dig into the mail archives and cvs commit messages to
> > > look for what each item of the official release note actually
> > > means. These work take at least 2 to 3 weeks with several people
> > > involved. The hardest part is what are fixed. The only useful
> > > information seems to be the cvs commit messages, however typical
> > > messages are something like "see recent discussions in the mail
> > > archive for more details". This is not very helpful at least for
> > > me. Once I proposed that we add a sequence number to each mail and the
> > > commit messages point to the number. This way we could easily trace
> > > what are the bug report and what are the actual intention for the
> > > fix. For some reason noboy was interested in. Maybe this is due to
> > > "coulture gap"... (In Japan giving a sequence number to each mail in
> > > mailing lists is quite common).
> > 
> > OK, if Tatsuo and SRA are having problems, I have to address it.  I can
> > supply a more detailed list to Tatsuo/SRA, or I can beef up the release
> > notes to contain more information.  Seems some in the community would
> > like to have this detail so I might as well do it and have it in the
> > official docs.  One idea would be to add a section at the bottom of the
> > release notes that goes into detail on changes listed in the release
> > notes above --- that way, people can still skim the 300-line release
> > notes, and if they want detailed information about the optimizer changes
> > or subtle pg_dump fixes, that will be at the bottom.
> > 
> > How does that sound?  I can start on this for 7.4 next week.  It
> > basically means going through the CVS logs again and pulling out
> > additional details.
> 
> Sounds good. However this kind of information could become huge and I
> am afraid it does not suite well in the official docs in the source
> tree. I think putiing it in somewhere in a web site (maybe
> http://developer.postgresql.org/?) might be more appropreate.
> What do you think?
> --
> Tatsuo Ishii
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  [email-masked]               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

---

From: Tom Lane <[email-masked]>
Date: Fri, 31 Oct 2003 00:43:48 -0500

Bruce Momjian <[email-masked]> writes:
> I need people to check this and help me with the items marked 'bjm'.

Okay, a few comments ...

<listitem><para> IN/NOT IN subqueries are now much more efficient</para>
  <para>
   In previous releases, IN/NOT IN subqueries were joined to the
   upper query by sequentially scanning the subquery looking for
   a join.  The 7.4 code uses the same sophisticated techniques
   used by ordinary joins and so is much faster, and is now faster
   than EXISTS subqueries.
  </para>
</listitem>

This might be overstatement.  How about "... is much faster.  An IN
will now usually be as fast as or faster than an equivalent EXISTS
subquery; this reverses the conventional wisdom that applied to previous
Postgres releases."

<listitem><para> Improved GROUP BY processing by using hash buckets</para>
  <para>
   In previous releases, GROUP BY totals were accumulated by
   sequentially scanning the list of groups looking for a match;
   the 7.4 code places GROUP BY values in hash buckets so the
   proper match can be found much quicker.  This is particularly
   significant in speeding up queries that have a large
   number of distinct GROUP BY values.
  </para>
</listitem>

This is backwards.  I suggest "In previous releases, GROUP BY required
sorting the input data to bring group members together.  7.4 can do it
that way, or can accumulate data into per-group hash buckets in-memory.
The hash technique avoids a sort and so can be much faster, if the
number of distinct GROUP BY values is not too large to fit in memory."

<listitem><para> ANSI joins are now better optimized</para>
   <para>
    Prior releases evaluated ANSI join syntax only in the order
    specified by the query;  7.4 allows full optimization of
    queries using ANSI join syntax, meaning the optimizer considers
    all possible join orderings and chooses the most efficient.
   </para>
</listitem>

This is correct only for inner joins.  Outer joins still follow the
syntax-implied ordering.  Not sure what the best rewording is.

<listitem><para> Full support for IPv6 connections and IPv6 address
data types</para>
   <para>
    Prior releases allowed only IPv6 connections and IP data types only
    supported IPv4 addresses. This release adds full IPv6 support in
    both of these areas.
   </para>
</listitem>

Surely "allowed only IPv4 connections".

<listitem><para> New protocol improves connection speed/reliability,
and adds error codes, status information, a binary protocol, error
reporting verbosity, and cleaner startup packets.</para>
</listitem>

I dunno anything about improving connection speed/reliability.  How
about "New client-to-server protocol adds error codes, more status
information, better support for binary data transmission, parameter
values separated from SQL commands, prepared statements available at the
protocol level, clean recovery from COPY failures, and cleaner startup
packets.  The older protocol is still supported by both servers and
clients."

<listitem><para>Align shared buffers on 32-byte boundary for copy speed improvement (Manfred Spraul)</para>
   <para>
    Certain CPU's perform faster data copies when addresses are 32-bit
    aligned.
   </para>
   </listitem>

bit -> byte

<listitem><para>Fix subquery aggregates of upper query columns to match SQL spec. (Tom)</para>
   <para>
    bjm
   </para>
   </listitem>

Try:

Fix aggregates in subqueries to match SQL spec

The SQL spec says that an aggregate function appearing within a nested
subquery belongs to the outer query if its argument contains only
outer-query variables.  Prior PG releases did not handle this fine point
correctly.

<listitem><para>Add option to prevent auto-addition of tables referenced in query (Nigel J.
  Andrews)  </para>
     <para>
      By default, tables mentioned in the query are automatically added
      to the FROM clause if they are not already there. This option
      disabled that behavior.
     </para>
   </listitem>

I'd suggest "... not already there.  This is compatible with
historical Postgres behavior but is contrary to the SQL spec.
This option allows selecting spec-compatible behavior."

<listitem><para>Multiple pggla_dump fixes, including tar format and large objects</para></listitem>

"pggla_dump"?

<listitem><para>Syntax errors now reported as 'syntax error' rather than 'parse error' (Tom)</para></listitem>

Is it worth giving this its own bullet point?  It's far down in the
noise compared to all the other message rewordings.

			regards, tom lane

---

From: Joe Conway <[email-masked]>
Date: Thu, 30 Oct 2003 21:54:10 -0800

> Allow polymorphic SQL functions (Joe)
 > bjm ??

What isn't clear about this? Should/can we refer to related sections of 
the manual?
http://developer.postgresql.org/docs/postgres/xfunc-sql.html#AEN28722
http://developer.postgresql.org/docs/postgres/extend-type-system.html#EXTEND-TYPES-POLYMORPHIC

 > Allow user defined aggregates to use polymorphic functions (Joe)
 > bjm ??

Same question. From this url:
http://developer.postgresql.org/docs/postgres/xaggr.html
see this paragraph:

  Aggregate functions may use polymorphic state transition functions or 
final functions, so that the same functions can be used to implement 
multiple aggregates. See Section 33.2.1  for an explanation of 
polymorphic functions. Going a step further, the aggregate function 
itself may be specified with a polymorphic base type and state type, 
allowing a single aggregate definition to serve for multiple input data 
types. Here is an example of a polymorphic aggregate:

CREATE AGGREGATE array_accum (
     sfunc = array_append,
     basetype = anyelement,
     stype = anyarray,
     initcond = '{}'
);

Joe

---

From: Kurt Roeckx <[email-masked]>
Date: Fri, 31 Oct 2003 18:39:47 +0100

> have a paragraph describing the change.  You can see the result at:

     * Full support for IPv6 connections and IPv6 address data types
       Prior releases allowed only IPv6 connections and IP data types
       only supported IPv4 addresses. This release adds full IPv6 support
       in both of these areas.

You probably want to say "Prior releases allowed only IPv4
connections".

Kurt

---

From: Bruce Momjian <[email-masked]>
Date: Fri, 31 Oct 2003 14:26:38 -0500 (EST)

> subquery; this reverses the conventional wisdom that applied to previous

Done.

> number of distinct GROUP BY values is not too large to fit in memory."

Done.

> Surely "allowed only IPv4 connections".

Yep, fixed.

> packets.  The older protocol is still supported by both servers and

Updated with your text.

I thought connections were faster because we passed fewer packets on
startup, and I thought you measured a speed improvement in connection
startup time.  Am I remembering wrong?

>     Certain CPU's perform faster data copies when addresses are 32-bit

Fixed.

> outer-query variables.  Prior PG releases did not handle this fine point

Updated.

> This option allows selecting spec-compatible behavior."

Updated.

> <listitem><para>Multiple pggla_dump fixes, including tar format and large objects</para></listitem>

Sure, you know, pggla_dump. :-)   Fixed.

> noise compared to all the other message rewordings.

I added it to the compatbility section at the top.

---

From: Bruce Momjian <[email-masked]>
Date: Fri, 31 Oct 2003 14:43:10 -0500 (EST)

Fixed.

---

From: Tom Lane <[email-masked]>
Date: Fri, 31 Oct 2003 14:47:54 -0500

> startup time.  Am I remembering wrong?

You're right, connection startup should be a little faster.  "Connection
speed" implies a more general performance improvement, though, and I
don't think we have a basis for claiming that.  If you want to say
faster startup, go ahead.

			regards, tom lane

---

From: Bruce Momjian <[email-masked]>
Date: Fri, 31 Oct 2003 14:49:00 -0500 (EST)

> CREATE AGGREGATE array_accum (

What had me really confused was the first release item:

How does an SQL function query the data types passed to it?  Once I saw
that I thought I didn't underestand what polymorphic functions were.

Right now we have:
	
	<listitem><para>Allow user defined aggregates to use polymorphic functions (Joe)</para>
	   <para>
	    bjm ??
	   </para>
	   </listitem>
	<listitem><para>Allow polymorphic user defined aggregates  (Joe)</para></listitem>

These seem like duplicates.  Are polymorphic functions currently most
useful for aggregates?  Why would someone want polymorphic aggregates? 
That is what I was hoping for.

---

From: Bruce Momjian <[email-masked]>
Date: Fri, 31 Oct 2003 14:51:12 -0500 (EST)

Updated.

---

From: elein <[email-masked]>
Date: Fri, 31 Oct 2003 12:55:12 -0800

I included some text about the information schema in
this issue of general bits.  I also did some documentation
of comparison of the changes in the postgresql.conf.

Anyone who wants to grab parts of those items in that
issue has my permission. I don't have time to re-edit
for the release note format.  But maybe there is some
clarification text you can use.

http://cookie.varlena.com:8080/varlena/GeneralBits/48.php

elein

On Thu, Oct 30, 2003 at 11:59:05PM -0500, Bruce Momjian wrote:
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to [email-masked]

---

From: Joe Conway <[email-masked]>
Date: Fri, 31 Oct 2003 14:10:26 -0800

Bruce Momjian wrote:
> How does an SQL function query the data types passed to it?  Once I
> saw that I thought I didn't underestand what polymorphic functions
> were.

It doesn't need to. For example:

CREATE OR REPLACE FUNCTION makearray(anyelement, anyelement) returns
anyarray as 'select ARRAY[$1, $2]' language sql;

regression=# select makearray(1,2);
  makearray
-----------
  {1,2}
(1 row)

regression=# select makearray('a'::text,'b');
  makearray
-----------
  {a,b}
(1 row)

> <listitem><para>Allow user defined aggregates to use polymorphic
> functions (Joe)</para> <listitem><para>Allow polymorphic user defined
> aggregates  (Joe)</para></listitem>
> 
> These seem like duplicates.

They aren't. The first says you could create an aggregate with defined
base and state datatypes, but where the state/final functions might be
polymorphic. The second says that the base and state types might be
polymorphic.

> Are polymorphic functions currently most useful for aggregates?  Why
> would someone want polymorphic aggregates? That is what I was hoping
> for.

Well, one example is a calculation aggregate (let's say median) which
you might want to use for any numeric data type. Or an array 
accumulator, e.g.

CREATE AGGREGATE myagg1
(
   BASETYPE = float8,
   SFUNC = array_append,
   STYPE = float8[],
   INITCOND = '{}'
);
CREATE AGGREGATE

CREATE AGGREGATE myagg1p
(
   BASETYPE = anyelement,
   SFUNC = array_append,
   STYPE = anyarray,
   INITCOND = '{}'
);
CREATE AGGREGATE

Joe

---

From: Bruce Momjian <[email-masked]>
Date: Mon, 10 Nov 2003 17:26:18 -0500 (EST)

OK, release notes updated to:

<listitem><para>Allow polymorphic PL/pgSQL functions (Tom, Joe)</para></listitem>
<listitem><para>Allow polymorphic SQL functions (Joe)</para>
   <para>
    Allow functions to accept arbitrary data types for input, and return arbitrary types.
   </para>
   </listitem>

Joe Conway wrote:
> TIP 9: the planner will ignore your desire to choose an index scan if your
>       joining column's datatypes do not match
>