Content-Type: text/plain;
  charset="iso-8859-1"
From: Steve Crawford <scrawford@pinpointresearch.com>
Organization: Pinpoint Research
To: Tom Lane <tgl@sss.pgh.pa.us>, pgsql-hackers@postgreSQL.org,
	pgsql-interfaces@postgreSQL.org
Subject: Re: More thoughts about FE/BE protocol
Date: Thu, 10 Apr 2003 07:55:01 -0800
References: <10478.1049927451@sss.pgh.pa.us>
In-Reply-To: <10478.1049927451@sss.pgh.pa.us>
MIME-Version: 1.0
Message-Id: <20030410155501.98D6C103C9@polaris.pinpointresearch.com>
Content-Transfer-Encoding: quoted-printable

What would be the recovery/re-sync mechanism for those cases where the=20
message is, either accidentally or maliciously, longer or shorter than the=
=20
described length?

Without proper timeout/recovery mechanisms a too-short message could cause=
=20
the receiver to effectively hang. A too long message could cause overflows,=
=20
loss of sync or other problems if the receiver attempts to interpret the=20
extra data as the next message header.

Cheers,
Steve

On Wednesday 09 April 2003 3:30 pm, Tom Lane wrote:
> I've been thinking some more about the FE/BE protocol redesign,
> specifically the desire to ensure that we can recover from error
> conditions without losing synchronization.  The fact that the existing
> protocol doesn't do very well at this shows up in several places:
> 	* having to drop and restart the connection after a COPY error
> 	* fastpath function calls also lose sync if there's an error
> 	* libpq gets terribly confused if it runs out of memory for
> 	  a query result
>
> I'm coming around to the idea that the cleanest solution is to require
> *all* protocol messages, in both directions, to have an initial length
> word.  That is, the general message format would look like
> 	<message type>		1 byte
> 	<payload length>	number of following bytes (4 bytes MSB-first)
> 	... message data as needed ...
>
> The advantage of doing it this way is that the recipient can absorb the
> whole message before starting to parse the contents; then, errors detected
> while processing the message contents don't cause us to lose protocol
> synchronization.  Also, even if the message is so large as to run the
> recipient out of memory, it can still use the <payload length> to count
> off the number of bytes it has to drop before looking for another message.
> This would make it quite a bit easier for libpq to cope with
> out-of-memory, as an example.
>
> These advantages seem enough to me to justify adding three or four bytes
> to the length of each message.  Does anyone have a problem with that?
>
>
> A related point is that I had been thinking of the new "extended query"
> facility (separate PARSE/BIND/DESCRIBE/EXECUTE steps) in terms of
> sending just one big message to the backend per interaction cycle, with
> the processing-step commands appearing as fields within that message.
> But putting a length word in front would effectively require the
> frontend to marshal the whole sequence before sending any of it.
> It seems better to send each of the processing-step commands as an
> independent message.  To do that, we need to introduce an additional
> processing step, call it SYNC, that substitutes for the functionality
> associated with the overall message boundary in the other way.
> Specifically:
> * ReadyForQuery (Z) is sent in response to SYNC.
> * If no BEGIN has been issued, SYNC is the point at which an implicit
>   COMMIT is done.
> * After an error occurs in an extended-query command, the backend reads
>   and discards messages until it finds a SYNC, then issues ReadyForQuery
>   and resumes processing messages.  This allows the frontend to be
>   certain what has been processed and what hasn't.
>
> Comments?
>
> 			regards, tom lane
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org