From: Steve Crawford <scrawford@pinpointresearch.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
To: pgsql-hackers@postgreSQL.org
To: pgsql-interfaces@postgreSQL.org
Subject: Re: More thoughts about FE/BE protocol
Date: Thu, 10 Apr 2003 07:55:01 -0800
Message-ID: <20030410155501.98D6C103C9@polaris.pinpointresearch.com> (raw)
In-Reply-To: <10478.1049927451@sss.pgh.pa.us>
References: <10478.1049927451@sss.pgh.pa.us>
What would be the recovery/re-sync mechanism for those cases where the
message is, either accidentally or maliciously, longer or shorter than the
described length?
Without proper timeout/recovery mechanisms a too-short message could cause
the receiver to effectively hang. A too long message could cause overflows,
loss of sync or other problems if the receiver attempts to interpret the
extra data as the next message header.
Cheers,
Steve
On Wednesday 09 April 2003 3:30 pm, Tom Lane wrote:
> I've been thinking some more about the FE/BE protocol redesign,
> specifically the desire to ensure that we can recover from error
> conditions without losing synchronization. The fact that the existing
> protocol doesn't do very well at this shows up in several places:
> * having to drop and restart the connection after a COPY error
> * fastpath function calls also lose sync if there's an error
> * libpq gets terribly confused if it runs out of memory for
> a query result
>
> I'm coming around to the idea that the cleanest solution is to require
> *all* protocol messages, in both directions, to have an initial length
> word. That is, the general message format would look like
> <message type> 1 byte
> <payload length> number of following bytes (4 bytes MSB-first)
> ... message data as needed ...
>
> The advantage of doing it this way is that the recipient can absorb the
> whole message before starting to parse the contents; then, errors detected
> while processing the message contents don't cause us to lose protocol
> synchronization. Also, even if the message is so large as to run the
> recipient out of memory, it can still use the <payload length> to count
> off the number of bytes it has to drop before looking for another message.
> This would make it quite a bit easier for libpq to cope with
> out-of-memory, as an example.
>
> These advantages seem enough to me to justify adding three or four bytes
> to the length of each message. Does anyone have a problem with that?
>
>
> A related point is that I had been thinking of the new "extended query"
> facility (separate PARSE/BIND/DESCRIBE/EXECUTE steps) in terms of
> sending just one big message to the backend per interaction cycle, with
> the processing-step commands appearing as fields within that message.
> But putting a length word in front would effectively require the
> frontend to marshal the whole sequence before sending any of it.
> It seems better to send each of the processing-step commands as an
> independent message. To do that, we need to introduce an additional
> processing step, call it SYNC, that substitutes for the functionality
> associated with the overall message boundary in the other way.
> Specifically:
> * ReadyForQuery (Z) is sent in response to SYNC.
> * If no BEGIN has been issued, SYNC is the point at which an implicit
> COMMIT is done.
> * After an error occurs in an extended-query command, the backend reads
> and discards messages until it finds a SYNC, then issues ReadyForQuery
> and resumes processing messages. This allows the frontend to be
> certain what has been processed and what hasn't.
>
> Comments?
>
> regards, tom lane
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-interfaces@postgresql.org
Cc: scrawford@pinpointresearch.com, tgl@sss.pgh.pa.us, pgsql-hackers@postgreSQL.org
Subject: Re: More thoughts about FE/BE protocol
In-Reply-To: <20030410155501.98D6C103C9@polaris.pinpointresearch.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox