public inbox for [email protected]
help / color / mirror / Atom feedFrom: Jacob Champion <[email protected]>
To: Jacob Burroughs <[email protected]>
Cc: Jelte Fennema-Nio <[email protected]>
Cc: Robert Haas <[email protected]>
Cc: Andrey M. Borodin <[email protected]>
Cc: PostgreSQL Hackers <[email protected]>
Subject: Re: libpq compression (part 3)
Date: Tue, 21 May 2024 12:42:57 -0700
Message-ID: <CAOYmi+mQGHm2wRO2wndAHTqFyM8iXfZ7YMBy993jWWMD2YKp5Q@mail.gmail.com> (raw)
In-Reply-To: <CACzsqT7pirbDjVPrYL4fciFDOkByXn+HkKHss-48_6mgMsY5gA@mail.gmail.com>
References: <CACzsqT4cJG0kaCbz24Sd=GAEgiQDpzU8yuD6vF25zo870+3M6g@mail.gmail.com>
<CA+TgmoZ0cq7SoiXWdv5A57gK-k3YXjGcDiRA+RvjCO976LD6iw@mail.gmail.com>
<CACzsqT5Y2cVES09bm4audEeh10bhsawSReeEgHwHA6NT2NV+BQ@mail.gmail.com>
<CAGECzQTPf3d4WP2B=t-kRQ3CQMzYSA6jneugkWqp7a3c1XYX0g@mail.gmail.com>
<CACzsqT5-7xfbz+Si35TBYHzerNX3XJVzAUH9AewQ+Pp13fYBoQ@mail.gmail.com>
<CA+TgmoY4LWvb2WYRT74UMaKZt+4g2NwCEOQ1KLe94EsphKWNTw@mail.gmail.com>
<CACzsqT6n0WNUGAK4R+Pn52+hAKWuinMqEjKnfkYkx9EroMwXeA@mail.gmail.com>
<CA+TgmoZy43rBVypyHujU-ZZuprv4Uh3zYCLO4Yzh2EjeR3A2AA@mail.gmail.com>
<CA+TgmoYiYC0Z57nctjGFeJD0yQi==imL37K5DA6wRWfu01h_ng@mail.gmail.com>
<CACzsqT7m+CYi8D=UYPTGHUqjwun==3puHuW8wEA81OyjFKHHEw@mail.gmail.com>
<CA+TgmoZ1eU35KwTZ6QPsfjMjpx4fkF8z5rVORfppc=Y+0ZHVVQ@mail.gmail.com>
<CACzsqT4een0jB1hH_vzwif-cA6OcLwbaju7rtUAaRe+BiVSUtw@mail.gmail.com>
<CA+TgmoaawjOaSUqW2Q53ezsBJRidXKHSmT-qqvahdNks1RNajA@mail.gmail.com>
<CACzsqT7S+Bk48xvGd9cvhyj0tHWNqzs_SPVs+rDNSsqfR7ymuw@mail.gmail.com>
<CACzsqT4-E4nYvS7fErmEsOcPVZTuGq4_y9C9iW7g=bwFfS3jgg@mail.gmail.com>
<CA+TgmoYu8EvvbR1amMmJDW_rPiSgC=11Qx2jJM1z-c0yaLR+Ag@mail.gmail.com>
<CACzsqT5dSZGzY5cmO88jiCd+T_B1xUCi1LWk6iXsd-Q9Xc=KXA@mail.gmail.com>
<CA+TgmobS=813B+7XZ+h-w2H8HZd0EAn-ug7HV2D7DF2FRTrTzQ@mail.gmail.com>
<CACzsqT5S4bAD7o5=X8+RHbwZcZzPipH6NG0Y953Yw5HZmVkeFw@mail.gmail.com>
<CA+Tgmobuj67e8z=6AdnrStvOtD2QSeTCRthKpjgeNkwJTa+kLQ@mail.gmail.com>
<CACzsqT6mhei=xiNwQ2JbVGs44bPg+S5KDit1LmiBzdbo=4NSLg@mail.gmail.com>
<CAOYmi+nLh2kj5=VCYQWJCe5feBPKNYhZ0Q1MaOoCTyoTA1sfLQ@mail.gmail.com>
<CAGECzQQ4iQ=bhtC4E_rVs6XYAgQMzcjvUr-rGRdKPkGT0VKqgA@mail.gmail.com>
<CAOYmi+=7J8+WuX5JnuhOHA9vYehbndUYaBW44u+a_x=nJgDw9Q@mail.gmail.com>
<CA+Tgmob5XzbKa7UaS0eKSv1ReopJ7XkVhBMS+qbLP91iqEQfnA@mail.gmail.com>
<CAOYmi+=Fyhdq7ePh9s-u9-hJ=iy9tnhWw-_zaN2a+uGYzfJrOQ@mail.gmail.com>
<CA+TgmoaNFFeyy8x7KKPUxn3VS8MixM7Y1kFjmcbYBqLCfoYjLQ@mail.gmail.com>
<CAOYmi+n6xQcA664DUDn5HsHTrT2u0hebfNYkgzJ=1J7zSOEx4A@mail.gmail.com>
<CA+TgmoYqD0zFyX_Zot5g12TNQUeMinJf-ri1B3t0GhG7kVXTvw@mail.gmail.com>
<[email protected]>
<CA+TgmoYX5hKOROc2Af5-ZGwaWxpY=r0WG=9Bc8xkDgRttXVLGQ@mail.gmail.com>
<CAOYmi+kA94o-VUmpQfuJVqf5zt5vMLu0Rjskb_iHE4W3O8OicA@mail.gmail.com>
<CAGECzQTMEry3HOqn_ajOZU4BnwAV1L67iSL9+2NQpwyFxm12tA@mail.gmail.com>
<CACzsqT6XbZ9bX=mgjHgv-h_6gZEhMB1rLbz0RT4=aWsoGaTvNw@mail.gmail.com>
<CAOYmi+=b_nyeUVn7BdzSSW85PTP1f405K4TBqRN6W881L=QSUQ@mail.gmail.com>
<CACzsqT7pirbDjVPrYL4fciFDOkByXn+HkKHss-48_6mgMsY5gA@mail.gmail.com>
On Tue, May 21, 2024 at 12:08 PM Jacob Burroughs
<[email protected]> wrote:
> I think I thought I was writing about something else when I wrote that
> :sigh:. I think what I really should have written was a version of
> the part below, which is that we use streaming decompression, only
> decompress 8kb at a time, and for pre-auth messages limit them to
> `PG_MAX_AUTH_TOKEN_LENGTH` (65535 bytes), which isn't really enough
> data to actually cause any real-world pain by needing to decompress vs
> the equivalent pain of sending invalid uncompressed auth packets.
Okay. So it sounds like your position is similar to Robert's from
earlier: prefer allowing unauthenticated compressed packets for
simplicity, as long as we think it's safe for the server. (Personally
I still think a client that compresses its password packets is doing
it wrong, and we could help them out by refusing that.)
> We own both the canonical client and server, so those are both covered
> here. I would think it would be the responsibility of any other
> system that maintains its own implementation of the postgres protocol
> and chooses to support the compression protocol to perform its own
> mitigations against potential compression security issues.
Sure, but if our official documentation is "here's an extremely
security-sensitive feature, figure it out!" then we've done a
disservice to the community.
> Should we
> put the fixed message size limits (that have de facto been part of the
> protocol since 2021, even if they weren't documented as such) into the
> protocol documentation?
Possibly? I don't know if the other PG-compatible implementations use
the same limits. It might be better to say "limits must exist".
> ( I don't really see how one could implement other tooling that used
> pg compression without using streaming compression, as the protocol
> never hands over a standalone blob of compressed data: all compressed
> data is always part of a stream, but even with streaming decompression
> you still need some kind of limits or you will just chew up memory.)
Well, that's a good point; I wasn't thinking about the streaming APIs
themselves. If the easiest way to implement decompression requires the
use of an API that shouts "hey, give me guardrails!", then that helps
quite a bit. I really need to look into the attack surface of the
three algorithms.
--Jacob
view thread (46+ messages)
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: libpq compression (part 3)
In-Reply-To: <CAOYmi+mQGHm2wRO2wndAHTqFyM8iXfZ7YMBy993jWWMD2YKp5Q@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox