Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1krmJg-0000SW-Ah for pgsql-hackers@arkaria.postgresql.org; Tue, 22 Dec 2020 18:23:24 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1krmJf-0003Zq-3S for pgsql-hackers@arkaria.postgresql.org; Tue, 22 Dec 2020 18:23:23 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1krmJe-0003Zj-Sh for pgsql-hackers@lists.postgresql.org; Tue, 22 Dec 2020 18:23:22 +0000 Received: from aart.rice.edu ([168.7.56.48]) by magus.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1krmJa-0001BS-Vb for pgsql-hackers@lists.postgresql.org; Tue, 22 Dec 2020 18:23:22 +0000 Received: by aart.rice.edu (Postfix, from userid 18612) id C0D6340EE818; Tue, 22 Dec 2020 12:23:16 -0600 (CST) Date: Tue, 22 Dec 2020 12:23:16 -0600 From: Kenneth Marshall To: Tomas Vondra Cc: Robert Haas , Daniil Zakhlystov , Konstantin Knizhnik , Denis Smirnov , Andrey Borodin , PostgreSQL Hackers Subject: Re: libpq compression Message-ID: <20201222182316.GG16082@aart.rice.edu> References: <6811D196-E2FB-40CC-B1C9-F19427FBF675@yandex-team.ru> <93027ee4-fefa-a8ca-2cc7-245c912bf824@postgrespro.ru> <62527092-16BD-479F-B503-FA527AF3B0C2@yandex-team.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk On Tue, Dec 22, 2020 at 07:15:23PM +0100, Tomas Vondra wrote: > > > On 12/22/20 6:56 PM, Robert Haas wrote: > >On Tue, Dec 22, 2020 at 6:24 AM Daniil Zakhlystov > > wrote: > >>When using bidirectional compression, Postgres resource usage correlates with the selected compression level. For example, here is the Postgresql application memory usage: > >> > >>No compression - 1.2 GiB > >> > >>ZSTD > >>zstd:1 - 1.4 GiB > >>zstd:7 - 4.0 GiB > >>zstd:13 - 17.7 GiB > >>zstd:19 - 56.3 GiB > >>zstd:20 - 109.8 GiB - did not succeed > >>zstd:21, zstd:22 > 140 GiB > >>Postgres process crashes (out of memory) > > > >Good grief. So, suppose we add compression and support zstd. Then, can > >unprivileged user capable of connecting to the database can negotiate > >for zstd level 1 and then choose to actually send data compressed at > >zstd level 22, crashing the server if it doesn't have a crapton of > >memory? Honestly, I wouldn't blame somebody for filing a CVE if we > >allowed that sort of thing to happen. I'm not sure what the solution > >is, but we can't leave a way for a malicious client to consume 140GB > >of memory on the server *per connection*. I assumed decompression > >memory was going to measured in kB or MB, not GB. Honestly, even at > >say L7, if you've got max_connections=100 and a user who wants to make > >trouble, you have a really big problem. > > > >Perhaps I'm being too pessimistic here, but man that's a lot of memory. > > > > Maybe I'm just confused, but my assumption was this means there's a > memory leak somewhere - that we're not resetting/freeing some piece > of memory, or so. Why would zstd need so much memory? It seems like > a pretty serious disadvantage, so how could it become so popular? > > > regards > Hi, It looks like the space needed for decompression is between 1kb and 3.75tb: https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#window_descriptor Sheesh! Looks like it would definitely need to be bounded to control resource use. Regards, Ken