pg.ddx.io pgsql-hackers@postgresql.org mailing list archive help / color / mirror / Atom feed From: Tomas Vondra <tomas.vondra@enterprisedb.com>
To: Robert Haas <robertmhaas@gmail.com>
To: Daniil Zakhlystov <usernamedt@yandex-team.ru>
Cc: Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
Cc: Denis Smirnov <sd@arenadata.io>
Cc: Andrey Borodin <x4mmm@yandex-team.ru>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: libpq compression
Date: Tue, 22 Dec 2020 19:15:23 +0100
Message-ID: <d889e3a2-d3f4-3f8e-aa81-c142e32ddf57@enterprisedb.com> (raw )
In-Reply-To: <CA+TgmoZbVH4R5JA9_pyTSdnFTq-w2B1JvomeD+r0oo0xp1DPtQ@mail.gmail.com >
References: <aad16e41-b3f9-e89d-fa57-fb4c694bec25@postgrespro.ru >
<612f227d-cb27-9238-5922-1500f8b2eb96@postgrespro.ru >
<160623927937.7563.15974279758478581796.pgcf@coridan.postgresql.org >
<CA+TgmoY5AMq8auemc0g3geCpVDeRnawW1A-jUeqT-5cE-_2MiA@mail.gmail.com >
<6811D196-E2FB-40CC-B1C9-F19427FBF675@yandex-team.ru >
<CA+TgmoZMTLydjP3iWBir20OrSXTugxS2CXC1WP1nWM5=Zqz0Nw@mail.gmail.com >
<A1D0D9A4-1FC4-4311-83C7-EEA90749EE07@yandex-team.ru >
<CA+TgmoZrM9T_W9YR-vLYZXj-vC_+F-DER65A87+pDXRwUFjJXg@mail.gmail.com >
<D5354E7A-3B9F-4D32-B3AB-F65058D36500@yandex-team.ru >
<CCAB1F57-A71D-4DEC-9A9C-F325B44BAC29@arenadata.io >
<93027ee4-fefa-a8ca-2cc7-245c912bf824@postgrespro.ru >
<62527092-16BD-479F-B503-FA527AF3B0C2@yandex-team.ru >
<CA+TgmoZbVH4R5JA9_pyTSdnFTq-w2B1JvomeD+r0oo0xp1DPtQ@mail.gmail.com >
On 12/22/20 6:56 PM, Robert Haas wrote:
> On Tue, Dec 22, 2020 at 6:24 AM Daniil Zakhlystov
> <usernamedt@yandex-team.ru> wrote:
>> When using bidirectional compression, Postgres resource usage correlates with the selected compression level. For example, here is the Postgresql application memory usage:
>>
>> No compression - 1.2 GiB
>>
>> ZSTD
>> zstd:1 - 1.4 GiB
>> zstd:7 - 4.0 GiB
>> zstd:13 - 17.7 GiB
>> zstd:19 - 56.3 GiB
>> zstd:20 - 109.8 GiB - did not succeed
>> zstd:21, zstd:22 > 140 GiB
>> Postgres process crashes (out of memory)
>
> Good grief. So, suppose we add compression and support zstd. Then, can
> unprivileged user capable of connecting to the database can negotiate
> for zstd level 1 and then choose to actually send data compressed at
> zstd level 22, crashing the server if it doesn't have a crapton of
> memory? Honestly, I wouldn't blame somebody for filing a CVE if we
> allowed that sort of thing to happen. I'm not sure what the solution
> is, but we can't leave a way for a malicious client to consume 140GB
> of memory on the server *per connection*. I assumed decompression
> memory was going to measured in kB or MB, not GB. Honestly, even at
> say L7, if you've got max_connections=100 and a user who wants to make
> trouble, you have a really big problem.
>
> Perhaps I'm being too pessimistic here, but man that's a lot of memory.
>
Maybe I'm just confused, but my assumption was this means there's a
memory leak somewhere - that we're not resetting/freeing some piece of
memory, or so. Why would zstd need so much memory? It seems like a
pretty serious disadvantage, so how could it become so popular?
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
view thread (289+ messages) latest in thread
Message-ID: <d889e3a2-d3f4-3f8e-aa81-c142e32ddf57@enterprisedb.com>
Permalink: ../d889e3a2-d3f4-3f8e-aa81-c142e32ddf57@enterprisedb.com/
Also on: postgresql.org/message-id/d889e3a2-d3f4-3f8e-aa81-c142e32ddf57@enterprisedb.com
copy link · copy postgr.es
reply Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: tomas.vondra@enterprisedb.com, robertmhaas@gmail.com, usernamedt@yandex-team.ru, k.knizhnik@postgrespro.ru, sd@arenadata.io, x4mmm@yandex-team.ru, pgsql-hackers@lists.postgresql.org
Subject: Re: libpq compression
In-Reply-To: <d889e3a2-d3f4-3f8e-aa81-c142e32ddf57@enterprisedb.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox