From: Andy Fan <zhihuifan1213@163.com>
To: Nikita Malakhov <hukutoc@gmail.com>
Cc: Tomas Vondra <tomas.vondra@enterprisedb.com>
Cc: Michael Zhilin <m.zhilin@postgrespro.ru>
Cc: Peter Smith <smithpb2250@gmail.com>
Cc: vignesh C <vignesh21@gmail.com>
Cc: Matthias van de Meent <boekewurm+postgres@gmail.com>
Cc: pgsql-hackers@lists.postgresql.org
Subject: Re: Shared detoast Datum proposal
Date: Mon, 26 Feb 2024 21:22:39 +0800
Message-ID: <87frxf1gld.fsf@163.com> (raw)
In-Reply-To: <CAN-LCVMU_AzYNDQ-MgX3xc5qCnhTgO1LWKUCygOu3neBM-y94g@mail.gmail.com>
References: <87il4jrk1l.fsf@163.com>
<87h6jxw41k.fsf@163.com>
<CALDaNm2EtgXGd+w81ZjTZBD_rLSURyGqXRoz7EFfhxEneOmWig@mail.gmail.com>
<87sf3andy2.fsf@163.com>
<87wmskm9qu.fsf@163.com>
<CAHut+PsehMtwQ4uE9VS2C7jB1yT0SZVOPU22u1=ma1K7dKFUow@mail.gmail.com>
<87v87kvboq.fsf@163.com>
<71321d83-391b-473b-aaf6-618c7f682b96@postgrespro.ru>
<87r0i7vos3.fsf@163.com>
<874je3hcwn.fsf@163.com>
<c066e988-38b9-460f-9e9d-aedd2482bff2@enterprisedb.com>
<8734tmg88e.fsf@163.com>
<7f4c8673-4c5d-4dbf-b8e5-0ec11cd2fba0@enterprisedb.com>
<87h6i2djf9.fsf@163.com>
<87jzms0xza.fsf@163.com>
<CAN-LCVMU_AzYNDQ-MgX3xc5qCnhTgO1LWKUCygOu3neBM-y94g@mail.gmail.com>
Hi,
> When we store and detoast large values, say, 1Gb - that's a very likely scenario,
> we have such cases from prod systems - we would end up in using a lot of shared
> memory to keep these values alive, only to discard them later.
First the feature can make sure if the original expression will not
detoast it, this feature will not detoast it as well. Based on this, if
there is a 1Gb datum, in the past, it took 1GB memory as well, but
it may be discarded quicker than the patched version. but patched
version will *not* keep it for too long time since once the
slot->tts_values[*] is invalidated, the memory will be released
immedately.
For example:
create table t(a text, b int);
insert into t select '1-gb-length-text' from generate_series(1, 100);
select * from t where a like '%gb%';
The patched version will take 1gb extra memory only.
Are you worried about this case or some other cases?
> Also, toasted values
> are not always being used immediately and as a whole, i.e. jsonb values are fully
> detoasted (we're working on this right now) to extract the smallest value from
> big json, and these values are not worth keeping in memory. For text values too,
> we often do not need the whole value to be detoasted.
I'm not sure how often this is true, espeically you metioned text data
type. I can't image why people acquire a piece of text and how can we
design a toasting system to fulfill such case without detoast the whole
as the first step. But for the jsonb or array data type, I think it is
more often. However if we design toasting like that, I'm not sure if it
would slow down other user case. for example detoasting the whole piece
use case. I am justing feeling both way has its user case, kind of heap
store and columna store.
FWIW, as I shown in the test case, this feature is not only helpful for
big datum, it is also helpful for small toast datum.
--
Best Regards
Andy Fan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: zhihuifan1213@163.com, hukutoc@gmail.com, tomas.vondra@enterprisedb.com, m.zhilin@postgrespro.ru, smithpb2250@gmail.com, vignesh21@gmail.com, boekewurm+postgres@gmail.com, pgsql-hackers@lists.postgresql.org
Subject: Re: Shared detoast Datum proposal
In-Reply-To: <87frxf1gld.fsf@163.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox