From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
To: Rushabh Lathia <rushabh.lathia@gmail.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Andrey Borodin <x4mmm@yandex-team.ru>
Cc: Alvaro Herrera <alvherre@2ndquadrant.com>
Cc: Binguo Bao <djydewang@gmail.com>
Cc: Paul Ramsey <pramsey@cleverelephant.ca>
Cc: Pgsql Hackers <pgsql-hackers@postgresql.org>
Subject: Re: Optimize partial TOAST decompression
Date: Thu, 14 Nov 2019 14:00:55 +0100
Message-ID: <20191114130055.dkrjcveigeviil7n@development> (raw)
In-Reply-To: <CAGPqQf2a4+6BKnLC+-b0kEc-02nBfosV=bf8qDRQa+y30N+tNg@mail.gmail.com>
References: <E11354D1-6980-408D-AC4B-E147EB7CCDED@yandex-team.ru>
<20190930172951.gdedvexnf4d2wv5e@development>
<123EF56B-F8EC-4868-B49B-095795095E7A@yandex-team.ru>
<20191001100805.wnnuj73kuzfwzs56@development>
<20191001123420.uwzxtqr7qvleyxef@development>
<20191001131803.j6uin7nho7t6vxzy@development>
<4411.1569939037@sss.pgh.pa.us>
<20191001145758.rofbwvguwxwpdxyb@development>
<2093.1569944075@sss.pgh.pa.us>
<CAGPqQf2a4+6BKnLC+-b0kEc-02nBfosV=bf8qDRQa+y30N+tNg@mail.gmail.com>
On Thu, Nov 14, 2019 at 03:27:42PM +0530, Rushabh Lathia wrote:
>Today I noticed strange behaviour, consider the following test:
>
>postgres@126111=#create table foo ( a text );
>CREATE TABLE
>postgres@126111=#insert into foo values ( repeat('PostgreSQL is the
>world''s best database and leading by an Open Source Community.', 8000));
>INSERT 0 1
>
>postgres@126111=#select substring(a from 639921 for 81) from foo;
> substring
>-----------
>
>(1 row)
>
Hmmm. I think the issue is heap_tuple_untoast_attr_slice is using the
wrong way to determine compressed size in the VARATT_IS_EXTERNAL_ONDISK
branch. It does this
max_size = pglz_maximum_compressed_size(sliceoffset + slicelength,
TOAST_COMPRESS_SIZE(attr));
But for the example you've posted TOAST_COMPRESS_SIZE(attr) returns 10,
which is obviously bogus because the TOAST table contains ~75kB of data.
I think it should be doing this instead:
max_size = pglz_maximum_compressed_size(sliceoffset + slicelength,
toast_pointer.va_extsize);
At least that fixes it for me.
I wonder if this actually explains the crashes 540f3168091 was supposed
to fix, but it just masked them instead.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.cindex 47a03fa98b..b796406239 100644--- a/src/backend/access/common/detoast.c+++ b/src/backend/access/common/detoast.c@@ -233,7 +233,7 @@ heap_tuple_untoast_attr_slice(struct varlena *attr,
* of a given length (after decompression).
*/
max_size = pglz_maximum_compressed_size(sliceoffset + slicelength,
- TOAST_COMPRESS_SIZE(attr));+ toast_pointer.va_extsize);
/*
* Fetch enough compressed slices (compressed marker will get set
Attachments:
[text/plain] detoasting.patch (567B, ../20191114130055.dkrjcveigeviil7n@development/2-detoasting.patch)
download | inline diff:diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.cindex 47a03fa98b..b796406239 100644--- a/src/backend/access/common/detoast.c+++ b/src/backend/access/common/detoast.c@@ -233,7 +233,7 @@ heap_tuple_untoast_attr_slice(struct varlena *attr,
* of a given length (after decompression).
*/
max_size = pglz_maximum_compressed_size(sliceoffset + slicelength,
- TOAST_COMPRESS_SIZE(attr));+ toast_pointer.va_extsize);
/*
* Fetch enough compressed slices (compressed marker will get set
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: tomas.vondra@2ndquadrant.com, rushabh.lathia@gmail.com, tgl@sss.pgh.pa.us, x4mmm@yandex-team.ru, alvherre@2ndquadrant.com, djydewang@gmail.com, pramsey@cleverelephant.ca
Subject: Re: Optimize partial TOAST decompression
In-Reply-To: <20191114130055.dkrjcveigeviil7n@development>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox