public inbox for [email protected]
help / color / mirror / Atom feedRe: patch: function xmltable
72+ messages / 6 participants
[nested] [flat]
* Re: patch: function xmltable
@ 2017-03-02 00:12 Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 72+ messages in thread
From: Alvaro Herrera @ 2017-03-02 00:12 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Craig Ringer <[email protected]>; pgsql-hackers
I've been giving this a look. I started by tweaking the docs once
again, and while verifying that the example works as expected, I
replayed what I have in sgml:
... begin SGML paste ...
<para>
For example, given the following XML document:
<screen><![CDATA[
<ROWS>
<ROW id="1">
<COUNTRY_ID>AU</COUNTRY_ID>
<COUNTRY_NAME>Australia</COUNTRY_NAME>
</ROW>
<ROW id="5">
<COUNTRY_ID>JP</COUNTRY_ID>
<COUNTRY_NAME>Japan</COUNTRY_NAME>
<PREMIER_NAME>Sinzo Abe</PREMIER_NAME>
</ROW>
<ROW id="6">
<COUNTRY_ID>SG</COUNTRY_ID>
<COUNTRY_NAME>Singapore</COUNTRY_NAME>
<SIZE unit="km">791</SIZE>
</ROW>
</ROWS>
]]></screen>
the following query produces the result shown below:
<screen><![CDATA[
SELECT xmltable.*
FROM (SELECT data FROM xmldata) x,
LATERAL xmltable('//ROWS/ROW'
PASSING data
COLUMNS id int PATH '@id',
ordinality FOR ORDINALITY,
country_name text PATH 'COUNTRY_NAME',
country_id text PATH 'COUNTRY_ID',
size float PATH 'SIZE[@unit = "km"]/text()',
unit text PATH 'SIZE/@unit',
premier_name text PATH 'PREMIER_NAME' DEFAULT 'not specified');
... end SGML paste ...
But the query doesn't actually return a table, but instead it fails with
this error:
ERROR: invalid input syntax for type double precision: ""
This is because of the "size" column (if I remove SIZE from the COLUMNS
clause, the query returns correctly). Apparently, for the rows where
SIZE is not given, we try to inssert an empty string instead of a NULL
value, which is what I expected.
I'm using your v44 code, but trimmed both the XML document used in SGML
as well as modified the query slightly to show additional features. But
those changes should not cause the above error ...
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: patch: function xmltable
@ 2017-03-02 07:04 Pavel Stehule <[email protected]>
parent: Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 72+ messages in thread
From: Pavel Stehule @ 2017-03-02 07:04 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Craig Ringer <[email protected]>; pgsql-hackers
Hi
2017-03-02 1:12 GMT+01:00 Alvaro Herrera <[email protected]>:
>
> I've been giving this a look. I started by tweaking the docs once
> again, and while verifying that the example works as expected, I
> replayed what I have in sgml:
>
> ... begin SGML paste ...
> <para>
> For example, given the following XML document:
> <screen><![CDATA[
> <ROWS>
> <ROW id="1">
> <COUNTRY_ID>AU</COUNTRY_ID>
> <COUNTRY_NAME>Australia</COUNTRY_NAME>
> </ROW>
> <ROW id="5">
> <COUNTRY_ID>JP</COUNTRY_ID>
> <COUNTRY_NAME>Japan</COUNTRY_NAME>
> <PREMIER_NAME>Sinzo Abe</PREMIER_NAME>
> </ROW>
> <ROW id="6">
> <COUNTRY_ID>SG</COUNTRY_ID>
> <COUNTRY_NAME>Singapore</COUNTRY_NAME>
> <SIZE unit="km">791</SIZE>
> </ROW>
> </ROWS>
> ]]></screen>
>
> the following query produces the result shown below:
>
> <screen><![CDATA[
> SELECT xmltable.*
> FROM (SELECT data FROM xmldata) x,
> LATERAL xmltable('//ROWS/ROW'
> PASSING data
> COLUMNS id int PATH '@id',
> ordinality FOR ORDINALITY,
> country_name text PATH 'COUNTRY_NAME',
> country_id text PATH 'COUNTRY_ID',
> size float PATH 'SIZE[@unit =
> "km"]/text()',
> unit text PATH 'SIZE/@unit',
> premier_name text PATH 'PREMIER_NAME'
> DEFAULT 'not specified');
> ... end SGML paste ...
>
>
> But the query doesn't actually return a table, but instead it fails with
> this error:
> ERROR: invalid input syntax for type double precision: ""
> This is because of the "size" column (if I remove SIZE from the COLUMNS
> clause, the query returns correctly). Apparently, for the rows where
> SIZE is not given, we try to inssert an empty string instead of a NULL
> value, which is what I expected.
>
> I'm using your v44 code, but trimmed both the XML document used in SGML
> as well as modified the query slightly to show additional features. But
> those changes should not cause the above error ...
>
The example in doc is obsolete. Following example works without problems.
SELECT xmltable.*
FROM (SELECT data FROM xmldata) x,
LATERAL xmltable('//ROWS/ROW'
PASSING data
COLUMNS id int PATH '@id',
ordinality FOR ORDINALITY,
country_name text PATH 'COUNTRY_NAME',
country_id text PATH 'COUNTRY_ID',
size float PATH 'SIZE[@unit = "km"]',
unit text PATH 'SIZE/@unit',
premier_name text PATH 'PREMIER_NAME'
DEFAULT 'not specified');
It is related to older variants of this patch, where I explicitly mapped
empty strings to NULL.
Now, I don't do it - I use libxml2 result with following mapping
No tag ... NULL
empty tag ... empty string
Important question is about mapping empty tags to Postgres. I prefer
current behave, because I have a possibility to differ between these states
on application level. If we returns NULL for empty tag, then there will not
be possible detect if XML has tag (although empty) or not. The change is
simple - just one row - but I am thinking so current behave is better.
There is possible risk of using /text() somewhere - it enforce a empty tag
with all negative impacts.
I prefer to fix doc in conformance with regress tests and append note about
mapping these corner cases from XML to relations.
What do you think about it?
Regards
Pavel
> --
> Álvaro Herrera https://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: patch: function xmltable
@ 2017-03-02 08:13 Pavel Stehule <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 72+ messages in thread
From: Pavel Stehule @ 2017-03-02 08:13 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Craig Ringer <[email protected]>; pgsql-hackers
2017-03-02 8:04 GMT+01:00 Pavel Stehule <[email protected]>:
> Hi
>
> 2017-03-02 1:12 GMT+01:00 Alvaro Herrera <[email protected]>:
>
>>
>> I've been giving this a look. I started by tweaking the docs once
>> again, and while verifying that the example works as expected, I
>> replayed what I have in sgml:
>>
>> ... begin SGML paste ...
>> <para>
>> For example, given the following XML document:
>> <screen><![CDATA[
>> <ROWS>
>> <ROW id="1">
>> <COUNTRY_ID>AU</COUNTRY_ID>
>> <COUNTRY_NAME>Australia</COUNTRY_NAME>
>> </ROW>
>> <ROW id="5">
>> <COUNTRY_ID>JP</COUNTRY_ID>
>> <COUNTRY_NAME>Japan</COUNTRY_NAME>
>> <PREMIER_NAME>Sinzo Abe</PREMIER_NAME>
>> </ROW>
>> <ROW id="6">
>> <COUNTRY_ID>SG</COUNTRY_ID>
>> <COUNTRY_NAME>Singapore</COUNTRY_NAME>
>> <SIZE unit="km">791</SIZE>
>> </ROW>
>> </ROWS>
>> ]]></screen>
>>
>> the following query produces the result shown below:
>>
>> <screen><![CDATA[
>> SELECT xmltable.*
>> FROM (SELECT data FROM xmldata) x,
>> LATERAL xmltable('//ROWS/ROW'
>> PASSING data
>> COLUMNS id int PATH '@id',
>> ordinality FOR ORDINALITY,
>> country_name text PATH 'COUNTRY_NAME',
>> country_id text PATH 'COUNTRY_ID',
>> size float PATH 'SIZE[@unit =
>> "km"]/text()',
>> unit text PATH 'SIZE/@unit',
>> premier_name text PATH 'PREMIER_NAME'
>> DEFAULT 'not specified');
>> ... end SGML paste ...
>>
>>
>> But the query doesn't actually return a table, but instead it fails with
>> this error:
>> ERROR: invalid input syntax for type double precision: ""
>> This is because of the "size" column (if I remove SIZE from the COLUMNS
>> clause, the query returns correctly). Apparently, for the rows where
>> SIZE is not given, we try to inssert an empty string instead of a NULL
>> value, which is what I expected.
>>
>> I'm using your v44 code, but trimmed both the XML document used in SGML
>> as well as modified the query slightly to show additional features. But
>> those changes should not cause the above error ...
>>
>
> The example in doc is obsolete. Following example works without problems.
>
> SELECT xmltable.*
>
> FROM (SELECT data FROM xmldata) x,
> LATERAL xmltable('//ROWS/ROW'
> PASSING data
> COLUMNS id int PATH '@id',
> ordinality FOR ORDINALITY,
> country_name text PATH 'COUNTRY_NAME',
> country_id text PATH 'COUNTRY_ID',
> size float PATH 'SIZE[@unit = "km"]',
> unit text PATH 'SIZE/@unit',
> premier_name text PATH 'PREMIER_NAME' DEFAULT 'not specified');
>
>
> It is related to older variants of this patch, where I explicitly mapped
> empty strings to NULL.
>
> Now, I don't do it - I use libxml2 result with following mapping
>
> No tag ... NULL
> empty tag ... empty string
>
> Important question is about mapping empty tags to Postgres. I prefer
> current behave, because I have a possibility to differ between these states
> on application level. If we returns NULL for empty tag, then there will not
> be possible detect if XML has tag (although empty) or not. The change is
> simple - just one row - but I am thinking so current behave is better.
> There is possible risk of using /text() somewhere - it enforce a empty tag
> with all negative impacts.
>
> I prefer to fix doc in conformance with regress tests and append note
> about mapping these corner cases from XML to relations.
>
> What do you think about it?
>
It is documented already
"If the <literal>PATH</> matches an empty tag the result is an empty string"
Attached new patch
cleaned documentation
regress tests is more robust
appended comment in src related to generating empty string for empty tag
Regards
Pavel
>
> Regards
>
> Pavel
>
>
>
>
>
>
>
>
>
>> --
>> Álvaro Herrera https://www.2ndQuadrant.com/
>> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>>
>
>
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[application/x-gzip] xmltable-45.patch.gz (33.1K, ../../CAFj8pRDmxH6xZwnFGKAcKKP=RFys+f3EC4SHHORi2=ud=tFBpg@mail.gmail.com/3-xmltable-45.patch.gz)
download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: patch: function xmltable
@ 2017-03-02 17:13 Alvaro Herrera <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 72+ messages in thread
From: Alvaro Herrera @ 2017-03-02 17:13 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Craig Ringer <[email protected]>; pgsql-hackers
Pavel Stehule wrote:
> It is documented already
>
> "If the <literal>PATH</> matches an empty tag the result is an empty string"
Hmm, okay. But what we have here is not an empty tag, but a tag that is
completely missing. I don't think those two cases should be treated in
the same way ...
> Attached new patch
>
> cleaned documentation
> regress tests is more robust
> appended comment in src related to generating empty string for empty tag
Thanks, I incorporated those changes. Here's v46. I rewrote the
documentation, and fixed a couple of incorrectly copied&pasted comments
in the new executor code; I think that one looks good. In the future we
could rewrite it to avoid the need for a tuplestore, but I think the
current approach is good enough for a pg10 implementation.
Barring serious problems, I intend to commit this later today.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[application/x-gunzip] xmltable-46.patch.gz (36.8K, ../../[email protected]/2-xmltable-46.patch.gz)
download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: patch: function xmltable
@ 2017-03-02 17:46 Pavel Stehule <[email protected]>
parent: Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 72+ messages in thread
From: Pavel Stehule @ 2017-03-02 17:46 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: pgsql-hackers; Craig Ringer <[email protected]>
Dne 2. 3. 2017 18:14 napsal uživatel "Alvaro Herrera" <
[email protected]>:
Pavel Stehule wrote:
> It is documented already
>
> "If the <literal>PATH</> matches an empty tag the result is an empty
string"
Hmm, okay. But what we have here is not an empty tag, but a tag that is
completely missing. I don't think those two cases should be treated in
the same way ...
this information is not propagated from libxml2.
> Attached new patch
>
> cleaned documentation
> regress tests is more robust
> appended comment in src related to generating empty string for empty tag
Thanks, I incorporated those changes. Here's v46. I rewrote the
documentation, and fixed a couple of incorrectly copied&pasted comments
in the new executor code; I think that one looks good. In the future we
could rewrite it to avoid the need for a tuplestore, but I think the
current approach is good enough for a pg10 implementation.
Barring serious problems, I intend to commit this later today.
thank you very much
regards
Pavel
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: patch: function xmltable
@ 2017-03-02 18:32 Alvaro Herrera <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 72+ messages in thread
From: Alvaro Herrera @ 2017-03-02 18:32 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: pgsql-hackers; Craig Ringer <[email protected]>
So in the old (non-executor-node) implementation, you could attach WITH
ORDINALITY to the xmltable expression and it would count the output
rows, regardless of which XML document it comes from. With the new
implementation, the grammar no longer accepts it. To count output rows,
you still need to use row_number(). Maybe this is okay. This is the
example from the docs, and I add another XML document with two more rows
for xmltable. Look at the three numbering columns ...
CREATE TABLE xmldata AS SELECT
xml $$
<ROWS>
<ROW id="1">
<COUNTRY_ID>AU</COUNTRY_ID>
<COUNTRY_NAME>Australia</COUNTRY_NAME>
</ROW>
<ROW id="5">
<COUNTRY_ID>JP</COUNTRY_ID>
<COUNTRY_NAME>Japan</COUNTRY_NAME>
<PREMIER_NAME>Shinzo Abe</PREMIER_NAME>
<SIZE unit="sq_mi">145935</SIZE>
</ROW>
<ROW id="6">
<COUNTRY_ID>SG</COUNTRY_ID>
<COUNTRY_NAME>Singapore</COUNTRY_NAME>
<SIZE unit="sq_km">697</SIZE>
</ROW>
</ROWS>
$$ AS data;
insert into xmldata values ($$
<ROWS><ROW id="2"><COUNTRY_ID>CL</COUNTRY_ID><COUNTRY_NAME>Chile</COUNTRY_NAME></ROW>
<ROW id="3"><COUNTRY_ID>AR</COUNTRY_ID><COUNTRY_NAME>Argentina</COUNTRY_NAME></ROW></ROWS>$$);
SELECT ROW_NUMBER() OVER (), xmltable.*
FROM xmldata,
XMLTABLE('//ROWS/ROW'
PASSING data
COLUMNS id int PATH '@id',
ordinality FOR ORDINALITY,
"COUNTRY_NAME" text,
country_id text PATH 'COUNTRY_ID',
size_sq_km float PATH 'SIZE[@unit = "sq_km"]',
size_other text PATH
'concat(SIZE[@unit!="sq_km"], " ", SIZE[@unit!="sq_km"]/@unit)',
premier_name text PATH 'PREMIER_NAME' DEFAULT 'not specified')
;
row_number │ id │ ordinality │ COUNTRY_NAME │ country_id │ size_sq_km │ size_other │ premier_name
────────────┼────┼────────────┼──────────────┼────────────┼────────────┼──────────────┼───────────────
1 │ 1 │ 1 │ Australia │ AU │ │ │ not specified
2 │ 5 │ 2 │ Japan │ JP │ │ 145935 sq_mi │ Shinzo Abe
3 │ 6 │ 3 │ Singapore │ SG │ 697 │ │ not specified
4 │ 2 │ 1 │ Chile │ CL │ │ │ not specified
5 │ 3 │ 2 │ Argentina │ AR │ │ │ not specified
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: patch: function xmltable
@ 2017-03-02 19:17 Pavel Stehule <[email protected]>
parent: Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 72+ messages in thread
From: Pavel Stehule @ 2017-03-02 19:17 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: pgsql-hackers; Craig Ringer <[email protected]>
2017-03-02 19:32 GMT+01:00 Alvaro Herrera <[email protected]>:
> So in the old (non-executor-node) implementation, you could attach WITH
> ORDINALITY to the xmltable expression and it would count the output
> rows, regardless of which XML document it comes from. With the new
> implementation, the grammar no longer accepts it. To count output rows,
> you still need to use row_number(). Maybe this is okay. This is the
> example from the docs, and I add another XML document with two more rows
> for xmltable. Look at the three numbering columns ...
>
It is expected - now tablefunc are not special case of SRF, so it lost all
SRF functionality. It is not critical lost - it supports internally FOR
ORDINALITY column, and classic ROW_NUMBER can be used. It can be enhanced
to support WITH ORDINALITY in future, but I have not any use case for it.
Regards
Pavel
>
> CREATE TABLE xmldata AS SELECT
> xml $$
> <ROWS>
> <ROW id="1">
> <COUNTRY_ID>AU</COUNTRY_ID>
> <COUNTRY_NAME>Australia</COUNTRY_NAME>
> </ROW>
> <ROW id="5">
> <COUNTRY_ID>JP</COUNTRY_ID>
> <COUNTRY_NAME>Japan</COUNTRY_NAME>
> <PREMIER_NAME>Shinzo Abe</PREMIER_NAME>
> <SIZE unit="sq_mi">145935</SIZE>
> </ROW>
> <ROW id="6">
> <COUNTRY_ID>SG</COUNTRY_ID>
> <COUNTRY_NAME>Singapore</COUNTRY_NAME>
> <SIZE unit="sq_km">697</SIZE>
> </ROW>
> </ROWS>
> $$ AS data;
>
> insert into xmldata values ($$
> <ROWS><ROW id="2"><COUNTRY_ID>CL</COUNTRY_ID><COUNTRY_NAME>
> Chile</COUNTRY_NAME></ROW>
> <ROW id="3"><COUNTRY_ID>AR</COUNTRY_ID><COUNTRY_NAME>
> Argentina</COUNTRY_NAME></ROW></ROWS>$$);
>
> SELECT ROW_NUMBER() OVER (), xmltable.*
> FROM xmldata,
> XMLTABLE('//ROWS/ROW'
> PASSING data
> COLUMNS id int PATH '@id',
> ordinality FOR ORDINALITY,
> "COUNTRY_NAME" text,
> country_id text PATH 'COUNTRY_ID',
> size_sq_km float PATH 'SIZE[@unit = "sq_km"]',
> size_other text PATH
> 'concat(SIZE[@unit!="sq_km"], " ",
> SIZE[@unit!="sq_km"]/@unit)',
> premier_name text PATH 'PREMIER_NAME' DEFAULT 'not
> specified')
> ;
>
> row_number │ id │ ordinality │ COUNTRY_NAME │ country_id │ size_sq_km │
> size_other │ premier_name
> ────────────┼────┼────────────┼──────────────┼────────────┼─
> ───────────┼──────────────┼───────────────
> 1 │ 1 │ 1 │ Australia │ AU │ │
> │ not specified
> 2 │ 5 │ 2 │ Japan │ JP │ │
> 145935 sq_mi │ Shinzo Abe
> 3 │ 6 │ 3 │ Singapore │ SG │ 697 │
> │ not specified
> 4 │ 2 │ 1 │ Chile │ CL │ │
> │ not specified
> 5 │ 3 │ 2 │ Argentina │ AR │ │
> │ not specified
>
>
> --
> Álvaro Herrera https://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: patch: function xmltable
@ 2017-03-02 21:35 Alvaro Herrera <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 72+ messages in thread
From: Alvaro Herrera @ 2017-03-02 21:35 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: pgsql-hackers; Craig Ringer <[email protected]>
Pavel Stehule wrote:
> 2017-03-02 19:32 GMT+01:00 Alvaro Herrera <[email protected]>:
>
> > So in the old (non-executor-node) implementation, you could attach WITH
> > ORDINALITY to the xmltable expression and it would count the output
> > rows, regardless of which XML document it comes from. With the new
> > implementation, the grammar no longer accepts it. To count output rows,
> > you still need to use row_number(). Maybe this is okay. This is the
> > example from the docs, and I add another XML document with two more rows
> > for xmltable. Look at the three numbering columns ...
> >
>
> It is expected - now tablefunc are not special case of SRF, so it lost all
> SRF functionality. It is not critical lost - it supports internally FOR
> ORDINALITY column, and classic ROW_NUMBER can be used. It can be enhanced
> to support WITH ORDINALITY in future, but I have not any use case for it.
Fine.
After looking at the new executor code a bit, I noticed that we don't
need the resultSlot anymore; we can use the ss_ScanTupleSlot instead.
Because resultSlot was being used in the xml.c code (which already
appeared a bit dubious to me), I changed the interface so that instead
the things that it read from it are passed as parameters -- namely, in
InitBuilder we pass natts, and in GetValue we pass typid and typmod.
Secondly, I noticed we have the FetchRow routine produce a minimal
tuple, put it in a slot; then its caller takes the slot and put the
tuple in the tuplestore. This is pointless; we can just have FetchRow
put the tuple in the tuplestore directly and not bother with any slot
manipulations there. This simplifies the code a bit.
Here's v47 with those changes.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[application/x-gunzip] xmltable-47.patch.gz (36.8K, ../../[email protected]/2-xmltable-47.patch.gz)
download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: patch: function xmltable
@ 2017-03-03 09:28 Pavel Stehule <[email protected]>
parent: Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 72+ messages in thread
From: Pavel Stehule @ 2017-03-03 09:28 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: pgsql-hackers; Craig Ringer <[email protected]>
2017-03-02 22:35 GMT+01:00 Alvaro Herrera <[email protected]>:
> Pavel Stehule wrote:
> > 2017-03-02 19:32 GMT+01:00 Alvaro Herrera <[email protected]>:
> >
> > > So in the old (non-executor-node) implementation, you could attach WITH
> > > ORDINALITY to the xmltable expression and it would count the output
> > > rows, regardless of which XML document it comes from. With the new
> > > implementation, the grammar no longer accepts it. To count output
> rows,
> > > you still need to use row_number(). Maybe this is okay. This is the
> > > example from the docs, and I add another XML document with two more
> rows
> > > for xmltable. Look at the three numbering columns ...
> > >
> >
> > It is expected - now tablefunc are not special case of SRF, so it lost
> all
> > SRF functionality. It is not critical lost - it supports internally FOR
> > ORDINALITY column, and classic ROW_NUMBER can be used. It can be enhanced
> > to support WITH ORDINALITY in future, but I have not any use case for it.
>
> Fine.
>
> After looking at the new executor code a bit, I noticed that we don't
> need the resultSlot anymore; we can use the ss_ScanTupleSlot instead.
> Because resultSlot was being used in the xml.c code (which already
> appeared a bit dubious to me), I changed the interface so that instead
> the things that it read from it are passed as parameters -- namely, in
> InitBuilder we pass natts, and in GetValue we pass typid and typmod.
>
I had similar feeling
>
> Secondly, I noticed we have the FetchRow routine produce a minimal
> tuple, put it in a slot; then its caller takes the slot and put the
> tuple in the tuplestore. This is pointless; we can just have FetchRow
> put the tuple in the tuplestore directly and not bother with any slot
> manipulations there. This simplifies the code a bit.
>
>
has sense
attached update with fixed tests
Regards
Pavel
> Here's v47 with those changes.
>
> --
> Álvaro Herrera https://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[application/x-gzip] xmltable-48.patch.gz (37.4K, ../../CAFj8pRCfSUihcWts78_HmcTFnAfgeCk304wZ6w+k+ADt=eQt5A@mail.gmail.com/3-xmltable-48.patch.gz)
download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: patch: function xmltable
@ 2017-03-03 18:15 Alvaro Herrera <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 72+ messages in thread
From: Alvaro Herrera @ 2017-03-03 18:15 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: pgsql-hackers; Craig Ringer <[email protected]>
Pavel Stehule wrote:
> attached update with fixed tests
Heh, I noticed that you removed the libxml "context" lines that
differentiate xml.out from xml_2.out when doing this. My implementation
emits those lines, so it was failing for me. I restored them.
I also changed a few things to avoid copying into TableFuncScanState
things that come from the TableFunc itself, since the executor state
node can grab them from the plan node. Let's do that. So instead of
"evalcols" the code now checks that the column list is empty; and also,
read the ordinality column number from the plan node.
I have to bounce this back to you one more time, hopefully the last one
I hope. Two things:
1. Please verify that pg_stat_statements behaves correctly. The patch
doesn't have changes to contrib/ so without testing I'm guessing that it
doesn't work. I think something very simple should do.
2. As I've complained many times, I find the way we manage an empty
COLUMNS clause pretty bad. The standard doesn't require that syntax
(COLUMNS is required), and I don't like the implementation, so why not
provide the feature in a different way? My proposal is to change the
column options in gram.y to be something like this:
xmltable_column_option_el:
IDENT b_expr
{ $$ = makeDefElem($1, $2, @1); }
| DEFAULT b_expr
{ $$ = makeDefElem("default", $2, @1); }
| FULL VALUE_P
{ $$ = makeDefElem("full_value", NULL, @1); }
| NOT NULL_P
{ $$ = makeDefElem("is_not_null", (Node *) makeInteger(true), @1); }
| NULL_P
{ $$ = makeDefElem("is_not_null", (Node *) makeInteger(false), @1); }
;
Note the FULL VALUE. Then we can process it like
else if (strcmp(defel->defname, "full_value") == 0)
{
if (fc->colexpr != NULL)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("FULL ROW may not be specified together with PATH"),
parser_errposition(defel->location)));
fc->full_row = true;
}
So if you want the full XML value of the row, you have to specify it,
.. XMLTABLE ( ... COLUMNS ..., whole_row xml FULL VALUE, ... )
This has the extra feature that you can add, say, an ORDINALITY column
together with the XML value, something that you cannot do with the
current implementation.
It doesn't have to be FULL VALUE, but I couldn't think of anything
better. (I didn't want to add any new keywords for this.) If you have
a better idea, let's discuss.
Code-wise, this completely removes the "else" block in transformRangeTableFunc
which I marked with an XXX comment. That's a good thing -- let's get
rid of that. Also, it should remove the need for the separate "if
!columns" case in tfuncLoadRows. All those cases would become part of
the normal code path instead of special cases. I think
XmlTableSetColumnFilter doesn't need any change (we just don't call if
for the FULL VALUE row); and XmlTableGetValue needs a special case that
if the column filter is NULL (i.e. SetColumnFilter wasn't called for
that column) then return the whole row.
Of course, this opens an implementation issue: how do you annotate
things from parse analysis till execution? The current TableFunc
structure doesn't help, because there are only lists of column names and
expressions; and we can't use the case of a NULL colexpr, because that
case is already used by the column filter being the column name (a
feature required by the standard). A simple way would be to have a new
"colno" struct member, to store a column number for the column marked
FULL VALUE (just like ordinalitycol). This means you can't have more
than one of those FULL VALUE columns, but that seems okay.
(Of course, this means that the two cases that have no COLUMNS in the
"xmltable" production in gram.y should go away).
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[application/x-gunzip] xmltable-49.patch.gz (36.6K, ../../[email protected]/2-xmltable-49.patch.gz)
download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: patch: function xmltable
@ 2017-03-03 18:42 Pavel Stehule <[email protected]>
parent: Alvaro Herrera <[email protected]>
0 siblings, 2 replies; 72+ messages in thread
From: Pavel Stehule @ 2017-03-03 18:42 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: pgsql-hackers; Craig Ringer <[email protected]>
2017-03-03 19:15 GMT+01:00 Alvaro Herrera <[email protected]>:
> Pavel Stehule wrote:
>
> > attached update with fixed tests
>
> Heh, I noticed that you removed the libxml "context" lines that
> differentiate xml.out from xml_2.out when doing this. My implementation
> emits those lines, so it was failing for me. I restored them.
>
> I also changed a few things to avoid copying into TableFuncScanState
> things that come from the TableFunc itself, since the executor state
> node can grab them from the plan node. Let's do that. So instead of
> "evalcols" the code now checks that the column list is empty; and also,
> read the ordinality column number from the plan node.
>
> I have to bounce this back to you one more time, hopefully the last one
> I hope. Two things:
>
> 1. Please verify that pg_stat_statements behaves correctly. The patch
> doesn't have changes to contrib/ so without testing I'm guessing that it
> doesn't work. I think something very simple should do.
>
> 2. As I've complained many times, I find the way we manage an empty
> COLUMNS clause pretty bad. The standard doesn't require that syntax
> (COLUMNS is required), and I don't like the implementation, so why not
> provide the feature in a different way? My proposal is to change the
> column options in gram.y to be something like this:
The clause COLUMNS is optional on Oracle and DB2
So I prefer a Oracle, DB2 design. If you are strongly against it, then we
can remove it to be ANSI/SQL only.
I am don't see an good idea to introduce third syntax.
> xmltable_column_option_el:
> IDENT b_expr
> { $$ = makeDefElem($1, $2, @1); }
> | DEFAULT b_expr
> { $$ = makeDefElem("default", $2, @1); }
> | FULL VALUE_P
> { $$ = makeDefElem("full_value", NULL,
> @1); }
> | NOT NULL_P
> { $$ = makeDefElem("is_not_null", (Node *)
> makeInteger(true), @1); }
> | NULL_P
> { $$ = makeDefElem("is_not_null", (Node *)
> makeInteger(false), @1); }
> ;
>
> Note the FULL VALUE. Then we can process it like
>
> else if (strcmp(defel->defname, "full_value") == 0)
> {
> if (fc->colexpr != NULL)
> ereport(ERROR,
>
> (errcode(ERRCODE_SYNTAX_ERROR),
> errmsg("FULL ROW
> may not be specified together with PATH"),
>
> parser_errposition(defel->location)));
> fc->full_row = true;
> }
>
> So if you want the full XML value of the row, you have to specify it,
>
> .. XMLTABLE ( ... COLUMNS ..., whole_row xml FULL VALUE, ... )
>
> This has the extra feature that you can add, say, an ORDINALITY column
> together with the XML value, something that you cannot do with the
> current implementation.
>
> It doesn't have to be FULL VALUE, but I couldn't think of anything
> better. (I didn't want to add any new keywords for this.) If you have
> a better idea, let's discuss.
>
I don't see a introduction own syntax as necessary solution here - use
Oracle, DB2 compatible syntax, or ANSI.
It is partially corner case - the benefit of this case is almost bigger
compatibility with mentioned databases.
>
> Code-wise, this completely removes the "else" block in
> transformRangeTableFunc
> which I marked with an XXX comment. That's a good thing -- let's get
> rid of that. Also, it should remove the need for the separate "if
> !columns" case in tfuncLoadRows. All those cases would become part of
> the normal code path instead of special cases. I think
> XmlTableSetColumnFilter doesn't need any change (we just don't call if
> for the FULL VALUE row); and XmlTableGetValue needs a special case that
> if the column filter is NULL (i.e. SetColumnFilter wasn't called for
> that column) then return the whole row.
>
>
> Of course, this opens an implementation issue: how do you annotate
> things from parse analysis till execution? The current TableFunc
> structure doesn't help, because there are only lists of column names and
> expressions; and we can't use the case of a NULL colexpr, because that
> case is already used by the column filter being the column name (a
> feature required by the standard). A simple way would be to have a new
> "colno" struct member, to store a column number for the column marked
> FULL VALUE (just like ordinalitycol). This means you can't have more
> than one of those FULL VALUE columns, but that seems okay.
>
>
> (Of course, this means that the two cases that have no COLUMNS in the
> "xmltable" production in gram.y should go away).
>
You are commiter, and you should to decide - as first I prefer current
state, as second a remove this part - it should be good for you too,
because code that you don't like will be left.
I dislike introduce new syntax - this case is not too important for this.
Regards
Pavel
> --
> Álvaro Herrera https://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: patch: function xmltable
@ 2017-03-03 19:23 Pavel Stehule <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 0 replies; 72+ messages in thread
From: Pavel Stehule @ 2017-03-03 19:23 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: pgsql-hackers; Craig Ringer <[email protected]>
2017-03-03 19:42 GMT+01:00 Pavel Stehule <[email protected]>:
>
>
> 2017-03-03 19:15 GMT+01:00 Alvaro Herrera <[email protected]>:
>
>> Pavel Stehule wrote:
>>
>> > attached update with fixed tests
>>
>> Heh, I noticed that you removed the libxml "context" lines that
>> differentiate xml.out from xml_2.out when doing this. My implementation
>> emits those lines, so it was failing for me. I restored them.
>>
>> I also changed a few things to avoid copying into TableFuncScanState
>> things that come from the TableFunc itself, since the executor state
>> node can grab them from the plan node. Let's do that. So instead of
>> "evalcols" the code now checks that the column list is empty; and also,
>> read the ordinality column number from the plan node.
>>
>> I have to bounce this back to you one more time, hopefully the last one
>> I hope. Two things:
>>
>> 1. Please verify that pg_stat_statements behaves correctly. The patch
>> doesn't have changes to contrib/ so without testing I'm guessing that it
>> doesn't work. I think something very simple should do.
>>
>> 2. As I've complained many times, I find the way we manage an empty
>> COLUMNS clause pretty bad. The standard doesn't require that syntax
>> (COLUMNS is required), and I don't like the implementation, so why not
>> provide the feature in a different way? My proposal is to change the
>> column options in gram.y to be something like this:
>
>
> The clause COLUMNS is optional on Oracle and DB2
>
> So I prefer a Oracle, DB2 design. If you are strongly against it, then we
> can remove it to be ANSI/SQL only.
>
> I am don't see an good idea to introduce third syntax.
>
>
>> xmltable_column_option_el:
>> IDENT b_expr
>> { $$ = makeDefElem($1, $2, @1); }
>> | DEFAULT b_expr
>> { $$ = makeDefElem("default", $2, @1); }
>> | FULL VALUE_P
>> { $$ = makeDefElem("full_value", NULL,
>> @1); }
>> | NOT NULL_P
>> { $$ = makeDefElem("is_not_null", (Node
>> *) makeInteger(true), @1); }
>> | NULL_P
>> { $$ = makeDefElem("is_not_null", (Node
>> *) makeInteger(false), @1); }
>> ;
>>
>> Note the FULL VALUE. Then we can process it like
>>
>> else if (strcmp(defel->defname, "full_value") ==
>> 0)
>> {
>> if (fc->colexpr != NULL)
>> ereport(ERROR,
>>
>> (errcode(ERRCODE_SYNTAX_ERROR),
>> errmsg("FULL ROW
>> may not be specified together with PATH"),
>>
>> parser_errposition(defel->location)));
>> fc->full_row = true;
>> }
>>
>> So if you want the full XML value of the row, you have to specify it,
>>
>> .. XMLTABLE ( ... COLUMNS ..., whole_row xml FULL VALUE, ... )
>>
>> This has the extra feature that you can add, say, an ORDINALITY column
>> together with the XML value, something that you cannot do with the
>> current implementation.
>>
>> It doesn't have to be FULL VALUE, but I couldn't think of anything
>> better. (I didn't want to add any new keywords for this.) If you have
>> a better idea, let's discuss.
>>
>
> I don't see a introduction own syntax as necessary solution here - use
> Oracle, DB2 compatible syntax, or ANSI.
>
> It is partially corner case - the benefit of this case is almost bigger
> compatibility with mentioned databases.
>
>
>>
>> Code-wise, this completely removes the "else" block in
>> transformRangeTableFunc
>> which I marked with an XXX comment. That's a good thing -- let's get
>> rid of that. Also, it should remove the need for the separate "if
>> !columns" case in tfuncLoadRows. All those cases would become part of
>> the normal code path instead of special cases. I think
>> XmlTableSetColumnFilter doesn't need any change (we just don't call if
>> for the FULL VALUE row); and XmlTableGetValue needs a special case that
>> if the column filter is NULL (i.e. SetColumnFilter wasn't called for
>> that column) then return the whole row.
>>
>>
>> Of course, this opens an implementation issue: how do you annotate
>> things from parse analysis till execution? The current TableFunc
>> structure doesn't help, because there are only lists of column names and
>> expressions; and we can't use the case of a NULL colexpr, because that
>> case is already used by the column filter being the column name (a
>> feature required by the standard). A simple way would be to have a new
>> "colno" struct member, to store a column number for the column marked
>> FULL VALUE (just like ordinalitycol). This means you can't have more
>> than one of those FULL VALUE columns, but that seems okay.
>>
>>
>> (Of course, this means that the two cases that have no COLUMNS in the
>> "xmltable" production in gram.y should go away).
>>
>
> You are commiter, and you should to decide - as first I prefer current
> state, as second a remove this part - it should be good for you too,
> because code that you don't like will be left.
>
> I dislike introduce new syntax - this case is not too important for this.
>
>
I am able to prepare reduced version if we do a agreement
Regards
Pavel
> Regards
>
> Pavel
>
>
>> --
>> Álvaro Herrera https://www.2ndQuadrant.com/
>> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>>
>
>
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: patch: function xmltable
@ 2017-03-03 20:04 Alvaro Herrera <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 2 replies; 72+ messages in thread
From: Alvaro Herrera @ 2017-03-03 20:04 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: pgsql-hackers; Craig Ringer <[email protected]>
Pavel Stehule wrote:
> 2017-03-03 19:15 GMT+01:00 Alvaro Herrera <[email protected]>:
> > 2. As I've complained many times, I find the way we manage an empty
> > COLUMNS clause pretty bad. The standard doesn't require that syntax
> > (COLUMNS is required), and I don't like the implementation, so why not
> > provide the feature in a different way? My proposal is to change the
> > column options in gram.y to be something like this:
>
> The clause COLUMNS is optional on Oracle and DB2
>
> So I prefer a Oracle, DB2 design. If you are strongly against it, then we
> can remove it to be ANSI/SQL only.
>
> I am don't see an good idea to introduce third syntax.
OK. I think trying to be syntax compatible with DB2 or Oracle is a lost
cause, because the syntax used in the XPath expressions seems different
-- I think Oracle uses XQuery (which we don't support) and DB2 uses ...
not sure what it is, but it doesn't work in our implementation
(stuff like '$d/employees/emp' in the row expression.)
In existing applications using those Oracle/DB2, is it common to omit
the COLUMNS clause? I searched for "xmltable oracle" and had a look at
the first few hits outside of the oracle docs:
http://viralpatel.net/blogs/oracle-xmltable-tutorial/
http://www.dba-oracle.com/t_xmltable.htm
http://stackoverflow.com/questions/12690868/how-to-use-xmltable-in-oracle
https://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:9533111800346252295
http://stackoverflow.com/questions/1222570/what-is-an-xmltable
https://community.oracle.com/thread/3955198
Not a single one of these omit the COLUMNS clause (though the second one
mentions that the clause can be omitted).
I also looked at a few samples with DB2 -- same thing; it is possible,
but is it common?
Anyway, I noticed that "xml PATH '.'" can be used to obtain the full XML
of the row, which I think is the feature I wanted, so I think we're
covered and we can omit the case with no COLUMNS, since we already have
the feature in another way. No need to implement anything further, and
we can rip out the special case I don't like. Example:
CREATE TABLE EMPLOYEES
(
id integer,
data XML
);
INSERT INTO EMPLOYEES
VALUES (1, '<Employees>
<Employee emplid="1111" type="admin">
<firstname>John</firstname>
<lastname>Watson</lastname>
<age>30</age>
<email>[email protected]</email>
</Employee>
<Employee emplid="2222" type="admin">
<firstname>Sherlock</firstname>
<lastname>Homes</lastname>
<age>32</age>
<email>[email protected]</email>
</Employee>
<Employee emplid="3333" type="user">
<firstname>Jim</firstname>
<lastname>Moriarty</lastname>
<age>52</age>
<email>[email protected]</email>
</Employee>
<Employee emplid="4444" type="user">
<firstname>Mycroft</firstname>
<lastname>Holmes</lastname>
<age>41</age>
<email>[email protected]</email>
</Employee>
</Employees>');
This is with COLUMNS omitted:
alvherre=# select xmltable.* from employees, xmltable('/Employees/Employee' passing data);
xmltable
──────────────────────────────────────────
<Employee emplid="1111" type="admin"> ↵
<firstname>John</firstname> ↵
<lastname>Watson</lastname> ↵
<age>30</age> ↵
<email>[email protected]</email>↵
</Employee>
<Employee emplid="2222" type="admin"> ↵
<firstname>Sherlock</firstname> ↵
<lastname>Homes</lastname> ↵
<age>32</age> ↵
<email>[email protected]</email> ↵
</Employee>
<Employee emplid="3333" type="user"> ↵
<firstname>Jim</firstname> ↵
<lastname>Moriarty</lastname> ↵
<age>52</age> ↵
<email>[email protected]</email> ↵
</Employee>
<Employee emplid="4444" type="user"> ↵
<firstname>Mycroft</firstname> ↵
<lastname>Holmes</lastname> ↵
<age>41</age> ↵
<email>[email protected]</email> ↵
</Employee>
and this is what you get with "xml PATH '.'" (I threw in ORDINALITY just
for fun):
alvherre=# select xmltable.* from employees, xmltable('/Employees/Employee' passing data columns row_number for ordinality, emp xml path '.');
row_number │ emp
────────────┼──────────────────────────────────────────
1 │ <Employee emplid="1111" type="admin"> ↵
│ <firstname>John</firstname> ↵
│ <lastname>Watson</lastname> ↵
│ <age>30</age> ↵
│ <email>[email protected]</email>↵
│ </Employee>
2 │ <Employee emplid="2222" type="admin"> ↵
│ <firstname>Sherlock</firstname> ↵
│ <lastname>Homes</lastname> ↵
│ <age>32</age> ↵
│ <email>[email protected]</email> ↵
│ </Employee>
3 │ <Employee emplid="3333" type="user"> ↵
│ <firstname>Jim</firstname> ↵
│ <lastname>Moriarty</lastname> ↵
│ <age>52</age> ↵
│ <email>[email protected]</email> ↵
│ </Employee>
4 │ <Employee emplid="4444" type="user"> ↵
│ <firstname>Mycroft</firstname> ↵
│ <lastname>Holmes</lastname> ↵
│ <age>41</age> ↵
│ <email>[email protected]</email> ↵
│ </Employee>
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: patch: function xmltable
@ 2017-03-03 21:41 Pavel Stehule <[email protected]>
parent: Alvaro Herrera <[email protected]>
1 sibling, 0 replies; 72+ messages in thread
From: Pavel Stehule @ 2017-03-03 21:41 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: pgsql-hackers; Craig Ringer <[email protected]>
2017-03-03 21:04 GMT+01:00 Alvaro Herrera <[email protected]>:
> Pavel Stehule wrote:
> > 2017-03-03 19:15 GMT+01:00 Alvaro Herrera <[email protected]>:
>
> > > 2. As I've complained many times, I find the way we manage an empty
> > > COLUMNS clause pretty bad. The standard doesn't require that syntax
> > > (COLUMNS is required), and I don't like the implementation, so why not
> > > provide the feature in a different way? My proposal is to change the
> > > column options in gram.y to be something like this:
> >
> > The clause COLUMNS is optional on Oracle and DB2
> >
> > So I prefer a Oracle, DB2 design. If you are strongly against it, then we
> > can remove it to be ANSI/SQL only.
> >
> > I am don't see an good idea to introduce third syntax.
>
> OK. I think trying to be syntax compatible with DB2 or Oracle is a lost
> cause, because the syntax used in the XPath expressions seems different
> -- I think Oracle uses XQuery (which we don't support) and DB2 uses ...
> not sure what it is, but it doesn't work in our implementation
> (stuff like '$d/employees/emp' in the row expression.)
>
100% compatibility is not possible - but XPath is subset of XQuery and in
reality - the full XQuery examples of XMLTABLE is not often.
Almost all examples of usage XMLTABLE, what I found in blogs, uses XPath
only
>
> In existing applications using those Oracle/DB2, is it common to omit
> the COLUMNS clause? I searched for "xmltable oracle" and had a look at
> the first few hits outside of the oracle docs:
> http://viralpatel.net/blogs/oracle-xmltable-tutorial/
> http://www.dba-oracle.com/t_xmltable.htm
> http://stackoverflow.com/questions/12690868/how-to-use-xmltable-in-oracle
> https://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:
> 9533111800346252295
> http://stackoverflow.com/questions/1222570/what-is-an-xmltable
> https://community.oracle.com/thread/3955198
>
> Not a single one of these omit the COLUMNS clause (though the second one
> mentions that the clause can be omitted).
>
> I also looked at a few samples with DB2 -- same thing; it is possible,
> but is it common?
>
I don't think so it is common - it is corner case - and I can live without
it well
>
> Anyway, I noticed that "xml PATH '.'" can be used to obtain the full XML
> of the row, which I think is the feature I wanted, so I think we're
> covered and we can omit the case with no COLUMNS, since we already have
> the feature in another way. No need to implement anything further, and
> we can rip out the special case I don't like. Example:
>
yes,
>
> CREATE TABLE EMPLOYEES
> (
> id integer,
> data XML
> );
> INSERT INTO EMPLOYEES
> VALUES (1, '<Employees>
> <Employee emplid="1111" type="admin">
> <firstname>John</firstname>
> <lastname>Watson</lastname>
> <age>30</age>
> <email>[email protected]</email>
> </Employee>
> <Employee emplid="2222" type="admin">
> <firstname>Sherlock</firstname>
> <lastname>Homes</lastname>
> <age>32</age>
> <email>[email protected]</email>
> </Employee>
> <Employee emplid="3333" type="user">
> <firstname>Jim</firstname>
> <lastname>Moriarty</lastname>
> <age>52</age>
> <email>[email protected]</email>
> </Employee>
> <Employee emplid="4444" type="user">
> <firstname>Mycroft</firstname>
> <lastname>Holmes</lastname>
> <age>41</age>
> <email>[email protected]</email>
> </Employee>
> </Employees>');
>
> This is with COLUMNS omitted:
>
> alvherre=# select xmltable.* from employees,
> xmltable('/Employees/Employee' passing data);
> xmltable
> ──────────────────────────────────────────
> <Employee emplid="1111" type="admin"> ↵
> <firstname>John</firstname> ↵
> <lastname>Watson</lastname> ↵
> <age>30</age> ↵
> <email>[email protected]</email>↵
> </Employee>
> <Employee emplid="2222" type="admin"> ↵
> <firstname>Sherlock</firstname> ↵
> <lastname>Homes</lastname> ↵
> <age>32</age> ↵
> <email>[email protected]</email> ↵
> </Employee>
> <Employee emplid="3333" type="user"> ↵
> <firstname>Jim</firstname> ↵
> <lastname>Moriarty</lastname> ↵
> <age>52</age> ↵
> <email>[email protected]</email> ↵
> </Employee>
> <Employee emplid="4444" type="user"> ↵
> <firstname>Mycroft</firstname> ↵
> <lastname>Holmes</lastname> ↵
> <age>41</age> ↵
> <email>[email protected]</email> ↵
> </Employee>
>
> and this is what you get with "xml PATH '.'" (I threw in ORDINALITY just
> for fun):
>
> alvherre=# select xmltable.* from employees,
> xmltable('/Employees/Employee' passing data columns row_number for
> ordinality, emp xml path '.');
> row_number │ emp
> ────────────┼──────────────────────────────────────────
> 1 │ <Employee emplid="1111" type="admin"> ↵
> │ <firstname>John</firstname> ↵
> │ <lastname>Watson</lastname> ↵
> │ <age>30</age> ↵
> │ <email>[email protected]</email>↵
> │ </Employee>
> 2 │ <Employee emplid="2222" type="admin"> ↵
> │ <firstname>Sherlock</firstname> ↵
> │ <lastname>Homes</lastname> ↵
> │ <age>32</age> ↵
> │ <email>[email protected]</email> ↵
> │ </Employee>
> 3 │ <Employee emplid="3333" type="user"> ↵
> │ <firstname>Jim</firstname> ↵
> │ <lastname>Moriarty</lastname> ↵
> │ <age>52</age> ↵
> │ <email>[email protected]</email> ↵
> │ </Employee>
> 4 │ <Employee emplid="4444" type="user"> ↵
> │ <firstname>Mycroft</firstname> ↵
> │ <lastname>Holmes</lastname> ↵
> │ <age>41</age> ↵
> │ <email>[email protected]</email> ↵
> │ </Employee>
>
> --
> Álvaro Herrera https://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: patch: function xmltable
@ 2017-03-05 09:32 Pavel Stehule <[email protected]>
parent: Alvaro Herrera <[email protected]>
1 sibling, 1 reply; 72+ messages in thread
From: Pavel Stehule @ 2017-03-05 09:32 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: pgsql-hackers; Craig Ringer <[email protected]>
Hi
I used your idea about special columns when COLUMNS are not explicitly
defined.
All lines that you are dislike removed.
Now, almost all code, related to this behave, is in next few lines.
+ /*
+ * Use implicit column when it is necessary. The COLUMNS clause is
optional
+ * on Oracle and DB2. In this case a result is complete row of XML type.
+ */
+ if (rtf->columns == NIL)
+ {
+ RangeTableFuncCol *fc = makeNode(RangeTableFuncCol);
+ A_Const *n = makeNode(A_Const);
+
+ fc->colname = "xmltable";
+ fc->typeName = makeTypeNameFromOid(XMLOID, -1);
+ n->val.type = T_String;
+ n->val.val.str = ".";
+ n->location = -1;
+
+ fc->colexpr = (Node *) n;
+ rtf->columns = list_make1(fc);
+ }
all regress tests passing.
Regards
Pavel
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[application/x-gzip] xmltable-50.patch.gz (36.5K, ../../CAFj8pRAjrX_dtdUdfH2DjEdsh3NZkXV3PLJTjUnyXKji9ChteQ@mail.gmail.com/3-xmltable-50.patch.gz)
download
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: patch: function xmltable
@ 2017-03-08 16:01 Alvaro Herrera <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 72+ messages in thread
From: Alvaro Herrera @ 2017-03-08 16:01 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: pgsql-hackers; Craig Ringer <[email protected]>
Pavel Stehule wrote:
> Hi
>
> I used your idea about special columns when COLUMNS are not explicitly
> defined.
>
> All lines that you are dislike removed.
I just pushed XMLTABLE, after some additional changes. Please test it
thoroughly and report any problems.
I didn't add the change you proposed here to keep COLUMNS optional;
instead, I just made COLUMNS mandatory. I think what you propose here
is not entirely out of the question, but you left out ruleutils.c
support for it, so I decided to leave it aside for now so that I could
get this patch out of my plate once and for all. If you really want
that feature, you can submit another patch for it and discuss with the
RMT whether it belongs in PG10 or not.
Some changes I made:
* I added some pg_stat_statements support. It works fine for simple
tests, but deeper testing of it would be appreciated.
* I removed the "buildercxt" memory context. It seemed mostly
pointless, and I was disturbed by the MemoryContextResetOnly().
Per-value memory still uses the per-value memory context, but the rest
of the stuff is in the per-query context, which should be pretty much
the same.
* Desultory stylistic changes
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: patch: function xmltable
@ 2017-03-08 16:10 Pavel Stehule <[email protected]>
parent: Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 72+ messages in thread
From: Pavel Stehule @ 2017-03-08 16:10 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: pgsql-hackers; Craig Ringer <[email protected]>
2017-03-08 17:01 GMT+01:00 Alvaro Herrera <[email protected]>:
> Pavel Stehule wrote:
> > Hi
> >
> > I used your idea about special columns when COLUMNS are not explicitly
> > defined.
> >
> > All lines that you are dislike removed.
>
> I just pushed XMLTABLE, after some additional changes. Please test it
> thoroughly and report any problems.
>
Thank you
>
> I didn't add the change you proposed here to keep COLUMNS optional;
> instead, I just made COLUMNS mandatory. I think what you propose here
> is not entirely out of the question, but you left out ruleutils.c
> support for it, so I decided to leave it aside for now so that I could
> get this patch out of my plate once and for all. If you really want
> that feature, you can submit another patch for it and discuss with the
> RMT whether it belongs in PG10 or not.
>
It is interesting feature - because it replaces XPATH function, but not
important enough.
For daily work the default schema support is much more interesting.
>
> Some changes I made:
> * I added some pg_stat_statements support. It works fine for simple
> tests, but deeper testing of it would be appreciated.
>
> * I removed the "buildercxt" memory context. It seemed mostly
> pointless, and I was disturbed by the MemoryContextResetOnly().
> Per-value memory still uses the per-value memory context, but the rest
> of the stuff is in the per-query context, which should be pretty much
> the same.
>
> * Desultory stylistic changes
>
ok
Regards
Pavel
>
> --
> Álvaro Herrera https://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: patch: function xmltable
@ 2017-03-08 16:32 Alvaro Herrera <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 72+ messages in thread
From: Alvaro Herrera @ 2017-03-08 16:32 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: pgsql-hackers; Craig Ringer <[email protected]>
Pavel Stehule wrote:
> 2017-03-08 17:01 GMT+01:00 Alvaro Herrera <[email protected]>:
> > I didn't add the change you proposed here to keep COLUMNS optional;
> > instead, I just made COLUMNS mandatory. I think what you propose here
> > is not entirely out of the question, but you left out ruleutils.c
> > support for it, so I decided to leave it aside for now so that I could
> > get this patch out of my plate once and for all. If you really want
> > that feature, you can submit another patch for it and discuss with the
> > RMT whether it belongs in PG10 or not.
>
> It is interesting feature - because it replaces XPATH function, but not
> important enough.
OK.
> For daily work the default schema support is much more interesting.
Let's see that one, then. It was part of the original submission so
depending on how the patch we looks can still cram it in. But other
patches have priority for me now.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: patch: function xmltable
@ 2017-03-08 16:46 Pavel Stehule <[email protected]>
parent: Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 72+ messages in thread
From: Pavel Stehule @ 2017-03-08 16:46 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: pgsql-hackers; Craig Ringer <[email protected]>
2017-03-08 17:32 GMT+01:00 Alvaro Herrera <[email protected]>:
> Pavel Stehule wrote:
> > 2017-03-08 17:01 GMT+01:00 Alvaro Herrera <[email protected]>:
>
> > > I didn't add the change you proposed here to keep COLUMNS optional;
> > > instead, I just made COLUMNS mandatory. I think what you propose here
> > > is not entirely out of the question, but you left out ruleutils.c
> > > support for it, so I decided to leave it aside for now so that I could
> > > get this patch out of my plate once and for all. If you really want
> > > that feature, you can submit another patch for it and discuss with the
> > > RMT whether it belongs in PG10 or not.
> >
> > It is interesting feature - because it replaces XPATH function, but not
> > important enough.
>
> OK.
>
> > For daily work the default schema support is much more interesting.
>
> Let's see that one, then. It was part of the original submission so
> depending on how the patch we looks can still cram it in. But other
> patches have priority for me now.
>
It is theme for 11
Thank you very much
Pavel
>
> --
> Álvaro Herrera https://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: patch: function xmltable
@ 2017-03-08 17:01 Alvaro Herrera <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Alvaro Herrera @ 2017-03-08 17:01 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: pgsql-hackers; Craig Ringer <[email protected]>
Pavel Stehule wrote:
> 2017-03-08 17:32 GMT+01:00 Alvaro Herrera <[email protected]>:
> > > For daily work the default schema support is much more interesting.
> >
> > Let's see that one, then. It was part of the original submission so
> > depending on how the patch we looks can still cram it in. But other
> > patches have priority for me now.
>
> It is theme for 11
Ah, great.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 01/12] Disallow compressed data inside container types
@ 2021-02-16 13:54 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-02-16 13:54 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the comprassed field but while contructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we onlle decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
---
src/backend/access/common/heaptuple.c | 28 ++++++---------------------
src/backend/access/heap/heaptoast.c | 16 ++++++++-------
src/backend/executor/execTuples.c | 11 -----------
src/include/access/heaptoast.h | 4 ++--
4 files changed, 17 insertions(+), 42 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index 24a27e387d..5c3194f96d 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -983,30 +983,14 @@ heap_expand_tuple(HeapTuple sourceTuple, TupleDesc tupleDesc)
Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
- HeapTupleHeader td;
-
- /*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
- */
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
- tuple->t_len,
- tupleDesc);
-
/*
- * Fast path for easy case: just make a palloc'd copy and insert the
- * correct composite-Datum header fields (since those may not be set if
- * the given tuple came from disk, rather than from heap_form_tuple).
+ * The tuple contains compressed/external TOAST pointers, so we have
+ * to inline those fields to meet the conventions for composite-type
+ * Datums.
*/
- td = (HeapTupleHeader) palloc(tuple->t_len);
- memcpy((char *) td, (char *) tuple->t_data, tuple->t_len);
-
- HeapTupleHeaderSetDatumLength(td, tuple->t_len);
- HeapTupleHeaderSetTypeId(td, tupleDesc->tdtypeid);
- HeapTupleHeaderSetTypMod(td, tupleDesc->tdtypmod);
-
- return PointerGetDatum(td);
+ return toast_flatten_tuple_to_datum(tuple->t_data,
+ tuple->t_len,
+ tupleDesc);
}
/*
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..dd162daab9 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -549,14 +549,15 @@ toast_flatten_tuple_to_datum(HeapTupleHeader tup,
/* ----------
* toast_build_flattened_tuple -
*
- * Build a tuple containing no out-of-line toasted fields.
- * (This does not eliminate compressed or short-header datums.)
+ * Build a tuple containing no compressed/out-of-line toasted fields.
+ * (This does not eliminate short-header datums.)
*
* This is essentially just like heap_form_tuple, except that it will
- * expand any external-data pointers beforehand.
+ * expand any compressed/external-data pointers beforehand.
*
- * It's not very clear whether it would be preferable to decompress
- * in-line compressed datums while at it. For now, we don't.
+ * It is not necessary to decompress the compressed data for the
+ * correctness, but reflects an expectation that compression will be more
+ * effective if applied to the whole tuple not individual fields.
* ----------
*/
HeapTuple
@@ -589,9 +590,10 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) ||
+ VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..ca7fbed576 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2189,13 +2189,6 @@ BuildTupleFromCStrings(AttInMetadata *attinmeta, char **values)
* memory context. Beware of code that changes context between the initial
* heap_form_tuple/etc call and calling HeapTuple(Header)GetDatum.
*
- * For performance-critical callers, it could be worthwhile to take extra
- * steps to ensure that there aren't TOAST pointers in the output of
- * heap_form_tuple to begin with. It's likely however that the costs of the
- * typcache lookup and tuple disassembly/reassembly are swamped by TOAST
- * dereference costs, so that the benefits of such extra effort would be
- * minimal.
- *
* XXX it would likely be better to create wrapper functions that produce
* a composite Datum from the field values in one step. However, there's
* enough code using the existing APIs that we couldn't get rid of this
@@ -2207,10 +2200,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.h
index 8b29f1a986..c80af45012 100644
--- a/src/include/access/heaptoast.h
+++ b/src/include/access/heaptoast.h
@@ -128,8 +128,8 @@ extern Datum toast_flatten_tuple_to_datum(HeapTupleHeader tup,
/* ----------
* toast_build_flattened_tuple -
*
- * Build a tuple containing no out-of-line toasted fields.
- * (This does not eliminate compressed or short-header datums.)
+ * Build a tuple containing no compressed/out-of-line toasted fields.
+ * (This does not eliminate short-header datums.)
* ----------
*/
extern HeapTuple toast_build_flattened_tuple(TupleDesc tupleDesc,
--
2.17.0
--9Ek0hoCL9XbhcSqy
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0002-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--cvVnyQ+4j833TQvp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* [PATCH 3/8] Disallow compressed data inside container types
@ 2021-03-04 11:03 Dilip Kumar <[email protected]>
0 siblings, 0 replies; 72+ messages in thread
From: Dilip Kumar @ 2021-03-04 11:03 UTC (permalink / raw)
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the compressed field but while constructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we only decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
Dilip Kumar based on idea from Robert Haas
---
src/backend/access/common/heaptuple.c | 9 +--
src/backend/access/heap/heaptoast.c | 4 +-
src/backend/executor/execExprInterp.c | 6 +-
src/backend/executor/execTuples.c | 4 --
src/backend/utils/adt/expandedrecord.c | 76 +++++++++-----------------
src/backend/utils/adt/jsonfuncs.c | 3 +-
src/include/funcapi.h | 4 +-
src/pl/plpgsql/src/pl_exec.c | 3 +-
8 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index c36c283253..eb9f016dfa 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -984,15 +984,12 @@ Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
/*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
+ * We have to inline any external/compressed data to meet the conventions
+ * for composite-type Datums.
*/
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
+ return toast_flatten_tuple_to_datum(tuple->t_data,
tuple->t_len,
tupleDesc);
- else
- return heap_copy_tuple_as_raw_datum(tuple, tupleDesc);
}
/* ----------------
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..b09462348b 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index c3754acca4..71e6f41fee 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op)
{
Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i);
- if (op->d.row.elemnulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i])))
+ if (op->d.row.elemnulls[i] || attr->attlen != -1)
continue;
op->d.row.elemvalues[i] =
@@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext
{
Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i);
- if (op->d.fieldstore.nulls[i] || attr->attlen != -1 ||
- !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i])))
+ if (op->d.fieldstore.nulls[i] || attr->attlen != -1)
continue;
op->d.fieldstore.values[i] = PointerGetDatum(
PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i]));
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..f11546468e 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c
index e19491ecf7..3cbc256671 100644
--- a/src/backend/utils/adt/expandedrecord.c
+++ b/src/backend/utils/adt/expandedrecord.c
@@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
erh->er_typmod = tupdesc->tdtypmod;
}
- /*
- * If we have a valid flattened value without out-of-line fields, we can
- * just use it as-is.
- */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- return erh->fvalue->t_len;
-
/* If we have a cached size value, believe that */
if (erh->flat_size)
return erh->flat_size;
@@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr)
tupdesc = erh->er_tupdesc;
/*
- * Composite datums mustn't contain any out-of-line values.
+ * Composite datums mustn't contain any out-of-line/compressed values.
*/
- if (erh->flags & ER_FLAG_HAVE_EXTERNAL)
+ for (i = 0; i < erh->nfields; i++)
{
- for (i = 0; i < erh->nfields; i++)
- {
- Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
- if (!erh->dnulls[i] &&
- !attr->attbyval && attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])))
- {
- /*
- * expanded_record_set_field_internal can do the actual work
- * of detoasting. It needn't recheck domain constraints.
- */
- expanded_record_set_field_internal(erh, i + 1,
- erh->dvalues[i], false,
- true,
- false);
- }
+ if (!erh->dnulls[i] &&
+ !attr->attbyval && attr->attlen == -1 &&
+ (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i]))))
+ {
+ /*
+ * expanded_record_set_field_internal can do the actual work
+ * of detoasting. It needn't recheck domain constraints.
+ */
+ expanded_record_set_field_internal(erh, i + 1,
+ erh->dvalues[i], false,
+ true,
+ false);
}
-
- /*
- * We have now removed all external field values, so we can clear the
- * flag about them. This won't cause ER_flatten_into() to mistakenly
- * take the fast path, since expanded_record_set_field() will have
- * cleared ER_FLAG_FVALUE_VALID.
- */
- erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
}
+ /*
+ * We have now removed all external field values, so we can clear the
+ * flag about them. This won't cause ER_flatten_into() to mistakenly
+ * take the fast path, since expanded_record_set_field() will have
+ * cleared ER_FLAG_FVALUE_VALID.
+ */
+ erh->flags &= ~ER_FLAG_HAVE_EXTERNAL;
+
/* Test if we currently have any null values */
hasnull = false;
for (i = 0; i < erh->nfields; i++)
@@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr,
Assert(erh->er_magic == ER_MAGIC);
- /* Easy if we have a valid flattened value without out-of-line fields */
- if (erh->flags & ER_FLAG_FVALUE_VALID &&
- !(erh->flags & ER_FLAG_HAVE_EXTERNAL))
- {
- Assert(allocated_size == erh->fvalue->t_len);
- memcpy(tuphdr, erh->fvalue->t_data, allocated_size);
- /* The original flattened value might not have datum header fields */
- HeapTupleHeaderSetDatumLength(tuphdr, allocated_size);
- HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid);
- HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod);
- return;
- }
-
/* Else allocation should match previous get_flat_size result */
Assert(allocated_size == erh->flat_size);
@@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber,
if (expand_external)
{
if (attr->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(newValue)))
+ (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) ||
+ VARATT_IS_COMPRESSED(DatumGetPointer(newValue))))
{
/* Detoasting should be done in short-lived context. */
oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh));
- newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue)));
+ newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue)));
MemoryContextSwitchTo(oldcxt);
}
else
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c
index c3d464f42b..821aa8fbdb 100644
--- a/src/backend/utils/adt/jsonfuncs.c
+++ b/src/backend/utils/adt/jsonfuncs.c
@@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc,
&field,
&nulls[i]);
- if (!nulls[i] && att->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(values[i])))
+ if (!nulls[i] && att->attlen == -1)
values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i]));
}
diff --git a/src/include/funcapi.h b/src/include/funcapi.h
index 8ba7ae211f..c869012873 100644
--- a/src/include/funcapi.h
+++ b/src/include/funcapi.h
@@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple);
* Macro declarations/inline functions:
* HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as
* HeapTupleHeaderGetDatum but the input tuple should not contain
- * external varlena
+ * external/compressed varlena
* HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum.
* HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum
- * but the input tuple should not contain external varlena
+ * but the input tuple should not contain external/compressed varlena
*
* Obsolete routines and macros:
* TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index fd073767bc..0519253cbe 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate,
&dvalues[i], &nulls[i]);
if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid)
return NULL;
- if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 &&
- VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i])))
+ if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1)
dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i]));
/* XXX should we insist on typmod match, too? */
}
--
2.17.0
--C94crkcyjafcjHxo
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-Built-in-compression-method.patch"
^ permalink raw reply [nested|flat] 72+ messages in thread
* about cross-compiling issue
@ 2024-05-23 09:08 =?utf-8?B?6ZmI5Lqa5p2w?= <[email protected]>
0 siblings, 2 replies; 72+ messages in thread
From: =?utf-8?B?6ZmI5Lqa5p2w?= @ 2024-05-23 09:08 UTC (permalink / raw)
To: pgsql-hackers
Hello, I have a question about cross-compiling. I get an error when doing initdb for postgresql for arm64 architecture devices.
The error information is Error relocating /data/postgresql/postgresql-16.3-arm64-v8a-build/tmp_install/usr/postgresql/arm64-v8a/lib/dict_snowball.so: palloc0: symbol not found.
In fact, the library exists in this directory, and the palloc symbol exists but is not defined.
Any tips to go around this issue?
Thanks!
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re: about cross-compiling issue
@ 2024-05-23 15:10 Tom Lane <[email protected]>
parent: =?utf-8?B?6ZmI5Lqa5p2w?= <[email protected]>
1 sibling, 0 replies; 72+ messages in thread
From: Tom Lane @ 2024-05-23 15:10 UTC (permalink / raw)
To: =?utf-8?B?6ZmI5Lqa5p2w?= <[email protected]>; +Cc: pgsql-hackers
"=?utf-8?B?6ZmI5Lqa5p2w?=" <[email protected]> writes:
> Hello, I have a question about cross-compiling. I get an error when doing initdb for postgresql for arm64 architecture devices.
> The error information is Error relocating /data/postgresql/postgresql-16.3-arm64-v8a-build/tmp_install/usr/postgresql/arm64-v8a/lib/dict_snowball.so: palloc0: symbol not found.
We don't really support cross-compiling, because there are too many
things that the configure script can't check for if it can't run a
test program. In this particular case I think what is biting you
is that configure won't add -Wl,--export-dynamic to the backend
link switches.
You might think that that shouldn't require a test program to
verify, but c-compiler.m4 says differently:
# Given a string, check if the compiler supports the string as a
# command-line option. If it does, add to the given variable.
# For reasons you'd really rather not know about, this checks whether
# you can link to a particular function, not just whether you can link.
# In fact, we must actually check that the resulting program runs :-(
This check dates to 2008, and maybe it's no longer necessary on
any modern system, but I'm unexcited about trying to find out.
regards, tom lane
^ permalink raw reply [nested|flat] 72+ messages in thread
* Re:about cross-compiling issue
@ 2024-05-24 06:02 Long Song <[email protected]>
parent: =?utf-8?B?6ZmI5Lqa5p2w?= <[email protected]>
1 sibling, 0 replies; 72+ messages in thread
From: Long Song @ 2024-05-24 06:02 UTC (permalink / raw)
To: =?GBK?B?s8LRx73c?= <[email protected]>; +Cc: pgsql-hackers
Hi Chen Yajie,
Your provided information is fuzzy, so I can only give some simple suggestions:
1. Use `file dict_snowball.so` to see the detail info of dict_snowball.so, maybe you can get some useful hint.
2. Use gdb to debug the initdb processing, then you can get more detail error info. That will help you figuring out the reason that initdb not working.
At 2024-05-23 16:08:05, "陈亚杰" <[email protected]> wrote:
Hello, I have a question about cross-compiling. I get an error when doing initdb for postgresql for arm64 architecture devices.
The error information is Error relocating /data/postgresql/postgresql-16.3-arm64-v8a-build/tmp_install/usr/postgresql/arm64-v8a/lib/dict_snowball.so: palloc0: symbol not found.
In fact, the library exists in this directory, and the palloc symbol exists but is not defined.
Any tips to go around this issue?
Thanks!
^ permalink raw reply [nested|flat] 72+ messages in thread
end of thread, other threads:[~2024-05-24 06:02 UTC | newest]
Thread overview: 72+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2017-03-02 00:12 Re: patch: function xmltable Alvaro Herrera <[email protected]>
2017-03-02 07:04 ` Pavel Stehule <[email protected]>
2017-03-02 08:13 ` Pavel Stehule <[email protected]>
2017-03-02 17:13 ` Alvaro Herrera <[email protected]>
2017-03-02 17:46 ` Pavel Stehule <[email protected]>
2017-03-02 18:32 ` Alvaro Herrera <[email protected]>
2017-03-02 19:17 ` Pavel Stehule <[email protected]>
2017-03-02 21:35 ` Alvaro Herrera <[email protected]>
2017-03-03 09:28 ` Pavel Stehule <[email protected]>
2017-03-03 18:15 ` Alvaro Herrera <[email protected]>
2017-03-03 18:42 ` Pavel Stehule <[email protected]>
2017-03-03 19:23 ` Pavel Stehule <[email protected]>
2017-03-03 20:04 ` Alvaro Herrera <[email protected]>
2017-03-03 21:41 ` Pavel Stehule <[email protected]>
2017-03-05 09:32 ` Pavel Stehule <[email protected]>
2017-03-08 16:01 ` Alvaro Herrera <[email protected]>
2017-03-08 16:10 ` Pavel Stehule <[email protected]>
2017-03-08 16:32 ` Alvaro Herrera <[email protected]>
2017-03-08 16:46 ` Pavel Stehule <[email protected]>
2017-03-08 17:01 ` Alvaro Herrera <[email protected]>
2021-02-16 13:54 [PATCH 01/12] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2021-03-04 11:03 [PATCH 3/8] Disallow compressed data inside container types Dilip Kumar <[email protected]>
2024-05-23 09:08 about cross-compiling issue =?utf-8?B?6ZmI5Lqa5p2w?= <[email protected]>
2024-05-23 15:10 ` Re: about cross-compiling issue Tom Lane <[email protected]>
2024-05-24 06:02 ` Re:about cross-compiling issue Long Song <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox