Feedback-ID: i131946ab:Fastmail
Message-ID: <b6f978bc-4d13-fd61-742b-69d3835b8c26@enterprisedb.com>
Date: Tue, 21 Mar 2023 18:05:15 +0100
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.9.0
Subject: Re: Transparent column encryption
Content-Language: en-US
To: Andres Freund <andres@anarazel.de>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
References: <CALDaNm393nCjM1GenCem=9kOfZxbSadsTSn48WDVomwPnfnCFw@mail.gmail.com>
 <1034b6f1-ccab-8a22-c843-71104ebedf01@enterprisedb.com>
 <40c43d0d-ae4b-fe34-2667-771b3718384e@enterprisedb.com>
 <47e17f03-6595-6c09-8978-7a1581cad6f2@enterprisedb.com>
 <7a430d89-233b-5cd7-6892-9fed06695d39@enterprisedb.com>
 <20230312001125.ijeu4imbwlhj5flf@awork3.anarazel.de>
 <43a88695-5e0b-4379-94cf-0392fdea4be9@enterprisedb.com>
 <20230313204119.4mkepdvixcxrwpsc@awork3.anarazel.de>
 <20230313211108.zge6vnqukexkih62@awork3.anarazel.de>
 <c2758429-b404-5feb-6ac4-2ced0b3ad755@enterprisedb.com>
 <20230316163648.q3wgjinmad6225t3@awork3.anarazel.de>
From: Peter Eisentraut <peter.eisentraut@enterprisedb.com>
In-Reply-To: <20230316163648.q3wgjinmad6225t3@awork3.anarazel.de>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Archived-At: <https://www.postgresql.org/message-id/b6f978bc-4d13-fd61-742b-69d3835b8c26%40enterprisedb.com>
Precedence: bulk

On 16.03.23 17:36, Andres Freund wrote:
> Maybe a daft question, but why do we need a separate type and typmod for
> encrypted columns? Why isn't the fact that the column is encrypted exactly one
> new field, and we use the existing type/typmod fields?

The way this is implemented is that for an encrypted column, the real 
atttypid and atttypmod are one of the encrypted special types 
(pg_encrypted_*).  That way, most of the system doesn't need to care 
about the details of encryption or whatnot, it just unpacks tuples etc. 
by looking at atttypid, atttyplen, etc., and queries on encrypted data 
behave normally by just looking at what operators etc. those types have. 
  This approach heavily contains the number of places that need to know 
about this feature at all.

>> Do we need to decouple tuple descriptors from pg_attribute altogether?
> 
> Yes. Very clearly. The amount of memory and runtime we spent on tupledescs is
> disproportionate. A second angle is that we build tupledescs way way too
> frequently. The executor infers them everywhere, so not even prepared
> statements protect against that.
> 
> 
>> How do we decide what goes into the tuple descriptor and what does not?  I'm
>> interested in addressing this, because obviously we do want the ability to
>> add more features in the future, but I don't know what the direction should
>> be.
> 
> We've had some prior discussion around this, see e.g.
> https://postgr.es/m/20210819114435.6r532qbadcsyfscp%40alap3.anarazel.de

This sounds like a good plan.