public inbox for [email protected]help / color / mirror / Atom feed
Re: Transparent column encryption 4+ messages / 3 participants [nested] [flat]
* Re: Transparent column encryption @ 2023-01-30 22:30 Jacob Champion <[email protected]> 0 siblings, 1 reply; 4+ messages in thread From: Jacob Champion @ 2023-01-30 22:30 UTC (permalink / raw) To: Peter Eisentraut <[email protected]>; +Cc: pgsql-hackers On Wed, Jan 25, 2023 at 11:00 AM Peter Eisentraut <[email protected]> wrote: > > When writing the paragraph on RSA-OAEP I was reminded that we didn't > > really dig into the asymmetric/symmetric discussion. Assuming that most > > first-time users will pick the builtin CMK encryption method, do we > > still want to have an asymmetric scheme implemented first instead of a > > symmetric keywrap? I'm still concerned about that public key, since it > > can't really be made public. > > I had started coding that, but one problem was that the openssl CLI > doesn't really provide any means to work with those kinds of keys. The > "openssl enc" command always wants to mix in a password. Without that, > there is no way to write a test case, and more crucially no way for > users to set up these kinds of keys. Unless we write our own tooling > for this, which, you know, the patch just passed 400k in size. Arrgh: https://github.com/openssl/openssl/issues/10605 > > The column encryption algorithm is set per-column -- but isn't it > > tightly coupled to the CEK, since the key length has to match? From a > > layperson perspective, using the same key to encrypt the same plaintext > > under two different algorithms (if they happen to have the same key > > length) seems like it might be cryptographically risky. Is there a > > reason I should be encouraged to do that? > > Not really. I was also initially confused by this setup, but that's how > other similar systems are set up, so I thought it would be confusing to > do it differently. Which systems let you mix and match keys and algorithms this way? I'd like to take a look at them. > > With the loss of \gencr it looks like we also lost a potential way to > > force encryption from within psql. Any plans to add that for v1? > > \gencr didn't do that either. We could do it. The libpq API supports > it. We just need to come up with some syntax for psql. Do you think people would rather set encryption for all parameters at once -- something like \encbind -- or have the ability to mix encrypted and unencrypted parameters? > > Are there plans to document client-side implementation requirements, to > > ensure cross-client compatibility? Things like the "PG\x00\x01" > > associated data are buried at the moment (or else I've missed them in > > the docs). If you're holding off until the feature is more finalized, > > that's fine too. > > This is documented in the protocol chapter, which I thought was the > right place. Did you want more documentation, or in a different place? I just missed it; sorry. > > Speaking of cross-client compatibility, I'm still disconcerted by the > > ability to write the value "hello world" into an encrypted integer > > column. Should clients be required to validate the text format, using > > the attrealtypid? > > Well, we can ask them to, but we can't really require them, in a > cryptographic sense. I'm not sure what more we can do. Right -- I just mean that clients need to pay more attention to it now, whereas before they may have delegated correctness to the server. The problem is documented in the context of deterministic encryption, but I think it applies to randomized as well. More concretely: should psql allow you to push arbitrary text into an encrypted \bind parameter, like it does now? > > It occurred to me when looking at the "unspecified" CMK scheme that the > > CEK doesn't really have to be an encryption key at all. In that case it > > can function more like a (possibly signed?) cookie for lookup, or even > > be ignored altogether if you don't want to use a wrapping scheme > > (similar to JWE's "direct" mode, maybe?). So now you have three ways to > > look up or determine a column encryption key (CMK realm, CMK name, CEK > > cookie)... is that a concept worth exploring in v1 and/or the documentation? > > I don't completely follow this. Yeah, I'm not expressing it very well. My feeling is that the organization system here -- a realm "contains" multiple CMKs, a CMK encrypts multiple CEKs -- is so general and flexible that it may need some suggested guardrails for people to use it sanely. I just don't know what those guardrails should be. I was motivated by the realization that CEKs don't even need to be keys. Thanks, --Jacob ^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: Transparent column encryption @ 2023-01-31 13:25 Peter Eisentraut <[email protected]> parent: Jacob Champion <[email protected]> 0 siblings, 1 reply; 4+ messages in thread From: Peter Eisentraut @ 2023-01-31 13:25 UTC (permalink / raw) To: Jacob Champion <[email protected]>; +Cc: pgsql-hackers On 30.01.23 23:30, Jacob Champion wrote: >>> The column encryption algorithm is set per-column -- but isn't it >>> tightly coupled to the CEK, since the key length has to match? From a >>> layperson perspective, using the same key to encrypt the same plaintext >>> under two different algorithms (if they happen to have the same key >>> length) seems like it might be cryptographically risky. Is there a >>> reason I should be encouraged to do that? >> >> Not really. I was also initially confused by this setup, but that's how >> other similar systems are set up, so I thought it would be confusing to >> do it differently. > > Which systems let you mix and match keys and algorithms this way? I'd > like to take a look at them. See here for example: https://learn.microsoft.com/en-us/sql/relational-databases/security/encryption/always-encrypted-data... >>> With the loss of \gencr it looks like we also lost a potential way to >>> force encryption from within psql. Any plans to add that for v1? >> >> \gencr didn't do that either. We could do it. The libpq API supports >> it. We just need to come up with some syntax for psql. > > Do you think people would rather set encryption for all parameters at > once -- something like \encbind -- or have the ability to mix > encrypted and unencrypted parameters? For pg_dump, I'd like a mode that makes all values parameters of an INSERT statement. But obviously not all of those will be encrypted. So I think we'd want a per-parameter syntax. > More concretely: should psql allow you to push arbitrary text into an > encrypted \bind parameter, like it does now? We don't have any data type awareness like that now in psql or libpq. It would be quite a change to start now. How would that deal with data type extensibility, is an obvious question to start with. Don't know. ^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: Transparent column encryption @ 2023-02-03 20:43 Jacob Champion <[email protected]> parent: Peter Eisentraut <[email protected]> 0 siblings, 0 replies; 4+ messages in thread From: Jacob Champion @ 2023-02-03 20:43 UTC (permalink / raw) To: Peter Eisentraut <[email protected]>; +Cc: pgsql-hackers On Tue, Jan 31, 2023 at 5:26 AM Peter Eisentraut <[email protected]> wrote: > See here for example: > https://learn.microsoft.com/en-us/sql/relational-databases/security/encryption/always-encrypted-data... Hm. I notice they haven't implemented more than one algorithm, so I wonder if they're going to be happy with their decision to mix-and-match when number two arrives. > For pg_dump, I'd like a mode that makes all values parameters of an > INSERT statement. But obviously not all of those will be encrypted. So > I think we'd want a per-parameter syntax. Makes sense. Maybe something that defaults to encrypted with opt-out per parameter? UPDATE t SET name = $1 WHERE id = $2 \encbind "Jacob" plaintext(24) > > More concretely: should psql allow you to push arbitrary text into an > > encrypted \bind parameter, like it does now? > > We don't have any data type awareness like that now in psql or libpq. > It would be quite a change to start now. I agree. It just feels weird that a misbehaving client can "attack" the other client implementations using it, and we don't make any attempt to mitigate it. --Jacob ^ permalink raw reply [nested|flat] 4+ messages in thread
* [PATCH v23 6/8] Row pattern recognition patch (docs). @ 2024-10-25 03:56 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 4+ messages in thread From: Tatsuo Ishii @ 2024-10-25 03:56 UTC (permalink / raw) --- doc/src/sgml/advanced.sgml | 82 ++++++++++++++++++++++++++++++++++++ doc/src/sgml/func.sgml | 54 ++++++++++++++++++++++++ doc/src/sgml/ref/select.sgml | 38 ++++++++++++++++- 3 files changed, 172 insertions(+), 2 deletions(-) diff --git a/doc/src/sgml/advanced.sgml b/doc/src/sgml/advanced.sgml index 755c9f1485..b0b1d1c51e 100644 --- a/doc/src/sgml/advanced.sgml +++ b/doc/src/sgml/advanced.sgml @@ -537,6 +537,88 @@ WHERE pos < 3; <literal>rank</literal> less than 3. </para> + <para> + Row pattern common syntax can be used to perform row pattern recognition + in a query. The row pattern common syntax includes two sub + clauses: <literal>DEFINE</literal> + and <literal>PATTERN</literal>. <literal>DEFINE</literal> defines + definition variables along with an expression. The expression must be a + logical expression, which means it must + return <literal>TRUE</literal>, <literal>FALSE</literal> + or <literal>NULL</literal>. The expression may comprise column references + and functions. Window functions, aggregate functions and subqueries are + not allowed. An example of <literal>DEFINE</literal> is as follows. + +<programlisting> +DEFINE + LOWPRICE AS price <= 100, + UP AS price > PREV(price), + DOWN AS price < PREV(price) +</programlisting> + + Note that <function>PREV</function> returns the price column in the + previous row if it's called in a context of row pattern recognition. Thus in + the second line the definition variable "UP" is <literal>TRUE</literal> + when the price column in the current row is greater than the price column + in the previous row. Likewise, "DOWN" is <literal>TRUE</literal> when when + the price column in the current row is lower than the price column in the + previous row. + </para> + <para> + Once <literal>DEFINE</literal> exists, <literal>PATTERN</literal> can be + used. <literal>PATTERN</literal> defines a sequence of rows that satisfies + certain conditions. For example following <literal>PATTERN</literal> + defines that a row starts with the condition "LOWPRICE", then one or more + rows satisfy "UP" and finally one or more rows satisfy "DOWN". Note that + "+" means one or more matches. Also you can use "*", which means zero or + more matches. If a sequence of rows which satisfies the PATTERN is found, + in the starting row of the sequence of rows all window functions and + aggregates are shown in the target list. Note that aggregations only look + into the matched rows, rather than whole frame. On the second or + subsequent rows all window functions are NULL. Aggregates are NULL or 0 + (count case) depending on its aggregation definition. For rows that do not + match on the PATTERN, all window functions and aggregates are shown AS + NULL too, except count showing 0. This is because the rows do not match, + thus they are in an empty frame. Example of a <literal>SELECT</literal> + using the <literal>DEFINE</literal> and <literal>PATTERN</literal> clause + is as follows. + +<programlisting> +SELECT company, tdate, price, + first_value(price) OVER w, + max(price) OVER w, + count(price) OVER w +FROM stock + WINDOW w AS ( + PARTITION BY company + ORDER BY tdate + ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING + AFTER MATCH SKIP PAST LAST ROW + INITIAL + PATTERN (LOWPRICE UP+ DOWN+) + DEFINE + LOWPRICE AS price <= 100, + UP AS price > PREV(price), + DOWN AS price < PREV(price) +); +</programlisting> +<screen> + company | tdate | price | first_value | max | count +----------+------------+-------+-------------+-----+------- + company1 | 2023-07-01 | 100 | 100 | 200 | 4 + company1 | 2023-07-02 | 200 | | | 0 + company1 | 2023-07-03 | 150 | | | 0 + company1 | 2023-07-04 | 140 | | | 0 + company1 | 2023-07-05 | 150 | | | 0 + company1 | 2023-07-06 | 90 | 90 | 130 | 4 + company1 | 2023-07-07 | 110 | | | 0 + company1 | 2023-07-08 | 130 | | | 0 + company1 | 2023-07-09 | 120 | | | 0 + company1 | 2023-07-10 | 130 | | | 0 +(10 rows) +</screen> + </para> + <para> When a query involves multiple window functions, it is possible to write out each one with a separate <literal>OVER</literal> clause, but this is diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 58dc06b68b..ef4fa7144f 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -23263,6 +23263,7 @@ SELECT count(*) FROM sometable; returns <literal>NULL</literal> if there is no such row. </para></entry> </row> + </tbody> </tgroup> </table> @@ -23302,6 +23303,59 @@ SELECT count(*) FROM sometable; Other frame specifications can be used to obtain other effects. </para> + <para> + Row pattern recognition navigation functions are listed in + <xref linkend="functions-rpr-navigation-table"/>. These functions + can be used to describe DEFINE clause of Row pattern recognition. + </para> + + <table id="functions-rpr-navigation-table"> + <title>Row Pattern Navigation Functions</title> + <tgroup cols="1"> + <thead> + <row> + <entry role="func_table_entry"><para role="func_signature"> + Function + </para> + <para> + Description + </para></entry> + </row> + </thead> + + <tbody> + <row> + <entry role="func_table_entry"><para role="func_signature"> + <indexterm> + <primary>prev</primary> + </indexterm> + <function>prev</function> ( <parameter>value</parameter> <type>anyelement</type> ) + <returnvalue>anyelement</returnvalue> + </para> + <para> + Returns the column value at the previous row; + returns NULL if there is no previous row in the window frame. + </para></entry> + </row> + + <row> + <entry role="func_table_entry"><para role="func_signature"> + <indexterm> + <primary>next</primary> + </indexterm> + <function>next</function> ( <parameter>value</parameter> <type>anyelement</type> ) + <returnvalue>anyelement</returnvalue> + </para> + <para> + Returns the column value at the next row; + returns NULL if there is no next row in the window frame. + </para></entry> + </row> + + </tbody> + </tgroup> + </table> + <note> <para> The SQL standard defines a <literal>RESPECT NULLS</literal> or diff --git a/doc/src/sgml/ref/select.sgml b/doc/src/sgml/ref/select.sgml index d7089eac0b..7e1c9989ba 100644 --- a/doc/src/sgml/ref/select.sgml +++ b/doc/src/sgml/ref/select.sgml @@ -969,8 +969,8 @@ WINDOW <replaceable class="parameter">window_name</replaceable> AS ( <replaceabl The <replaceable class="parameter">frame_clause</replaceable> can be one of <synopsis> -{ RANGE | ROWS | GROUPS } <replaceable>frame_start</replaceable> [ <replaceable>frame_exclusion</replaceable> ] -{ RANGE | ROWS | GROUPS } BETWEEN <replaceable>frame_start</replaceable> AND <replaceable>frame_end</replaceable> [ <replaceable>frame_exclusion</replaceable> ] +{ RANGE | ROWS | GROUPS } <replaceable>frame_start</replaceable> [ <replaceable>frame_exclusion</replaceable> ] [row_pattern_common_syntax] +{ RANGE | ROWS | GROUPS } BETWEEN <replaceable>frame_start</replaceable> AND <replaceable>frame_end</replaceable> [ <replaceable>frame_exclusion</replaceable> ] [row_pattern_common_syntax] </synopsis> where <replaceable>frame_start</replaceable> @@ -1077,6 +1077,40 @@ EXCLUDE NO OTHERS a given peer group will be in the frame or excluded from it. </para> + <para> + The + optional <replaceable class="parameter">row_pattern_common_syntax</replaceable> + defines the <firstterm>row pattern recognition condition</firstterm> for + this + window. <replaceable class="parameter">row_pattern_common_syntax</replaceable> + includes following subclauses. <literal>AFTER MATCH SKIP PAST LAST + ROW</literal> or <literal>AFTER MATCH SKIP TO NEXT ROW</literal> controls + how to proceed to next row position after a match + found. With <literal>AFTER MATCH SKIP PAST LAST ROW</literal> (the + default) next row position is next to the last row of previous match. On + the other hand, with <literal>AFTER MATCH SKIP TO NEXT ROW</literal> next + row position is always next to the last row of previous + match. <literal>DEFINE</literal> defines definition variables along with a + boolean expression. <literal>PATTERN</literal> defines a sequence of rows + that satisfies certain conditions using variables defined + in <literal>DEFINE</literal> clause. If the variable is not defined in + the <literal>DEFINE</literal> clause, it is implicitly assumed + following is defined in the <literal>DEFINE</literal> clause. + +<synopsis> +<literal>variable_name</literal> AS TRUE +</synopsis> + + Note that the maximu number of variables defined + in <literal>DEFINE</literal> clause is 26. + +<synopsis> +[ AFTER MATCH SKIP PAST LAST ROW | AFTER MATCH SKIP TO NEXT ROW ] +PATTERN <replaceable class="parameter">pattern_variable_name</replaceable>[+] [, ...] +DEFINE <replaceable class="parameter">definition_varible_name</replaceable> AS <replaceable class="parameter">expression</replaceable> [, ...] +</synopsis> + </para> + <para> The purpose of a <literal>WINDOW</literal> clause is to specify the behavior of <firstterm>window functions</firstterm> appearing in the query's -- 2.25.1 ----Next_Part(Fri_Oct_25_13_04_53_2024_648)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v23-0007-Row-pattern-recognition-patch-tests.patch" ^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2024-10-25 03:56 UTC | newest] Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2023-01-30 22:30 Re: Transparent column encryption Jacob Champion <[email protected]> 2023-01-31 13:25 ` Peter Eisentraut <[email protected]> 2023-02-03 20:43 ` Jacob Champion <[email protected]> 2024-10-25 03:56 [PATCH v23 6/8] Row pattern recognition patch (docs). Tatsuo Ishii <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox