public inbox for [email protected]
help / color / mirror / Atom feedRe: Super PathKeys (Allowing sort order through precision loss functions)
88+ messages / 14 participants
[nested] [flat]
* Re: Super PathKeys (Allowing sort order through precision loss functions)
@ 2018-11-01 01:40 David Rowley <[email protected]>
0 siblings, 1 reply; 88+ messages in thread
From: David Rowley @ 2018-11-01 01:40 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Robert Haas <[email protected]>; Tom Lane <[email protected]>; PostgreSQL Hackers <[email protected]>
On 1 November 2018 at 12:24, Andres Freund <[email protected]> wrote:
> FWIW, I kind of wonder if we built proper infrastructure to allow to
> make such inferrences from function calls, whether it could also be made
> to support the transformation of LIKEs into indexable <= >= clauses.
Perhaps, but I doubt it would be the same function to do both. Surely
I need something that accepts details about the function call as
arguments and returns an Expr * of the argument that we can derive the
order of the return value from, or NULL. I think the transformation
you need might be more along the lines of returning a List * of quals
that can substitute an OpExpr containing a function call. I'm not that
clear on how we'd know the new quals were better than the existing
ones. For example extract('year', dt) = 2018 could be transformed to
dt >= '2018-01-01' AND dt < '2019-01-01', but how would we know that
was better. There might be an index on extract('year', dt).
--
David Rowley http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: Super PathKeys (Allowing sort order through precision loss functions)
@ 2018-11-01 09:44 Tomas Vondra <[email protected]>
parent: David Rowley <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Tomas Vondra @ 2018-11-01 09:44 UTC (permalink / raw)
To: David Rowley <[email protected]>; Andres Freund <[email protected]>; +Cc: Robert Haas <[email protected]>; Tom Lane <[email protected]>; PostgreSQL Hackers <[email protected]>
On 11/01/2018 02:40 AM, David Rowley wrote:
> On 1 November 2018 at 12:24, Andres Freund <[email protected]> wrote:
>> FWIW, I kind of wonder if we built proper infrastructure to allow to
>> make such inferrences from function calls, whether it could also be made
>> to support the transformation of LIKEs into indexable <= >= clauses.
>
> Perhaps, but I doubt it would be the same function to do both. Surely
> I need something that accepts details about the function call as
> arguments and returns an Expr * of the argument that we can derive the
> order of the return value from, or NULL. I think the transformation
> you need might be more along the lines of returning a List * of quals
> that can substitute an OpExpr containing a function call. I'm not that
> clear on how we'd know the new quals were better than the existing
> ones. For example extract('year', dt) = 2018 could be transformed to
> dt >= '2018-01-01' AND dt < '2019-01-01', but how would we know that
> was better. There might be an index on extract('year', dt).
>
IMHO there is only a handful of "useful" transformations of this kind,
depending on the operator class of an index. So when building index
paths and checking which quals may be used as index conditions, we'd do
try transforming incompatible quals and leave the rest up to the
existing create_index_path machinery (which already makes the decision
about which quals to use for index search etc.)
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: Changing the autovacuum launcher scheduling; oldest table first algorithm
@ 2018-11-29 17:21 Dmitry Dolgov <[email protected]>
0 siblings, 1 reply; 88+ messages in thread
From: Dmitry Dolgov @ 2018-11-29 17:21 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Michael Paquier <[email protected]>; [email protected]; pgsql-hackers; [email protected]; [email protected]
> On Tue, Oct 2, 2018 at 4:42 AM Michael Paquier <[email protected]> wrote:
>
> On Thu, Jun 28, 2018 at 04:20:53PM +0900, Masahiko Sawada wrote:
> > If there is an up-to-date information meaning either that there is no
> > tables needing vacuum or that there is only table needing vacuum but
> > being vacuumed by other worker, AV launcher can launches new one to
> > other database.
>
> I am not completely sure what we want to do with this patch in
> particular as there are many approaches and things which can be
> discussed. For now, the latest patch proposed does not apply, so I am
> moving it to next CF, waiting for its author.
Nothing changed since then, but also the patch got not enough review to say
that there was substantial feedback. I'll move it to the next CF.
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: Changing the autovacuum launcher scheduling; oldest table first algorithm
@ 2018-11-30 01:48 Michael Paquier <[email protected]>
parent: Dmitry Dolgov <[email protected]>
0 siblings, 1 reply; 88+ messages in thread
From: Michael Paquier @ 2018-11-30 01:48 UTC (permalink / raw)
To: Dmitry Dolgov <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; [email protected]; pgsql-hackers; [email protected]; [email protected]
On Thu, Nov 29, 2018 at 06:21:34PM +0100, Dmitry Dolgov wrote:
> Nothing changed since then, but also the patch got not enough review to say
> that there was substantial feedback. I'll move it to the next CF.
I would have suggested to mark the patch as returned with feedback
instead as the thing does not apply for some time now, and the author
has been notified about a rebase.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: Changing the autovacuum launcher scheduling; oldest table first algorithm
@ 2018-11-30 02:00 Masahiko Sawada <[email protected]>
parent: Michael Paquier <[email protected]>
0 siblings, 1 reply; 88+ messages in thread
From: Masahiko Sawada @ 2018-11-30 02:00 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Dmitry Dolgov <[email protected]>; Alvaro Herrera <[email protected]>; pgsql-hackers; Tsunakawa, Takayuki <[email protected]>; Ildus Kurbangaliev <[email protected]>
On Fri, Nov 30, 2018 at 10:48 AM Michael Paquier <[email protected]> wrote:
>
> On Thu, Nov 29, 2018 at 06:21:34PM +0100, Dmitry Dolgov wrote:
> > Nothing changed since then, but also the patch got not enough review to say
> > that there was substantial feedback. I'll move it to the next CF.
>
> I would have suggested to mark the patch as returned with feedback
> instead as the thing does not apply for some time now, and the author
> has been notified about a rebase.
Agreed and sorry for the late reply.
The design of this patch would need to be reconsidered based on
suggestions and discussions we had before. I'll propose it again after
considerations.
Regards,
--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: Changing the autovacuum launcher scheduling; oldest table first algorithm
@ 2018-11-30 09:40 Dmitry Dolgov <[email protected]>
parent: Masahiko Sawada <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Dmitry Dolgov @ 2018-11-30 09:40 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Michael Paquier <[email protected]>; [email protected]; pgsql-hackers; [email protected]; [email protected]
> On Fri, Nov 30, 2018 at 3:05 AM Masahiko Sawada <[email protected]> wrote:
>
> On Fri, Nov 30, 2018 at 10:48 AM Michael Paquier <[email protected]> wrote:
> >
> > On Thu, Nov 29, 2018 at 06:21:34PM +0100, Dmitry Dolgov wrote:
> > > Nothing changed since then, but also the patch got not enough review to say
> > > that there was substantial feedback. I'll move it to the next CF.
> >
> > I would have suggested to mark the patch as returned with feedback
> > instead as the thing does not apply for some time now, and the author
> > has been notified about a rebase.
>
> Agreed and sorry for the late reply.
>
> The design of this patch would need to be reconsidered based on
> suggestions and discussions we had before. I'll propose it again after
> considerations.
Well, taking into account this information, then yes, it makes sense and I'll
mark it as "Returned with feedback", thanks!
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v12 03/11] Document historic behavior about hiding directories and special files
@ 2020-03-06 23:12 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Justin Pryzby @ 2020-03-06 23:12 UTC (permalink / raw)
Should backpatch to v12: tmpdir
---
doc/src/sgml/func.sgml | 1 +
1 file changed, 1 insertion(+)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index ace95fe661..2c6142a0e0 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21489,6 +21489,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
default only superusers and members of the <literal>pg_monitor</literal>
role can use this function. Access may be granted to others using
<command>GRANT</command>.
+ Filenames beginning with a dot, directories, and other special files are not shown.
</para>
<indexterm>
--
2.17.0
--wIc/V6YLA2QdyfT4
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v12-0004-Add-tests-exercizing-pg_ls_tmpdir.patch"
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: [PATCH] Compression dictionaries for JSONB
@ 2022-04-22 15:21 Zhihong Yu <[email protected]>
0 siblings, 1 reply; 88+ messages in thread
From: Zhihong Yu @ 2022-04-22 15:21 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>; Alvaro Herrera <[email protected]>; Matthias van de Meent <[email protected]>
On Fri, Apr 22, 2022 at 1:30 AM Aleksander Alekseev <
[email protected]> wrote:
> Hi hackers,
>
> This is a follow-up thread to `RFC: compression dictionaries for JSONB`
> [1]. I would like to share my current progress in order to get early
> feedback. The patch is currently in a draft state but implements the basic
> functionality. I did my best to account for all the great feedback I
> previously got from Alvaro and Matthias.
>
> Usage example:
>
> ```
> CREATE TYPE mydict AS DICTIONARY OF jsonb ('aaa', 'bbb');
>
> SELECT '{"aaa":"bbb"}' :: mydict;
> mydict
> ----------------
> {"aaa": "bbb"}
>
> SELECT ('{"aaa":"bbb"}' :: mydict) -> 'aaa';
> ?column?
> ----------
> "bbb"
> ```
>
> Here `mydict` works as a transparent replacement for `jsonb`. However, its
> internal representation differs. The provided dictionary entries ('aaa',
> 'bbb') are stored in the new catalog table:
>
> ```
> SELECT * FROM pg_dict;
> oid | dicttypid | dictentry
> -------+-----------+-----------
> 39476 | 39475 | aaa
> 39477 | 39475 | bbb
> (2 rows)
> ```
>
> When `mydict` sees 'aaa' in the document, it replaces it with the
> corresponding code, in this case - 39476. For more details regarding the
> compression algorithm and choosen compromises please see the comments in
> the patch.
>
> In pg_type `mydict` has typtype = TYPTYPE_DICT. It works the same way as
> TYPTYPE_BASE with only difference: corresponding `<type>_in`
> (pg_type.typinput) and `<another-type>_<type>` (pg_cast.castfunc)
> procedures receive the dictionary Oid as a `typmod` argument. This way the
> procedures can distinguish `mydict1` from `mydict2` and use the proper
> compression dictionary.
>
> The approach with alternative `typmod` role is arguably a bit hacky, but
> it was the less invasive way to implement the feature I've found. I'm open
> to alternative suggestions.
>
> Current limitations (todo):
> - ALTER TYPE is not implemented
> - Tests and documentation are missing
> - Autocomplete is missing
>
> Future work (out of scope of this patch):
> - Support types other than JSONB: TEXT, XML, etc
> - Automatically updated dictionaries, e.g. during VACUUM
> - Alternative compression algorithms. Note that this will not require any
> further changes in the catalog, only the values we write to pg_type and
> pg_cast will differ.
>
> Open questions:
> - Dictionary entries are currently stored as NameData, the same type that
> is used for enums. Are we OK with the accompanying limitations? Any
> alternative suggestions?
> - All in all, am I moving the right direction?
>
> Your feedback is very much welcomed!
>
> [1]:
> https://postgr.es/m/CAJ7c6TPx7N-bVw0dZ1ASCDQKZJHhBYkT6w4HV1LzfS%2BUUTUfmA%40mail.gmail.com
>
> --
> Best regards,
> Aleksander Alekseev
>
Hi,
For src/backend/catalog/pg_dict.c, please add license header.
+ elog(ERROR, "skipbytes > decoded_size - outoffset");
Include the values for skipbytes, decoded_size and outoffset.
Cheers
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: [PATCH] Compression dictionaries for JSONB
@ 2022-04-25 13:15 Aleksander Alekseev <[email protected]>
parent: Zhihong Yu <[email protected]>
0 siblings, 1 reply; 88+ messages in thread
From: Aleksander Alekseev @ 2022-04-25 13:15 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Zhihong Yu <[email protected]>; Matthias van de Meent <[email protected]>
Hi Zhihong,
Many thanks for your feedback!
> For src/backend/catalog/pg_dict.c, please add license header.
Fixed.
> + elog(ERROR, "skipbytes > decoded_size - outoffset");
>
> Include the values for skipbytes, decoded_size and outoffset.
In fact, this code should never be executed, and if somehow it will
be, this information will not help us much to debug the issue. I made
corresponding changes to the error message and added the comments.
Here it the 2nd version of the patch:
- Includes changes named above
- Fixes a warning reported by cfbot
- Fixes some FIXME's
- The path includes some simple tests now
- A proper commit message was added
Please note that this is still a draft. Feedback is welcome.
--
Best regards,
Aleksander Alekseev
Attachments:
[application/octet-stream] v2-0001-Compression-dictionaries-for-JSONB.patch (49.8K, ../../CAJ7c6TPvcXhqsLEcuJBV96it8Zh_YbbxJ5F2ugZFySypStaoLA@mail.gmail.com/2-v2-0001-Compression-dictionaries-for-JSONB.patch)
download | inline diff:
From 56436bf15f43b349366fcdea962cf812777e0954 Mon Sep 17 00:00:00 2001
From: Aleksander Alekseev <[email protected]>
Date: Tue, 12 Apr 2022 15:58:11 +0300
Subject: [PATCH v2] Compression dictionaries for JSONB
Usage example:
CREATE TYPE mydict AS DITIONARY OF jsonb ('aaa', 'bbb', ...);
SELECT ('{"aaa":"bbb"}' :: mydict) -> 'aaa';
The created type works as a transparent replacement for JSONB. However,
its internal representation differs. The provided dictionary entries ('aaa',
'bbb', ..) are stored in pg_dict catalog table. When `mydict` sees one of
the entries in the document, it replaces it with the corresponding pg_dict
entry Oid. For more details regarding the compression algorithm and choosen
compromises please see the comments in the code.
In pg_type `mydict` has typtype = TYPTYPE_DICT. It works the same way as
TYPTYPE_BASE with only difference: corresponding `<type>_in`
(pg_type.typinput) and `<another-type>_<type>` (pg_cast.castfunc)
procedures receive the dictionary Oid as a `typmod` argument. This way the
procedures can distinguish `mydict1` from `mydict2` and use the proper
compression dictionary.
The approach with alternative `typmod` role is arguably a bit hacky, but it
was the least invasive way to implement the feature we've come up with.
(NOTE: dear committer, please bump the catalog version!)
Author: Aleksander Alekseev <[email protected]>
Reviewed-by: FIXME
Discussion: https://postgr.es/m/CAJ7c6TOtAB0z1UrksvGTStNE-herK-43bj22%3D5xVBg7S4vr5rQ%40mail.gmail.com
Discussion: https://postgr.es/m/CAJ7c6TPx7N-bVw0dZ1ASCDQKZJHhBYkT6w4HV1LzfS%2BUUTUfmA%40mail.gmail.com
---
src/backend/catalog/Makefile | 3 +-
src/backend/catalog/pg_dict.c | 175 +++++++
src/backend/commands/typecmds.c | 151 +++++-
src/backend/executor/functions.c | 1 +
src/backend/nodes/copyfuncs.c | 15 +
src/backend/nodes/equalfuncs.c | 13 +
src/backend/parser/gram.y | 8 +
src/backend/parser/parse_coerce.c | 17 +-
src/backend/parser/parse_type.c | 16 +
src/backend/tcop/utility.c | 13 +
src/backend/utils/adt/Makefile | 1 +
src/backend/utils/adt/dictionaries.c | 474 ++++++++++++++++++
src/backend/utils/fmgr/funcapi.c | 1 +
src/bin/pg_dump/pg_dump.c | 2 +
src/include/catalog/pg_dict.h | 76 +++
src/include/catalog/pg_proc.dat | 20 +
src/include/catalog/pg_type.h | 1 +
src/include/commands/typecmds.h | 1 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 12 +
src/pl/plpgsql/src/pl_comp.c | 1 +
.../modules/test_oat_hooks/test_oat_hooks.c | 1 +
src/test/regress/expected/dict.out | 36 ++
src/test/regress/expected/oidjoins.out | 1 +
src/test/regress/expected/opr_sanity.out | 14 +-
src/test/regress/parallel_schedule | 2 +-
src/test/regress/sql/dict.sql | 14 +
27 files changed, 1059 insertions(+), 11 deletions(-)
create mode 100644 src/backend/catalog/pg_dict.c
create mode 100644 src/backend/utils/adt/dictionaries.c
create mode 100644 src/include/catalog/pg_dict.h
create mode 100644 src/test/regress/expected/dict.out
create mode 100644 src/test/regress/sql/dict.sql
diff --git a/src/backend/catalog/Makefile b/src/backend/catalog/Makefile
index 89a0221ec9..3b4330148e 100644
--- a/src/backend/catalog/Makefile
+++ b/src/backend/catalog/Makefile
@@ -33,6 +33,7 @@ OBJS = \
pg_conversion.o \
pg_db_role_setting.o \
pg_depend.o \
+ pg_dict.o \
pg_enum.o \
pg_inherits.o \
pg_largeobject.o \
@@ -61,7 +62,7 @@ CATALOG_HEADERS := \
pg_language.h pg_largeobject_metadata.h pg_largeobject.h pg_aggregate.h \
pg_statistic.h pg_statistic_ext.h pg_statistic_ext_data.h \
pg_rewrite.h pg_trigger.h pg_event_trigger.h pg_description.h \
- pg_cast.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \
+ pg_cast.h pg_dict.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \
pg_database.h pg_db_role_setting.h pg_tablespace.h \
pg_authid.h pg_auth_members.h pg_shdepend.h pg_shdescription.h \
pg_ts_config.h pg_ts_config_map.h pg_ts_dict.h \
diff --git a/src/backend/catalog/pg_dict.c b/src/backend/catalog/pg_dict.c
new file mode 100644
index 0000000000..ef76e565cd
--- /dev/null
+++ b/src/backend/catalog/pg_dict.c
@@ -0,0 +1,175 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_dict.c
+ * routines to support manipulation of the pg_dict relation
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/catalog/pg_dict.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "access/genam.h"
+#include "access/skey.h"
+#include "access/table.h"
+#include "catalog/catalog.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_dict.h"
+#include "catalog/pg_type.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/rel.h"
+
+/*
+ * Creates an entry in pg_dict for each of the supplied values.
+ * vals is a list of String values.
+ */
+void
+DictEntriesCreate(Oid dictTypeOid, List *vals)
+{
+ Relation pg_dict;
+ NameData dictentry;
+ Datum values[Natts_pg_dict];
+ bool nulls[Natts_pg_dict];
+ ListCell *lc;
+ HeapTuple tup;
+
+ if (vals == NIL)
+ {
+ /* The list is empty; do nothing. */
+ return;
+ }
+
+ memset(nulls, false, sizeof(nulls));
+
+ /*
+ * We don't check the list of values for duplicates here. If there are
+ * any, the user will get an unique-index violation.
+ */
+
+ pg_dict = table_open(DictRelationId, RowExclusiveLock);
+ foreach(lc, vals)
+ {
+ Oid oid = GetNewOidWithIndex(pg_dict, DictOidIndexId,
+ Anum_pg_dict_oid);
+ char *entry = strVal(lfirst(lc));
+
+ /*
+ * Entries are stored in a name field, for easier syscache lookup, so
+ * check the length to make sure it's within range.
+ */
+ if (strlen(entry) > (NAMEDATALEN - 1))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_NAME),
+ errmsg("invalid dict entry \"%s\"", entry),
+ errdetail("Entries must be %d bytes or less.",
+ NAMEDATALEN - 1)));
+
+ namestrcpy(&dictentry, entry);
+
+ values[Anum_pg_dict_oid - 1] = ObjectIdGetDatum(oid);
+ values[Anum_pg_dict_dicttypid - 1] = ObjectIdGetDatum(dictTypeOid);
+ values[Anum_pg_dict_dictentry - 1] = NameGetDatum(&dictentry);
+
+ tup = heap_form_tuple(RelationGetDescr(pg_dict), values, nulls);
+
+ CatalogTupleInsert(pg_dict, tup);
+ heap_freetuple(tup);
+ }
+
+ /* clean up */
+ table_close(pg_dict, RowExclusiveLock);
+}
+
+/*
+ * Returns all the entries for the dictinary with given Oid. Entries are sorted
+ * by dictentry. Note that shorter entries are considered smaller, i.e. 'abc'
+ * goes before 'abcdef'. The memory is allocated in the caller's memory context.
+ *
+ * If there are no entries a valid but empty dictionary is returned.
+ */
+Dictionary
+DictEntriesRead(Oid dictTypeOid)
+{
+ Relation pg_dict;
+ ScanKeyData key[1];
+ SysScanDesc scan;
+ HeapTuple tup;
+ uint32 entries_allocated = 8;
+ Dictionary dict = (Dictionary)palloc(sizeof(DictionaryData));
+
+ dict->nentries = 0;
+ dict->entries = (DictEntry*)palloc(entries_allocated*sizeof(DictEntry));
+
+ pg_dict = table_open(DictRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_dict_dicttypid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(dictTypeOid));
+
+ scan = systable_beginscan(pg_dict, DictTypIdEntryIndexId, true,
+ NULL, 1, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ if(dict->nentries == entries_allocated)
+ {
+ entries_allocated = entries_allocated * 2;
+ dict->entries = (DictEntry*)repalloc(dict->entries, entries_allocated*sizeof(DictEntry));
+ }
+
+ dict->entries[dict->nentries].oid = ((Form_pg_dict) GETSTRUCT(tup))->oid;
+ dict->entries[dict->nentries].dictentry = ((Form_pg_dict) GETSTRUCT(tup))->dictentry;
+ /*
+ * This is arguably not the fastest way to determine the length of
+ * the entry. Alternatively, we could store a precalculated value in
+ * the catalog. However, usually it's a good idea to keep things simple
+ * until somebody discovers a bottleneck in this exact place and
+ * proposes a concrete fix.
+ */
+ dict->entries[dict->nentries].length = (uint32)strlen(dict->entries[dict->nentries].dictentry.data);
+ dict->nentries++;
+ }
+
+ systable_endscan(scan);
+ table_close(pg_dict, RowExclusiveLock);
+
+ return dict;
+}
+
+/*
+ * Deletes all the entries for the dictinary with given Oid.
+ */
+void
+DictEntriesDelete(Oid dictTypeOid)
+{
+ Relation pg_dict;
+ ScanKeyData key[1];
+ SysScanDesc scan;
+ HeapTuple tup;
+
+ pg_dict = table_open(DictRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_dict_dicttypid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(dictTypeOid));
+
+ scan = systable_beginscan(pg_dict, DictTypIdEntryIndexId, true,
+ NULL, 1, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ CatalogTupleDelete(pg_dict, &tup->t_self);
+ }
+
+ systable_endscan(scan);
+ table_close(pg_dict, RowExclusiveLock);
+}
diff --git a/src/backend/commands/typecmds.c b/src/backend/commands/typecmds.c
index 9b92b04242..5bbbbe09af 100644
--- a/src/backend/commands/typecmds.c
+++ b/src/backend/commands/typecmds.c
@@ -46,6 +46,7 @@
#include "catalog/pg_collation.h"
#include "catalog/pg_constraint.h"
#include "catalog/pg_depend.h"
+#include "catalog/pg_dict.h"
#include "catalog/pg_enum.h"
#include "catalog/pg_language.h"
#include "catalog/pg_namespace.h"
@@ -679,6 +680,14 @@ RemoveTypeById(Oid typeOid)
if (((Form_pg_type) GETSTRUCT(tup))->typtype == TYPTYPE_RANGE)
RangeDelete(typeOid);
+ /*
+ * If it is a dictionary type, delete the pg_dict entries too; we don't
+ * bother with making a dependency entry for that, so it has to be done
+ * "by hand" here.
+ */
+ if (((Form_pg_type) GETSTRUCT(tup))->typtype == TYPTYPE_DICT)
+ DictEntriesDelete(typeOid);
+
ReleaseSysCache(tup);
table_close(relation, RowExclusiveLock);
@@ -763,8 +772,8 @@ DefineDomain(CreateDomainStmt *stmt)
/*
* Base type must be a plain base type, a composite type, another domain,
- * an enum or a range type. Domains over pseudotypes would create a
- * security hole. (It would be shorter to code this to just check for
+ * a dict, an enum or a range type. Domains over pseudotypes would create
+ * a security hole. (It would be shorter to code this to just check for
* pseudotypes; but it seems safer to call out the specific typtypes that
* are supported, rather than assume that all future typtypes would be
* automatically supported.)
@@ -772,6 +781,7 @@ DefineDomain(CreateDomainStmt *stmt)
typtype = baseType->typtype;
if (typtype != TYPTYPE_BASE &&
typtype != TYPTYPE_COMPOSITE &&
+ typtype != TYPTYPE_DICT &&
typtype != TYPTYPE_DOMAIN &&
typtype != TYPTYPE_ENUM &&
typtype != TYPTYPE_RANGE &&
@@ -1129,6 +1139,143 @@ DefineDomain(CreateDomainStmt *stmt)
}
+/*
+ * DefineDictionary
+ * Registers a new dictionary.
+ */
+ObjectAddress
+DefineDictionary(CreateDictionaryStmt * stmt)
+{
+ char *dictName;
+ char *dictArrayName;
+ Oid dictNamespace;
+ AclResult aclresult;
+ Oid old_type_oid;
+ Oid dictArrayOid;
+ ObjectAddress dictTypeAddr;
+
+ /* Convert list of names to a name and namespace */
+ dictNamespace = QualifiedNameGetCreationNamespace(stmt->typeName,
+ &dictName);
+
+ /* Check we have creation rights in target namespace */
+ aclresult = pg_namespace_aclcheck(dictNamespace, GetUserId(), ACL_CREATE);
+ if (aclresult != ACLCHECK_OK)
+ aclcheck_error(aclresult, OBJECT_SCHEMA,
+ get_namespace_name(dictNamespace));
+
+ /* AALEKSEEV TODO make sure this is allowed only for JSONB */
+
+ /*
+ * Check for collision with an existing type name. If there is one and
+ * it's an autogenerated array, we can rename it out of the way.
+ */
+ old_type_oid = GetSysCacheOid2(TYPENAMENSP, Anum_pg_type_oid,
+ CStringGetDatum(dictName),
+ ObjectIdGetDatum(dictNamespace));
+ if (OidIsValid(old_type_oid))
+ {
+ if (!moveArrayTypeName(old_type_oid, dictName, dictNamespace))
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("type \"%s\" already exists", dictName)));
+ }
+
+ /* Allocate OID for array type */
+ dictArrayOid = AssignTypeArrayOid();
+
+ /* Create the pg_type entry */
+ dictTypeAddr =
+ TypeCreate(InvalidOid, /* no predetermined type OID */
+ dictName, /* type name */
+ dictNamespace, /* namespace */
+ InvalidOid, /* relation oid (n/a here) */
+ 0, /* relation kind (ditto) */
+ GetUserId(), /* owner's ID */
+ -1, /* internal size: varlena, as for JSONB */
+ TYPTYPE_DICT, /* type-type: dictionary */
+ TYPCATEGORY_USER, /* type-category: user, as for JSONB */
+ false, /* dict types are never preferred */
+ DEFAULT_TYPDELIM, /* array element delimiter */
+ F_DICTIONARY_IN, /* input procedure */
+ F_DICTIONARY_OUT, /* output procedure */
+ InvalidOid, /* receive procedure: none */
+ InvalidOid, /* send procedure: none */
+ InvalidOid, /* typmodin procedure - none */
+ InvalidOid, /* typmodout procedure - none */
+ InvalidOid, /* analyze procedure - default */
+ InvalidOid, /* subscript procedure - none */
+ InvalidOid, /* element type ID */
+ false, /* this is not an array type */
+ dictArrayOid, /* array type we are about to create */
+ InvalidOid, /* base type ID (only for domains) */
+ NULL, /* never a default type value */
+ NULL, /* binary default isn't sent either */
+ false, /* passed by value: same as for JSONB */
+ TYPALIGN_INT, /* int alignment */
+ TYPSTORAGE_EXTENDED, /* TOAST strategy: fully toastable, as
+ * JSONB */
+ -1, /* typMod (Domains only) */
+ 0, /* Array dimensions of typbasetype */
+ false, /* Type NOT NULL */
+ InvalidOid); /* type's collation */
+
+ /*
+ * Create the array type that goes with it.
+ */
+ dictArrayName = makeArrayTypeName(dictName, dictNamespace);
+
+ TypeCreate(dictArrayOid, /* force assignment of this type OID */
+ dictArrayName, /* type name */
+ dictNamespace, /* namespace */
+ InvalidOid, /* relation oid (n/a here) */
+ 0, /* relation kind (ditto) */
+ GetUserId(), /* owner's ID */
+ -1, /* internal size (always varlena) */
+ TYPTYPE_BASE, /* type-type (base type) */
+ TYPCATEGORY_ARRAY, /* type-category (array) */
+ false, /* array types are never preferred */
+ DEFAULT_TYPDELIM, /* array element delimiter */
+ F_ARRAY_IN, /* input procedure */
+ F_ARRAY_OUT, /* output procedure */
+ F_ARRAY_RECV, /* receive procedure */
+ F_ARRAY_SEND, /* send procedure */
+ InvalidOid, /* typmodin procedure - none */
+ InvalidOid, /* typmodout procedure - none */
+ F_ARRAY_TYPANALYZE, /* analyze procedure */
+ F_ARRAY_SUBSCRIPT_HANDLER, /* array subscript procedure */
+ dictTypeAddr.objectId, /* element type ID */
+ true, /* yes this is an array type */
+ InvalidOid, /* no further array type */
+ InvalidOid, /* base type ID */
+ NULL, /* never a default type value */
+ NULL, /* binary default isn't sent either */
+ false, /* passed by value: same as for JSONB */
+ TYPALIGN_INT, /* enums have int align, so do their arrays */
+ TYPSTORAGE_EXTENDED, /* ARRAY is always toastable */
+ -1, /* typMod (Domains only) */
+ 0, /* Array dimensions of typbasetype */
+ false, /* Type NOT NULL */
+ InvalidOid); /* type's collation */
+
+ pfree(dictArrayName);
+
+ /* Enter the dict's entries into pg_dict */
+ DictEntriesCreate(dictTypeAddr.objectId, stmt->vals);
+
+ /* Create casts to and from JSONB */
+ CastCreate(JSONBOID, dictTypeAddr.objectId, F_JSONB_DICTIONARY, 'a', 'f', DEPENDENCY_INTERNAL);
+ CastCreate(dictTypeAddr.objectId, JSONBOID, F_DICTIONARY_JSONB, 'i', 'f', DEPENDENCY_INTERNAL);
+
+ /*
+ * Create explicit cast to bytea. This is convenient for debugging purposes.
+ * Casting bytea to a dictionary type is dangerous and thus not supported.
+ */
+ CastCreate(dictTypeAddr.objectId, BYTEAOID, F_DICTIONARY_BYTEA, 'e', 'f', DEPENDENCY_INTERNAL);
+
+ return dictTypeAddr;
+}
+
/*
* DefineEnum
* Registers a new enum.
diff --git a/src/backend/executor/functions.c b/src/backend/executor/functions.c
index f9460ae506..7ad3de3a00 100644
--- a/src/backend/executor/functions.c
+++ b/src/backend/executor/functions.c
@@ -1708,6 +1708,7 @@ check_sql_fn_retval(List *queryTreeLists,
if (fn_typtype == TYPTYPE_BASE ||
fn_typtype == TYPTYPE_DOMAIN ||
+ fn_typtype == TYPTYPE_DICT ||
fn_typtype == TYPTYPE_ENUM ||
fn_typtype == TYPTYPE_RANGE ||
fn_typtype == TYPTYPE_MULTIRANGE)
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 836f427ea8..04b9d573f4 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4440,6 +4440,18 @@ _copyCompositeTypeStmt(const CompositeTypeStmt *from)
return newnode;
}
+static CreateDictionaryStmt *
+_copyCreateDictionaryStmt(const CreateDictionaryStmt *from)
+{
+ CreateDictionaryStmt *newnode = makeNode(CreateDictionaryStmt);
+
+ COPY_NODE_FIELD(typeName);
+ COPY_NODE_FIELD(baseTypeName);
+ COPY_NODE_FIELD(vals);
+
+ return newnode;
+}
+
static CreateEnumStmt *
_copyCreateEnumStmt(const CreateEnumStmt *from)
{
@@ -6181,6 +6193,9 @@ copyObjectImpl(const void *from)
case T_CompositeTypeStmt:
retval = _copyCompositeTypeStmt(from);
break;
+ case T_CreateDictionaryStmt:
+ retval = _copyCreateDictionaryStmt(from);
+ break;
case T_CreateEnumStmt:
retval = _copyCreateEnumStmt(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index e013c1bbfe..381d8b46f5 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -1962,6 +1962,16 @@ _equalCompositeTypeStmt(const CompositeTypeStmt *a, const CompositeTypeStmt *b)
return true;
}
+static bool
+_equalCreateDictionaryStmt(const CreateDictionaryStmt *a, const CreateDictionaryStmt *b)
+{
+ COMPARE_NODE_FIELD(typeName);
+ COMPARE_NODE_FIELD(baseTypeName);
+ COMPARE_NODE_FIELD(vals);
+
+ return true;
+}
+
static bool
_equalCreateEnumStmt(const CreateEnumStmt *a, const CreateEnumStmt *b)
{
@@ -3984,6 +3994,9 @@ equal(const void *a, const void *b)
case T_CompositeTypeStmt:
retval = _equalCompositeTypeStmt(a, b);
break;
+ case T_CreateDictionaryStmt:
+ retval = _equalCreateDictionaryStmt(a, b);
+ break;
case T_CreateEnumStmt:
retval = _equalCreateEnumStmt(a, b);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index c9941d9cb4..2ac632e025 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -5972,6 +5972,14 @@ DefineStmt:
n->coldeflist = $6;
$$ = (Node *)n;
}
+ | CREATE TYPE_P any_name AS DICTIONARY OF any_name '(' opt_enum_val_list ')'
+ {
+ CreateDictionaryStmt *n = makeNode(CreateDictionaryStmt);
+ n->typeName = $3;
+ n->baseTypeName = $7;
+ n->vals = $9;
+ $$ = (Node *)n;
+ }
| CREATE TYPE_P any_name AS ENUM_P '(' opt_enum_val_list ')'
{
CreateEnumStmt *n = makeNode(CreateEnumStmt);
diff --git a/src/backend/parser/parse_coerce.c b/src/backend/parser/parse_coerce.c
index c4e958e4aa..e489dd9421 100644
--- a/src/backend/parser/parse_coerce.c
+++ b/src/backend/parser/parse_coerce.c
@@ -279,7 +279,22 @@ coerce_type(ParseState *pstate, Node *node,
if (baseTypeId == INTERVALOID)
inputTypeMod = baseTypeMod;
else
- inputTypeMod = -1;
+ {
+ if((int32)targetTypeId == targetTypeMod)
+ {
+ /*
+ * For dictionaries internally we set typmod to a dictionary
+ * OID. This allows dictionary_in() and jsonb_dictionary() to
+ * distinguish one dictionary from another.
+ *
+ * It also allows us to recognize that the target type is
+ * a dictionary here and pass a correct typmod.
+ */
+ inputTypeMod = targetTypeMod;
+ }
+ else
+ inputTypeMod = -1;
+ }
baseType = typeidType(baseTypeId);
diff --git a/src/backend/parser/parse_type.c b/src/backend/parser/parse_type.c
index 307114a30d..868b24fd9f 100644
--- a/src/backend/parser/parse_type.c
+++ b/src/backend/parser/parse_type.c
@@ -259,6 +259,10 @@ LookupTypeNameOid(ParseState *pstate, const TypeName *typeName, bool missing_ok)
* This is equivalent to LookupTypeName, except that this will report
* a suitable error message if the type cannot be found or is not defined.
* Callers of this can therefore assume the result is a fully valid type.
+ *
+ * For a dictionary type it's OID is returned as a typmod. This is used by
+ * dictionary_in() and jsonb_dictionary() to distinguish one dictionary from
+ * another.
*/
Type
typenameType(ParseState *pstate, const TypeName *typeName, int32 *typmod_p)
@@ -305,6 +309,10 @@ typenameTypeId(ParseState *pstate, const TypeName *typeName)
*
* This is equivalent to typenameType, but we only hand back the type OID
* and typmod, not the syscache entry.
+ *
+ * For a dictionary type it's OID is returned as a typmod. This is used by
+ * dictionary_in() and jsonb_dictionary() to distinguish one dictionary from
+ * another.
*/
void
typenameTypeIdAndMod(ParseState *pstate, const TypeName *typeName,
@@ -327,6 +335,10 @@ typenameTypeIdAndMod(ParseState *pstate, const TypeName *typeName,
* looked up, and is passed as "typ".
*
* pstate is only used for error location info, and may be NULL.
+ *
+ * For a dictionary type it's OID is returned as a typmod. This is used by
+ * dictionary_in() and jsonb_dictionary() to distinguish one dictionary from
+ * another.
*/
static int32
typenameTypeMod(ParseState *pstate, const TypeName *typeName, Type typ)
@@ -339,6 +351,10 @@ typenameTypeMod(ParseState *pstate, const TypeName *typeName, Type typ)
ArrayType *arrtypmod;
ParseCallbackState pcbstate;
+ /* For a dictionary return it's OID. */
+ if (((Form_pg_type) GETSTRUCT(typ))->typtype == TYPTYPE_DICT)
+ return (int32)((Form_pg_type) GETSTRUCT(typ))->oid;
+
/* Return prespecified typmod if no typmod expressions */
if (typeName->typmods == NIL)
return typeName->typemod;
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index 0e7b7b3138..cffef5d707 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -174,6 +174,7 @@ ClassifyUtilityCommandAsReadOnly(Node *parsetree)
case T_CreateCastStmt:
case T_CreateConversionStmt:
case T_CreateDomainStmt:
+ case T_CreateDictionaryStmt:
case T_CreateEnumStmt:
case T_CreateEventTrigStmt:
case T_CreateExtensionStmt:
@@ -1621,6 +1622,10 @@ ProcessUtilitySlow(ParseState *pstate,
}
break;
+ case T_CreateDictionaryStmt: /* CREATE TYPE AS DICTIONARY OF */
+ address = DefineDictionary((CreateDictionaryStmt *) parsetree);
+ break;
+
case T_CreateEnumStmt: /* CREATE TYPE AS ENUM */
address = DefineEnum((CreateEnumStmt *) parsetree);
break;
@@ -2766,6 +2771,10 @@ CreateCommandTag(Node *parsetree)
tag = CMDTAG_CREATE_TYPE;
break;
+ case T_CreateDictionaryStmt:
+ tag = CMDTAG_CREATE_TYPE;
+ break;
+
case T_CreateEnumStmt:
tag = CMDTAG_CREATE_TYPE;
break;
@@ -3413,6 +3422,10 @@ GetCommandLogLevel(Node *parsetree)
lev = LOGSTMT_DDL;
break;
+ case T_CreateDictionaryStmt:
+ lev = LOGSTMT_DDL;
+ break;
+
case T_CreateEnumStmt:
lev = LOGSTMT_DDL;
break;
diff --git a/src/backend/utils/adt/Makefile b/src/backend/utils/adt/Makefile
index 7c722ea2ce..a7ec3d89f3 100644
--- a/src/backend/utils/adt/Makefile
+++ b/src/backend/utils/adt/Makefile
@@ -30,6 +30,7 @@ OBJS = \
datetime.o \
datum.o \
dbsize.o \
+ dictionaries.o \
domains.o \
encode.o \
enum.o \
diff --git a/src/backend/utils/adt/dictionaries.c b/src/backend/utils/adt/dictionaries.c
new file mode 100644
index 0000000000..bef09b8f93
--- /dev/null
+++ b/src/backend/utils/adt/dictionaries.c
@@ -0,0 +1,474 @@
+/*-------------------------------------------------------------------------
+ *
+ * dictionaries.c
+ * Conversion functions for dictionary types.
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/adt/dictionaries.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "catalog/pg_dict.h"
+#include "utils/fmgrprotos.h"
+#include "utils/jsonb.h"
+
+/*
+ * When compressing a data we treat it as a BLOB, in other words we don't
+ * assume anything regarding its internal representation. This is not
+ * necessarily the best and/or the only possible approach. However, it is
+ * universal and can be reused for JSONB, TEXT, XML and other types. Alternative
+ * compression methods can be introduced in the future, so that the user will
+ * be able to choose the best one for the task.
+ *
+ * The compressed data is stored in the following format:
+ *
+ * (struct varlena)
+ * dictionary_id [uint32]
+ * decompressed_size [uint32]
+ * algorithm_version [uint8]
+ *
+ * (repeated) {
+ * number of bytes to copy as-is [uint8]
+ * ... bytes to copy as-is ...
+ * dictionary entry id, or 0 to skip [uint32]
+ * }
+ *
+ * Compressed data is a variable-length type and thus has the 'struct varlena'
+ * header. The code below doesn't consider it being a part of the payload
+ * (e.g. see the DICT_COMP_HEADER_SIZE definition).
+ *
+ * Storing dictionary_id may seem redundant, but without it dictionary_out()
+ * and dictionary_jsonb() have no way to distinguish one dictionary type from
+ * another. dictionary_in() and jsonb_dictionary() get the dictionary id through
+ * the 'typemod' argument.
+ *
+ * Currently algorithm_version is always 0. In the future it will allow us to
+ * introduce new features (e.g. usage of varints) and/or lazily migrate the
+ * data to other compression methods.
+ *
+ * Compression and decompression are implemented based on a binary search over
+ * DictEntry[] array (see pg_dict.h). For compression the array is sorted by
+ * dictentries and for decompression - by oid's.
+ *
+ */
+
+/* Size of the header in the compressed data */
+#define DICT_COMP_HEADER_SIZE (sizeof(uint32)*2 + sizeof(uint8))
+
+/* Extracts dictionary_id from the compressed data */
+#define DICT_COMP_DICTIONARY_ID(hdrp) \
+ (*(uint32*)hdrp)
+
+/* Extracts decompressed_size from the compressed data */
+#define DICT_COMP_DECOMPRESSED_SIZE(hdrp) \
+ (*(uint32*)((uint8*)hdrp + sizeof(uint32)))
+
+/* Extracts algorithm_version from the compressed data */
+#define DICT_COMP_ALGORITHM_VERSION(hdrp) \
+ (*(uint8*)((uint8*)hdrp + sizeof(uint32)*2))
+
+/* Current algorithm_version */
+#define DICT_COMP_CURRENT_ALGORITHM_VERSION 0
+
+/*
+ * bsearch_arg() callback for finding DictEntry by oid.
+ */
+static int
+find_by_oid_cb(const void *key, const void *current, void *arg)
+{
+ /* Note that oids are unsigned, so we should be careful here */
+ if (*(Oid *) key < ((DictEntry *) current)->oid)
+ return -1;
+ else if (*(Oid *) key == ((DictEntry *) current)->oid)
+ return 0; /* found! */
+ else
+ return 1;
+}
+
+/*
+ * qsort() callback for sorting DictEntry[] by oids.
+ */
+static int
+sort_by_oids_cb(const void *left, const void *right)
+{
+ /* Note that oids are unsigned, so we should be careful here */
+ if (((DictEntry *) left)->oid < ((DictEntry *) right)->oid)
+ return -1;
+
+ /* Oids are unique so this callback will never return 0 */
+ return 1;
+}
+
+/*
+ * Finds a DictEntry which dictentry field matches *data and returns its Oid.
+ * If there are several matching entries, the largest is returned. The length
+ * of the found entry is written to *found_length on success. On failure
+ * InvalidOid is returned and found_length is zeroed.
+ *
+ * The implementation is similar to bsearch_arg(). The procedure can't be used
+ * directly because we are looking not by the exact match. We could generalize
+ * this case but the signature of the function becomes so complicated that it
+ * doesn't seem to worth the effort.
+ */
+static Oid
+compress_find_oid(Dictionary dict, const uint8 *data, Size data_size, Size *found_length)
+{
+ int res;
+ int32 left = 0;
+ int32 right = dict->nentries - 1;
+ Size best_length = 0;
+ Oid best_match = InvalidOid;
+
+ while (left <= right)
+ {
+ int32 current = (left + right) / 2;
+ Size nbytes = (Size)dict->entries[current].length;
+
+ if (nbytes > data_size)
+ {
+ /* current can be less or greater depending on the prefix */
+ res = memcmp(dict->entries[current].dictentry.data, data, data_size);
+
+ /* if prefixes match, current is greater */
+ if (res == 0)
+ res = 1;
+ }
+ else
+ res = memcmp(dict->entries[current].dictentry.data, data, nbytes);
+
+ if (res == 0) /* match found */
+ {
+ best_length = nbytes;
+ best_match = dict->entries[current].oid;
+
+ if (nbytes == data_size)
+ break;
+
+ /* maybe there is a larger match */
+ left = current + 1;
+ }
+ else if (res < 0) /* current is less */
+ left = current + 1;
+ else /* current is greater */
+ right = current - 1;
+ }
+
+ *found_length = best_length;
+ return best_match;
+}
+
+/*
+ * Finds a DictEntry by Oid using a binary search. The dictionary should be
+ * sorted by oids before the call. Returns NULL if nothing was found.
+ */
+static DictEntry *
+decompress_find_dictentry(Dictionary dict, Oid oid)
+{
+ return (DictEntry *) bsearch_arg(&oid, dict->entries, dict->nentries, sizeof(DictEntry), find_by_oid_cb, NULL);
+}
+
+/*
+ * Estimates the worst-case compressed size of the data of given size.
+ * Worst-case scenario happens when the dictionary consists of single-character
+ * entries. In this case every byte will be encoded as 6 bytes:
+ * 0x00, (0 bytes to copy as-is), 4 bytes of the entry Oid
+ *
+ * This procedure doesn't account for the header size.
+ *
+ * AALEKSEEV TODO FIXME don't allow dictionary entries shorter than 4 bytes?
+ */
+static Size
+worst_case_compressed_size(Size insize)
+{
+ return insize * 6;
+}
+
+/*
+ * Compresses the data using the provided dictionary. The dictionary should
+ * be sorted by dictentries before the call. Output buffer should be at least
+ * worst_case_compressed_size(src_size) bytes in size.
+ */
+static void
+compress(Dictionary dict,
+ const void *src_data_, Size src_size,
+ void *encoded_data_, Size *pencoded_size)
+{
+ Size nbytes;
+ Size inoffset;
+ Size outskipoffset = 0;
+ Size outoffset = 1;
+ uint8 skipbytes = 0;
+ const uint8 *src_data = src_data_;
+ uint8 *encoded_data = ((uint8 *) encoded_data_);
+
+ for (inoffset = 0; inoffset < src_size;)
+ {
+ Oid code = compress_find_oid(dict, &(src_data[inoffset]),
+ src_size - inoffset, &nbytes);
+
+ if (code == InvalidOid)
+ {
+ skipbytes++;
+ encoded_data[outoffset] = src_data[inoffset];
+ outoffset++;
+ inoffset++;
+
+ if (skipbytes == 255)
+ {
+ encoded_data[outskipoffset] = skipbytes;
+ encoded_data[outoffset++] = 0; /* InvalidOid */
+ encoded_data[outoffset++] = 0;
+ encoded_data[outoffset++] = 0;
+ encoded_data[outoffset++] = 0;
+ outskipoffset = outoffset++;
+ skipbytes = 0;
+ }
+ }
+ else
+ {
+ encoded_data[outskipoffset] = skipbytes;
+ encoded_data[outoffset++] = (code >> 24) & 0xFF;
+ encoded_data[outoffset++] = (code >> 16) & 0xFF;
+ encoded_data[outoffset++] = (code >> 8) & 0xFF;
+ encoded_data[outoffset++] = code & 0xFF;
+ outskipoffset = outoffset++;
+ skipbytes = 0;
+ inoffset += nbytes;
+ }
+ }
+
+ /* Double check that we didn't write out of buffer */
+ Assert(outoffset < worst_case_compressed_size(src_size));
+
+ encoded_data[outskipoffset] = skipbytes;
+ *pencoded_size = outoffset;
+}
+
+/*
+ * Report an internal error in decompress() procedure below.
+ * Under normal circumstances this should never happen.
+ */
+static void
+decompress_error()
+{
+ ereport(ERROR,
+ (
+ errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("Unable to decompress a dictionary type"),
+ errdetail("The compressed data seems to be corrupted"),
+ errhint("Please report the steps to reproduce the issue to pgsql-bugs@")
+ ));
+}
+
+/*
+ * Decompresses the data using the provided dictionary. The dictionary should
+ * be sorted by oids before the call.
+ */
+static void
+decompress(Dictionary dict,
+ const void *encoded_data_, Size encoded_size,
+ void *decoded_data_, Size decoded_size)
+{
+ Size inoffset = 0;
+ Size outoffset = 0;
+ Oid code;
+ uint8 skipbytes;
+ const uint8 *encoded_data = ((uint8 *) encoded_data_);
+ uint8 *decoded_data = decoded_data_;
+
+ for (inoffset = 0; inoffset < encoded_size;)
+ {
+ skipbytes = encoded_data[inoffset++];
+
+ if (skipbytes > decoded_size - outoffset)
+ decompress_error();
+
+ if (skipbytes > encoded_size - inoffset)
+ decompress_error();
+
+ memcpy(
+ &(decoded_data[outoffset]),
+ &(encoded_data[inoffset]),
+ skipbytes
+ );
+
+ outoffset += skipbytes;
+ inoffset += skipbytes;
+
+ if (encoded_size == inoffset && decoded_size == outoffset)
+ break; /* end of input - its OK */
+
+ if (encoded_size - inoffset < 4)
+ decompress_error();
+
+ code = (Oid) encoded_data[inoffset++];
+ code = (code << 8) | (Oid) encoded_data[inoffset++];
+ code = (code << 8) | (Oid) encoded_data[inoffset++];
+ code = (code << 8) | (Oid) encoded_data[inoffset++];
+
+ if (code != InvalidOid)
+ {
+ Size entrylen;
+ DictEntry *entry = decompress_find_dictentry(dict, code);
+
+ if (entry == NULL)
+ decompress_error();
+
+ Assert(entry->oid == code);
+ entrylen = (Size)entry->length;
+
+ if (entrylen > decoded_size - outoffset)
+ decompress_error();
+
+ memcpy(
+ &(decoded_data[outoffset]),
+ entry->dictentry.data,
+ entrylen
+ );
+
+ outoffset += entrylen;
+ }
+ }
+}
+
+/*
+ * Converts a cstring to a dictionary.
+ */
+Datum
+dictionary_in(PG_FUNCTION_ARGS)
+{
+ const char *instr = PG_GETARG_CSTRING(0);
+#ifdef NOT_USED
+ Oid typelem = PG_GETARG_OID(1);
+#endif
+ int32 typmod = PG_GETARG_INT32(2);
+ Jsonb *jsonb;
+ bytea *dict;
+
+ Assert(typmod != -1);
+
+ jsonb = DatumGetJsonbP(DirectFunctionCall1(jsonb_in, CStringGetDatum(instr)));
+ dict = DatumGetByteaP(DirectFunctionCall2(jsonb_dictionary, JsonbPGetDatum(jsonb), Int32GetDatum(typmod)));
+
+ PG_RETURN_BYTEA_P(dict);
+}
+
+/*
+ * Converts a dictionary to a cstring.
+ */
+Datum
+dictionary_out(PG_FUNCTION_ARGS)
+{
+ bytea *dict = PG_GETARG_BYTEA_P(0);
+ Jsonb *jsonb = DatumGetJsonbP(DirectFunctionCall1(dictionary_jsonb, PointerGetDatum(dict)));
+ const char *outstr = DatumGetCString(DirectFunctionCall1(jsonb_out, JsonbPGetDatum(jsonb)));
+
+ PG_RETURN_CSTRING(outstr);
+}
+
+/*
+ * Coverts JSONB to a dictionary type.
+ */
+Datum
+jsonb_dictionary(PG_FUNCTION_ARGS)
+{
+ Dictionary dict;
+ Jsonb *jsonb = PG_GETARG_JSONB_P(0);
+ int32 typmod = PG_GETARG_INT32(1);
+ uint8 *jsonb_data = (uint8 *) VARDATA(jsonb);
+ Size jsonb_data_size = VARSIZE(jsonb) - VARHDRSZ;
+ uint8 *encoded_buff,
+ *encoded_header,
+ *encoded_data;
+ Size encoded_size,
+ encoded_buff_size;
+
+ Assert(typmod != -1);
+
+ dict = DictEntriesRead(typmod);
+ qsort((void *) dict->entries, dict->nentries, sizeof(DictEntry), sort_by_oids_cb);
+
+ encoded_buff_size = VARHDRSZ + DICT_COMP_HEADER_SIZE + worst_case_compressed_size(jsonb_data_size);
+ encoded_buff = palloc(encoded_buff_size);
+ encoded_header = (uint8 *) VARDATA(encoded_buff);
+ encoded_data = encoded_header + DICT_COMP_HEADER_SIZE;
+
+ DICT_COMP_DICTIONARY_ID(encoded_header) = typmod;
+ DICT_COMP_DECOMPRESSED_SIZE(encoded_header) = jsonb_data_size;
+ DICT_COMP_ALGORITHM_VERSION(encoded_header) = DICT_COMP_CURRENT_ALGORITHM_VERSION;
+
+ encoded_size = encoded_buff_size - VARHDRSZ - DICT_COMP_HEADER_SIZE;
+
+ compress(dict, jsonb_data, jsonb_data_size,
+ encoded_data, &encoded_size);
+
+ encoded_size += VARHDRSZ + DICT_COMP_HEADER_SIZE;
+
+ encoded_buff = repalloc(encoded_buff, encoded_size);
+ SET_VARSIZE(encoded_buff, encoded_size);
+
+ pfree(dict);
+ PG_RETURN_BYTEA_P(encoded_buff);
+}
+
+/*
+ * Converts a dictionary type to JSONB.
+ */
+Datum
+dictionary_jsonb(PG_FUNCTION_ARGS)
+{
+ bytea *encoded_buff = PG_GETARG_BYTEA_P(0);
+ uint8 *encoded_header = (uint8 *) VARDATA(encoded_buff);
+ uint8 *encoded_data = encoded_header + DICT_COMP_HEADER_SIZE;
+ Size encoded_size = VARSIZE(encoded_buff) - VARHDRSZ - DICT_COMP_HEADER_SIZE;
+ int alg_version = DICT_COMP_ALGORITHM_VERSION(encoded_header);
+ Oid dictOid; /* cannot read until algorithm version is
+ * checked */
+ uint32 decoded_size; /* cannot read until algorithm version is
+ * checked */
+ Jsonb *jsonb;
+ uint8 *jsonb_data;
+ Dictionary dict;
+
+ if (alg_version > DICT_COMP_CURRENT_ALGORITHM_VERSION)
+ ereport(ERROR,
+ (
+ errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("Unsupported compression algorithm version"),
+ errdetail("Saved algorithm version is %d, current version is %d",
+ alg_version, DICT_COMP_CURRENT_ALGORITHM_VERSION),
+ errhint("The data is either corrupted or imported from "
+ "the future version of PostgreSQL")
+ ));
+
+ dictOid = DICT_COMP_DICTIONARY_ID(encoded_header);
+ decoded_size = DICT_COMP_DECOMPRESSED_SIZE(encoded_header);
+ dict = DictEntriesRead(dictOid);
+
+ jsonb = palloc(decoded_size + VARHDRSZ);
+ jsonb_data = (uint8 *) VARDATA(jsonb);
+
+ decompress(dict, encoded_data, encoded_size, jsonb_data, decoded_size);
+
+ decoded_size += VARHDRSZ;
+ SET_VARSIZE(jsonb, decoded_size);
+
+ pfree(dict);
+ PG_RETURN_JSONB_P(jsonb);
+}
+
+/*
+ * Converts a dictionary type to bytea.
+ */
+Datum
+dictionary_bytea(PG_FUNCTION_ARGS)
+{
+ bytea *compressed_data = PG_GETARG_BYTEA_P(0);
+ PG_RETURN_BYTEA_P(compressed_data);
+}
\ No newline at end of file
diff --git a/src/backend/utils/fmgr/funcapi.c b/src/backend/utils/fmgr/funcapi.c
index 9197b0f1e2..4f9dc52428 100644
--- a/src/backend/utils/fmgr/funcapi.c
+++ b/src/backend/utils/fmgr/funcapi.c
@@ -1292,6 +1292,7 @@ get_type_func_class(Oid typid, Oid *base_typeid)
case TYPTYPE_COMPOSITE:
return TYPEFUNC_COMPOSITE;
case TYPTYPE_BASE:
+ case TYPTYPE_DICT:
case TYPTYPE_ENUM:
case TYPTYPE_RANGE:
case TYPTYPE_MULTIRANGE:
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 786d592e2b..493ef56c30 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -10177,6 +10177,8 @@ dumpType(Archive *fout, const TypeInfo *tyinfo)
dumpDomain(fout, tyinfo);
else if (tyinfo->typtype == TYPTYPE_COMPOSITE)
dumpCompositeType(fout, tyinfo);
+ else if (tyinfo->typtype == TYPTYPE_DICT)
+ pg_log_error("AALEKSEEV TODO FIXME not implemented");
else if (tyinfo->typtype == TYPTYPE_ENUM)
dumpEnumType(fout, tyinfo);
else if (tyinfo->typtype == TYPTYPE_RANGE)
diff --git a/src/include/catalog/pg_dict.h b/src/include/catalog/pg_dict.h
new file mode 100644
index 0000000000..d05303f724
--- /dev/null
+++ b/src/include/catalog/pg_dict.h
@@ -0,0 +1,76 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_dict.h
+ * definition of the "dict" system catalog (pg_dict)
+ *
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ * src/include/catalog/pg_dict.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef PG_DICT_H
+#define PG_DICT_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_dict_d.h"
+
+#include "nodes/pg_list.h"
+
+/* ----------------
+ * pg_dict definition. cpp turns this into
+ * typedef struct FormData_pg_dict
+ * ----------------
+ */
+CATALOG(pg_dict,9861,DictRelationId)
+{
+ Oid oid; /* oid */
+ Oid dicttypid BKI_LOOKUP(pg_type); /* OID of owning dict type */
+ NameData dictentry; /* text representation of the dictionary entry */
+} FormData_pg_dict;
+
+/* ----------------
+ * Form_pg_dict corresponds to a pointer to a tuple with
+ * the format of pg_dict relation.
+ * ----------------
+ */
+typedef FormData_pg_dict *Form_pg_dict;
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_dict_oid_index, 9862, DictOidIndexId, on pg_dict using btree(oid oid_ops));
+DECLARE_UNIQUE_INDEX(pg_dict_typid_entry_index, 9863, DictTypIdEntryIndexId, on pg_dict using btree(dicttypid oid_ops, dictentry name_ops));
+
+
+/*
+ * DictEntry type represents one entry in the given dictionary.
+ */
+typedef struct
+{
+ Oid oid; /* entry Oid */
+ uint32 length; /* entry length */
+ NameData dictentry; /* the entry */
+} DictEntry;
+
+/*
+ * Dictionary type represents all the entries in the given dictionary.
+ */
+typedef struct
+{
+ uint32 nentries; /* number of entries */
+ DictEntry *entries; /* array of entries */
+} DictionaryData;
+
+typedef DictionaryData *Dictionary;
+
+/*
+ * prototypes for functions in pg_dict.c
+ */
+extern void DictEntriesCreate(Oid dictTypeOid, List *vals);
+extern Dictionary DictEntriesRead(Oid dictTypeOid);
+extern void DictEntriesDelete(Oid dictTypeOid);
+
+#endif /* PG_DICT_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 6d378ff785..c6ec8737df 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10022,6 +10022,26 @@
proname => 'jsonb_path_match_opr', prorettype => 'bool',
proargtypes => 'jsonb jsonpath', prosrc => 'jsonb_path_match_opr' },
+# dictionaries
+{ oid => '9864', descr => 'Converts a cstring to a dictionary',
+ proname => 'dictionary_in', prorettype => 'any', proargtypes => 'cstring oid int4',
+ prosrc => 'dictionary_in' },
+{ oid => '9865', descr => 'Converts a dictionary to a cstring',
+ proname => 'dictionary_out', prorettype => 'cstring', proargtypes => 'any',
+ prosrc => 'dictionary_out' },
+{ oid => '9866', descr => 'Coverts JSONB to a dictionary type',
+ proname => 'jsonb_dictionary', proisstrict => 'f', provolatile => 's',
+ prorettype => 'any', proargtypes => 'jsonb int4',
+ prosrc => 'jsonb_dictionary' },
+{ oid => '9867', descr => 'Converts a dictionary type to JSONB',
+ proname => 'dictionary_jsonb', proisstrict => 'f', provolatile => 's',
+ prorettype => 'jsonb', proargtypes => 'any',
+ prosrc => 'dictionary_jsonb' },
+{ oid => '9868', descr => 'Converts a dictionary type to a byte array',
+ proname => 'dictionary_bytea', proisstrict => 'f', provolatile => 's',
+ prorettype => 'jsonb', proargtypes => 'any',
+ prosrc => 'dictionary_bytea' },
+
# historical int8/txid_snapshot variants of xid8 functions
{ oid => '2939', descr => 'I/O',
proname => 'txid_snapshot_in', prorettype => 'txid_snapshot',
diff --git a/src/include/catalog/pg_type.h b/src/include/catalog/pg_type.h
index 48a2559137..91c6bfae99 100644
--- a/src/include/catalog/pg_type.h
+++ b/src/include/catalog/pg_type.h
@@ -272,6 +272,7 @@ DECLARE_UNIQUE_INDEX(pg_type_typname_nsp_index, 2704, TypeNameNspIndexId, on pg_
*/
#define TYPTYPE_BASE 'b' /* base type (ordinary scalar type) */
#define TYPTYPE_COMPOSITE 'c' /* composite (e.g., table's rowtype) */
+#define TYPTYPE_DICT 'D' /* dictionary */
#define TYPTYPE_DOMAIN 'd' /* domain over another type */
#define TYPTYPE_ENUM 'e' /* enumerated type */
#define TYPTYPE_MULTIRANGE 'm' /* multirange type */
diff --git a/src/include/commands/typecmds.h b/src/include/commands/typecmds.h
index a17bedb851..5cc8e5ff99 100644
--- a/src/include/commands/typecmds.h
+++ b/src/include/commands/typecmds.h
@@ -24,6 +24,7 @@
extern ObjectAddress DefineType(ParseState *pstate, List *names, List *parameters);
extern void RemoveTypeById(Oid typeOid);
extern ObjectAddress DefineDomain(CreateDomainStmt *stmt);
+extern ObjectAddress DefineDictionary(CreateDictionaryStmt *stmt);
extern ObjectAddress DefineEnum(CreateEnumStmt *stmt);
extern ObjectAddress DefineRange(ParseState *pstate, CreateRangeStmt *stmt);
extern ObjectAddress AlterEnum(AlterEnumStmt *stmt);
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 340d28f4e1..ad1e3ad9f5 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -408,6 +408,7 @@ typedef enum NodeTag
T_DropOwnedStmt,
T_ReassignOwnedStmt,
T_CompositeTypeStmt,
+ T_CreateDictionaryStmt,
T_CreateEnumStmt,
T_CreateRangeStmt,
T_AlterEnumStmt,
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index da02658c81..5e3aacd8a7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3567,6 +3567,18 @@ typedef struct CompositeTypeStmt
List *coldeflist; /* list of ColumnDef nodes */
} CompositeTypeStmt;
+/* ----------------------
+ * Create Type Statement, dictionary types
+ * ----------------------
+ */
+typedef struct CreateDictionaryStmt
+{
+ NodeTag type;
+ List *typeName; /* qualified name (list of String) */
+ List *baseTypeName; /* qualified base type name (list of String) */
+ List *vals; /* dictionary entries (list of String) */
+} CreateDictionaryStmt;
+
/* ----------------------
* Create Type Statement, enum types
* ----------------------
diff --git a/src/pl/plpgsql/src/pl_comp.c b/src/pl/plpgsql/src/pl_comp.c
index b791c23f06..897051c1f7 100644
--- a/src/pl/plpgsql/src/pl_comp.c
+++ b/src/pl/plpgsql/src/pl_comp.c
@@ -2129,6 +2129,7 @@ build_datatype(HeapTuple typeTup, int32 typmod,
switch (typeStruct->typtype)
{
case TYPTYPE_BASE:
+ case TYPTYPE_DICT:
case TYPTYPE_ENUM:
case TYPTYPE_RANGE:
case TYPTYPE_MULTIRANGE:
diff --git a/src/test/modules/test_oat_hooks/test_oat_hooks.c b/src/test/modules/test_oat_hooks/test_oat_hooks.c
index 551da5d498..1cab22183a 100644
--- a/src/test/modules/test_oat_hooks/test_oat_hooks.c
+++ b/src/test/modules/test_oat_hooks/test_oat_hooks.c
@@ -772,6 +772,7 @@ nodetag_to_string(NodeTag tag)
case T_DropOwnedStmt: return "DropOwnedStmt"; break;
case T_ReassignOwnedStmt: return "ReassignOwnedStmt"; break;
case T_CompositeTypeStmt: return "CompositeTypeStmt"; break;
+ case T_CreateDictionaryStmt: return "CreateDictionaryStmt"; break;
case T_CreateEnumStmt: return "CreateEnumStmt"; break;
case T_CreateRangeStmt: return "CreateRangeStmt"; break;
case T_AlterEnumStmt: return "AlterEnumStmt"; break;
diff --git a/src/test/regress/expected/dict.out b/src/test/regress/expected/dict.out
new file mode 100644
index 0000000000..13fe15810f
--- /dev/null
+++ b/src/test/regress/expected/dict.out
@@ -0,0 +1,36 @@
+--
+-- Compression dictionaries tests
+--
+-- Simple happy path
+CREATE TYPE mydict AS DICTIONARY OF JSONB ('abcdef', 'ghijkl');
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
+ dictentry
+-----------
+ abcdef
+ ghijkl
+(2 rows)
+
+SELECT '{"abcdef":"ghijkl"}' :: mydict;
+ mydict
+----------------------
+ {"abcdef": "ghijkl"}
+(1 row)
+
+SELECT '{"abcdef":"ghijkl"}' :: mydict :: jsonb;
+ jsonb
+----------------------
+ {"abcdef": "ghijkl"}
+(1 row)
+
+SELECT ('{"abcdef":"ghijkl"}' :: mydict) -> 'abcdef';
+ ?column?
+----------
+ "ghijkl"
+(1 row)
+
+DROP TYPE mydict;
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
+ dictentry
+-----------
+(0 rows)
+
diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out
index 215eb899be..91bc491a6c 100644
--- a/src/test/regress/expected/oidjoins.out
+++ b/src/test/regress/expected/oidjoins.out
@@ -182,6 +182,7 @@ NOTICE: checking pg_description {classoid} => pg_class {oid}
NOTICE: checking pg_cast {castsource} => pg_type {oid}
NOTICE: checking pg_cast {casttarget} => pg_type {oid}
NOTICE: checking pg_cast {castfunc} => pg_proc {oid}
+NOTICE: checking pg_dict {dicttypid} => pg_type {oid}
NOTICE: checking pg_enum {enumtypid} => pg_type {oid}
NOTICE: checking pg_namespace {nspowner} => pg_authid {oid}
NOTICE: checking pg_conversion {connamespace} => pg_namespace {oid}
diff --git a/src/test/regress/expected/opr_sanity.out b/src/test/regress/expected/opr_sanity.out
index 86d755aa44..5dc4d0ab01 100644
--- a/src/test/regress/expected/opr_sanity.out
+++ b/src/test/regress/expected/opr_sanity.out
@@ -388,11 +388,12 @@ WHERE 'cstring'::regtype = ANY (p1.proargtypes)
AND NOT EXISTS(SELECT 1 FROM pg_conversion WHERE conproc = p1.oid)
AND p1.oid != 'shell_in(cstring)'::regprocedure
ORDER BY 1;
- oid | proname
-------+--------------
+ oid | proname
+------+---------------
2293 | cstring_out
2501 | cstring_send
-(2 rows)
+ 9864 | dictionary_in
+(3 rows)
-- Likewise, look for functions that return cstring and aren't datatype output
-- functions nor typmod output functions.
@@ -405,11 +406,12 @@ WHERE p1.prorettype = 'cstring'::regtype
AND NOT EXISTS(SELECT 1 FROM pg_type WHERE typmodout = p1.oid)
AND p1.oid != 'shell_out(void)'::regprocedure
ORDER BY 1;
- oid | proname
-------+--------------
+ oid | proname
+------+----------------
2292 | cstring_in
2500 | cstring_recv
-(2 rows)
+ 9865 | dictionary_out
+(3 rows)
-- Check for length inconsistencies between the various argument-info arrays.
SELECT p1.oid, p1.proname
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 103e11483d..2fa15a2080 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -111,7 +111,7 @@ test: select_views portals_p2 foreign_key cluster dependency guc bitmapops combo
# ----------
# Another group of parallel tests (JSON related)
# ----------
-test: json jsonb json_encoding jsonpath jsonpath_encoding jsonb_jsonpath sqljson json_sqljson jsonb_sqljson
+test: dict json jsonb json_encoding jsonpath jsonpath_encoding jsonb_jsonpath sqljson json_sqljson jsonb_sqljson
# ----------
# Another group of parallel tests
diff --git a/src/test/regress/sql/dict.sql b/src/test/regress/sql/dict.sql
new file mode 100644
index 0000000000..35e1848296
--- /dev/null
+++ b/src/test/regress/sql/dict.sql
@@ -0,0 +1,14 @@
+--
+-- Compression dictionaries tests
+--
+
+-- Simple happy path
+CREATE TYPE mydict AS DICTIONARY OF JSONB ('abcdef', 'ghijkl');
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
+
+SELECT '{"abcdef":"ghijkl"}' :: mydict;
+SELECT '{"abcdef":"ghijkl"}' :: mydict :: jsonb;
+SELECT ('{"abcdef":"ghijkl"}' :: mydict) -> 'abcdef';
+
+DROP TYPE mydict;
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
\ No newline at end of file
--
2.35.1
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: [PATCH] Compression dictionaries for JSONB
@ 2022-05-13 08:08 Aleksander Alekseev <[email protected]>
parent: Aleksander Alekseev <[email protected]>
0 siblings, 1 reply; 88+ messages in thread
From: Aleksander Alekseev @ 2022-05-13 08:08 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Zhihong Yu <[email protected]>; Matthias van de Meent <[email protected]>
Hi hackers,
> Here it the 2nd version of the patch:
>
> - Includes changes named above
> - Fixes a warning reported by cfbot
> - Fixes some FIXME's
> - The path includes some simple tests now
> - A proper commit message was added
Here is the rebased version of the patch. Changes compared to v2 are minimal.
> Open questions:
> - Dictionary entries are currently stored as NameData, the same type that is
> used for enums. Are we OK with the accompanying limitations? Any alternative
> suggestions?
> - All in all, am I moving the right direction?
I would like to receive a little bit more feedback before investing more time
into this effort. This will allow me, if necessary, to alter the overall design
more easily.
--
Best regards,
Aleksander Alekseev
Attachments:
[application/octet-stream] v3-0001-Compression-dictionaries-for-JSONB.patch (50.5K, ../../CAJ7c6TNBSN0BG_PJBz46mHMAtkam0MJKL5nWTBFhwKA+3N=4LA@mail.gmail.com/2-v3-0001-Compression-dictionaries-for-JSONB.patch)
download | inline diff:
From f525c1b14eac7bbdb6c56fd6c40ba1048ea7f9ce Mon Sep 17 00:00:00 2001
From: Aleksander Alekseev <[email protected]>
Date: Tue, 12 Apr 2022 15:58:11 +0300
Subject: [PATCH v3] Compression dictionaries for JSONB
Usage example:
CREATE TYPE mydict AS DITIONARY OF jsonb ('aaa', 'bbb', ...);
SELECT ('{"aaa":"bbb"}' :: mydict) -> 'aaa';
The created type works as a drop-in replacement for JSONB. However,
its internal representation differs. The provided dictionary entries ('aaa',
'bbb', ..) are stored in pg_dict catalog table. When `mydict` sees one of
the entries in the document, it replaces it with the corresponding pg_dict
entry Oid. For more details regarding the compression algorithm and choosen
compromises please see the comments in the code.
In pg_type `mydict` has typtype = TYPTYPE_DICT. It works the same way
as TYPTYPE_BASE with only difference: corresponding `<type>_in`
(pg_type.typinput) and `<another-type>_<type>` (pg_cast.castfunc)
procedures receive the dictionary Oid as a `typmod` argument. This way the
procedures can distinguish `mydict1` from `mydict2` and use the proper
compression dictionary.
(NOTE: dear committer, please bump the catalog version!)
Author: Aleksander Alekseev <[email protected]>
Reviewed-by: FIXME
Discussion: https://postgr.es/m/CAJ7c6TOtAB0z1UrksvGTStNE-herK-43bj22%3D5xVBg7S4vr5rQ%40mail.gmail.com
Discussion: https://postgr.es/m/CAJ7c6TPx7N-bVw0dZ1ASCDQKZJHhBYkT6w4HV1LzfS%2BUUTUfmA%40mail.gmail.com
---
src/backend/catalog/Makefile | 3 +-
src/backend/catalog/pg_dict.c | 175 +++++++
src/backend/commands/typecmds.c | 160 +++++-
src/backend/executor/functions.c | 1 +
src/backend/nodes/copyfuncs.c | 15 +
src/backend/nodes/equalfuncs.c | 13 +
src/backend/parser/gram.y | 8 +
src/backend/parser/parse_coerce.c | 17 +-
src/backend/parser/parse_type.c | 16 +
src/backend/tcop/utility.c | 13 +
src/backend/utils/adt/Makefile | 1 +
src/backend/utils/adt/dictionaries.c | 479 ++++++++++++++++++
src/backend/utils/fmgr/funcapi.c | 1 +
src/bin/pg_dump/pg_dump.c | 2 +
src/include/catalog/pg_dict.h | 76 +++
src/include/catalog/pg_proc.dat | 20 +
src/include/catalog/pg_type.h | 1 +
src/include/commands/typecmds.h | 1 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 12 +
src/pl/plpgsql/src/pl_comp.c | 1 +
.../modules/test_oat_hooks/test_oat_hooks.c | 3 +
src/test/regress/expected/dict.out | 47 ++
src/test/regress/expected/oidjoins.out | 1 +
src/test/regress/expected/opr_sanity.out | 14 +-
src/test/regress/parallel_schedule | 2 +-
src/test/regress/sql/dict.sql | 20 +
27 files changed, 1092 insertions(+), 11 deletions(-)
create mode 100644 src/backend/catalog/pg_dict.c
create mode 100644 src/backend/utils/adt/dictionaries.c
create mode 100644 src/include/catalog/pg_dict.h
create mode 100644 src/test/regress/expected/dict.out
create mode 100644 src/test/regress/sql/dict.sql
diff --git a/src/backend/catalog/Makefile b/src/backend/catalog/Makefile
index 89a0221ec9..3b4330148e 100644
--- a/src/backend/catalog/Makefile
+++ b/src/backend/catalog/Makefile
@@ -33,6 +33,7 @@ OBJS = \
pg_conversion.o \
pg_db_role_setting.o \
pg_depend.o \
+ pg_dict.o \
pg_enum.o \
pg_inherits.o \
pg_largeobject.o \
@@ -61,7 +62,7 @@ CATALOG_HEADERS := \
pg_language.h pg_largeobject_metadata.h pg_largeobject.h pg_aggregate.h \
pg_statistic.h pg_statistic_ext.h pg_statistic_ext_data.h \
pg_rewrite.h pg_trigger.h pg_event_trigger.h pg_description.h \
- pg_cast.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \
+ pg_cast.h pg_dict.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \
pg_database.h pg_db_role_setting.h pg_tablespace.h \
pg_authid.h pg_auth_members.h pg_shdepend.h pg_shdescription.h \
pg_ts_config.h pg_ts_config_map.h pg_ts_dict.h \
diff --git a/src/backend/catalog/pg_dict.c b/src/backend/catalog/pg_dict.c
new file mode 100644
index 0000000000..ef76e565cd
--- /dev/null
+++ b/src/backend/catalog/pg_dict.c
@@ -0,0 +1,175 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_dict.c
+ * routines to support manipulation of the pg_dict relation
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/catalog/pg_dict.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "access/genam.h"
+#include "access/skey.h"
+#include "access/table.h"
+#include "catalog/catalog.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_dict.h"
+#include "catalog/pg_type.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/rel.h"
+
+/*
+ * Creates an entry in pg_dict for each of the supplied values.
+ * vals is a list of String values.
+ */
+void
+DictEntriesCreate(Oid dictTypeOid, List *vals)
+{
+ Relation pg_dict;
+ NameData dictentry;
+ Datum values[Natts_pg_dict];
+ bool nulls[Natts_pg_dict];
+ ListCell *lc;
+ HeapTuple tup;
+
+ if (vals == NIL)
+ {
+ /* The list is empty; do nothing. */
+ return;
+ }
+
+ memset(nulls, false, sizeof(nulls));
+
+ /*
+ * We don't check the list of values for duplicates here. If there are
+ * any, the user will get an unique-index violation.
+ */
+
+ pg_dict = table_open(DictRelationId, RowExclusiveLock);
+ foreach(lc, vals)
+ {
+ Oid oid = GetNewOidWithIndex(pg_dict, DictOidIndexId,
+ Anum_pg_dict_oid);
+ char *entry = strVal(lfirst(lc));
+
+ /*
+ * Entries are stored in a name field, for easier syscache lookup, so
+ * check the length to make sure it's within range.
+ */
+ if (strlen(entry) > (NAMEDATALEN - 1))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_NAME),
+ errmsg("invalid dict entry \"%s\"", entry),
+ errdetail("Entries must be %d bytes or less.",
+ NAMEDATALEN - 1)));
+
+ namestrcpy(&dictentry, entry);
+
+ values[Anum_pg_dict_oid - 1] = ObjectIdGetDatum(oid);
+ values[Anum_pg_dict_dicttypid - 1] = ObjectIdGetDatum(dictTypeOid);
+ values[Anum_pg_dict_dictentry - 1] = NameGetDatum(&dictentry);
+
+ tup = heap_form_tuple(RelationGetDescr(pg_dict), values, nulls);
+
+ CatalogTupleInsert(pg_dict, tup);
+ heap_freetuple(tup);
+ }
+
+ /* clean up */
+ table_close(pg_dict, RowExclusiveLock);
+}
+
+/*
+ * Returns all the entries for the dictinary with given Oid. Entries are sorted
+ * by dictentry. Note that shorter entries are considered smaller, i.e. 'abc'
+ * goes before 'abcdef'. The memory is allocated in the caller's memory context.
+ *
+ * If there are no entries a valid but empty dictionary is returned.
+ */
+Dictionary
+DictEntriesRead(Oid dictTypeOid)
+{
+ Relation pg_dict;
+ ScanKeyData key[1];
+ SysScanDesc scan;
+ HeapTuple tup;
+ uint32 entries_allocated = 8;
+ Dictionary dict = (Dictionary)palloc(sizeof(DictionaryData));
+
+ dict->nentries = 0;
+ dict->entries = (DictEntry*)palloc(entries_allocated*sizeof(DictEntry));
+
+ pg_dict = table_open(DictRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_dict_dicttypid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(dictTypeOid));
+
+ scan = systable_beginscan(pg_dict, DictTypIdEntryIndexId, true,
+ NULL, 1, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ if(dict->nentries == entries_allocated)
+ {
+ entries_allocated = entries_allocated * 2;
+ dict->entries = (DictEntry*)repalloc(dict->entries, entries_allocated*sizeof(DictEntry));
+ }
+
+ dict->entries[dict->nentries].oid = ((Form_pg_dict) GETSTRUCT(tup))->oid;
+ dict->entries[dict->nentries].dictentry = ((Form_pg_dict) GETSTRUCT(tup))->dictentry;
+ /*
+ * This is arguably not the fastest way to determine the length of
+ * the entry. Alternatively, we could store a precalculated value in
+ * the catalog. However, usually it's a good idea to keep things simple
+ * until somebody discovers a bottleneck in this exact place and
+ * proposes a concrete fix.
+ */
+ dict->entries[dict->nentries].length = (uint32)strlen(dict->entries[dict->nentries].dictentry.data);
+ dict->nentries++;
+ }
+
+ systable_endscan(scan);
+ table_close(pg_dict, RowExclusiveLock);
+
+ return dict;
+}
+
+/*
+ * Deletes all the entries for the dictinary with given Oid.
+ */
+void
+DictEntriesDelete(Oid dictTypeOid)
+{
+ Relation pg_dict;
+ ScanKeyData key[1];
+ SysScanDesc scan;
+ HeapTuple tup;
+
+ pg_dict = table_open(DictRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_dict_dicttypid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(dictTypeOid));
+
+ scan = systable_beginscan(pg_dict, DictTypIdEntryIndexId, true,
+ NULL, 1, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ CatalogTupleDelete(pg_dict, &tup->t_self);
+ }
+
+ systable_endscan(scan);
+ table_close(pg_dict, RowExclusiveLock);
+}
diff --git a/src/backend/commands/typecmds.c b/src/backend/commands/typecmds.c
index 9b92b04242..3115fb7336 100644
--- a/src/backend/commands/typecmds.c
+++ b/src/backend/commands/typecmds.c
@@ -46,6 +46,7 @@
#include "catalog/pg_collation.h"
#include "catalog/pg_constraint.h"
#include "catalog/pg_depend.h"
+#include "catalog/pg_dict.h"
#include "catalog/pg_enum.h"
#include "catalog/pg_language.h"
#include "catalog/pg_namespace.h"
@@ -679,6 +680,14 @@ RemoveTypeById(Oid typeOid)
if (((Form_pg_type) GETSTRUCT(tup))->typtype == TYPTYPE_RANGE)
RangeDelete(typeOid);
+ /*
+ * If it is a dictionary type, delete the pg_dict entries too; we don't
+ * bother with making a dependency entry for that, so it has to be done
+ * "by hand" here.
+ */
+ if (((Form_pg_type) GETSTRUCT(tup))->typtype == TYPTYPE_DICT)
+ DictEntriesDelete(typeOid);
+
ReleaseSysCache(tup);
table_close(relation, RowExclusiveLock);
@@ -763,8 +772,8 @@ DefineDomain(CreateDomainStmt *stmt)
/*
* Base type must be a plain base type, a composite type, another domain,
- * an enum or a range type. Domains over pseudotypes would create a
- * security hole. (It would be shorter to code this to just check for
+ * a dict, an enum or a range type. Domains over pseudotypes would create
+ * a security hole. (It would be shorter to code this to just check for
* pseudotypes; but it seems safer to call out the specific typtypes that
* are supported, rather than assume that all future typtypes would be
* automatically supported.)
@@ -772,6 +781,7 @@ DefineDomain(CreateDomainStmt *stmt)
typtype = baseType->typtype;
if (typtype != TYPTYPE_BASE &&
typtype != TYPTYPE_COMPOSITE &&
+ typtype != TYPTYPE_DICT &&
typtype != TYPTYPE_DOMAIN &&
typtype != TYPTYPE_ENUM &&
typtype != TYPTYPE_RANGE &&
@@ -1129,6 +1139,152 @@ DefineDomain(CreateDomainStmt *stmt)
}
+/*
+ * DefineDictionary
+ * Registers a new dictionary.
+ */
+ObjectAddress
+DefineDictionary(CreateDictionaryStmt *stmt)
+{
+ char *dictName;
+ char *dictBaseTypeName;
+ char *dictArrayName;
+ Oid dictNamespace;
+ AclResult aclresult;
+ Oid old_type_oid;
+ Oid dictArrayOid;
+ ObjectAddress dictTypeAddr;
+
+ Assert(list_length(stmt->baseTypeName) == 1);
+ dictBaseTypeName = strVal(linitial(stmt->baseTypeName));
+ if(pg_strcasecmp(dictBaseTypeName, "jsonb") != 0)
+ ereport(ERROR,
+ (
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("Dictionary types currently can be used only with JSONB")
+ )
+ );
+
+ /* Convert list of names to a name and namespace */
+ dictNamespace = QualifiedNameGetCreationNamespace(stmt->typeName,
+ &dictName);
+
+ /* Check we have creation rights in target namespace */
+ aclresult = pg_namespace_aclcheck(dictNamespace, GetUserId(), ACL_CREATE);
+ if (aclresult != ACLCHECK_OK)
+ aclcheck_error(aclresult, OBJECT_SCHEMA,
+ get_namespace_name(dictNamespace));
+
+ /*
+ * Check for collision with an existing type name. If there is one and
+ * it's an autogenerated array, we can rename it out of the way.
+ */
+ old_type_oid = GetSysCacheOid2(TYPENAMENSP, Anum_pg_type_oid,
+ CStringGetDatum(dictName),
+ ObjectIdGetDatum(dictNamespace));
+ if (OidIsValid(old_type_oid))
+ {
+ if (!moveArrayTypeName(old_type_oid, dictName, dictNamespace))
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("type \"%s\" already exists", dictName)));
+ }
+
+ /* Allocate OID for array type */
+ dictArrayOid = AssignTypeArrayOid();
+
+ /* Create the pg_type entry */
+ dictTypeAddr =
+ TypeCreate(InvalidOid, /* no predetermined type OID */
+ dictName, /* type name */
+ dictNamespace, /* namespace */
+ InvalidOid, /* relation oid (n/a here) */
+ 0, /* relation kind (ditto) */
+ GetUserId(), /* owner's ID */
+ -1, /* internal size: varlena, as for JSONB */
+ TYPTYPE_DICT, /* type-type: dictionary */
+ TYPCATEGORY_USER, /* type-category: user, as for JSONB */
+ false, /* dict types are never preferred */
+ DEFAULT_TYPDELIM, /* array element delimiter */
+ F_DICTIONARY_IN, /* input procedure */
+ F_DICTIONARY_OUT, /* output procedure */
+ InvalidOid, /* receive procedure: none */
+ InvalidOid, /* send procedure: none */
+ InvalidOid, /* typmodin procedure - none */
+ InvalidOid, /* typmodout procedure - none */
+ InvalidOid, /* analyze procedure - default */
+ InvalidOid, /* subscript procedure - none */
+ InvalidOid, /* element type ID */
+ false, /* this is not an array type */
+ dictArrayOid, /* array type we are about to create */
+ InvalidOid, /* base type ID (only for domains) */
+ NULL, /* never a default type value */
+ NULL, /* binary default isn't sent either */
+ false, /* passed by value: same as for JSONB */
+ TYPALIGN_INT, /* int alignment */
+ TYPSTORAGE_EXTENDED, /* TOAST strategy: fully toastable, as
+ * JSONB */
+ -1, /* typMod (Domains only) */
+ 0, /* Array dimensions of typbasetype */
+ false, /* Type NOT NULL */
+ InvalidOid); /* type's collation */
+
+ /*
+ * Create the array type that goes with it.
+ */
+ dictArrayName = makeArrayTypeName(dictName, dictNamespace);
+
+ TypeCreate(dictArrayOid, /* force assignment of this type OID */
+ dictArrayName, /* type name */
+ dictNamespace, /* namespace */
+ InvalidOid, /* relation oid (n/a here) */
+ 0, /* relation kind (ditto) */
+ GetUserId(), /* owner's ID */
+ -1, /* internal size (always varlena) */
+ TYPTYPE_BASE, /* type-type (base type) */
+ TYPCATEGORY_ARRAY, /* type-category (array) */
+ false, /* array types are never preferred */
+ DEFAULT_TYPDELIM, /* array element delimiter */
+ F_ARRAY_IN, /* input procedure */
+ F_ARRAY_OUT, /* output procedure */
+ F_ARRAY_RECV, /* receive procedure */
+ F_ARRAY_SEND, /* send procedure */
+ InvalidOid, /* typmodin procedure - none */
+ InvalidOid, /* typmodout procedure - none */
+ F_ARRAY_TYPANALYZE, /* analyze procedure */
+ F_ARRAY_SUBSCRIPT_HANDLER, /* array subscript procedure */
+ dictTypeAddr.objectId, /* element type ID */
+ true, /* yes this is an array type */
+ InvalidOid, /* no further array type */
+ InvalidOid, /* base type ID */
+ NULL, /* never a default type value */
+ NULL, /* binary default isn't sent either */
+ false, /* passed by value: same as for JSONB */
+ TYPALIGN_INT, /* enums have int align, so do their arrays */
+ TYPSTORAGE_EXTENDED, /* ARRAY is always toastable */
+ -1, /* typMod (Domains only) */
+ 0, /* Array dimensions of typbasetype */
+ false, /* Type NOT NULL */
+ InvalidOid); /* type's collation */
+
+ pfree(dictArrayName);
+
+ /* Enter the dict's entries into pg_dict */
+ DictEntriesCreate(dictTypeAddr.objectId, stmt->vals);
+
+ /* Create casts to and from JSONB */
+ CastCreate(JSONBOID, dictTypeAddr.objectId, F_JSONB_DICTIONARY, 'a', 'f', DEPENDENCY_INTERNAL);
+ CastCreate(dictTypeAddr.objectId, JSONBOID, F_DICTIONARY_JSONB, 'i', 'f', DEPENDENCY_INTERNAL);
+
+ /*
+ * Create explicit cast to bytea. This is convenient for debugging purposes.
+ * Casting bytea to a dictionary type is dangerous and thus not supported.
+ */
+ CastCreate(dictTypeAddr.objectId, BYTEAOID, F_DICTIONARY_BYTEA, 'e', 'f', DEPENDENCY_INTERNAL);
+
+ return dictTypeAddr;
+}
+
/*
* DefineEnum
* Registers a new enum.
diff --git a/src/backend/executor/functions.c b/src/backend/executor/functions.c
index f9460ae506..7ad3de3a00 100644
--- a/src/backend/executor/functions.c
+++ b/src/backend/executor/functions.c
@@ -1708,6 +1708,7 @@ check_sql_fn_retval(List *queryTreeLists,
if (fn_typtype == TYPTYPE_BASE ||
fn_typtype == TYPTYPE_DOMAIN ||
+ fn_typtype == TYPTYPE_DICT ||
fn_typtype == TYPTYPE_ENUM ||
fn_typtype == TYPTYPE_RANGE ||
fn_typtype == TYPTYPE_MULTIRANGE)
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 205506305b..9905a1a828 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4440,6 +4440,18 @@ _copyCompositeTypeStmt(const CompositeTypeStmt *from)
return newnode;
}
+static CreateDictionaryStmt *
+_copyCreateDictionaryStmt(const CreateDictionaryStmt *from)
+{
+ CreateDictionaryStmt *newnode = makeNode(CreateDictionaryStmt);
+
+ COPY_NODE_FIELD(typeName);
+ COPY_NODE_FIELD(baseTypeName);
+ COPY_NODE_FIELD(vals);
+
+ return newnode;
+}
+
static CreateEnumStmt *
_copyCreateEnumStmt(const CreateEnumStmt *from)
{
@@ -6181,6 +6193,9 @@ copyObjectImpl(const void *from)
case T_CompositeTypeStmt:
retval = _copyCompositeTypeStmt(from);
break;
+ case T_CreateDictionaryStmt:
+ retval = _copyCreateDictionaryStmt(from);
+ break;
case T_CreateEnumStmt:
retval = _copyCreateEnumStmt(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 9688b22a4b..357b10a106 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -1962,6 +1962,16 @@ _equalCompositeTypeStmt(const CompositeTypeStmt *a, const CompositeTypeStmt *b)
return true;
}
+static bool
+_equalCreateDictionaryStmt(const CreateDictionaryStmt *a, const CreateDictionaryStmt *b)
+{
+ COMPARE_NODE_FIELD(typeName);
+ COMPARE_NODE_FIELD(baseTypeName);
+ COMPARE_NODE_FIELD(vals);
+
+ return true;
+}
+
static bool
_equalCreateEnumStmt(const CreateEnumStmt *a, const CreateEnumStmt *b)
{
@@ -3983,6 +3993,9 @@ equal(const void *a, const void *b)
case T_CompositeTypeStmt:
retval = _equalCompositeTypeStmt(a, b);
break;
+ case T_CreateDictionaryStmt:
+ retval = _equalCreateDictionaryStmt(a, b);
+ break;
case T_CreateEnumStmt:
retval = _equalCreateEnumStmt(a, b);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 989db0dbec..5c8a7e4586 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -6237,6 +6237,14 @@ DefineStmt:
n->coldeflist = $6;
$$ = (Node *) n;
}
+ | CREATE TYPE_P any_name AS DICTIONARY OF any_name '(' opt_enum_val_list ')'
+ {
+ CreateDictionaryStmt *n = makeNode(CreateDictionaryStmt);
+ n->typeName = $3;
+ n->baseTypeName = $7;
+ n->vals = $9;
+ $$ = (Node *)n;
+ }
| CREATE TYPE_P any_name AS ENUM_P '(' opt_enum_val_list ')'
{
CreateEnumStmt *n = makeNode(CreateEnumStmt);
diff --git a/src/backend/parser/parse_coerce.c b/src/backend/parser/parse_coerce.c
index c4e958e4aa..e489dd9421 100644
--- a/src/backend/parser/parse_coerce.c
+++ b/src/backend/parser/parse_coerce.c
@@ -279,7 +279,22 @@ coerce_type(ParseState *pstate, Node *node,
if (baseTypeId == INTERVALOID)
inputTypeMod = baseTypeMod;
else
- inputTypeMod = -1;
+ {
+ if((int32)targetTypeId == targetTypeMod)
+ {
+ /*
+ * For dictionaries internally we set typmod to a dictionary
+ * OID. This allows dictionary_in() and jsonb_dictionary() to
+ * distinguish one dictionary from another.
+ *
+ * It also allows us to recognize that the target type is
+ * a dictionary here and pass a correct typmod.
+ */
+ inputTypeMod = targetTypeMod;
+ }
+ else
+ inputTypeMod = -1;
+ }
baseType = typeidType(baseTypeId);
diff --git a/src/backend/parser/parse_type.c b/src/backend/parser/parse_type.c
index 307114a30d..868b24fd9f 100644
--- a/src/backend/parser/parse_type.c
+++ b/src/backend/parser/parse_type.c
@@ -259,6 +259,10 @@ LookupTypeNameOid(ParseState *pstate, const TypeName *typeName, bool missing_ok)
* This is equivalent to LookupTypeName, except that this will report
* a suitable error message if the type cannot be found or is not defined.
* Callers of this can therefore assume the result is a fully valid type.
+ *
+ * For a dictionary type it's OID is returned as a typmod. This is used by
+ * dictionary_in() and jsonb_dictionary() to distinguish one dictionary from
+ * another.
*/
Type
typenameType(ParseState *pstate, const TypeName *typeName, int32 *typmod_p)
@@ -305,6 +309,10 @@ typenameTypeId(ParseState *pstate, const TypeName *typeName)
*
* This is equivalent to typenameType, but we only hand back the type OID
* and typmod, not the syscache entry.
+ *
+ * For a dictionary type it's OID is returned as a typmod. This is used by
+ * dictionary_in() and jsonb_dictionary() to distinguish one dictionary from
+ * another.
*/
void
typenameTypeIdAndMod(ParseState *pstate, const TypeName *typeName,
@@ -327,6 +335,10 @@ typenameTypeIdAndMod(ParseState *pstate, const TypeName *typeName,
* looked up, and is passed as "typ".
*
* pstate is only used for error location info, and may be NULL.
+ *
+ * For a dictionary type it's OID is returned as a typmod. This is used by
+ * dictionary_in() and jsonb_dictionary() to distinguish one dictionary from
+ * another.
*/
static int32
typenameTypeMod(ParseState *pstate, const TypeName *typeName, Type typ)
@@ -339,6 +351,10 @@ typenameTypeMod(ParseState *pstate, const TypeName *typeName, Type typ)
ArrayType *arrtypmod;
ParseCallbackState pcbstate;
+ /* For a dictionary return it's OID. */
+ if (((Form_pg_type) GETSTRUCT(typ))->typtype == TYPTYPE_DICT)
+ return (int32)((Form_pg_type) GETSTRUCT(typ))->oid;
+
/* Return prespecified typmod if no typmod expressions */
if (typeName->typmods == NIL)
return typeName->typemod;
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index 0e7b7b3138..cffef5d707 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -174,6 +174,7 @@ ClassifyUtilityCommandAsReadOnly(Node *parsetree)
case T_CreateCastStmt:
case T_CreateConversionStmt:
case T_CreateDomainStmt:
+ case T_CreateDictionaryStmt:
case T_CreateEnumStmt:
case T_CreateEventTrigStmt:
case T_CreateExtensionStmt:
@@ -1621,6 +1622,10 @@ ProcessUtilitySlow(ParseState *pstate,
}
break;
+ case T_CreateDictionaryStmt: /* CREATE TYPE AS DICTIONARY OF */
+ address = DefineDictionary((CreateDictionaryStmt *) parsetree);
+ break;
+
case T_CreateEnumStmt: /* CREATE TYPE AS ENUM */
address = DefineEnum((CreateEnumStmt *) parsetree);
break;
@@ -2766,6 +2771,10 @@ CreateCommandTag(Node *parsetree)
tag = CMDTAG_CREATE_TYPE;
break;
+ case T_CreateDictionaryStmt:
+ tag = CMDTAG_CREATE_TYPE;
+ break;
+
case T_CreateEnumStmt:
tag = CMDTAG_CREATE_TYPE;
break;
@@ -3413,6 +3422,10 @@ GetCommandLogLevel(Node *parsetree)
lev = LOGSTMT_DDL;
break;
+ case T_CreateDictionaryStmt:
+ lev = LOGSTMT_DDL;
+ break;
+
case T_CreateEnumStmt:
lev = LOGSTMT_DDL;
break;
diff --git a/src/backend/utils/adt/Makefile b/src/backend/utils/adt/Makefile
index 7c722ea2ce..a7ec3d89f3 100644
--- a/src/backend/utils/adt/Makefile
+++ b/src/backend/utils/adt/Makefile
@@ -30,6 +30,7 @@ OBJS = \
datetime.o \
datum.o \
dbsize.o \
+ dictionaries.o \
domains.o \
encode.o \
enum.o \
diff --git a/src/backend/utils/adt/dictionaries.c b/src/backend/utils/adt/dictionaries.c
new file mode 100644
index 0000000000..364ed7cb20
--- /dev/null
+++ b/src/backend/utils/adt/dictionaries.c
@@ -0,0 +1,479 @@
+/*-------------------------------------------------------------------------
+ *
+ * dictionaries.c
+ * Conversion functions for dictionary types.
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/adt/dictionaries.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "catalog/pg_dict.h"
+#include "utils/fmgrprotos.h"
+#include "utils/jsonb.h"
+
+/*
+ * When compressing a data we treat it as a BLOB, in other words we don't
+ * assume anything regarding its internal representation. This is not
+ * necessarily the best and/or the only possible approach. However, it is
+ * universal and can be reused for JSONB, TEXT, XML and other types. Alternative
+ * compression methods can be introduced in the future, so that the user will
+ * be able to choose the best one for the task.
+ *
+ * The compressed data is stored in the following format:
+ *
+ * (struct varlena)
+ * dictionary_id [uint32]
+ * decompressed_size [uint32]
+ * algorithm_version [uint8]
+ *
+ * (repeated) {
+ * number of bytes to copy as-is [uint8]
+ * ... bytes to copy as-is ...
+ * dictionary entry id, or 0 to skip [uint32]
+ * }
+ *
+ * Compressed data is a variable-length type and thus has the 'struct varlena'
+ * header. The code below doesn't consider it being a part of the payload
+ * (e.g. see the DICT_COMP_HEADER_SIZE definition).
+ *
+ * Storing dictionary_id may seem redundant, but without it dictionary_out()
+ * and dictionary_jsonb() have no way to distinguish one dictionary type from
+ * another. dictionary_in() and jsonb_dictionary() get the dictionary id through
+ * the 'typemod' argument.
+ *
+ * Currently algorithm_version is always 0. In the future it will allow us to
+ * introduce new features (e.g. usage of varints) and/or lazily migrate the
+ * data to other compression methods.
+ *
+ * Compression and decompression are implemented based on a binary search over
+ * DictEntry[] array (see pg_dict.h). For compression the array is sorted by
+ * dictentries and for decompression - by oid's.
+ *
+ */
+
+/* Size of the header in the compressed data */
+#define DICT_COMP_HEADER_SIZE (sizeof(uint32)*2 + sizeof(uint8))
+
+/* Extracts dictionary_id from the compressed data */
+#define DICT_COMP_DICTIONARY_ID(hdrp) \
+ (*(uint32*)hdrp)
+
+/* Extracts decompressed_size from the compressed data */
+#define DICT_COMP_DECOMPRESSED_SIZE(hdrp) \
+ (*(uint32*)((uint8*)hdrp + sizeof(uint32)))
+
+/* Extracts algorithm_version from the compressed data */
+#define DICT_COMP_ALGORITHM_VERSION(hdrp) \
+ (*(uint8*)((uint8*)hdrp + sizeof(uint32)*2))
+
+/* Current algorithm_version */
+#define DICT_COMP_CURRENT_ALGORITHM_VERSION 0
+
+/*
+ * bsearch_arg() callback for finding DictEntry by oid.
+ */
+static int
+find_by_oid_cb(const void *key, const void *current, void *arg)
+{
+ /* Note that oids are unsigned, so we should be careful here */
+ if (*(Oid *) key < ((DictEntry *) current)->oid)
+ return -1;
+ else if (*(Oid *) key == ((DictEntry *) current)->oid)
+ return 0; /* found! */
+ else
+ return 1;
+}
+
+/*
+ * qsort() callback for sorting DictEntry[] by oids.
+ */
+static int
+sort_by_oids_cb(const void *left, const void *right)
+{
+ /* Note that oids are unsigned, so we should be careful here */
+ if (((DictEntry *) left)->oid < ((DictEntry *) right)->oid)
+ return -1;
+
+ /* Oids are unique so this callback will never return 0 */
+ return 1;
+}
+
+/*
+ * Finds a DictEntry which dictentry field matches *data and returns its Oid.
+ * If there are several matching entries, the largest is returned. The length
+ * of the found entry is written to *found_length on success. On failure
+ * InvalidOid is returned and found_length is zeroed.
+ *
+ * The implementation is similar to bsearch_arg(). The procedure can't be used
+ * directly because we are looking not by the exact match. We could generalize
+ * this case but the signature of the function becomes so complicated that it
+ * doesn't seem to worth the effort.
+ */
+static Oid
+compress_find_oid(Dictionary dict, const uint8 *data, Size data_size, Size *found_length)
+{
+ int res;
+ int32 left = 0;
+ int32 right = dict->nentries - 1;
+ Size best_length = 0;
+ Oid best_match = InvalidOid;
+
+ while (left <= right)
+ {
+ int32 current = (left + right) / 2;
+ Size nbytes = (Size)dict->entries[current].length;
+
+ if (nbytes > data_size)
+ {
+ /* current can be less or greater depending on the prefix */
+ res = memcmp(dict->entries[current].dictentry.data, data, data_size);
+
+ /* if prefixes match, current is greater */
+ if (res == 0)
+ res = 1;
+ }
+ else
+ res = memcmp(dict->entries[current].dictentry.data, data, nbytes);
+
+ if (res == 0) /* match found */
+ {
+ best_length = nbytes;
+ best_match = dict->entries[current].oid;
+
+ if (nbytes == data_size)
+ break;
+
+ /* maybe there is a larger match */
+ left = current + 1;
+ }
+ else if (res < 0) /* current is less */
+ left = current + 1;
+ else /* current is greater */
+ right = current - 1;
+ }
+
+ *found_length = best_length;
+ return best_match;
+}
+
+/*
+ * Finds a DictEntry by Oid using a binary search. The dictionary should be
+ * sorted by oids before the call. Returns NULL if nothing was found.
+ */
+static DictEntry *
+decompress_find_dictentry(Dictionary dict, Oid oid)
+{
+ return (DictEntry *) bsearch_arg(&oid, dict->entries, dict->nentries, sizeof(DictEntry), find_by_oid_cb, NULL);
+}
+
+/*
+ * Estimates the worst-case compressed size of the data of given size.
+ * Worst-case scenario happens when the dictionary consists of single-character
+ * entries. In this case every byte will be encoded as 6 bytes:
+ * 0x00, (0 bytes to copy as-is), 4 bytes of the entry Oid
+ *
+ * This procedure doesn't account for the header size.
+ *
+ * AALEKSEEV FIXME don't use dictionary entries shorter than 5 bytes
+ */
+static Size
+worst_case_compressed_size(Size insize)
+{
+ return insize * 6;
+}
+
+/*
+ * Compresses the data using the provided dictionary. The dictionary should
+ * be sorted by dictentries before the call. Output buffer should be at least
+ * worst_case_compressed_size(src_size) bytes in size.
+ */
+static void
+compress(Dictionary dict,
+ const void *src_data_, Size src_size,
+ void *encoded_data_, Size *pencoded_size)
+{
+ Size nbytes;
+ Size inoffset;
+ Size outskipoffset = 0;
+ Size outoffset = 1;
+ uint8 skipbytes = 0;
+ const uint8 *src_data = src_data_;
+ uint8 *encoded_data = ((uint8 *) encoded_data_);
+
+ for (inoffset = 0; inoffset < src_size;)
+ {
+ Oid code = compress_find_oid(dict, &(src_data[inoffset]),
+ src_size - inoffset, &nbytes);
+
+ if (code == InvalidOid)
+ {
+ skipbytes++;
+ encoded_data[outoffset] = src_data[inoffset];
+ outoffset++;
+ inoffset++;
+
+ if (skipbytes == 255)
+ {
+ encoded_data[outskipoffset] = skipbytes;
+ encoded_data[outoffset++] = 0; /* InvalidOid */
+ encoded_data[outoffset++] = 0;
+ encoded_data[outoffset++] = 0;
+ encoded_data[outoffset++] = 0;
+ outskipoffset = outoffset++;
+ skipbytes = 0;
+ }
+ }
+ else
+ {
+ encoded_data[outskipoffset] = skipbytes;
+ encoded_data[outoffset++] = (code >> 24) & 0xFF;
+ encoded_data[outoffset++] = (code >> 16) & 0xFF;
+ encoded_data[outoffset++] = (code >> 8) & 0xFF;
+ encoded_data[outoffset++] = code & 0xFF;
+ outskipoffset = outoffset++;
+ skipbytes = 0;
+ inoffset += nbytes;
+ }
+ }
+
+ /* Double check that we didn't write out of buffer */
+ Assert(outoffset < worst_case_compressed_size(src_size));
+
+ encoded_data[outskipoffset] = skipbytes;
+ *pencoded_size = outoffset;
+}
+
+/*
+ * Report an internal error in decompress() procedure below.
+ * Under normal circumstances this should never happen.
+ */
+static void
+decompress_error()
+{
+ ereport(ERROR,
+ (
+ errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("Unable to decompress a dictionary type"),
+ errdetail("The compressed data seems to be corrupted"),
+ errhint("Please report the steps to reproduce the issue to pgsql-bugs@")
+ ));
+}
+
+/*
+ * Decompresses the data using the provided dictionary. The dictionary should
+ * be sorted by oids before the call.
+ */
+static void
+decompress(Dictionary dict,
+ const void *encoded_data_, Size encoded_size,
+ void *decoded_data_, Size decoded_size)
+{
+ Size inoffset = 0;
+ Size outoffset = 0;
+ Oid code;
+ uint8 skipbytes;
+ const uint8 *encoded_data = ((uint8 *) encoded_data_);
+ uint8 *decoded_data = decoded_data_;
+
+ for (inoffset = 0; inoffset < encoded_size;)
+ {
+ skipbytes = encoded_data[inoffset++];
+
+ if (skipbytes > decoded_size - outoffset)
+ decompress_error();
+
+ if (skipbytes > encoded_size - inoffset)
+ decompress_error();
+
+ memcpy(
+ &(decoded_data[outoffset]),
+ &(encoded_data[inoffset]),
+ skipbytes
+ );
+
+ outoffset += skipbytes;
+ inoffset += skipbytes;
+
+ if ((encoded_size == inoffset) && (decoded_size == outoffset))
+ break; /* end of input - its OK */
+
+ if (encoded_size - inoffset < 4)
+ decompress_error();
+
+ code = (Oid) encoded_data[inoffset++];
+ code = (code << 8) | (Oid) encoded_data[inoffset++];
+ code = (code << 8) | (Oid) encoded_data[inoffset++];
+ code = (code << 8) | (Oid) encoded_data[inoffset++];
+
+ if (code != InvalidOid)
+ {
+ Size entrylen;
+ DictEntry *entry = decompress_find_dictentry(dict, code);
+
+ if (entry == NULL)
+ decompress_error();
+
+ Assert(entry->oid == code);
+ entrylen = (Size)entry->length;
+
+ if (entrylen > decoded_size - outoffset)
+ decompress_error();
+
+ memcpy(
+ &(decoded_data[outoffset]),
+ entry->dictentry.data,
+ entrylen
+ );
+
+ outoffset += entrylen;
+ }
+ }
+
+ Assert(decoded_size == outoffset);
+}
+
+/*
+ * Converts a cstring to a dictionary.
+ */
+Datum
+dictionary_in(PG_FUNCTION_ARGS)
+{
+ const char *instr = PG_GETARG_CSTRING(0);
+#ifdef NOT_USED
+ Oid typelem = PG_GETARG_OID(1);
+#endif
+ int32 typmod = PG_GETARG_INT32(2);
+ Jsonb *jsonb;
+ bytea *dict;
+
+ Assert(typmod != -1);
+
+ jsonb = DatumGetJsonbP(DirectFunctionCall1(jsonb_in, CStringGetDatum(instr)));
+ dict = DatumGetByteaP(DirectFunctionCall2(jsonb_dictionary, JsonbPGetDatum(jsonb), Int32GetDatum(typmod)));
+
+ PG_RETURN_BYTEA_P(dict);
+}
+
+/*
+ * Converts a dictionary to a cstring.
+ */
+Datum
+dictionary_out(PG_FUNCTION_ARGS)
+{
+ bytea *dict = PG_GETARG_BYTEA_P(0);
+ Jsonb *jsonb = DatumGetJsonbP(DirectFunctionCall1(dictionary_jsonb, PointerGetDatum(dict)));
+ const char *outstr = DatumGetCString(DirectFunctionCall1(jsonb_out, JsonbPGetDatum(jsonb)));
+
+ PG_RETURN_CSTRING(outstr);
+}
+
+/*
+ * Coverts JSONB to a dictionary type.
+ *
+ * AALEKSEEV FIXME: if the compressed document ends up being larger than
+ * the original one write a corresponding flag and copy all the data as-is.
+ */
+Datum
+jsonb_dictionary(PG_FUNCTION_ARGS)
+{
+ Dictionary dict;
+ Jsonb *jsonb = PG_GETARG_JSONB_P(0);
+ int32 typmod = PG_GETARG_INT32(1);
+ uint8 *jsonb_data = (uint8 *) VARDATA(jsonb);
+ Size jsonb_data_size = VARSIZE(jsonb) - VARHDRSZ;
+ uint8 *encoded_buff,
+ *encoded_header,
+ *encoded_data;
+ Size encoded_size,
+ encoded_buff_size;
+
+ Assert(typmod != -1);
+
+ dict = DictEntriesRead(typmod);
+ qsort((void *) dict->entries, dict->nentries, sizeof(DictEntry), sort_by_oids_cb);
+
+ encoded_buff_size = VARHDRSZ + DICT_COMP_HEADER_SIZE + worst_case_compressed_size(jsonb_data_size);
+ encoded_buff = palloc(encoded_buff_size);
+ encoded_header = (uint8 *) VARDATA(encoded_buff);
+ encoded_data = encoded_header + DICT_COMP_HEADER_SIZE;
+
+ DICT_COMP_DICTIONARY_ID(encoded_header) = typmod;
+ DICT_COMP_DECOMPRESSED_SIZE(encoded_header) = jsonb_data_size;
+ DICT_COMP_ALGORITHM_VERSION(encoded_header) = DICT_COMP_CURRENT_ALGORITHM_VERSION;
+
+ encoded_size = encoded_buff_size - VARHDRSZ - DICT_COMP_HEADER_SIZE;
+
+ compress(dict, jsonb_data, jsonb_data_size,
+ encoded_data, &encoded_size);
+
+ encoded_size += VARHDRSZ + DICT_COMP_HEADER_SIZE;
+
+ encoded_buff = repalloc(encoded_buff, encoded_size);
+ SET_VARSIZE(encoded_buff, encoded_size);
+
+ pfree(dict);
+ PG_RETURN_BYTEA_P(encoded_buff);
+}
+
+/*
+ * Converts a dictionary type to JSONB.
+ */
+Datum
+dictionary_jsonb(PG_FUNCTION_ARGS)
+{
+ bytea *encoded_buff = PG_GETARG_BYTEA_P(0);
+ uint8 *encoded_header = (uint8 *) VARDATA(encoded_buff);
+ uint8 *encoded_data = encoded_header + DICT_COMP_HEADER_SIZE;
+ Size encoded_size = VARSIZE(encoded_buff) - VARHDRSZ - DICT_COMP_HEADER_SIZE;
+ int alg_version = DICT_COMP_ALGORITHM_VERSION(encoded_header);
+ Oid dictOid; /* cannot read until algorithm version is
+ * checked */
+ uint32 decoded_size; /* cannot read until algorithm version is
+ * checked */
+ Jsonb *jsonb;
+ uint8 *jsonb_data;
+ Dictionary dict;
+
+ if (alg_version > DICT_COMP_CURRENT_ALGORITHM_VERSION)
+ ereport(ERROR,
+ (
+ errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("Unsupported compression algorithm version"),
+ errdetail("Saved algorithm version is %d, current version is %d",
+ alg_version, DICT_COMP_CURRENT_ALGORITHM_VERSION),
+ errhint("The data is either corrupted or imported from "
+ "the future version of PostgreSQL")
+ ));
+
+ dictOid = DICT_COMP_DICTIONARY_ID(encoded_header);
+ decoded_size = DICT_COMP_DECOMPRESSED_SIZE(encoded_header);
+ dict = DictEntriesRead(dictOid);
+
+ jsonb = palloc(decoded_size + VARHDRSZ);
+ jsonb_data = (uint8 *) VARDATA(jsonb);
+
+ decompress(dict, encoded_data, encoded_size, jsonb_data, decoded_size);
+
+ decoded_size += VARHDRSZ;
+ SET_VARSIZE(jsonb, decoded_size);
+
+ pfree(dict);
+ PG_RETURN_JSONB_P(jsonb);
+}
+
+/*
+ * Converts a dictionary type to bytea.
+ */
+Datum
+dictionary_bytea(PG_FUNCTION_ARGS)
+{
+ bytea *compressed_data = PG_GETARG_BYTEA_P(0);
+ PG_RETURN_BYTEA_P(compressed_data);
+}
\ No newline at end of file
diff --git a/src/backend/utils/fmgr/funcapi.c b/src/backend/utils/fmgr/funcapi.c
index 9197b0f1e2..4f9dc52428 100644
--- a/src/backend/utils/fmgr/funcapi.c
+++ b/src/backend/utils/fmgr/funcapi.c
@@ -1292,6 +1292,7 @@ get_type_func_class(Oid typid, Oid *base_typeid)
case TYPTYPE_COMPOSITE:
return TYPEFUNC_COMPOSITE;
case TYPTYPE_BASE:
+ case TYPTYPE_DICT:
case TYPTYPE_ENUM:
case TYPTYPE_RANGE:
case TYPTYPE_MULTIRANGE:
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7cc9c72e49..1c85b3f7b0 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -10178,6 +10178,8 @@ dumpType(Archive *fout, const TypeInfo *tyinfo)
dumpDomain(fout, tyinfo);
else if (tyinfo->typtype == TYPTYPE_COMPOSITE)
dumpCompositeType(fout, tyinfo);
+ else if (tyinfo->typtype == TYPTYPE_DICT)
+ pg_log_error("AALEKSEEV TODO FIXME not implemented");
else if (tyinfo->typtype == TYPTYPE_ENUM)
dumpEnumType(fout, tyinfo);
else if (tyinfo->typtype == TYPTYPE_RANGE)
diff --git a/src/include/catalog/pg_dict.h b/src/include/catalog/pg_dict.h
new file mode 100644
index 0000000000..d05303f724
--- /dev/null
+++ b/src/include/catalog/pg_dict.h
@@ -0,0 +1,76 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_dict.h
+ * definition of the "dict" system catalog (pg_dict)
+ *
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ * src/include/catalog/pg_dict.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef PG_DICT_H
+#define PG_DICT_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_dict_d.h"
+
+#include "nodes/pg_list.h"
+
+/* ----------------
+ * pg_dict definition. cpp turns this into
+ * typedef struct FormData_pg_dict
+ * ----------------
+ */
+CATALOG(pg_dict,9861,DictRelationId)
+{
+ Oid oid; /* oid */
+ Oid dicttypid BKI_LOOKUP(pg_type); /* OID of owning dict type */
+ NameData dictentry; /* text representation of the dictionary entry */
+} FormData_pg_dict;
+
+/* ----------------
+ * Form_pg_dict corresponds to a pointer to a tuple with
+ * the format of pg_dict relation.
+ * ----------------
+ */
+typedef FormData_pg_dict *Form_pg_dict;
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_dict_oid_index, 9862, DictOidIndexId, on pg_dict using btree(oid oid_ops));
+DECLARE_UNIQUE_INDEX(pg_dict_typid_entry_index, 9863, DictTypIdEntryIndexId, on pg_dict using btree(dicttypid oid_ops, dictentry name_ops));
+
+
+/*
+ * DictEntry type represents one entry in the given dictionary.
+ */
+typedef struct
+{
+ Oid oid; /* entry Oid */
+ uint32 length; /* entry length */
+ NameData dictentry; /* the entry */
+} DictEntry;
+
+/*
+ * Dictionary type represents all the entries in the given dictionary.
+ */
+typedef struct
+{
+ uint32 nentries; /* number of entries */
+ DictEntry *entries; /* array of entries */
+} DictionaryData;
+
+typedef DictionaryData *Dictionary;
+
+/*
+ * prototypes for functions in pg_dict.c
+ */
+extern void DictEntriesCreate(Oid dictTypeOid, List *vals);
+extern Dictionary DictEntriesRead(Oid dictTypeOid);
+extern void DictEntriesDelete(Oid dictTypeOid);
+
+#endif /* PG_DICT_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index babe16f00a..81a020bc2f 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10024,6 +10024,26 @@
proname => 'jsonb_path_match_opr', prorettype => 'bool',
proargtypes => 'jsonb jsonpath', prosrc => 'jsonb_path_match_opr' },
+# dictionaries
+{ oid => '9864', descr => 'Converts a cstring to a dictionary',
+ proname => 'dictionary_in', prorettype => 'any', proargtypes => 'cstring oid int4',
+ prosrc => 'dictionary_in' },
+{ oid => '9865', descr => 'Converts a dictionary to a cstring',
+ proname => 'dictionary_out', prorettype => 'cstring', proargtypes => 'any',
+ prosrc => 'dictionary_out' },
+{ oid => '9866', descr => 'Coverts JSONB to a dictionary type',
+ proname => 'jsonb_dictionary', proisstrict => 'f', provolatile => 's',
+ prorettype => 'any', proargtypes => 'jsonb int4',
+ prosrc => 'jsonb_dictionary' },
+{ oid => '9867', descr => 'Converts a dictionary type to JSONB',
+ proname => 'dictionary_jsonb', proisstrict => 'f', provolatile => 's',
+ prorettype => 'jsonb', proargtypes => 'any',
+ prosrc => 'dictionary_jsonb' },
+{ oid => '9868', descr => 'Converts a dictionary type to a byte array',
+ proname => 'dictionary_bytea', proisstrict => 'f', provolatile => 's',
+ prorettype => 'jsonb', proargtypes => 'any',
+ prosrc => 'dictionary_bytea' },
+
# historical int8/txid_snapshot variants of xid8 functions
{ oid => '2939', descr => 'I/O',
proname => 'txid_snapshot_in', prorettype => 'txid_snapshot',
diff --git a/src/include/catalog/pg_type.h b/src/include/catalog/pg_type.h
index 48a2559137..91c6bfae99 100644
--- a/src/include/catalog/pg_type.h
+++ b/src/include/catalog/pg_type.h
@@ -272,6 +272,7 @@ DECLARE_UNIQUE_INDEX(pg_type_typname_nsp_index, 2704, TypeNameNspIndexId, on pg_
*/
#define TYPTYPE_BASE 'b' /* base type (ordinary scalar type) */
#define TYPTYPE_COMPOSITE 'c' /* composite (e.g., table's rowtype) */
+#define TYPTYPE_DICT 'D' /* dictionary */
#define TYPTYPE_DOMAIN 'd' /* domain over another type */
#define TYPTYPE_ENUM 'e' /* enumerated type */
#define TYPTYPE_MULTIRANGE 'm' /* multirange type */
diff --git a/src/include/commands/typecmds.h b/src/include/commands/typecmds.h
index a17bedb851..5cc8e5ff99 100644
--- a/src/include/commands/typecmds.h
+++ b/src/include/commands/typecmds.h
@@ -24,6 +24,7 @@
extern ObjectAddress DefineType(ParseState *pstate, List *names, List *parameters);
extern void RemoveTypeById(Oid typeOid);
extern ObjectAddress DefineDomain(CreateDomainStmt *stmt);
+extern ObjectAddress DefineDictionary(CreateDictionaryStmt *stmt);
extern ObjectAddress DefineEnum(CreateEnumStmt *stmt);
extern ObjectAddress DefineRange(ParseState *pstate, CreateRangeStmt *stmt);
extern ObjectAddress AlterEnum(AlterEnumStmt *stmt);
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index b3b407579b..031369cee3 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -408,6 +408,7 @@ typedef enum NodeTag
T_DropOwnedStmt,
T_ReassignOwnedStmt,
T_CompositeTypeStmt,
+ T_CreateDictionaryStmt,
T_CreateEnumStmt,
T_CreateRangeStmt,
T_AlterEnumStmt,
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 73f635b455..4e2e7af20d 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3563,6 +3563,18 @@ typedef struct CompositeTypeStmt
List *coldeflist; /* list of ColumnDef nodes */
} CompositeTypeStmt;
+/* ----------------------
+ * Create Type Statement, dictionary types
+ * ----------------------
+ */
+typedef struct CreateDictionaryStmt
+{
+ NodeTag type;
+ List *typeName; /* qualified name (list of String) */
+ List *baseTypeName; /* qualified base type name (list of String) */
+ List *vals; /* dictionary entries (list of String) */
+} CreateDictionaryStmt;
+
/* ----------------------
* Create Type Statement, enum types
* ----------------------
diff --git a/src/pl/plpgsql/src/pl_comp.c b/src/pl/plpgsql/src/pl_comp.c
index b791c23f06..897051c1f7 100644
--- a/src/pl/plpgsql/src/pl_comp.c
+++ b/src/pl/plpgsql/src/pl_comp.c
@@ -2129,6 +2129,7 @@ build_datatype(HeapTuple typeTup, int32 typmod,
switch (typeStruct->typtype)
{
case TYPTYPE_BASE:
+ case TYPTYPE_DICT:
case TYPTYPE_ENUM:
case TYPTYPE_RANGE:
case TYPTYPE_MULTIRANGE:
diff --git a/src/test/modules/test_oat_hooks/test_oat_hooks.c b/src/test/modules/test_oat_hooks/test_oat_hooks.c
index 7ef272cc7a..0dceca435a 100644
--- a/src/test/modules/test_oat_hooks/test_oat_hooks.c
+++ b/src/test/modules/test_oat_hooks/test_oat_hooks.c
@@ -1380,6 +1380,9 @@ nodetag_to_string(NodeTag tag)
case T_CompositeTypeStmt:
return "CompositeTypeStmt";
break;
+ case T_CreateDictionaryStmt:
+ return "CreateDictionaryStmt";
+ break;
case T_CreateEnumStmt:
return "CreateEnumStmt";
break;
diff --git a/src/test/regress/expected/dict.out b/src/test/regress/expected/dict.out
new file mode 100644
index 0000000000..699e76b654
--- /dev/null
+++ b/src/test/regress/expected/dict.out
@@ -0,0 +1,47 @@
+--
+-- Compression dictionaries tests
+--
+-- Dictionary types currently can be used only with JSONB
+\set ON_ERROR_STOP 0
+CREATE TYPE textdict AS DICTIONARY OF TEXT ('abcdef', 'ghijkl');
+ERROR: Dictionary types currently can be used only with JSONB
+\set ON_ERROR_STOP 1
+-- Simple happy path
+CREATE TYPE mydict AS DICTIONARY OF JSONB ('abcdef', 'ghijkl');
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
+ dictentry
+-----------
+ abcdef
+ ghijkl
+(2 rows)
+
+SELECT '{"abcdef":"ghijkl"}' :: mydict;
+ mydict
+----------------------
+ {"abcdef": "ghijkl"}
+(1 row)
+
+SELECT '{"abcdef":"ghijkl"}' :: mydict :: jsonb;
+ jsonb
+----------------------
+ {"abcdef": "ghijkl"}
+(1 row)
+
+SELECT '{"abcdef":"ghijkl"}' :: jsonb :: mydict;
+ mydict
+----------------------
+ {"abcdef": "ghijkl"}
+(1 row)
+
+SELECT ('{"abcdef":"ghijkl"}' :: mydict) -> 'abcdef';
+ ?column?
+----------
+ "ghijkl"
+(1 row)
+
+DROP TYPE mydict;
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
+ dictentry
+-----------
+(0 rows)
+
diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out
index 215eb899be..91bc491a6c 100644
--- a/src/test/regress/expected/oidjoins.out
+++ b/src/test/regress/expected/oidjoins.out
@@ -182,6 +182,7 @@ NOTICE: checking pg_description {classoid} => pg_class {oid}
NOTICE: checking pg_cast {castsource} => pg_type {oid}
NOTICE: checking pg_cast {casttarget} => pg_type {oid}
NOTICE: checking pg_cast {castfunc} => pg_proc {oid}
+NOTICE: checking pg_dict {dicttypid} => pg_type {oid}
NOTICE: checking pg_enum {enumtypid} => pg_type {oid}
NOTICE: checking pg_namespace {nspowner} => pg_authid {oid}
NOTICE: checking pg_conversion {connamespace} => pg_namespace {oid}
diff --git a/src/test/regress/expected/opr_sanity.out b/src/test/regress/expected/opr_sanity.out
index 86d755aa44..5dc4d0ab01 100644
--- a/src/test/regress/expected/opr_sanity.out
+++ b/src/test/regress/expected/opr_sanity.out
@@ -388,11 +388,12 @@ WHERE 'cstring'::regtype = ANY (p1.proargtypes)
AND NOT EXISTS(SELECT 1 FROM pg_conversion WHERE conproc = p1.oid)
AND p1.oid != 'shell_in(cstring)'::regprocedure
ORDER BY 1;
- oid | proname
-------+--------------
+ oid | proname
+------+---------------
2293 | cstring_out
2501 | cstring_send
-(2 rows)
+ 9864 | dictionary_in
+(3 rows)
-- Likewise, look for functions that return cstring and aren't datatype output
-- functions nor typmod output functions.
@@ -405,11 +406,12 @@ WHERE p1.prorettype = 'cstring'::regtype
AND NOT EXISTS(SELECT 1 FROM pg_type WHERE typmodout = p1.oid)
AND p1.oid != 'shell_out(void)'::regprocedure
ORDER BY 1;
- oid | proname
-------+--------------
+ oid | proname
+------+----------------
2292 | cstring_in
2500 | cstring_recv
-(2 rows)
+ 9865 | dictionary_out
+(3 rows)
-- Check for length inconsistencies between the various argument-info arrays.
SELECT p1.oid, p1.proname
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 103e11483d..2fa15a2080 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -111,7 +111,7 @@ test: select_views portals_p2 foreign_key cluster dependency guc bitmapops combo
# ----------
# Another group of parallel tests (JSON related)
# ----------
-test: json jsonb json_encoding jsonpath jsonpath_encoding jsonb_jsonpath sqljson json_sqljson jsonb_sqljson
+test: dict json jsonb json_encoding jsonpath jsonpath_encoding jsonb_jsonpath sqljson json_sqljson jsonb_sqljson
# ----------
# Another group of parallel tests
diff --git a/src/test/regress/sql/dict.sql b/src/test/regress/sql/dict.sql
new file mode 100644
index 0000000000..27ecd6a451
--- /dev/null
+++ b/src/test/regress/sql/dict.sql
@@ -0,0 +1,20 @@
+--
+-- Compression dictionaries tests
+--
+
+-- Dictionary types currently can be used only with JSONB
+\set ON_ERROR_STOP 0
+CREATE TYPE textdict AS DICTIONARY OF TEXT ('abcdef', 'ghijkl');
+\set ON_ERROR_STOP 1
+
+-- Simple happy path
+CREATE TYPE mydict AS DICTIONARY OF JSONB ('abcdef', 'ghijkl');
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
+
+SELECT '{"abcdef":"ghijkl"}' :: mydict;
+SELECT '{"abcdef":"ghijkl"}' :: mydict :: jsonb;
+SELECT '{"abcdef":"ghijkl"}' :: jsonb :: mydict;
+SELECT ('{"abcdef":"ghijkl"}' :: mydict) -> 'abcdef';
+
+DROP TYPE mydict;
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
--
2.35.1
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: [PATCH] Compression dictionaries for JSONB
@ 2022-06-05 19:51 Matthias van de Meent <[email protected]>
parent: Aleksander Alekseev <[email protected]>
0 siblings, 1 reply; 88+ messages in thread
From: Matthias van de Meent @ 2022-06-05 19:51 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>; Alvaro Herrera <[email protected]>; Zhihong Yu <[email protected]>; Matthias van de Meent <[email protected]>
On Fri, 13 May 2022 at 10:09, Aleksander Alekseev
<[email protected]> wrote:
>
> Hi hackers,
>
> > Here it the 2nd version of the patch:
> >
> > - Includes changes named above
> > - Fixes a warning reported by cfbot
> > - Fixes some FIXME's
> > - The path includes some simple tests now
> > - A proper commit message was added
>
> Here is the rebased version of the patch. Changes compared to v2 are minimal.
>
> > Open questions:
> > - Dictionary entries are currently stored as NameData, the same type that is
> > used for enums. Are we OK with the accompanying limitations? Any alternative
> > suggestions?
> > - All in all, am I moving the right direction?
>
> I would like to receive a little bit more feedback before investing more time
> into this effort. This will allow me, if necessary, to alter the overall design
> more easily.
Sorry for the delayed reply. After the last thread, I've put some time
in looking into the "pluggable toaster" patches, which appears to want
to provide related things: Compressing typed data using an extensible
API. I think that that API is a better approach to increase the
compression ratio for JSONB.
That does not mean that I think that the basis of this patch is
incorrect, just that the current API (through new entries in the
pg_type and pg_casts catalogs) is not the right direction if/when
we're going to have a pluggable toaster API. The bulk of the patch
should still be usable, but I think that the way it interfaces with
the CREATE TABLE (column ...) APIs would need reworking to build on
top of the api's of the "pluggable toaster" patches (so, creating
toasters instead of types). I think that would allow for an overall
better user experience and better performance due to decreased need
for fully decompressed type casting.
Kind regards,
Matthias van de Meent.
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: [PATCH] Compression dictionaries for JSONB
@ 2022-06-15 12:38 Aleksander Alekseev <[email protected]>
parent: Matthias van de Meent <[email protected]>
0 siblings, 1 reply; 88+ messages in thread
From: Aleksander Alekseev @ 2022-06-15 12:38 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>; +Cc: Matthias van de Meent <[email protected]>; Alvaro Herrera <[email protected]>; Zhihong Yu <[email protected]>; Teodor Sigaev <[email protected]>
Hi Matthias,
> The bulk of the patch
> should still be usable, but I think that the way it interfaces with
> the CREATE TABLE (column ...) APIs would need reworking to build on
> top of the api's of the "pluggable toaster" patches (so, creating
> toasters instead of types). I think that would allow for an overall
> better user experience and better performance due to decreased need
> for fully decompressed type casting.
Many thanks for the feedback.
The "pluggable TOASTer" patch looks very interesting indeed. I'm
currently trying to make heads and tails of it and trying to figure
out if it can be used as a base for compression dictionaries,
especially for implementing the partial decompression. Hopefully I
will be able to contribute to it and to the dependent patch [1] in the
upcoming CF, at least as a tester/reviewer. Focusing our efforts on
[1] for now seems to be a good strategy.
My current impression of your idea is somewhat mixed at this point though.
Teodor's goal is to allow creating _extensions_ that implement
alternative TOAST strategies, which use alternative compression
algorithms and/or use the knowledge of the binary representation of
the particular type. For sure, this would be a nice thing to have.
However, during the discussion of the "compression dictionaries" RFC
the consensus was reached that the community wants to see it as a
_built_in_ functionality rather than an extension. Otherwise we could
simply add ZSON to /contrib/ as it was originally proposed.
So if we are going to keep "compression dictionaries" a built-in
functionality, putting artificial constraints on its particular
implementation, or adding artificial dependencies of two rather
complicated patches, is arguably a controversial idea. Especially
considering the fact that it was shown that the feature can be
implemented without these dependencies, in a very non-invasive way.
These are just my initial thoughts I would like to share though. I may
change my mind after diving deeper into a "pluggable TOASTer" patch.
I cc:'ed Teodor in case he would like to share his insights on the topic.
[1]: https://commitfest.postgresql.org/38/3479/
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: [PATCH] Compression dictionaries for JSONB
@ 2022-06-17 15:04 Aleksander Alekseev <[email protected]>
parent: Aleksander Alekseev <[email protected]>
0 siblings, 1 reply; 88+ messages in thread
From: Aleksander Alekseev @ 2022-06-17 15:04 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>; +Cc: Matthias van de Meent <[email protected]>; Alvaro Herrera <[email protected]>; Zhihong Yu <[email protected]>; Teodor Sigaev <[email protected]>
Hi Matthias,
> These are just my initial thoughts I would like to share though. I may
> change my mind after diving deeper into a "pluggable TOASTer" patch.
I familiarized myself with the "pluggable TOASTer" thread and joined
the discussion [1].
I'm afraid so far I failed to understand your suggestion to base
"compression dictionaries" patch on "pluggable TOASTer", considering
the fair amount of push-back it got from the community, not to mention
a somewhat raw state of the patchset. It's true that Teodor and I are
trying to address similar problems. This however doesn't mean that
there should be a dependency between these patches.
Also, I completely agree with Tomas [2]:
> My main point is that we should not be making too many radical
> changes at once - it makes it much harder to actually get anything done.
IMO the patches don't depend on each other but rather complement each
other. The user can switch between different TOAST methods, and the
compression dictionaries can work on top of different TOAST methods.
Although there is also a high-level idea (according to the
presentations) to share common data between different TOASTed values,
similarly to what compression dictionaries do, by looking at the
current feedback and considering the overall complexity and the amount
of open questions (e.g. interaction with different TableAMs, etc), I
seriously doubt that this particular part of "pluggable TOASTer" will
end-up in the core.
[1]: https://postgr.es/m/CAJ7c6TOMPiRs-CZ%3DA9hyzxOyqHhKXxLD8qCF5%2BGJuLjQBzOX4A%40mail.gmail.com
[2]: https://postgr.es/m/9ef14537-b33b-c63a-9938-e2b413db0a4c%40enterprisedb.com
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: [PATCH] Compression dictionaries for JSONB
@ 2022-07-04 12:45 Matthias van de Meent <[email protected]>
parent: Aleksander Alekseev <[email protected]>
0 siblings, 1 reply; 88+ messages in thread
From: Matthias van de Meent @ 2022-07-04 12:45 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>; Alvaro Herrera <[email protected]>; Zhihong Yu <[email protected]>; Teodor Sigaev <[email protected]>
Hi Alexander,
On Fri, 17 Jun 2022 at 17:04, Aleksander Alekseev
<[email protected]> wrote:
>> These are just my initial thoughts I would like to share though. I may
>> change my mind after diving deeper into a "pluggable TOASTer" patch.
>
> I familiarized myself with the "pluggable TOASTer" thread and joined
> the discussion [1].
>
> I'm afraid so far I failed to understand your suggestion to base
> "compression dictionaries" patch on "pluggable TOASTer", considering
> the fair amount of push-back it got from the community, not to mention
> a somewhat raw state of the patchset. It's true that Teodor and I are
> trying to address similar problems. This however doesn't mean that
> there should be a dependency between these patches.
The reason I think this is better implemented as a pluggable toaster
is because casts are necessarily opaque and require O(sizeofdata)
copies or processing. The toaster infrastructure that is proposed in
[0] seems to improve on the O(sizeofdata) requirement for toast, but
that will not work with casts.
> Also, I completely agree with Tomas [2]:
>
>> My main point is that we should not be making too many radical
>> changes at once - it makes it much harder to actually get anything done.
>
> IMO the patches don't depend on each other but rather complement each
> other. The user can switch between different TOAST methods, and the
> compression dictionaries can work on top of different TOAST methods.
I don't think that is possible (or at least, not as performant). To
treat type X' as type X and use it as a stored medium instead, you
must have either the whole binary representation of X, or have access
to the internals of type X. I find it difficult to believe that casts
can be done without a full detoast (or otherwise without deep
knowledge about internal structure of the data type such as 'type A is
binary compatible with type X'), and as such I think this feature
'compression dictionaries' is competing with the 'pluggable toaster'
feature, if the one is used on top of the other. That is, the
dictionary is still created like in the proposed patches (though
preferably without the 64-byte NAMELEN limit), but the usage will be
through "TOASTER my_dict_enabled_toaster".
Additionally, I don't think we've ever accepted two different
implementations of the same concept, at least not without first having
good arguments why both competing implementations have obvious
benefits over the other, and both implementations being incompatible.
> Although there is also a high-level idea (according to the
> presentations) to share common data between different TOASTed values,
> similarly to what compression dictionaries do, by looking at the
> current feedback and considering the overall complexity and the amount
> of open questions (e.g. interaction with different TableAMs, etc), I
> seriously doubt that this particular part of "pluggable TOASTer" will
> end-up in the core.
Yes, and that's why I think that this where this dictionary
infrastructure could provide value, as an alternative or extension to
the proposed jsonb toaster in the 'pluggable toaster' thread.
Kind regards,
Matthias van de Meent
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: [PATCH] Compression dictionaries for JSONB
@ 2022-07-04 14:00 Aleksander Alekseev <[email protected]>
parent: Matthias van de Meent <[email protected]>
0 siblings, 1 reply; 88+ messages in thread
From: Aleksander Alekseev @ 2022-07-04 14:00 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Matthias van de Meent <[email protected]>; Zhihong Yu <[email protected]>; Teodor Sigaev <[email protected]>
Hi Matthias,
> > Although there is also a high-level idea (according to the
> > presentations) to share common data between different TOASTed values,
> > similarly to what compression dictionaries do, by looking at the
> > current feedback and considering the overall complexity and the amount
> > of open questions (e.g. interaction with different TableAMs, etc), I
> > seriously doubt that this particular part of "pluggable TOASTer" will
> > end-up in the core.
>
> Yes, and that's why I think that this where this dictionary
> infrastructure could provide value, as an alternative or extension to
> the proposed jsonb toaster in the 'pluggable toaster' thread.
OK, I see your point now. And I think this is a very good point.
Basing "Compression dictionaries" on the API provided by "pluggable
TOASTer" can also be less hacky than what I'm currently doing with
`typmod` argument. I'm going to switch the implementation at some
point, unless anyone will object to the idea.
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: [PATCH] Compression dictionaries for JSONB
@ 2022-07-11 14:44 Aleksander Alekseev <[email protected]>
parent: Aleksander Alekseev <[email protected]>
0 siblings, 1 reply; 88+ messages in thread
From: Aleksander Alekseev @ 2022-07-11 14:44 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Matthias van de Meent <[email protected]>; Zhihong Yu <[email protected]>; Teodor Sigaev <[email protected]>
Hi hackers,
> OK, I see your point now. And I think this is a very good point.
> Basing "Compression dictionaries" on the API provided by "pluggable
> TOASTer" can also be less hacky than what I'm currently doing with
> `typmod` argument. I'm going to switch the implementation at some
> point, unless anyone will object to the idea.
Here is the rebased patch. I reworked the memory management a bit but
other than that there are no new changes.
So far we seem to have a consensus to:
1. Use bytea instead of NameData to store dictionary entries;
2. Assign monotonically ascending IDs to the entries instead of using
Oids, as it is done with pg_class.relnatts. In order to do this we
should either add a corresponding column to pg_type, or add a new
catalog table, e.g. pg_dict_meta. Personally I don't have a strong
opinion on what is better. Thoughts?
Both changes should be straightforward to implement and also are a
good exercise to newcomers.
I invite anyone interested to join this effort as a co-author! (since,
honestly, rewriting the same feature over and over again alone is
quite boring :D).
--
Best regards,
Aleksander Alekseev
Attachments:
[application/octet-stream] v4-0001-Compression-dictionaries-for-JSONB.patch (49.0K, ../../CAJ7c6TMJpvGER4qaK__20BbzGKw5Z=twOKxPgemSV97T4eSmpQ@mail.gmail.com/2-v4-0001-Compression-dictionaries-for-JSONB.patch)
download | inline diff:
From 22f5ab2ea6e9dcfc4994fbc2111030804c90c7eb Mon Sep 17 00:00:00 2001
From: Aleksander Alekseev <[email protected]>
Date: Tue, 12 Apr 2022 15:58:11 +0300
Subject: [PATCH v4] Compression dictionaries for JSONB
Usage example:
CREATE TYPE mydict AS DITIONARY OF jsonb ('aaa', 'bbb', ...);
SELECT ('{"aaa":"bbb"}' :: mydict) -> 'aaa';
The created type works as a drop-in replacement for JSONB. However,
its internal representation differs. The provided dictionary entries ('aaa',
'bbb', ..) are stored in pg_dict catalog table. When `mydict` sees one of
the entries in the document, it replaces it with the corresponding pg_dict
entry Oid. For more details regarding the compression algorithm and choosen
compromises please see the comments in the code.
In pg_type `mydict` has typtype = TYPTYPE_DICT. It works the same way
as TYPTYPE_BASE with only difference: corresponding `<type>_in`
(pg_type.typinput) and `<another-type>_<type>` (pg_cast.castfunc)
procedures receive the dictionary Oid as a `typmod` argument. This way the
procedures can distinguish `mydict1` from `mydict2` and use the proper
compression dictionary.
(NOTE: dear committer, please bump the catalog version!)
Author: Aleksander Alekseev <[email protected]>
Author: TODO FIXME add your name and email here
Reviewed-by: FIXME
Discussion: https://postgr.es/m/CAJ7c6TOtAB0z1UrksvGTStNE-herK-43bj22%3D5xVBg7S4vr5rQ%40mail.gmail.com
Discussion: https://postgr.es/m/CAJ7c6TPx7N-bVw0dZ1ASCDQKZJHhBYkT6w4HV1LzfS%2BUUTUfmA%40mail.gmail.com
---
src/backend/catalog/Makefile | 3 +-
src/backend/catalog/pg_dict.c | 197 +++++++
src/backend/commands/typecmds.c | 160 +++++-
src/backend/executor/functions.c | 1 +
src/backend/parser/gram.y | 8 +
src/backend/parser/parse_coerce.c | 17 +-
src/backend/parser/parse_type.c | 16 +
src/backend/tcop/utility.c | 13 +
src/backend/utils/adt/Makefile | 1 +
src/backend/utils/adt/dictionaries.c | 483 ++++++++++++++++++
src/backend/utils/fmgr/funcapi.c | 1 +
src/bin/pg_dump/pg_dump.c | 2 +
src/include/catalog/pg_dict.h | 78 +++
src/include/catalog/pg_proc.dat | 20 +
src/include/catalog/pg_type.h | 1 +
src/include/commands/typecmds.h | 1 +
src/include/nodes/parsenodes.h | 12 +
src/pl/plpgsql/src/pl_comp.c | 1 +
.../modules/test_oat_hooks/test_oat_hooks.c | 3 +
src/test/regress/expected/dict.out | 47 ++
src/test/regress/expected/oidjoins.out | 1 +
src/test/regress/expected/opr_sanity.out | 14 +-
src/test/regress/parallel_schedule | 2 +-
src/test/regress/sql/dict.sql | 20 +
24 files changed, 1091 insertions(+), 11 deletions(-)
create mode 100644 src/backend/catalog/pg_dict.c
create mode 100644 src/backend/utils/adt/dictionaries.c
create mode 100644 src/include/catalog/pg_dict.h
create mode 100644 src/test/regress/expected/dict.out
create mode 100644 src/test/regress/sql/dict.sql
diff --git a/src/backend/catalog/Makefile b/src/backend/catalog/Makefile
index 89a0221ec9..3b4330148e 100644
--- a/src/backend/catalog/Makefile
+++ b/src/backend/catalog/Makefile
@@ -33,6 +33,7 @@ OBJS = \
pg_conversion.o \
pg_db_role_setting.o \
pg_depend.o \
+ pg_dict.o \
pg_enum.o \
pg_inherits.o \
pg_largeobject.o \
@@ -61,7 +62,7 @@ CATALOG_HEADERS := \
pg_language.h pg_largeobject_metadata.h pg_largeobject.h pg_aggregate.h \
pg_statistic.h pg_statistic_ext.h pg_statistic_ext_data.h \
pg_rewrite.h pg_trigger.h pg_event_trigger.h pg_description.h \
- pg_cast.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \
+ pg_cast.h pg_dict.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \
pg_database.h pg_db_role_setting.h pg_tablespace.h \
pg_authid.h pg_auth_members.h pg_shdepend.h pg_shdescription.h \
pg_ts_config.h pg_ts_config_map.h pg_ts_dict.h \
diff --git a/src/backend/catalog/pg_dict.c b/src/backend/catalog/pg_dict.c
new file mode 100644
index 0000000000..456f3b5177
--- /dev/null
+++ b/src/backend/catalog/pg_dict.c
@@ -0,0 +1,197 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_dict.c
+ * routines to support manipulation of the pg_dict relation
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/catalog/pg_dict.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "access/genam.h"
+#include "access/skey.h"
+#include "access/table.h"
+#include "catalog/catalog.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_dict.h"
+#include "catalog/pg_type.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/memutils.h"
+#include "utils/rel.h"
+
+/*
+ * Creates an entry in pg_dict for each of the supplied values.
+ * vals is a list of String values.
+ */
+void
+DictEntriesCreate(Oid dictTypeOid, List *vals)
+{
+ Relation pg_dict;
+ NameData dictentry;
+ Datum values[Natts_pg_dict];
+ bool nulls[Natts_pg_dict];
+ ListCell *lc;
+ HeapTuple tup;
+
+ if (vals == NIL)
+ {
+ /* The list is empty; do nothing. */
+ return;
+ }
+
+ memset(nulls, false, sizeof(nulls));
+
+ /*
+ * We don't check the list of values for duplicates here. If there are
+ * any, the user will get an unique-index violation.
+ */
+
+ pg_dict = table_open(DictRelationId, RowExclusiveLock);
+ foreach(lc, vals)
+ {
+ Oid oid = GetNewOidWithIndex(pg_dict, DictOidIndexId,
+ Anum_pg_dict_oid);
+ char *entry = strVal(lfirst(lc));
+
+ /*
+ * Entries are stored in a name field, for easier syscache lookup, so
+ * check the length to make sure it's within range.
+ */
+ if (strlen(entry) > (NAMEDATALEN - 1))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_NAME),
+ errmsg("invalid dict entry \"%s\"", entry),
+ errdetail("Entries must be %d bytes or less.",
+ NAMEDATALEN - 1)));
+
+ /* AALEKSEEV TODO use bytea instead */
+ namestrcpy(&dictentry, entry);
+
+ values[Anum_pg_dict_oid - 1] = ObjectIdGetDatum(oid);
+ values[Anum_pg_dict_dicttypid - 1] = ObjectIdGetDatum(dictTypeOid);
+ values[Anum_pg_dict_dictentry - 1] = NameGetDatum(&dictentry);
+
+ tup = heap_form_tuple(RelationGetDescr(pg_dict), values, nulls);
+
+ CatalogTupleInsert(pg_dict, tup);
+ heap_freetuple(tup);
+ }
+
+ /* clean up */
+ table_close(pg_dict, RowExclusiveLock);
+}
+
+/*
+ * Returns all the entries for the dictinary with given Oid. Entries are sorted
+ * by dictentry. Note that shorter entries are considered smaller, i.e. 'abc'
+ * goes before 'abcdef'. The memory is allocated in a child memory context of
+ * the caller's memory context. It can be freed with DictEntriesFree().
+ *
+ * If there are no entries a valid but empty dictionary is returned.
+ */
+Dictionary
+DictEntriesRead(Oid dictTypeOid)
+{
+ Relation pg_dict;
+ ScanKeyData key[1];
+ SysScanDesc scan;
+ HeapTuple tup;
+ uint32 entries_allocated;
+ Dictionary dict;
+ MemoryContext myctx, oldctx;
+
+ myctx = AllocSetContextCreate(CurrentMemoryContext, "DictEntriesCtx", ALLOCSET_DEFAULT_SIZES);
+ oldctx = MemoryContextSwitchTo(myctx);
+
+ dict = (Dictionary)palloc(sizeof(DictionaryData));
+
+ entries_allocated = 8;
+ dict->nentries = 0;
+ dict->entries = (DictEntry*)palloc(entries_allocated*sizeof(DictEntry));
+
+ pg_dict = table_open(DictRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_dict_dicttypid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(dictTypeOid));
+
+ scan = systable_beginscan(pg_dict, DictTypIdEntryIndexId, true,
+ NULL, 1, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ if(dict->nentries == entries_allocated)
+ {
+ entries_allocated = entries_allocated * 2;
+ dict->entries = (DictEntry*)repalloc(dict->entries, entries_allocated*sizeof(DictEntry));
+ }
+
+ dict->entries[dict->nentries].oid = ((Form_pg_dict) GETSTRUCT(tup))->oid;
+ /*
+ * This is arguably not the fastest way to determine the length of
+ * the entry. Alternatively, we could store a precalculated value in
+ * the catalog. However, usually it's a good idea to keep things simple
+ * until somebody discovers a bottleneck in this exact place and
+ * proposes a concrete fix.
+ */
+ dict->entries[dict->nentries].length = (uint32)strlen(((Form_pg_dict) GETSTRUCT(tup))->dictentry.data);
+ dict->entries[dict->nentries].data = palloc(dict->entries[dict->nentries].length);
+ memcpy(dict->entries[dict->nentries].data, ((Form_pg_dict) GETSTRUCT(tup))->dictentry.data, dict->entries[dict->nentries].length);
+ dict->nentries++;
+ }
+
+ systable_endscan(scan);
+ table_close(pg_dict, RowExclusiveLock);
+
+ MemoryContextSwitchTo(oldctx);
+
+ return dict;
+}
+
+/*
+ * Frees the memory allocated for the dictionary.
+ */
+void
+DictEntriesFree(Dictionary dict)
+{
+ MemoryContextDelete(GetMemoryChunkContext(dict));
+}
+
+/*
+ * Deletes all the entries for the dictinary with given Oid.
+ */
+void
+DictEntriesDelete(Oid dictTypeOid)
+{
+ Relation pg_dict;
+ ScanKeyData key[1];
+ SysScanDesc scan;
+ HeapTuple tup;
+
+ pg_dict = table_open(DictRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_dict_dicttypid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(dictTypeOid));
+
+ scan = systable_beginscan(pg_dict, DictTypIdEntryIndexId, true,
+ NULL, 1, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ CatalogTupleDelete(pg_dict, &tup->t_self);
+ }
+
+ systable_endscan(scan);
+ table_close(pg_dict, RowExclusiveLock);
+}
diff --git a/src/backend/commands/typecmds.c b/src/backend/commands/typecmds.c
index 4f5e7c7ccb..7e8fc57e6f 100644
--- a/src/backend/commands/typecmds.c
+++ b/src/backend/commands/typecmds.c
@@ -46,6 +46,7 @@
#include "catalog/pg_collation.h"
#include "catalog/pg_constraint.h"
#include "catalog/pg_depend.h"
+#include "catalog/pg_dict.h"
#include "catalog/pg_enum.h"
#include "catalog/pg_language.h"
#include "catalog/pg_namespace.h"
@@ -679,6 +680,14 @@ RemoveTypeById(Oid typeOid)
if (((Form_pg_type) GETSTRUCT(tup))->typtype == TYPTYPE_RANGE)
RangeDelete(typeOid);
+ /*
+ * If it is a dictionary type, delete the pg_dict entries too; we don't
+ * bother with making a dependency entry for that, so it has to be done
+ * "by hand" here.
+ */
+ if (((Form_pg_type) GETSTRUCT(tup))->typtype == TYPTYPE_DICT)
+ DictEntriesDelete(typeOid);
+
ReleaseSysCache(tup);
table_close(relation, RowExclusiveLock);
@@ -763,8 +772,8 @@ DefineDomain(CreateDomainStmt *stmt)
/*
* Base type must be a plain base type, a composite type, another domain,
- * an enum or a range type. Domains over pseudotypes would create a
- * security hole. (It would be shorter to code this to just check for
+ * a dict, an enum or a range type. Domains over pseudotypes would create
+ * a security hole. (It would be shorter to code this to just check for
* pseudotypes; but it seems safer to call out the specific typtypes that
* are supported, rather than assume that all future typtypes would be
* automatically supported.)
@@ -772,6 +781,7 @@ DefineDomain(CreateDomainStmt *stmt)
typtype = baseType->typtype;
if (typtype != TYPTYPE_BASE &&
typtype != TYPTYPE_COMPOSITE &&
+ typtype != TYPTYPE_DICT &&
typtype != TYPTYPE_DOMAIN &&
typtype != TYPTYPE_ENUM &&
typtype != TYPTYPE_RANGE &&
@@ -1129,6 +1139,152 @@ DefineDomain(CreateDomainStmt *stmt)
}
+/*
+ * DefineDictionary
+ * Registers a new dictionary.
+ */
+ObjectAddress
+DefineDictionary(CreateDictionaryStmt *stmt)
+{
+ char *dictName;
+ char *dictBaseTypeName;
+ char *dictArrayName;
+ Oid dictNamespace;
+ AclResult aclresult;
+ Oid old_type_oid;
+ Oid dictArrayOid;
+ ObjectAddress dictTypeAddr;
+
+ Assert(list_length(stmt->baseTypeName) == 1);
+ dictBaseTypeName = strVal(linitial(stmt->baseTypeName));
+ if(pg_strcasecmp(dictBaseTypeName, "jsonb") != 0)
+ ereport(ERROR,
+ (
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("Dictionary types currently can be used only with JSONB")
+ )
+ );
+
+ /* Convert list of names to a name and namespace */
+ dictNamespace = QualifiedNameGetCreationNamespace(stmt->typeName,
+ &dictName);
+
+ /* Check we have creation rights in target namespace */
+ aclresult = pg_namespace_aclcheck(dictNamespace, GetUserId(), ACL_CREATE);
+ if (aclresult != ACLCHECK_OK)
+ aclcheck_error(aclresult, OBJECT_SCHEMA,
+ get_namespace_name(dictNamespace));
+
+ /*
+ * Check for collision with an existing type name. If there is one and
+ * it's an autogenerated array, we can rename it out of the way.
+ */
+ old_type_oid = GetSysCacheOid2(TYPENAMENSP, Anum_pg_type_oid,
+ CStringGetDatum(dictName),
+ ObjectIdGetDatum(dictNamespace));
+ if (OidIsValid(old_type_oid))
+ {
+ if (!moveArrayTypeName(old_type_oid, dictName, dictNamespace))
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("type \"%s\" already exists", dictName)));
+ }
+
+ /* Allocate OID for array type */
+ dictArrayOid = AssignTypeArrayOid();
+
+ /* Create the pg_type entry */
+ dictTypeAddr =
+ TypeCreate(InvalidOid, /* no predetermined type OID */
+ dictName, /* type name */
+ dictNamespace, /* namespace */
+ InvalidOid, /* relation oid (n/a here) */
+ 0, /* relation kind (ditto) */
+ GetUserId(), /* owner's ID */
+ -1, /* internal size: varlena, as for JSONB */
+ TYPTYPE_DICT, /* type-type: dictionary */
+ TYPCATEGORY_USER, /* type-category: user, as for JSONB */
+ false, /* dict types are never preferred */
+ DEFAULT_TYPDELIM, /* array element delimiter */
+ F_DICTIONARY_IN, /* input procedure */
+ F_DICTIONARY_OUT, /* output procedure */
+ InvalidOid, /* receive procedure: none */
+ InvalidOid, /* send procedure: none */
+ InvalidOid, /* typmodin procedure - none */
+ InvalidOid, /* typmodout procedure - none */
+ InvalidOid, /* analyze procedure - default */
+ InvalidOid, /* subscript procedure - none */
+ InvalidOid, /* element type ID */
+ false, /* this is not an array type */
+ dictArrayOid, /* array type we are about to create */
+ InvalidOid, /* base type ID (only for domains) */
+ NULL, /* never a default type value */
+ NULL, /* binary default isn't sent either */
+ false, /* passed by value: same as for JSONB */
+ TYPALIGN_INT, /* int alignment */
+ TYPSTORAGE_EXTENDED, /* TOAST strategy: fully toastable, as
+ * JSONB */
+ -1, /* typMod (Domains only) */
+ 0, /* Array dimensions of typbasetype */
+ false, /* Type NOT NULL */
+ InvalidOid); /* type's collation */
+
+ /*
+ * Create the array type that goes with it.
+ */
+ dictArrayName = makeArrayTypeName(dictName, dictNamespace);
+
+ TypeCreate(dictArrayOid, /* force assignment of this type OID */
+ dictArrayName, /* type name */
+ dictNamespace, /* namespace */
+ InvalidOid, /* relation oid (n/a here) */
+ 0, /* relation kind (ditto) */
+ GetUserId(), /* owner's ID */
+ -1, /* internal size (always varlena) */
+ TYPTYPE_BASE, /* type-type (base type) */
+ TYPCATEGORY_ARRAY, /* type-category (array) */
+ false, /* array types are never preferred */
+ DEFAULT_TYPDELIM, /* array element delimiter */
+ F_ARRAY_IN, /* input procedure */
+ F_ARRAY_OUT, /* output procedure */
+ F_ARRAY_RECV, /* receive procedure */
+ F_ARRAY_SEND, /* send procedure */
+ InvalidOid, /* typmodin procedure - none */
+ InvalidOid, /* typmodout procedure - none */
+ F_ARRAY_TYPANALYZE, /* analyze procedure */
+ F_ARRAY_SUBSCRIPT_HANDLER, /* array subscript procedure */
+ dictTypeAddr.objectId, /* element type ID */
+ true, /* yes this is an array type */
+ InvalidOid, /* no further array type */
+ InvalidOid, /* base type ID */
+ NULL, /* never a default type value */
+ NULL, /* binary default isn't sent either */
+ false, /* passed by value: same as for JSONB */
+ TYPALIGN_INT, /* enums have int align, so do their arrays */
+ TYPSTORAGE_EXTENDED, /* ARRAY is always toastable */
+ -1, /* typMod (Domains only) */
+ 0, /* Array dimensions of typbasetype */
+ false, /* Type NOT NULL */
+ InvalidOid); /* type's collation */
+
+ pfree(dictArrayName);
+
+ /* Enter the dict's entries into pg_dict */
+ DictEntriesCreate(dictTypeAddr.objectId, stmt->vals);
+
+ /* Create casts to and from JSONB */
+ CastCreate(JSONBOID, dictTypeAddr.objectId, F_JSONB_DICTIONARY, 'a', 'f', DEPENDENCY_INTERNAL);
+ CastCreate(dictTypeAddr.objectId, JSONBOID, F_DICTIONARY_JSONB, 'i', 'f', DEPENDENCY_INTERNAL);
+
+ /*
+ * Create explicit cast to bytea. This is convenient for debugging purposes.
+ * Casting bytea to a dictionary type is dangerous and thus not supported.
+ */
+ CastCreate(dictTypeAddr.objectId, BYTEAOID, F_DICTIONARY_BYTEA, 'e', 'f', DEPENDENCY_INTERNAL);
+
+ return dictTypeAddr;
+}
+
/*
* DefineEnum
* Registers a new enum.
diff --git a/src/backend/executor/functions.c b/src/backend/executor/functions.c
index f9460ae506..7ad3de3a00 100644
--- a/src/backend/executor/functions.c
+++ b/src/backend/executor/functions.c
@@ -1708,6 +1708,7 @@ check_sql_fn_retval(List *queryTreeLists,
if (fn_typtype == TYPTYPE_BASE ||
fn_typtype == TYPTYPE_DOMAIN ||
+ fn_typtype == TYPTYPE_DICT ||
fn_typtype == TYPTYPE_ENUM ||
fn_typtype == TYPTYPE_RANGE ||
fn_typtype == TYPTYPE_MULTIRANGE)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 0523013f53..51ab3e89db 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -6237,6 +6237,14 @@ DefineStmt:
n->coldeflist = $6;
$$ = (Node *) n;
}
+ | CREATE TYPE_P any_name AS DICTIONARY OF any_name '(' opt_enum_val_list ')'
+ {
+ CreateDictionaryStmt *n = makeNode(CreateDictionaryStmt);
+ n->typeName = $3;
+ n->baseTypeName = $7;
+ n->vals = $9;
+ $$ = (Node *)n;
+ }
| CREATE TYPE_P any_name AS ENUM_P '(' opt_enum_val_list ')'
{
CreateEnumStmt *n = makeNode(CreateEnumStmt);
diff --git a/src/backend/parser/parse_coerce.c b/src/backend/parser/parse_coerce.c
index c4e958e4aa..e489dd9421 100644
--- a/src/backend/parser/parse_coerce.c
+++ b/src/backend/parser/parse_coerce.c
@@ -279,7 +279,22 @@ coerce_type(ParseState *pstate, Node *node,
if (baseTypeId == INTERVALOID)
inputTypeMod = baseTypeMod;
else
- inputTypeMod = -1;
+ {
+ if((int32)targetTypeId == targetTypeMod)
+ {
+ /*
+ * For dictionaries internally we set typmod to a dictionary
+ * OID. This allows dictionary_in() and jsonb_dictionary() to
+ * distinguish one dictionary from another.
+ *
+ * It also allows us to recognize that the target type is
+ * a dictionary here and pass a correct typmod.
+ */
+ inputTypeMod = targetTypeMod;
+ }
+ else
+ inputTypeMod = -1;
+ }
baseType = typeidType(baseTypeId);
diff --git a/src/backend/parser/parse_type.c b/src/backend/parser/parse_type.c
index f7ad689459..2188002535 100644
--- a/src/backend/parser/parse_type.c
+++ b/src/backend/parser/parse_type.c
@@ -259,6 +259,10 @@ LookupTypeNameOid(ParseState *pstate, const TypeName *typeName, bool missing_ok)
* This is equivalent to LookupTypeName, except that this will report
* a suitable error message if the type cannot be found or is not defined.
* Callers of this can therefore assume the result is a fully valid type.
+ *
+ * For a dictionary type it's OID is returned as a typmod. This is used by
+ * dictionary_in() and jsonb_dictionary() to distinguish one dictionary from
+ * another.
*/
Type
typenameType(ParseState *pstate, const TypeName *typeName, int32 *typmod_p)
@@ -305,6 +309,10 @@ typenameTypeId(ParseState *pstate, const TypeName *typeName)
*
* This is equivalent to typenameType, but we only hand back the type OID
* and typmod, not the syscache entry.
+ *
+ * For a dictionary type it's OID is returned as a typmod. This is used by
+ * dictionary_in() and jsonb_dictionary() to distinguish one dictionary from
+ * another.
*/
void
typenameTypeIdAndMod(ParseState *pstate, const TypeName *typeName,
@@ -327,6 +335,10 @@ typenameTypeIdAndMod(ParseState *pstate, const TypeName *typeName,
* looked up, and is passed as "typ".
*
* pstate is only used for error location info, and may be NULL.
+ *
+ * For a dictionary type it's OID is returned as a typmod. This is used by
+ * dictionary_in() and jsonb_dictionary() to distinguish one dictionary from
+ * another.
*/
static int32
typenameTypeMod(ParseState *pstate, const TypeName *typeName, Type typ)
@@ -339,6 +351,10 @@ typenameTypeMod(ParseState *pstate, const TypeName *typeName, Type typ)
ArrayType *arrtypmod;
ParseCallbackState pcbstate;
+ /* For a dictionary return it's OID. */
+ if (((Form_pg_type) GETSTRUCT(typ))->typtype == TYPTYPE_DICT)
+ return (int32)((Form_pg_type) GETSTRUCT(typ))->oid;
+
/* Return prespecified typmod if no typmod expressions */
if (typeName->typmods == NIL)
return typeName->typemod;
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index 6b0a865262..40cb80eb4a 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -174,6 +174,7 @@ ClassifyUtilityCommandAsReadOnly(Node *parsetree)
case T_CreateCastStmt:
case T_CreateConversionStmt:
case T_CreateDomainStmt:
+ case T_CreateDictionaryStmt:
case T_CreateEnumStmt:
case T_CreateEventTrigStmt:
case T_CreateExtensionStmt:
@@ -1621,6 +1622,10 @@ ProcessUtilitySlow(ParseState *pstate,
}
break;
+ case T_CreateDictionaryStmt: /* CREATE TYPE AS DICTIONARY OF */
+ address = DefineDictionary((CreateDictionaryStmt *) parsetree);
+ break;
+
case T_CreateEnumStmt: /* CREATE TYPE AS ENUM */
address = DefineEnum((CreateEnumStmt *) parsetree);
break;
@@ -2772,6 +2777,10 @@ CreateCommandTag(Node *parsetree)
tag = CMDTAG_CREATE_TYPE;
break;
+ case T_CreateDictionaryStmt:
+ tag = CMDTAG_CREATE_TYPE;
+ break;
+
case T_CreateEnumStmt:
tag = CMDTAG_CREATE_TYPE;
break;
@@ -3419,6 +3428,10 @@ GetCommandLogLevel(Node *parsetree)
lev = LOGSTMT_DDL;
break;
+ case T_CreateDictionaryStmt:
+ lev = LOGSTMT_DDL;
+ break;
+
case T_CreateEnumStmt:
lev = LOGSTMT_DDL;
break;
diff --git a/src/backend/utils/adt/Makefile b/src/backend/utils/adt/Makefile
index 7c722ea2ce..a7ec3d89f3 100644
--- a/src/backend/utils/adt/Makefile
+++ b/src/backend/utils/adt/Makefile
@@ -30,6 +30,7 @@ OBJS = \
datetime.o \
datum.o \
dbsize.o \
+ dictionaries.o \
domains.o \
encode.o \
enum.o \
diff --git a/src/backend/utils/adt/dictionaries.c b/src/backend/utils/adt/dictionaries.c
new file mode 100644
index 0000000000..28de59b0b9
--- /dev/null
+++ b/src/backend/utils/adt/dictionaries.c
@@ -0,0 +1,483 @@
+/*-------------------------------------------------------------------------
+ *
+ * dictionaries.c
+ * Conversion functions for dictionary types.
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/adt/dictionaries.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "catalog/pg_dict.h"
+#include "utils/fmgrprotos.h"
+#include "utils/jsonb.h"
+
+/*
+ * When compressing a data we treat it as a BLOB, in other words we don't
+ * assume anything regarding its internal representation. This is not
+ * necessarily the best and/or the only possible approach. However, it is
+ * universal and can be reused for JSONB, TEXT, XML and other types. Alternative
+ * compression methods can be introduced in the future, so that the user will
+ * be able to choose the best one for the task.
+ *
+ * The compressed data is stored in the following format:
+ *
+ * (struct varlena)
+ * dictionary_id [uint32]
+ * decompressed_size [uint32]
+ * algorithm_version [uint8]
+ *
+ * (repeated) {
+ * number of bytes to copy as-is [uint8]
+ * ... bytes to copy as-is ...
+ * dictionary entry id, or 0 to skip [uint32]
+ * }
+ *
+ * Compressed data is a variable-length type and thus has the 'struct varlena'
+ * header. The code below doesn't consider it being a part of the payload
+ * (e.g. see the DICT_COMP_HEADER_SIZE definition).
+ *
+ * Storing dictionary_id may seem redundant, but without it dictionary_out()
+ * and dictionary_jsonb() have no way to distinguish one dictionary type from
+ * another. dictionary_in() and jsonb_dictionary() get the dictionary id through
+ * the 'typemod' argument.
+ *
+ * Currently algorithm_version is always 0. In the future it will allow us to
+ * introduce new features (e.g. usage of varints) and/or lazily migrate the
+ * data to other compression methods.
+ *
+ * Compression and decompression are implemented based on a binary search over
+ * DictEntry[] array (see pg_dict.h). For compression the array is sorted by
+ * dictentries and for decompression - by oid's.
+ *
+ */
+
+/* Size of the header in the compressed data */
+#define DICT_COMP_HEADER_SIZE (sizeof(uint32)*2 + sizeof(uint8))
+
+/* Extracts dictionary_id from the compressed data */
+#define DICT_COMP_DICTIONARY_ID(hdrp) \
+ (*(uint32*)hdrp)
+
+/* Extracts decompressed_size from the compressed data */
+#define DICT_COMP_DECOMPRESSED_SIZE(hdrp) \
+ (*(uint32*)((uint8*)hdrp + sizeof(uint32)))
+
+/* Extracts algorithm_version from the compressed data */
+#define DICT_COMP_ALGORITHM_VERSION(hdrp) \
+ (*(uint8*)((uint8*)hdrp + sizeof(uint32)*2))
+
+/* Current algorithm_version */
+#define DICT_COMP_CURRENT_ALGORITHM_VERSION 0
+
+/*
+ * bsearch_arg() callback for finding DictEntry by oid.
+ */
+static int
+find_by_oid_cb(const void *key, const void *current, void *arg)
+{
+ /* Note that oids are unsigned, so we should be careful here */
+ if (*(Oid *) key < ((DictEntry *) current)->oid)
+ return -1;
+ else if (*(Oid *) key == ((DictEntry *) current)->oid)
+ return 0; /* found! */
+ else
+ return 1;
+}
+
+/*
+ * qsort() callback for sorting DictEntry[] by oids.
+ */
+static int
+sort_by_oids_cb(const void *left, const void *right)
+{
+ /* Note that oids are unsigned, so we should be careful here */
+ if (((DictEntry *) left)->oid < ((DictEntry *) right)->oid)
+ return -1;
+
+ /* Oids are unique so this callback will never return 0 */
+ return 1;
+}
+
+/*
+ * Finds a DictEntry which dictentry field matches *data and returns its Oid.
+ * If there are several matching entries, the largest is returned. The length
+ * of the found entry is written to *found_length on success. On failure
+ * InvalidOid is returned and found_length is zeroed.
+ *
+ * The implementation is similar to bsearch_arg(). The procedure can't be used
+ * directly because we are looking not by the exact match. We could generalize
+ * this case but the signature of the function becomes so complicated that it
+ * doesn't seem to worth the effort.
+ */
+static Oid
+compress_find_oid(Dictionary dict, const uint8 *data, Size data_size, Size *found_length)
+{
+ int res;
+ int32 left = 0;
+ int32 right = dict->nentries - 1;
+ Size best_length = 0;
+ Oid best_match = InvalidOid;
+
+ while (left <= right)
+ {
+ int32 current = (left + right) / 2;
+ Size nbytes = (Size)dict->entries[current].length;
+
+ if (nbytes > data_size)
+ {
+ /* current can be less or greater depending on the prefix */
+ res = memcmp(dict->entries[current].data, data, data_size);
+
+ /* if prefixes match, current is greater */
+ if (res == 0)
+ res = 1;
+ }
+ else
+ res = memcmp(dict->entries[current].data, data, nbytes);
+
+ if (res == 0) /* match found */
+ {
+ best_length = nbytes;
+ best_match = dict->entries[current].oid;
+
+ if (nbytes == data_size)
+ break;
+
+ /* maybe there is a larger match */
+ left = current + 1;
+ }
+ else if (res < 0) /* current is less */
+ left = current + 1;
+ else /* current is greater */
+ right = current - 1;
+ }
+
+ *found_length = best_length;
+ return best_match;
+}
+
+/*
+ * Finds a DictEntry by Oid using a binary search. The dictionary should be
+ * sorted by oids before the call. Returns NULL if nothing was found.
+ */
+static DictEntry *
+decompress_find_dictentry(Dictionary dict, Oid oid)
+{
+ return (DictEntry *) bsearch_arg(&oid, dict->entries, dict->nentries, sizeof(DictEntry), find_by_oid_cb, NULL);
+}
+
+/*
+ * Estimates the worst-case compressed size of the data of given size.
+ * Worst-case scenario happens when the dictionary consists of single-character
+ * entries. In this case every byte will be encoded as 6 bytes:
+ * 0x00, (0 bytes to copy as-is), 4 bytes of the entry Oid
+ *
+ * This procedure doesn't account for the header size.
+ *
+ * AALEKSEEV FIXME don't use dictionary entries shorter than 5 bytes
+ */
+static Size
+worst_case_compressed_size(Size insize)
+{
+ return insize * 6;
+}
+
+/*
+ * Compresses the data using the provided dictionary. The dictionary should
+ * be sorted by dictentries before the call. Output buffer should be at least
+ * worst_case_compressed_size(src_size) bytes in size.
+ */
+static void
+compress(Dictionary dict,
+ const void *src_data_, Size src_size,
+ void *encoded_data_, Size *pencoded_size)
+{
+ Size nbytes;
+ Size inoffset;
+ Size outskipoffset = 0;
+ Size outoffset = 1;
+ uint8 skipbytes = 0;
+ const uint8 *src_data = src_data_;
+ uint8 *encoded_data = ((uint8 *) encoded_data_);
+
+ for (inoffset = 0; inoffset < src_size;)
+ {
+ Oid code = compress_find_oid(dict, &(src_data[inoffset]),
+ src_size - inoffset, &nbytes);
+
+ if (code == InvalidOid)
+ {
+ skipbytes++;
+ encoded_data[outoffset] = src_data[inoffset];
+ outoffset++;
+ inoffset++;
+
+ if (skipbytes == 255)
+ {
+ encoded_data[outskipoffset] = skipbytes;
+ encoded_data[outoffset++] = 0; /* InvalidOid */
+ encoded_data[outoffset++] = 0;
+ encoded_data[outoffset++] = 0;
+ encoded_data[outoffset++] = 0;
+ outskipoffset = outoffset++;
+ skipbytes = 0;
+ }
+ }
+ else
+ {
+ encoded_data[outskipoffset] = skipbytes;
+ encoded_data[outoffset++] = (code >> 24) & 0xFF;
+ encoded_data[outoffset++] = (code >> 16) & 0xFF;
+ encoded_data[outoffset++] = (code >> 8) & 0xFF;
+ encoded_data[outoffset++] = code & 0xFF;
+ outskipoffset = outoffset++;
+ skipbytes = 0;
+ inoffset += nbytes;
+ }
+ }
+
+ /* Double check that we didn't write out of buffer */
+ Assert(outoffset < worst_case_compressed_size(src_size));
+
+ encoded_data[outskipoffset] = skipbytes;
+ *pencoded_size = outoffset;
+}
+
+/*
+ * Report an internal error in decompress() procedure below.
+ * Under normal circumstances this should never happen.
+ */
+static void
+decompress_error()
+{
+ ereport(ERROR,
+ (
+ errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("Unable to decompress a dictionary type"),
+ errdetail("The compressed data seems to be corrupted"),
+ errhint("Please report the steps to reproduce the issue to pgsql-bugs@")
+ ));
+}
+
+/*
+ * Decompresses the data using the provided dictionary. The dictionary should
+ * be sorted by oids before the call.
+ */
+static void
+decompress(Dictionary dict,
+ const void *encoded_data_, Size encoded_size,
+ void *decoded_data_, Size decoded_size)
+{
+ Size inoffset = 0;
+ Size outoffset = 0;
+ Oid code;
+ uint8 skipbytes;
+ const uint8 *encoded_data = ((uint8 *) encoded_data_);
+ uint8 *decoded_data = decoded_data_;
+
+ for (inoffset = 0; inoffset < encoded_size;)
+ {
+ skipbytes = encoded_data[inoffset++];
+
+ if (skipbytes > decoded_size - outoffset)
+ decompress_error();
+
+ if (skipbytes > encoded_size - inoffset)
+ decompress_error();
+
+ memcpy(
+ &(decoded_data[outoffset]),
+ &(encoded_data[inoffset]),
+ skipbytes
+ );
+
+ outoffset += skipbytes;
+ inoffset += skipbytes;
+
+ if ((encoded_size == inoffset) && (decoded_size == outoffset))
+ break; /* end of input - its OK */
+
+ if (encoded_size - inoffset < 4)
+ decompress_error();
+
+ code = (Oid) encoded_data[inoffset++];
+ code = (code << 8) | (Oid) encoded_data[inoffset++];
+ code = (code << 8) | (Oid) encoded_data[inoffset++];
+ code = (code << 8) | (Oid) encoded_data[inoffset++];
+
+ if (code != InvalidOid)
+ {
+ Size entrylen;
+ DictEntry *entry = decompress_find_dictentry(dict, code);
+
+ if (entry == NULL)
+ decompress_error();
+
+ Assert(entry->oid == code);
+ entrylen = (Size)entry->length;
+
+ if (entrylen > decoded_size - outoffset)
+ decompress_error();
+
+ memcpy(
+ &(decoded_data[outoffset]),
+ entry->data,
+ entrylen
+ );
+
+ outoffset += entrylen;
+ }
+ }
+
+ Assert(decoded_size == outoffset);
+}
+
+/*
+ * Converts a cstring to a dictionary.
+ */
+Datum
+dictionary_in(PG_FUNCTION_ARGS)
+{
+ const char *instr = PG_GETARG_CSTRING(0);
+#ifdef NOT_USED
+ Oid typelem = PG_GETARG_OID(1);
+#endif
+ int32 typmod = PG_GETARG_INT32(2);
+ Jsonb *jsonb;
+ bytea *dict;
+
+ Assert(typmod != -1);
+
+ jsonb = DatumGetJsonbP(DirectFunctionCall1(jsonb_in, CStringGetDatum(instr)));
+ dict = DatumGetByteaP(DirectFunctionCall2(jsonb_dictionary, JsonbPGetDatum(jsonb), Int32GetDatum(typmod)));
+
+ PG_RETURN_BYTEA_P(dict);
+}
+
+/*
+ * Converts a dictionary to a cstring.
+ */
+Datum
+dictionary_out(PG_FUNCTION_ARGS)
+{
+ bytea *dict = PG_GETARG_BYTEA_P(0);
+ Jsonb *jsonb = DatumGetJsonbP(DirectFunctionCall1(dictionary_jsonb, PointerGetDatum(dict)));
+ const char *outstr = DatumGetCString(DirectFunctionCall1(jsonb_out, JsonbPGetDatum(jsonb)));
+
+ PG_RETURN_CSTRING(outstr);
+}
+
+/*
+ * Coverts JSONB to a dictionary type.
+ *
+ * AALEKSEEV FIXME: if the compressed document ends up being larger than
+ * the original one write a corresponding flag and copy all the data as-is.
+ */
+Datum
+jsonb_dictionary(PG_FUNCTION_ARGS)
+{
+ Dictionary dict;
+ Jsonb *jsonb = PG_GETARG_JSONB_P(0);
+ int32 typmod = PG_GETARG_INT32(1);
+ uint8 *jsonb_data = (uint8 *) VARDATA(jsonb);
+ Size jsonb_data_size = VARSIZE(jsonb) - VARHDRSZ;
+ uint8 *encoded_buff,
+ *encoded_header,
+ *encoded_data;
+ Size encoded_size,
+ encoded_buff_size;
+ MemoryContext myctx, oldctx;
+
+ Assert(typmod != -1);
+
+ dict = DictEntriesRead(typmod);
+ qsort((void *) dict->entries, dict->nentries, sizeof(DictEntry), sort_by_oids_cb);
+
+ encoded_buff_size = VARHDRSZ + DICT_COMP_HEADER_SIZE + worst_case_compressed_size(jsonb_data_size);
+ encoded_buff = palloc(encoded_buff_size);
+ encoded_header = (uint8 *) VARDATA(encoded_buff);
+ encoded_data = encoded_header + DICT_COMP_HEADER_SIZE;
+
+ DICT_COMP_DICTIONARY_ID(encoded_header) = typmod;
+ DICT_COMP_DECOMPRESSED_SIZE(encoded_header) = jsonb_data_size;
+ DICT_COMP_ALGORITHM_VERSION(encoded_header) = DICT_COMP_CURRENT_ALGORITHM_VERSION;
+
+ encoded_size = encoded_buff_size - VARHDRSZ - DICT_COMP_HEADER_SIZE;
+
+ compress(dict, jsonb_data, jsonb_data_size,
+ encoded_data, &encoded_size);
+
+ encoded_size += VARHDRSZ + DICT_COMP_HEADER_SIZE;
+
+ encoded_buff = repalloc(encoded_buff, encoded_size);
+ SET_VARSIZE(encoded_buff, encoded_size);
+
+ DictEntriesFree(dict);
+
+ PG_RETURN_BYTEA_P(encoded_buff);
+}
+
+/*
+ * Converts a dictionary type to JSONB.
+ */
+Datum
+dictionary_jsonb(PG_FUNCTION_ARGS)
+{
+ bytea *encoded_buff = PG_GETARG_BYTEA_P(0);
+ uint8 *encoded_header = (uint8 *) VARDATA(encoded_buff);
+ uint8 *encoded_data = encoded_header + DICT_COMP_HEADER_SIZE;
+ Size encoded_size = VARSIZE(encoded_buff) - VARHDRSZ - DICT_COMP_HEADER_SIZE;
+ int alg_version = DICT_COMP_ALGORITHM_VERSION(encoded_header);
+ Oid dictOid; /* cannot read until algorithm version is
+ * checked */
+ uint32 decoded_size; /* cannot read until algorithm version is
+ * checked */
+ Jsonb *jsonb;
+ uint8 *jsonb_data;
+ Dictionary dict;
+
+ if (alg_version > DICT_COMP_CURRENT_ALGORITHM_VERSION)
+ ereport(ERROR,
+ (
+ errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("Unsupported compression algorithm version"),
+ errdetail("Saved algorithm version is %d, current version is %d",
+ alg_version, DICT_COMP_CURRENT_ALGORITHM_VERSION),
+ errhint("The data is either corrupted or imported from "
+ "the future version of PostgreSQL")
+ ));
+
+ dictOid = DICT_COMP_DICTIONARY_ID(encoded_header);
+ decoded_size = DICT_COMP_DECOMPRESSED_SIZE(encoded_header);
+
+ dict = DictEntriesRead(dictOid);
+
+ jsonb = palloc(decoded_size + VARHDRSZ);
+ jsonb_data = (uint8 *) VARDATA(jsonb);
+
+ decompress(dict, encoded_data, encoded_size, jsonb_data, decoded_size);
+
+ decoded_size += VARHDRSZ;
+ SET_VARSIZE(jsonb, decoded_size);
+
+ DictEntriesFree(dict);
+
+ PG_RETURN_JSONB_P(jsonb);
+}
+
+/*
+ * Converts a dictionary type to bytea.
+ */
+Datum
+dictionary_bytea(PG_FUNCTION_ARGS)
+{
+ bytea *compressed_data = PG_GETARG_BYTEA_P(0);
+ PG_RETURN_BYTEA_P(compressed_data);
+}
\ No newline at end of file
diff --git a/src/backend/utils/fmgr/funcapi.c b/src/backend/utils/fmgr/funcapi.c
index a1fe50ffca..e20c45f485 100644
--- a/src/backend/utils/fmgr/funcapi.c
+++ b/src/backend/utils/fmgr/funcapi.c
@@ -1292,6 +1292,7 @@ get_type_func_class(Oid typid, Oid *base_typeid)
case TYPTYPE_COMPOSITE:
return TYPEFUNC_COMPOSITE;
case TYPTYPE_BASE:
+ case TYPTYPE_DICT:
case TYPTYPE_ENUM:
case TYPTYPE_RANGE:
case TYPTYPE_MULTIRANGE:
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index e4fdb6b75b..fa1458416f 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -10183,6 +10183,8 @@ dumpType(Archive *fout, const TypeInfo *tyinfo)
dumpDomain(fout, tyinfo);
else if (tyinfo->typtype == TYPTYPE_COMPOSITE)
dumpCompositeType(fout, tyinfo);
+ else if (tyinfo->typtype == TYPTYPE_DICT)
+ pg_log_error("AALEKSEEV TODO FIXME not implemented");
else if (tyinfo->typtype == TYPTYPE_ENUM)
dumpEnumType(fout, tyinfo);
else if (tyinfo->typtype == TYPTYPE_RANGE)
diff --git a/src/include/catalog/pg_dict.h b/src/include/catalog/pg_dict.h
new file mode 100644
index 0000000000..fbad803468
--- /dev/null
+++ b/src/include/catalog/pg_dict.h
@@ -0,0 +1,78 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_dict.h
+ * definition of the "dict" system catalog (pg_dict)
+ *
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ * src/include/catalog/pg_dict.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef PG_DICT_H
+#define PG_DICT_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_dict_d.h"
+
+#include "nodes/pg_list.h"
+
+/* ----------------
+ * pg_dict definition. cpp turns this into
+ * typedef struct FormData_pg_dict
+ * ----------------
+ */
+CATALOG(pg_dict,9861,DictRelationId)
+{
+ Oid oid; /* oid */
+ Oid dicttypid BKI_LOOKUP(pg_type); /* OID of owning dict type */
+ /* AALEKSEEV TODO refactor to bytea */
+ NameData dictentry; /* text representation of the dictionary entry */
+} FormData_pg_dict;
+
+/* ----------------
+ * Form_pg_dict corresponds to a pointer to a tuple with
+ * the format of pg_dict relation.
+ * ----------------
+ */
+typedef FormData_pg_dict *Form_pg_dict;
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_dict_oid_index, 9862, DictOidIndexId, on pg_dict using btree(oid oid_ops));
+DECLARE_UNIQUE_INDEX(pg_dict_typid_entry_index, 9863, DictTypIdEntryIndexId, on pg_dict using btree(dicttypid oid_ops, dictentry name_ops));
+
+
+/*
+ * DictEntry type represents one entry in the given dictionary.
+ */
+typedef struct
+{
+ Oid oid; /* entry id */
+ uint32 length; /* entry length */
+ uint8* data; /* entry data */
+} DictEntry;
+
+/*
+ * Dictionary type represents all the entries in the given dictionary.
+ */
+typedef struct
+{
+ uint32 nentries; /* number of entries */
+ DictEntry *entries; /* array of entries */
+} DictionaryData;
+
+typedef DictionaryData *Dictionary;
+
+/*
+ * prototypes for functions in pg_dict.c
+ */
+extern void DictEntriesCreate(Oid dictTypeOid, List *vals);
+extern void DictEntriesFree(Dictionary dict);
+extern Dictionary DictEntriesRead(Oid dictTypeOid);
+extern void DictEntriesDelete(Oid dictTypeOid);
+
+#endif /* PG_DICT_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 2e41f4d9e8..33d51e44e2 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10024,6 +10024,26 @@
proname => 'jsonb_path_match_opr', prorettype => 'bool',
proargtypes => 'jsonb jsonpath', prosrc => 'jsonb_path_match_opr' },
+# dictionaries
+{ oid => '9864', descr => 'Converts a cstring to a dictionary',
+ proname => 'dictionary_in', prorettype => 'any', proargtypes => 'cstring oid int4',
+ prosrc => 'dictionary_in' },
+{ oid => '9865', descr => 'Converts a dictionary to a cstring',
+ proname => 'dictionary_out', prorettype => 'cstring', proargtypes => 'any',
+ prosrc => 'dictionary_out' },
+{ oid => '9866', descr => 'Coverts JSONB to a dictionary type',
+ proname => 'jsonb_dictionary', proisstrict => 'f', provolatile => 's',
+ prorettype => 'any', proargtypes => 'jsonb int4',
+ prosrc => 'jsonb_dictionary' },
+{ oid => '9867', descr => 'Converts a dictionary type to JSONB',
+ proname => 'dictionary_jsonb', proisstrict => 'f', provolatile => 's',
+ prorettype => 'jsonb', proargtypes => 'any',
+ prosrc => 'dictionary_jsonb' },
+{ oid => '9868', descr => 'Converts a dictionary type to a byte array',
+ proname => 'dictionary_bytea', proisstrict => 'f', provolatile => 's',
+ prorettype => 'jsonb', proargtypes => 'any',
+ prosrc => 'dictionary_bytea' },
+
# historical int8/txid_snapshot variants of xid8 functions
{ oid => '2939', descr => 'I/O',
proname => 'txid_snapshot_in', prorettype => 'txid_snapshot',
diff --git a/src/include/catalog/pg_type.h b/src/include/catalog/pg_type.h
index 48a2559137..91c6bfae99 100644
--- a/src/include/catalog/pg_type.h
+++ b/src/include/catalog/pg_type.h
@@ -272,6 +272,7 @@ DECLARE_UNIQUE_INDEX(pg_type_typname_nsp_index, 2704, TypeNameNspIndexId, on pg_
*/
#define TYPTYPE_BASE 'b' /* base type (ordinary scalar type) */
#define TYPTYPE_COMPOSITE 'c' /* composite (e.g., table's rowtype) */
+#define TYPTYPE_DICT 'D' /* dictionary */
#define TYPTYPE_DOMAIN 'd' /* domain over another type */
#define TYPTYPE_ENUM 'e' /* enumerated type */
#define TYPTYPE_MULTIRANGE 'm' /* multirange type */
diff --git a/src/include/commands/typecmds.h b/src/include/commands/typecmds.h
index a17bedb851..5cc8e5ff99 100644
--- a/src/include/commands/typecmds.h
+++ b/src/include/commands/typecmds.h
@@ -24,6 +24,7 @@
extern ObjectAddress DefineType(ParseState *pstate, List *names, List *parameters);
extern void RemoveTypeById(Oid typeOid);
extern ObjectAddress DefineDomain(CreateDomainStmt *stmt);
+extern ObjectAddress DefineDictionary(CreateDictionaryStmt *stmt);
extern ObjectAddress DefineEnum(CreateEnumStmt *stmt);
extern ObjectAddress DefineRange(ParseState *pstate, CreateRangeStmt *stmt);
extern ObjectAddress AlterEnum(AlterEnumStmt *stmt);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 0b6a7bb365..b6c1d548ef 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3579,6 +3579,18 @@ typedef struct CompositeTypeStmt
List *coldeflist; /* list of ColumnDef nodes */
} CompositeTypeStmt;
+/* ----------------------
+ * Create Type Statement, dictionary types
+ * ----------------------
+ */
+typedef struct CreateDictionaryStmt
+{
+ NodeTag type;
+ List *typeName; /* qualified name (list of String) */
+ List *baseTypeName; /* qualified base type name (list of String) */
+ List *vals; /* dictionary entries (list of String) */
+} CreateDictionaryStmt;
+
/* ----------------------
* Create Type Statement, enum types
* ----------------------
diff --git a/src/pl/plpgsql/src/pl_comp.c b/src/pl/plpgsql/src/pl_comp.c
index b791c23f06..897051c1f7 100644
--- a/src/pl/plpgsql/src/pl_comp.c
+++ b/src/pl/plpgsql/src/pl_comp.c
@@ -2129,6 +2129,7 @@ build_datatype(HeapTuple typeTup, int32 typmod,
switch (typeStruct->typtype)
{
case TYPTYPE_BASE:
+ case TYPTYPE_DICT:
case TYPTYPE_ENUM:
case TYPTYPE_RANGE:
case TYPTYPE_MULTIRANGE:
diff --git a/src/test/modules/test_oat_hooks/test_oat_hooks.c b/src/test/modules/test_oat_hooks/test_oat_hooks.c
index 0ad77e743d..d5784efb5c 100644
--- a/src/test/modules/test_oat_hooks/test_oat_hooks.c
+++ b/src/test/modules/test_oat_hooks/test_oat_hooks.c
@@ -1368,6 +1368,9 @@ nodetag_to_string(NodeTag tag)
case T_CompositeTypeStmt:
return "CompositeTypeStmt";
break;
+ case T_CreateDictionaryStmt:
+ return "CreateDictionaryStmt";
+ break;
case T_CreateEnumStmt:
return "CreateEnumStmt";
break;
diff --git a/src/test/regress/expected/dict.out b/src/test/regress/expected/dict.out
new file mode 100644
index 0000000000..699e76b654
--- /dev/null
+++ b/src/test/regress/expected/dict.out
@@ -0,0 +1,47 @@
+--
+-- Compression dictionaries tests
+--
+-- Dictionary types currently can be used only with JSONB
+\set ON_ERROR_STOP 0
+CREATE TYPE textdict AS DICTIONARY OF TEXT ('abcdef', 'ghijkl');
+ERROR: Dictionary types currently can be used only with JSONB
+\set ON_ERROR_STOP 1
+-- Simple happy path
+CREATE TYPE mydict AS DICTIONARY OF JSONB ('abcdef', 'ghijkl');
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
+ dictentry
+-----------
+ abcdef
+ ghijkl
+(2 rows)
+
+SELECT '{"abcdef":"ghijkl"}' :: mydict;
+ mydict
+----------------------
+ {"abcdef": "ghijkl"}
+(1 row)
+
+SELECT '{"abcdef":"ghijkl"}' :: mydict :: jsonb;
+ jsonb
+----------------------
+ {"abcdef": "ghijkl"}
+(1 row)
+
+SELECT '{"abcdef":"ghijkl"}' :: jsonb :: mydict;
+ mydict
+----------------------
+ {"abcdef": "ghijkl"}
+(1 row)
+
+SELECT ('{"abcdef":"ghijkl"}' :: mydict) -> 'abcdef';
+ ?column?
+----------
+ "ghijkl"
+(1 row)
+
+DROP TYPE mydict;
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
+ dictentry
+-----------
+(0 rows)
+
diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out
index 215eb899be..91bc491a6c 100644
--- a/src/test/regress/expected/oidjoins.out
+++ b/src/test/regress/expected/oidjoins.out
@@ -182,6 +182,7 @@ NOTICE: checking pg_description {classoid} => pg_class {oid}
NOTICE: checking pg_cast {castsource} => pg_type {oid}
NOTICE: checking pg_cast {casttarget} => pg_type {oid}
NOTICE: checking pg_cast {castfunc} => pg_proc {oid}
+NOTICE: checking pg_dict {dicttypid} => pg_type {oid}
NOTICE: checking pg_enum {enumtypid} => pg_type {oid}
NOTICE: checking pg_namespace {nspowner} => pg_authid {oid}
NOTICE: checking pg_conversion {connamespace} => pg_namespace {oid}
diff --git a/src/test/regress/expected/opr_sanity.out b/src/test/regress/expected/opr_sanity.out
index 86d755aa44..5dc4d0ab01 100644
--- a/src/test/regress/expected/opr_sanity.out
+++ b/src/test/regress/expected/opr_sanity.out
@@ -388,11 +388,12 @@ WHERE 'cstring'::regtype = ANY (p1.proargtypes)
AND NOT EXISTS(SELECT 1 FROM pg_conversion WHERE conproc = p1.oid)
AND p1.oid != 'shell_in(cstring)'::regprocedure
ORDER BY 1;
- oid | proname
-------+--------------
+ oid | proname
+------+---------------
2293 | cstring_out
2501 | cstring_send
-(2 rows)
+ 9864 | dictionary_in
+(3 rows)
-- Likewise, look for functions that return cstring and aren't datatype output
-- functions nor typmod output functions.
@@ -405,11 +406,12 @@ WHERE p1.prorettype = 'cstring'::regtype
AND NOT EXISTS(SELECT 1 FROM pg_type WHERE typmodout = p1.oid)
AND p1.oid != 'shell_out(void)'::regprocedure
ORDER BY 1;
- oid | proname
-------+--------------
+ oid | proname
+------+----------------
2292 | cstring_in
2500 | cstring_recv
-(2 rows)
+ 9865 | dictionary_out
+(3 rows)
-- Check for length inconsistencies between the various argument-info arrays.
SELECT p1.oid, p1.proname
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 103e11483d..2fa15a2080 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -111,7 +111,7 @@ test: select_views portals_p2 foreign_key cluster dependency guc bitmapops combo
# ----------
# Another group of parallel tests (JSON related)
# ----------
-test: json jsonb json_encoding jsonpath jsonpath_encoding jsonb_jsonpath sqljson json_sqljson jsonb_sqljson
+test: dict json jsonb json_encoding jsonpath jsonpath_encoding jsonb_jsonpath sqljson json_sqljson jsonb_sqljson
# ----------
# Another group of parallel tests
diff --git a/src/test/regress/sql/dict.sql b/src/test/regress/sql/dict.sql
new file mode 100644
index 0000000000..27ecd6a451
--- /dev/null
+++ b/src/test/regress/sql/dict.sql
@@ -0,0 +1,20 @@
+--
+-- Compression dictionaries tests
+--
+
+-- Dictionary types currently can be used only with JSONB
+\set ON_ERROR_STOP 0
+CREATE TYPE textdict AS DICTIONARY OF TEXT ('abcdef', 'ghijkl');
+\set ON_ERROR_STOP 1
+
+-- Simple happy path
+CREATE TYPE mydict AS DICTIONARY OF JSONB ('abcdef', 'ghijkl');
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
+
+SELECT '{"abcdef":"ghijkl"}' :: mydict;
+SELECT '{"abcdef":"ghijkl"}' :: mydict :: jsonb;
+SELECT '{"abcdef":"ghijkl"}' :: jsonb :: mydict;
+SELECT ('{"abcdef":"ghijkl"}' :: mydict) -> 'abcdef';
+
+DROP TYPE mydict;
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
--
2.36.1
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: [PATCH] Compression dictionaries for JSONB
@ 2022-07-11 15:41 Aleksander Alekseev <[email protected]>
parent: Aleksander Alekseev <[email protected]>
0 siblings, 1 reply; 88+ messages in thread
From: Aleksander Alekseev @ 2022-07-11 15:41 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Matthias van de Meent <[email protected]>; Zhihong Yu <[email protected]>; Teodor Sigaev <[email protected]>
Hi hackers,
> I invite anyone interested to join this effort as a co-author!
Here is v5. Same as v4 but with a fixed compiler warning (thanks,
cfbot). Sorry for the noise.
--
Best regards,
Aleksander Alekseev
Attachments:
[application/octet-stream] v5-0001-Compression-dictionaries-for-JSONB.patch (48.9K, ../../CAJ7c6TNb_hp5sehj0J0mE_SxWPXvGpZgOv_uLp1GLPsQFAoi8w@mail.gmail.com/2-v5-0001-Compression-dictionaries-for-JSONB.patch)
download | inline diff:
From f0612fcf9b356f4cd38efeb28ec9dc5ddca1f372 Mon Sep 17 00:00:00 2001
From: Aleksander Alekseev <[email protected]>
Date: Tue, 12 Apr 2022 15:58:11 +0300
Subject: [PATCH v5] Compression dictionaries for JSONB
Usage example:
CREATE TYPE mydict AS DITIONARY OF jsonb ('aaa', 'bbb', ...);
SELECT ('{"aaa":"bbb"}' :: mydict) -> 'aaa';
The created type works as a drop-in replacement for JSONB. However,
its internal representation differs. The provided dictionary entries ('aaa',
'bbb', ..) are stored in pg_dict catalog table. When `mydict` sees one of
the entries in the document, it replaces it with the corresponding pg_dict
entry Oid. For more details regarding the compression algorithm and choosen
compromises please see the comments in the code.
In pg_type `mydict` has typtype = TYPTYPE_DICT. It works the same way
as TYPTYPE_BASE with only difference: corresponding `<type>_in`
(pg_type.typinput) and `<another-type>_<type>` (pg_cast.castfunc)
procedures receive the dictionary Oid as a `typmod` argument. This way the
procedures can distinguish `mydict1` from `mydict2` and use the proper
compression dictionary.
(NOTE: dear committer, please bump the catalog version!)
Author: Aleksander Alekseev <[email protected]>
Author: TODO FIXME add your name and email here
Reviewed-by: FIXME
Discussion: https://postgr.es/m/CAJ7c6TOtAB0z1UrksvGTStNE-herK-43bj22%3D5xVBg7S4vr5rQ%40mail.gmail.com
Discussion: https://postgr.es/m/CAJ7c6TPx7N-bVw0dZ1ASCDQKZJHhBYkT6w4HV1LzfS%2BUUTUfmA%40mail.gmail.com
---
src/backend/catalog/Makefile | 3 +-
src/backend/catalog/pg_dict.c | 197 +++++++
src/backend/commands/typecmds.c | 160 +++++-
src/backend/executor/functions.c | 1 +
src/backend/parser/gram.y | 8 +
src/backend/parser/parse_coerce.c | 17 +-
src/backend/parser/parse_type.c | 16 +
src/backend/tcop/utility.c | 13 +
src/backend/utils/adt/Makefile | 1 +
src/backend/utils/adt/dictionaries.c | 482 ++++++++++++++++++
src/backend/utils/fmgr/funcapi.c | 1 +
src/bin/pg_dump/pg_dump.c | 2 +
src/include/catalog/pg_dict.h | 78 +++
src/include/catalog/pg_proc.dat | 20 +
src/include/catalog/pg_type.h | 1 +
src/include/commands/typecmds.h | 1 +
src/include/nodes/parsenodes.h | 12 +
src/pl/plpgsql/src/pl_comp.c | 1 +
.../modules/test_oat_hooks/test_oat_hooks.c | 3 +
src/test/regress/expected/dict.out | 47 ++
src/test/regress/expected/oidjoins.out | 1 +
src/test/regress/expected/opr_sanity.out | 14 +-
src/test/regress/parallel_schedule | 2 +-
src/test/regress/sql/dict.sql | 20 +
24 files changed, 1090 insertions(+), 11 deletions(-)
create mode 100644 src/backend/catalog/pg_dict.c
create mode 100644 src/backend/utils/adt/dictionaries.c
create mode 100644 src/include/catalog/pg_dict.h
create mode 100644 src/test/regress/expected/dict.out
create mode 100644 src/test/regress/sql/dict.sql
diff --git a/src/backend/catalog/Makefile b/src/backend/catalog/Makefile
index 89a0221ec9..3b4330148e 100644
--- a/src/backend/catalog/Makefile
+++ b/src/backend/catalog/Makefile
@@ -33,6 +33,7 @@ OBJS = \
pg_conversion.o \
pg_db_role_setting.o \
pg_depend.o \
+ pg_dict.o \
pg_enum.o \
pg_inherits.o \
pg_largeobject.o \
@@ -61,7 +62,7 @@ CATALOG_HEADERS := \
pg_language.h pg_largeobject_metadata.h pg_largeobject.h pg_aggregate.h \
pg_statistic.h pg_statistic_ext.h pg_statistic_ext_data.h \
pg_rewrite.h pg_trigger.h pg_event_trigger.h pg_description.h \
- pg_cast.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \
+ pg_cast.h pg_dict.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \
pg_database.h pg_db_role_setting.h pg_tablespace.h \
pg_authid.h pg_auth_members.h pg_shdepend.h pg_shdescription.h \
pg_ts_config.h pg_ts_config_map.h pg_ts_dict.h \
diff --git a/src/backend/catalog/pg_dict.c b/src/backend/catalog/pg_dict.c
new file mode 100644
index 0000000000..456f3b5177
--- /dev/null
+++ b/src/backend/catalog/pg_dict.c
@@ -0,0 +1,197 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_dict.c
+ * routines to support manipulation of the pg_dict relation
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/catalog/pg_dict.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "access/genam.h"
+#include "access/skey.h"
+#include "access/table.h"
+#include "catalog/catalog.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_dict.h"
+#include "catalog/pg_type.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/memutils.h"
+#include "utils/rel.h"
+
+/*
+ * Creates an entry in pg_dict for each of the supplied values.
+ * vals is a list of String values.
+ */
+void
+DictEntriesCreate(Oid dictTypeOid, List *vals)
+{
+ Relation pg_dict;
+ NameData dictentry;
+ Datum values[Natts_pg_dict];
+ bool nulls[Natts_pg_dict];
+ ListCell *lc;
+ HeapTuple tup;
+
+ if (vals == NIL)
+ {
+ /* The list is empty; do nothing. */
+ return;
+ }
+
+ memset(nulls, false, sizeof(nulls));
+
+ /*
+ * We don't check the list of values for duplicates here. If there are
+ * any, the user will get an unique-index violation.
+ */
+
+ pg_dict = table_open(DictRelationId, RowExclusiveLock);
+ foreach(lc, vals)
+ {
+ Oid oid = GetNewOidWithIndex(pg_dict, DictOidIndexId,
+ Anum_pg_dict_oid);
+ char *entry = strVal(lfirst(lc));
+
+ /*
+ * Entries are stored in a name field, for easier syscache lookup, so
+ * check the length to make sure it's within range.
+ */
+ if (strlen(entry) > (NAMEDATALEN - 1))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_NAME),
+ errmsg("invalid dict entry \"%s\"", entry),
+ errdetail("Entries must be %d bytes or less.",
+ NAMEDATALEN - 1)));
+
+ /* AALEKSEEV TODO use bytea instead */
+ namestrcpy(&dictentry, entry);
+
+ values[Anum_pg_dict_oid - 1] = ObjectIdGetDatum(oid);
+ values[Anum_pg_dict_dicttypid - 1] = ObjectIdGetDatum(dictTypeOid);
+ values[Anum_pg_dict_dictentry - 1] = NameGetDatum(&dictentry);
+
+ tup = heap_form_tuple(RelationGetDescr(pg_dict), values, nulls);
+
+ CatalogTupleInsert(pg_dict, tup);
+ heap_freetuple(tup);
+ }
+
+ /* clean up */
+ table_close(pg_dict, RowExclusiveLock);
+}
+
+/*
+ * Returns all the entries for the dictinary with given Oid. Entries are sorted
+ * by dictentry. Note that shorter entries are considered smaller, i.e. 'abc'
+ * goes before 'abcdef'. The memory is allocated in a child memory context of
+ * the caller's memory context. It can be freed with DictEntriesFree().
+ *
+ * If there are no entries a valid but empty dictionary is returned.
+ */
+Dictionary
+DictEntriesRead(Oid dictTypeOid)
+{
+ Relation pg_dict;
+ ScanKeyData key[1];
+ SysScanDesc scan;
+ HeapTuple tup;
+ uint32 entries_allocated;
+ Dictionary dict;
+ MemoryContext myctx, oldctx;
+
+ myctx = AllocSetContextCreate(CurrentMemoryContext, "DictEntriesCtx", ALLOCSET_DEFAULT_SIZES);
+ oldctx = MemoryContextSwitchTo(myctx);
+
+ dict = (Dictionary)palloc(sizeof(DictionaryData));
+
+ entries_allocated = 8;
+ dict->nentries = 0;
+ dict->entries = (DictEntry*)palloc(entries_allocated*sizeof(DictEntry));
+
+ pg_dict = table_open(DictRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_dict_dicttypid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(dictTypeOid));
+
+ scan = systable_beginscan(pg_dict, DictTypIdEntryIndexId, true,
+ NULL, 1, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ if(dict->nentries == entries_allocated)
+ {
+ entries_allocated = entries_allocated * 2;
+ dict->entries = (DictEntry*)repalloc(dict->entries, entries_allocated*sizeof(DictEntry));
+ }
+
+ dict->entries[dict->nentries].oid = ((Form_pg_dict) GETSTRUCT(tup))->oid;
+ /*
+ * This is arguably not the fastest way to determine the length of
+ * the entry. Alternatively, we could store a precalculated value in
+ * the catalog. However, usually it's a good idea to keep things simple
+ * until somebody discovers a bottleneck in this exact place and
+ * proposes a concrete fix.
+ */
+ dict->entries[dict->nentries].length = (uint32)strlen(((Form_pg_dict) GETSTRUCT(tup))->dictentry.data);
+ dict->entries[dict->nentries].data = palloc(dict->entries[dict->nentries].length);
+ memcpy(dict->entries[dict->nentries].data, ((Form_pg_dict) GETSTRUCT(tup))->dictentry.data, dict->entries[dict->nentries].length);
+ dict->nentries++;
+ }
+
+ systable_endscan(scan);
+ table_close(pg_dict, RowExclusiveLock);
+
+ MemoryContextSwitchTo(oldctx);
+
+ return dict;
+}
+
+/*
+ * Frees the memory allocated for the dictionary.
+ */
+void
+DictEntriesFree(Dictionary dict)
+{
+ MemoryContextDelete(GetMemoryChunkContext(dict));
+}
+
+/*
+ * Deletes all the entries for the dictinary with given Oid.
+ */
+void
+DictEntriesDelete(Oid dictTypeOid)
+{
+ Relation pg_dict;
+ ScanKeyData key[1];
+ SysScanDesc scan;
+ HeapTuple tup;
+
+ pg_dict = table_open(DictRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_dict_dicttypid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(dictTypeOid));
+
+ scan = systable_beginscan(pg_dict, DictTypIdEntryIndexId, true,
+ NULL, 1, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ CatalogTupleDelete(pg_dict, &tup->t_self);
+ }
+
+ systable_endscan(scan);
+ table_close(pg_dict, RowExclusiveLock);
+}
diff --git a/src/backend/commands/typecmds.c b/src/backend/commands/typecmds.c
index 4f5e7c7ccb..7e8fc57e6f 100644
--- a/src/backend/commands/typecmds.c
+++ b/src/backend/commands/typecmds.c
@@ -46,6 +46,7 @@
#include "catalog/pg_collation.h"
#include "catalog/pg_constraint.h"
#include "catalog/pg_depend.h"
+#include "catalog/pg_dict.h"
#include "catalog/pg_enum.h"
#include "catalog/pg_language.h"
#include "catalog/pg_namespace.h"
@@ -679,6 +680,14 @@ RemoveTypeById(Oid typeOid)
if (((Form_pg_type) GETSTRUCT(tup))->typtype == TYPTYPE_RANGE)
RangeDelete(typeOid);
+ /*
+ * If it is a dictionary type, delete the pg_dict entries too; we don't
+ * bother with making a dependency entry for that, so it has to be done
+ * "by hand" here.
+ */
+ if (((Form_pg_type) GETSTRUCT(tup))->typtype == TYPTYPE_DICT)
+ DictEntriesDelete(typeOid);
+
ReleaseSysCache(tup);
table_close(relation, RowExclusiveLock);
@@ -763,8 +772,8 @@ DefineDomain(CreateDomainStmt *stmt)
/*
* Base type must be a plain base type, a composite type, another domain,
- * an enum or a range type. Domains over pseudotypes would create a
- * security hole. (It would be shorter to code this to just check for
+ * a dict, an enum or a range type. Domains over pseudotypes would create
+ * a security hole. (It would be shorter to code this to just check for
* pseudotypes; but it seems safer to call out the specific typtypes that
* are supported, rather than assume that all future typtypes would be
* automatically supported.)
@@ -772,6 +781,7 @@ DefineDomain(CreateDomainStmt *stmt)
typtype = baseType->typtype;
if (typtype != TYPTYPE_BASE &&
typtype != TYPTYPE_COMPOSITE &&
+ typtype != TYPTYPE_DICT &&
typtype != TYPTYPE_DOMAIN &&
typtype != TYPTYPE_ENUM &&
typtype != TYPTYPE_RANGE &&
@@ -1129,6 +1139,152 @@ DefineDomain(CreateDomainStmt *stmt)
}
+/*
+ * DefineDictionary
+ * Registers a new dictionary.
+ */
+ObjectAddress
+DefineDictionary(CreateDictionaryStmt *stmt)
+{
+ char *dictName;
+ char *dictBaseTypeName;
+ char *dictArrayName;
+ Oid dictNamespace;
+ AclResult aclresult;
+ Oid old_type_oid;
+ Oid dictArrayOid;
+ ObjectAddress dictTypeAddr;
+
+ Assert(list_length(stmt->baseTypeName) == 1);
+ dictBaseTypeName = strVal(linitial(stmt->baseTypeName));
+ if(pg_strcasecmp(dictBaseTypeName, "jsonb") != 0)
+ ereport(ERROR,
+ (
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("Dictionary types currently can be used only with JSONB")
+ )
+ );
+
+ /* Convert list of names to a name and namespace */
+ dictNamespace = QualifiedNameGetCreationNamespace(stmt->typeName,
+ &dictName);
+
+ /* Check we have creation rights in target namespace */
+ aclresult = pg_namespace_aclcheck(dictNamespace, GetUserId(), ACL_CREATE);
+ if (aclresult != ACLCHECK_OK)
+ aclcheck_error(aclresult, OBJECT_SCHEMA,
+ get_namespace_name(dictNamespace));
+
+ /*
+ * Check for collision with an existing type name. If there is one and
+ * it's an autogenerated array, we can rename it out of the way.
+ */
+ old_type_oid = GetSysCacheOid2(TYPENAMENSP, Anum_pg_type_oid,
+ CStringGetDatum(dictName),
+ ObjectIdGetDatum(dictNamespace));
+ if (OidIsValid(old_type_oid))
+ {
+ if (!moveArrayTypeName(old_type_oid, dictName, dictNamespace))
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("type \"%s\" already exists", dictName)));
+ }
+
+ /* Allocate OID for array type */
+ dictArrayOid = AssignTypeArrayOid();
+
+ /* Create the pg_type entry */
+ dictTypeAddr =
+ TypeCreate(InvalidOid, /* no predetermined type OID */
+ dictName, /* type name */
+ dictNamespace, /* namespace */
+ InvalidOid, /* relation oid (n/a here) */
+ 0, /* relation kind (ditto) */
+ GetUserId(), /* owner's ID */
+ -1, /* internal size: varlena, as for JSONB */
+ TYPTYPE_DICT, /* type-type: dictionary */
+ TYPCATEGORY_USER, /* type-category: user, as for JSONB */
+ false, /* dict types are never preferred */
+ DEFAULT_TYPDELIM, /* array element delimiter */
+ F_DICTIONARY_IN, /* input procedure */
+ F_DICTIONARY_OUT, /* output procedure */
+ InvalidOid, /* receive procedure: none */
+ InvalidOid, /* send procedure: none */
+ InvalidOid, /* typmodin procedure - none */
+ InvalidOid, /* typmodout procedure - none */
+ InvalidOid, /* analyze procedure - default */
+ InvalidOid, /* subscript procedure - none */
+ InvalidOid, /* element type ID */
+ false, /* this is not an array type */
+ dictArrayOid, /* array type we are about to create */
+ InvalidOid, /* base type ID (only for domains) */
+ NULL, /* never a default type value */
+ NULL, /* binary default isn't sent either */
+ false, /* passed by value: same as for JSONB */
+ TYPALIGN_INT, /* int alignment */
+ TYPSTORAGE_EXTENDED, /* TOAST strategy: fully toastable, as
+ * JSONB */
+ -1, /* typMod (Domains only) */
+ 0, /* Array dimensions of typbasetype */
+ false, /* Type NOT NULL */
+ InvalidOid); /* type's collation */
+
+ /*
+ * Create the array type that goes with it.
+ */
+ dictArrayName = makeArrayTypeName(dictName, dictNamespace);
+
+ TypeCreate(dictArrayOid, /* force assignment of this type OID */
+ dictArrayName, /* type name */
+ dictNamespace, /* namespace */
+ InvalidOid, /* relation oid (n/a here) */
+ 0, /* relation kind (ditto) */
+ GetUserId(), /* owner's ID */
+ -1, /* internal size (always varlena) */
+ TYPTYPE_BASE, /* type-type (base type) */
+ TYPCATEGORY_ARRAY, /* type-category (array) */
+ false, /* array types are never preferred */
+ DEFAULT_TYPDELIM, /* array element delimiter */
+ F_ARRAY_IN, /* input procedure */
+ F_ARRAY_OUT, /* output procedure */
+ F_ARRAY_RECV, /* receive procedure */
+ F_ARRAY_SEND, /* send procedure */
+ InvalidOid, /* typmodin procedure - none */
+ InvalidOid, /* typmodout procedure - none */
+ F_ARRAY_TYPANALYZE, /* analyze procedure */
+ F_ARRAY_SUBSCRIPT_HANDLER, /* array subscript procedure */
+ dictTypeAddr.objectId, /* element type ID */
+ true, /* yes this is an array type */
+ InvalidOid, /* no further array type */
+ InvalidOid, /* base type ID */
+ NULL, /* never a default type value */
+ NULL, /* binary default isn't sent either */
+ false, /* passed by value: same as for JSONB */
+ TYPALIGN_INT, /* enums have int align, so do their arrays */
+ TYPSTORAGE_EXTENDED, /* ARRAY is always toastable */
+ -1, /* typMod (Domains only) */
+ 0, /* Array dimensions of typbasetype */
+ false, /* Type NOT NULL */
+ InvalidOid); /* type's collation */
+
+ pfree(dictArrayName);
+
+ /* Enter the dict's entries into pg_dict */
+ DictEntriesCreate(dictTypeAddr.objectId, stmt->vals);
+
+ /* Create casts to and from JSONB */
+ CastCreate(JSONBOID, dictTypeAddr.objectId, F_JSONB_DICTIONARY, 'a', 'f', DEPENDENCY_INTERNAL);
+ CastCreate(dictTypeAddr.objectId, JSONBOID, F_DICTIONARY_JSONB, 'i', 'f', DEPENDENCY_INTERNAL);
+
+ /*
+ * Create explicit cast to bytea. This is convenient for debugging purposes.
+ * Casting bytea to a dictionary type is dangerous and thus not supported.
+ */
+ CastCreate(dictTypeAddr.objectId, BYTEAOID, F_DICTIONARY_BYTEA, 'e', 'f', DEPENDENCY_INTERNAL);
+
+ return dictTypeAddr;
+}
+
/*
* DefineEnum
* Registers a new enum.
diff --git a/src/backend/executor/functions.c b/src/backend/executor/functions.c
index f9460ae506..7ad3de3a00 100644
--- a/src/backend/executor/functions.c
+++ b/src/backend/executor/functions.c
@@ -1708,6 +1708,7 @@ check_sql_fn_retval(List *queryTreeLists,
if (fn_typtype == TYPTYPE_BASE ||
fn_typtype == TYPTYPE_DOMAIN ||
+ fn_typtype == TYPTYPE_DICT ||
fn_typtype == TYPTYPE_ENUM ||
fn_typtype == TYPTYPE_RANGE ||
fn_typtype == TYPTYPE_MULTIRANGE)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 0523013f53..51ab3e89db 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -6237,6 +6237,14 @@ DefineStmt:
n->coldeflist = $6;
$$ = (Node *) n;
}
+ | CREATE TYPE_P any_name AS DICTIONARY OF any_name '(' opt_enum_val_list ')'
+ {
+ CreateDictionaryStmt *n = makeNode(CreateDictionaryStmt);
+ n->typeName = $3;
+ n->baseTypeName = $7;
+ n->vals = $9;
+ $$ = (Node *)n;
+ }
| CREATE TYPE_P any_name AS ENUM_P '(' opt_enum_val_list ')'
{
CreateEnumStmt *n = makeNode(CreateEnumStmt);
diff --git a/src/backend/parser/parse_coerce.c b/src/backend/parser/parse_coerce.c
index c4e958e4aa..e489dd9421 100644
--- a/src/backend/parser/parse_coerce.c
+++ b/src/backend/parser/parse_coerce.c
@@ -279,7 +279,22 @@ coerce_type(ParseState *pstate, Node *node,
if (baseTypeId == INTERVALOID)
inputTypeMod = baseTypeMod;
else
- inputTypeMod = -1;
+ {
+ if((int32)targetTypeId == targetTypeMod)
+ {
+ /*
+ * For dictionaries internally we set typmod to a dictionary
+ * OID. This allows dictionary_in() and jsonb_dictionary() to
+ * distinguish one dictionary from another.
+ *
+ * It also allows us to recognize that the target type is
+ * a dictionary here and pass a correct typmod.
+ */
+ inputTypeMod = targetTypeMod;
+ }
+ else
+ inputTypeMod = -1;
+ }
baseType = typeidType(baseTypeId);
diff --git a/src/backend/parser/parse_type.c b/src/backend/parser/parse_type.c
index f7ad689459..2188002535 100644
--- a/src/backend/parser/parse_type.c
+++ b/src/backend/parser/parse_type.c
@@ -259,6 +259,10 @@ LookupTypeNameOid(ParseState *pstate, const TypeName *typeName, bool missing_ok)
* This is equivalent to LookupTypeName, except that this will report
* a suitable error message if the type cannot be found or is not defined.
* Callers of this can therefore assume the result is a fully valid type.
+ *
+ * For a dictionary type it's OID is returned as a typmod. This is used by
+ * dictionary_in() and jsonb_dictionary() to distinguish one dictionary from
+ * another.
*/
Type
typenameType(ParseState *pstate, const TypeName *typeName, int32 *typmod_p)
@@ -305,6 +309,10 @@ typenameTypeId(ParseState *pstate, const TypeName *typeName)
*
* This is equivalent to typenameType, but we only hand back the type OID
* and typmod, not the syscache entry.
+ *
+ * For a dictionary type it's OID is returned as a typmod. This is used by
+ * dictionary_in() and jsonb_dictionary() to distinguish one dictionary from
+ * another.
*/
void
typenameTypeIdAndMod(ParseState *pstate, const TypeName *typeName,
@@ -327,6 +335,10 @@ typenameTypeIdAndMod(ParseState *pstate, const TypeName *typeName,
* looked up, and is passed as "typ".
*
* pstate is only used for error location info, and may be NULL.
+ *
+ * For a dictionary type it's OID is returned as a typmod. This is used by
+ * dictionary_in() and jsonb_dictionary() to distinguish one dictionary from
+ * another.
*/
static int32
typenameTypeMod(ParseState *pstate, const TypeName *typeName, Type typ)
@@ -339,6 +351,10 @@ typenameTypeMod(ParseState *pstate, const TypeName *typeName, Type typ)
ArrayType *arrtypmod;
ParseCallbackState pcbstate;
+ /* For a dictionary return it's OID. */
+ if (((Form_pg_type) GETSTRUCT(typ))->typtype == TYPTYPE_DICT)
+ return (int32)((Form_pg_type) GETSTRUCT(typ))->oid;
+
/* Return prespecified typmod if no typmod expressions */
if (typeName->typmods == NIL)
return typeName->typemod;
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index 6b0a865262..40cb80eb4a 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -174,6 +174,7 @@ ClassifyUtilityCommandAsReadOnly(Node *parsetree)
case T_CreateCastStmt:
case T_CreateConversionStmt:
case T_CreateDomainStmt:
+ case T_CreateDictionaryStmt:
case T_CreateEnumStmt:
case T_CreateEventTrigStmt:
case T_CreateExtensionStmt:
@@ -1621,6 +1622,10 @@ ProcessUtilitySlow(ParseState *pstate,
}
break;
+ case T_CreateDictionaryStmt: /* CREATE TYPE AS DICTIONARY OF */
+ address = DefineDictionary((CreateDictionaryStmt *) parsetree);
+ break;
+
case T_CreateEnumStmt: /* CREATE TYPE AS ENUM */
address = DefineEnum((CreateEnumStmt *) parsetree);
break;
@@ -2772,6 +2777,10 @@ CreateCommandTag(Node *parsetree)
tag = CMDTAG_CREATE_TYPE;
break;
+ case T_CreateDictionaryStmt:
+ tag = CMDTAG_CREATE_TYPE;
+ break;
+
case T_CreateEnumStmt:
tag = CMDTAG_CREATE_TYPE;
break;
@@ -3419,6 +3428,10 @@ GetCommandLogLevel(Node *parsetree)
lev = LOGSTMT_DDL;
break;
+ case T_CreateDictionaryStmt:
+ lev = LOGSTMT_DDL;
+ break;
+
case T_CreateEnumStmt:
lev = LOGSTMT_DDL;
break;
diff --git a/src/backend/utils/adt/Makefile b/src/backend/utils/adt/Makefile
index 7c722ea2ce..a7ec3d89f3 100644
--- a/src/backend/utils/adt/Makefile
+++ b/src/backend/utils/adt/Makefile
@@ -30,6 +30,7 @@ OBJS = \
datetime.o \
datum.o \
dbsize.o \
+ dictionaries.o \
domains.o \
encode.o \
enum.o \
diff --git a/src/backend/utils/adt/dictionaries.c b/src/backend/utils/adt/dictionaries.c
new file mode 100644
index 0000000000..2bb46471a4
--- /dev/null
+++ b/src/backend/utils/adt/dictionaries.c
@@ -0,0 +1,482 @@
+/*-------------------------------------------------------------------------
+ *
+ * dictionaries.c
+ * Conversion functions for dictionary types.
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/adt/dictionaries.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "catalog/pg_dict.h"
+#include "utils/fmgrprotos.h"
+#include "utils/jsonb.h"
+
+/*
+ * When compressing a data we treat it as a BLOB, in other words we don't
+ * assume anything regarding its internal representation. This is not
+ * necessarily the best and/or the only possible approach. However, it is
+ * universal and can be reused for JSONB, TEXT, XML and other types. Alternative
+ * compression methods can be introduced in the future, so that the user will
+ * be able to choose the best one for the task.
+ *
+ * The compressed data is stored in the following format:
+ *
+ * (struct varlena)
+ * dictionary_id [uint32]
+ * decompressed_size [uint32]
+ * algorithm_version [uint8]
+ *
+ * (repeated) {
+ * number of bytes to copy as-is [uint8]
+ * ... bytes to copy as-is ...
+ * dictionary entry id, or 0 to skip [uint32]
+ * }
+ *
+ * Compressed data is a variable-length type and thus has the 'struct varlena'
+ * header. The code below doesn't consider it being a part of the payload
+ * (e.g. see the DICT_COMP_HEADER_SIZE definition).
+ *
+ * Storing dictionary_id may seem redundant, but without it dictionary_out()
+ * and dictionary_jsonb() have no way to distinguish one dictionary type from
+ * another. dictionary_in() and jsonb_dictionary() get the dictionary id through
+ * the 'typemod' argument.
+ *
+ * Currently algorithm_version is always 0. In the future it will allow us to
+ * introduce new features (e.g. usage of varints) and/or lazily migrate the
+ * data to other compression methods.
+ *
+ * Compression and decompression are implemented based on a binary search over
+ * DictEntry[] array (see pg_dict.h). For compression the array is sorted by
+ * dictentries and for decompression - by oid's.
+ *
+ */
+
+/* Size of the header in the compressed data */
+#define DICT_COMP_HEADER_SIZE (sizeof(uint32)*2 + sizeof(uint8))
+
+/* Extracts dictionary_id from the compressed data */
+#define DICT_COMP_DICTIONARY_ID(hdrp) \
+ (*(uint32*)hdrp)
+
+/* Extracts decompressed_size from the compressed data */
+#define DICT_COMP_DECOMPRESSED_SIZE(hdrp) \
+ (*(uint32*)((uint8*)hdrp + sizeof(uint32)))
+
+/* Extracts algorithm_version from the compressed data */
+#define DICT_COMP_ALGORITHM_VERSION(hdrp) \
+ (*(uint8*)((uint8*)hdrp + sizeof(uint32)*2))
+
+/* Current algorithm_version */
+#define DICT_COMP_CURRENT_ALGORITHM_VERSION 0
+
+/*
+ * bsearch_arg() callback for finding DictEntry by oid.
+ */
+static int
+find_by_oid_cb(const void *key, const void *current, void *arg)
+{
+ /* Note that oids are unsigned, so we should be careful here */
+ if (*(Oid *) key < ((DictEntry *) current)->oid)
+ return -1;
+ else if (*(Oid *) key == ((DictEntry *) current)->oid)
+ return 0; /* found! */
+ else
+ return 1;
+}
+
+/*
+ * qsort() callback for sorting DictEntry[] by oids.
+ */
+static int
+sort_by_oids_cb(const void *left, const void *right)
+{
+ /* Note that oids are unsigned, so we should be careful here */
+ if (((DictEntry *) left)->oid < ((DictEntry *) right)->oid)
+ return -1;
+
+ /* Oids are unique so this callback will never return 0 */
+ return 1;
+}
+
+/*
+ * Finds a DictEntry which dictentry field matches *data and returns its Oid.
+ * If there are several matching entries, the largest is returned. The length
+ * of the found entry is written to *found_length on success. On failure
+ * InvalidOid is returned and found_length is zeroed.
+ *
+ * The implementation is similar to bsearch_arg(). The procedure can't be used
+ * directly because we are looking not by the exact match. We could generalize
+ * this case but the signature of the function becomes so complicated that it
+ * doesn't seem to worth the effort.
+ */
+static Oid
+compress_find_oid(Dictionary dict, const uint8 *data, Size data_size, Size *found_length)
+{
+ int res;
+ int32 left = 0;
+ int32 right = dict->nentries - 1;
+ Size best_length = 0;
+ Oid best_match = InvalidOid;
+
+ while (left <= right)
+ {
+ int32 current = (left + right) / 2;
+ Size nbytes = (Size)dict->entries[current].length;
+
+ if (nbytes > data_size)
+ {
+ /* current can be less or greater depending on the prefix */
+ res = memcmp(dict->entries[current].data, data, data_size);
+
+ /* if prefixes match, current is greater */
+ if (res == 0)
+ res = 1;
+ }
+ else
+ res = memcmp(dict->entries[current].data, data, nbytes);
+
+ if (res == 0) /* match found */
+ {
+ best_length = nbytes;
+ best_match = dict->entries[current].oid;
+
+ if (nbytes == data_size)
+ break;
+
+ /* maybe there is a larger match */
+ left = current + 1;
+ }
+ else if (res < 0) /* current is less */
+ left = current + 1;
+ else /* current is greater */
+ right = current - 1;
+ }
+
+ *found_length = best_length;
+ return best_match;
+}
+
+/*
+ * Finds a DictEntry by Oid using a binary search. The dictionary should be
+ * sorted by oids before the call. Returns NULL if nothing was found.
+ */
+static DictEntry *
+decompress_find_dictentry(Dictionary dict, Oid oid)
+{
+ return (DictEntry *) bsearch_arg(&oid, dict->entries, dict->nentries, sizeof(DictEntry), find_by_oid_cb, NULL);
+}
+
+/*
+ * Estimates the worst-case compressed size of the data of given size.
+ * Worst-case scenario happens when the dictionary consists of single-character
+ * entries. In this case every byte will be encoded as 6 bytes:
+ * 0x00, (0 bytes to copy as-is), 4 bytes of the entry Oid
+ *
+ * This procedure doesn't account for the header size.
+ *
+ * AALEKSEEV FIXME don't use dictionary entries shorter than 5 bytes
+ */
+static Size
+worst_case_compressed_size(Size insize)
+{
+ return insize * 6;
+}
+
+/*
+ * Compresses the data using the provided dictionary. The dictionary should
+ * be sorted by dictentries before the call. Output buffer should be at least
+ * worst_case_compressed_size(src_size) bytes in size.
+ */
+static void
+compress(Dictionary dict,
+ const void *src_data_, Size src_size,
+ void *encoded_data_, Size *pencoded_size)
+{
+ Size nbytes;
+ Size inoffset;
+ Size outskipoffset = 0;
+ Size outoffset = 1;
+ uint8 skipbytes = 0;
+ const uint8 *src_data = src_data_;
+ uint8 *encoded_data = ((uint8 *) encoded_data_);
+
+ for (inoffset = 0; inoffset < src_size;)
+ {
+ Oid code = compress_find_oid(dict, &(src_data[inoffset]),
+ src_size - inoffset, &nbytes);
+
+ if (code == InvalidOid)
+ {
+ skipbytes++;
+ encoded_data[outoffset] = src_data[inoffset];
+ outoffset++;
+ inoffset++;
+
+ if (skipbytes == 255)
+ {
+ encoded_data[outskipoffset] = skipbytes;
+ encoded_data[outoffset++] = 0; /* InvalidOid */
+ encoded_data[outoffset++] = 0;
+ encoded_data[outoffset++] = 0;
+ encoded_data[outoffset++] = 0;
+ outskipoffset = outoffset++;
+ skipbytes = 0;
+ }
+ }
+ else
+ {
+ encoded_data[outskipoffset] = skipbytes;
+ encoded_data[outoffset++] = (code >> 24) & 0xFF;
+ encoded_data[outoffset++] = (code >> 16) & 0xFF;
+ encoded_data[outoffset++] = (code >> 8) & 0xFF;
+ encoded_data[outoffset++] = code & 0xFF;
+ outskipoffset = outoffset++;
+ skipbytes = 0;
+ inoffset += nbytes;
+ }
+ }
+
+ /* Double check that we didn't write out of buffer */
+ Assert(outoffset < worst_case_compressed_size(src_size));
+
+ encoded_data[outskipoffset] = skipbytes;
+ *pencoded_size = outoffset;
+}
+
+/*
+ * Report an internal error in decompress() procedure below.
+ * Under normal circumstances this should never happen.
+ */
+static void
+decompress_error()
+{
+ ereport(ERROR,
+ (
+ errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("Unable to decompress a dictionary type"),
+ errdetail("The compressed data seems to be corrupted"),
+ errhint("Please report the steps to reproduce the issue to pgsql-bugs@")
+ ));
+}
+
+/*
+ * Decompresses the data using the provided dictionary. The dictionary should
+ * be sorted by oids before the call.
+ */
+static void
+decompress(Dictionary dict,
+ const void *encoded_data_, Size encoded_size,
+ void *decoded_data_, Size decoded_size)
+{
+ Size inoffset = 0;
+ Size outoffset = 0;
+ Oid code;
+ uint8 skipbytes;
+ const uint8 *encoded_data = ((uint8 *) encoded_data_);
+ uint8 *decoded_data = decoded_data_;
+
+ for (inoffset = 0; inoffset < encoded_size;)
+ {
+ skipbytes = encoded_data[inoffset++];
+
+ if (skipbytes > decoded_size - outoffset)
+ decompress_error();
+
+ if (skipbytes > encoded_size - inoffset)
+ decompress_error();
+
+ memcpy(
+ &(decoded_data[outoffset]),
+ &(encoded_data[inoffset]),
+ skipbytes
+ );
+
+ outoffset += skipbytes;
+ inoffset += skipbytes;
+
+ if ((encoded_size == inoffset) && (decoded_size == outoffset))
+ break; /* end of input - its OK */
+
+ if (encoded_size - inoffset < 4)
+ decompress_error();
+
+ code = (Oid) encoded_data[inoffset++];
+ code = (code << 8) | (Oid) encoded_data[inoffset++];
+ code = (code << 8) | (Oid) encoded_data[inoffset++];
+ code = (code << 8) | (Oid) encoded_data[inoffset++];
+
+ if (code != InvalidOid)
+ {
+ Size entrylen;
+ DictEntry *entry = decompress_find_dictentry(dict, code);
+
+ if (entry == NULL)
+ decompress_error();
+
+ Assert(entry->oid == code);
+ entrylen = (Size)entry->length;
+
+ if (entrylen > decoded_size - outoffset)
+ decompress_error();
+
+ memcpy(
+ &(decoded_data[outoffset]),
+ entry->data,
+ entrylen
+ );
+
+ outoffset += entrylen;
+ }
+ }
+
+ Assert(decoded_size == outoffset);
+}
+
+/*
+ * Converts a cstring to a dictionary.
+ */
+Datum
+dictionary_in(PG_FUNCTION_ARGS)
+{
+ const char *instr = PG_GETARG_CSTRING(0);
+#ifdef NOT_USED
+ Oid typelem = PG_GETARG_OID(1);
+#endif
+ int32 typmod = PG_GETARG_INT32(2);
+ Jsonb *jsonb;
+ bytea *dict;
+
+ Assert(typmod != -1);
+
+ jsonb = DatumGetJsonbP(DirectFunctionCall1(jsonb_in, CStringGetDatum(instr)));
+ dict = DatumGetByteaP(DirectFunctionCall2(jsonb_dictionary, JsonbPGetDatum(jsonb), Int32GetDatum(typmod)));
+
+ PG_RETURN_BYTEA_P(dict);
+}
+
+/*
+ * Converts a dictionary to a cstring.
+ */
+Datum
+dictionary_out(PG_FUNCTION_ARGS)
+{
+ bytea *dict = PG_GETARG_BYTEA_P(0);
+ Jsonb *jsonb = DatumGetJsonbP(DirectFunctionCall1(dictionary_jsonb, PointerGetDatum(dict)));
+ const char *outstr = DatumGetCString(DirectFunctionCall1(jsonb_out, JsonbPGetDatum(jsonb)));
+
+ PG_RETURN_CSTRING(outstr);
+}
+
+/*
+ * Coverts JSONB to a dictionary type.
+ *
+ * AALEKSEEV FIXME: if the compressed document ends up being larger than
+ * the original one write a corresponding flag and copy all the data as-is.
+ */
+Datum
+jsonb_dictionary(PG_FUNCTION_ARGS)
+{
+ Dictionary dict;
+ Jsonb *jsonb = PG_GETARG_JSONB_P(0);
+ int32 typmod = PG_GETARG_INT32(1);
+ uint8 *jsonb_data = (uint8 *) VARDATA(jsonb);
+ Size jsonb_data_size = VARSIZE(jsonb) - VARHDRSZ;
+ uint8 *encoded_buff,
+ *encoded_header,
+ *encoded_data;
+ Size encoded_size,
+ encoded_buff_size;
+
+ Assert(typmod != -1);
+
+ dict = DictEntriesRead(typmod);
+ qsort((void *) dict->entries, dict->nentries, sizeof(DictEntry), sort_by_oids_cb);
+
+ encoded_buff_size = VARHDRSZ + DICT_COMP_HEADER_SIZE + worst_case_compressed_size(jsonb_data_size);
+ encoded_buff = palloc(encoded_buff_size);
+ encoded_header = (uint8 *) VARDATA(encoded_buff);
+ encoded_data = encoded_header + DICT_COMP_HEADER_SIZE;
+
+ DICT_COMP_DICTIONARY_ID(encoded_header) = typmod;
+ DICT_COMP_DECOMPRESSED_SIZE(encoded_header) = jsonb_data_size;
+ DICT_COMP_ALGORITHM_VERSION(encoded_header) = DICT_COMP_CURRENT_ALGORITHM_VERSION;
+
+ encoded_size = encoded_buff_size - VARHDRSZ - DICT_COMP_HEADER_SIZE;
+
+ compress(dict, jsonb_data, jsonb_data_size,
+ encoded_data, &encoded_size);
+
+ encoded_size += VARHDRSZ + DICT_COMP_HEADER_SIZE;
+
+ encoded_buff = repalloc(encoded_buff, encoded_size);
+ SET_VARSIZE(encoded_buff, encoded_size);
+
+ DictEntriesFree(dict);
+
+ PG_RETURN_BYTEA_P(encoded_buff);
+}
+
+/*
+ * Converts a dictionary type to JSONB.
+ */
+Datum
+dictionary_jsonb(PG_FUNCTION_ARGS)
+{
+ bytea *encoded_buff = PG_GETARG_BYTEA_P(0);
+ uint8 *encoded_header = (uint8 *) VARDATA(encoded_buff);
+ uint8 *encoded_data = encoded_header + DICT_COMP_HEADER_SIZE;
+ Size encoded_size = VARSIZE(encoded_buff) - VARHDRSZ - DICT_COMP_HEADER_SIZE;
+ int alg_version = DICT_COMP_ALGORITHM_VERSION(encoded_header);
+ Oid dictOid; /* cannot read until algorithm version is
+ * checked */
+ uint32 decoded_size; /* cannot read until algorithm version is
+ * checked */
+ Jsonb *jsonb;
+ uint8 *jsonb_data;
+ Dictionary dict;
+
+ if (alg_version > DICT_COMP_CURRENT_ALGORITHM_VERSION)
+ ereport(ERROR,
+ (
+ errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("Unsupported compression algorithm version"),
+ errdetail("Saved algorithm version is %d, current version is %d",
+ alg_version, DICT_COMP_CURRENT_ALGORITHM_VERSION),
+ errhint("The data is either corrupted or imported from "
+ "the future version of PostgreSQL")
+ ));
+
+ dictOid = DICT_COMP_DICTIONARY_ID(encoded_header);
+ decoded_size = DICT_COMP_DECOMPRESSED_SIZE(encoded_header);
+
+ dict = DictEntriesRead(dictOid);
+
+ jsonb = palloc(decoded_size + VARHDRSZ);
+ jsonb_data = (uint8 *) VARDATA(jsonb);
+
+ decompress(dict, encoded_data, encoded_size, jsonb_data, decoded_size);
+
+ decoded_size += VARHDRSZ;
+ SET_VARSIZE(jsonb, decoded_size);
+
+ DictEntriesFree(dict);
+
+ PG_RETURN_JSONB_P(jsonb);
+}
+
+/*
+ * Converts a dictionary type to bytea.
+ */
+Datum
+dictionary_bytea(PG_FUNCTION_ARGS)
+{
+ bytea *compressed_data = PG_GETARG_BYTEA_P(0);
+ PG_RETURN_BYTEA_P(compressed_data);
+}
\ No newline at end of file
diff --git a/src/backend/utils/fmgr/funcapi.c b/src/backend/utils/fmgr/funcapi.c
index a1fe50ffca..e20c45f485 100644
--- a/src/backend/utils/fmgr/funcapi.c
+++ b/src/backend/utils/fmgr/funcapi.c
@@ -1292,6 +1292,7 @@ get_type_func_class(Oid typid, Oid *base_typeid)
case TYPTYPE_COMPOSITE:
return TYPEFUNC_COMPOSITE;
case TYPTYPE_BASE:
+ case TYPTYPE_DICT:
case TYPTYPE_ENUM:
case TYPTYPE_RANGE:
case TYPTYPE_MULTIRANGE:
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index e4fdb6b75b..fa1458416f 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -10183,6 +10183,8 @@ dumpType(Archive *fout, const TypeInfo *tyinfo)
dumpDomain(fout, tyinfo);
else if (tyinfo->typtype == TYPTYPE_COMPOSITE)
dumpCompositeType(fout, tyinfo);
+ else if (tyinfo->typtype == TYPTYPE_DICT)
+ pg_log_error("AALEKSEEV TODO FIXME not implemented");
else if (tyinfo->typtype == TYPTYPE_ENUM)
dumpEnumType(fout, tyinfo);
else if (tyinfo->typtype == TYPTYPE_RANGE)
diff --git a/src/include/catalog/pg_dict.h b/src/include/catalog/pg_dict.h
new file mode 100644
index 0000000000..fbad803468
--- /dev/null
+++ b/src/include/catalog/pg_dict.h
@@ -0,0 +1,78 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_dict.h
+ * definition of the "dict" system catalog (pg_dict)
+ *
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ * src/include/catalog/pg_dict.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef PG_DICT_H
+#define PG_DICT_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_dict_d.h"
+
+#include "nodes/pg_list.h"
+
+/* ----------------
+ * pg_dict definition. cpp turns this into
+ * typedef struct FormData_pg_dict
+ * ----------------
+ */
+CATALOG(pg_dict,9861,DictRelationId)
+{
+ Oid oid; /* oid */
+ Oid dicttypid BKI_LOOKUP(pg_type); /* OID of owning dict type */
+ /* AALEKSEEV TODO refactor to bytea */
+ NameData dictentry; /* text representation of the dictionary entry */
+} FormData_pg_dict;
+
+/* ----------------
+ * Form_pg_dict corresponds to a pointer to a tuple with
+ * the format of pg_dict relation.
+ * ----------------
+ */
+typedef FormData_pg_dict *Form_pg_dict;
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_dict_oid_index, 9862, DictOidIndexId, on pg_dict using btree(oid oid_ops));
+DECLARE_UNIQUE_INDEX(pg_dict_typid_entry_index, 9863, DictTypIdEntryIndexId, on pg_dict using btree(dicttypid oid_ops, dictentry name_ops));
+
+
+/*
+ * DictEntry type represents one entry in the given dictionary.
+ */
+typedef struct
+{
+ Oid oid; /* entry id */
+ uint32 length; /* entry length */
+ uint8* data; /* entry data */
+} DictEntry;
+
+/*
+ * Dictionary type represents all the entries in the given dictionary.
+ */
+typedef struct
+{
+ uint32 nentries; /* number of entries */
+ DictEntry *entries; /* array of entries */
+} DictionaryData;
+
+typedef DictionaryData *Dictionary;
+
+/*
+ * prototypes for functions in pg_dict.c
+ */
+extern void DictEntriesCreate(Oid dictTypeOid, List *vals);
+extern void DictEntriesFree(Dictionary dict);
+extern Dictionary DictEntriesRead(Oid dictTypeOid);
+extern void DictEntriesDelete(Oid dictTypeOid);
+
+#endif /* PG_DICT_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 2e41f4d9e8..33d51e44e2 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10024,6 +10024,26 @@
proname => 'jsonb_path_match_opr', prorettype => 'bool',
proargtypes => 'jsonb jsonpath', prosrc => 'jsonb_path_match_opr' },
+# dictionaries
+{ oid => '9864', descr => 'Converts a cstring to a dictionary',
+ proname => 'dictionary_in', prorettype => 'any', proargtypes => 'cstring oid int4',
+ prosrc => 'dictionary_in' },
+{ oid => '9865', descr => 'Converts a dictionary to a cstring',
+ proname => 'dictionary_out', prorettype => 'cstring', proargtypes => 'any',
+ prosrc => 'dictionary_out' },
+{ oid => '9866', descr => 'Coverts JSONB to a dictionary type',
+ proname => 'jsonb_dictionary', proisstrict => 'f', provolatile => 's',
+ prorettype => 'any', proargtypes => 'jsonb int4',
+ prosrc => 'jsonb_dictionary' },
+{ oid => '9867', descr => 'Converts a dictionary type to JSONB',
+ proname => 'dictionary_jsonb', proisstrict => 'f', provolatile => 's',
+ prorettype => 'jsonb', proargtypes => 'any',
+ prosrc => 'dictionary_jsonb' },
+{ oid => '9868', descr => 'Converts a dictionary type to a byte array',
+ proname => 'dictionary_bytea', proisstrict => 'f', provolatile => 's',
+ prorettype => 'jsonb', proargtypes => 'any',
+ prosrc => 'dictionary_bytea' },
+
# historical int8/txid_snapshot variants of xid8 functions
{ oid => '2939', descr => 'I/O',
proname => 'txid_snapshot_in', prorettype => 'txid_snapshot',
diff --git a/src/include/catalog/pg_type.h b/src/include/catalog/pg_type.h
index 48a2559137..91c6bfae99 100644
--- a/src/include/catalog/pg_type.h
+++ b/src/include/catalog/pg_type.h
@@ -272,6 +272,7 @@ DECLARE_UNIQUE_INDEX(pg_type_typname_nsp_index, 2704, TypeNameNspIndexId, on pg_
*/
#define TYPTYPE_BASE 'b' /* base type (ordinary scalar type) */
#define TYPTYPE_COMPOSITE 'c' /* composite (e.g., table's rowtype) */
+#define TYPTYPE_DICT 'D' /* dictionary */
#define TYPTYPE_DOMAIN 'd' /* domain over another type */
#define TYPTYPE_ENUM 'e' /* enumerated type */
#define TYPTYPE_MULTIRANGE 'm' /* multirange type */
diff --git a/src/include/commands/typecmds.h b/src/include/commands/typecmds.h
index a17bedb851..5cc8e5ff99 100644
--- a/src/include/commands/typecmds.h
+++ b/src/include/commands/typecmds.h
@@ -24,6 +24,7 @@
extern ObjectAddress DefineType(ParseState *pstate, List *names, List *parameters);
extern void RemoveTypeById(Oid typeOid);
extern ObjectAddress DefineDomain(CreateDomainStmt *stmt);
+extern ObjectAddress DefineDictionary(CreateDictionaryStmt *stmt);
extern ObjectAddress DefineEnum(CreateEnumStmt *stmt);
extern ObjectAddress DefineRange(ParseState *pstate, CreateRangeStmt *stmt);
extern ObjectAddress AlterEnum(AlterEnumStmt *stmt);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 0b6a7bb365..b6c1d548ef 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3579,6 +3579,18 @@ typedef struct CompositeTypeStmt
List *coldeflist; /* list of ColumnDef nodes */
} CompositeTypeStmt;
+/* ----------------------
+ * Create Type Statement, dictionary types
+ * ----------------------
+ */
+typedef struct CreateDictionaryStmt
+{
+ NodeTag type;
+ List *typeName; /* qualified name (list of String) */
+ List *baseTypeName; /* qualified base type name (list of String) */
+ List *vals; /* dictionary entries (list of String) */
+} CreateDictionaryStmt;
+
/* ----------------------
* Create Type Statement, enum types
* ----------------------
diff --git a/src/pl/plpgsql/src/pl_comp.c b/src/pl/plpgsql/src/pl_comp.c
index b791c23f06..897051c1f7 100644
--- a/src/pl/plpgsql/src/pl_comp.c
+++ b/src/pl/plpgsql/src/pl_comp.c
@@ -2129,6 +2129,7 @@ build_datatype(HeapTuple typeTup, int32 typmod,
switch (typeStruct->typtype)
{
case TYPTYPE_BASE:
+ case TYPTYPE_DICT:
case TYPTYPE_ENUM:
case TYPTYPE_RANGE:
case TYPTYPE_MULTIRANGE:
diff --git a/src/test/modules/test_oat_hooks/test_oat_hooks.c b/src/test/modules/test_oat_hooks/test_oat_hooks.c
index 0ad77e743d..d5784efb5c 100644
--- a/src/test/modules/test_oat_hooks/test_oat_hooks.c
+++ b/src/test/modules/test_oat_hooks/test_oat_hooks.c
@@ -1368,6 +1368,9 @@ nodetag_to_string(NodeTag tag)
case T_CompositeTypeStmt:
return "CompositeTypeStmt";
break;
+ case T_CreateDictionaryStmt:
+ return "CreateDictionaryStmt";
+ break;
case T_CreateEnumStmt:
return "CreateEnumStmt";
break;
diff --git a/src/test/regress/expected/dict.out b/src/test/regress/expected/dict.out
new file mode 100644
index 0000000000..699e76b654
--- /dev/null
+++ b/src/test/regress/expected/dict.out
@@ -0,0 +1,47 @@
+--
+-- Compression dictionaries tests
+--
+-- Dictionary types currently can be used only with JSONB
+\set ON_ERROR_STOP 0
+CREATE TYPE textdict AS DICTIONARY OF TEXT ('abcdef', 'ghijkl');
+ERROR: Dictionary types currently can be used only with JSONB
+\set ON_ERROR_STOP 1
+-- Simple happy path
+CREATE TYPE mydict AS DICTIONARY OF JSONB ('abcdef', 'ghijkl');
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
+ dictentry
+-----------
+ abcdef
+ ghijkl
+(2 rows)
+
+SELECT '{"abcdef":"ghijkl"}' :: mydict;
+ mydict
+----------------------
+ {"abcdef": "ghijkl"}
+(1 row)
+
+SELECT '{"abcdef":"ghijkl"}' :: mydict :: jsonb;
+ jsonb
+----------------------
+ {"abcdef": "ghijkl"}
+(1 row)
+
+SELECT '{"abcdef":"ghijkl"}' :: jsonb :: mydict;
+ mydict
+----------------------
+ {"abcdef": "ghijkl"}
+(1 row)
+
+SELECT ('{"abcdef":"ghijkl"}' :: mydict) -> 'abcdef';
+ ?column?
+----------
+ "ghijkl"
+(1 row)
+
+DROP TYPE mydict;
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
+ dictentry
+-----------
+(0 rows)
+
diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out
index 215eb899be..91bc491a6c 100644
--- a/src/test/regress/expected/oidjoins.out
+++ b/src/test/regress/expected/oidjoins.out
@@ -182,6 +182,7 @@ NOTICE: checking pg_description {classoid} => pg_class {oid}
NOTICE: checking pg_cast {castsource} => pg_type {oid}
NOTICE: checking pg_cast {casttarget} => pg_type {oid}
NOTICE: checking pg_cast {castfunc} => pg_proc {oid}
+NOTICE: checking pg_dict {dicttypid} => pg_type {oid}
NOTICE: checking pg_enum {enumtypid} => pg_type {oid}
NOTICE: checking pg_namespace {nspowner} => pg_authid {oid}
NOTICE: checking pg_conversion {connamespace} => pg_namespace {oid}
diff --git a/src/test/regress/expected/opr_sanity.out b/src/test/regress/expected/opr_sanity.out
index 86d755aa44..5dc4d0ab01 100644
--- a/src/test/regress/expected/opr_sanity.out
+++ b/src/test/regress/expected/opr_sanity.out
@@ -388,11 +388,12 @@ WHERE 'cstring'::regtype = ANY (p1.proargtypes)
AND NOT EXISTS(SELECT 1 FROM pg_conversion WHERE conproc = p1.oid)
AND p1.oid != 'shell_in(cstring)'::regprocedure
ORDER BY 1;
- oid | proname
-------+--------------
+ oid | proname
+------+---------------
2293 | cstring_out
2501 | cstring_send
-(2 rows)
+ 9864 | dictionary_in
+(3 rows)
-- Likewise, look for functions that return cstring and aren't datatype output
-- functions nor typmod output functions.
@@ -405,11 +406,12 @@ WHERE p1.prorettype = 'cstring'::regtype
AND NOT EXISTS(SELECT 1 FROM pg_type WHERE typmodout = p1.oid)
AND p1.oid != 'shell_out(void)'::regprocedure
ORDER BY 1;
- oid | proname
-------+--------------
+ oid | proname
+------+----------------
2292 | cstring_in
2500 | cstring_recv
-(2 rows)
+ 9865 | dictionary_out
+(3 rows)
-- Check for length inconsistencies between the various argument-info arrays.
SELECT p1.oid, p1.proname
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 103e11483d..2fa15a2080 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -111,7 +111,7 @@ test: select_views portals_p2 foreign_key cluster dependency guc bitmapops combo
# ----------
# Another group of parallel tests (JSON related)
# ----------
-test: json jsonb json_encoding jsonpath jsonpath_encoding jsonb_jsonpath sqljson json_sqljson jsonb_sqljson
+test: dict json jsonb json_encoding jsonpath jsonpath_encoding jsonb_jsonpath sqljson json_sqljson jsonb_sqljson
# ----------
# Another group of parallel tests
diff --git a/src/test/regress/sql/dict.sql b/src/test/regress/sql/dict.sql
new file mode 100644
index 0000000000..27ecd6a451
--- /dev/null
+++ b/src/test/regress/sql/dict.sql
@@ -0,0 +1,20 @@
+--
+-- Compression dictionaries tests
+--
+
+-- Dictionary types currently can be used only with JSONB
+\set ON_ERROR_STOP 0
+CREATE TYPE textdict AS DICTIONARY OF TEXT ('abcdef', 'ghijkl');
+\set ON_ERROR_STOP 1
+
+-- Simple happy path
+CREATE TYPE mydict AS DICTIONARY OF JSONB ('abcdef', 'ghijkl');
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
+
+SELECT '{"abcdef":"ghijkl"}' :: mydict;
+SELECT '{"abcdef":"ghijkl"}' :: mydict :: jsonb;
+SELECT '{"abcdef":"ghijkl"}' :: jsonb :: mydict;
+SELECT ('{"abcdef":"ghijkl"}' :: mydict) -> 'abcdef';
+
+DROP TYPE mydict;
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
--
2.36.1
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: [PATCH] Compression dictionaries for JSONB
@ 2022-07-12 10:25 Nikita Malakhov <[email protected]>
parent: Aleksander Alekseev <[email protected]>
0 siblings, 1 reply; 88+ messages in thread
From: Nikita Malakhov @ 2022-07-12 10:25 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>; Alvaro Herrera <[email protected]>; Matthias van de Meent <[email protected]>; Zhihong Yu <[email protected]>; Teodor Sigaev <[email protected]>
Hi hackers!
Aleksander, please point me in the right direction if it was mentioned
before, I have a few questions:
1) It is not clear for me, how do you see the life cycle of such a
dictionary? If it is meant to keep growing without
cleaning up/rebuilding it could affect performance in an undesirable way,
along with keeping unused data without
any means to get rid of them.
Also, I agree with Simon Riggs, using OIDs from the general pool for
dictionary entries is a bad idea.
2) From (1) follows another question - I haven't seen any means for getting
rid of unused keys (or any other means
for dictionary cleanup). How could it be done?
3) Is the possible scenario legal - by some means a dictionary does not
contain some keys for entries? What happens then?
4) If one dictionary is used by several tables - I see future issues in
concurrent dictionary updates. This will for sure
affect performance and can cause unpredictable behavior for queries.
If you have any questions on Pluggable TOAST don't hesitate to ask me and
on JSONB Toaster you can ask Nikita Glukhov.
Thank you!
Regards,
Nikita Malakhov
Postgres Professional
https://postgrespro.ru/
On Mon, Jul 11, 2022 at 6:41 PM Aleksander Alekseev <
[email protected]> wrote:
> Hi hackers,
>
> > I invite anyone interested to join this effort as a co-author!
>
> Here is v5. Same as v4 but with a fixed compiler warning (thanks,
> cfbot). Sorry for the noise.
>
> --
> Best regards,
> Aleksander Alekseev
>
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: [PATCH] Compression dictionaries for JSONB
@ 2022-07-12 12:15 Aleksander Alekseev <[email protected]>
parent: Nikita Malakhov <[email protected]>
0 siblings, 1 reply; 88+ messages in thread
From: Aleksander Alekseev @ 2022-07-12 12:15 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>; +Cc: Nikita Malakhov <[email protected]>; Alvaro Herrera <[email protected]>; Matthias van de Meent <[email protected]>; Zhihong Yu <[email protected]>; Teodor Sigaev <[email protected]>
Hi Nikita,
> Aleksander, please point me in the right direction if it was mentioned before, I have a few questions:
Thanks for your feedback. These are good questions indeed.
> 1) It is not clear for me, how do you see the life cycle of such a dictionary? If it is meant to keep growing without
> cleaning up/rebuilding it could affect performance in an undesirable way, along with keeping unused data without
> any means to get rid of them.
> 2) From (1) follows another question - I haven't seen any means for getting rid of unused keys (or any other means
> for dictionary cleanup). How could it be done?
Good point. This was not a problem for ZSON since the dictionary size
was limited to 2**16 entries, the dictionary was immutable, and the
dictionaries had versions. For compression dictionaries we removed the
2**16 entries limit and also decided to get rid of versions. The idea
was that you can simply continue adding new entries, but no one
thought about the fact that this will consume the memory required to
decompress the document indefinitely.
Maybe we should return to the idea of limited dictionary size and
versions. Objections?
> 4) If one dictionary is used by several tables - I see future issues in concurrent dictionary updates. This will for sure
> affect performance and can cause unpredictable behavior for queries.
You are right. Another reason to return to the idea of dictionary versions.
> Also, I agree with Simon Riggs, using OIDs from the general pool for dictionary entries is a bad idea.
Yep, we agreed to stop using OIDs for this, however this was not
changed in the patch at this point. Please don't hesitate joining the
effort if you want to. I wouldn't mind taking a short break from this
patch.
> 3) Is the possible scenario legal - by some means a dictionary does not contain some keys for entries? What happens then?
No, we should either forbid removing dictionary entries or check that
all the existing documents are not using the entries being removed.
> If you have any questions on Pluggable TOAST don't hesitate to ask me and on JSONB Toaster you can ask Nikita Glukhov.
Will do! Thanks for working on this and I'm looking forward to the
next version of the patch for the next round of review.
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: [PATCH] Compression dictionaries for JSONB
@ 2022-07-17 18:15 Nikita Malakhov <[email protected]>
parent: Aleksander Alekseev <[email protected]>
0 siblings, 2 replies; 88+ messages in thread
From: Nikita Malakhov @ 2022-07-17 18:15 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>; Alvaro Herrera <[email protected]>; Matthias van de Meent <[email protected]>; Zhihong Yu <[email protected]>; Teodor Sigaev <[email protected]>
Hi hackers!
Aleksander, I've carefully gone over discussion and still have some
questions to ask -
1) Is there any means of measuring overhead of dictionaries over vanilla
implementation? IMO it is a must because
JSON is a widely used functionality. Also, as it was mentioned before, to
check the dictionary value must be detoasted;
2) Storing dictionaries in one table. As I wrote before, this will surely
lead to locks and waits while inserting and updating
dictionaries, and could cause serious performance issues. And vacuuming
this table will lead to locks for all tables using
dictionaries until vacuum is complete;
3) JSON documents in production environments could be very complex and use
thousands of keys, so creating dictionary
directly in SQL statement is not very good approach, so it's another reason
to have means for creating dictionaries as a
separate tables and/or passing them as files or so;
4) Suggested mechanics, if put on top of the TOAST, could not benefit from
knowledge if internal JSON structure, which
is seen as important drawback in spite of extensive research work done on
working with JSON schema (storing, validating,
etc.), and also it cannot recognize and help to compress duplicated parts
of JSON document;
5) A small test issue - if dictionaried' JSON has a key which is equal to
OID used in a dictionary for some other key?
In Pluggable TOAST we suggest that as an improvement compression should be
put inside the Toaster as an option,
thus the Toaster could have maximum benefits from knowledge of data
internal structure (and in future use JSON Schema).
For using in special Toaster for JSON datatype compression dictionaries
seem to be very valuable addition, but now I
have to agree that this feature in current state is competing with
Pluggable TOAST.
Thank you!
Regards,
Nikita Malakhov
Postgres Professional
https://postgrespro.ru/
On Tue, Jul 12, 2022 at 3:15 PM Aleksander Alekseev <
[email protected]> wrote:
> Hi Nikita,
>
> > Aleksander, please point me in the right direction if it was mentioned
> before, I have a few questions:
>
> Thanks for your feedback. These are good questions indeed.
>
> > 1) It is not clear for me, how do you see the life cycle of such a
> dictionary? If it is meant to keep growing without
> > cleaning up/rebuilding it could affect performance in an undesirable
> way, along with keeping unused data without
> > any means to get rid of them.
> > 2) From (1) follows another question - I haven't seen any means for
> getting rid of unused keys (or any other means
> > for dictionary cleanup). How could it be done?
>
> Good point. This was not a problem for ZSON since the dictionary size
> was limited to 2**16 entries, the dictionary was immutable, and the
> dictionaries had versions. For compression dictionaries we removed the
> 2**16 entries limit and also decided to get rid of versions. The idea
> was that you can simply continue adding new entries, but no one
> thought about the fact that this will consume the memory required to
> decompress the document indefinitely.
>
> Maybe we should return to the idea of limited dictionary size and
> versions. Objections?
>
> > 4) If one dictionary is used by several tables - I see future issues in
> concurrent dictionary updates. This will for sure
> > affect performance and can cause unpredictable behavior for queries.
>
> You are right. Another reason to return to the idea of dictionary versions.
>
> > Also, I agree with Simon Riggs, using OIDs from the general pool for
> dictionary entries is a bad idea.
>
> Yep, we agreed to stop using OIDs for this, however this was not
> changed in the patch at this point. Please don't hesitate joining the
> effort if you want to. I wouldn't mind taking a short break from this
> patch.
>
> > 3) Is the possible scenario legal - by some means a dictionary does not
> contain some keys for entries? What happens then?
>
> No, we should either forbid removing dictionary entries or check that
> all the existing documents are not using the entries being removed.
>
> > If you have any questions on Pluggable TOAST don't hesitate to ask me
> and on JSONB Toaster you can ask Nikita Glukhov.
>
> Will do! Thanks for working on this and I'm looking forward to the
> next version of the patch for the next round of review.
>
> --
> Best regards,
> Aleksander Alekseev
>
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: [PATCH] Compression dictionaries for JSONB
@ 2022-07-18 12:26 Aleksander Alekseev <[email protected]>
parent: Nikita Malakhov <[email protected]>
1 sibling, 0 replies; 88+ messages in thread
From: Aleksander Alekseev @ 2022-07-18 12:26 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>; +Cc: Nikita Malakhov <[email protected]>; Alvaro Herrera <[email protected]>; Matthias van de Meent <[email protected]>; Zhihong Yu <[email protected]>; Teodor Sigaev <[email protected]>
Hi Nikita,
Thanks for your feedback!
> Aleksander, I've carefully gone over discussion and still have some questions to ask -
>
> 1) Is there any means of measuring overhead of dictionaries over vanilla implementation? IMO it is a must because
> JSON is a widely used functionality. Also, as it was mentioned before, to check the dictionary value must be detoasted;
Not sure what overhead you have in mind. The patch doesn't affect the
vanilla JSONB implementation.
> 2) Storing dictionaries in one table. As I wrote before, this will surely lead to locks and waits while inserting and updating
> dictionaries, and could cause serious performance issues. And vacuuming this table will lead to locks for all tables using
> dictionaries until vacuum is complete;
I believe this is true to some degree. But doesn't the same generally
apply to the rest of catalog tables?
I'm not that concerned about inserting/updating since this is a rare
operation. Vacuuming shouldn't be such a problem unless the user
creates/deletes dictionaries all the time.
Am I missing something?
> 3) JSON documents in production environments could be very complex and use thousands of keys, so creating dictionary
> directly in SQL statement is not very good approach, so it's another reason to have means for creating dictionaries as a
> separate tables and/or passing them as files or so;
Yes, it was proposed to update dictionaries automatically e.g. during
the VACUUM of the table that contains compressed documents. This is
simply out of scope of this particular patch. It was argued that the
manual update should be supported too, which is implemented in this
patch.
> 4) Suggested mechanics, if put on top of the TOAST, could not benefit from knowledge if internal JSON structure, which
> is seen as important drawback in spite of extensive research work done on working with JSON schema (storing, validating,
> etc.), and also it cannot recognize and help to compress duplicated parts of JSON document;
Could you please elaborate on this a bit and/or maybe give an example? ...
> In Pluggable TOAST we suggest that as an improvement compression should be put inside the Toaster as an option,
> thus the Toaster could have maximum benefits from knowledge of data internal structure (and in future use JSON Schema).
... Current implementation doesn't use the knowledge of JSONB format,
that's true. This is because previously we agreed there is no "one
size fits all" compression method, thus several are going to be
supported eventually. The current algorithm was chosen merely as the
one that is going to work good enough for any data type, not just
JSONB. Nothing prevents an alternative compression method from using
the knowledge of JSONB structure.
As, I believe, Matthias pointed out above, only partial decompression
would be a challenge. This is indeed something that would be better to
implement somewhere closer to the TOAST level. Other than that I'm not
sure what you mean.
> 5) A small test issue - if dictionaried' JSON has a key which is equal to OID used in a dictionary for some other key?
Again, I'm having difficulties understanding the case you are
describing. Could you give a specific example?
> For using in special Toaster for JSON datatype compression dictionaries seem to be very valuable addition, but now I
> have to agree that this feature in current state is competing with Pluggable TOAST.
I disagree with the word "competing" here. Again, Matthias had a very
good point about this above.
In short, pluggable TOAST is a low-level internal mechanism, but it
doesn't provide a good interface for the end user and has several open
issues. The most important one IMO is how it is supposed to work with
pluggable AMs in the general case. "Compression dictionaries" have a
good user interface, and the implementation is not that important. The
current implementation uses casts, as the only option available at the
moment. But nothing prevents it from using Pluggable TOAST if this
will produce a cleaner code (I believe it will) and will allow
delivering partial decompression (this is yet to be figured out).
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: [PATCH] Compression dictionaries for JSONB
@ 2022-07-27 07:36 Simon Riggs <[email protected]>
parent: Nikita Malakhov <[email protected]>
1 sibling, 1 reply; 88+ messages in thread
From: Simon Riggs @ 2022-07-27 07:36 UTC (permalink / raw)
To: Nikita Malakhov <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; PostgreSQL Hackers <[email protected]>; Alvaro Herrera <[email protected]>; Matthias van de Meent <[email protected]>; Zhihong Yu <[email protected]>; Teodor Sigaev <[email protected]>
On Sun, 17 Jul 2022 at 19:15, Nikita Malakhov <[email protected]> wrote:
> we suggest that as an improvement compression should be put inside the Toaster as an option,
> thus the Toaster could have maximum benefits from knowledge of data internal structure (and in future use JSON Schema).
Very much agreed.
> For using in special Toaster for JSON datatype compression dictionaries seem to be very valuable addition, but now I
> have to agree that this feature in current state is competing with Pluggable TOAST.
But I don't understand this.
Why does storing a compression dictionary in the catalog prevent that
dictionary from being used within the toaster?
--
Simon Riggs http://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: [PATCH] Compression dictionaries for JSONB
@ 2022-07-27 08:30 Matthias van de Meent <[email protected]>
parent: Simon Riggs <[email protected]>
0 siblings, 1 reply; 88+ messages in thread
From: Matthias van de Meent @ 2022-07-27 08:30 UTC (permalink / raw)
To: Simon Riggs <[email protected]>; +Cc: Nikita Malakhov <[email protected]>; Aleksander Alekseev <[email protected]>; PostgreSQL Hackers <[email protected]>; Alvaro Herrera <[email protected]>; Zhihong Yu <[email protected]>; Teodor Sigaev <[email protected]>
On Wed, 27 Jul 2022 at 09:36, Simon Riggs <[email protected]> wrote:
>
> On Sun, 17 Jul 2022 at 19:15, Nikita Malakhov <[email protected]> wrote:
>
> > For using in special Toaster for JSON datatype compression dictionaries seem to be very valuable addition, but now I
> > have to agree that this feature in current state is competing with Pluggable TOAST.
>
> But I don't understand this.
>
> Why does storing a compression dictionary in the catalog prevent that
> dictionary from being used within the toaster?
The point is not that compression dictionaries in the catalog are bad
- I think it makes a lot of sense - but that the typecast -based usage
of those dictionaries in user tables (like the UI provided by zson)
effectively competes with the toaster: It tries to store the data in a
more compressed manner than the toaster currently can because it has
additional knowledge about the values being toasted.
The main difference between casting and toasting however is that
casting is fairly because it has a significantly higher memory
overhead: both the fully decompressed and the compressed values are
stored in memory at the same time at some point when you cast a value,
while only the decompressed value is stored in full in memory when
(de)toasting.
And, considering that there is an open proposal for extending the
toaster mechanism, I think that it is not specifically efficient to
work with the relatively expensive typecast -based infrastructure if
this dictionary compression can instead be added using the proposed
extensible toasting mechanism at relatively low overhead.
Kind regards,
Matthias van de Meent
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: [PATCH] Compression dictionaries for JSONB
@ 2022-08-01 11:25 Aleksander Alekseev <[email protected]>
parent: Matthias van de Meent <[email protected]>
0 siblings, 1 reply; 88+ messages in thread
From: Aleksander Alekseev @ 2022-08-01 11:25 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>; +Cc: Simon Riggs <[email protected]>; Matthias van de Meent <[email protected]>; Nikita Malakhov <[email protected]>; Alvaro Herrera <[email protected]>; Zhihong Yu <[email protected]>; Teodor Sigaev <[email protected]>
Hi hackers,
> So far we seem to have a consensus to:
>
> 1. Use bytea instead of NameData to store dictionary entries;
>
> 2. Assign monotonically ascending IDs to the entries instead of using
> Oids, as it is done with pg_class.relnatts. In order to do this we
> should either add a corresponding column to pg_type, or add a new
> catalog table, e.g. pg_dict_meta. Personally I don't have a strong
> opinion on what is better. Thoughts?
>
> Both changes should be straightforward to implement and also are a
> good exercise to newcomers.
>
> I invite anyone interested to join this effort as a co-author! (since,
> honestly, rewriting the same feature over and over again alone is
> quite boring :D).
cfbot complained that v5 doesn't apply anymore. Here is the rebased
version of the patch.
> Good point. This was not a problem for ZSON since the dictionary size
> was limited to 2**16 entries, the dictionary was immutable, and the
> dictionaries had versions. For compression dictionaries we removed the
> 2**16 entries limit and also decided to get rid of versions. The idea
> was that you can simply continue adding new entries, but no one
> thought about the fact that this will consume the memory required to
> decompress the document indefinitely.
>
> Maybe we should return to the idea of limited dictionary size and
> versions. Objections?
> [ ...]
> You are right. Another reason to return to the idea of dictionary versions.
Since no one objected so far and/or proposed a better idea I assume
this can be added to the list of TODOs as well.
--
Best regards,
Aleksander Alekseev
Attachments:
[application/octet-stream] v6-0001-Compression-dictionaries-for-JSONB.patch (48.3K, ../../CAJ7c6TOx9NrLpu1SS9p2osY+5-6s=PTm_iFAQCpfgHr9H2hFWA@mail.gmail.com/2-v6-0001-Compression-dictionaries-for-JSONB.patch)
download | inline diff:
From b7a28ae68a4db7b7192d682b31a56937ecc1ee1b Mon Sep 17 00:00:00 2001
From: Aleksander Alekseev <[email protected]>
Date: Tue, 12 Apr 2022 15:58:11 +0300
Subject: [PATCH v6] Compression dictionaries for JSONB
Usage example:
CREATE TYPE mydict AS DITIONARY OF jsonb ('aaa', 'bbb', ...);
SELECT ('{"aaa":"bbb"}' :: mydict) -> 'aaa';
The created type works as a drop-in replacement for JSONB. However,
its internal representation differs. The provided dictionary entries ('aaa',
'bbb', ..) are stored in pg_dict catalog table. When `mydict` sees one of
the entries in the document, it replaces it with the corresponding pg_dict
entry Oid. For more details regarding the compression algorithm and choosen
compromises please see the comments in the code.
In pg_type `mydict` has typtype = TYPTYPE_DICT. It works the same way
as TYPTYPE_BASE with only difference: corresponding `<type>_in`
(pg_type.typinput) and `<another-type>_<type>` (pg_cast.castfunc)
procedures receive the dictionary Oid as a `typmod` argument. This way the
procedures can distinguish `mydict1` from `mydict2` and use the proper
compression dictionary.
(NOTE: dear committer, please bump the catalog version!)
Author: Aleksander Alekseev <[email protected]>
Author: TODO FIXME add your name and email here
Reviewed-by: FIXME
Discussion: https://postgr.es/m/CAJ7c6TOtAB0z1UrksvGTStNE-herK-43bj22%3D5xVBg7S4vr5rQ%40mail.gmail.com
Discussion: https://postgr.es/m/CAJ7c6TPx7N-bVw0dZ1ASCDQKZJHhBYkT6w4HV1LzfS%2BUUTUfmA%40mail.gmail.com
---
src/backend/catalog/Makefile | 3 +-
src/backend/catalog/pg_dict.c | 197 +++++++++
src/backend/commands/typecmds.c | 160 +++++++-
src/backend/executor/functions.c | 1 +
src/backend/parser/gram.y | 8 +
src/backend/parser/parse_coerce.c | 17 +-
src/backend/parser/parse_type.c | 16 +
src/backend/tcop/utility.c | 13 +
src/backend/utils/adt/Makefile | 1 +
src/backend/utils/adt/dictionaries.c | 482 +++++++++++++++++++++++
src/backend/utils/fmgr/funcapi.c | 1 +
src/bin/pg_dump/pg_dump.c | 2 +
src/include/catalog/pg_dict.h | 78 ++++
src/include/catalog/pg_proc.dat | 20 +
src/include/catalog/pg_type.h | 1 +
src/include/commands/typecmds.h | 1 +
src/include/nodes/parsenodes.h | 12 +
src/pl/plpgsql/src/pl_comp.c | 1 +
src/test/regress/expected/dict.out | 47 +++
src/test/regress/expected/oidjoins.out | 1 +
src/test/regress/expected/opr_sanity.out | 14 +-
src/test/regress/parallel_schedule | 2 +-
src/test/regress/sql/dict.sql | 20 +
23 files changed, 1087 insertions(+), 11 deletions(-)
create mode 100644 src/backend/catalog/pg_dict.c
create mode 100644 src/backend/utils/adt/dictionaries.c
create mode 100644 src/include/catalog/pg_dict.h
create mode 100644 src/test/regress/expected/dict.out
create mode 100644 src/test/regress/sql/dict.sql
diff --git a/src/backend/catalog/Makefile b/src/backend/catalog/Makefile
index 89a0221ec9..3b4330148e 100644
--- a/src/backend/catalog/Makefile
+++ b/src/backend/catalog/Makefile
@@ -33,6 +33,7 @@ OBJS = \
pg_conversion.o \
pg_db_role_setting.o \
pg_depend.o \
+ pg_dict.o \
pg_enum.o \
pg_inherits.o \
pg_largeobject.o \
@@ -61,7 +62,7 @@ CATALOG_HEADERS := \
pg_language.h pg_largeobject_metadata.h pg_largeobject.h pg_aggregate.h \
pg_statistic.h pg_statistic_ext.h pg_statistic_ext_data.h \
pg_rewrite.h pg_trigger.h pg_event_trigger.h pg_description.h \
- pg_cast.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \
+ pg_cast.h pg_dict.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \
pg_database.h pg_db_role_setting.h pg_tablespace.h \
pg_authid.h pg_auth_members.h pg_shdepend.h pg_shdescription.h \
pg_ts_config.h pg_ts_config_map.h pg_ts_dict.h \
diff --git a/src/backend/catalog/pg_dict.c b/src/backend/catalog/pg_dict.c
new file mode 100644
index 0000000000..456f3b5177
--- /dev/null
+++ b/src/backend/catalog/pg_dict.c
@@ -0,0 +1,197 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_dict.c
+ * routines to support manipulation of the pg_dict relation
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/catalog/pg_dict.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "access/genam.h"
+#include "access/skey.h"
+#include "access/table.h"
+#include "catalog/catalog.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_dict.h"
+#include "catalog/pg_type.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/memutils.h"
+#include "utils/rel.h"
+
+/*
+ * Creates an entry in pg_dict for each of the supplied values.
+ * vals is a list of String values.
+ */
+void
+DictEntriesCreate(Oid dictTypeOid, List *vals)
+{
+ Relation pg_dict;
+ NameData dictentry;
+ Datum values[Natts_pg_dict];
+ bool nulls[Natts_pg_dict];
+ ListCell *lc;
+ HeapTuple tup;
+
+ if (vals == NIL)
+ {
+ /* The list is empty; do nothing. */
+ return;
+ }
+
+ memset(nulls, false, sizeof(nulls));
+
+ /*
+ * We don't check the list of values for duplicates here. If there are
+ * any, the user will get an unique-index violation.
+ */
+
+ pg_dict = table_open(DictRelationId, RowExclusiveLock);
+ foreach(lc, vals)
+ {
+ Oid oid = GetNewOidWithIndex(pg_dict, DictOidIndexId,
+ Anum_pg_dict_oid);
+ char *entry = strVal(lfirst(lc));
+
+ /*
+ * Entries are stored in a name field, for easier syscache lookup, so
+ * check the length to make sure it's within range.
+ */
+ if (strlen(entry) > (NAMEDATALEN - 1))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_NAME),
+ errmsg("invalid dict entry \"%s\"", entry),
+ errdetail("Entries must be %d bytes or less.",
+ NAMEDATALEN - 1)));
+
+ /* AALEKSEEV TODO use bytea instead */
+ namestrcpy(&dictentry, entry);
+
+ values[Anum_pg_dict_oid - 1] = ObjectIdGetDatum(oid);
+ values[Anum_pg_dict_dicttypid - 1] = ObjectIdGetDatum(dictTypeOid);
+ values[Anum_pg_dict_dictentry - 1] = NameGetDatum(&dictentry);
+
+ tup = heap_form_tuple(RelationGetDescr(pg_dict), values, nulls);
+
+ CatalogTupleInsert(pg_dict, tup);
+ heap_freetuple(tup);
+ }
+
+ /* clean up */
+ table_close(pg_dict, RowExclusiveLock);
+}
+
+/*
+ * Returns all the entries for the dictinary with given Oid. Entries are sorted
+ * by dictentry. Note that shorter entries are considered smaller, i.e. 'abc'
+ * goes before 'abcdef'. The memory is allocated in a child memory context of
+ * the caller's memory context. It can be freed with DictEntriesFree().
+ *
+ * If there are no entries a valid but empty dictionary is returned.
+ */
+Dictionary
+DictEntriesRead(Oid dictTypeOid)
+{
+ Relation pg_dict;
+ ScanKeyData key[1];
+ SysScanDesc scan;
+ HeapTuple tup;
+ uint32 entries_allocated;
+ Dictionary dict;
+ MemoryContext myctx, oldctx;
+
+ myctx = AllocSetContextCreate(CurrentMemoryContext, "DictEntriesCtx", ALLOCSET_DEFAULT_SIZES);
+ oldctx = MemoryContextSwitchTo(myctx);
+
+ dict = (Dictionary)palloc(sizeof(DictionaryData));
+
+ entries_allocated = 8;
+ dict->nentries = 0;
+ dict->entries = (DictEntry*)palloc(entries_allocated*sizeof(DictEntry));
+
+ pg_dict = table_open(DictRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_dict_dicttypid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(dictTypeOid));
+
+ scan = systable_beginscan(pg_dict, DictTypIdEntryIndexId, true,
+ NULL, 1, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ if(dict->nentries == entries_allocated)
+ {
+ entries_allocated = entries_allocated * 2;
+ dict->entries = (DictEntry*)repalloc(dict->entries, entries_allocated*sizeof(DictEntry));
+ }
+
+ dict->entries[dict->nentries].oid = ((Form_pg_dict) GETSTRUCT(tup))->oid;
+ /*
+ * This is arguably not the fastest way to determine the length of
+ * the entry. Alternatively, we could store a precalculated value in
+ * the catalog. However, usually it's a good idea to keep things simple
+ * until somebody discovers a bottleneck in this exact place and
+ * proposes a concrete fix.
+ */
+ dict->entries[dict->nentries].length = (uint32)strlen(((Form_pg_dict) GETSTRUCT(tup))->dictentry.data);
+ dict->entries[dict->nentries].data = palloc(dict->entries[dict->nentries].length);
+ memcpy(dict->entries[dict->nentries].data, ((Form_pg_dict) GETSTRUCT(tup))->dictentry.data, dict->entries[dict->nentries].length);
+ dict->nentries++;
+ }
+
+ systable_endscan(scan);
+ table_close(pg_dict, RowExclusiveLock);
+
+ MemoryContextSwitchTo(oldctx);
+
+ return dict;
+}
+
+/*
+ * Frees the memory allocated for the dictionary.
+ */
+void
+DictEntriesFree(Dictionary dict)
+{
+ MemoryContextDelete(GetMemoryChunkContext(dict));
+}
+
+/*
+ * Deletes all the entries for the dictinary with given Oid.
+ */
+void
+DictEntriesDelete(Oid dictTypeOid)
+{
+ Relation pg_dict;
+ ScanKeyData key[1];
+ SysScanDesc scan;
+ HeapTuple tup;
+
+ pg_dict = table_open(DictRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_dict_dicttypid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(dictTypeOid));
+
+ scan = systable_beginscan(pg_dict, DictTypIdEntryIndexId, true,
+ NULL, 1, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ CatalogTupleDelete(pg_dict, &tup->t_self);
+ }
+
+ systable_endscan(scan);
+ table_close(pg_dict, RowExclusiveLock);
+}
diff --git a/src/backend/commands/typecmds.c b/src/backend/commands/typecmds.c
index d0d87a1184..cd3d00de3d 100644
--- a/src/backend/commands/typecmds.c
+++ b/src/backend/commands/typecmds.c
@@ -46,6 +46,7 @@
#include "catalog/pg_collation.h"
#include "catalog/pg_constraint.h"
#include "catalog/pg_depend.h"
+#include "catalog/pg_dict.h"
#include "catalog/pg_enum.h"
#include "catalog/pg_language.h"
#include "catalog/pg_namespace.h"
@@ -679,6 +680,14 @@ RemoveTypeById(Oid typeOid)
if (((Form_pg_type) GETSTRUCT(tup))->typtype == TYPTYPE_RANGE)
RangeDelete(typeOid);
+ /*
+ * If it is a dictionary type, delete the pg_dict entries too; we don't
+ * bother with making a dependency entry for that, so it has to be done
+ * "by hand" here.
+ */
+ if (((Form_pg_type) GETSTRUCT(tup))->typtype == TYPTYPE_DICT)
+ DictEntriesDelete(typeOid);
+
ReleaseSysCache(tup);
table_close(relation, RowExclusiveLock);
@@ -763,8 +772,8 @@ DefineDomain(CreateDomainStmt *stmt)
/*
* Base type must be a plain base type, a composite type, another domain,
- * an enum or a range type. Domains over pseudotypes would create a
- * security hole. (It would be shorter to code this to just check for
+ * a dict, an enum or a range type. Domains over pseudotypes would create
+ * a security hole. (It would be shorter to code this to just check for
* pseudotypes; but it seems safer to call out the specific typtypes that
* are supported, rather than assume that all future typtypes would be
* automatically supported.)
@@ -772,6 +781,7 @@ DefineDomain(CreateDomainStmt *stmt)
typtype = baseType->typtype;
if (typtype != TYPTYPE_BASE &&
typtype != TYPTYPE_COMPOSITE &&
+ typtype != TYPTYPE_DICT &&
typtype != TYPTYPE_DOMAIN &&
typtype != TYPTYPE_ENUM &&
typtype != TYPTYPE_RANGE &&
@@ -1129,6 +1139,152 @@ DefineDomain(CreateDomainStmt *stmt)
}
+/*
+ * DefineDictionary
+ * Registers a new dictionary.
+ */
+ObjectAddress
+DefineDictionary(CreateDictionaryStmt *stmt)
+{
+ char *dictName;
+ char *dictBaseTypeName;
+ char *dictArrayName;
+ Oid dictNamespace;
+ AclResult aclresult;
+ Oid old_type_oid;
+ Oid dictArrayOid;
+ ObjectAddress dictTypeAddr;
+
+ Assert(list_length(stmt->baseTypeName) == 1);
+ dictBaseTypeName = strVal(linitial(stmt->baseTypeName));
+ if(pg_strcasecmp(dictBaseTypeName, "jsonb") != 0)
+ ereport(ERROR,
+ (
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("Dictionary types currently can be used only with JSONB")
+ )
+ );
+
+ /* Convert list of names to a name and namespace */
+ dictNamespace = QualifiedNameGetCreationNamespace(stmt->typeName,
+ &dictName);
+
+ /* Check we have creation rights in target namespace */
+ aclresult = pg_namespace_aclcheck(dictNamespace, GetUserId(), ACL_CREATE);
+ if (aclresult != ACLCHECK_OK)
+ aclcheck_error(aclresult, OBJECT_SCHEMA,
+ get_namespace_name(dictNamespace));
+
+ /*
+ * Check for collision with an existing type name. If there is one and
+ * it's an autogenerated array, we can rename it out of the way.
+ */
+ old_type_oid = GetSysCacheOid2(TYPENAMENSP, Anum_pg_type_oid,
+ CStringGetDatum(dictName),
+ ObjectIdGetDatum(dictNamespace));
+ if (OidIsValid(old_type_oid))
+ {
+ if (!moveArrayTypeName(old_type_oid, dictName, dictNamespace))
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("type \"%s\" already exists", dictName)));
+ }
+
+ /* Allocate OID for array type */
+ dictArrayOid = AssignTypeArrayOid();
+
+ /* Create the pg_type entry */
+ dictTypeAddr =
+ TypeCreate(InvalidOid, /* no predetermined type OID */
+ dictName, /* type name */
+ dictNamespace, /* namespace */
+ InvalidOid, /* relation oid (n/a here) */
+ 0, /* relation kind (ditto) */
+ GetUserId(), /* owner's ID */
+ -1, /* internal size: varlena, as for JSONB */
+ TYPTYPE_DICT, /* type-type: dictionary */
+ TYPCATEGORY_USER, /* type-category: user, as for JSONB */
+ false, /* dict types are never preferred */
+ DEFAULT_TYPDELIM, /* array element delimiter */
+ F_DICTIONARY_IN, /* input procedure */
+ F_DICTIONARY_OUT, /* output procedure */
+ InvalidOid, /* receive procedure: none */
+ InvalidOid, /* send procedure: none */
+ InvalidOid, /* typmodin procedure - none */
+ InvalidOid, /* typmodout procedure - none */
+ InvalidOid, /* analyze procedure - default */
+ InvalidOid, /* subscript procedure - none */
+ InvalidOid, /* element type ID */
+ false, /* this is not an array type */
+ dictArrayOid, /* array type we are about to create */
+ InvalidOid, /* base type ID (only for domains) */
+ NULL, /* never a default type value */
+ NULL, /* binary default isn't sent either */
+ false, /* passed by value: same as for JSONB */
+ TYPALIGN_INT, /* int alignment */
+ TYPSTORAGE_EXTENDED, /* TOAST strategy: fully toastable, as
+ * JSONB */
+ -1, /* typMod (Domains only) */
+ 0, /* Array dimensions of typbasetype */
+ false, /* Type NOT NULL */
+ InvalidOid); /* type's collation */
+
+ /*
+ * Create the array type that goes with it.
+ */
+ dictArrayName = makeArrayTypeName(dictName, dictNamespace);
+
+ TypeCreate(dictArrayOid, /* force assignment of this type OID */
+ dictArrayName, /* type name */
+ dictNamespace, /* namespace */
+ InvalidOid, /* relation oid (n/a here) */
+ 0, /* relation kind (ditto) */
+ GetUserId(), /* owner's ID */
+ -1, /* internal size (always varlena) */
+ TYPTYPE_BASE, /* type-type (base type) */
+ TYPCATEGORY_ARRAY, /* type-category (array) */
+ false, /* array types are never preferred */
+ DEFAULT_TYPDELIM, /* array element delimiter */
+ F_ARRAY_IN, /* input procedure */
+ F_ARRAY_OUT, /* output procedure */
+ F_ARRAY_RECV, /* receive procedure */
+ F_ARRAY_SEND, /* send procedure */
+ InvalidOid, /* typmodin procedure - none */
+ InvalidOid, /* typmodout procedure - none */
+ F_ARRAY_TYPANALYZE, /* analyze procedure */
+ F_ARRAY_SUBSCRIPT_HANDLER, /* array subscript procedure */
+ dictTypeAddr.objectId, /* element type ID */
+ true, /* yes this is an array type */
+ InvalidOid, /* no further array type */
+ InvalidOid, /* base type ID */
+ NULL, /* never a default type value */
+ NULL, /* binary default isn't sent either */
+ false, /* passed by value: same as for JSONB */
+ TYPALIGN_INT, /* enums have int align, so do their arrays */
+ TYPSTORAGE_EXTENDED, /* ARRAY is always toastable */
+ -1, /* typMod (Domains only) */
+ 0, /* Array dimensions of typbasetype */
+ false, /* Type NOT NULL */
+ InvalidOid); /* type's collation */
+
+ pfree(dictArrayName);
+
+ /* Enter the dict's entries into pg_dict */
+ DictEntriesCreate(dictTypeAddr.objectId, stmt->vals);
+
+ /* Create casts to and from JSONB */
+ CastCreate(JSONBOID, dictTypeAddr.objectId, F_JSONB_DICTIONARY, 'a', 'f', DEPENDENCY_INTERNAL);
+ CastCreate(dictTypeAddr.objectId, JSONBOID, F_DICTIONARY_JSONB, 'i', 'f', DEPENDENCY_INTERNAL);
+
+ /*
+ * Create explicit cast to bytea. This is convenient for debugging purposes.
+ * Casting bytea to a dictionary type is dangerous and thus not supported.
+ */
+ CastCreate(dictTypeAddr.objectId, BYTEAOID, F_DICTIONARY_BYTEA, 'e', 'f', DEPENDENCY_INTERNAL);
+
+ return dictTypeAddr;
+}
+
/*
* DefineEnum
* Registers a new enum.
diff --git a/src/backend/executor/functions.c b/src/backend/executor/functions.c
index 076226868f..fae9f5dd5a 100644
--- a/src/backend/executor/functions.c
+++ b/src/backend/executor/functions.c
@@ -1706,6 +1706,7 @@ check_sql_fn_retval(List *queryTreeLists,
if (fn_typtype == TYPTYPE_BASE ||
fn_typtype == TYPTYPE_DOMAIN ||
+ fn_typtype == TYPTYPE_DICT ||
fn_typtype == TYPTYPE_ENUM ||
fn_typtype == TYPTYPE_RANGE ||
fn_typtype == TYPTYPE_MULTIRANGE)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index f9037761f9..b492cc8a47 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -6267,6 +6267,14 @@ DefineStmt:
n->coldeflist = $6;
$$ = (Node *) n;
}
+ | CREATE TYPE_P any_name AS DICTIONARY OF any_name '(' opt_enum_val_list ')'
+ {
+ CreateDictionaryStmt *n = makeNode(CreateDictionaryStmt);
+ n->typeName = $3;
+ n->baseTypeName = $7;
+ n->vals = $9;
+ $$ = (Node *)n;
+ }
| CREATE TYPE_P any_name AS ENUM_P '(' opt_enum_val_list ')'
{
CreateEnumStmt *n = makeNode(CreateEnumStmt);
diff --git a/src/backend/parser/parse_coerce.c b/src/backend/parser/parse_coerce.c
index c4e958e4aa..e489dd9421 100644
--- a/src/backend/parser/parse_coerce.c
+++ b/src/backend/parser/parse_coerce.c
@@ -279,7 +279,22 @@ coerce_type(ParseState *pstate, Node *node,
if (baseTypeId == INTERVALOID)
inputTypeMod = baseTypeMod;
else
- inputTypeMod = -1;
+ {
+ if((int32)targetTypeId == targetTypeMod)
+ {
+ /*
+ * For dictionaries internally we set typmod to a dictionary
+ * OID. This allows dictionary_in() and jsonb_dictionary() to
+ * distinguish one dictionary from another.
+ *
+ * It also allows us to recognize that the target type is
+ * a dictionary here and pass a correct typmod.
+ */
+ inputTypeMod = targetTypeMod;
+ }
+ else
+ inputTypeMod = -1;
+ }
baseType = typeidType(baseTypeId);
diff --git a/src/backend/parser/parse_type.c b/src/backend/parser/parse_type.c
index f7ad689459..2188002535 100644
--- a/src/backend/parser/parse_type.c
+++ b/src/backend/parser/parse_type.c
@@ -259,6 +259,10 @@ LookupTypeNameOid(ParseState *pstate, const TypeName *typeName, bool missing_ok)
* This is equivalent to LookupTypeName, except that this will report
* a suitable error message if the type cannot be found or is not defined.
* Callers of this can therefore assume the result is a fully valid type.
+ *
+ * For a dictionary type it's OID is returned as a typmod. This is used by
+ * dictionary_in() and jsonb_dictionary() to distinguish one dictionary from
+ * another.
*/
Type
typenameType(ParseState *pstate, const TypeName *typeName, int32 *typmod_p)
@@ -305,6 +309,10 @@ typenameTypeId(ParseState *pstate, const TypeName *typeName)
*
* This is equivalent to typenameType, but we only hand back the type OID
* and typmod, not the syscache entry.
+ *
+ * For a dictionary type it's OID is returned as a typmod. This is used by
+ * dictionary_in() and jsonb_dictionary() to distinguish one dictionary from
+ * another.
*/
void
typenameTypeIdAndMod(ParseState *pstate, const TypeName *typeName,
@@ -327,6 +335,10 @@ typenameTypeIdAndMod(ParseState *pstate, const TypeName *typeName,
* looked up, and is passed as "typ".
*
* pstate is only used for error location info, and may be NULL.
+ *
+ * For a dictionary type it's OID is returned as a typmod. This is used by
+ * dictionary_in() and jsonb_dictionary() to distinguish one dictionary from
+ * another.
*/
static int32
typenameTypeMod(ParseState *pstate, const TypeName *typeName, Type typ)
@@ -339,6 +351,10 @@ typenameTypeMod(ParseState *pstate, const TypeName *typeName, Type typ)
ArrayType *arrtypmod;
ParseCallbackState pcbstate;
+ /* For a dictionary return it's OID. */
+ if (((Form_pg_type) GETSTRUCT(typ))->typtype == TYPTYPE_DICT)
+ return (int32)((Form_pg_type) GETSTRUCT(typ))->oid;
+
/* Return prespecified typmod if no typmod expressions */
if (typeName->typmods == NIL)
return typeName->typemod;
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index 6b0a865262..40cb80eb4a 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -174,6 +174,7 @@ ClassifyUtilityCommandAsReadOnly(Node *parsetree)
case T_CreateCastStmt:
case T_CreateConversionStmt:
case T_CreateDomainStmt:
+ case T_CreateDictionaryStmt:
case T_CreateEnumStmt:
case T_CreateEventTrigStmt:
case T_CreateExtensionStmt:
@@ -1621,6 +1622,10 @@ ProcessUtilitySlow(ParseState *pstate,
}
break;
+ case T_CreateDictionaryStmt: /* CREATE TYPE AS DICTIONARY OF */
+ address = DefineDictionary((CreateDictionaryStmt *) parsetree);
+ break;
+
case T_CreateEnumStmt: /* CREATE TYPE AS ENUM */
address = DefineEnum((CreateEnumStmt *) parsetree);
break;
@@ -2772,6 +2777,10 @@ CreateCommandTag(Node *parsetree)
tag = CMDTAG_CREATE_TYPE;
break;
+ case T_CreateDictionaryStmt:
+ tag = CMDTAG_CREATE_TYPE;
+ break;
+
case T_CreateEnumStmt:
tag = CMDTAG_CREATE_TYPE;
break;
@@ -3419,6 +3428,10 @@ GetCommandLogLevel(Node *parsetree)
lev = LOGSTMT_DDL;
break;
+ case T_CreateDictionaryStmt:
+ lev = LOGSTMT_DDL;
+ break;
+
case T_CreateEnumStmt:
lev = LOGSTMT_DDL;
break;
diff --git a/src/backend/utils/adt/Makefile b/src/backend/utils/adt/Makefile
index 7c722ea2ce..a7ec3d89f3 100644
--- a/src/backend/utils/adt/Makefile
+++ b/src/backend/utils/adt/Makefile
@@ -30,6 +30,7 @@ OBJS = \
datetime.o \
datum.o \
dbsize.o \
+ dictionaries.o \
domains.o \
encode.o \
enum.o \
diff --git a/src/backend/utils/adt/dictionaries.c b/src/backend/utils/adt/dictionaries.c
new file mode 100644
index 0000000000..2bb46471a4
--- /dev/null
+++ b/src/backend/utils/adt/dictionaries.c
@@ -0,0 +1,482 @@
+/*-------------------------------------------------------------------------
+ *
+ * dictionaries.c
+ * Conversion functions for dictionary types.
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/adt/dictionaries.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "catalog/pg_dict.h"
+#include "utils/fmgrprotos.h"
+#include "utils/jsonb.h"
+
+/*
+ * When compressing a data we treat it as a BLOB, in other words we don't
+ * assume anything regarding its internal representation. This is not
+ * necessarily the best and/or the only possible approach. However, it is
+ * universal and can be reused for JSONB, TEXT, XML and other types. Alternative
+ * compression methods can be introduced in the future, so that the user will
+ * be able to choose the best one for the task.
+ *
+ * The compressed data is stored in the following format:
+ *
+ * (struct varlena)
+ * dictionary_id [uint32]
+ * decompressed_size [uint32]
+ * algorithm_version [uint8]
+ *
+ * (repeated) {
+ * number of bytes to copy as-is [uint8]
+ * ... bytes to copy as-is ...
+ * dictionary entry id, or 0 to skip [uint32]
+ * }
+ *
+ * Compressed data is a variable-length type and thus has the 'struct varlena'
+ * header. The code below doesn't consider it being a part of the payload
+ * (e.g. see the DICT_COMP_HEADER_SIZE definition).
+ *
+ * Storing dictionary_id may seem redundant, but without it dictionary_out()
+ * and dictionary_jsonb() have no way to distinguish one dictionary type from
+ * another. dictionary_in() and jsonb_dictionary() get the dictionary id through
+ * the 'typemod' argument.
+ *
+ * Currently algorithm_version is always 0. In the future it will allow us to
+ * introduce new features (e.g. usage of varints) and/or lazily migrate the
+ * data to other compression methods.
+ *
+ * Compression and decompression are implemented based on a binary search over
+ * DictEntry[] array (see pg_dict.h). For compression the array is sorted by
+ * dictentries and for decompression - by oid's.
+ *
+ */
+
+/* Size of the header in the compressed data */
+#define DICT_COMP_HEADER_SIZE (sizeof(uint32)*2 + sizeof(uint8))
+
+/* Extracts dictionary_id from the compressed data */
+#define DICT_COMP_DICTIONARY_ID(hdrp) \
+ (*(uint32*)hdrp)
+
+/* Extracts decompressed_size from the compressed data */
+#define DICT_COMP_DECOMPRESSED_SIZE(hdrp) \
+ (*(uint32*)((uint8*)hdrp + sizeof(uint32)))
+
+/* Extracts algorithm_version from the compressed data */
+#define DICT_COMP_ALGORITHM_VERSION(hdrp) \
+ (*(uint8*)((uint8*)hdrp + sizeof(uint32)*2))
+
+/* Current algorithm_version */
+#define DICT_COMP_CURRENT_ALGORITHM_VERSION 0
+
+/*
+ * bsearch_arg() callback for finding DictEntry by oid.
+ */
+static int
+find_by_oid_cb(const void *key, const void *current, void *arg)
+{
+ /* Note that oids are unsigned, so we should be careful here */
+ if (*(Oid *) key < ((DictEntry *) current)->oid)
+ return -1;
+ else if (*(Oid *) key == ((DictEntry *) current)->oid)
+ return 0; /* found! */
+ else
+ return 1;
+}
+
+/*
+ * qsort() callback for sorting DictEntry[] by oids.
+ */
+static int
+sort_by_oids_cb(const void *left, const void *right)
+{
+ /* Note that oids are unsigned, so we should be careful here */
+ if (((DictEntry *) left)->oid < ((DictEntry *) right)->oid)
+ return -1;
+
+ /* Oids are unique so this callback will never return 0 */
+ return 1;
+}
+
+/*
+ * Finds a DictEntry which dictentry field matches *data and returns its Oid.
+ * If there are several matching entries, the largest is returned. The length
+ * of the found entry is written to *found_length on success. On failure
+ * InvalidOid is returned and found_length is zeroed.
+ *
+ * The implementation is similar to bsearch_arg(). The procedure can't be used
+ * directly because we are looking not by the exact match. We could generalize
+ * this case but the signature of the function becomes so complicated that it
+ * doesn't seem to worth the effort.
+ */
+static Oid
+compress_find_oid(Dictionary dict, const uint8 *data, Size data_size, Size *found_length)
+{
+ int res;
+ int32 left = 0;
+ int32 right = dict->nentries - 1;
+ Size best_length = 0;
+ Oid best_match = InvalidOid;
+
+ while (left <= right)
+ {
+ int32 current = (left + right) / 2;
+ Size nbytes = (Size)dict->entries[current].length;
+
+ if (nbytes > data_size)
+ {
+ /* current can be less or greater depending on the prefix */
+ res = memcmp(dict->entries[current].data, data, data_size);
+
+ /* if prefixes match, current is greater */
+ if (res == 0)
+ res = 1;
+ }
+ else
+ res = memcmp(dict->entries[current].data, data, nbytes);
+
+ if (res == 0) /* match found */
+ {
+ best_length = nbytes;
+ best_match = dict->entries[current].oid;
+
+ if (nbytes == data_size)
+ break;
+
+ /* maybe there is a larger match */
+ left = current + 1;
+ }
+ else if (res < 0) /* current is less */
+ left = current + 1;
+ else /* current is greater */
+ right = current - 1;
+ }
+
+ *found_length = best_length;
+ return best_match;
+}
+
+/*
+ * Finds a DictEntry by Oid using a binary search. The dictionary should be
+ * sorted by oids before the call. Returns NULL if nothing was found.
+ */
+static DictEntry *
+decompress_find_dictentry(Dictionary dict, Oid oid)
+{
+ return (DictEntry *) bsearch_arg(&oid, dict->entries, dict->nentries, sizeof(DictEntry), find_by_oid_cb, NULL);
+}
+
+/*
+ * Estimates the worst-case compressed size of the data of given size.
+ * Worst-case scenario happens when the dictionary consists of single-character
+ * entries. In this case every byte will be encoded as 6 bytes:
+ * 0x00, (0 bytes to copy as-is), 4 bytes of the entry Oid
+ *
+ * This procedure doesn't account for the header size.
+ *
+ * AALEKSEEV FIXME don't use dictionary entries shorter than 5 bytes
+ */
+static Size
+worst_case_compressed_size(Size insize)
+{
+ return insize * 6;
+}
+
+/*
+ * Compresses the data using the provided dictionary. The dictionary should
+ * be sorted by dictentries before the call. Output buffer should be at least
+ * worst_case_compressed_size(src_size) bytes in size.
+ */
+static void
+compress(Dictionary dict,
+ const void *src_data_, Size src_size,
+ void *encoded_data_, Size *pencoded_size)
+{
+ Size nbytes;
+ Size inoffset;
+ Size outskipoffset = 0;
+ Size outoffset = 1;
+ uint8 skipbytes = 0;
+ const uint8 *src_data = src_data_;
+ uint8 *encoded_data = ((uint8 *) encoded_data_);
+
+ for (inoffset = 0; inoffset < src_size;)
+ {
+ Oid code = compress_find_oid(dict, &(src_data[inoffset]),
+ src_size - inoffset, &nbytes);
+
+ if (code == InvalidOid)
+ {
+ skipbytes++;
+ encoded_data[outoffset] = src_data[inoffset];
+ outoffset++;
+ inoffset++;
+
+ if (skipbytes == 255)
+ {
+ encoded_data[outskipoffset] = skipbytes;
+ encoded_data[outoffset++] = 0; /* InvalidOid */
+ encoded_data[outoffset++] = 0;
+ encoded_data[outoffset++] = 0;
+ encoded_data[outoffset++] = 0;
+ outskipoffset = outoffset++;
+ skipbytes = 0;
+ }
+ }
+ else
+ {
+ encoded_data[outskipoffset] = skipbytes;
+ encoded_data[outoffset++] = (code >> 24) & 0xFF;
+ encoded_data[outoffset++] = (code >> 16) & 0xFF;
+ encoded_data[outoffset++] = (code >> 8) & 0xFF;
+ encoded_data[outoffset++] = code & 0xFF;
+ outskipoffset = outoffset++;
+ skipbytes = 0;
+ inoffset += nbytes;
+ }
+ }
+
+ /* Double check that we didn't write out of buffer */
+ Assert(outoffset < worst_case_compressed_size(src_size));
+
+ encoded_data[outskipoffset] = skipbytes;
+ *pencoded_size = outoffset;
+}
+
+/*
+ * Report an internal error in decompress() procedure below.
+ * Under normal circumstances this should never happen.
+ */
+static void
+decompress_error()
+{
+ ereport(ERROR,
+ (
+ errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("Unable to decompress a dictionary type"),
+ errdetail("The compressed data seems to be corrupted"),
+ errhint("Please report the steps to reproduce the issue to pgsql-bugs@")
+ ));
+}
+
+/*
+ * Decompresses the data using the provided dictionary. The dictionary should
+ * be sorted by oids before the call.
+ */
+static void
+decompress(Dictionary dict,
+ const void *encoded_data_, Size encoded_size,
+ void *decoded_data_, Size decoded_size)
+{
+ Size inoffset = 0;
+ Size outoffset = 0;
+ Oid code;
+ uint8 skipbytes;
+ const uint8 *encoded_data = ((uint8 *) encoded_data_);
+ uint8 *decoded_data = decoded_data_;
+
+ for (inoffset = 0; inoffset < encoded_size;)
+ {
+ skipbytes = encoded_data[inoffset++];
+
+ if (skipbytes > decoded_size - outoffset)
+ decompress_error();
+
+ if (skipbytes > encoded_size - inoffset)
+ decompress_error();
+
+ memcpy(
+ &(decoded_data[outoffset]),
+ &(encoded_data[inoffset]),
+ skipbytes
+ );
+
+ outoffset += skipbytes;
+ inoffset += skipbytes;
+
+ if ((encoded_size == inoffset) && (decoded_size == outoffset))
+ break; /* end of input - its OK */
+
+ if (encoded_size - inoffset < 4)
+ decompress_error();
+
+ code = (Oid) encoded_data[inoffset++];
+ code = (code << 8) | (Oid) encoded_data[inoffset++];
+ code = (code << 8) | (Oid) encoded_data[inoffset++];
+ code = (code << 8) | (Oid) encoded_data[inoffset++];
+
+ if (code != InvalidOid)
+ {
+ Size entrylen;
+ DictEntry *entry = decompress_find_dictentry(dict, code);
+
+ if (entry == NULL)
+ decompress_error();
+
+ Assert(entry->oid == code);
+ entrylen = (Size)entry->length;
+
+ if (entrylen > decoded_size - outoffset)
+ decompress_error();
+
+ memcpy(
+ &(decoded_data[outoffset]),
+ entry->data,
+ entrylen
+ );
+
+ outoffset += entrylen;
+ }
+ }
+
+ Assert(decoded_size == outoffset);
+}
+
+/*
+ * Converts a cstring to a dictionary.
+ */
+Datum
+dictionary_in(PG_FUNCTION_ARGS)
+{
+ const char *instr = PG_GETARG_CSTRING(0);
+#ifdef NOT_USED
+ Oid typelem = PG_GETARG_OID(1);
+#endif
+ int32 typmod = PG_GETARG_INT32(2);
+ Jsonb *jsonb;
+ bytea *dict;
+
+ Assert(typmod != -1);
+
+ jsonb = DatumGetJsonbP(DirectFunctionCall1(jsonb_in, CStringGetDatum(instr)));
+ dict = DatumGetByteaP(DirectFunctionCall2(jsonb_dictionary, JsonbPGetDatum(jsonb), Int32GetDatum(typmod)));
+
+ PG_RETURN_BYTEA_P(dict);
+}
+
+/*
+ * Converts a dictionary to a cstring.
+ */
+Datum
+dictionary_out(PG_FUNCTION_ARGS)
+{
+ bytea *dict = PG_GETARG_BYTEA_P(0);
+ Jsonb *jsonb = DatumGetJsonbP(DirectFunctionCall1(dictionary_jsonb, PointerGetDatum(dict)));
+ const char *outstr = DatumGetCString(DirectFunctionCall1(jsonb_out, JsonbPGetDatum(jsonb)));
+
+ PG_RETURN_CSTRING(outstr);
+}
+
+/*
+ * Coverts JSONB to a dictionary type.
+ *
+ * AALEKSEEV FIXME: if the compressed document ends up being larger than
+ * the original one write a corresponding flag and copy all the data as-is.
+ */
+Datum
+jsonb_dictionary(PG_FUNCTION_ARGS)
+{
+ Dictionary dict;
+ Jsonb *jsonb = PG_GETARG_JSONB_P(0);
+ int32 typmod = PG_GETARG_INT32(1);
+ uint8 *jsonb_data = (uint8 *) VARDATA(jsonb);
+ Size jsonb_data_size = VARSIZE(jsonb) - VARHDRSZ;
+ uint8 *encoded_buff,
+ *encoded_header,
+ *encoded_data;
+ Size encoded_size,
+ encoded_buff_size;
+
+ Assert(typmod != -1);
+
+ dict = DictEntriesRead(typmod);
+ qsort((void *) dict->entries, dict->nentries, sizeof(DictEntry), sort_by_oids_cb);
+
+ encoded_buff_size = VARHDRSZ + DICT_COMP_HEADER_SIZE + worst_case_compressed_size(jsonb_data_size);
+ encoded_buff = palloc(encoded_buff_size);
+ encoded_header = (uint8 *) VARDATA(encoded_buff);
+ encoded_data = encoded_header + DICT_COMP_HEADER_SIZE;
+
+ DICT_COMP_DICTIONARY_ID(encoded_header) = typmod;
+ DICT_COMP_DECOMPRESSED_SIZE(encoded_header) = jsonb_data_size;
+ DICT_COMP_ALGORITHM_VERSION(encoded_header) = DICT_COMP_CURRENT_ALGORITHM_VERSION;
+
+ encoded_size = encoded_buff_size - VARHDRSZ - DICT_COMP_HEADER_SIZE;
+
+ compress(dict, jsonb_data, jsonb_data_size,
+ encoded_data, &encoded_size);
+
+ encoded_size += VARHDRSZ + DICT_COMP_HEADER_SIZE;
+
+ encoded_buff = repalloc(encoded_buff, encoded_size);
+ SET_VARSIZE(encoded_buff, encoded_size);
+
+ DictEntriesFree(dict);
+
+ PG_RETURN_BYTEA_P(encoded_buff);
+}
+
+/*
+ * Converts a dictionary type to JSONB.
+ */
+Datum
+dictionary_jsonb(PG_FUNCTION_ARGS)
+{
+ bytea *encoded_buff = PG_GETARG_BYTEA_P(0);
+ uint8 *encoded_header = (uint8 *) VARDATA(encoded_buff);
+ uint8 *encoded_data = encoded_header + DICT_COMP_HEADER_SIZE;
+ Size encoded_size = VARSIZE(encoded_buff) - VARHDRSZ - DICT_COMP_HEADER_SIZE;
+ int alg_version = DICT_COMP_ALGORITHM_VERSION(encoded_header);
+ Oid dictOid; /* cannot read until algorithm version is
+ * checked */
+ uint32 decoded_size; /* cannot read until algorithm version is
+ * checked */
+ Jsonb *jsonb;
+ uint8 *jsonb_data;
+ Dictionary dict;
+
+ if (alg_version > DICT_COMP_CURRENT_ALGORITHM_VERSION)
+ ereport(ERROR,
+ (
+ errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("Unsupported compression algorithm version"),
+ errdetail("Saved algorithm version is %d, current version is %d",
+ alg_version, DICT_COMP_CURRENT_ALGORITHM_VERSION),
+ errhint("The data is either corrupted or imported from "
+ "the future version of PostgreSQL")
+ ));
+
+ dictOid = DICT_COMP_DICTIONARY_ID(encoded_header);
+ decoded_size = DICT_COMP_DECOMPRESSED_SIZE(encoded_header);
+
+ dict = DictEntriesRead(dictOid);
+
+ jsonb = palloc(decoded_size + VARHDRSZ);
+ jsonb_data = (uint8 *) VARDATA(jsonb);
+
+ decompress(dict, encoded_data, encoded_size, jsonb_data, decoded_size);
+
+ decoded_size += VARHDRSZ;
+ SET_VARSIZE(jsonb, decoded_size);
+
+ DictEntriesFree(dict);
+
+ PG_RETURN_JSONB_P(jsonb);
+}
+
+/*
+ * Converts a dictionary type to bytea.
+ */
+Datum
+dictionary_bytea(PG_FUNCTION_ARGS)
+{
+ bytea *compressed_data = PG_GETARG_BYTEA_P(0);
+ PG_RETURN_BYTEA_P(compressed_data);
+}
\ No newline at end of file
diff --git a/src/backend/utils/fmgr/funcapi.c b/src/backend/utils/fmgr/funcapi.c
index a1fe50ffca..e20c45f485 100644
--- a/src/backend/utils/fmgr/funcapi.c
+++ b/src/backend/utils/fmgr/funcapi.c
@@ -1292,6 +1292,7 @@ get_type_func_class(Oid typid, Oid *base_typeid)
case TYPTYPE_COMPOSITE:
return TYPEFUNC_COMPOSITE;
case TYPTYPE_BASE:
+ case TYPTYPE_DICT:
case TYPTYPE_ENUM:
case TYPTYPE_RANGE:
case TYPTYPE_MULTIRANGE:
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index da6605175a..0b60d47b5f 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -10217,6 +10217,8 @@ dumpType(Archive *fout, const TypeInfo *tyinfo)
dumpDomain(fout, tyinfo);
else if (tyinfo->typtype == TYPTYPE_COMPOSITE)
dumpCompositeType(fout, tyinfo);
+ else if (tyinfo->typtype == TYPTYPE_DICT)
+ pg_log_error("AALEKSEEV TODO FIXME not implemented");
else if (tyinfo->typtype == TYPTYPE_ENUM)
dumpEnumType(fout, tyinfo);
else if (tyinfo->typtype == TYPTYPE_RANGE)
diff --git a/src/include/catalog/pg_dict.h b/src/include/catalog/pg_dict.h
new file mode 100644
index 0000000000..fbad803468
--- /dev/null
+++ b/src/include/catalog/pg_dict.h
@@ -0,0 +1,78 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_dict.h
+ * definition of the "dict" system catalog (pg_dict)
+ *
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ * src/include/catalog/pg_dict.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef PG_DICT_H
+#define PG_DICT_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_dict_d.h"
+
+#include "nodes/pg_list.h"
+
+/* ----------------
+ * pg_dict definition. cpp turns this into
+ * typedef struct FormData_pg_dict
+ * ----------------
+ */
+CATALOG(pg_dict,9861,DictRelationId)
+{
+ Oid oid; /* oid */
+ Oid dicttypid BKI_LOOKUP(pg_type); /* OID of owning dict type */
+ /* AALEKSEEV TODO refactor to bytea */
+ NameData dictentry; /* text representation of the dictionary entry */
+} FormData_pg_dict;
+
+/* ----------------
+ * Form_pg_dict corresponds to a pointer to a tuple with
+ * the format of pg_dict relation.
+ * ----------------
+ */
+typedef FormData_pg_dict *Form_pg_dict;
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_dict_oid_index, 9862, DictOidIndexId, on pg_dict using btree(oid oid_ops));
+DECLARE_UNIQUE_INDEX(pg_dict_typid_entry_index, 9863, DictTypIdEntryIndexId, on pg_dict using btree(dicttypid oid_ops, dictentry name_ops));
+
+
+/*
+ * DictEntry type represents one entry in the given dictionary.
+ */
+typedef struct
+{
+ Oid oid; /* entry id */
+ uint32 length; /* entry length */
+ uint8* data; /* entry data */
+} DictEntry;
+
+/*
+ * Dictionary type represents all the entries in the given dictionary.
+ */
+typedef struct
+{
+ uint32 nentries; /* number of entries */
+ DictEntry *entries; /* array of entries */
+} DictionaryData;
+
+typedef DictionaryData *Dictionary;
+
+/*
+ * prototypes for functions in pg_dict.c
+ */
+extern void DictEntriesCreate(Oid dictTypeOid, List *vals);
+extern void DictEntriesFree(Dictionary dict);
+extern Dictionary DictEntriesRead(Oid dictTypeOid);
+extern void DictEntriesDelete(Oid dictTypeOid);
+
+#endif /* PG_DICT_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index be47583122..0e3c5cd870 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10032,6 +10032,26 @@
proname => 'jsonb_path_match_opr', prorettype => 'bool',
proargtypes => 'jsonb jsonpath', prosrc => 'jsonb_path_match_opr' },
+# dictionaries
+{ oid => '9864', descr => 'Converts a cstring to a dictionary',
+ proname => 'dictionary_in', prorettype => 'any', proargtypes => 'cstring oid int4',
+ prosrc => 'dictionary_in' },
+{ oid => '9865', descr => 'Converts a dictionary to a cstring',
+ proname => 'dictionary_out', prorettype => 'cstring', proargtypes => 'any',
+ prosrc => 'dictionary_out' },
+{ oid => '9866', descr => 'Coverts JSONB to a dictionary type',
+ proname => 'jsonb_dictionary', proisstrict => 'f', provolatile => 's',
+ prorettype => 'any', proargtypes => 'jsonb int4',
+ prosrc => 'jsonb_dictionary' },
+{ oid => '9867', descr => 'Converts a dictionary type to JSONB',
+ proname => 'dictionary_jsonb', proisstrict => 'f', provolatile => 's',
+ prorettype => 'jsonb', proargtypes => 'any',
+ prosrc => 'dictionary_jsonb' },
+{ oid => '9868', descr => 'Converts a dictionary type to a byte array',
+ proname => 'dictionary_bytea', proisstrict => 'f', provolatile => 's',
+ prorettype => 'jsonb', proargtypes => 'any',
+ prosrc => 'dictionary_bytea' },
+
# historical int8/txid_snapshot variants of xid8 functions
{ oid => '2939', descr => 'I/O',
proname => 'txid_snapshot_in', prorettype => 'txid_snapshot',
diff --git a/src/include/catalog/pg_type.h b/src/include/catalog/pg_type.h
index 48a2559137..91c6bfae99 100644
--- a/src/include/catalog/pg_type.h
+++ b/src/include/catalog/pg_type.h
@@ -272,6 +272,7 @@ DECLARE_UNIQUE_INDEX(pg_type_typname_nsp_index, 2704, TypeNameNspIndexId, on pg_
*/
#define TYPTYPE_BASE 'b' /* base type (ordinary scalar type) */
#define TYPTYPE_COMPOSITE 'c' /* composite (e.g., table's rowtype) */
+#define TYPTYPE_DICT 'D' /* dictionary */
#define TYPTYPE_DOMAIN 'd' /* domain over another type */
#define TYPTYPE_ENUM 'e' /* enumerated type */
#define TYPTYPE_MULTIRANGE 'm' /* multirange type */
diff --git a/src/include/commands/typecmds.h b/src/include/commands/typecmds.h
index a17bedb851..5cc8e5ff99 100644
--- a/src/include/commands/typecmds.h
+++ b/src/include/commands/typecmds.h
@@ -24,6 +24,7 @@
extern ObjectAddress DefineType(ParseState *pstate, List *names, List *parameters);
extern void RemoveTypeById(Oid typeOid);
extern ObjectAddress DefineDomain(CreateDomainStmt *stmt);
+extern ObjectAddress DefineDictionary(CreateDictionaryStmt *stmt);
extern ObjectAddress DefineEnum(CreateEnumStmt *stmt);
extern ObjectAddress DefineRange(ParseState *pstate, CreateRangeStmt *stmt);
extern ObjectAddress AlterEnum(AlterEnumStmt *stmt);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 98fe1abaa2..157686c399 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3580,6 +3580,18 @@ typedef struct CompositeTypeStmt
List *coldeflist; /* list of ColumnDef nodes */
} CompositeTypeStmt;
+/* ----------------------
+ * Create Type Statement, dictionary types
+ * ----------------------
+ */
+typedef struct CreateDictionaryStmt
+{
+ NodeTag type;
+ List *typeName; /* qualified name (list of String) */
+ List *baseTypeName; /* qualified base type name (list of String) */
+ List *vals; /* dictionary entries (list of String) */
+} CreateDictionaryStmt;
+
/* ----------------------
* Create Type Statement, enum types
* ----------------------
diff --git a/src/pl/plpgsql/src/pl_comp.c b/src/pl/plpgsql/src/pl_comp.c
index 61fbdf0686..72e3fa5906 100644
--- a/src/pl/plpgsql/src/pl_comp.c
+++ b/src/pl/plpgsql/src/pl_comp.c
@@ -2123,6 +2123,7 @@ build_datatype(HeapTuple typeTup, int32 typmod,
switch (typeStruct->typtype)
{
case TYPTYPE_BASE:
+ case TYPTYPE_DICT:
case TYPTYPE_ENUM:
case TYPTYPE_RANGE:
case TYPTYPE_MULTIRANGE:
diff --git a/src/test/regress/expected/dict.out b/src/test/regress/expected/dict.out
new file mode 100644
index 0000000000..699e76b654
--- /dev/null
+++ b/src/test/regress/expected/dict.out
@@ -0,0 +1,47 @@
+--
+-- Compression dictionaries tests
+--
+-- Dictionary types currently can be used only with JSONB
+\set ON_ERROR_STOP 0
+CREATE TYPE textdict AS DICTIONARY OF TEXT ('abcdef', 'ghijkl');
+ERROR: Dictionary types currently can be used only with JSONB
+\set ON_ERROR_STOP 1
+-- Simple happy path
+CREATE TYPE mydict AS DICTIONARY OF JSONB ('abcdef', 'ghijkl');
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
+ dictentry
+-----------
+ abcdef
+ ghijkl
+(2 rows)
+
+SELECT '{"abcdef":"ghijkl"}' :: mydict;
+ mydict
+----------------------
+ {"abcdef": "ghijkl"}
+(1 row)
+
+SELECT '{"abcdef":"ghijkl"}' :: mydict :: jsonb;
+ jsonb
+----------------------
+ {"abcdef": "ghijkl"}
+(1 row)
+
+SELECT '{"abcdef":"ghijkl"}' :: jsonb :: mydict;
+ mydict
+----------------------
+ {"abcdef": "ghijkl"}
+(1 row)
+
+SELECT ('{"abcdef":"ghijkl"}' :: mydict) -> 'abcdef';
+ ?column?
+----------
+ "ghijkl"
+(1 row)
+
+DROP TYPE mydict;
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
+ dictentry
+-----------
+(0 rows)
+
diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out
index 215eb899be..91bc491a6c 100644
--- a/src/test/regress/expected/oidjoins.out
+++ b/src/test/regress/expected/oidjoins.out
@@ -182,6 +182,7 @@ NOTICE: checking pg_description {classoid} => pg_class {oid}
NOTICE: checking pg_cast {castsource} => pg_type {oid}
NOTICE: checking pg_cast {casttarget} => pg_type {oid}
NOTICE: checking pg_cast {castfunc} => pg_proc {oid}
+NOTICE: checking pg_dict {dicttypid} => pg_type {oid}
NOTICE: checking pg_enum {enumtypid} => pg_type {oid}
NOTICE: checking pg_namespace {nspowner} => pg_authid {oid}
NOTICE: checking pg_conversion {connamespace} => pg_namespace {oid}
diff --git a/src/test/regress/expected/opr_sanity.out b/src/test/regress/expected/opr_sanity.out
index 86d755aa44..5dc4d0ab01 100644
--- a/src/test/regress/expected/opr_sanity.out
+++ b/src/test/regress/expected/opr_sanity.out
@@ -388,11 +388,12 @@ WHERE 'cstring'::regtype = ANY (p1.proargtypes)
AND NOT EXISTS(SELECT 1 FROM pg_conversion WHERE conproc = p1.oid)
AND p1.oid != 'shell_in(cstring)'::regprocedure
ORDER BY 1;
- oid | proname
-------+--------------
+ oid | proname
+------+---------------
2293 | cstring_out
2501 | cstring_send
-(2 rows)
+ 9864 | dictionary_in
+(3 rows)
-- Likewise, look for functions that return cstring and aren't datatype output
-- functions nor typmod output functions.
@@ -405,11 +406,12 @@ WHERE p1.prorettype = 'cstring'::regtype
AND NOT EXISTS(SELECT 1 FROM pg_type WHERE typmodout = p1.oid)
AND p1.oid != 'shell_out(void)'::regprocedure
ORDER BY 1;
- oid | proname
-------+--------------
+ oid | proname
+------+----------------
2292 | cstring_in
2500 | cstring_recv
-(2 rows)
+ 9865 | dictionary_out
+(3 rows)
-- Check for length inconsistencies between the various argument-info arrays.
SELECT p1.oid, p1.proname
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 103e11483d..2fa15a2080 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -111,7 +111,7 @@ test: select_views portals_p2 foreign_key cluster dependency guc bitmapops combo
# ----------
# Another group of parallel tests (JSON related)
# ----------
-test: json jsonb json_encoding jsonpath jsonpath_encoding jsonb_jsonpath sqljson json_sqljson jsonb_sqljson
+test: dict json jsonb json_encoding jsonpath jsonpath_encoding jsonb_jsonpath sqljson json_sqljson jsonb_sqljson
# ----------
# Another group of parallel tests
diff --git a/src/test/regress/sql/dict.sql b/src/test/regress/sql/dict.sql
new file mode 100644
index 0000000000..27ecd6a451
--- /dev/null
+++ b/src/test/regress/sql/dict.sql
@@ -0,0 +1,20 @@
+--
+-- Compression dictionaries tests
+--
+
+-- Dictionary types currently can be used only with JSONB
+\set ON_ERROR_STOP 0
+CREATE TYPE textdict AS DICTIONARY OF TEXT ('abcdef', 'ghijkl');
+\set ON_ERROR_STOP 1
+
+-- Simple happy path
+CREATE TYPE mydict AS DICTIONARY OF JSONB ('abcdef', 'ghijkl');
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
+
+SELECT '{"abcdef":"ghijkl"}' :: mydict;
+SELECT '{"abcdef":"ghijkl"}' :: mydict :: jsonb;
+SELECT '{"abcdef":"ghijkl"}' :: jsonb :: mydict;
+SELECT ('{"abcdef":"ghijkl"}' :: mydict) -> 'abcdef';
+
+DROP TYPE mydict;
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
--
2.37.1
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: [PATCH] Compression dictionaries for JSONB
@ 2022-08-19 07:57 Nikita Malakhov <[email protected]>
parent: Aleksander Alekseev <[email protected]>
0 siblings, 1 reply; 88+ messages in thread
From: Nikita Malakhov @ 2022-08-19 07:57 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>; Simon Riggs <[email protected]>; Matthias van de Meent <[email protected]>; Alvaro Herrera <[email protected]>; Zhihong Yu <[email protected]>; Teodor Sigaev <[email protected]>
Hi hackers!
I've got a partly question, partly proposal for the future development of
this
feature:
What if we use pg_dict table not to store dictionaries but to store
dictionaries'
meta, and actual dictionaries to be stored in separate tables like it is
done with
TOAST tables (i.e. pg_dict.<dictionary 1 entry> --> pg_dict_16385 table)?
Thus we can kill several birds with one stone - we deal with concurrent
dictionaries' updates - which looks like very serious issue for now, they
do not
affect each other and overall DB performance while using, we get around SQL
statement size restriction, could effectively deal with versions in
dictionaries
and even dictionaries' versions, as well as dictionary size restriction, we
can
use it for duplicated JSON parts, and even we can provide an API to work
with dictionaries and dictionary tables which later could be usable even
for
working with JSON schemas as well (maybe, with some extension)?
Overall structure could look like this:
pg_dict
|
|---- dictionary 1 meta
| |--name
| |--size
| |--etc
| |--dictionary table name (i.e. pg_dict_16385)
| |
| |----> pg_dict_16385
|
|---- dictionary 2 meta
| |--name
| |--size
| |--etc
| |--dictionary table name (i.e. pg_dict_16386)
| |
| |----> pg_dict_16386
...
where dictionary table could look like
pg_dict_16385
|
|---- key 1
| |-value
|
|---- key 2
| |-value
...
And with a special DICT API we would have means to access, cache, store our
dictionaries in a uniform way from different levels. In this implementation
it also
looks as a very valuable addition for our JSONb Toaster.
JSON schema processing is a very promising feature and we have to keep up
with major competitors like Oracle which are already working on it.
On Mon, Aug 1, 2022 at 2:25 PM Aleksander Alekseev <[email protected]>
wrote:
> Hi hackers,
>
> > So far we seem to have a consensus to:
> >
> > 1. Use bytea instead of NameData to store dictionary entries;
> >
> > 2. Assign monotonically ascending IDs to the entries instead of using
> > Oids, as it is done with pg_class.relnatts. In order to do this we
> > should either add a corresponding column to pg_type, or add a new
> > catalog table, e.g. pg_dict_meta. Personally I don't have a strong
> > opinion on what is better. Thoughts?
> >
> > Both changes should be straightforward to implement and also are a
> > good exercise to newcomers.
> >
> > I invite anyone interested to join this effort as a co-author! (since,
> > honestly, rewriting the same feature over and over again alone is
> > quite boring :D).
>
> cfbot complained that v5 doesn't apply anymore. Here is the rebased
> version of the patch.
>
> > Good point. This was not a problem for ZSON since the dictionary size
> > was limited to 2**16 entries, the dictionary was immutable, and the
> > dictionaries had versions. For compression dictionaries we removed the
> > 2**16 entries limit and also decided to get rid of versions. The idea
> > was that you can simply continue adding new entries, but no one
> > thought about the fact that this will consume the memory required to
> > decompress the document indefinitely.
> >
> > Maybe we should return to the idea of limited dictionary size and
> > versions. Objections?
> > [ ...]
> > You are right. Another reason to return to the idea of dictionary
> versions.
>
> Since no one objected so far and/or proposed a better idea I assume
> this can be added to the list of TODOs as well.
>
> --
> Best regards,
> Aleksander Alekseev
>
--
Regards,
Nikita Malakhov
https://postgrespro.ru/
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: [PATCH] Compression dictionaries for JSONB
@ 2022-09-02 10:50 Aleksander Alekseev <[email protected]>
parent: Nikita Malakhov <[email protected]>
0 siblings, 1 reply; 88+ messages in thread
From: Aleksander Alekseev @ 2022-09-02 10:50 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>; +Cc: Simon Riggs <[email protected]>; Nikita Malakhov <[email protected]>; Matthias van de Meent <[email protected]>; Alvaro Herrera <[email protected]>; Zhihong Yu <[email protected]>; Teodor Sigaev <[email protected]>
Hi hackers,
Here is the rebased version of the patch.
> I invite anyone interested to join this effort as a co-author! (since,
> honestly, rewriting the same feature over and over again alone is
> quite boring :D).
> Overall structure could look like this:
> pg_dict
> |
> |---- dictionary 1 meta
> | |--name
> | |--size
> | |--etc
> | |--dictionary table name (i.e. pg_dict_16385)
> | |
> | |----> pg_dict_16385
> |
> |---- dictionary 2 meta
> | |--name
> | |--size
> | |--etc
> | |--dictionary table name (i.e. pg_dict_16386)
> | |
> | |----> pg_dict_16386
For the record, Nikita and I agreed offlist that Nikita will join this
effort as a co-author in order to implement the suggested improvements
(and perhaps some improvements that were not suggested yet). Meanwhile
I'm going to keep the current version of the patch up to date with the
`master` branch.
--
Best regards,
Aleksander Alekseev
Attachments:
[application/octet-stream] v7-0001-Compression-dictionaries-for-JSONB.patch (48.2K, ../../CAJ7c6TOnCgvK-HNZ5yRDa6kCidut6O0-SooJqFubtfJBMgxLzQ@mail.gmail.com/2-v7-0001-Compression-dictionaries-for-JSONB.patch)
download | inline diff:
From d349264fcdaf56a0e61e2aff4e74bd782191f8ed Mon Sep 17 00:00:00 2001
From: Aleksander Alekseev <[email protected]>
Date: Tue, 12 Apr 2022 15:58:11 +0300
Subject: [PATCH v7] Compression dictionaries for JSONB
Usage example:
CREATE TYPE mydict AS DITIONARY OF jsonb ('aaa', 'bbb', ...);
SELECT ('{"aaa":"bbb"}' :: mydict) -> 'aaa';
The created type works as a drop-in replacement for JSONB. However,
its internal representation differs. The provided dictionary entries ('aaa',
'bbb', ..) are stored in pg_dict catalog table. When `mydict` sees one of
the entries in the document, it replaces it with the corresponding pg_dict
entry Oid. For more details regarding the compression algorithm and choosen
compromises please see the comments in the code.
In pg_type `mydict` has typtype = TYPTYPE_DICT. It works the same way
as TYPTYPE_BASE with only difference: corresponding `<type>_in`
(pg_type.typinput) and `<another-type>_<type>` (pg_cast.castfunc)
procedures receive the dictionary Oid as a `typmod` argument. This way the
procedures can distinguish `mydict1` from `mydict2` and use the proper
compression dictionary.
(NOTE: dear committer, please bump the catalog version!)
Author: Aleksander Alekseev <[email protected]>
Author: TODO FIXME add your name and email here
Reviewed-by: FIXME
Discussion: https://postgr.es/m/CAJ7c6TOtAB0z1UrksvGTStNE-herK-43bj22%3D5xVBg7S4vr5rQ%40mail.gmail.com
Discussion: https://postgr.es/m/CAJ7c6TPx7N-bVw0dZ1ASCDQKZJHhBYkT6w4HV1LzfS%2BUUTUfmA%40mail.gmail.com
---
src/backend/catalog/Makefile | 3 +-
src/backend/catalog/pg_dict.c | 197 +++++++++
src/backend/commands/typecmds.c | 160 +++++++-
src/backend/executor/functions.c | 1 +
src/backend/parser/gram.y | 8 +
src/backend/parser/parse_coerce.c | 17 +-
src/backend/parser/parse_type.c | 16 +
src/backend/tcop/utility.c | 13 +
src/backend/utils/adt/Makefile | 1 +
src/backend/utils/adt/dictionaries.c | 482 +++++++++++++++++++++++
src/backend/utils/fmgr/funcapi.c | 1 +
src/bin/pg_dump/pg_dump.c | 2 +
src/include/catalog/pg_dict.h | 78 ++++
src/include/catalog/pg_proc.dat | 20 +
src/include/catalog/pg_type.h | 1 +
src/include/commands/typecmds.h | 1 +
src/include/nodes/parsenodes.h | 12 +
src/pl/plpgsql/src/pl_comp.c | 1 +
src/test/regress/expected/dict.out | 47 +++
src/test/regress/expected/oidjoins.out | 1 +
src/test/regress/expected/opr_sanity.out | 14 +-
src/test/regress/parallel_schedule | 2 +-
src/test/regress/sql/dict.sql | 20 +
23 files changed, 1087 insertions(+), 11 deletions(-)
create mode 100644 src/backend/catalog/pg_dict.c
create mode 100644 src/backend/utils/adt/dictionaries.c
create mode 100644 src/include/catalog/pg_dict.h
create mode 100644 src/test/regress/expected/dict.out
create mode 100644 src/test/regress/sql/dict.sql
diff --git a/src/backend/catalog/Makefile b/src/backend/catalog/Makefile
index 89a0221ec9..3b4330148e 100644
--- a/src/backend/catalog/Makefile
+++ b/src/backend/catalog/Makefile
@@ -33,6 +33,7 @@ OBJS = \
pg_conversion.o \
pg_db_role_setting.o \
pg_depend.o \
+ pg_dict.o \
pg_enum.o \
pg_inherits.o \
pg_largeobject.o \
@@ -61,7 +62,7 @@ CATALOG_HEADERS := \
pg_language.h pg_largeobject_metadata.h pg_largeobject.h pg_aggregate.h \
pg_statistic.h pg_statistic_ext.h pg_statistic_ext_data.h \
pg_rewrite.h pg_trigger.h pg_event_trigger.h pg_description.h \
- pg_cast.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \
+ pg_cast.h pg_dict.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \
pg_database.h pg_db_role_setting.h pg_tablespace.h \
pg_authid.h pg_auth_members.h pg_shdepend.h pg_shdescription.h \
pg_ts_config.h pg_ts_config_map.h pg_ts_dict.h \
diff --git a/src/backend/catalog/pg_dict.c b/src/backend/catalog/pg_dict.c
new file mode 100644
index 0000000000..456f3b5177
--- /dev/null
+++ b/src/backend/catalog/pg_dict.c
@@ -0,0 +1,197 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_dict.c
+ * routines to support manipulation of the pg_dict relation
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/catalog/pg_dict.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "access/genam.h"
+#include "access/skey.h"
+#include "access/table.h"
+#include "catalog/catalog.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_dict.h"
+#include "catalog/pg_type.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/memutils.h"
+#include "utils/rel.h"
+
+/*
+ * Creates an entry in pg_dict for each of the supplied values.
+ * vals is a list of String values.
+ */
+void
+DictEntriesCreate(Oid dictTypeOid, List *vals)
+{
+ Relation pg_dict;
+ NameData dictentry;
+ Datum values[Natts_pg_dict];
+ bool nulls[Natts_pg_dict];
+ ListCell *lc;
+ HeapTuple tup;
+
+ if (vals == NIL)
+ {
+ /* The list is empty; do nothing. */
+ return;
+ }
+
+ memset(nulls, false, sizeof(nulls));
+
+ /*
+ * We don't check the list of values for duplicates here. If there are
+ * any, the user will get an unique-index violation.
+ */
+
+ pg_dict = table_open(DictRelationId, RowExclusiveLock);
+ foreach(lc, vals)
+ {
+ Oid oid = GetNewOidWithIndex(pg_dict, DictOidIndexId,
+ Anum_pg_dict_oid);
+ char *entry = strVal(lfirst(lc));
+
+ /*
+ * Entries are stored in a name field, for easier syscache lookup, so
+ * check the length to make sure it's within range.
+ */
+ if (strlen(entry) > (NAMEDATALEN - 1))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_NAME),
+ errmsg("invalid dict entry \"%s\"", entry),
+ errdetail("Entries must be %d bytes or less.",
+ NAMEDATALEN - 1)));
+
+ /* AALEKSEEV TODO use bytea instead */
+ namestrcpy(&dictentry, entry);
+
+ values[Anum_pg_dict_oid - 1] = ObjectIdGetDatum(oid);
+ values[Anum_pg_dict_dicttypid - 1] = ObjectIdGetDatum(dictTypeOid);
+ values[Anum_pg_dict_dictentry - 1] = NameGetDatum(&dictentry);
+
+ tup = heap_form_tuple(RelationGetDescr(pg_dict), values, nulls);
+
+ CatalogTupleInsert(pg_dict, tup);
+ heap_freetuple(tup);
+ }
+
+ /* clean up */
+ table_close(pg_dict, RowExclusiveLock);
+}
+
+/*
+ * Returns all the entries for the dictinary with given Oid. Entries are sorted
+ * by dictentry. Note that shorter entries are considered smaller, i.e. 'abc'
+ * goes before 'abcdef'. The memory is allocated in a child memory context of
+ * the caller's memory context. It can be freed with DictEntriesFree().
+ *
+ * If there are no entries a valid but empty dictionary is returned.
+ */
+Dictionary
+DictEntriesRead(Oid dictTypeOid)
+{
+ Relation pg_dict;
+ ScanKeyData key[1];
+ SysScanDesc scan;
+ HeapTuple tup;
+ uint32 entries_allocated;
+ Dictionary dict;
+ MemoryContext myctx, oldctx;
+
+ myctx = AllocSetContextCreate(CurrentMemoryContext, "DictEntriesCtx", ALLOCSET_DEFAULT_SIZES);
+ oldctx = MemoryContextSwitchTo(myctx);
+
+ dict = (Dictionary)palloc(sizeof(DictionaryData));
+
+ entries_allocated = 8;
+ dict->nentries = 0;
+ dict->entries = (DictEntry*)palloc(entries_allocated*sizeof(DictEntry));
+
+ pg_dict = table_open(DictRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_dict_dicttypid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(dictTypeOid));
+
+ scan = systable_beginscan(pg_dict, DictTypIdEntryIndexId, true,
+ NULL, 1, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ if(dict->nentries == entries_allocated)
+ {
+ entries_allocated = entries_allocated * 2;
+ dict->entries = (DictEntry*)repalloc(dict->entries, entries_allocated*sizeof(DictEntry));
+ }
+
+ dict->entries[dict->nentries].oid = ((Form_pg_dict) GETSTRUCT(tup))->oid;
+ /*
+ * This is arguably not the fastest way to determine the length of
+ * the entry. Alternatively, we could store a precalculated value in
+ * the catalog. However, usually it's a good idea to keep things simple
+ * until somebody discovers a bottleneck in this exact place and
+ * proposes a concrete fix.
+ */
+ dict->entries[dict->nentries].length = (uint32)strlen(((Form_pg_dict) GETSTRUCT(tup))->dictentry.data);
+ dict->entries[dict->nentries].data = palloc(dict->entries[dict->nentries].length);
+ memcpy(dict->entries[dict->nentries].data, ((Form_pg_dict) GETSTRUCT(tup))->dictentry.data, dict->entries[dict->nentries].length);
+ dict->nentries++;
+ }
+
+ systable_endscan(scan);
+ table_close(pg_dict, RowExclusiveLock);
+
+ MemoryContextSwitchTo(oldctx);
+
+ return dict;
+}
+
+/*
+ * Frees the memory allocated for the dictionary.
+ */
+void
+DictEntriesFree(Dictionary dict)
+{
+ MemoryContextDelete(GetMemoryChunkContext(dict));
+}
+
+/*
+ * Deletes all the entries for the dictinary with given Oid.
+ */
+void
+DictEntriesDelete(Oid dictTypeOid)
+{
+ Relation pg_dict;
+ ScanKeyData key[1];
+ SysScanDesc scan;
+ HeapTuple tup;
+
+ pg_dict = table_open(DictRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_dict_dicttypid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(dictTypeOid));
+
+ scan = systable_beginscan(pg_dict, DictTypIdEntryIndexId, true,
+ NULL, 1, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ CatalogTupleDelete(pg_dict, &tup->t_self);
+ }
+
+ systable_endscan(scan);
+ table_close(pg_dict, RowExclusiveLock);
+}
diff --git a/src/backend/commands/typecmds.c b/src/backend/commands/typecmds.c
index 33b64fd279..2481a05df1 100644
--- a/src/backend/commands/typecmds.c
+++ b/src/backend/commands/typecmds.c
@@ -46,6 +46,7 @@
#include "catalog/pg_collation.h"
#include "catalog/pg_constraint.h"
#include "catalog/pg_depend.h"
+#include "catalog/pg_dict.h"
#include "catalog/pg_enum.h"
#include "catalog/pg_language.h"
#include "catalog/pg_namespace.h"
@@ -679,6 +680,14 @@ RemoveTypeById(Oid typeOid)
if (((Form_pg_type) GETSTRUCT(tup))->typtype == TYPTYPE_RANGE)
RangeDelete(typeOid);
+ /*
+ * If it is a dictionary type, delete the pg_dict entries too; we don't
+ * bother with making a dependency entry for that, so it has to be done
+ * "by hand" here.
+ */
+ if (((Form_pg_type) GETSTRUCT(tup))->typtype == TYPTYPE_DICT)
+ DictEntriesDelete(typeOid);
+
ReleaseSysCache(tup);
table_close(relation, RowExclusiveLock);
@@ -763,8 +772,8 @@ DefineDomain(CreateDomainStmt *stmt)
/*
* Base type must be a plain base type, a composite type, another domain,
- * an enum or a range type. Domains over pseudotypes would create a
- * security hole. (It would be shorter to code this to just check for
+ * a dict, an enum or a range type. Domains over pseudotypes would create
+ * a security hole. (It would be shorter to code this to just check for
* pseudotypes; but it seems safer to call out the specific typtypes that
* are supported, rather than assume that all future typtypes would be
* automatically supported.)
@@ -772,6 +781,7 @@ DefineDomain(CreateDomainStmt *stmt)
typtype = baseType->typtype;
if (typtype != TYPTYPE_BASE &&
typtype != TYPTYPE_COMPOSITE &&
+ typtype != TYPTYPE_DICT &&
typtype != TYPTYPE_DOMAIN &&
typtype != TYPTYPE_ENUM &&
typtype != TYPTYPE_RANGE &&
@@ -1129,6 +1139,152 @@ DefineDomain(CreateDomainStmt *stmt)
}
+/*
+ * DefineDictionary
+ * Registers a new dictionary.
+ */
+ObjectAddress
+DefineDictionary(CreateDictionaryStmt *stmt)
+{
+ char *dictName;
+ char *dictBaseTypeName;
+ char *dictArrayName;
+ Oid dictNamespace;
+ AclResult aclresult;
+ Oid old_type_oid;
+ Oid dictArrayOid;
+ ObjectAddress dictTypeAddr;
+
+ Assert(list_length(stmt->baseTypeName) == 1);
+ dictBaseTypeName = strVal(linitial(stmt->baseTypeName));
+ if(pg_strcasecmp(dictBaseTypeName, "jsonb") != 0)
+ ereport(ERROR,
+ (
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("Dictionary types currently can be used only with JSONB")
+ )
+ );
+
+ /* Convert list of names to a name and namespace */
+ dictNamespace = QualifiedNameGetCreationNamespace(stmt->typeName,
+ &dictName);
+
+ /* Check we have creation rights in target namespace */
+ aclresult = pg_namespace_aclcheck(dictNamespace, GetUserId(), ACL_CREATE);
+ if (aclresult != ACLCHECK_OK)
+ aclcheck_error(aclresult, OBJECT_SCHEMA,
+ get_namespace_name(dictNamespace));
+
+ /*
+ * Check for collision with an existing type name. If there is one and
+ * it's an autogenerated array, we can rename it out of the way.
+ */
+ old_type_oid = GetSysCacheOid2(TYPENAMENSP, Anum_pg_type_oid,
+ CStringGetDatum(dictName),
+ ObjectIdGetDatum(dictNamespace));
+ if (OidIsValid(old_type_oid))
+ {
+ if (!moveArrayTypeName(old_type_oid, dictName, dictNamespace))
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("type \"%s\" already exists", dictName)));
+ }
+
+ /* Allocate OID for array type */
+ dictArrayOid = AssignTypeArrayOid();
+
+ /* Create the pg_type entry */
+ dictTypeAddr =
+ TypeCreate(InvalidOid, /* no predetermined type OID */
+ dictName, /* type name */
+ dictNamespace, /* namespace */
+ InvalidOid, /* relation oid (n/a here) */
+ 0, /* relation kind (ditto) */
+ GetUserId(), /* owner's ID */
+ -1, /* internal size: varlena, as for JSONB */
+ TYPTYPE_DICT, /* type-type: dictionary */
+ TYPCATEGORY_USER, /* type-category: user, as for JSONB */
+ false, /* dict types are never preferred */
+ DEFAULT_TYPDELIM, /* array element delimiter */
+ F_DICTIONARY_IN, /* input procedure */
+ F_DICTIONARY_OUT, /* output procedure */
+ InvalidOid, /* receive procedure: none */
+ InvalidOid, /* send procedure: none */
+ InvalidOid, /* typmodin procedure - none */
+ InvalidOid, /* typmodout procedure - none */
+ InvalidOid, /* analyze procedure - default */
+ InvalidOid, /* subscript procedure - none */
+ InvalidOid, /* element type ID */
+ false, /* this is not an array type */
+ dictArrayOid, /* array type we are about to create */
+ InvalidOid, /* base type ID (only for domains) */
+ NULL, /* never a default type value */
+ NULL, /* binary default isn't sent either */
+ false, /* passed by value: same as for JSONB */
+ TYPALIGN_INT, /* int alignment */
+ TYPSTORAGE_EXTENDED, /* TOAST strategy: fully toastable, as
+ * JSONB */
+ -1, /* typMod (Domains only) */
+ 0, /* Array dimensions of typbasetype */
+ false, /* Type NOT NULL */
+ InvalidOid); /* type's collation */
+
+ /*
+ * Create the array type that goes with it.
+ */
+ dictArrayName = makeArrayTypeName(dictName, dictNamespace);
+
+ TypeCreate(dictArrayOid, /* force assignment of this type OID */
+ dictArrayName, /* type name */
+ dictNamespace, /* namespace */
+ InvalidOid, /* relation oid (n/a here) */
+ 0, /* relation kind (ditto) */
+ GetUserId(), /* owner's ID */
+ -1, /* internal size (always varlena) */
+ TYPTYPE_BASE, /* type-type (base type) */
+ TYPCATEGORY_ARRAY, /* type-category (array) */
+ false, /* array types are never preferred */
+ DEFAULT_TYPDELIM, /* array element delimiter */
+ F_ARRAY_IN, /* input procedure */
+ F_ARRAY_OUT, /* output procedure */
+ F_ARRAY_RECV, /* receive procedure */
+ F_ARRAY_SEND, /* send procedure */
+ InvalidOid, /* typmodin procedure - none */
+ InvalidOid, /* typmodout procedure - none */
+ F_ARRAY_TYPANALYZE, /* analyze procedure */
+ F_ARRAY_SUBSCRIPT_HANDLER, /* array subscript procedure */
+ dictTypeAddr.objectId, /* element type ID */
+ true, /* yes this is an array type */
+ InvalidOid, /* no further array type */
+ InvalidOid, /* base type ID */
+ NULL, /* never a default type value */
+ NULL, /* binary default isn't sent either */
+ false, /* passed by value: same as for JSONB */
+ TYPALIGN_INT, /* enums have int align, so do their arrays */
+ TYPSTORAGE_EXTENDED, /* ARRAY is always toastable */
+ -1, /* typMod (Domains only) */
+ 0, /* Array dimensions of typbasetype */
+ false, /* Type NOT NULL */
+ InvalidOid); /* type's collation */
+
+ pfree(dictArrayName);
+
+ /* Enter the dict's entries into pg_dict */
+ DictEntriesCreate(dictTypeAddr.objectId, stmt->vals);
+
+ /* Create casts to and from JSONB */
+ CastCreate(JSONBOID, dictTypeAddr.objectId, F_JSONB_DICTIONARY, 'a', 'f', DEPENDENCY_INTERNAL);
+ CastCreate(dictTypeAddr.objectId, JSONBOID, F_DICTIONARY_JSONB, 'i', 'f', DEPENDENCY_INTERNAL);
+
+ /*
+ * Create explicit cast to bytea. This is convenient for debugging purposes.
+ * Casting bytea to a dictionary type is dangerous and thus not supported.
+ */
+ CastCreate(dictTypeAddr.objectId, BYTEAOID, F_DICTIONARY_BYTEA, 'e', 'f', DEPENDENCY_INTERNAL);
+
+ return dictTypeAddr;
+}
+
/*
* DefineEnum
* Registers a new enum.
diff --git a/src/backend/executor/functions.c b/src/backend/executor/functions.c
index e134a82ff7..9674cdfe4d 100644
--- a/src/backend/executor/functions.c
+++ b/src/backend/executor/functions.c
@@ -1721,6 +1721,7 @@ check_sql_fn_retval(List *queryTreeLists,
if (fn_typtype == TYPTYPE_BASE ||
fn_typtype == TYPTYPE_DOMAIN ||
+ fn_typtype == TYPTYPE_DICT ||
fn_typtype == TYPTYPE_ENUM ||
fn_typtype == TYPTYPE_RANGE ||
fn_typtype == TYPTYPE_MULTIRANGE)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 0492ff9a66..9ccc26945c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -6155,6 +6155,14 @@ DefineStmt:
n->coldeflist = $6;
$$ = (Node *) n;
}
+ | CREATE TYPE_P any_name AS DICTIONARY OF any_name '(' opt_enum_val_list ')'
+ {
+ CreateDictionaryStmt *n = makeNode(CreateDictionaryStmt);
+ n->typeName = $3;
+ n->baseTypeName = $7;
+ n->vals = $9;
+ $$ = (Node *)n;
+ }
| CREATE TYPE_P any_name AS ENUM_P '(' opt_enum_val_list ')'
{
CreateEnumStmt *n = makeNode(CreateEnumStmt);
diff --git a/src/backend/parser/parse_coerce.c b/src/backend/parser/parse_coerce.c
index c4e958e4aa..e489dd9421 100644
--- a/src/backend/parser/parse_coerce.c
+++ b/src/backend/parser/parse_coerce.c
@@ -279,7 +279,22 @@ coerce_type(ParseState *pstate, Node *node,
if (baseTypeId == INTERVALOID)
inputTypeMod = baseTypeMod;
else
- inputTypeMod = -1;
+ {
+ if((int32)targetTypeId == targetTypeMod)
+ {
+ /*
+ * For dictionaries internally we set typmod to a dictionary
+ * OID. This allows dictionary_in() and jsonb_dictionary() to
+ * distinguish one dictionary from another.
+ *
+ * It also allows us to recognize that the target type is
+ * a dictionary here and pass a correct typmod.
+ */
+ inputTypeMod = targetTypeMod;
+ }
+ else
+ inputTypeMod = -1;
+ }
baseType = typeidType(baseTypeId);
diff --git a/src/backend/parser/parse_type.c b/src/backend/parser/parse_type.c
index f7ad689459..2188002535 100644
--- a/src/backend/parser/parse_type.c
+++ b/src/backend/parser/parse_type.c
@@ -259,6 +259,10 @@ LookupTypeNameOid(ParseState *pstate, const TypeName *typeName, bool missing_ok)
* This is equivalent to LookupTypeName, except that this will report
* a suitable error message if the type cannot be found or is not defined.
* Callers of this can therefore assume the result is a fully valid type.
+ *
+ * For a dictionary type it's OID is returned as a typmod. This is used by
+ * dictionary_in() and jsonb_dictionary() to distinguish one dictionary from
+ * another.
*/
Type
typenameType(ParseState *pstate, const TypeName *typeName, int32 *typmod_p)
@@ -305,6 +309,10 @@ typenameTypeId(ParseState *pstate, const TypeName *typeName)
*
* This is equivalent to typenameType, but we only hand back the type OID
* and typmod, not the syscache entry.
+ *
+ * For a dictionary type it's OID is returned as a typmod. This is used by
+ * dictionary_in() and jsonb_dictionary() to distinguish one dictionary from
+ * another.
*/
void
typenameTypeIdAndMod(ParseState *pstate, const TypeName *typeName,
@@ -327,6 +335,10 @@ typenameTypeIdAndMod(ParseState *pstate, const TypeName *typeName,
* looked up, and is passed as "typ".
*
* pstate is only used for error location info, and may be NULL.
+ *
+ * For a dictionary type it's OID is returned as a typmod. This is used by
+ * dictionary_in() and jsonb_dictionary() to distinguish one dictionary from
+ * another.
*/
static int32
typenameTypeMod(ParseState *pstate, const TypeName *typeName, Type typ)
@@ -339,6 +351,10 @@ typenameTypeMod(ParseState *pstate, const TypeName *typeName, Type typ)
ArrayType *arrtypmod;
ParseCallbackState pcbstate;
+ /* For a dictionary return it's OID. */
+ if (((Form_pg_type) GETSTRUCT(typ))->typtype == TYPTYPE_DICT)
+ return (int32)((Form_pg_type) GETSTRUCT(typ))->oid;
+
/* Return prespecified typmod if no typmod expressions */
if (typeName->typmods == NIL)
return typeName->typemod;
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index aa00815787..183c8a8172 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -174,6 +174,7 @@ ClassifyUtilityCommandAsReadOnly(Node *parsetree)
case T_CreateCastStmt:
case T_CreateConversionStmt:
case T_CreateDomainStmt:
+ case T_CreateDictionaryStmt:
case T_CreateEnumStmt:
case T_CreateEventTrigStmt:
case T_CreateExtensionStmt:
@@ -1621,6 +1622,10 @@ ProcessUtilitySlow(ParseState *pstate,
}
break;
+ case T_CreateDictionaryStmt: /* CREATE TYPE AS DICTIONARY OF */
+ address = DefineDictionary((CreateDictionaryStmt *) parsetree);
+ break;
+
case T_CreateEnumStmt: /* CREATE TYPE AS ENUM */
address = DefineEnum((CreateEnumStmt *) parsetree);
break;
@@ -2772,6 +2777,10 @@ CreateCommandTag(Node *parsetree)
tag = CMDTAG_CREATE_TYPE;
break;
+ case T_CreateDictionaryStmt:
+ tag = CMDTAG_CREATE_TYPE;
+ break;
+
case T_CreateEnumStmt:
tag = CMDTAG_CREATE_TYPE;
break;
@@ -3419,6 +3428,10 @@ GetCommandLogLevel(Node *parsetree)
lev = LOGSTMT_DDL;
break;
+ case T_CreateDictionaryStmt:
+ lev = LOGSTMT_DDL;
+ break;
+
case T_CreateEnumStmt:
lev = LOGSTMT_DDL;
break;
diff --git a/src/backend/utils/adt/Makefile b/src/backend/utils/adt/Makefile
index 7c722ea2ce..a7ec3d89f3 100644
--- a/src/backend/utils/adt/Makefile
+++ b/src/backend/utils/adt/Makefile
@@ -30,6 +30,7 @@ OBJS = \
datetime.o \
datum.o \
dbsize.o \
+ dictionaries.o \
domains.o \
encode.o \
enum.o \
diff --git a/src/backend/utils/adt/dictionaries.c b/src/backend/utils/adt/dictionaries.c
new file mode 100644
index 0000000000..2bb46471a4
--- /dev/null
+++ b/src/backend/utils/adt/dictionaries.c
@@ -0,0 +1,482 @@
+/*-------------------------------------------------------------------------
+ *
+ * dictionaries.c
+ * Conversion functions for dictionary types.
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/adt/dictionaries.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "catalog/pg_dict.h"
+#include "utils/fmgrprotos.h"
+#include "utils/jsonb.h"
+
+/*
+ * When compressing a data we treat it as a BLOB, in other words we don't
+ * assume anything regarding its internal representation. This is not
+ * necessarily the best and/or the only possible approach. However, it is
+ * universal and can be reused for JSONB, TEXT, XML and other types. Alternative
+ * compression methods can be introduced in the future, so that the user will
+ * be able to choose the best one for the task.
+ *
+ * The compressed data is stored in the following format:
+ *
+ * (struct varlena)
+ * dictionary_id [uint32]
+ * decompressed_size [uint32]
+ * algorithm_version [uint8]
+ *
+ * (repeated) {
+ * number of bytes to copy as-is [uint8]
+ * ... bytes to copy as-is ...
+ * dictionary entry id, or 0 to skip [uint32]
+ * }
+ *
+ * Compressed data is a variable-length type and thus has the 'struct varlena'
+ * header. The code below doesn't consider it being a part of the payload
+ * (e.g. see the DICT_COMP_HEADER_SIZE definition).
+ *
+ * Storing dictionary_id may seem redundant, but without it dictionary_out()
+ * and dictionary_jsonb() have no way to distinguish one dictionary type from
+ * another. dictionary_in() and jsonb_dictionary() get the dictionary id through
+ * the 'typemod' argument.
+ *
+ * Currently algorithm_version is always 0. In the future it will allow us to
+ * introduce new features (e.g. usage of varints) and/or lazily migrate the
+ * data to other compression methods.
+ *
+ * Compression and decompression are implemented based on a binary search over
+ * DictEntry[] array (see pg_dict.h). For compression the array is sorted by
+ * dictentries and for decompression - by oid's.
+ *
+ */
+
+/* Size of the header in the compressed data */
+#define DICT_COMP_HEADER_SIZE (sizeof(uint32)*2 + sizeof(uint8))
+
+/* Extracts dictionary_id from the compressed data */
+#define DICT_COMP_DICTIONARY_ID(hdrp) \
+ (*(uint32*)hdrp)
+
+/* Extracts decompressed_size from the compressed data */
+#define DICT_COMP_DECOMPRESSED_SIZE(hdrp) \
+ (*(uint32*)((uint8*)hdrp + sizeof(uint32)))
+
+/* Extracts algorithm_version from the compressed data */
+#define DICT_COMP_ALGORITHM_VERSION(hdrp) \
+ (*(uint8*)((uint8*)hdrp + sizeof(uint32)*2))
+
+/* Current algorithm_version */
+#define DICT_COMP_CURRENT_ALGORITHM_VERSION 0
+
+/*
+ * bsearch_arg() callback for finding DictEntry by oid.
+ */
+static int
+find_by_oid_cb(const void *key, const void *current, void *arg)
+{
+ /* Note that oids are unsigned, so we should be careful here */
+ if (*(Oid *) key < ((DictEntry *) current)->oid)
+ return -1;
+ else if (*(Oid *) key == ((DictEntry *) current)->oid)
+ return 0; /* found! */
+ else
+ return 1;
+}
+
+/*
+ * qsort() callback for sorting DictEntry[] by oids.
+ */
+static int
+sort_by_oids_cb(const void *left, const void *right)
+{
+ /* Note that oids are unsigned, so we should be careful here */
+ if (((DictEntry *) left)->oid < ((DictEntry *) right)->oid)
+ return -1;
+
+ /* Oids are unique so this callback will never return 0 */
+ return 1;
+}
+
+/*
+ * Finds a DictEntry which dictentry field matches *data and returns its Oid.
+ * If there are several matching entries, the largest is returned. The length
+ * of the found entry is written to *found_length on success. On failure
+ * InvalidOid is returned and found_length is zeroed.
+ *
+ * The implementation is similar to bsearch_arg(). The procedure can't be used
+ * directly because we are looking not by the exact match. We could generalize
+ * this case but the signature of the function becomes so complicated that it
+ * doesn't seem to worth the effort.
+ */
+static Oid
+compress_find_oid(Dictionary dict, const uint8 *data, Size data_size, Size *found_length)
+{
+ int res;
+ int32 left = 0;
+ int32 right = dict->nentries - 1;
+ Size best_length = 0;
+ Oid best_match = InvalidOid;
+
+ while (left <= right)
+ {
+ int32 current = (left + right) / 2;
+ Size nbytes = (Size)dict->entries[current].length;
+
+ if (nbytes > data_size)
+ {
+ /* current can be less or greater depending on the prefix */
+ res = memcmp(dict->entries[current].data, data, data_size);
+
+ /* if prefixes match, current is greater */
+ if (res == 0)
+ res = 1;
+ }
+ else
+ res = memcmp(dict->entries[current].data, data, nbytes);
+
+ if (res == 0) /* match found */
+ {
+ best_length = nbytes;
+ best_match = dict->entries[current].oid;
+
+ if (nbytes == data_size)
+ break;
+
+ /* maybe there is a larger match */
+ left = current + 1;
+ }
+ else if (res < 0) /* current is less */
+ left = current + 1;
+ else /* current is greater */
+ right = current - 1;
+ }
+
+ *found_length = best_length;
+ return best_match;
+}
+
+/*
+ * Finds a DictEntry by Oid using a binary search. The dictionary should be
+ * sorted by oids before the call. Returns NULL if nothing was found.
+ */
+static DictEntry *
+decompress_find_dictentry(Dictionary dict, Oid oid)
+{
+ return (DictEntry *) bsearch_arg(&oid, dict->entries, dict->nentries, sizeof(DictEntry), find_by_oid_cb, NULL);
+}
+
+/*
+ * Estimates the worst-case compressed size of the data of given size.
+ * Worst-case scenario happens when the dictionary consists of single-character
+ * entries. In this case every byte will be encoded as 6 bytes:
+ * 0x00, (0 bytes to copy as-is), 4 bytes of the entry Oid
+ *
+ * This procedure doesn't account for the header size.
+ *
+ * AALEKSEEV FIXME don't use dictionary entries shorter than 5 bytes
+ */
+static Size
+worst_case_compressed_size(Size insize)
+{
+ return insize * 6;
+}
+
+/*
+ * Compresses the data using the provided dictionary. The dictionary should
+ * be sorted by dictentries before the call. Output buffer should be at least
+ * worst_case_compressed_size(src_size) bytes in size.
+ */
+static void
+compress(Dictionary dict,
+ const void *src_data_, Size src_size,
+ void *encoded_data_, Size *pencoded_size)
+{
+ Size nbytes;
+ Size inoffset;
+ Size outskipoffset = 0;
+ Size outoffset = 1;
+ uint8 skipbytes = 0;
+ const uint8 *src_data = src_data_;
+ uint8 *encoded_data = ((uint8 *) encoded_data_);
+
+ for (inoffset = 0; inoffset < src_size;)
+ {
+ Oid code = compress_find_oid(dict, &(src_data[inoffset]),
+ src_size - inoffset, &nbytes);
+
+ if (code == InvalidOid)
+ {
+ skipbytes++;
+ encoded_data[outoffset] = src_data[inoffset];
+ outoffset++;
+ inoffset++;
+
+ if (skipbytes == 255)
+ {
+ encoded_data[outskipoffset] = skipbytes;
+ encoded_data[outoffset++] = 0; /* InvalidOid */
+ encoded_data[outoffset++] = 0;
+ encoded_data[outoffset++] = 0;
+ encoded_data[outoffset++] = 0;
+ outskipoffset = outoffset++;
+ skipbytes = 0;
+ }
+ }
+ else
+ {
+ encoded_data[outskipoffset] = skipbytes;
+ encoded_data[outoffset++] = (code >> 24) & 0xFF;
+ encoded_data[outoffset++] = (code >> 16) & 0xFF;
+ encoded_data[outoffset++] = (code >> 8) & 0xFF;
+ encoded_data[outoffset++] = code & 0xFF;
+ outskipoffset = outoffset++;
+ skipbytes = 0;
+ inoffset += nbytes;
+ }
+ }
+
+ /* Double check that we didn't write out of buffer */
+ Assert(outoffset < worst_case_compressed_size(src_size));
+
+ encoded_data[outskipoffset] = skipbytes;
+ *pencoded_size = outoffset;
+}
+
+/*
+ * Report an internal error in decompress() procedure below.
+ * Under normal circumstances this should never happen.
+ */
+static void
+decompress_error()
+{
+ ereport(ERROR,
+ (
+ errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("Unable to decompress a dictionary type"),
+ errdetail("The compressed data seems to be corrupted"),
+ errhint("Please report the steps to reproduce the issue to pgsql-bugs@")
+ ));
+}
+
+/*
+ * Decompresses the data using the provided dictionary. The dictionary should
+ * be sorted by oids before the call.
+ */
+static void
+decompress(Dictionary dict,
+ const void *encoded_data_, Size encoded_size,
+ void *decoded_data_, Size decoded_size)
+{
+ Size inoffset = 0;
+ Size outoffset = 0;
+ Oid code;
+ uint8 skipbytes;
+ const uint8 *encoded_data = ((uint8 *) encoded_data_);
+ uint8 *decoded_data = decoded_data_;
+
+ for (inoffset = 0; inoffset < encoded_size;)
+ {
+ skipbytes = encoded_data[inoffset++];
+
+ if (skipbytes > decoded_size - outoffset)
+ decompress_error();
+
+ if (skipbytes > encoded_size - inoffset)
+ decompress_error();
+
+ memcpy(
+ &(decoded_data[outoffset]),
+ &(encoded_data[inoffset]),
+ skipbytes
+ );
+
+ outoffset += skipbytes;
+ inoffset += skipbytes;
+
+ if ((encoded_size == inoffset) && (decoded_size == outoffset))
+ break; /* end of input - its OK */
+
+ if (encoded_size - inoffset < 4)
+ decompress_error();
+
+ code = (Oid) encoded_data[inoffset++];
+ code = (code << 8) | (Oid) encoded_data[inoffset++];
+ code = (code << 8) | (Oid) encoded_data[inoffset++];
+ code = (code << 8) | (Oid) encoded_data[inoffset++];
+
+ if (code != InvalidOid)
+ {
+ Size entrylen;
+ DictEntry *entry = decompress_find_dictentry(dict, code);
+
+ if (entry == NULL)
+ decompress_error();
+
+ Assert(entry->oid == code);
+ entrylen = (Size)entry->length;
+
+ if (entrylen > decoded_size - outoffset)
+ decompress_error();
+
+ memcpy(
+ &(decoded_data[outoffset]),
+ entry->data,
+ entrylen
+ );
+
+ outoffset += entrylen;
+ }
+ }
+
+ Assert(decoded_size == outoffset);
+}
+
+/*
+ * Converts a cstring to a dictionary.
+ */
+Datum
+dictionary_in(PG_FUNCTION_ARGS)
+{
+ const char *instr = PG_GETARG_CSTRING(0);
+#ifdef NOT_USED
+ Oid typelem = PG_GETARG_OID(1);
+#endif
+ int32 typmod = PG_GETARG_INT32(2);
+ Jsonb *jsonb;
+ bytea *dict;
+
+ Assert(typmod != -1);
+
+ jsonb = DatumGetJsonbP(DirectFunctionCall1(jsonb_in, CStringGetDatum(instr)));
+ dict = DatumGetByteaP(DirectFunctionCall2(jsonb_dictionary, JsonbPGetDatum(jsonb), Int32GetDatum(typmod)));
+
+ PG_RETURN_BYTEA_P(dict);
+}
+
+/*
+ * Converts a dictionary to a cstring.
+ */
+Datum
+dictionary_out(PG_FUNCTION_ARGS)
+{
+ bytea *dict = PG_GETARG_BYTEA_P(0);
+ Jsonb *jsonb = DatumGetJsonbP(DirectFunctionCall1(dictionary_jsonb, PointerGetDatum(dict)));
+ const char *outstr = DatumGetCString(DirectFunctionCall1(jsonb_out, JsonbPGetDatum(jsonb)));
+
+ PG_RETURN_CSTRING(outstr);
+}
+
+/*
+ * Coverts JSONB to a dictionary type.
+ *
+ * AALEKSEEV FIXME: if the compressed document ends up being larger than
+ * the original one write a corresponding flag and copy all the data as-is.
+ */
+Datum
+jsonb_dictionary(PG_FUNCTION_ARGS)
+{
+ Dictionary dict;
+ Jsonb *jsonb = PG_GETARG_JSONB_P(0);
+ int32 typmod = PG_GETARG_INT32(1);
+ uint8 *jsonb_data = (uint8 *) VARDATA(jsonb);
+ Size jsonb_data_size = VARSIZE(jsonb) - VARHDRSZ;
+ uint8 *encoded_buff,
+ *encoded_header,
+ *encoded_data;
+ Size encoded_size,
+ encoded_buff_size;
+
+ Assert(typmod != -1);
+
+ dict = DictEntriesRead(typmod);
+ qsort((void *) dict->entries, dict->nentries, sizeof(DictEntry), sort_by_oids_cb);
+
+ encoded_buff_size = VARHDRSZ + DICT_COMP_HEADER_SIZE + worst_case_compressed_size(jsonb_data_size);
+ encoded_buff = palloc(encoded_buff_size);
+ encoded_header = (uint8 *) VARDATA(encoded_buff);
+ encoded_data = encoded_header + DICT_COMP_HEADER_SIZE;
+
+ DICT_COMP_DICTIONARY_ID(encoded_header) = typmod;
+ DICT_COMP_DECOMPRESSED_SIZE(encoded_header) = jsonb_data_size;
+ DICT_COMP_ALGORITHM_VERSION(encoded_header) = DICT_COMP_CURRENT_ALGORITHM_VERSION;
+
+ encoded_size = encoded_buff_size - VARHDRSZ - DICT_COMP_HEADER_SIZE;
+
+ compress(dict, jsonb_data, jsonb_data_size,
+ encoded_data, &encoded_size);
+
+ encoded_size += VARHDRSZ + DICT_COMP_HEADER_SIZE;
+
+ encoded_buff = repalloc(encoded_buff, encoded_size);
+ SET_VARSIZE(encoded_buff, encoded_size);
+
+ DictEntriesFree(dict);
+
+ PG_RETURN_BYTEA_P(encoded_buff);
+}
+
+/*
+ * Converts a dictionary type to JSONB.
+ */
+Datum
+dictionary_jsonb(PG_FUNCTION_ARGS)
+{
+ bytea *encoded_buff = PG_GETARG_BYTEA_P(0);
+ uint8 *encoded_header = (uint8 *) VARDATA(encoded_buff);
+ uint8 *encoded_data = encoded_header + DICT_COMP_HEADER_SIZE;
+ Size encoded_size = VARSIZE(encoded_buff) - VARHDRSZ - DICT_COMP_HEADER_SIZE;
+ int alg_version = DICT_COMP_ALGORITHM_VERSION(encoded_header);
+ Oid dictOid; /* cannot read until algorithm version is
+ * checked */
+ uint32 decoded_size; /* cannot read until algorithm version is
+ * checked */
+ Jsonb *jsonb;
+ uint8 *jsonb_data;
+ Dictionary dict;
+
+ if (alg_version > DICT_COMP_CURRENT_ALGORITHM_VERSION)
+ ereport(ERROR,
+ (
+ errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("Unsupported compression algorithm version"),
+ errdetail("Saved algorithm version is %d, current version is %d",
+ alg_version, DICT_COMP_CURRENT_ALGORITHM_VERSION),
+ errhint("The data is either corrupted or imported from "
+ "the future version of PostgreSQL")
+ ));
+
+ dictOid = DICT_COMP_DICTIONARY_ID(encoded_header);
+ decoded_size = DICT_COMP_DECOMPRESSED_SIZE(encoded_header);
+
+ dict = DictEntriesRead(dictOid);
+
+ jsonb = palloc(decoded_size + VARHDRSZ);
+ jsonb_data = (uint8 *) VARDATA(jsonb);
+
+ decompress(dict, encoded_data, encoded_size, jsonb_data, decoded_size);
+
+ decoded_size += VARHDRSZ;
+ SET_VARSIZE(jsonb, decoded_size);
+
+ DictEntriesFree(dict);
+
+ PG_RETURN_JSONB_P(jsonb);
+}
+
+/*
+ * Converts a dictionary type to bytea.
+ */
+Datum
+dictionary_bytea(PG_FUNCTION_ARGS)
+{
+ bytea *compressed_data = PG_GETARG_BYTEA_P(0);
+ PG_RETURN_BYTEA_P(compressed_data);
+}
\ No newline at end of file
diff --git a/src/backend/utils/fmgr/funcapi.c b/src/backend/utils/fmgr/funcapi.c
index a1fe50ffca..e20c45f485 100644
--- a/src/backend/utils/fmgr/funcapi.c
+++ b/src/backend/utils/fmgr/funcapi.c
@@ -1292,6 +1292,7 @@ get_type_func_class(Oid typid, Oid *base_typeid)
case TYPTYPE_COMPOSITE:
return TYPEFUNC_COMPOSITE;
case TYPTYPE_BASE:
+ case TYPTYPE_DICT:
case TYPTYPE_ENUM:
case TYPTYPE_RANGE:
case TYPTYPE_MULTIRANGE:
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index d25709ad5f..c0e5c67689 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -10217,6 +10217,8 @@ dumpType(Archive *fout, const TypeInfo *tyinfo)
dumpDomain(fout, tyinfo);
else if (tyinfo->typtype == TYPTYPE_COMPOSITE)
dumpCompositeType(fout, tyinfo);
+ else if (tyinfo->typtype == TYPTYPE_DICT)
+ pg_log_error("AALEKSEEV TODO FIXME not implemented");
else if (tyinfo->typtype == TYPTYPE_ENUM)
dumpEnumType(fout, tyinfo);
else if (tyinfo->typtype == TYPTYPE_RANGE)
diff --git a/src/include/catalog/pg_dict.h b/src/include/catalog/pg_dict.h
new file mode 100644
index 0000000000..fbad803468
--- /dev/null
+++ b/src/include/catalog/pg_dict.h
@@ -0,0 +1,78 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_dict.h
+ * definition of the "dict" system catalog (pg_dict)
+ *
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ * src/include/catalog/pg_dict.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef PG_DICT_H
+#define PG_DICT_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_dict_d.h"
+
+#include "nodes/pg_list.h"
+
+/* ----------------
+ * pg_dict definition. cpp turns this into
+ * typedef struct FormData_pg_dict
+ * ----------------
+ */
+CATALOG(pg_dict,9861,DictRelationId)
+{
+ Oid oid; /* oid */
+ Oid dicttypid BKI_LOOKUP(pg_type); /* OID of owning dict type */
+ /* AALEKSEEV TODO refactor to bytea */
+ NameData dictentry; /* text representation of the dictionary entry */
+} FormData_pg_dict;
+
+/* ----------------
+ * Form_pg_dict corresponds to a pointer to a tuple with
+ * the format of pg_dict relation.
+ * ----------------
+ */
+typedef FormData_pg_dict *Form_pg_dict;
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_dict_oid_index, 9862, DictOidIndexId, on pg_dict using btree(oid oid_ops));
+DECLARE_UNIQUE_INDEX(pg_dict_typid_entry_index, 9863, DictTypIdEntryIndexId, on pg_dict using btree(dicttypid oid_ops, dictentry name_ops));
+
+
+/*
+ * DictEntry type represents one entry in the given dictionary.
+ */
+typedef struct
+{
+ Oid oid; /* entry id */
+ uint32 length; /* entry length */
+ uint8* data; /* entry data */
+} DictEntry;
+
+/*
+ * Dictionary type represents all the entries in the given dictionary.
+ */
+typedef struct
+{
+ uint32 nentries; /* number of entries */
+ DictEntry *entries; /* array of entries */
+} DictionaryData;
+
+typedef DictionaryData *Dictionary;
+
+/*
+ * prototypes for functions in pg_dict.c
+ */
+extern void DictEntriesCreate(Oid dictTypeOid, List *vals);
+extern void DictEntriesFree(Dictionary dict);
+extern Dictionary DictEntriesRead(Oid dictTypeOid);
+extern void DictEntriesDelete(Oid dictTypeOid);
+
+#endif /* PG_DICT_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index a07e737a33..ac69b63e0f 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -9958,6 +9958,26 @@
proname => 'jsonb_path_match_opr', prorettype => 'bool',
proargtypes => 'jsonb jsonpath', prosrc => 'jsonb_path_match_opr' },
+# dictionaries
+{ oid => '9864', descr => 'Converts a cstring to a dictionary',
+ proname => 'dictionary_in', prorettype => 'any', proargtypes => 'cstring oid int4',
+ prosrc => 'dictionary_in' },
+{ oid => '9865', descr => 'Converts a dictionary to a cstring',
+ proname => 'dictionary_out', prorettype => 'cstring', proargtypes => 'any',
+ prosrc => 'dictionary_out' },
+{ oid => '9866', descr => 'Coverts JSONB to a dictionary type',
+ proname => 'jsonb_dictionary', proisstrict => 'f', provolatile => 's',
+ prorettype => 'any', proargtypes => 'jsonb int4',
+ prosrc => 'jsonb_dictionary' },
+{ oid => '9867', descr => 'Converts a dictionary type to JSONB',
+ proname => 'dictionary_jsonb', proisstrict => 'f', provolatile => 's',
+ prorettype => 'jsonb', proargtypes => 'any',
+ prosrc => 'dictionary_jsonb' },
+{ oid => '9868', descr => 'Converts a dictionary type to a byte array',
+ proname => 'dictionary_bytea', proisstrict => 'f', provolatile => 's',
+ prorettype => 'jsonb', proargtypes => 'any',
+ prosrc => 'dictionary_bytea' },
+
# historical int8/txid_snapshot variants of xid8 functions
{ oid => '2939', descr => 'I/O',
proname => 'txid_snapshot_in', prorettype => 'txid_snapshot',
diff --git a/src/include/catalog/pg_type.h b/src/include/catalog/pg_type.h
index 48a2559137..91c6bfae99 100644
--- a/src/include/catalog/pg_type.h
+++ b/src/include/catalog/pg_type.h
@@ -272,6 +272,7 @@ DECLARE_UNIQUE_INDEX(pg_type_typname_nsp_index, 2704, TypeNameNspIndexId, on pg_
*/
#define TYPTYPE_BASE 'b' /* base type (ordinary scalar type) */
#define TYPTYPE_COMPOSITE 'c' /* composite (e.g., table's rowtype) */
+#define TYPTYPE_DICT 'D' /* dictionary */
#define TYPTYPE_DOMAIN 'd' /* domain over another type */
#define TYPTYPE_ENUM 'e' /* enumerated type */
#define TYPTYPE_MULTIRANGE 'm' /* multirange type */
diff --git a/src/include/commands/typecmds.h b/src/include/commands/typecmds.h
index a17bedb851..5cc8e5ff99 100644
--- a/src/include/commands/typecmds.h
+++ b/src/include/commands/typecmds.h
@@ -24,6 +24,7 @@
extern ObjectAddress DefineType(ParseState *pstate, List *names, List *parameters);
extern void RemoveTypeById(Oid typeOid);
extern ObjectAddress DefineDomain(CreateDomainStmt *stmt);
+extern ObjectAddress DefineDictionary(CreateDictionaryStmt *stmt);
extern ObjectAddress DefineEnum(CreateEnumStmt *stmt);
extern ObjectAddress DefineRange(ParseState *pstate, CreateRangeStmt *stmt);
extern ObjectAddress AlterEnum(AlterEnumStmt *stmt);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 6958306a7d..b9a23987fa 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3294,6 +3294,18 @@ typedef struct CompositeTypeStmt
List *coldeflist; /* list of ColumnDef nodes */
} CompositeTypeStmt;
+/* ----------------------
+ * Create Type Statement, dictionary types
+ * ----------------------
+ */
+typedef struct CreateDictionaryStmt
+{
+ NodeTag type;
+ List *typeName; /* qualified name (list of String) */
+ List *baseTypeName; /* qualified base type name (list of String) */
+ List *vals; /* dictionary entries (list of String) */
+} CreateDictionaryStmt;
+
/* ----------------------
* Create Type Statement, enum types
* ----------------------
diff --git a/src/pl/plpgsql/src/pl_comp.c b/src/pl/plpgsql/src/pl_comp.c
index 61fbdf0686..72e3fa5906 100644
--- a/src/pl/plpgsql/src/pl_comp.c
+++ b/src/pl/plpgsql/src/pl_comp.c
@@ -2123,6 +2123,7 @@ build_datatype(HeapTuple typeTup, int32 typmod,
switch (typeStruct->typtype)
{
case TYPTYPE_BASE:
+ case TYPTYPE_DICT:
case TYPTYPE_ENUM:
case TYPTYPE_RANGE:
case TYPTYPE_MULTIRANGE:
diff --git a/src/test/regress/expected/dict.out b/src/test/regress/expected/dict.out
new file mode 100644
index 0000000000..699e76b654
--- /dev/null
+++ b/src/test/regress/expected/dict.out
@@ -0,0 +1,47 @@
+--
+-- Compression dictionaries tests
+--
+-- Dictionary types currently can be used only with JSONB
+\set ON_ERROR_STOP 0
+CREATE TYPE textdict AS DICTIONARY OF TEXT ('abcdef', 'ghijkl');
+ERROR: Dictionary types currently can be used only with JSONB
+\set ON_ERROR_STOP 1
+-- Simple happy path
+CREATE TYPE mydict AS DICTIONARY OF JSONB ('abcdef', 'ghijkl');
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
+ dictentry
+-----------
+ abcdef
+ ghijkl
+(2 rows)
+
+SELECT '{"abcdef":"ghijkl"}' :: mydict;
+ mydict
+----------------------
+ {"abcdef": "ghijkl"}
+(1 row)
+
+SELECT '{"abcdef":"ghijkl"}' :: mydict :: jsonb;
+ jsonb
+----------------------
+ {"abcdef": "ghijkl"}
+(1 row)
+
+SELECT '{"abcdef":"ghijkl"}' :: jsonb :: mydict;
+ mydict
+----------------------
+ {"abcdef": "ghijkl"}
+(1 row)
+
+SELECT ('{"abcdef":"ghijkl"}' :: mydict) -> 'abcdef';
+ ?column?
+----------
+ "ghijkl"
+(1 row)
+
+DROP TYPE mydict;
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
+ dictentry
+-----------
+(0 rows)
+
diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out
index 215eb899be..91bc491a6c 100644
--- a/src/test/regress/expected/oidjoins.out
+++ b/src/test/regress/expected/oidjoins.out
@@ -182,6 +182,7 @@ NOTICE: checking pg_description {classoid} => pg_class {oid}
NOTICE: checking pg_cast {castsource} => pg_type {oid}
NOTICE: checking pg_cast {casttarget} => pg_type {oid}
NOTICE: checking pg_cast {castfunc} => pg_proc {oid}
+NOTICE: checking pg_dict {dicttypid} => pg_type {oid}
NOTICE: checking pg_enum {enumtypid} => pg_type {oid}
NOTICE: checking pg_namespace {nspowner} => pg_authid {oid}
NOTICE: checking pg_conversion {connamespace} => pg_namespace {oid}
diff --git a/src/test/regress/expected/opr_sanity.out b/src/test/regress/expected/opr_sanity.out
index 330eb0f765..f435feb108 100644
--- a/src/test/regress/expected/opr_sanity.out
+++ b/src/test/regress/expected/opr_sanity.out
@@ -388,11 +388,12 @@ WHERE 'cstring'::regtype = ANY (p1.proargtypes)
AND NOT EXISTS(SELECT 1 FROM pg_conversion WHERE conproc = p1.oid)
AND p1.oid != 'shell_in(cstring)'::regprocedure
ORDER BY 1;
- oid | proname
-------+--------------
+ oid | proname
+------+---------------
2293 | cstring_out
2501 | cstring_send
-(2 rows)
+ 9864 | dictionary_in
+(3 rows)
-- Likewise, look for functions that return cstring and aren't datatype output
-- functions nor typmod output functions.
@@ -405,11 +406,12 @@ WHERE p1.prorettype = 'cstring'::regtype
AND NOT EXISTS(SELECT 1 FROM pg_type WHERE typmodout = p1.oid)
AND p1.oid != 'shell_out(void)'::regprocedure
ORDER BY 1;
- oid | proname
-------+--------------
+ oid | proname
+------+----------------
2292 | cstring_in
2500 | cstring_recv
-(2 rows)
+ 9865 | dictionary_out
+(3 rows)
-- Check for length inconsistencies between the various argument-info arrays.
SELECT p1.oid, p1.proname
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 9f644a0c1b..3542f461f9 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -111,7 +111,7 @@ test: select_views portals_p2 foreign_key cluster dependency guc bitmapops combo
# ----------
# Another group of parallel tests (JSON related)
# ----------
-test: json jsonb json_encoding jsonpath jsonpath_encoding jsonb_jsonpath
+test: dict json jsonb json_encoding jsonpath jsonpath_encoding jsonb_jsonpath
# ----------
# Another group of parallel tests
diff --git a/src/test/regress/sql/dict.sql b/src/test/regress/sql/dict.sql
new file mode 100644
index 0000000000..27ecd6a451
--- /dev/null
+++ b/src/test/regress/sql/dict.sql
@@ -0,0 +1,20 @@
+--
+-- Compression dictionaries tests
+--
+
+-- Dictionary types currently can be used only with JSONB
+\set ON_ERROR_STOP 0
+CREATE TYPE textdict AS DICTIONARY OF TEXT ('abcdef', 'ghijkl');
+\set ON_ERROR_STOP 1
+
+-- Simple happy path
+CREATE TYPE mydict AS DICTIONARY OF JSONB ('abcdef', 'ghijkl');
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
+
+SELECT '{"abcdef":"ghijkl"}' :: mydict;
+SELECT '{"abcdef":"ghijkl"}' :: mydict :: jsonb;
+SELECT '{"abcdef":"ghijkl"}' :: jsonb :: mydict;
+SELECT ('{"abcdef":"ghijkl"}' :: mydict) -> 'abcdef';
+
+DROP TYPE mydict;
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
--
2.37.2
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: [PATCH] Compression dictionaries for JSONB
@ 2022-10-06 10:29 Aleksander Alekseev <[email protected]>
parent: Aleksander Alekseev <[email protected]>
0 siblings, 1 reply; 88+ messages in thread
From: Aleksander Alekseev @ 2022-10-06 10:29 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>; +Cc: Simon Riggs <[email protected]>; Nikita Malakhov <[email protected]>; Matthias van de Meent <[email protected]>; Alvaro Herrera <[email protected]>; Zhihong Yu <[email protected]>; Teodor Sigaev <[email protected]>
Hi hackers,
> For the record, Nikita and I agreed offlist that Nikita will join this
> effort as a co-author in order to implement the suggested improvements
> (and perhaps some improvements that were not suggested yet). Meanwhile
> I'm going to keep the current version of the patch up to date with the
> `master` branch.
Here is an updated patch with added Meson support.
--
Best regards,
Aleksander Alekseev
Attachments:
[application/octet-stream] v8-0001-Compression-dictionaries-for-JSONB.patch (49.4K, ../../CAJ7c6TPX05danTdnxqjEH=ohhcO4WBgqVriFwzXWmcoSaDOTmg@mail.gmail.com/2-v8-0001-Compression-dictionaries-for-JSONB.patch)
download | inline diff:
From 57ff1fffce80fe4541fd621acee0965b0941aa84 Mon Sep 17 00:00:00 2001
From: Aleksander Alekseev <[email protected]>
Date: Tue, 12 Apr 2022 15:58:11 +0300
Subject: [PATCH v8] Compression dictionaries for JSONB
Usage example:
CREATE TYPE mydict AS DITIONARY OF jsonb ('aaa', 'bbb', ...);
SELECT ('{"aaa":"bbb"}' :: mydict) -> 'aaa';
The created type works as a drop-in replacement for JSONB. However,
its internal representation differs. The provided dictionary entries ('aaa',
'bbb', ..) are stored in pg_dict catalog table. When `mydict` sees one of
the entries in the document, it replaces it with the corresponding pg_dict
entry Oid. For more details regarding the compression algorithm and choosen
compromises please see the comments in the code.
In pg_type `mydict` has typtype = TYPTYPE_DICT. It works the same way
as TYPTYPE_BASE with only difference: corresponding `<type>_in`
(pg_type.typinput) and `<another-type>_<type>` (pg_cast.castfunc)
procedures receive the dictionary Oid as a `typmod` argument. This way the
procedures can distinguish `mydict1` from `mydict2` and use the proper
compression dictionary.
(NOTE: dear committer, please bump the catalog version!)
Author: Aleksander Alekseev <[email protected]>
Author: TODO FIXME add your name and email here
Reviewed-by: FIXME
Discussion: https://postgr.es/m/CAJ7c6TOtAB0z1UrksvGTStNE-herK-43bj22%3D5xVBg7S4vr5rQ%40mail.gmail.com
Discussion: https://postgr.es/m/CAJ7c6TPx7N-bVw0dZ1ASCDQKZJHhBYkT6w4HV1LzfS%2BUUTUfmA%40mail.gmail.com
---
src/backend/catalog/Makefile | 3 +-
src/backend/catalog/meson.build | 1 +
src/backend/catalog/pg_dict.c | 197 +++++++++
src/backend/commands/typecmds.c | 160 +++++++-
src/backend/executor/functions.c | 1 +
src/backend/parser/gram.y | 8 +
src/backend/parser/parse_coerce.c | 17 +-
src/backend/parser/parse_type.c | 16 +
src/backend/tcop/utility.c | 13 +
src/backend/utils/adt/Makefile | 1 +
src/backend/utils/adt/dictionaries.c | 482 +++++++++++++++++++++++
src/backend/utils/adt/meson.build | 1 +
src/backend/utils/fmgr/funcapi.c | 1 +
src/bin/pg_dump/pg_dump.c | 2 +
src/include/catalog/meson.build | 1 +
src/include/catalog/pg_dict.h | 78 ++++
src/include/catalog/pg_proc.dat | 20 +
src/include/catalog/pg_type.h | 1 +
src/include/commands/typecmds.h | 1 +
src/include/nodes/parsenodes.h | 12 +
src/pl/plpgsql/src/pl_comp.c | 1 +
src/test/regress/expected/dict.out | 47 +++
src/test/regress/expected/oidjoins.out | 1 +
src/test/regress/expected/opr_sanity.out | 14 +-
src/test/regress/parallel_schedule | 2 +-
src/test/regress/sql/dict.sql | 20 +
26 files changed, 1090 insertions(+), 11 deletions(-)
create mode 100644 src/backend/catalog/pg_dict.c
create mode 100644 src/backend/utils/adt/dictionaries.c
create mode 100644 src/include/catalog/pg_dict.h
create mode 100644 src/test/regress/expected/dict.out
create mode 100644 src/test/regress/sql/dict.sql
diff --git a/src/backend/catalog/Makefile b/src/backend/catalog/Makefile
index 89a0221ec9..3b4330148e 100644
--- a/src/backend/catalog/Makefile
+++ b/src/backend/catalog/Makefile
@@ -33,6 +33,7 @@ OBJS = \
pg_conversion.o \
pg_db_role_setting.o \
pg_depend.o \
+ pg_dict.o \
pg_enum.o \
pg_inherits.o \
pg_largeobject.o \
@@ -61,7 +62,7 @@ CATALOG_HEADERS := \
pg_language.h pg_largeobject_metadata.h pg_largeobject.h pg_aggregate.h \
pg_statistic.h pg_statistic_ext.h pg_statistic_ext_data.h \
pg_rewrite.h pg_trigger.h pg_event_trigger.h pg_description.h \
- pg_cast.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \
+ pg_cast.h pg_dict.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \
pg_database.h pg_db_role_setting.h pg_tablespace.h \
pg_authid.h pg_auth_members.h pg_shdepend.h pg_shdescription.h \
pg_ts_config.h pg_ts_config_map.h pg_ts_dict.h \
diff --git a/src/backend/catalog/meson.build b/src/backend/catalog/meson.build
index 0874791451..ab98236c4c 100644
--- a/src/backend/catalog/meson.build
+++ b/src/backend/catalog/meson.build
@@ -18,6 +18,7 @@ backend_sources += files(
'pg_conversion.c',
'pg_db_role_setting.c',
'pg_depend.c',
+ 'pg_dict.c',
'pg_enum.c',
'pg_inherits.c',
'pg_largeobject.c',
diff --git a/src/backend/catalog/pg_dict.c b/src/backend/catalog/pg_dict.c
new file mode 100644
index 0000000000..456f3b5177
--- /dev/null
+++ b/src/backend/catalog/pg_dict.c
@@ -0,0 +1,197 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_dict.c
+ * routines to support manipulation of the pg_dict relation
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/catalog/pg_dict.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "access/genam.h"
+#include "access/skey.h"
+#include "access/table.h"
+#include "catalog/catalog.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_dict.h"
+#include "catalog/pg_type.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/memutils.h"
+#include "utils/rel.h"
+
+/*
+ * Creates an entry in pg_dict for each of the supplied values.
+ * vals is a list of String values.
+ */
+void
+DictEntriesCreate(Oid dictTypeOid, List *vals)
+{
+ Relation pg_dict;
+ NameData dictentry;
+ Datum values[Natts_pg_dict];
+ bool nulls[Natts_pg_dict];
+ ListCell *lc;
+ HeapTuple tup;
+
+ if (vals == NIL)
+ {
+ /* The list is empty; do nothing. */
+ return;
+ }
+
+ memset(nulls, false, sizeof(nulls));
+
+ /*
+ * We don't check the list of values for duplicates here. If there are
+ * any, the user will get an unique-index violation.
+ */
+
+ pg_dict = table_open(DictRelationId, RowExclusiveLock);
+ foreach(lc, vals)
+ {
+ Oid oid = GetNewOidWithIndex(pg_dict, DictOidIndexId,
+ Anum_pg_dict_oid);
+ char *entry = strVal(lfirst(lc));
+
+ /*
+ * Entries are stored in a name field, for easier syscache lookup, so
+ * check the length to make sure it's within range.
+ */
+ if (strlen(entry) > (NAMEDATALEN - 1))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_NAME),
+ errmsg("invalid dict entry \"%s\"", entry),
+ errdetail("Entries must be %d bytes or less.",
+ NAMEDATALEN - 1)));
+
+ /* AALEKSEEV TODO use bytea instead */
+ namestrcpy(&dictentry, entry);
+
+ values[Anum_pg_dict_oid - 1] = ObjectIdGetDatum(oid);
+ values[Anum_pg_dict_dicttypid - 1] = ObjectIdGetDatum(dictTypeOid);
+ values[Anum_pg_dict_dictentry - 1] = NameGetDatum(&dictentry);
+
+ tup = heap_form_tuple(RelationGetDescr(pg_dict), values, nulls);
+
+ CatalogTupleInsert(pg_dict, tup);
+ heap_freetuple(tup);
+ }
+
+ /* clean up */
+ table_close(pg_dict, RowExclusiveLock);
+}
+
+/*
+ * Returns all the entries for the dictinary with given Oid. Entries are sorted
+ * by dictentry. Note that shorter entries are considered smaller, i.e. 'abc'
+ * goes before 'abcdef'. The memory is allocated in a child memory context of
+ * the caller's memory context. It can be freed with DictEntriesFree().
+ *
+ * If there are no entries a valid but empty dictionary is returned.
+ */
+Dictionary
+DictEntriesRead(Oid dictTypeOid)
+{
+ Relation pg_dict;
+ ScanKeyData key[1];
+ SysScanDesc scan;
+ HeapTuple tup;
+ uint32 entries_allocated;
+ Dictionary dict;
+ MemoryContext myctx, oldctx;
+
+ myctx = AllocSetContextCreate(CurrentMemoryContext, "DictEntriesCtx", ALLOCSET_DEFAULT_SIZES);
+ oldctx = MemoryContextSwitchTo(myctx);
+
+ dict = (Dictionary)palloc(sizeof(DictionaryData));
+
+ entries_allocated = 8;
+ dict->nentries = 0;
+ dict->entries = (DictEntry*)palloc(entries_allocated*sizeof(DictEntry));
+
+ pg_dict = table_open(DictRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_dict_dicttypid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(dictTypeOid));
+
+ scan = systable_beginscan(pg_dict, DictTypIdEntryIndexId, true,
+ NULL, 1, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ if(dict->nentries == entries_allocated)
+ {
+ entries_allocated = entries_allocated * 2;
+ dict->entries = (DictEntry*)repalloc(dict->entries, entries_allocated*sizeof(DictEntry));
+ }
+
+ dict->entries[dict->nentries].oid = ((Form_pg_dict) GETSTRUCT(tup))->oid;
+ /*
+ * This is arguably not the fastest way to determine the length of
+ * the entry. Alternatively, we could store a precalculated value in
+ * the catalog. However, usually it's a good idea to keep things simple
+ * until somebody discovers a bottleneck in this exact place and
+ * proposes a concrete fix.
+ */
+ dict->entries[dict->nentries].length = (uint32)strlen(((Form_pg_dict) GETSTRUCT(tup))->dictentry.data);
+ dict->entries[dict->nentries].data = palloc(dict->entries[dict->nentries].length);
+ memcpy(dict->entries[dict->nentries].data, ((Form_pg_dict) GETSTRUCT(tup))->dictentry.data, dict->entries[dict->nentries].length);
+ dict->nentries++;
+ }
+
+ systable_endscan(scan);
+ table_close(pg_dict, RowExclusiveLock);
+
+ MemoryContextSwitchTo(oldctx);
+
+ return dict;
+}
+
+/*
+ * Frees the memory allocated for the dictionary.
+ */
+void
+DictEntriesFree(Dictionary dict)
+{
+ MemoryContextDelete(GetMemoryChunkContext(dict));
+}
+
+/*
+ * Deletes all the entries for the dictinary with given Oid.
+ */
+void
+DictEntriesDelete(Oid dictTypeOid)
+{
+ Relation pg_dict;
+ ScanKeyData key[1];
+ SysScanDesc scan;
+ HeapTuple tup;
+
+ pg_dict = table_open(DictRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_dict_dicttypid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(dictTypeOid));
+
+ scan = systable_beginscan(pg_dict, DictTypIdEntryIndexId, true,
+ NULL, 1, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ CatalogTupleDelete(pg_dict, &tup->t_self);
+ }
+
+ systable_endscan(scan);
+ table_close(pg_dict, RowExclusiveLock);
+}
diff --git a/src/backend/commands/typecmds.c b/src/backend/commands/typecmds.c
index 33b64fd279..2481a05df1 100644
--- a/src/backend/commands/typecmds.c
+++ b/src/backend/commands/typecmds.c
@@ -46,6 +46,7 @@
#include "catalog/pg_collation.h"
#include "catalog/pg_constraint.h"
#include "catalog/pg_depend.h"
+#include "catalog/pg_dict.h"
#include "catalog/pg_enum.h"
#include "catalog/pg_language.h"
#include "catalog/pg_namespace.h"
@@ -679,6 +680,14 @@ RemoveTypeById(Oid typeOid)
if (((Form_pg_type) GETSTRUCT(tup))->typtype == TYPTYPE_RANGE)
RangeDelete(typeOid);
+ /*
+ * If it is a dictionary type, delete the pg_dict entries too; we don't
+ * bother with making a dependency entry for that, so it has to be done
+ * "by hand" here.
+ */
+ if (((Form_pg_type) GETSTRUCT(tup))->typtype == TYPTYPE_DICT)
+ DictEntriesDelete(typeOid);
+
ReleaseSysCache(tup);
table_close(relation, RowExclusiveLock);
@@ -763,8 +772,8 @@ DefineDomain(CreateDomainStmt *stmt)
/*
* Base type must be a plain base type, a composite type, another domain,
- * an enum or a range type. Domains over pseudotypes would create a
- * security hole. (It would be shorter to code this to just check for
+ * a dict, an enum or a range type. Domains over pseudotypes would create
+ * a security hole. (It would be shorter to code this to just check for
* pseudotypes; but it seems safer to call out the specific typtypes that
* are supported, rather than assume that all future typtypes would be
* automatically supported.)
@@ -772,6 +781,7 @@ DefineDomain(CreateDomainStmt *stmt)
typtype = baseType->typtype;
if (typtype != TYPTYPE_BASE &&
typtype != TYPTYPE_COMPOSITE &&
+ typtype != TYPTYPE_DICT &&
typtype != TYPTYPE_DOMAIN &&
typtype != TYPTYPE_ENUM &&
typtype != TYPTYPE_RANGE &&
@@ -1129,6 +1139,152 @@ DefineDomain(CreateDomainStmt *stmt)
}
+/*
+ * DefineDictionary
+ * Registers a new dictionary.
+ */
+ObjectAddress
+DefineDictionary(CreateDictionaryStmt *stmt)
+{
+ char *dictName;
+ char *dictBaseTypeName;
+ char *dictArrayName;
+ Oid dictNamespace;
+ AclResult aclresult;
+ Oid old_type_oid;
+ Oid dictArrayOid;
+ ObjectAddress dictTypeAddr;
+
+ Assert(list_length(stmt->baseTypeName) == 1);
+ dictBaseTypeName = strVal(linitial(stmt->baseTypeName));
+ if(pg_strcasecmp(dictBaseTypeName, "jsonb") != 0)
+ ereport(ERROR,
+ (
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("Dictionary types currently can be used only with JSONB")
+ )
+ );
+
+ /* Convert list of names to a name and namespace */
+ dictNamespace = QualifiedNameGetCreationNamespace(stmt->typeName,
+ &dictName);
+
+ /* Check we have creation rights in target namespace */
+ aclresult = pg_namespace_aclcheck(dictNamespace, GetUserId(), ACL_CREATE);
+ if (aclresult != ACLCHECK_OK)
+ aclcheck_error(aclresult, OBJECT_SCHEMA,
+ get_namespace_name(dictNamespace));
+
+ /*
+ * Check for collision with an existing type name. If there is one and
+ * it's an autogenerated array, we can rename it out of the way.
+ */
+ old_type_oid = GetSysCacheOid2(TYPENAMENSP, Anum_pg_type_oid,
+ CStringGetDatum(dictName),
+ ObjectIdGetDatum(dictNamespace));
+ if (OidIsValid(old_type_oid))
+ {
+ if (!moveArrayTypeName(old_type_oid, dictName, dictNamespace))
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("type \"%s\" already exists", dictName)));
+ }
+
+ /* Allocate OID for array type */
+ dictArrayOid = AssignTypeArrayOid();
+
+ /* Create the pg_type entry */
+ dictTypeAddr =
+ TypeCreate(InvalidOid, /* no predetermined type OID */
+ dictName, /* type name */
+ dictNamespace, /* namespace */
+ InvalidOid, /* relation oid (n/a here) */
+ 0, /* relation kind (ditto) */
+ GetUserId(), /* owner's ID */
+ -1, /* internal size: varlena, as for JSONB */
+ TYPTYPE_DICT, /* type-type: dictionary */
+ TYPCATEGORY_USER, /* type-category: user, as for JSONB */
+ false, /* dict types are never preferred */
+ DEFAULT_TYPDELIM, /* array element delimiter */
+ F_DICTIONARY_IN, /* input procedure */
+ F_DICTIONARY_OUT, /* output procedure */
+ InvalidOid, /* receive procedure: none */
+ InvalidOid, /* send procedure: none */
+ InvalidOid, /* typmodin procedure - none */
+ InvalidOid, /* typmodout procedure - none */
+ InvalidOid, /* analyze procedure - default */
+ InvalidOid, /* subscript procedure - none */
+ InvalidOid, /* element type ID */
+ false, /* this is not an array type */
+ dictArrayOid, /* array type we are about to create */
+ InvalidOid, /* base type ID (only for domains) */
+ NULL, /* never a default type value */
+ NULL, /* binary default isn't sent either */
+ false, /* passed by value: same as for JSONB */
+ TYPALIGN_INT, /* int alignment */
+ TYPSTORAGE_EXTENDED, /* TOAST strategy: fully toastable, as
+ * JSONB */
+ -1, /* typMod (Domains only) */
+ 0, /* Array dimensions of typbasetype */
+ false, /* Type NOT NULL */
+ InvalidOid); /* type's collation */
+
+ /*
+ * Create the array type that goes with it.
+ */
+ dictArrayName = makeArrayTypeName(dictName, dictNamespace);
+
+ TypeCreate(dictArrayOid, /* force assignment of this type OID */
+ dictArrayName, /* type name */
+ dictNamespace, /* namespace */
+ InvalidOid, /* relation oid (n/a here) */
+ 0, /* relation kind (ditto) */
+ GetUserId(), /* owner's ID */
+ -1, /* internal size (always varlena) */
+ TYPTYPE_BASE, /* type-type (base type) */
+ TYPCATEGORY_ARRAY, /* type-category (array) */
+ false, /* array types are never preferred */
+ DEFAULT_TYPDELIM, /* array element delimiter */
+ F_ARRAY_IN, /* input procedure */
+ F_ARRAY_OUT, /* output procedure */
+ F_ARRAY_RECV, /* receive procedure */
+ F_ARRAY_SEND, /* send procedure */
+ InvalidOid, /* typmodin procedure - none */
+ InvalidOid, /* typmodout procedure - none */
+ F_ARRAY_TYPANALYZE, /* analyze procedure */
+ F_ARRAY_SUBSCRIPT_HANDLER, /* array subscript procedure */
+ dictTypeAddr.objectId, /* element type ID */
+ true, /* yes this is an array type */
+ InvalidOid, /* no further array type */
+ InvalidOid, /* base type ID */
+ NULL, /* never a default type value */
+ NULL, /* binary default isn't sent either */
+ false, /* passed by value: same as for JSONB */
+ TYPALIGN_INT, /* enums have int align, so do their arrays */
+ TYPSTORAGE_EXTENDED, /* ARRAY is always toastable */
+ -1, /* typMod (Domains only) */
+ 0, /* Array dimensions of typbasetype */
+ false, /* Type NOT NULL */
+ InvalidOid); /* type's collation */
+
+ pfree(dictArrayName);
+
+ /* Enter the dict's entries into pg_dict */
+ DictEntriesCreate(dictTypeAddr.objectId, stmt->vals);
+
+ /* Create casts to and from JSONB */
+ CastCreate(JSONBOID, dictTypeAddr.objectId, F_JSONB_DICTIONARY, 'a', 'f', DEPENDENCY_INTERNAL);
+ CastCreate(dictTypeAddr.objectId, JSONBOID, F_DICTIONARY_JSONB, 'i', 'f', DEPENDENCY_INTERNAL);
+
+ /*
+ * Create explicit cast to bytea. This is convenient for debugging purposes.
+ * Casting bytea to a dictionary type is dangerous and thus not supported.
+ */
+ CastCreate(dictTypeAddr.objectId, BYTEAOID, F_DICTIONARY_BYTEA, 'e', 'f', DEPENDENCY_INTERNAL);
+
+ return dictTypeAddr;
+}
+
/*
* DefineEnum
* Registers a new enum.
diff --git a/src/backend/executor/functions.c b/src/backend/executor/functions.c
index e134a82ff7..9674cdfe4d 100644
--- a/src/backend/executor/functions.c
+++ b/src/backend/executor/functions.c
@@ -1721,6 +1721,7 @@ check_sql_fn_retval(List *queryTreeLists,
if (fn_typtype == TYPTYPE_BASE ||
fn_typtype == TYPTYPE_DOMAIN ||
+ fn_typtype == TYPTYPE_DICT ||
fn_typtype == TYPTYPE_ENUM ||
fn_typtype == TYPTYPE_RANGE ||
fn_typtype == TYPTYPE_MULTIRANGE)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 94d5142a4a..7a673a139c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -6152,6 +6152,14 @@ DefineStmt:
n->coldeflist = $6;
$$ = (Node *) n;
}
+ | CREATE TYPE_P any_name AS DICTIONARY OF any_name '(' opt_enum_val_list ')'
+ {
+ CreateDictionaryStmt *n = makeNode(CreateDictionaryStmt);
+ n->typeName = $3;
+ n->baseTypeName = $7;
+ n->vals = $9;
+ $$ = (Node *)n;
+ }
| CREATE TYPE_P any_name AS ENUM_P '(' opt_enum_val_list ')'
{
CreateEnumStmt *n = makeNode(CreateEnumStmt);
diff --git a/src/backend/parser/parse_coerce.c b/src/backend/parser/parse_coerce.c
index c4e958e4aa..e489dd9421 100644
--- a/src/backend/parser/parse_coerce.c
+++ b/src/backend/parser/parse_coerce.c
@@ -279,7 +279,22 @@ coerce_type(ParseState *pstate, Node *node,
if (baseTypeId == INTERVALOID)
inputTypeMod = baseTypeMod;
else
- inputTypeMod = -1;
+ {
+ if((int32)targetTypeId == targetTypeMod)
+ {
+ /*
+ * For dictionaries internally we set typmod to a dictionary
+ * OID. This allows dictionary_in() and jsonb_dictionary() to
+ * distinguish one dictionary from another.
+ *
+ * It also allows us to recognize that the target type is
+ * a dictionary here and pass a correct typmod.
+ */
+ inputTypeMod = targetTypeMod;
+ }
+ else
+ inputTypeMod = -1;
+ }
baseType = typeidType(baseTypeId);
diff --git a/src/backend/parser/parse_type.c b/src/backend/parser/parse_type.c
index f7ad689459..2188002535 100644
--- a/src/backend/parser/parse_type.c
+++ b/src/backend/parser/parse_type.c
@@ -259,6 +259,10 @@ LookupTypeNameOid(ParseState *pstate, const TypeName *typeName, bool missing_ok)
* This is equivalent to LookupTypeName, except that this will report
* a suitable error message if the type cannot be found or is not defined.
* Callers of this can therefore assume the result is a fully valid type.
+ *
+ * For a dictionary type it's OID is returned as a typmod. This is used by
+ * dictionary_in() and jsonb_dictionary() to distinguish one dictionary from
+ * another.
*/
Type
typenameType(ParseState *pstate, const TypeName *typeName, int32 *typmod_p)
@@ -305,6 +309,10 @@ typenameTypeId(ParseState *pstate, const TypeName *typeName)
*
* This is equivalent to typenameType, but we only hand back the type OID
* and typmod, not the syscache entry.
+ *
+ * For a dictionary type it's OID is returned as a typmod. This is used by
+ * dictionary_in() and jsonb_dictionary() to distinguish one dictionary from
+ * another.
*/
void
typenameTypeIdAndMod(ParseState *pstate, const TypeName *typeName,
@@ -327,6 +335,10 @@ typenameTypeIdAndMod(ParseState *pstate, const TypeName *typeName,
* looked up, and is passed as "typ".
*
* pstate is only used for error location info, and may be NULL.
+ *
+ * For a dictionary type it's OID is returned as a typmod. This is used by
+ * dictionary_in() and jsonb_dictionary() to distinguish one dictionary from
+ * another.
*/
static int32
typenameTypeMod(ParseState *pstate, const TypeName *typeName, Type typ)
@@ -339,6 +351,10 @@ typenameTypeMod(ParseState *pstate, const TypeName *typeName, Type typ)
ArrayType *arrtypmod;
ParseCallbackState pcbstate;
+ /* For a dictionary return it's OID. */
+ if (((Form_pg_type) GETSTRUCT(typ))->typtype == TYPTYPE_DICT)
+ return (int32)((Form_pg_type) GETSTRUCT(typ))->oid;
+
/* Return prespecified typmod if no typmod expressions */
if (typeName->typmods == NIL)
return typeName->typemod;
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index 247d0816ad..ec8a70c3fc 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -174,6 +174,7 @@ ClassifyUtilityCommandAsReadOnly(Node *parsetree)
case T_CreateCastStmt:
case T_CreateConversionStmt:
case T_CreateDomainStmt:
+ case T_CreateDictionaryStmt:
case T_CreateEnumStmt:
case T_CreateEventTrigStmt:
case T_CreateExtensionStmt:
@@ -1621,6 +1622,10 @@ ProcessUtilitySlow(ParseState *pstate,
}
break;
+ case T_CreateDictionaryStmt: /* CREATE TYPE AS DICTIONARY OF */
+ address = DefineDictionary((CreateDictionaryStmt *) parsetree);
+ break;
+
case T_CreateEnumStmt: /* CREATE TYPE AS ENUM */
address = DefineEnum((CreateEnumStmt *) parsetree);
break;
@@ -2772,6 +2777,10 @@ CreateCommandTag(Node *parsetree)
tag = CMDTAG_CREATE_TYPE;
break;
+ case T_CreateDictionaryStmt:
+ tag = CMDTAG_CREATE_TYPE;
+ break;
+
case T_CreateEnumStmt:
tag = CMDTAG_CREATE_TYPE;
break;
@@ -3419,6 +3428,10 @@ GetCommandLogLevel(Node *parsetree)
lev = LOGSTMT_DDL;
break;
+ case T_CreateDictionaryStmt:
+ lev = LOGSTMT_DDL;
+ break;
+
case T_CreateEnumStmt:
lev = LOGSTMT_DDL;
break;
diff --git a/src/backend/utils/adt/Makefile b/src/backend/utils/adt/Makefile
index 0de0bbb1b8..58e25fd8bc 100644
--- a/src/backend/utils/adt/Makefile
+++ b/src/backend/utils/adt/Makefile
@@ -30,6 +30,7 @@ OBJS = \
datetime.o \
datum.o \
dbsize.o \
+ dictionaries.o \
domains.o \
encode.o \
enum.o \
diff --git a/src/backend/utils/adt/dictionaries.c b/src/backend/utils/adt/dictionaries.c
new file mode 100644
index 0000000000..2bb46471a4
--- /dev/null
+++ b/src/backend/utils/adt/dictionaries.c
@@ -0,0 +1,482 @@
+/*-------------------------------------------------------------------------
+ *
+ * dictionaries.c
+ * Conversion functions for dictionary types.
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/adt/dictionaries.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "catalog/pg_dict.h"
+#include "utils/fmgrprotos.h"
+#include "utils/jsonb.h"
+
+/*
+ * When compressing a data we treat it as a BLOB, in other words we don't
+ * assume anything regarding its internal representation. This is not
+ * necessarily the best and/or the only possible approach. However, it is
+ * universal and can be reused for JSONB, TEXT, XML and other types. Alternative
+ * compression methods can be introduced in the future, so that the user will
+ * be able to choose the best one for the task.
+ *
+ * The compressed data is stored in the following format:
+ *
+ * (struct varlena)
+ * dictionary_id [uint32]
+ * decompressed_size [uint32]
+ * algorithm_version [uint8]
+ *
+ * (repeated) {
+ * number of bytes to copy as-is [uint8]
+ * ... bytes to copy as-is ...
+ * dictionary entry id, or 0 to skip [uint32]
+ * }
+ *
+ * Compressed data is a variable-length type and thus has the 'struct varlena'
+ * header. The code below doesn't consider it being a part of the payload
+ * (e.g. see the DICT_COMP_HEADER_SIZE definition).
+ *
+ * Storing dictionary_id may seem redundant, but without it dictionary_out()
+ * and dictionary_jsonb() have no way to distinguish one dictionary type from
+ * another. dictionary_in() and jsonb_dictionary() get the dictionary id through
+ * the 'typemod' argument.
+ *
+ * Currently algorithm_version is always 0. In the future it will allow us to
+ * introduce new features (e.g. usage of varints) and/or lazily migrate the
+ * data to other compression methods.
+ *
+ * Compression and decompression are implemented based on a binary search over
+ * DictEntry[] array (see pg_dict.h). For compression the array is sorted by
+ * dictentries and for decompression - by oid's.
+ *
+ */
+
+/* Size of the header in the compressed data */
+#define DICT_COMP_HEADER_SIZE (sizeof(uint32)*2 + sizeof(uint8))
+
+/* Extracts dictionary_id from the compressed data */
+#define DICT_COMP_DICTIONARY_ID(hdrp) \
+ (*(uint32*)hdrp)
+
+/* Extracts decompressed_size from the compressed data */
+#define DICT_COMP_DECOMPRESSED_SIZE(hdrp) \
+ (*(uint32*)((uint8*)hdrp + sizeof(uint32)))
+
+/* Extracts algorithm_version from the compressed data */
+#define DICT_COMP_ALGORITHM_VERSION(hdrp) \
+ (*(uint8*)((uint8*)hdrp + sizeof(uint32)*2))
+
+/* Current algorithm_version */
+#define DICT_COMP_CURRENT_ALGORITHM_VERSION 0
+
+/*
+ * bsearch_arg() callback for finding DictEntry by oid.
+ */
+static int
+find_by_oid_cb(const void *key, const void *current, void *arg)
+{
+ /* Note that oids are unsigned, so we should be careful here */
+ if (*(Oid *) key < ((DictEntry *) current)->oid)
+ return -1;
+ else if (*(Oid *) key == ((DictEntry *) current)->oid)
+ return 0; /* found! */
+ else
+ return 1;
+}
+
+/*
+ * qsort() callback for sorting DictEntry[] by oids.
+ */
+static int
+sort_by_oids_cb(const void *left, const void *right)
+{
+ /* Note that oids are unsigned, so we should be careful here */
+ if (((DictEntry *) left)->oid < ((DictEntry *) right)->oid)
+ return -1;
+
+ /* Oids are unique so this callback will never return 0 */
+ return 1;
+}
+
+/*
+ * Finds a DictEntry which dictentry field matches *data and returns its Oid.
+ * If there are several matching entries, the largest is returned. The length
+ * of the found entry is written to *found_length on success. On failure
+ * InvalidOid is returned and found_length is zeroed.
+ *
+ * The implementation is similar to bsearch_arg(). The procedure can't be used
+ * directly because we are looking not by the exact match. We could generalize
+ * this case but the signature of the function becomes so complicated that it
+ * doesn't seem to worth the effort.
+ */
+static Oid
+compress_find_oid(Dictionary dict, const uint8 *data, Size data_size, Size *found_length)
+{
+ int res;
+ int32 left = 0;
+ int32 right = dict->nentries - 1;
+ Size best_length = 0;
+ Oid best_match = InvalidOid;
+
+ while (left <= right)
+ {
+ int32 current = (left + right) / 2;
+ Size nbytes = (Size)dict->entries[current].length;
+
+ if (nbytes > data_size)
+ {
+ /* current can be less or greater depending on the prefix */
+ res = memcmp(dict->entries[current].data, data, data_size);
+
+ /* if prefixes match, current is greater */
+ if (res == 0)
+ res = 1;
+ }
+ else
+ res = memcmp(dict->entries[current].data, data, nbytes);
+
+ if (res == 0) /* match found */
+ {
+ best_length = nbytes;
+ best_match = dict->entries[current].oid;
+
+ if (nbytes == data_size)
+ break;
+
+ /* maybe there is a larger match */
+ left = current + 1;
+ }
+ else if (res < 0) /* current is less */
+ left = current + 1;
+ else /* current is greater */
+ right = current - 1;
+ }
+
+ *found_length = best_length;
+ return best_match;
+}
+
+/*
+ * Finds a DictEntry by Oid using a binary search. The dictionary should be
+ * sorted by oids before the call. Returns NULL if nothing was found.
+ */
+static DictEntry *
+decompress_find_dictentry(Dictionary dict, Oid oid)
+{
+ return (DictEntry *) bsearch_arg(&oid, dict->entries, dict->nentries, sizeof(DictEntry), find_by_oid_cb, NULL);
+}
+
+/*
+ * Estimates the worst-case compressed size of the data of given size.
+ * Worst-case scenario happens when the dictionary consists of single-character
+ * entries. In this case every byte will be encoded as 6 bytes:
+ * 0x00, (0 bytes to copy as-is), 4 bytes of the entry Oid
+ *
+ * This procedure doesn't account for the header size.
+ *
+ * AALEKSEEV FIXME don't use dictionary entries shorter than 5 bytes
+ */
+static Size
+worst_case_compressed_size(Size insize)
+{
+ return insize * 6;
+}
+
+/*
+ * Compresses the data using the provided dictionary. The dictionary should
+ * be sorted by dictentries before the call. Output buffer should be at least
+ * worst_case_compressed_size(src_size) bytes in size.
+ */
+static void
+compress(Dictionary dict,
+ const void *src_data_, Size src_size,
+ void *encoded_data_, Size *pencoded_size)
+{
+ Size nbytes;
+ Size inoffset;
+ Size outskipoffset = 0;
+ Size outoffset = 1;
+ uint8 skipbytes = 0;
+ const uint8 *src_data = src_data_;
+ uint8 *encoded_data = ((uint8 *) encoded_data_);
+
+ for (inoffset = 0; inoffset < src_size;)
+ {
+ Oid code = compress_find_oid(dict, &(src_data[inoffset]),
+ src_size - inoffset, &nbytes);
+
+ if (code == InvalidOid)
+ {
+ skipbytes++;
+ encoded_data[outoffset] = src_data[inoffset];
+ outoffset++;
+ inoffset++;
+
+ if (skipbytes == 255)
+ {
+ encoded_data[outskipoffset] = skipbytes;
+ encoded_data[outoffset++] = 0; /* InvalidOid */
+ encoded_data[outoffset++] = 0;
+ encoded_data[outoffset++] = 0;
+ encoded_data[outoffset++] = 0;
+ outskipoffset = outoffset++;
+ skipbytes = 0;
+ }
+ }
+ else
+ {
+ encoded_data[outskipoffset] = skipbytes;
+ encoded_data[outoffset++] = (code >> 24) & 0xFF;
+ encoded_data[outoffset++] = (code >> 16) & 0xFF;
+ encoded_data[outoffset++] = (code >> 8) & 0xFF;
+ encoded_data[outoffset++] = code & 0xFF;
+ outskipoffset = outoffset++;
+ skipbytes = 0;
+ inoffset += nbytes;
+ }
+ }
+
+ /* Double check that we didn't write out of buffer */
+ Assert(outoffset < worst_case_compressed_size(src_size));
+
+ encoded_data[outskipoffset] = skipbytes;
+ *pencoded_size = outoffset;
+}
+
+/*
+ * Report an internal error in decompress() procedure below.
+ * Under normal circumstances this should never happen.
+ */
+static void
+decompress_error()
+{
+ ereport(ERROR,
+ (
+ errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("Unable to decompress a dictionary type"),
+ errdetail("The compressed data seems to be corrupted"),
+ errhint("Please report the steps to reproduce the issue to pgsql-bugs@")
+ ));
+}
+
+/*
+ * Decompresses the data using the provided dictionary. The dictionary should
+ * be sorted by oids before the call.
+ */
+static void
+decompress(Dictionary dict,
+ const void *encoded_data_, Size encoded_size,
+ void *decoded_data_, Size decoded_size)
+{
+ Size inoffset = 0;
+ Size outoffset = 0;
+ Oid code;
+ uint8 skipbytes;
+ const uint8 *encoded_data = ((uint8 *) encoded_data_);
+ uint8 *decoded_data = decoded_data_;
+
+ for (inoffset = 0; inoffset < encoded_size;)
+ {
+ skipbytes = encoded_data[inoffset++];
+
+ if (skipbytes > decoded_size - outoffset)
+ decompress_error();
+
+ if (skipbytes > encoded_size - inoffset)
+ decompress_error();
+
+ memcpy(
+ &(decoded_data[outoffset]),
+ &(encoded_data[inoffset]),
+ skipbytes
+ );
+
+ outoffset += skipbytes;
+ inoffset += skipbytes;
+
+ if ((encoded_size == inoffset) && (decoded_size == outoffset))
+ break; /* end of input - its OK */
+
+ if (encoded_size - inoffset < 4)
+ decompress_error();
+
+ code = (Oid) encoded_data[inoffset++];
+ code = (code << 8) | (Oid) encoded_data[inoffset++];
+ code = (code << 8) | (Oid) encoded_data[inoffset++];
+ code = (code << 8) | (Oid) encoded_data[inoffset++];
+
+ if (code != InvalidOid)
+ {
+ Size entrylen;
+ DictEntry *entry = decompress_find_dictentry(dict, code);
+
+ if (entry == NULL)
+ decompress_error();
+
+ Assert(entry->oid == code);
+ entrylen = (Size)entry->length;
+
+ if (entrylen > decoded_size - outoffset)
+ decompress_error();
+
+ memcpy(
+ &(decoded_data[outoffset]),
+ entry->data,
+ entrylen
+ );
+
+ outoffset += entrylen;
+ }
+ }
+
+ Assert(decoded_size == outoffset);
+}
+
+/*
+ * Converts a cstring to a dictionary.
+ */
+Datum
+dictionary_in(PG_FUNCTION_ARGS)
+{
+ const char *instr = PG_GETARG_CSTRING(0);
+#ifdef NOT_USED
+ Oid typelem = PG_GETARG_OID(1);
+#endif
+ int32 typmod = PG_GETARG_INT32(2);
+ Jsonb *jsonb;
+ bytea *dict;
+
+ Assert(typmod != -1);
+
+ jsonb = DatumGetJsonbP(DirectFunctionCall1(jsonb_in, CStringGetDatum(instr)));
+ dict = DatumGetByteaP(DirectFunctionCall2(jsonb_dictionary, JsonbPGetDatum(jsonb), Int32GetDatum(typmod)));
+
+ PG_RETURN_BYTEA_P(dict);
+}
+
+/*
+ * Converts a dictionary to a cstring.
+ */
+Datum
+dictionary_out(PG_FUNCTION_ARGS)
+{
+ bytea *dict = PG_GETARG_BYTEA_P(0);
+ Jsonb *jsonb = DatumGetJsonbP(DirectFunctionCall1(dictionary_jsonb, PointerGetDatum(dict)));
+ const char *outstr = DatumGetCString(DirectFunctionCall1(jsonb_out, JsonbPGetDatum(jsonb)));
+
+ PG_RETURN_CSTRING(outstr);
+}
+
+/*
+ * Coverts JSONB to a dictionary type.
+ *
+ * AALEKSEEV FIXME: if the compressed document ends up being larger than
+ * the original one write a corresponding flag and copy all the data as-is.
+ */
+Datum
+jsonb_dictionary(PG_FUNCTION_ARGS)
+{
+ Dictionary dict;
+ Jsonb *jsonb = PG_GETARG_JSONB_P(0);
+ int32 typmod = PG_GETARG_INT32(1);
+ uint8 *jsonb_data = (uint8 *) VARDATA(jsonb);
+ Size jsonb_data_size = VARSIZE(jsonb) - VARHDRSZ;
+ uint8 *encoded_buff,
+ *encoded_header,
+ *encoded_data;
+ Size encoded_size,
+ encoded_buff_size;
+
+ Assert(typmod != -1);
+
+ dict = DictEntriesRead(typmod);
+ qsort((void *) dict->entries, dict->nentries, sizeof(DictEntry), sort_by_oids_cb);
+
+ encoded_buff_size = VARHDRSZ + DICT_COMP_HEADER_SIZE + worst_case_compressed_size(jsonb_data_size);
+ encoded_buff = palloc(encoded_buff_size);
+ encoded_header = (uint8 *) VARDATA(encoded_buff);
+ encoded_data = encoded_header + DICT_COMP_HEADER_SIZE;
+
+ DICT_COMP_DICTIONARY_ID(encoded_header) = typmod;
+ DICT_COMP_DECOMPRESSED_SIZE(encoded_header) = jsonb_data_size;
+ DICT_COMP_ALGORITHM_VERSION(encoded_header) = DICT_COMP_CURRENT_ALGORITHM_VERSION;
+
+ encoded_size = encoded_buff_size - VARHDRSZ - DICT_COMP_HEADER_SIZE;
+
+ compress(dict, jsonb_data, jsonb_data_size,
+ encoded_data, &encoded_size);
+
+ encoded_size += VARHDRSZ + DICT_COMP_HEADER_SIZE;
+
+ encoded_buff = repalloc(encoded_buff, encoded_size);
+ SET_VARSIZE(encoded_buff, encoded_size);
+
+ DictEntriesFree(dict);
+
+ PG_RETURN_BYTEA_P(encoded_buff);
+}
+
+/*
+ * Converts a dictionary type to JSONB.
+ */
+Datum
+dictionary_jsonb(PG_FUNCTION_ARGS)
+{
+ bytea *encoded_buff = PG_GETARG_BYTEA_P(0);
+ uint8 *encoded_header = (uint8 *) VARDATA(encoded_buff);
+ uint8 *encoded_data = encoded_header + DICT_COMP_HEADER_SIZE;
+ Size encoded_size = VARSIZE(encoded_buff) - VARHDRSZ - DICT_COMP_HEADER_SIZE;
+ int alg_version = DICT_COMP_ALGORITHM_VERSION(encoded_header);
+ Oid dictOid; /* cannot read until algorithm version is
+ * checked */
+ uint32 decoded_size; /* cannot read until algorithm version is
+ * checked */
+ Jsonb *jsonb;
+ uint8 *jsonb_data;
+ Dictionary dict;
+
+ if (alg_version > DICT_COMP_CURRENT_ALGORITHM_VERSION)
+ ereport(ERROR,
+ (
+ errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("Unsupported compression algorithm version"),
+ errdetail("Saved algorithm version is %d, current version is %d",
+ alg_version, DICT_COMP_CURRENT_ALGORITHM_VERSION),
+ errhint("The data is either corrupted or imported from "
+ "the future version of PostgreSQL")
+ ));
+
+ dictOid = DICT_COMP_DICTIONARY_ID(encoded_header);
+ decoded_size = DICT_COMP_DECOMPRESSED_SIZE(encoded_header);
+
+ dict = DictEntriesRead(dictOid);
+
+ jsonb = palloc(decoded_size + VARHDRSZ);
+ jsonb_data = (uint8 *) VARDATA(jsonb);
+
+ decompress(dict, encoded_data, encoded_size, jsonb_data, decoded_size);
+
+ decoded_size += VARHDRSZ;
+ SET_VARSIZE(jsonb, decoded_size);
+
+ DictEntriesFree(dict);
+
+ PG_RETURN_JSONB_P(jsonb);
+}
+
+/*
+ * Converts a dictionary type to bytea.
+ */
+Datum
+dictionary_bytea(PG_FUNCTION_ARGS)
+{
+ bytea *compressed_data = PG_GETARG_BYTEA_P(0);
+ PG_RETURN_BYTEA_P(compressed_data);
+}
\ No newline at end of file
diff --git a/src/backend/utils/adt/meson.build b/src/backend/utils/adt/meson.build
index ed9ceadfef..2ed48c1406 100644
--- a/src/backend/utils/adt/meson.build
+++ b/src/backend/utils/adt/meson.build
@@ -17,6 +17,7 @@ backend_sources += files(
'datetime.c',
'datum.c',
'dbsize.c',
+ 'dictionaries.c',
'domains.c',
'encode.c',
'enum.c',
diff --git a/src/backend/utils/fmgr/funcapi.c b/src/backend/utils/fmgr/funcapi.c
index a1fe50ffca..e20c45f485 100644
--- a/src/backend/utils/fmgr/funcapi.c
+++ b/src/backend/utils/fmgr/funcapi.c
@@ -1292,6 +1292,7 @@ get_type_func_class(Oid typid, Oid *base_typeid)
case TYPTYPE_COMPOSITE:
return TYPEFUNC_COMPOSITE;
case TYPTYPE_BASE:
+ case TYPTYPE_DICT:
case TYPTYPE_ENUM:
case TYPTYPE_RANGE:
case TYPTYPE_MULTIRANGE:
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index bd9b066e4e..109190d3a8 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -10218,6 +10218,8 @@ dumpType(Archive *fout, const TypeInfo *tyinfo)
dumpDomain(fout, tyinfo);
else if (tyinfo->typtype == TYPTYPE_COMPOSITE)
dumpCompositeType(fout, tyinfo);
+ else if (tyinfo->typtype == TYPTYPE_DICT)
+ pg_log_error("AALEKSEEV TODO FIXME not implemented");
else if (tyinfo->typtype == TYPTYPE_ENUM)
dumpEnumType(fout, tyinfo);
else if (tyinfo->typtype == TYPTYPE_RANGE)
diff --git a/src/include/catalog/meson.build b/src/include/catalog/meson.build
index 45ffa99692..8d76de46f3 100644
--- a/src/include/catalog/meson.build
+++ b/src/include/catalog/meson.build
@@ -25,6 +25,7 @@ catalog_headers = [
'pg_event_trigger.h',
'pg_description.h',
'pg_cast.h',
+ 'pg_dict.h',
'pg_enum.h',
'pg_namespace.h',
'pg_conversion.h',
diff --git a/src/include/catalog/pg_dict.h b/src/include/catalog/pg_dict.h
new file mode 100644
index 0000000000..fbad803468
--- /dev/null
+++ b/src/include/catalog/pg_dict.h
@@ -0,0 +1,78 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_dict.h
+ * definition of the "dict" system catalog (pg_dict)
+ *
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ * src/include/catalog/pg_dict.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef PG_DICT_H
+#define PG_DICT_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_dict_d.h"
+
+#include "nodes/pg_list.h"
+
+/* ----------------
+ * pg_dict definition. cpp turns this into
+ * typedef struct FormData_pg_dict
+ * ----------------
+ */
+CATALOG(pg_dict,9861,DictRelationId)
+{
+ Oid oid; /* oid */
+ Oid dicttypid BKI_LOOKUP(pg_type); /* OID of owning dict type */
+ /* AALEKSEEV TODO refactor to bytea */
+ NameData dictentry; /* text representation of the dictionary entry */
+} FormData_pg_dict;
+
+/* ----------------
+ * Form_pg_dict corresponds to a pointer to a tuple with
+ * the format of pg_dict relation.
+ * ----------------
+ */
+typedef FormData_pg_dict *Form_pg_dict;
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_dict_oid_index, 9862, DictOidIndexId, on pg_dict using btree(oid oid_ops));
+DECLARE_UNIQUE_INDEX(pg_dict_typid_entry_index, 9863, DictTypIdEntryIndexId, on pg_dict using btree(dicttypid oid_ops, dictentry name_ops));
+
+
+/*
+ * DictEntry type represents one entry in the given dictionary.
+ */
+typedef struct
+{
+ Oid oid; /* entry id */
+ uint32 length; /* entry length */
+ uint8* data; /* entry data */
+} DictEntry;
+
+/*
+ * Dictionary type represents all the entries in the given dictionary.
+ */
+typedef struct
+{
+ uint32 nentries; /* number of entries */
+ DictEntry *entries; /* array of entries */
+} DictionaryData;
+
+typedef DictionaryData *Dictionary;
+
+/*
+ * prototypes for functions in pg_dict.c
+ */
+extern void DictEntriesCreate(Oid dictTypeOid, List *vals);
+extern void DictEntriesFree(Dictionary dict);
+extern Dictionary DictEntriesRead(Oid dictTypeOid);
+extern void DictEntriesDelete(Oid dictTypeOid);
+
+#endif /* PG_DICT_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 68bb032d3e..b2ca9d6524 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -9961,6 +9961,26 @@
proname => 'jsonb_path_match_opr', prorettype => 'bool',
proargtypes => 'jsonb jsonpath', prosrc => 'jsonb_path_match_opr' },
+# dictionaries
+{ oid => '9864', descr => 'Converts a cstring to a dictionary',
+ proname => 'dictionary_in', prorettype => 'any', proargtypes => 'cstring oid int4',
+ prosrc => 'dictionary_in' },
+{ oid => '9865', descr => 'Converts a dictionary to a cstring',
+ proname => 'dictionary_out', prorettype => 'cstring', proargtypes => 'any',
+ prosrc => 'dictionary_out' },
+{ oid => '9866', descr => 'Coverts JSONB to a dictionary type',
+ proname => 'jsonb_dictionary', proisstrict => 'f', provolatile => 's',
+ prorettype => 'any', proargtypes => 'jsonb int4',
+ prosrc => 'jsonb_dictionary' },
+{ oid => '9867', descr => 'Converts a dictionary type to JSONB',
+ proname => 'dictionary_jsonb', proisstrict => 'f', provolatile => 's',
+ prorettype => 'jsonb', proargtypes => 'any',
+ prosrc => 'dictionary_jsonb' },
+{ oid => '9868', descr => 'Converts a dictionary type to a byte array',
+ proname => 'dictionary_bytea', proisstrict => 'f', provolatile => 's',
+ prorettype => 'jsonb', proargtypes => 'any',
+ prosrc => 'dictionary_bytea' },
+
# historical int8/txid_snapshot variants of xid8 functions
{ oid => '2939', descr => 'I/O',
proname => 'txid_snapshot_in', prorettype => 'txid_snapshot',
diff --git a/src/include/catalog/pg_type.h b/src/include/catalog/pg_type.h
index 48a2559137..91c6bfae99 100644
--- a/src/include/catalog/pg_type.h
+++ b/src/include/catalog/pg_type.h
@@ -272,6 +272,7 @@ DECLARE_UNIQUE_INDEX(pg_type_typname_nsp_index, 2704, TypeNameNspIndexId, on pg_
*/
#define TYPTYPE_BASE 'b' /* base type (ordinary scalar type) */
#define TYPTYPE_COMPOSITE 'c' /* composite (e.g., table's rowtype) */
+#define TYPTYPE_DICT 'D' /* dictionary */
#define TYPTYPE_DOMAIN 'd' /* domain over another type */
#define TYPTYPE_ENUM 'e' /* enumerated type */
#define TYPTYPE_MULTIRANGE 'm' /* multirange type */
diff --git a/src/include/commands/typecmds.h b/src/include/commands/typecmds.h
index 532bc49c38..a3ef87741d 100644
--- a/src/include/commands/typecmds.h
+++ b/src/include/commands/typecmds.h
@@ -24,6 +24,7 @@
extern ObjectAddress DefineType(ParseState *pstate, List *names, List *parameters);
extern void RemoveTypeById(Oid typeOid);
extern ObjectAddress DefineDomain(CreateDomainStmt *stmt);
+extern ObjectAddress DefineDictionary(CreateDictionaryStmt *stmt);
extern ObjectAddress DefineEnum(CreateEnumStmt *stmt);
extern ObjectAddress DefineRange(ParseState *pstate, CreateRangeStmt *stmt);
extern ObjectAddress AlterEnum(AlterEnumStmt *stmt);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 633e7671b3..575445a750 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3295,6 +3295,18 @@ typedef struct CompositeTypeStmt
List *coldeflist; /* list of ColumnDef nodes */
} CompositeTypeStmt;
+/* ----------------------
+ * Create Type Statement, dictionary types
+ * ----------------------
+ */
+typedef struct CreateDictionaryStmt
+{
+ NodeTag type;
+ List *typeName; /* qualified name (list of String) */
+ List *baseTypeName; /* qualified base type name (list of String) */
+ List *vals; /* dictionary entries (list of String) */
+} CreateDictionaryStmt;
+
/* ----------------------
* Create Type Statement, enum types
* ----------------------
diff --git a/src/pl/plpgsql/src/pl_comp.c b/src/pl/plpgsql/src/pl_comp.c
index b286f2a50c..2754098f7f 100644
--- a/src/pl/plpgsql/src/pl_comp.c
+++ b/src/pl/plpgsql/src/pl_comp.c
@@ -2123,6 +2123,7 @@ build_datatype(HeapTuple typeTup, int32 typmod,
switch (typeStruct->typtype)
{
case TYPTYPE_BASE:
+ case TYPTYPE_DICT:
case TYPTYPE_ENUM:
case TYPTYPE_RANGE:
case TYPTYPE_MULTIRANGE:
diff --git a/src/test/regress/expected/dict.out b/src/test/regress/expected/dict.out
new file mode 100644
index 0000000000..699e76b654
--- /dev/null
+++ b/src/test/regress/expected/dict.out
@@ -0,0 +1,47 @@
+--
+-- Compression dictionaries tests
+--
+-- Dictionary types currently can be used only with JSONB
+\set ON_ERROR_STOP 0
+CREATE TYPE textdict AS DICTIONARY OF TEXT ('abcdef', 'ghijkl');
+ERROR: Dictionary types currently can be used only with JSONB
+\set ON_ERROR_STOP 1
+-- Simple happy path
+CREATE TYPE mydict AS DICTIONARY OF JSONB ('abcdef', 'ghijkl');
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
+ dictentry
+-----------
+ abcdef
+ ghijkl
+(2 rows)
+
+SELECT '{"abcdef":"ghijkl"}' :: mydict;
+ mydict
+----------------------
+ {"abcdef": "ghijkl"}
+(1 row)
+
+SELECT '{"abcdef":"ghijkl"}' :: mydict :: jsonb;
+ jsonb
+----------------------
+ {"abcdef": "ghijkl"}
+(1 row)
+
+SELECT '{"abcdef":"ghijkl"}' :: jsonb :: mydict;
+ mydict
+----------------------
+ {"abcdef": "ghijkl"}
+(1 row)
+
+SELECT ('{"abcdef":"ghijkl"}' :: mydict) -> 'abcdef';
+ ?column?
+----------
+ "ghijkl"
+(1 row)
+
+DROP TYPE mydict;
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
+ dictentry
+-----------
+(0 rows)
+
diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out
index 215eb899be..91bc491a6c 100644
--- a/src/test/regress/expected/oidjoins.out
+++ b/src/test/regress/expected/oidjoins.out
@@ -182,6 +182,7 @@ NOTICE: checking pg_description {classoid} => pg_class {oid}
NOTICE: checking pg_cast {castsource} => pg_type {oid}
NOTICE: checking pg_cast {casttarget} => pg_type {oid}
NOTICE: checking pg_cast {castfunc} => pg_proc {oid}
+NOTICE: checking pg_dict {dicttypid} => pg_type {oid}
NOTICE: checking pg_enum {enumtypid} => pg_type {oid}
NOTICE: checking pg_namespace {nspowner} => pg_authid {oid}
NOTICE: checking pg_conversion {connamespace} => pg_namespace {oid}
diff --git a/src/test/regress/expected/opr_sanity.out b/src/test/regress/expected/opr_sanity.out
index 330eb0f765..f435feb108 100644
--- a/src/test/regress/expected/opr_sanity.out
+++ b/src/test/regress/expected/opr_sanity.out
@@ -388,11 +388,12 @@ WHERE 'cstring'::regtype = ANY (p1.proargtypes)
AND NOT EXISTS(SELECT 1 FROM pg_conversion WHERE conproc = p1.oid)
AND p1.oid != 'shell_in(cstring)'::regprocedure
ORDER BY 1;
- oid | proname
-------+--------------
+ oid | proname
+------+---------------
2293 | cstring_out
2501 | cstring_send
-(2 rows)
+ 9864 | dictionary_in
+(3 rows)
-- Likewise, look for functions that return cstring and aren't datatype output
-- functions nor typmod output functions.
@@ -405,11 +406,12 @@ WHERE p1.prorettype = 'cstring'::regtype
AND NOT EXISTS(SELECT 1 FROM pg_type WHERE typmodout = p1.oid)
AND p1.oid != 'shell_out(void)'::regprocedure
ORDER BY 1;
- oid | proname
-------+--------------
+ oid | proname
+------+----------------
2292 | cstring_in
2500 | cstring_recv
-(2 rows)
+ 9865 | dictionary_out
+(3 rows)
-- Check for length inconsistencies between the various argument-info arrays.
SELECT p1.oid, p1.proname
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 9f644a0c1b..3542f461f9 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -111,7 +111,7 @@ test: select_views portals_p2 foreign_key cluster dependency guc bitmapops combo
# ----------
# Another group of parallel tests (JSON related)
# ----------
-test: json jsonb json_encoding jsonpath jsonpath_encoding jsonb_jsonpath
+test: dict json jsonb json_encoding jsonpath jsonpath_encoding jsonb_jsonpath
# ----------
# Another group of parallel tests
diff --git a/src/test/regress/sql/dict.sql b/src/test/regress/sql/dict.sql
new file mode 100644
index 0000000000..27ecd6a451
--- /dev/null
+++ b/src/test/regress/sql/dict.sql
@@ -0,0 +1,20 @@
+--
+-- Compression dictionaries tests
+--
+
+-- Dictionary types currently can be used only with JSONB
+\set ON_ERROR_STOP 0
+CREATE TYPE textdict AS DICTIONARY OF TEXT ('abcdef', 'ghijkl');
+\set ON_ERROR_STOP 1
+
+-- Simple happy path
+CREATE TYPE mydict AS DICTIONARY OF JSONB ('abcdef', 'ghijkl');
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
+
+SELECT '{"abcdef":"ghijkl"}' :: mydict;
+SELECT '{"abcdef":"ghijkl"}' :: mydict :: jsonb;
+SELECT '{"abcdef":"ghijkl"}' :: jsonb :: mydict;
+SELECT ('{"abcdef":"ghijkl"}' :: mydict) -> 'abcdef';
+
+DROP TYPE mydict;
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
--
2.37.3
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: [PATCH] Compression dictionaries for JSONB
@ 2022-11-04 08:37 Aleksander Alekseev <[email protected]>
parent: Aleksander Alekseev <[email protected]>
0 siblings, 1 reply; 88+ messages in thread
From: Aleksander Alekseev @ 2022-11-04 08:37 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>; +Cc: Simon Riggs <[email protected]>; Nikita Malakhov <[email protected]>; Matthias van de Meent <[email protected]>; Alvaro Herrera <[email protected]>; Zhihong Yu <[email protected]>; Teodor Sigaev <[email protected]>
Hi hackers,
> For the record, Nikita and I agreed offlist that Nikita will join this
> effort as a co-author in order to implement the suggested improvements
> (and perhaps some improvements that were not suggested yet). Meanwhile
> I'm going to keep the current version of the patch up to date with the
> `master` branch.
8272749e added a few more arguments to CastCreate(). Here is the rebased patch.
--
Best regards,
Aleksander Alekseev
Attachments:
[application/octet-stream] v9-0001-Compression-dictionaries-for-JSONB.patch (49.7K, ../../CAJ7c6TMhHCNBBFtYMa6d-SU=j0PdpW0Mwp7rtF_fm3HpCsOsKw@mail.gmail.com/2-v9-0001-Compression-dictionaries-for-JSONB.patch)
download | inline diff:
From 9f606df68379556b5bf155fcc69f979d7b9f743e Mon Sep 17 00:00:00 2001
From: Aleksander Alekseev <[email protected]>
Date: Tue, 12 Apr 2022 15:58:11 +0300
Subject: [PATCH v9] Compression dictionaries for JSONB
Usage example:
CREATE TYPE mydict AS DITIONARY OF jsonb ('aaa', 'bbb', ...);
SELECT ('{"aaa":"bbb"}' :: mydict) -> 'aaa';
The created type works as a drop-in replacement for JSONB. However,
its internal representation differs. The provided dictionary entries ('aaa',
'bbb', ..) are stored in pg_dict catalog table. When `mydict` sees one of
the entries in the document, it replaces it with the corresponding pg_dict
entry Oid. For more details regarding the compression algorithm and choosen
compromises please see the comments in the code.
In pg_type `mydict` has typtype = TYPTYPE_DICT. It works the same way
as TYPTYPE_BASE with only difference: corresponding `<type>_in`
(pg_type.typinput) and `<another-type>_<type>` (pg_cast.castfunc)
procedures receive the dictionary Oid as a `typmod` argument. This way the
procedures can distinguish `mydict1` from `mydict2` and use the proper
compression dictionary.
(NOTE: dear committer, please bump the catalog version!)
Author: Aleksander Alekseev <[email protected]>
Reviewed-by: Matthias van de Meent, Nikita Malakhov
Discussion: https://postgr.es/m/CAJ7c6TOtAB0z1UrksvGTStNE-herK-43bj22%3D5xVBg7S4vr5rQ%40mail.gmail.com
Discussion: https://postgr.es/m/CAJ7c6TPx7N-bVw0dZ1ASCDQKZJHhBYkT6w4HV1LzfS%2BUUTUfmA%40mail.gmail.com
---
src/backend/catalog/Makefile | 3 +-
src/backend/catalog/meson.build | 1 +
src/backend/catalog/pg_dict.c | 197 +++++++++
src/backend/commands/typecmds.c | 171 +++++++-
src/backend/executor/functions.c | 1 +
src/backend/parser/gram.y | 8 +
src/backend/parser/parse_coerce.c | 17 +-
src/backend/parser/parse_type.c | 16 +
src/backend/tcop/utility.c | 13 +
src/backend/utils/adt/Makefile | 1 +
src/backend/utils/adt/dictionaries.c | 482 +++++++++++++++++++++++
src/backend/utils/adt/meson.build | 1 +
src/backend/utils/fmgr/funcapi.c | 1 +
src/bin/pg_dump/pg_dump.c | 2 +
src/include/catalog/meson.build | 1 +
src/include/catalog/pg_dict.h | 78 ++++
src/include/catalog/pg_proc.dat | 20 +
src/include/catalog/pg_type.h | 1 +
src/include/commands/typecmds.h | 1 +
src/include/nodes/parsenodes.h | 12 +
src/pl/plpgsql/src/pl_comp.c | 1 +
src/test/regress/expected/dict.out | 47 +++
src/test/regress/expected/oidjoins.out | 1 +
src/test/regress/expected/opr_sanity.out | 14 +-
src/test/regress/parallel_schedule | 2 +-
src/test/regress/sql/dict.sql | 20 +
26 files changed, 1101 insertions(+), 11 deletions(-)
create mode 100644 src/backend/catalog/pg_dict.c
create mode 100644 src/backend/utils/adt/dictionaries.c
create mode 100644 src/include/catalog/pg_dict.h
create mode 100644 src/test/regress/expected/dict.out
create mode 100644 src/test/regress/sql/dict.sql
diff --git a/src/backend/catalog/Makefile b/src/backend/catalog/Makefile
index 89a0221ec9..3b4330148e 100644
--- a/src/backend/catalog/Makefile
+++ b/src/backend/catalog/Makefile
@@ -33,6 +33,7 @@ OBJS = \
pg_conversion.o \
pg_db_role_setting.o \
pg_depend.o \
+ pg_dict.o \
pg_enum.o \
pg_inherits.o \
pg_largeobject.o \
@@ -61,7 +62,7 @@ CATALOG_HEADERS := \
pg_language.h pg_largeobject_metadata.h pg_largeobject.h pg_aggregate.h \
pg_statistic.h pg_statistic_ext.h pg_statistic_ext_data.h \
pg_rewrite.h pg_trigger.h pg_event_trigger.h pg_description.h \
- pg_cast.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \
+ pg_cast.h pg_dict.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \
pg_database.h pg_db_role_setting.h pg_tablespace.h \
pg_authid.h pg_auth_members.h pg_shdepend.h pg_shdescription.h \
pg_ts_config.h pg_ts_config_map.h pg_ts_dict.h \
diff --git a/src/backend/catalog/meson.build b/src/backend/catalog/meson.build
index 0874791451..ab98236c4c 100644
--- a/src/backend/catalog/meson.build
+++ b/src/backend/catalog/meson.build
@@ -18,6 +18,7 @@ backend_sources += files(
'pg_conversion.c',
'pg_db_role_setting.c',
'pg_depend.c',
+ 'pg_dict.c',
'pg_enum.c',
'pg_inherits.c',
'pg_largeobject.c',
diff --git a/src/backend/catalog/pg_dict.c b/src/backend/catalog/pg_dict.c
new file mode 100644
index 0000000000..456f3b5177
--- /dev/null
+++ b/src/backend/catalog/pg_dict.c
@@ -0,0 +1,197 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_dict.c
+ * routines to support manipulation of the pg_dict relation
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/catalog/pg_dict.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "access/genam.h"
+#include "access/skey.h"
+#include "access/table.h"
+#include "catalog/catalog.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_dict.h"
+#include "catalog/pg_type.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/memutils.h"
+#include "utils/rel.h"
+
+/*
+ * Creates an entry in pg_dict for each of the supplied values.
+ * vals is a list of String values.
+ */
+void
+DictEntriesCreate(Oid dictTypeOid, List *vals)
+{
+ Relation pg_dict;
+ NameData dictentry;
+ Datum values[Natts_pg_dict];
+ bool nulls[Natts_pg_dict];
+ ListCell *lc;
+ HeapTuple tup;
+
+ if (vals == NIL)
+ {
+ /* The list is empty; do nothing. */
+ return;
+ }
+
+ memset(nulls, false, sizeof(nulls));
+
+ /*
+ * We don't check the list of values for duplicates here. If there are
+ * any, the user will get an unique-index violation.
+ */
+
+ pg_dict = table_open(DictRelationId, RowExclusiveLock);
+ foreach(lc, vals)
+ {
+ Oid oid = GetNewOidWithIndex(pg_dict, DictOidIndexId,
+ Anum_pg_dict_oid);
+ char *entry = strVal(lfirst(lc));
+
+ /*
+ * Entries are stored in a name field, for easier syscache lookup, so
+ * check the length to make sure it's within range.
+ */
+ if (strlen(entry) > (NAMEDATALEN - 1))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_NAME),
+ errmsg("invalid dict entry \"%s\"", entry),
+ errdetail("Entries must be %d bytes or less.",
+ NAMEDATALEN - 1)));
+
+ /* AALEKSEEV TODO use bytea instead */
+ namestrcpy(&dictentry, entry);
+
+ values[Anum_pg_dict_oid - 1] = ObjectIdGetDatum(oid);
+ values[Anum_pg_dict_dicttypid - 1] = ObjectIdGetDatum(dictTypeOid);
+ values[Anum_pg_dict_dictentry - 1] = NameGetDatum(&dictentry);
+
+ tup = heap_form_tuple(RelationGetDescr(pg_dict), values, nulls);
+
+ CatalogTupleInsert(pg_dict, tup);
+ heap_freetuple(tup);
+ }
+
+ /* clean up */
+ table_close(pg_dict, RowExclusiveLock);
+}
+
+/*
+ * Returns all the entries for the dictinary with given Oid. Entries are sorted
+ * by dictentry. Note that shorter entries are considered smaller, i.e. 'abc'
+ * goes before 'abcdef'. The memory is allocated in a child memory context of
+ * the caller's memory context. It can be freed with DictEntriesFree().
+ *
+ * If there are no entries a valid but empty dictionary is returned.
+ */
+Dictionary
+DictEntriesRead(Oid dictTypeOid)
+{
+ Relation pg_dict;
+ ScanKeyData key[1];
+ SysScanDesc scan;
+ HeapTuple tup;
+ uint32 entries_allocated;
+ Dictionary dict;
+ MemoryContext myctx, oldctx;
+
+ myctx = AllocSetContextCreate(CurrentMemoryContext, "DictEntriesCtx", ALLOCSET_DEFAULT_SIZES);
+ oldctx = MemoryContextSwitchTo(myctx);
+
+ dict = (Dictionary)palloc(sizeof(DictionaryData));
+
+ entries_allocated = 8;
+ dict->nentries = 0;
+ dict->entries = (DictEntry*)palloc(entries_allocated*sizeof(DictEntry));
+
+ pg_dict = table_open(DictRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_dict_dicttypid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(dictTypeOid));
+
+ scan = systable_beginscan(pg_dict, DictTypIdEntryIndexId, true,
+ NULL, 1, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ if(dict->nentries == entries_allocated)
+ {
+ entries_allocated = entries_allocated * 2;
+ dict->entries = (DictEntry*)repalloc(dict->entries, entries_allocated*sizeof(DictEntry));
+ }
+
+ dict->entries[dict->nentries].oid = ((Form_pg_dict) GETSTRUCT(tup))->oid;
+ /*
+ * This is arguably not the fastest way to determine the length of
+ * the entry. Alternatively, we could store a precalculated value in
+ * the catalog. However, usually it's a good idea to keep things simple
+ * until somebody discovers a bottleneck in this exact place and
+ * proposes a concrete fix.
+ */
+ dict->entries[dict->nentries].length = (uint32)strlen(((Form_pg_dict) GETSTRUCT(tup))->dictentry.data);
+ dict->entries[dict->nentries].data = palloc(dict->entries[dict->nentries].length);
+ memcpy(dict->entries[dict->nentries].data, ((Form_pg_dict) GETSTRUCT(tup))->dictentry.data, dict->entries[dict->nentries].length);
+ dict->nentries++;
+ }
+
+ systable_endscan(scan);
+ table_close(pg_dict, RowExclusiveLock);
+
+ MemoryContextSwitchTo(oldctx);
+
+ return dict;
+}
+
+/*
+ * Frees the memory allocated for the dictionary.
+ */
+void
+DictEntriesFree(Dictionary dict)
+{
+ MemoryContextDelete(GetMemoryChunkContext(dict));
+}
+
+/*
+ * Deletes all the entries for the dictinary with given Oid.
+ */
+void
+DictEntriesDelete(Oid dictTypeOid)
+{
+ Relation pg_dict;
+ ScanKeyData key[1];
+ SysScanDesc scan;
+ HeapTuple tup;
+
+ pg_dict = table_open(DictRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_dict_dicttypid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(dictTypeOid));
+
+ scan = systable_beginscan(pg_dict, DictTypIdEntryIndexId, true,
+ NULL, 1, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ CatalogTupleDelete(pg_dict, &tup->t_self);
+ }
+
+ systable_endscan(scan);
+ table_close(pg_dict, RowExclusiveLock);
+}
diff --git a/src/backend/commands/typecmds.c b/src/backend/commands/typecmds.c
index b7c3dded17..e0d82416ea 100644
--- a/src/backend/commands/typecmds.c
+++ b/src/backend/commands/typecmds.c
@@ -46,6 +46,7 @@
#include "catalog/pg_collation.h"
#include "catalog/pg_constraint.h"
#include "catalog/pg_depend.h"
+#include "catalog/pg_dict.h"
#include "catalog/pg_enum.h"
#include "catalog/pg_language.h"
#include "catalog/pg_namespace.h"
@@ -679,6 +680,14 @@ RemoveTypeById(Oid typeOid)
if (((Form_pg_type) GETSTRUCT(tup))->typtype == TYPTYPE_RANGE)
RangeDelete(typeOid);
+ /*
+ * If it is a dictionary type, delete the pg_dict entries too; we don't
+ * bother with making a dependency entry for that, so it has to be done
+ * "by hand" here.
+ */
+ if (((Form_pg_type) GETSTRUCT(tup))->typtype == TYPTYPE_DICT)
+ DictEntriesDelete(typeOid);
+
ReleaseSysCache(tup);
table_close(relation, RowExclusiveLock);
@@ -763,8 +772,8 @@ DefineDomain(CreateDomainStmt *stmt)
/*
* Base type must be a plain base type, a composite type, another domain,
- * an enum or a range type. Domains over pseudotypes would create a
- * security hole. (It would be shorter to code this to just check for
+ * a dict, an enum or a range type. Domains over pseudotypes would create
+ * a security hole. (It would be shorter to code this to just check for
* pseudotypes; but it seems safer to call out the specific typtypes that
* are supported, rather than assume that all future typtypes would be
* automatically supported.)
@@ -772,6 +781,7 @@ DefineDomain(CreateDomainStmt *stmt)
typtype = baseType->typtype;
if (typtype != TYPTYPE_BASE &&
typtype != TYPTYPE_COMPOSITE &&
+ typtype != TYPTYPE_DICT &&
typtype != TYPTYPE_DOMAIN &&
typtype != TYPTYPE_ENUM &&
typtype != TYPTYPE_RANGE &&
@@ -1129,6 +1139,163 @@ DefineDomain(CreateDomainStmt *stmt)
}
+/*
+ * DefineDictionary
+ * Registers a new dictionary.
+ */
+ObjectAddress
+DefineDictionary(CreateDictionaryStmt * stmt)
+{
+ char *dictName;
+ char *dictBaseTypeName;
+ char *dictArrayName;
+ Oid dictNamespace;
+ AclResult aclresult;
+ Oid old_type_oid;
+ Oid dictArrayOid;
+ ObjectAddress dictTypeAddr;
+
+ Assert(list_length(stmt->baseTypeName) == 1);
+ dictBaseTypeName = strVal(linitial(stmt->baseTypeName));
+ if (pg_strcasecmp(dictBaseTypeName, "jsonb") != 0)
+ ereport(ERROR,
+ (
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("Dictionary types currently can be used only with JSONB")
+ )
+ );
+
+ /* Convert list of names to a name and namespace */
+ dictNamespace = QualifiedNameGetCreationNamespace(stmt->typeName,
+ &dictName);
+
+ /* Check we have creation rights in target namespace */
+ aclresult = pg_namespace_aclcheck(dictNamespace, GetUserId(), ACL_CREATE);
+ if (aclresult != ACLCHECK_OK)
+ aclcheck_error(aclresult, OBJECT_SCHEMA,
+ get_namespace_name(dictNamespace));
+
+ /*
+ * Check for collision with an existing type name. If there is one and
+ * it's an autogenerated array, we can rename it out of the way.
+ */
+ old_type_oid = GetSysCacheOid2(TYPENAMENSP, Anum_pg_type_oid,
+ CStringGetDatum(dictName),
+ ObjectIdGetDatum(dictNamespace));
+ if (OidIsValid(old_type_oid))
+ {
+ if (!moveArrayTypeName(old_type_oid, dictName, dictNamespace))
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("type \"%s\" already exists", dictName)));
+ }
+
+ /* Allocate OID for array type */
+ dictArrayOid = AssignTypeArrayOid();
+
+ /* Create the pg_type entry */
+ dictTypeAddr =
+ TypeCreate(InvalidOid, /* no predetermined type OID */
+ dictName, /* type name */
+ dictNamespace, /* namespace */
+ InvalidOid, /* relation oid (n/a here) */
+ 0, /* relation kind (ditto) */
+ GetUserId(), /* owner's ID */
+ -1, /* internal size: varlena, as for JSONB */
+ TYPTYPE_DICT, /* type-type: dictionary */
+ TYPCATEGORY_USER, /* type-category: user, as for JSONB */
+ false, /* dict types are never preferred */
+ DEFAULT_TYPDELIM, /* array element delimiter */
+ F_DICTIONARY_IN, /* input procedure */
+ F_DICTIONARY_OUT, /* output procedure */
+ InvalidOid, /* receive procedure: none */
+ InvalidOid, /* send procedure: none */
+ InvalidOid, /* typmodin procedure - none */
+ InvalidOid, /* typmodout procedure - none */
+ InvalidOid, /* analyze procedure - default */
+ InvalidOid, /* subscript procedure - none */
+ InvalidOid, /* element type ID */
+ false, /* this is not an array type */
+ dictArrayOid, /* array type we are about to create */
+ InvalidOid, /* base type ID (only for domains) */
+ NULL, /* never a default type value */
+ NULL, /* binary default isn't sent either */
+ false, /* passed by value: same as for JSONB */
+ TYPALIGN_INT, /* int alignment */
+ TYPSTORAGE_EXTENDED, /* TOAST strategy: fully toastable, as
+ * JSONB */
+ -1, /* typMod (Domains only) */
+ 0, /* Array dimensions of typbasetype */
+ false, /* Type NOT NULL */
+ InvalidOid); /* type's collation */
+
+ /*
+ * Create the array type that goes with it.
+ */
+ dictArrayName = makeArrayTypeName(dictName, dictNamespace);
+
+ TypeCreate(dictArrayOid, /* force assignment of this type OID */
+ dictArrayName, /* type name */
+ dictNamespace, /* namespace */
+ InvalidOid, /* relation oid (n/a here) */
+ 0, /* relation kind (ditto) */
+ GetUserId(), /* owner's ID */
+ -1, /* internal size (always varlena) */
+ TYPTYPE_BASE, /* type-type (base type) */
+ TYPCATEGORY_ARRAY, /* type-category (array) */
+ false, /* array types are never preferred */
+ DEFAULT_TYPDELIM, /* array element delimiter */
+ F_ARRAY_IN, /* input procedure */
+ F_ARRAY_OUT, /* output procedure */
+ F_ARRAY_RECV, /* receive procedure */
+ F_ARRAY_SEND, /* send procedure */
+ InvalidOid, /* typmodin procedure - none */
+ InvalidOid, /* typmodout procedure - none */
+ F_ARRAY_TYPANALYZE, /* analyze procedure */
+ F_ARRAY_SUBSCRIPT_HANDLER, /* array subscript procedure */
+ dictTypeAddr.objectId, /* element type ID */
+ true, /* yes this is an array type */
+ InvalidOid, /* no further array type */
+ InvalidOid, /* base type ID */
+ NULL, /* never a default type value */
+ NULL, /* binary default isn't sent either */
+ false, /* passed by value: same as for JSONB */
+ TYPALIGN_INT, /* enums have int align, so do their arrays */
+ TYPSTORAGE_EXTENDED, /* ARRAY is always toastable */
+ -1, /* typMod (Domains only) */
+ 0, /* Array dimensions of typbasetype */
+ false, /* Type NOT NULL */
+ InvalidOid); /* type's collation */
+
+ pfree(dictArrayName);
+
+ /* Enter the dict's entries into pg_dict */
+ DictEntriesCreate(dictTypeAddr.objectId, stmt->vals);
+
+ /* Create casts to and from JSONB */
+ CastCreate(JSONBOID, dictTypeAddr.objectId, F_JSONB_DICTIONARY,
+ InvalidOid, InvalidOid,
+ COERCION_CODE_ASSIGNMENT, COERCION_METHOD_FUNCTION,
+ DEPENDENCY_INTERNAL);
+
+ CastCreate(dictTypeAddr.objectId, JSONBOID, F_DICTIONARY_JSONB,
+ InvalidOid, InvalidOid,
+ COERCION_METHOD_INOUT, COERCION_METHOD_FUNCTION,
+ DEPENDENCY_INTERNAL);
+
+ /*
+ * Create explicit cast to bytea. This is convenient for debugging
+ * purposes. Casting bytea to a dictionary type is dangerous and thus not
+ * supported.
+ */
+ CastCreate(dictTypeAddr.objectId, BYTEAOID, F_DICTIONARY_BYTEA,
+ InvalidOid, InvalidOid,
+ COERCION_CODE_EXPLICIT, COERCION_METHOD_FUNCTION,
+ DEPENDENCY_INTERNAL);
+
+ return dictTypeAddr;
+}
+
/*
* DefineEnum
* Registers a new enum.
diff --git a/src/backend/executor/functions.c b/src/backend/executor/functions.c
index e134a82ff7..9674cdfe4d 100644
--- a/src/backend/executor/functions.c
+++ b/src/backend/executor/functions.c
@@ -1721,6 +1721,7 @@ check_sql_fn_retval(List *queryTreeLists,
if (fn_typtype == TYPTYPE_BASE ||
fn_typtype == TYPTYPE_DOMAIN ||
+ fn_typtype == TYPTYPE_DICT ||
fn_typtype == TYPTYPE_ENUM ||
fn_typtype == TYPTYPE_RANGE ||
fn_typtype == TYPTYPE_MULTIRANGE)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 6ca23f88c4..4aec75e55f 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -6153,6 +6153,14 @@ DefineStmt:
n->coldeflist = $6;
$$ = (Node *) n;
}
+ | CREATE TYPE_P any_name AS DICTIONARY OF any_name '(' opt_enum_val_list ')'
+ {
+ CreateDictionaryStmt *n = makeNode(CreateDictionaryStmt);
+ n->typeName = $3;
+ n->baseTypeName = $7;
+ n->vals = $9;
+ $$ = (Node *)n;
+ }
| CREATE TYPE_P any_name AS ENUM_P '(' opt_enum_val_list ')'
{
CreateEnumStmt *n = makeNode(CreateEnumStmt);
diff --git a/src/backend/parser/parse_coerce.c b/src/backend/parser/parse_coerce.c
index 60908111c8..8952e1469c 100644
--- a/src/backend/parser/parse_coerce.c
+++ b/src/backend/parser/parse_coerce.c
@@ -279,7 +279,22 @@ coerce_type(ParseState *pstate, Node *node,
if (baseTypeId == INTERVALOID)
inputTypeMod = baseTypeMod;
else
- inputTypeMod = -1;
+ {
+ if((int32)targetTypeId == targetTypeMod)
+ {
+ /*
+ * For dictionaries internally we set typmod to a dictionary
+ * OID. This allows dictionary_in() and jsonb_dictionary() to
+ * distinguish one dictionary from another.
+ *
+ * It also allows us to recognize that the target type is
+ * a dictionary here and pass a correct typmod.
+ */
+ inputTypeMod = targetTypeMod;
+ }
+ else
+ inputTypeMod = -1;
+ }
baseType = typeidType(baseTypeId);
diff --git a/src/backend/parser/parse_type.c b/src/backend/parser/parse_type.c
index f7ad689459..2188002535 100644
--- a/src/backend/parser/parse_type.c
+++ b/src/backend/parser/parse_type.c
@@ -259,6 +259,10 @@ LookupTypeNameOid(ParseState *pstate, const TypeName *typeName, bool missing_ok)
* This is equivalent to LookupTypeName, except that this will report
* a suitable error message if the type cannot be found or is not defined.
* Callers of this can therefore assume the result is a fully valid type.
+ *
+ * For a dictionary type it's OID is returned as a typmod. This is used by
+ * dictionary_in() and jsonb_dictionary() to distinguish one dictionary from
+ * another.
*/
Type
typenameType(ParseState *pstate, const TypeName *typeName, int32 *typmod_p)
@@ -305,6 +309,10 @@ typenameTypeId(ParseState *pstate, const TypeName *typeName)
*
* This is equivalent to typenameType, but we only hand back the type OID
* and typmod, not the syscache entry.
+ *
+ * For a dictionary type it's OID is returned as a typmod. This is used by
+ * dictionary_in() and jsonb_dictionary() to distinguish one dictionary from
+ * another.
*/
void
typenameTypeIdAndMod(ParseState *pstate, const TypeName *typeName,
@@ -327,6 +335,10 @@ typenameTypeIdAndMod(ParseState *pstate, const TypeName *typeName,
* looked up, and is passed as "typ".
*
* pstate is only used for error location info, and may be NULL.
+ *
+ * For a dictionary type it's OID is returned as a typmod. This is used by
+ * dictionary_in() and jsonb_dictionary() to distinguish one dictionary from
+ * another.
*/
static int32
typenameTypeMod(ParseState *pstate, const TypeName *typeName, Type typ)
@@ -339,6 +351,10 @@ typenameTypeMod(ParseState *pstate, const TypeName *typeName, Type typ)
ArrayType *arrtypmod;
ParseCallbackState pcbstate;
+ /* For a dictionary return it's OID. */
+ if (((Form_pg_type) GETSTRUCT(typ))->typtype == TYPTYPE_DICT)
+ return (int32)((Form_pg_type) GETSTRUCT(typ))->oid;
+
/* Return prespecified typmod if no typmod expressions */
if (typeName->typmods == NIL)
return typeName->typemod;
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index 247d0816ad..ec8a70c3fc 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -174,6 +174,7 @@ ClassifyUtilityCommandAsReadOnly(Node *parsetree)
case T_CreateCastStmt:
case T_CreateConversionStmt:
case T_CreateDomainStmt:
+ case T_CreateDictionaryStmt:
case T_CreateEnumStmt:
case T_CreateEventTrigStmt:
case T_CreateExtensionStmt:
@@ -1621,6 +1622,10 @@ ProcessUtilitySlow(ParseState *pstate,
}
break;
+ case T_CreateDictionaryStmt: /* CREATE TYPE AS DICTIONARY OF */
+ address = DefineDictionary((CreateDictionaryStmt *) parsetree);
+ break;
+
case T_CreateEnumStmt: /* CREATE TYPE AS ENUM */
address = DefineEnum((CreateEnumStmt *) parsetree);
break;
@@ -2772,6 +2777,10 @@ CreateCommandTag(Node *parsetree)
tag = CMDTAG_CREATE_TYPE;
break;
+ case T_CreateDictionaryStmt:
+ tag = CMDTAG_CREATE_TYPE;
+ break;
+
case T_CreateEnumStmt:
tag = CMDTAG_CREATE_TYPE;
break;
@@ -3419,6 +3428,10 @@ GetCommandLogLevel(Node *parsetree)
lev = LOGSTMT_DDL;
break;
+ case T_CreateDictionaryStmt:
+ lev = LOGSTMT_DDL;
+ break;
+
case T_CreateEnumStmt:
lev = LOGSTMT_DDL;
break;
diff --git a/src/backend/utils/adt/Makefile b/src/backend/utils/adt/Makefile
index 0de0bbb1b8..58e25fd8bc 100644
--- a/src/backend/utils/adt/Makefile
+++ b/src/backend/utils/adt/Makefile
@@ -30,6 +30,7 @@ OBJS = \
datetime.o \
datum.o \
dbsize.o \
+ dictionaries.o \
domains.o \
encode.o \
enum.o \
diff --git a/src/backend/utils/adt/dictionaries.c b/src/backend/utils/adt/dictionaries.c
new file mode 100644
index 0000000000..2bb46471a4
--- /dev/null
+++ b/src/backend/utils/adt/dictionaries.c
@@ -0,0 +1,482 @@
+/*-------------------------------------------------------------------------
+ *
+ * dictionaries.c
+ * Conversion functions for dictionary types.
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/adt/dictionaries.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "catalog/pg_dict.h"
+#include "utils/fmgrprotos.h"
+#include "utils/jsonb.h"
+
+/*
+ * When compressing a data we treat it as a BLOB, in other words we don't
+ * assume anything regarding its internal representation. This is not
+ * necessarily the best and/or the only possible approach. However, it is
+ * universal and can be reused for JSONB, TEXT, XML and other types. Alternative
+ * compression methods can be introduced in the future, so that the user will
+ * be able to choose the best one for the task.
+ *
+ * The compressed data is stored in the following format:
+ *
+ * (struct varlena)
+ * dictionary_id [uint32]
+ * decompressed_size [uint32]
+ * algorithm_version [uint8]
+ *
+ * (repeated) {
+ * number of bytes to copy as-is [uint8]
+ * ... bytes to copy as-is ...
+ * dictionary entry id, or 0 to skip [uint32]
+ * }
+ *
+ * Compressed data is a variable-length type and thus has the 'struct varlena'
+ * header. The code below doesn't consider it being a part of the payload
+ * (e.g. see the DICT_COMP_HEADER_SIZE definition).
+ *
+ * Storing dictionary_id may seem redundant, but without it dictionary_out()
+ * and dictionary_jsonb() have no way to distinguish one dictionary type from
+ * another. dictionary_in() and jsonb_dictionary() get the dictionary id through
+ * the 'typemod' argument.
+ *
+ * Currently algorithm_version is always 0. In the future it will allow us to
+ * introduce new features (e.g. usage of varints) and/or lazily migrate the
+ * data to other compression methods.
+ *
+ * Compression and decompression are implemented based on a binary search over
+ * DictEntry[] array (see pg_dict.h). For compression the array is sorted by
+ * dictentries and for decompression - by oid's.
+ *
+ */
+
+/* Size of the header in the compressed data */
+#define DICT_COMP_HEADER_SIZE (sizeof(uint32)*2 + sizeof(uint8))
+
+/* Extracts dictionary_id from the compressed data */
+#define DICT_COMP_DICTIONARY_ID(hdrp) \
+ (*(uint32*)hdrp)
+
+/* Extracts decompressed_size from the compressed data */
+#define DICT_COMP_DECOMPRESSED_SIZE(hdrp) \
+ (*(uint32*)((uint8*)hdrp + sizeof(uint32)))
+
+/* Extracts algorithm_version from the compressed data */
+#define DICT_COMP_ALGORITHM_VERSION(hdrp) \
+ (*(uint8*)((uint8*)hdrp + sizeof(uint32)*2))
+
+/* Current algorithm_version */
+#define DICT_COMP_CURRENT_ALGORITHM_VERSION 0
+
+/*
+ * bsearch_arg() callback for finding DictEntry by oid.
+ */
+static int
+find_by_oid_cb(const void *key, const void *current, void *arg)
+{
+ /* Note that oids are unsigned, so we should be careful here */
+ if (*(Oid *) key < ((DictEntry *) current)->oid)
+ return -1;
+ else if (*(Oid *) key == ((DictEntry *) current)->oid)
+ return 0; /* found! */
+ else
+ return 1;
+}
+
+/*
+ * qsort() callback for sorting DictEntry[] by oids.
+ */
+static int
+sort_by_oids_cb(const void *left, const void *right)
+{
+ /* Note that oids are unsigned, so we should be careful here */
+ if (((DictEntry *) left)->oid < ((DictEntry *) right)->oid)
+ return -1;
+
+ /* Oids are unique so this callback will never return 0 */
+ return 1;
+}
+
+/*
+ * Finds a DictEntry which dictentry field matches *data and returns its Oid.
+ * If there are several matching entries, the largest is returned. The length
+ * of the found entry is written to *found_length on success. On failure
+ * InvalidOid is returned and found_length is zeroed.
+ *
+ * The implementation is similar to bsearch_arg(). The procedure can't be used
+ * directly because we are looking not by the exact match. We could generalize
+ * this case but the signature of the function becomes so complicated that it
+ * doesn't seem to worth the effort.
+ */
+static Oid
+compress_find_oid(Dictionary dict, const uint8 *data, Size data_size, Size *found_length)
+{
+ int res;
+ int32 left = 0;
+ int32 right = dict->nentries - 1;
+ Size best_length = 0;
+ Oid best_match = InvalidOid;
+
+ while (left <= right)
+ {
+ int32 current = (left + right) / 2;
+ Size nbytes = (Size)dict->entries[current].length;
+
+ if (nbytes > data_size)
+ {
+ /* current can be less or greater depending on the prefix */
+ res = memcmp(dict->entries[current].data, data, data_size);
+
+ /* if prefixes match, current is greater */
+ if (res == 0)
+ res = 1;
+ }
+ else
+ res = memcmp(dict->entries[current].data, data, nbytes);
+
+ if (res == 0) /* match found */
+ {
+ best_length = nbytes;
+ best_match = dict->entries[current].oid;
+
+ if (nbytes == data_size)
+ break;
+
+ /* maybe there is a larger match */
+ left = current + 1;
+ }
+ else if (res < 0) /* current is less */
+ left = current + 1;
+ else /* current is greater */
+ right = current - 1;
+ }
+
+ *found_length = best_length;
+ return best_match;
+}
+
+/*
+ * Finds a DictEntry by Oid using a binary search. The dictionary should be
+ * sorted by oids before the call. Returns NULL if nothing was found.
+ */
+static DictEntry *
+decompress_find_dictentry(Dictionary dict, Oid oid)
+{
+ return (DictEntry *) bsearch_arg(&oid, dict->entries, dict->nentries, sizeof(DictEntry), find_by_oid_cb, NULL);
+}
+
+/*
+ * Estimates the worst-case compressed size of the data of given size.
+ * Worst-case scenario happens when the dictionary consists of single-character
+ * entries. In this case every byte will be encoded as 6 bytes:
+ * 0x00, (0 bytes to copy as-is), 4 bytes of the entry Oid
+ *
+ * This procedure doesn't account for the header size.
+ *
+ * AALEKSEEV FIXME don't use dictionary entries shorter than 5 bytes
+ */
+static Size
+worst_case_compressed_size(Size insize)
+{
+ return insize * 6;
+}
+
+/*
+ * Compresses the data using the provided dictionary. The dictionary should
+ * be sorted by dictentries before the call. Output buffer should be at least
+ * worst_case_compressed_size(src_size) bytes in size.
+ */
+static void
+compress(Dictionary dict,
+ const void *src_data_, Size src_size,
+ void *encoded_data_, Size *pencoded_size)
+{
+ Size nbytes;
+ Size inoffset;
+ Size outskipoffset = 0;
+ Size outoffset = 1;
+ uint8 skipbytes = 0;
+ const uint8 *src_data = src_data_;
+ uint8 *encoded_data = ((uint8 *) encoded_data_);
+
+ for (inoffset = 0; inoffset < src_size;)
+ {
+ Oid code = compress_find_oid(dict, &(src_data[inoffset]),
+ src_size - inoffset, &nbytes);
+
+ if (code == InvalidOid)
+ {
+ skipbytes++;
+ encoded_data[outoffset] = src_data[inoffset];
+ outoffset++;
+ inoffset++;
+
+ if (skipbytes == 255)
+ {
+ encoded_data[outskipoffset] = skipbytes;
+ encoded_data[outoffset++] = 0; /* InvalidOid */
+ encoded_data[outoffset++] = 0;
+ encoded_data[outoffset++] = 0;
+ encoded_data[outoffset++] = 0;
+ outskipoffset = outoffset++;
+ skipbytes = 0;
+ }
+ }
+ else
+ {
+ encoded_data[outskipoffset] = skipbytes;
+ encoded_data[outoffset++] = (code >> 24) & 0xFF;
+ encoded_data[outoffset++] = (code >> 16) & 0xFF;
+ encoded_data[outoffset++] = (code >> 8) & 0xFF;
+ encoded_data[outoffset++] = code & 0xFF;
+ outskipoffset = outoffset++;
+ skipbytes = 0;
+ inoffset += nbytes;
+ }
+ }
+
+ /* Double check that we didn't write out of buffer */
+ Assert(outoffset < worst_case_compressed_size(src_size));
+
+ encoded_data[outskipoffset] = skipbytes;
+ *pencoded_size = outoffset;
+}
+
+/*
+ * Report an internal error in decompress() procedure below.
+ * Under normal circumstances this should never happen.
+ */
+static void
+decompress_error()
+{
+ ereport(ERROR,
+ (
+ errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("Unable to decompress a dictionary type"),
+ errdetail("The compressed data seems to be corrupted"),
+ errhint("Please report the steps to reproduce the issue to pgsql-bugs@")
+ ));
+}
+
+/*
+ * Decompresses the data using the provided dictionary. The dictionary should
+ * be sorted by oids before the call.
+ */
+static void
+decompress(Dictionary dict,
+ const void *encoded_data_, Size encoded_size,
+ void *decoded_data_, Size decoded_size)
+{
+ Size inoffset = 0;
+ Size outoffset = 0;
+ Oid code;
+ uint8 skipbytes;
+ const uint8 *encoded_data = ((uint8 *) encoded_data_);
+ uint8 *decoded_data = decoded_data_;
+
+ for (inoffset = 0; inoffset < encoded_size;)
+ {
+ skipbytes = encoded_data[inoffset++];
+
+ if (skipbytes > decoded_size - outoffset)
+ decompress_error();
+
+ if (skipbytes > encoded_size - inoffset)
+ decompress_error();
+
+ memcpy(
+ &(decoded_data[outoffset]),
+ &(encoded_data[inoffset]),
+ skipbytes
+ );
+
+ outoffset += skipbytes;
+ inoffset += skipbytes;
+
+ if ((encoded_size == inoffset) && (decoded_size == outoffset))
+ break; /* end of input - its OK */
+
+ if (encoded_size - inoffset < 4)
+ decompress_error();
+
+ code = (Oid) encoded_data[inoffset++];
+ code = (code << 8) | (Oid) encoded_data[inoffset++];
+ code = (code << 8) | (Oid) encoded_data[inoffset++];
+ code = (code << 8) | (Oid) encoded_data[inoffset++];
+
+ if (code != InvalidOid)
+ {
+ Size entrylen;
+ DictEntry *entry = decompress_find_dictentry(dict, code);
+
+ if (entry == NULL)
+ decompress_error();
+
+ Assert(entry->oid == code);
+ entrylen = (Size)entry->length;
+
+ if (entrylen > decoded_size - outoffset)
+ decompress_error();
+
+ memcpy(
+ &(decoded_data[outoffset]),
+ entry->data,
+ entrylen
+ );
+
+ outoffset += entrylen;
+ }
+ }
+
+ Assert(decoded_size == outoffset);
+}
+
+/*
+ * Converts a cstring to a dictionary.
+ */
+Datum
+dictionary_in(PG_FUNCTION_ARGS)
+{
+ const char *instr = PG_GETARG_CSTRING(0);
+#ifdef NOT_USED
+ Oid typelem = PG_GETARG_OID(1);
+#endif
+ int32 typmod = PG_GETARG_INT32(2);
+ Jsonb *jsonb;
+ bytea *dict;
+
+ Assert(typmod != -1);
+
+ jsonb = DatumGetJsonbP(DirectFunctionCall1(jsonb_in, CStringGetDatum(instr)));
+ dict = DatumGetByteaP(DirectFunctionCall2(jsonb_dictionary, JsonbPGetDatum(jsonb), Int32GetDatum(typmod)));
+
+ PG_RETURN_BYTEA_P(dict);
+}
+
+/*
+ * Converts a dictionary to a cstring.
+ */
+Datum
+dictionary_out(PG_FUNCTION_ARGS)
+{
+ bytea *dict = PG_GETARG_BYTEA_P(0);
+ Jsonb *jsonb = DatumGetJsonbP(DirectFunctionCall1(dictionary_jsonb, PointerGetDatum(dict)));
+ const char *outstr = DatumGetCString(DirectFunctionCall1(jsonb_out, JsonbPGetDatum(jsonb)));
+
+ PG_RETURN_CSTRING(outstr);
+}
+
+/*
+ * Coverts JSONB to a dictionary type.
+ *
+ * AALEKSEEV FIXME: if the compressed document ends up being larger than
+ * the original one write a corresponding flag and copy all the data as-is.
+ */
+Datum
+jsonb_dictionary(PG_FUNCTION_ARGS)
+{
+ Dictionary dict;
+ Jsonb *jsonb = PG_GETARG_JSONB_P(0);
+ int32 typmod = PG_GETARG_INT32(1);
+ uint8 *jsonb_data = (uint8 *) VARDATA(jsonb);
+ Size jsonb_data_size = VARSIZE(jsonb) - VARHDRSZ;
+ uint8 *encoded_buff,
+ *encoded_header,
+ *encoded_data;
+ Size encoded_size,
+ encoded_buff_size;
+
+ Assert(typmod != -1);
+
+ dict = DictEntriesRead(typmod);
+ qsort((void *) dict->entries, dict->nentries, sizeof(DictEntry), sort_by_oids_cb);
+
+ encoded_buff_size = VARHDRSZ + DICT_COMP_HEADER_SIZE + worst_case_compressed_size(jsonb_data_size);
+ encoded_buff = palloc(encoded_buff_size);
+ encoded_header = (uint8 *) VARDATA(encoded_buff);
+ encoded_data = encoded_header + DICT_COMP_HEADER_SIZE;
+
+ DICT_COMP_DICTIONARY_ID(encoded_header) = typmod;
+ DICT_COMP_DECOMPRESSED_SIZE(encoded_header) = jsonb_data_size;
+ DICT_COMP_ALGORITHM_VERSION(encoded_header) = DICT_COMP_CURRENT_ALGORITHM_VERSION;
+
+ encoded_size = encoded_buff_size - VARHDRSZ - DICT_COMP_HEADER_SIZE;
+
+ compress(dict, jsonb_data, jsonb_data_size,
+ encoded_data, &encoded_size);
+
+ encoded_size += VARHDRSZ + DICT_COMP_HEADER_SIZE;
+
+ encoded_buff = repalloc(encoded_buff, encoded_size);
+ SET_VARSIZE(encoded_buff, encoded_size);
+
+ DictEntriesFree(dict);
+
+ PG_RETURN_BYTEA_P(encoded_buff);
+}
+
+/*
+ * Converts a dictionary type to JSONB.
+ */
+Datum
+dictionary_jsonb(PG_FUNCTION_ARGS)
+{
+ bytea *encoded_buff = PG_GETARG_BYTEA_P(0);
+ uint8 *encoded_header = (uint8 *) VARDATA(encoded_buff);
+ uint8 *encoded_data = encoded_header + DICT_COMP_HEADER_SIZE;
+ Size encoded_size = VARSIZE(encoded_buff) - VARHDRSZ - DICT_COMP_HEADER_SIZE;
+ int alg_version = DICT_COMP_ALGORITHM_VERSION(encoded_header);
+ Oid dictOid; /* cannot read until algorithm version is
+ * checked */
+ uint32 decoded_size; /* cannot read until algorithm version is
+ * checked */
+ Jsonb *jsonb;
+ uint8 *jsonb_data;
+ Dictionary dict;
+
+ if (alg_version > DICT_COMP_CURRENT_ALGORITHM_VERSION)
+ ereport(ERROR,
+ (
+ errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("Unsupported compression algorithm version"),
+ errdetail("Saved algorithm version is %d, current version is %d",
+ alg_version, DICT_COMP_CURRENT_ALGORITHM_VERSION),
+ errhint("The data is either corrupted or imported from "
+ "the future version of PostgreSQL")
+ ));
+
+ dictOid = DICT_COMP_DICTIONARY_ID(encoded_header);
+ decoded_size = DICT_COMP_DECOMPRESSED_SIZE(encoded_header);
+
+ dict = DictEntriesRead(dictOid);
+
+ jsonb = palloc(decoded_size + VARHDRSZ);
+ jsonb_data = (uint8 *) VARDATA(jsonb);
+
+ decompress(dict, encoded_data, encoded_size, jsonb_data, decoded_size);
+
+ decoded_size += VARHDRSZ;
+ SET_VARSIZE(jsonb, decoded_size);
+
+ DictEntriesFree(dict);
+
+ PG_RETURN_JSONB_P(jsonb);
+}
+
+/*
+ * Converts a dictionary type to bytea.
+ */
+Datum
+dictionary_bytea(PG_FUNCTION_ARGS)
+{
+ bytea *compressed_data = PG_GETARG_BYTEA_P(0);
+ PG_RETURN_BYTEA_P(compressed_data);
+}
\ No newline at end of file
diff --git a/src/backend/utils/adt/meson.build b/src/backend/utils/adt/meson.build
index ed9ceadfef..2ed48c1406 100644
--- a/src/backend/utils/adt/meson.build
+++ b/src/backend/utils/adt/meson.build
@@ -17,6 +17,7 @@ backend_sources += files(
'datetime.c',
'datum.c',
'dbsize.c',
+ 'dictionaries.c',
'domains.c',
'encode.c',
'enum.c',
diff --git a/src/backend/utils/fmgr/funcapi.c b/src/backend/utils/fmgr/funcapi.c
index 87cbb1d3e3..efb3e739e4 100644
--- a/src/backend/utils/fmgr/funcapi.c
+++ b/src/backend/utils/fmgr/funcapi.c
@@ -1326,6 +1326,7 @@ get_type_func_class(Oid typid, Oid *base_typeid)
case TYPTYPE_COMPOSITE:
return TYPEFUNC_COMPOSITE;
case TYPTYPE_BASE:
+ case TYPTYPE_DICT:
case TYPTYPE_ENUM:
case TYPTYPE_RANGE:
case TYPTYPE_MULTIRANGE:
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index da427f4d4a..38638b88cc 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -10218,6 +10218,8 @@ dumpType(Archive *fout, const TypeInfo *tyinfo)
dumpDomain(fout, tyinfo);
else if (tyinfo->typtype == TYPTYPE_COMPOSITE)
dumpCompositeType(fout, tyinfo);
+ else if (tyinfo->typtype == TYPTYPE_DICT)
+ pg_log_error("AALEKSEEV TODO FIXME not implemented");
else if (tyinfo->typtype == TYPTYPE_ENUM)
dumpEnumType(fout, tyinfo);
else if (tyinfo->typtype == TYPTYPE_RANGE)
diff --git a/src/include/catalog/meson.build b/src/include/catalog/meson.build
index 45ffa99692..8d76de46f3 100644
--- a/src/include/catalog/meson.build
+++ b/src/include/catalog/meson.build
@@ -25,6 +25,7 @@ catalog_headers = [
'pg_event_trigger.h',
'pg_description.h',
'pg_cast.h',
+ 'pg_dict.h',
'pg_enum.h',
'pg_namespace.h',
'pg_conversion.h',
diff --git a/src/include/catalog/pg_dict.h b/src/include/catalog/pg_dict.h
new file mode 100644
index 0000000000..fbad803468
--- /dev/null
+++ b/src/include/catalog/pg_dict.h
@@ -0,0 +1,78 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_dict.h
+ * definition of the "dict" system catalog (pg_dict)
+ *
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ * src/include/catalog/pg_dict.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef PG_DICT_H
+#define PG_DICT_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_dict_d.h"
+
+#include "nodes/pg_list.h"
+
+/* ----------------
+ * pg_dict definition. cpp turns this into
+ * typedef struct FormData_pg_dict
+ * ----------------
+ */
+CATALOG(pg_dict,9861,DictRelationId)
+{
+ Oid oid; /* oid */
+ Oid dicttypid BKI_LOOKUP(pg_type); /* OID of owning dict type */
+ /* AALEKSEEV TODO refactor to bytea */
+ NameData dictentry; /* text representation of the dictionary entry */
+} FormData_pg_dict;
+
+/* ----------------
+ * Form_pg_dict corresponds to a pointer to a tuple with
+ * the format of pg_dict relation.
+ * ----------------
+ */
+typedef FormData_pg_dict *Form_pg_dict;
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_dict_oid_index, 9862, DictOidIndexId, on pg_dict using btree(oid oid_ops));
+DECLARE_UNIQUE_INDEX(pg_dict_typid_entry_index, 9863, DictTypIdEntryIndexId, on pg_dict using btree(dicttypid oid_ops, dictentry name_ops));
+
+
+/*
+ * DictEntry type represents one entry in the given dictionary.
+ */
+typedef struct
+{
+ Oid oid; /* entry id */
+ uint32 length; /* entry length */
+ uint8* data; /* entry data */
+} DictEntry;
+
+/*
+ * Dictionary type represents all the entries in the given dictionary.
+ */
+typedef struct
+{
+ uint32 nentries; /* number of entries */
+ DictEntry *entries; /* array of entries */
+} DictionaryData;
+
+typedef DictionaryData *Dictionary;
+
+/*
+ * prototypes for functions in pg_dict.c
+ */
+extern void DictEntriesCreate(Oid dictTypeOid, List *vals);
+extern void DictEntriesFree(Dictionary dict);
+extern Dictionary DictEntriesRead(Oid dictTypeOid);
+extern void DictEntriesDelete(Oid dictTypeOid);
+
+#endif /* PG_DICT_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 20f5aa56ea..5c31bb58a3 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -9966,6 +9966,26 @@
proname => 'jsonb_path_match_opr', prorettype => 'bool',
proargtypes => 'jsonb jsonpath', prosrc => 'jsonb_path_match_opr' },
+# dictionaries
+{ oid => '9864', descr => 'Converts a cstring to a dictionary',
+ proname => 'dictionary_in', prorettype => 'any', proargtypes => 'cstring oid int4',
+ prosrc => 'dictionary_in' },
+{ oid => '9865', descr => 'Converts a dictionary to a cstring',
+ proname => 'dictionary_out', prorettype => 'cstring', proargtypes => 'any',
+ prosrc => 'dictionary_out' },
+{ oid => '9866', descr => 'Coverts JSONB to a dictionary type',
+ proname => 'jsonb_dictionary', proisstrict => 'f', provolatile => 's',
+ prorettype => 'any', proargtypes => 'jsonb int4',
+ prosrc => 'jsonb_dictionary' },
+{ oid => '9867', descr => 'Converts a dictionary type to JSONB',
+ proname => 'dictionary_jsonb', proisstrict => 'f', provolatile => 's',
+ prorettype => 'jsonb', proargtypes => 'any',
+ prosrc => 'dictionary_jsonb' },
+{ oid => '9868', descr => 'Converts a dictionary type to a byte array',
+ proname => 'dictionary_bytea', proisstrict => 'f', provolatile => 's',
+ prorettype => 'jsonb', proargtypes => 'any',
+ prosrc => 'dictionary_bytea' },
+
# historical int8/txid_snapshot variants of xid8 functions
{ oid => '2939', descr => 'I/O',
proname => 'txid_snapshot_in', prorettype => 'txid_snapshot',
diff --git a/src/include/catalog/pg_type.h b/src/include/catalog/pg_type.h
index 48a2559137..91c6bfae99 100644
--- a/src/include/catalog/pg_type.h
+++ b/src/include/catalog/pg_type.h
@@ -272,6 +272,7 @@ DECLARE_UNIQUE_INDEX(pg_type_typname_nsp_index, 2704, TypeNameNspIndexId, on pg_
*/
#define TYPTYPE_BASE 'b' /* base type (ordinary scalar type) */
#define TYPTYPE_COMPOSITE 'c' /* composite (e.g., table's rowtype) */
+#define TYPTYPE_DICT 'D' /* dictionary */
#define TYPTYPE_DOMAIN 'd' /* domain over another type */
#define TYPTYPE_ENUM 'e' /* enumerated type */
#define TYPTYPE_MULTIRANGE 'm' /* multirange type */
diff --git a/src/include/commands/typecmds.h b/src/include/commands/typecmds.h
index 532bc49c38..a3ef87741d 100644
--- a/src/include/commands/typecmds.h
+++ b/src/include/commands/typecmds.h
@@ -24,6 +24,7 @@
extern ObjectAddress DefineType(ParseState *pstate, List *names, List *parameters);
extern void RemoveTypeById(Oid typeOid);
extern ObjectAddress DefineDomain(CreateDomainStmt *stmt);
+extern ObjectAddress DefineDictionary(CreateDictionaryStmt *stmt);
extern ObjectAddress DefineEnum(CreateEnumStmt *stmt);
extern ObjectAddress DefineRange(ParseState *pstate, CreateRangeStmt *stmt);
extern ObjectAddress AlterEnum(AlterEnumStmt *stmt);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 7caff62af7..8b127086c9 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3296,6 +3296,18 @@ typedef struct CompositeTypeStmt
List *coldeflist; /* list of ColumnDef nodes */
} CompositeTypeStmt;
+/* ----------------------
+ * Create Type Statement, dictionary types
+ * ----------------------
+ */
+typedef struct CreateDictionaryStmt
+{
+ NodeTag type;
+ List *typeName; /* qualified name (list of String) */
+ List *baseTypeName; /* qualified base type name (list of String) */
+ List *vals; /* dictionary entries (list of String) */
+} CreateDictionaryStmt;
+
/* ----------------------
* Create Type Statement, enum types
* ----------------------
diff --git a/src/pl/plpgsql/src/pl_comp.c b/src/pl/plpgsql/src/pl_comp.c
index b286f2a50c..2754098f7f 100644
--- a/src/pl/plpgsql/src/pl_comp.c
+++ b/src/pl/plpgsql/src/pl_comp.c
@@ -2123,6 +2123,7 @@ build_datatype(HeapTuple typeTup, int32 typmod,
switch (typeStruct->typtype)
{
case TYPTYPE_BASE:
+ case TYPTYPE_DICT:
case TYPTYPE_ENUM:
case TYPTYPE_RANGE:
case TYPTYPE_MULTIRANGE:
diff --git a/src/test/regress/expected/dict.out b/src/test/regress/expected/dict.out
new file mode 100644
index 0000000000..699e76b654
--- /dev/null
+++ b/src/test/regress/expected/dict.out
@@ -0,0 +1,47 @@
+--
+-- Compression dictionaries tests
+--
+-- Dictionary types currently can be used only with JSONB
+\set ON_ERROR_STOP 0
+CREATE TYPE textdict AS DICTIONARY OF TEXT ('abcdef', 'ghijkl');
+ERROR: Dictionary types currently can be used only with JSONB
+\set ON_ERROR_STOP 1
+-- Simple happy path
+CREATE TYPE mydict AS DICTIONARY OF JSONB ('abcdef', 'ghijkl');
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
+ dictentry
+-----------
+ abcdef
+ ghijkl
+(2 rows)
+
+SELECT '{"abcdef":"ghijkl"}' :: mydict;
+ mydict
+----------------------
+ {"abcdef": "ghijkl"}
+(1 row)
+
+SELECT '{"abcdef":"ghijkl"}' :: mydict :: jsonb;
+ jsonb
+----------------------
+ {"abcdef": "ghijkl"}
+(1 row)
+
+SELECT '{"abcdef":"ghijkl"}' :: jsonb :: mydict;
+ mydict
+----------------------
+ {"abcdef": "ghijkl"}
+(1 row)
+
+SELECT ('{"abcdef":"ghijkl"}' :: mydict) -> 'abcdef';
+ ?column?
+----------
+ "ghijkl"
+(1 row)
+
+DROP TYPE mydict;
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
+ dictentry
+-----------
+(0 rows)
+
diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out
index 215eb899be..91bc491a6c 100644
--- a/src/test/regress/expected/oidjoins.out
+++ b/src/test/regress/expected/oidjoins.out
@@ -182,6 +182,7 @@ NOTICE: checking pg_description {classoid} => pg_class {oid}
NOTICE: checking pg_cast {castsource} => pg_type {oid}
NOTICE: checking pg_cast {casttarget} => pg_type {oid}
NOTICE: checking pg_cast {castfunc} => pg_proc {oid}
+NOTICE: checking pg_dict {dicttypid} => pg_type {oid}
NOTICE: checking pg_enum {enumtypid} => pg_type {oid}
NOTICE: checking pg_namespace {nspowner} => pg_authid {oid}
NOTICE: checking pg_conversion {connamespace} => pg_namespace {oid}
diff --git a/src/test/regress/expected/opr_sanity.out b/src/test/regress/expected/opr_sanity.out
index 330eb0f765..f435feb108 100644
--- a/src/test/regress/expected/opr_sanity.out
+++ b/src/test/regress/expected/opr_sanity.out
@@ -388,11 +388,12 @@ WHERE 'cstring'::regtype = ANY (p1.proargtypes)
AND NOT EXISTS(SELECT 1 FROM pg_conversion WHERE conproc = p1.oid)
AND p1.oid != 'shell_in(cstring)'::regprocedure
ORDER BY 1;
- oid | proname
-------+--------------
+ oid | proname
+------+---------------
2293 | cstring_out
2501 | cstring_send
-(2 rows)
+ 9864 | dictionary_in
+(3 rows)
-- Likewise, look for functions that return cstring and aren't datatype output
-- functions nor typmod output functions.
@@ -405,11 +406,12 @@ WHERE p1.prorettype = 'cstring'::regtype
AND NOT EXISTS(SELECT 1 FROM pg_type WHERE typmodout = p1.oid)
AND p1.oid != 'shell_out(void)'::regprocedure
ORDER BY 1;
- oid | proname
-------+--------------
+ oid | proname
+------+----------------
2292 | cstring_in
2500 | cstring_recv
-(2 rows)
+ 9865 | dictionary_out
+(3 rows)
-- Check for length inconsistencies between the various argument-info arrays.
SELECT p1.oid, p1.proname
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 9a139f1e24..385398545b 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -111,7 +111,7 @@ test: select_views portals_p2 foreign_key cluster dependency guc bitmapops combo
# ----------
# Another group of parallel tests (JSON related)
# ----------
-test: json jsonb json_encoding jsonpath jsonpath_encoding jsonb_jsonpath
+test: dict json jsonb json_encoding jsonpath jsonpath_encoding jsonb_jsonpath
# ----------
# Another group of parallel tests
diff --git a/src/test/regress/sql/dict.sql b/src/test/regress/sql/dict.sql
new file mode 100644
index 0000000000..27ecd6a451
--- /dev/null
+++ b/src/test/regress/sql/dict.sql
@@ -0,0 +1,20 @@
+--
+-- Compression dictionaries tests
+--
+
+-- Dictionary types currently can be used only with JSONB
+\set ON_ERROR_STOP 0
+CREATE TYPE textdict AS DICTIONARY OF TEXT ('abcdef', 'ghijkl');
+\set ON_ERROR_STOP 1
+
+-- Simple happy path
+CREATE TYPE mydict AS DICTIONARY OF JSONB ('abcdef', 'ghijkl');
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
+
+SELECT '{"abcdef":"ghijkl"}' :: mydict;
+SELECT '{"abcdef":"ghijkl"}' :: mydict :: jsonb;
+SELECT '{"abcdef":"ghijkl"}' :: jsonb :: mydict;
+SELECT ('{"abcdef":"ghijkl"}' :: mydict) -> 'abcdef';
+
+DROP TYPE mydict;
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
--
2.38.1
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: [PATCH] Compression dictionaries for JSONB
@ 2022-11-17 10:36 Aleksander Alekseev <[email protected]>
parent: Aleksander Alekseev <[email protected]>
0 siblings, 1 reply; 88+ messages in thread
From: Aleksander Alekseev @ 2022-11-17 10:36 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>; +Cc: Simon Riggs <[email protected]>; Nikita Malakhov <[email protected]>; Matthias van de Meent <[email protected]>; Jacob Champion <[email protected]>; Alvaro Herrera <[email protected]>; Zhihong Yu <[email protected]>
Hi hackers,
> 8272749e added a few more arguments to CastCreate(). Here is the rebased patch.
After merging afbfc029 [1] the patch needed a rebase. PFA v10.
The patch is still in a PoC state and this is exactly why comments and
suggestions from the community are most welcome! Particularly I would
like to know:
1. Would you call it a wanted feature considering the existence of
Pluggable TOASTer patchset which (besides other things) tries to
introduce type-aware TOASTers for EXTERNAL attributes? I know what
Simon's [2] and Nikita's latest answers were, and I know my personal
opinion on this [3][4], but I would like to hear from the rest of the
community.
2. How should we make sure a dictionary will not consume all the
available memory? Limiting the amount of dictionary entries to pow(2,
16) and having dictionary versions seems to work OK for ZSON. However
it was pointed out that this may be an unwanted limitation for the
in-core implementation.
[1]: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=c727f511;hp=afbfc02983f86c4d71825...
[2]: https://www.postgresql.org/message-id/CANbhV-HpCF852WcZuU0wyh1jMU4p6XLbV6rCRkZpnpeKQ9OenQ%40mail.gma...
[3]: https://www.postgresql.org/message-id/CAJ7c6TN-N3%3DPSykmOjmW1EAf9YyyHFDHEznX-5VORsWUvVN-5w%40mail.g...
[4]: https://www.postgresql.org/message-id/CAJ7c6TO2XTTk3cu5w6ePHfhYQkoNpw7u1jeqHf%3DGwn%2BoWci8eA%40mail...
--
Best regards,
Aleksander Alekseev
Attachments:
[application/octet-stream] v10-0001-Compression-dictionaries-for-JSONB.patch (49.7K, ../../CAJ7c6TOOn+gK9bh7OsmVD5ZwT+0KYRt7+Htzb8JkRu-N_JWB7A@mail.gmail.com/2-v10-0001-Compression-dictionaries-for-JSONB.patch)
download | inline diff:
From d44617b861758b54b922645bc040965b70f0e49f Mon Sep 17 00:00:00 2001
From: Aleksander Alekseev <[email protected]>
Date: Tue, 12 Apr 2022 15:58:11 +0300
Subject: [PATCH v10] Compression dictionaries for JSONB
Usage example:
CREATE TYPE mydict AS DITIONARY OF jsonb ('aaa', 'bbb', ...);
SELECT ('{"aaa":"bbb"}' :: mydict) -> 'aaa';
The created type works as a drop-in replacement for JSONB. However,
its internal representation differs. The provided dictionary entries ('aaa',
'bbb', ..) are stored in pg_dict catalog table. When `mydict` sees one of
the entries in the document, it replaces it with the corresponding pg_dict
entry Oid. For more details regarding the compression algorithm and choosen
compromises please see the comments in the code.
In pg_type `mydict` has typtype = TYPTYPE_DICT. It works the same way
as TYPTYPE_BASE with only difference: corresponding `<type>_in`
(pg_type.typinput) and `<another-type>_<type>` (pg_cast.castfunc)
procedures receive the dictionary Oid as a `typmod` argument. This way the
procedures can distinguish `mydict1` from `mydict2` and use the proper
compression dictionary.
(NOTE: dear committer, please bump the catalog version!)
Author: Aleksander Alekseev <[email protected]>
Reviewed-by: Matthias van de Meent, Nikita Malakhov
Discussion: https://postgr.es/m/CAJ7c6TOtAB0z1UrksvGTStNE-herK-43bj22%3D5xVBg7S4vr5rQ%40mail.gmail.com
Discussion: https://postgr.es/m/CAJ7c6TPx7N-bVw0dZ1ASCDQKZJHhBYkT6w4HV1LzfS%2BUUTUfmA%40mail.gmail.com
---
src/backend/catalog/Makefile | 3 +-
src/backend/catalog/meson.build | 1 +
src/backend/catalog/pg_dict.c | 197 +++++++++
src/backend/commands/typecmds.c | 171 +++++++-
src/backend/executor/functions.c | 1 +
src/backend/parser/gram.y | 8 +
src/backend/parser/parse_coerce.c | 17 +-
src/backend/parser/parse_type.c | 16 +
src/backend/tcop/utility.c | 13 +
src/backend/utils/adt/Makefile | 1 +
src/backend/utils/adt/dictionaries.c | 482 +++++++++++++++++++++++
src/backend/utils/adt/meson.build | 1 +
src/backend/utils/fmgr/funcapi.c | 1 +
src/bin/pg_dump/pg_dump.c | 2 +
src/include/catalog/meson.build | 1 +
src/include/catalog/pg_dict.h | 78 ++++
src/include/catalog/pg_proc.dat | 20 +
src/include/catalog/pg_type.h | 1 +
src/include/commands/typecmds.h | 1 +
src/include/nodes/parsenodes.h | 12 +
src/pl/plpgsql/src/pl_comp.c | 1 +
src/test/regress/expected/dict.out | 47 +++
src/test/regress/expected/oidjoins.out | 1 +
src/test/regress/expected/opr_sanity.out | 14 +-
src/test/regress/parallel_schedule | 2 +-
src/test/regress/sql/dict.sql | 20 +
26 files changed, 1101 insertions(+), 11 deletions(-)
create mode 100644 src/backend/catalog/pg_dict.c
create mode 100644 src/backend/utils/adt/dictionaries.c
create mode 100644 src/include/catalog/pg_dict.h
create mode 100644 src/test/regress/expected/dict.out
create mode 100644 src/test/regress/sql/dict.sql
diff --git a/src/backend/catalog/Makefile b/src/backend/catalog/Makefile
index 89a0221ec9..3b4330148e 100644
--- a/src/backend/catalog/Makefile
+++ b/src/backend/catalog/Makefile
@@ -33,6 +33,7 @@ OBJS = \
pg_conversion.o \
pg_db_role_setting.o \
pg_depend.o \
+ pg_dict.o \
pg_enum.o \
pg_inherits.o \
pg_largeobject.o \
@@ -61,7 +62,7 @@ CATALOG_HEADERS := \
pg_language.h pg_largeobject_metadata.h pg_largeobject.h pg_aggregate.h \
pg_statistic.h pg_statistic_ext.h pg_statistic_ext_data.h \
pg_rewrite.h pg_trigger.h pg_event_trigger.h pg_description.h \
- pg_cast.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \
+ pg_cast.h pg_dict.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \
pg_database.h pg_db_role_setting.h pg_tablespace.h \
pg_authid.h pg_auth_members.h pg_shdepend.h pg_shdescription.h \
pg_ts_config.h pg_ts_config_map.h pg_ts_dict.h \
diff --git a/src/backend/catalog/meson.build b/src/backend/catalog/meson.build
index 0874791451..ab98236c4c 100644
--- a/src/backend/catalog/meson.build
+++ b/src/backend/catalog/meson.build
@@ -18,6 +18,7 @@ backend_sources += files(
'pg_conversion.c',
'pg_db_role_setting.c',
'pg_depend.c',
+ 'pg_dict.c',
'pg_enum.c',
'pg_inherits.c',
'pg_largeobject.c',
diff --git a/src/backend/catalog/pg_dict.c b/src/backend/catalog/pg_dict.c
new file mode 100644
index 0000000000..456f3b5177
--- /dev/null
+++ b/src/backend/catalog/pg_dict.c
@@ -0,0 +1,197 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_dict.c
+ * routines to support manipulation of the pg_dict relation
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/catalog/pg_dict.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "access/genam.h"
+#include "access/skey.h"
+#include "access/table.h"
+#include "catalog/catalog.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_dict.h"
+#include "catalog/pg_type.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/memutils.h"
+#include "utils/rel.h"
+
+/*
+ * Creates an entry in pg_dict for each of the supplied values.
+ * vals is a list of String values.
+ */
+void
+DictEntriesCreate(Oid dictTypeOid, List *vals)
+{
+ Relation pg_dict;
+ NameData dictentry;
+ Datum values[Natts_pg_dict];
+ bool nulls[Natts_pg_dict];
+ ListCell *lc;
+ HeapTuple tup;
+
+ if (vals == NIL)
+ {
+ /* The list is empty; do nothing. */
+ return;
+ }
+
+ memset(nulls, false, sizeof(nulls));
+
+ /*
+ * We don't check the list of values for duplicates here. If there are
+ * any, the user will get an unique-index violation.
+ */
+
+ pg_dict = table_open(DictRelationId, RowExclusiveLock);
+ foreach(lc, vals)
+ {
+ Oid oid = GetNewOidWithIndex(pg_dict, DictOidIndexId,
+ Anum_pg_dict_oid);
+ char *entry = strVal(lfirst(lc));
+
+ /*
+ * Entries are stored in a name field, for easier syscache lookup, so
+ * check the length to make sure it's within range.
+ */
+ if (strlen(entry) > (NAMEDATALEN - 1))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_NAME),
+ errmsg("invalid dict entry \"%s\"", entry),
+ errdetail("Entries must be %d bytes or less.",
+ NAMEDATALEN - 1)));
+
+ /* AALEKSEEV TODO use bytea instead */
+ namestrcpy(&dictentry, entry);
+
+ values[Anum_pg_dict_oid - 1] = ObjectIdGetDatum(oid);
+ values[Anum_pg_dict_dicttypid - 1] = ObjectIdGetDatum(dictTypeOid);
+ values[Anum_pg_dict_dictentry - 1] = NameGetDatum(&dictentry);
+
+ tup = heap_form_tuple(RelationGetDescr(pg_dict), values, nulls);
+
+ CatalogTupleInsert(pg_dict, tup);
+ heap_freetuple(tup);
+ }
+
+ /* clean up */
+ table_close(pg_dict, RowExclusiveLock);
+}
+
+/*
+ * Returns all the entries for the dictinary with given Oid. Entries are sorted
+ * by dictentry. Note that shorter entries are considered smaller, i.e. 'abc'
+ * goes before 'abcdef'. The memory is allocated in a child memory context of
+ * the caller's memory context. It can be freed with DictEntriesFree().
+ *
+ * If there are no entries a valid but empty dictionary is returned.
+ */
+Dictionary
+DictEntriesRead(Oid dictTypeOid)
+{
+ Relation pg_dict;
+ ScanKeyData key[1];
+ SysScanDesc scan;
+ HeapTuple tup;
+ uint32 entries_allocated;
+ Dictionary dict;
+ MemoryContext myctx, oldctx;
+
+ myctx = AllocSetContextCreate(CurrentMemoryContext, "DictEntriesCtx", ALLOCSET_DEFAULT_SIZES);
+ oldctx = MemoryContextSwitchTo(myctx);
+
+ dict = (Dictionary)palloc(sizeof(DictionaryData));
+
+ entries_allocated = 8;
+ dict->nentries = 0;
+ dict->entries = (DictEntry*)palloc(entries_allocated*sizeof(DictEntry));
+
+ pg_dict = table_open(DictRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_dict_dicttypid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(dictTypeOid));
+
+ scan = systable_beginscan(pg_dict, DictTypIdEntryIndexId, true,
+ NULL, 1, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ if(dict->nentries == entries_allocated)
+ {
+ entries_allocated = entries_allocated * 2;
+ dict->entries = (DictEntry*)repalloc(dict->entries, entries_allocated*sizeof(DictEntry));
+ }
+
+ dict->entries[dict->nentries].oid = ((Form_pg_dict) GETSTRUCT(tup))->oid;
+ /*
+ * This is arguably not the fastest way to determine the length of
+ * the entry. Alternatively, we could store a precalculated value in
+ * the catalog. However, usually it's a good idea to keep things simple
+ * until somebody discovers a bottleneck in this exact place and
+ * proposes a concrete fix.
+ */
+ dict->entries[dict->nentries].length = (uint32)strlen(((Form_pg_dict) GETSTRUCT(tup))->dictentry.data);
+ dict->entries[dict->nentries].data = palloc(dict->entries[dict->nentries].length);
+ memcpy(dict->entries[dict->nentries].data, ((Form_pg_dict) GETSTRUCT(tup))->dictentry.data, dict->entries[dict->nentries].length);
+ dict->nentries++;
+ }
+
+ systable_endscan(scan);
+ table_close(pg_dict, RowExclusiveLock);
+
+ MemoryContextSwitchTo(oldctx);
+
+ return dict;
+}
+
+/*
+ * Frees the memory allocated for the dictionary.
+ */
+void
+DictEntriesFree(Dictionary dict)
+{
+ MemoryContextDelete(GetMemoryChunkContext(dict));
+}
+
+/*
+ * Deletes all the entries for the dictinary with given Oid.
+ */
+void
+DictEntriesDelete(Oid dictTypeOid)
+{
+ Relation pg_dict;
+ ScanKeyData key[1];
+ SysScanDesc scan;
+ HeapTuple tup;
+
+ pg_dict = table_open(DictRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_dict_dicttypid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(dictTypeOid));
+
+ scan = systable_beginscan(pg_dict, DictTypIdEntryIndexId, true,
+ NULL, 1, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ CatalogTupleDelete(pg_dict, &tup->t_self);
+ }
+
+ systable_endscan(scan);
+ table_close(pg_dict, RowExclusiveLock);
+}
diff --git a/src/backend/commands/typecmds.c b/src/backend/commands/typecmds.c
index ecc8b3f44c..803638c5b6 100644
--- a/src/backend/commands/typecmds.c
+++ b/src/backend/commands/typecmds.c
@@ -46,6 +46,7 @@
#include "catalog/pg_collation.h"
#include "catalog/pg_constraint.h"
#include "catalog/pg_depend.h"
+#include "catalog/pg_dict.h"
#include "catalog/pg_enum.h"
#include "catalog/pg_language.h"
#include "catalog/pg_namespace.h"
@@ -679,6 +680,14 @@ RemoveTypeById(Oid typeOid)
if (((Form_pg_type) GETSTRUCT(tup))->typtype == TYPTYPE_RANGE)
RangeDelete(typeOid);
+ /*
+ * If it is a dictionary type, delete the pg_dict entries too; we don't
+ * bother with making a dependency entry for that, so it has to be done
+ * "by hand" here.
+ */
+ if (((Form_pg_type) GETSTRUCT(tup))->typtype == TYPTYPE_DICT)
+ DictEntriesDelete(typeOid);
+
ReleaseSysCache(tup);
table_close(relation, RowExclusiveLock);
@@ -763,8 +772,8 @@ DefineDomain(CreateDomainStmt *stmt)
/*
* Base type must be a plain base type, a composite type, another domain,
- * an enum or a range type. Domains over pseudotypes would create a
- * security hole. (It would be shorter to code this to just check for
+ * a dict, an enum or a range type. Domains over pseudotypes would create
+ * a security hole. (It would be shorter to code this to just check for
* pseudotypes; but it seems safer to call out the specific typtypes that
* are supported, rather than assume that all future typtypes would be
* automatically supported.)
@@ -772,6 +781,7 @@ DefineDomain(CreateDomainStmt *stmt)
typtype = baseType->typtype;
if (typtype != TYPTYPE_BASE &&
typtype != TYPTYPE_COMPOSITE &&
+ typtype != TYPTYPE_DICT &&
typtype != TYPTYPE_DOMAIN &&
typtype != TYPTYPE_ENUM &&
typtype != TYPTYPE_RANGE &&
@@ -1129,6 +1139,163 @@ DefineDomain(CreateDomainStmt *stmt)
}
+/*
+ * DefineDictionary
+ * Registers a new dictionary.
+ */
+ObjectAddress
+DefineDictionary(CreateDictionaryStmt * stmt)
+{
+ char *dictName;
+ char *dictBaseTypeName;
+ char *dictArrayName;
+ Oid dictNamespace;
+ AclResult aclresult;
+ Oid old_type_oid;
+ Oid dictArrayOid;
+ ObjectAddress dictTypeAddr;
+
+ Assert(list_length(stmt->baseTypeName) == 1);
+ dictBaseTypeName = strVal(linitial(stmt->baseTypeName));
+ if (pg_strcasecmp(dictBaseTypeName, "jsonb") != 0)
+ ereport(ERROR,
+ (
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("Dictionary types currently can be used only with JSONB")
+ )
+ );
+
+ /* Convert list of names to a name and namespace */
+ dictNamespace = QualifiedNameGetCreationNamespace(stmt->typeName,
+ &dictName);
+
+ /* Check we have creation rights in target namespace */
+ aclresult = object_aclcheck(NamespaceRelationId, dictNamespace, GetUserId(), ACL_CREATE);
+ if (aclresult != ACLCHECK_OK)
+ aclcheck_error(aclresult, OBJECT_SCHEMA,
+ get_namespace_name(dictNamespace));
+
+ /*
+ * Check for collision with an existing type name. If there is one and
+ * it's an autogenerated array, we can rename it out of the way.
+ */
+ old_type_oid = GetSysCacheOid2(TYPENAMENSP, Anum_pg_type_oid,
+ CStringGetDatum(dictName),
+ ObjectIdGetDatum(dictNamespace));
+ if (OidIsValid(old_type_oid))
+ {
+ if (!moveArrayTypeName(old_type_oid, dictName, dictNamespace))
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("type \"%s\" already exists", dictName)));
+ }
+
+ /* Allocate OID for array type */
+ dictArrayOid = AssignTypeArrayOid();
+
+ /* Create the pg_type entry */
+ dictTypeAddr =
+ TypeCreate(InvalidOid, /* no predetermined type OID */
+ dictName, /* type name */
+ dictNamespace, /* namespace */
+ InvalidOid, /* relation oid (n/a here) */
+ 0, /* relation kind (ditto) */
+ GetUserId(), /* owner's ID */
+ -1, /* internal size: varlena, as for JSONB */
+ TYPTYPE_DICT, /* type-type: dictionary */
+ TYPCATEGORY_USER, /* type-category: user, as for JSONB */
+ false, /* dict types are never preferred */
+ DEFAULT_TYPDELIM, /* array element delimiter */
+ F_DICTIONARY_IN, /* input procedure */
+ F_DICTIONARY_OUT, /* output procedure */
+ InvalidOid, /* receive procedure: none */
+ InvalidOid, /* send procedure: none */
+ InvalidOid, /* typmodin procedure - none */
+ InvalidOid, /* typmodout procedure - none */
+ InvalidOid, /* analyze procedure - default */
+ InvalidOid, /* subscript procedure - none */
+ InvalidOid, /* element type ID */
+ false, /* this is not an array type */
+ dictArrayOid, /* array type we are about to create */
+ InvalidOid, /* base type ID (only for domains) */
+ NULL, /* never a default type value */
+ NULL, /* binary default isn't sent either */
+ false, /* passed by value: same as for JSONB */
+ TYPALIGN_INT, /* int alignment */
+ TYPSTORAGE_EXTENDED, /* TOAST strategy: fully toastable, as
+ * JSONB */
+ -1, /* typMod (Domains only) */
+ 0, /* Array dimensions of typbasetype */
+ false, /* Type NOT NULL */
+ InvalidOid); /* type's collation */
+
+ /*
+ * Create the array type that goes with it.
+ */
+ dictArrayName = makeArrayTypeName(dictName, dictNamespace);
+
+ TypeCreate(dictArrayOid, /* force assignment of this type OID */
+ dictArrayName, /* type name */
+ dictNamespace, /* namespace */
+ InvalidOid, /* relation oid (n/a here) */
+ 0, /* relation kind (ditto) */
+ GetUserId(), /* owner's ID */
+ -1, /* internal size (always varlena) */
+ TYPTYPE_BASE, /* type-type (base type) */
+ TYPCATEGORY_ARRAY, /* type-category (array) */
+ false, /* array types are never preferred */
+ DEFAULT_TYPDELIM, /* array element delimiter */
+ F_ARRAY_IN, /* input procedure */
+ F_ARRAY_OUT, /* output procedure */
+ F_ARRAY_RECV, /* receive procedure */
+ F_ARRAY_SEND, /* send procedure */
+ InvalidOid, /* typmodin procedure - none */
+ InvalidOid, /* typmodout procedure - none */
+ F_ARRAY_TYPANALYZE, /* analyze procedure */
+ F_ARRAY_SUBSCRIPT_HANDLER, /* array subscript procedure */
+ dictTypeAddr.objectId, /* element type ID */
+ true, /* yes this is an array type */
+ InvalidOid, /* no further array type */
+ InvalidOid, /* base type ID */
+ NULL, /* never a default type value */
+ NULL, /* binary default isn't sent either */
+ false, /* passed by value: same as for JSONB */
+ TYPALIGN_INT, /* enums have int align, so do their arrays */
+ TYPSTORAGE_EXTENDED, /* ARRAY is always toastable */
+ -1, /* typMod (Domains only) */
+ 0, /* Array dimensions of typbasetype */
+ false, /* Type NOT NULL */
+ InvalidOid); /* type's collation */
+
+ pfree(dictArrayName);
+
+ /* Enter the dict's entries into pg_dict */
+ DictEntriesCreate(dictTypeAddr.objectId, stmt->vals);
+
+ /* Create casts to and from JSONB */
+ CastCreate(JSONBOID, dictTypeAddr.objectId, F_JSONB_DICTIONARY,
+ InvalidOid, InvalidOid,
+ COERCION_CODE_ASSIGNMENT, COERCION_METHOD_FUNCTION,
+ DEPENDENCY_INTERNAL);
+
+ CastCreate(dictTypeAddr.objectId, JSONBOID, F_DICTIONARY_JSONB,
+ InvalidOid, InvalidOid,
+ COERCION_METHOD_INOUT, COERCION_METHOD_FUNCTION,
+ DEPENDENCY_INTERNAL);
+
+ /*
+ * Create explicit cast to bytea. This is convenient for debugging
+ * purposes. Casting bytea to a dictionary type is dangerous and thus not
+ * supported.
+ */
+ CastCreate(dictTypeAddr.objectId, BYTEAOID, F_DICTIONARY_BYTEA,
+ InvalidOid, InvalidOid,
+ COERCION_CODE_EXPLICIT, COERCION_METHOD_FUNCTION,
+ DEPENDENCY_INTERNAL);
+
+ return dictTypeAddr;
+}
+
/*
* DefineEnum
* Registers a new enum.
diff --git a/src/backend/executor/functions.c b/src/backend/executor/functions.c
index e134a82ff7..9674cdfe4d 100644
--- a/src/backend/executor/functions.c
+++ b/src/backend/executor/functions.c
@@ -1721,6 +1721,7 @@ check_sql_fn_retval(List *queryTreeLists,
if (fn_typtype == TYPTYPE_BASE ||
fn_typtype == TYPTYPE_DOMAIN ||
+ fn_typtype == TYPTYPE_DICT ||
fn_typtype == TYPTYPE_ENUM ||
fn_typtype == TYPTYPE_RANGE ||
fn_typtype == TYPTYPE_MULTIRANGE)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 2a910ded15..513dbafcf3 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -6154,6 +6154,14 @@ DefineStmt:
n->coldeflist = $6;
$$ = (Node *) n;
}
+ | CREATE TYPE_P any_name AS DICTIONARY OF any_name '(' opt_enum_val_list ')'
+ {
+ CreateDictionaryStmt *n = makeNode(CreateDictionaryStmt);
+ n->typeName = $3;
+ n->baseTypeName = $7;
+ n->vals = $9;
+ $$ = (Node *)n;
+ }
| CREATE TYPE_P any_name AS ENUM_P '(' opt_enum_val_list ')'
{
CreateEnumStmt *n = makeNode(CreateEnumStmt);
diff --git a/src/backend/parser/parse_coerce.c b/src/backend/parser/parse_coerce.c
index 60908111c8..8952e1469c 100644
--- a/src/backend/parser/parse_coerce.c
+++ b/src/backend/parser/parse_coerce.c
@@ -279,7 +279,22 @@ coerce_type(ParseState *pstate, Node *node,
if (baseTypeId == INTERVALOID)
inputTypeMod = baseTypeMod;
else
- inputTypeMod = -1;
+ {
+ if((int32)targetTypeId == targetTypeMod)
+ {
+ /*
+ * For dictionaries internally we set typmod to a dictionary
+ * OID. This allows dictionary_in() and jsonb_dictionary() to
+ * distinguish one dictionary from another.
+ *
+ * It also allows us to recognize that the target type is
+ * a dictionary here and pass a correct typmod.
+ */
+ inputTypeMod = targetTypeMod;
+ }
+ else
+ inputTypeMod = -1;
+ }
baseType = typeidType(baseTypeId);
diff --git a/src/backend/parser/parse_type.c b/src/backend/parser/parse_type.c
index f7ad689459..2188002535 100644
--- a/src/backend/parser/parse_type.c
+++ b/src/backend/parser/parse_type.c
@@ -259,6 +259,10 @@ LookupTypeNameOid(ParseState *pstate, const TypeName *typeName, bool missing_ok)
* This is equivalent to LookupTypeName, except that this will report
* a suitable error message if the type cannot be found or is not defined.
* Callers of this can therefore assume the result is a fully valid type.
+ *
+ * For a dictionary type it's OID is returned as a typmod. This is used by
+ * dictionary_in() and jsonb_dictionary() to distinguish one dictionary from
+ * another.
*/
Type
typenameType(ParseState *pstate, const TypeName *typeName, int32 *typmod_p)
@@ -305,6 +309,10 @@ typenameTypeId(ParseState *pstate, const TypeName *typeName)
*
* This is equivalent to typenameType, but we only hand back the type OID
* and typmod, not the syscache entry.
+ *
+ * For a dictionary type it's OID is returned as a typmod. This is used by
+ * dictionary_in() and jsonb_dictionary() to distinguish one dictionary from
+ * another.
*/
void
typenameTypeIdAndMod(ParseState *pstate, const TypeName *typeName,
@@ -327,6 +335,10 @@ typenameTypeIdAndMod(ParseState *pstate, const TypeName *typeName,
* looked up, and is passed as "typ".
*
* pstate is only used for error location info, and may be NULL.
+ *
+ * For a dictionary type it's OID is returned as a typmod. This is used by
+ * dictionary_in() and jsonb_dictionary() to distinguish one dictionary from
+ * another.
*/
static int32
typenameTypeMod(ParseState *pstate, const TypeName *typeName, Type typ)
@@ -339,6 +351,10 @@ typenameTypeMod(ParseState *pstate, const TypeName *typeName, Type typ)
ArrayType *arrtypmod;
ParseCallbackState pcbstate;
+ /* For a dictionary return it's OID. */
+ if (((Form_pg_type) GETSTRUCT(typ))->typtype == TYPTYPE_DICT)
+ return (int32)((Form_pg_type) GETSTRUCT(typ))->oid;
+
/* Return prespecified typmod if no typmod expressions */
if (typeName->typmods == NIL)
return typeName->typemod;
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index 247d0816ad..ec8a70c3fc 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -174,6 +174,7 @@ ClassifyUtilityCommandAsReadOnly(Node *parsetree)
case T_CreateCastStmt:
case T_CreateConversionStmt:
case T_CreateDomainStmt:
+ case T_CreateDictionaryStmt:
case T_CreateEnumStmt:
case T_CreateEventTrigStmt:
case T_CreateExtensionStmt:
@@ -1621,6 +1622,10 @@ ProcessUtilitySlow(ParseState *pstate,
}
break;
+ case T_CreateDictionaryStmt: /* CREATE TYPE AS DICTIONARY OF */
+ address = DefineDictionary((CreateDictionaryStmt *) parsetree);
+ break;
+
case T_CreateEnumStmt: /* CREATE TYPE AS ENUM */
address = DefineEnum((CreateEnumStmt *) parsetree);
break;
@@ -2772,6 +2777,10 @@ CreateCommandTag(Node *parsetree)
tag = CMDTAG_CREATE_TYPE;
break;
+ case T_CreateDictionaryStmt:
+ tag = CMDTAG_CREATE_TYPE;
+ break;
+
case T_CreateEnumStmt:
tag = CMDTAG_CREATE_TYPE;
break;
@@ -3419,6 +3428,10 @@ GetCommandLogLevel(Node *parsetree)
lev = LOGSTMT_DDL;
break;
+ case T_CreateDictionaryStmt:
+ lev = LOGSTMT_DDL;
+ break;
+
case T_CreateEnumStmt:
lev = LOGSTMT_DDL;
break;
diff --git a/src/backend/utils/adt/Makefile b/src/backend/utils/adt/Makefile
index 0de0bbb1b8..58e25fd8bc 100644
--- a/src/backend/utils/adt/Makefile
+++ b/src/backend/utils/adt/Makefile
@@ -30,6 +30,7 @@ OBJS = \
datetime.o \
datum.o \
dbsize.o \
+ dictionaries.o \
domains.o \
encode.o \
enum.o \
diff --git a/src/backend/utils/adt/dictionaries.c b/src/backend/utils/adt/dictionaries.c
new file mode 100644
index 0000000000..2bb46471a4
--- /dev/null
+++ b/src/backend/utils/adt/dictionaries.c
@@ -0,0 +1,482 @@
+/*-------------------------------------------------------------------------
+ *
+ * dictionaries.c
+ * Conversion functions for dictionary types.
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/adt/dictionaries.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "catalog/pg_dict.h"
+#include "utils/fmgrprotos.h"
+#include "utils/jsonb.h"
+
+/*
+ * When compressing a data we treat it as a BLOB, in other words we don't
+ * assume anything regarding its internal representation. This is not
+ * necessarily the best and/or the only possible approach. However, it is
+ * universal and can be reused for JSONB, TEXT, XML and other types. Alternative
+ * compression methods can be introduced in the future, so that the user will
+ * be able to choose the best one for the task.
+ *
+ * The compressed data is stored in the following format:
+ *
+ * (struct varlena)
+ * dictionary_id [uint32]
+ * decompressed_size [uint32]
+ * algorithm_version [uint8]
+ *
+ * (repeated) {
+ * number of bytes to copy as-is [uint8]
+ * ... bytes to copy as-is ...
+ * dictionary entry id, or 0 to skip [uint32]
+ * }
+ *
+ * Compressed data is a variable-length type and thus has the 'struct varlena'
+ * header. The code below doesn't consider it being a part of the payload
+ * (e.g. see the DICT_COMP_HEADER_SIZE definition).
+ *
+ * Storing dictionary_id may seem redundant, but without it dictionary_out()
+ * and dictionary_jsonb() have no way to distinguish one dictionary type from
+ * another. dictionary_in() and jsonb_dictionary() get the dictionary id through
+ * the 'typemod' argument.
+ *
+ * Currently algorithm_version is always 0. In the future it will allow us to
+ * introduce new features (e.g. usage of varints) and/or lazily migrate the
+ * data to other compression methods.
+ *
+ * Compression and decompression are implemented based on a binary search over
+ * DictEntry[] array (see pg_dict.h). For compression the array is sorted by
+ * dictentries and for decompression - by oid's.
+ *
+ */
+
+/* Size of the header in the compressed data */
+#define DICT_COMP_HEADER_SIZE (sizeof(uint32)*2 + sizeof(uint8))
+
+/* Extracts dictionary_id from the compressed data */
+#define DICT_COMP_DICTIONARY_ID(hdrp) \
+ (*(uint32*)hdrp)
+
+/* Extracts decompressed_size from the compressed data */
+#define DICT_COMP_DECOMPRESSED_SIZE(hdrp) \
+ (*(uint32*)((uint8*)hdrp + sizeof(uint32)))
+
+/* Extracts algorithm_version from the compressed data */
+#define DICT_COMP_ALGORITHM_VERSION(hdrp) \
+ (*(uint8*)((uint8*)hdrp + sizeof(uint32)*2))
+
+/* Current algorithm_version */
+#define DICT_COMP_CURRENT_ALGORITHM_VERSION 0
+
+/*
+ * bsearch_arg() callback for finding DictEntry by oid.
+ */
+static int
+find_by_oid_cb(const void *key, const void *current, void *arg)
+{
+ /* Note that oids are unsigned, so we should be careful here */
+ if (*(Oid *) key < ((DictEntry *) current)->oid)
+ return -1;
+ else if (*(Oid *) key == ((DictEntry *) current)->oid)
+ return 0; /* found! */
+ else
+ return 1;
+}
+
+/*
+ * qsort() callback for sorting DictEntry[] by oids.
+ */
+static int
+sort_by_oids_cb(const void *left, const void *right)
+{
+ /* Note that oids are unsigned, so we should be careful here */
+ if (((DictEntry *) left)->oid < ((DictEntry *) right)->oid)
+ return -1;
+
+ /* Oids are unique so this callback will never return 0 */
+ return 1;
+}
+
+/*
+ * Finds a DictEntry which dictentry field matches *data and returns its Oid.
+ * If there are several matching entries, the largest is returned. The length
+ * of the found entry is written to *found_length on success. On failure
+ * InvalidOid is returned and found_length is zeroed.
+ *
+ * The implementation is similar to bsearch_arg(). The procedure can't be used
+ * directly because we are looking not by the exact match. We could generalize
+ * this case but the signature of the function becomes so complicated that it
+ * doesn't seem to worth the effort.
+ */
+static Oid
+compress_find_oid(Dictionary dict, const uint8 *data, Size data_size, Size *found_length)
+{
+ int res;
+ int32 left = 0;
+ int32 right = dict->nentries - 1;
+ Size best_length = 0;
+ Oid best_match = InvalidOid;
+
+ while (left <= right)
+ {
+ int32 current = (left + right) / 2;
+ Size nbytes = (Size)dict->entries[current].length;
+
+ if (nbytes > data_size)
+ {
+ /* current can be less or greater depending on the prefix */
+ res = memcmp(dict->entries[current].data, data, data_size);
+
+ /* if prefixes match, current is greater */
+ if (res == 0)
+ res = 1;
+ }
+ else
+ res = memcmp(dict->entries[current].data, data, nbytes);
+
+ if (res == 0) /* match found */
+ {
+ best_length = nbytes;
+ best_match = dict->entries[current].oid;
+
+ if (nbytes == data_size)
+ break;
+
+ /* maybe there is a larger match */
+ left = current + 1;
+ }
+ else if (res < 0) /* current is less */
+ left = current + 1;
+ else /* current is greater */
+ right = current - 1;
+ }
+
+ *found_length = best_length;
+ return best_match;
+}
+
+/*
+ * Finds a DictEntry by Oid using a binary search. The dictionary should be
+ * sorted by oids before the call. Returns NULL if nothing was found.
+ */
+static DictEntry *
+decompress_find_dictentry(Dictionary dict, Oid oid)
+{
+ return (DictEntry *) bsearch_arg(&oid, dict->entries, dict->nentries, sizeof(DictEntry), find_by_oid_cb, NULL);
+}
+
+/*
+ * Estimates the worst-case compressed size of the data of given size.
+ * Worst-case scenario happens when the dictionary consists of single-character
+ * entries. In this case every byte will be encoded as 6 bytes:
+ * 0x00, (0 bytes to copy as-is), 4 bytes of the entry Oid
+ *
+ * This procedure doesn't account for the header size.
+ *
+ * AALEKSEEV FIXME don't use dictionary entries shorter than 5 bytes
+ */
+static Size
+worst_case_compressed_size(Size insize)
+{
+ return insize * 6;
+}
+
+/*
+ * Compresses the data using the provided dictionary. The dictionary should
+ * be sorted by dictentries before the call. Output buffer should be at least
+ * worst_case_compressed_size(src_size) bytes in size.
+ */
+static void
+compress(Dictionary dict,
+ const void *src_data_, Size src_size,
+ void *encoded_data_, Size *pencoded_size)
+{
+ Size nbytes;
+ Size inoffset;
+ Size outskipoffset = 0;
+ Size outoffset = 1;
+ uint8 skipbytes = 0;
+ const uint8 *src_data = src_data_;
+ uint8 *encoded_data = ((uint8 *) encoded_data_);
+
+ for (inoffset = 0; inoffset < src_size;)
+ {
+ Oid code = compress_find_oid(dict, &(src_data[inoffset]),
+ src_size - inoffset, &nbytes);
+
+ if (code == InvalidOid)
+ {
+ skipbytes++;
+ encoded_data[outoffset] = src_data[inoffset];
+ outoffset++;
+ inoffset++;
+
+ if (skipbytes == 255)
+ {
+ encoded_data[outskipoffset] = skipbytes;
+ encoded_data[outoffset++] = 0; /* InvalidOid */
+ encoded_data[outoffset++] = 0;
+ encoded_data[outoffset++] = 0;
+ encoded_data[outoffset++] = 0;
+ outskipoffset = outoffset++;
+ skipbytes = 0;
+ }
+ }
+ else
+ {
+ encoded_data[outskipoffset] = skipbytes;
+ encoded_data[outoffset++] = (code >> 24) & 0xFF;
+ encoded_data[outoffset++] = (code >> 16) & 0xFF;
+ encoded_data[outoffset++] = (code >> 8) & 0xFF;
+ encoded_data[outoffset++] = code & 0xFF;
+ outskipoffset = outoffset++;
+ skipbytes = 0;
+ inoffset += nbytes;
+ }
+ }
+
+ /* Double check that we didn't write out of buffer */
+ Assert(outoffset < worst_case_compressed_size(src_size));
+
+ encoded_data[outskipoffset] = skipbytes;
+ *pencoded_size = outoffset;
+}
+
+/*
+ * Report an internal error in decompress() procedure below.
+ * Under normal circumstances this should never happen.
+ */
+static void
+decompress_error()
+{
+ ereport(ERROR,
+ (
+ errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("Unable to decompress a dictionary type"),
+ errdetail("The compressed data seems to be corrupted"),
+ errhint("Please report the steps to reproduce the issue to pgsql-bugs@")
+ ));
+}
+
+/*
+ * Decompresses the data using the provided dictionary. The dictionary should
+ * be sorted by oids before the call.
+ */
+static void
+decompress(Dictionary dict,
+ const void *encoded_data_, Size encoded_size,
+ void *decoded_data_, Size decoded_size)
+{
+ Size inoffset = 0;
+ Size outoffset = 0;
+ Oid code;
+ uint8 skipbytes;
+ const uint8 *encoded_data = ((uint8 *) encoded_data_);
+ uint8 *decoded_data = decoded_data_;
+
+ for (inoffset = 0; inoffset < encoded_size;)
+ {
+ skipbytes = encoded_data[inoffset++];
+
+ if (skipbytes > decoded_size - outoffset)
+ decompress_error();
+
+ if (skipbytes > encoded_size - inoffset)
+ decompress_error();
+
+ memcpy(
+ &(decoded_data[outoffset]),
+ &(encoded_data[inoffset]),
+ skipbytes
+ );
+
+ outoffset += skipbytes;
+ inoffset += skipbytes;
+
+ if ((encoded_size == inoffset) && (decoded_size == outoffset))
+ break; /* end of input - its OK */
+
+ if (encoded_size - inoffset < 4)
+ decompress_error();
+
+ code = (Oid) encoded_data[inoffset++];
+ code = (code << 8) | (Oid) encoded_data[inoffset++];
+ code = (code << 8) | (Oid) encoded_data[inoffset++];
+ code = (code << 8) | (Oid) encoded_data[inoffset++];
+
+ if (code != InvalidOid)
+ {
+ Size entrylen;
+ DictEntry *entry = decompress_find_dictentry(dict, code);
+
+ if (entry == NULL)
+ decompress_error();
+
+ Assert(entry->oid == code);
+ entrylen = (Size)entry->length;
+
+ if (entrylen > decoded_size - outoffset)
+ decompress_error();
+
+ memcpy(
+ &(decoded_data[outoffset]),
+ entry->data,
+ entrylen
+ );
+
+ outoffset += entrylen;
+ }
+ }
+
+ Assert(decoded_size == outoffset);
+}
+
+/*
+ * Converts a cstring to a dictionary.
+ */
+Datum
+dictionary_in(PG_FUNCTION_ARGS)
+{
+ const char *instr = PG_GETARG_CSTRING(0);
+#ifdef NOT_USED
+ Oid typelem = PG_GETARG_OID(1);
+#endif
+ int32 typmod = PG_GETARG_INT32(2);
+ Jsonb *jsonb;
+ bytea *dict;
+
+ Assert(typmod != -1);
+
+ jsonb = DatumGetJsonbP(DirectFunctionCall1(jsonb_in, CStringGetDatum(instr)));
+ dict = DatumGetByteaP(DirectFunctionCall2(jsonb_dictionary, JsonbPGetDatum(jsonb), Int32GetDatum(typmod)));
+
+ PG_RETURN_BYTEA_P(dict);
+}
+
+/*
+ * Converts a dictionary to a cstring.
+ */
+Datum
+dictionary_out(PG_FUNCTION_ARGS)
+{
+ bytea *dict = PG_GETARG_BYTEA_P(0);
+ Jsonb *jsonb = DatumGetJsonbP(DirectFunctionCall1(dictionary_jsonb, PointerGetDatum(dict)));
+ const char *outstr = DatumGetCString(DirectFunctionCall1(jsonb_out, JsonbPGetDatum(jsonb)));
+
+ PG_RETURN_CSTRING(outstr);
+}
+
+/*
+ * Coverts JSONB to a dictionary type.
+ *
+ * AALEKSEEV FIXME: if the compressed document ends up being larger than
+ * the original one write a corresponding flag and copy all the data as-is.
+ */
+Datum
+jsonb_dictionary(PG_FUNCTION_ARGS)
+{
+ Dictionary dict;
+ Jsonb *jsonb = PG_GETARG_JSONB_P(0);
+ int32 typmod = PG_GETARG_INT32(1);
+ uint8 *jsonb_data = (uint8 *) VARDATA(jsonb);
+ Size jsonb_data_size = VARSIZE(jsonb) - VARHDRSZ;
+ uint8 *encoded_buff,
+ *encoded_header,
+ *encoded_data;
+ Size encoded_size,
+ encoded_buff_size;
+
+ Assert(typmod != -1);
+
+ dict = DictEntriesRead(typmod);
+ qsort((void *) dict->entries, dict->nentries, sizeof(DictEntry), sort_by_oids_cb);
+
+ encoded_buff_size = VARHDRSZ + DICT_COMP_HEADER_SIZE + worst_case_compressed_size(jsonb_data_size);
+ encoded_buff = palloc(encoded_buff_size);
+ encoded_header = (uint8 *) VARDATA(encoded_buff);
+ encoded_data = encoded_header + DICT_COMP_HEADER_SIZE;
+
+ DICT_COMP_DICTIONARY_ID(encoded_header) = typmod;
+ DICT_COMP_DECOMPRESSED_SIZE(encoded_header) = jsonb_data_size;
+ DICT_COMP_ALGORITHM_VERSION(encoded_header) = DICT_COMP_CURRENT_ALGORITHM_VERSION;
+
+ encoded_size = encoded_buff_size - VARHDRSZ - DICT_COMP_HEADER_SIZE;
+
+ compress(dict, jsonb_data, jsonb_data_size,
+ encoded_data, &encoded_size);
+
+ encoded_size += VARHDRSZ + DICT_COMP_HEADER_SIZE;
+
+ encoded_buff = repalloc(encoded_buff, encoded_size);
+ SET_VARSIZE(encoded_buff, encoded_size);
+
+ DictEntriesFree(dict);
+
+ PG_RETURN_BYTEA_P(encoded_buff);
+}
+
+/*
+ * Converts a dictionary type to JSONB.
+ */
+Datum
+dictionary_jsonb(PG_FUNCTION_ARGS)
+{
+ bytea *encoded_buff = PG_GETARG_BYTEA_P(0);
+ uint8 *encoded_header = (uint8 *) VARDATA(encoded_buff);
+ uint8 *encoded_data = encoded_header + DICT_COMP_HEADER_SIZE;
+ Size encoded_size = VARSIZE(encoded_buff) - VARHDRSZ - DICT_COMP_HEADER_SIZE;
+ int alg_version = DICT_COMP_ALGORITHM_VERSION(encoded_header);
+ Oid dictOid; /* cannot read until algorithm version is
+ * checked */
+ uint32 decoded_size; /* cannot read until algorithm version is
+ * checked */
+ Jsonb *jsonb;
+ uint8 *jsonb_data;
+ Dictionary dict;
+
+ if (alg_version > DICT_COMP_CURRENT_ALGORITHM_VERSION)
+ ereport(ERROR,
+ (
+ errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("Unsupported compression algorithm version"),
+ errdetail("Saved algorithm version is %d, current version is %d",
+ alg_version, DICT_COMP_CURRENT_ALGORITHM_VERSION),
+ errhint("The data is either corrupted or imported from "
+ "the future version of PostgreSQL")
+ ));
+
+ dictOid = DICT_COMP_DICTIONARY_ID(encoded_header);
+ decoded_size = DICT_COMP_DECOMPRESSED_SIZE(encoded_header);
+
+ dict = DictEntriesRead(dictOid);
+
+ jsonb = palloc(decoded_size + VARHDRSZ);
+ jsonb_data = (uint8 *) VARDATA(jsonb);
+
+ decompress(dict, encoded_data, encoded_size, jsonb_data, decoded_size);
+
+ decoded_size += VARHDRSZ;
+ SET_VARSIZE(jsonb, decoded_size);
+
+ DictEntriesFree(dict);
+
+ PG_RETURN_JSONB_P(jsonb);
+}
+
+/*
+ * Converts a dictionary type to bytea.
+ */
+Datum
+dictionary_bytea(PG_FUNCTION_ARGS)
+{
+ bytea *compressed_data = PG_GETARG_BYTEA_P(0);
+ PG_RETURN_BYTEA_P(compressed_data);
+}
\ No newline at end of file
diff --git a/src/backend/utils/adt/meson.build b/src/backend/utils/adt/meson.build
index ed9ceadfef..2ed48c1406 100644
--- a/src/backend/utils/adt/meson.build
+++ b/src/backend/utils/adt/meson.build
@@ -17,6 +17,7 @@ backend_sources += files(
'datetime.c',
'datum.c',
'dbsize.c',
+ 'dictionaries.c',
'domains.c',
'encode.c',
'enum.c',
diff --git a/src/backend/utils/fmgr/funcapi.c b/src/backend/utils/fmgr/funcapi.c
index 87cbb1d3e3..efb3e739e4 100644
--- a/src/backend/utils/fmgr/funcapi.c
+++ b/src/backend/utils/fmgr/funcapi.c
@@ -1326,6 +1326,7 @@ get_type_func_class(Oid typid, Oid *base_typeid)
case TYPTYPE_COMPOSITE:
return TYPEFUNC_COMPOSITE;
case TYPTYPE_BASE:
+ case TYPTYPE_DICT:
case TYPTYPE_ENUM:
case TYPTYPE_RANGE:
case TYPTYPE_MULTIRANGE:
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index da427f4d4a..38638b88cc 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -10218,6 +10218,8 @@ dumpType(Archive *fout, const TypeInfo *tyinfo)
dumpDomain(fout, tyinfo);
else if (tyinfo->typtype == TYPTYPE_COMPOSITE)
dumpCompositeType(fout, tyinfo);
+ else if (tyinfo->typtype == TYPTYPE_DICT)
+ pg_log_error("AALEKSEEV TODO FIXME not implemented");
else if (tyinfo->typtype == TYPTYPE_ENUM)
dumpEnumType(fout, tyinfo);
else if (tyinfo->typtype == TYPTYPE_RANGE)
diff --git a/src/include/catalog/meson.build b/src/include/catalog/meson.build
index 45ffa99692..8d76de46f3 100644
--- a/src/include/catalog/meson.build
+++ b/src/include/catalog/meson.build
@@ -25,6 +25,7 @@ catalog_headers = [
'pg_event_trigger.h',
'pg_description.h',
'pg_cast.h',
+ 'pg_dict.h',
'pg_enum.h',
'pg_namespace.h',
'pg_conversion.h',
diff --git a/src/include/catalog/pg_dict.h b/src/include/catalog/pg_dict.h
new file mode 100644
index 0000000000..fbad803468
--- /dev/null
+++ b/src/include/catalog/pg_dict.h
@@ -0,0 +1,78 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_dict.h
+ * definition of the "dict" system catalog (pg_dict)
+ *
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ * src/include/catalog/pg_dict.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef PG_DICT_H
+#define PG_DICT_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_dict_d.h"
+
+#include "nodes/pg_list.h"
+
+/* ----------------
+ * pg_dict definition. cpp turns this into
+ * typedef struct FormData_pg_dict
+ * ----------------
+ */
+CATALOG(pg_dict,9861,DictRelationId)
+{
+ Oid oid; /* oid */
+ Oid dicttypid BKI_LOOKUP(pg_type); /* OID of owning dict type */
+ /* AALEKSEEV TODO refactor to bytea */
+ NameData dictentry; /* text representation of the dictionary entry */
+} FormData_pg_dict;
+
+/* ----------------
+ * Form_pg_dict corresponds to a pointer to a tuple with
+ * the format of pg_dict relation.
+ * ----------------
+ */
+typedef FormData_pg_dict *Form_pg_dict;
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_dict_oid_index, 9862, DictOidIndexId, on pg_dict using btree(oid oid_ops));
+DECLARE_UNIQUE_INDEX(pg_dict_typid_entry_index, 9863, DictTypIdEntryIndexId, on pg_dict using btree(dicttypid oid_ops, dictentry name_ops));
+
+
+/*
+ * DictEntry type represents one entry in the given dictionary.
+ */
+typedef struct
+{
+ Oid oid; /* entry id */
+ uint32 length; /* entry length */
+ uint8* data; /* entry data */
+} DictEntry;
+
+/*
+ * Dictionary type represents all the entries in the given dictionary.
+ */
+typedef struct
+{
+ uint32 nentries; /* number of entries */
+ DictEntry *entries; /* array of entries */
+} DictionaryData;
+
+typedef DictionaryData *Dictionary;
+
+/*
+ * prototypes for functions in pg_dict.c
+ */
+extern void DictEntriesCreate(Oid dictTypeOid, List *vals);
+extern void DictEntriesFree(Dictionary dict);
+extern Dictionary DictEntriesRead(Oid dictTypeOid);
+extern void DictEntriesDelete(Oid dictTypeOid);
+
+#endif /* PG_DICT_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 9dbe9ec801..62629cf3d6 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -9966,6 +9966,26 @@
proname => 'jsonb_path_match_opr', prorettype => 'bool',
proargtypes => 'jsonb jsonpath', prosrc => 'jsonb_path_match_opr' },
+# dictionaries
+{ oid => '9864', descr => 'Converts a cstring to a dictionary',
+ proname => 'dictionary_in', prorettype => 'any', proargtypes => 'cstring oid int4',
+ prosrc => 'dictionary_in' },
+{ oid => '9865', descr => 'Converts a dictionary to a cstring',
+ proname => 'dictionary_out', prorettype => 'cstring', proargtypes => 'any',
+ prosrc => 'dictionary_out' },
+{ oid => '9866', descr => 'Coverts JSONB to a dictionary type',
+ proname => 'jsonb_dictionary', proisstrict => 'f', provolatile => 's',
+ prorettype => 'any', proargtypes => 'jsonb int4',
+ prosrc => 'jsonb_dictionary' },
+{ oid => '9867', descr => 'Converts a dictionary type to JSONB',
+ proname => 'dictionary_jsonb', proisstrict => 'f', provolatile => 's',
+ prorettype => 'jsonb', proargtypes => 'any',
+ prosrc => 'dictionary_jsonb' },
+{ oid => '9868', descr => 'Converts a dictionary type to a byte array',
+ proname => 'dictionary_bytea', proisstrict => 'f', provolatile => 's',
+ prorettype => 'jsonb', proargtypes => 'any',
+ prosrc => 'dictionary_bytea' },
+
# historical int8/txid_snapshot variants of xid8 functions
{ oid => '2939', descr => 'I/O',
proname => 'txid_snapshot_in', prorettype => 'txid_snapshot',
diff --git a/src/include/catalog/pg_type.h b/src/include/catalog/pg_type.h
index 48a2559137..91c6bfae99 100644
--- a/src/include/catalog/pg_type.h
+++ b/src/include/catalog/pg_type.h
@@ -272,6 +272,7 @@ DECLARE_UNIQUE_INDEX(pg_type_typname_nsp_index, 2704, TypeNameNspIndexId, on pg_
*/
#define TYPTYPE_BASE 'b' /* base type (ordinary scalar type) */
#define TYPTYPE_COMPOSITE 'c' /* composite (e.g., table's rowtype) */
+#define TYPTYPE_DICT 'D' /* dictionary */
#define TYPTYPE_DOMAIN 'd' /* domain over another type */
#define TYPTYPE_ENUM 'e' /* enumerated type */
#define TYPTYPE_MULTIRANGE 'm' /* multirange type */
diff --git a/src/include/commands/typecmds.h b/src/include/commands/typecmds.h
index 532bc49c38..a3ef87741d 100644
--- a/src/include/commands/typecmds.h
+++ b/src/include/commands/typecmds.h
@@ -24,6 +24,7 @@
extern ObjectAddress DefineType(ParseState *pstate, List *names, List *parameters);
extern void RemoveTypeById(Oid typeOid);
extern ObjectAddress DefineDomain(CreateDomainStmt *stmt);
+extern ObjectAddress DefineDictionary(CreateDictionaryStmt *stmt);
extern ObjectAddress DefineEnum(CreateEnumStmt *stmt);
extern ObjectAddress DefineRange(ParseState *pstate, CreateRangeStmt *stmt);
extern ObjectAddress AlterEnum(AlterEnumStmt *stmt);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 7caff62af7..8b127086c9 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3296,6 +3296,18 @@ typedef struct CompositeTypeStmt
List *coldeflist; /* list of ColumnDef nodes */
} CompositeTypeStmt;
+/* ----------------------
+ * Create Type Statement, dictionary types
+ * ----------------------
+ */
+typedef struct CreateDictionaryStmt
+{
+ NodeTag type;
+ List *typeName; /* qualified name (list of String) */
+ List *baseTypeName; /* qualified base type name (list of String) */
+ List *vals; /* dictionary entries (list of String) */
+} CreateDictionaryStmt;
+
/* ----------------------
* Create Type Statement, enum types
* ----------------------
diff --git a/src/pl/plpgsql/src/pl_comp.c b/src/pl/plpgsql/src/pl_comp.c
index b286f2a50c..2754098f7f 100644
--- a/src/pl/plpgsql/src/pl_comp.c
+++ b/src/pl/plpgsql/src/pl_comp.c
@@ -2123,6 +2123,7 @@ build_datatype(HeapTuple typeTup, int32 typmod,
switch (typeStruct->typtype)
{
case TYPTYPE_BASE:
+ case TYPTYPE_DICT:
case TYPTYPE_ENUM:
case TYPTYPE_RANGE:
case TYPTYPE_MULTIRANGE:
diff --git a/src/test/regress/expected/dict.out b/src/test/regress/expected/dict.out
new file mode 100644
index 0000000000..699e76b654
--- /dev/null
+++ b/src/test/regress/expected/dict.out
@@ -0,0 +1,47 @@
+--
+-- Compression dictionaries tests
+--
+-- Dictionary types currently can be used only with JSONB
+\set ON_ERROR_STOP 0
+CREATE TYPE textdict AS DICTIONARY OF TEXT ('abcdef', 'ghijkl');
+ERROR: Dictionary types currently can be used only with JSONB
+\set ON_ERROR_STOP 1
+-- Simple happy path
+CREATE TYPE mydict AS DICTIONARY OF JSONB ('abcdef', 'ghijkl');
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
+ dictentry
+-----------
+ abcdef
+ ghijkl
+(2 rows)
+
+SELECT '{"abcdef":"ghijkl"}' :: mydict;
+ mydict
+----------------------
+ {"abcdef": "ghijkl"}
+(1 row)
+
+SELECT '{"abcdef":"ghijkl"}' :: mydict :: jsonb;
+ jsonb
+----------------------
+ {"abcdef": "ghijkl"}
+(1 row)
+
+SELECT '{"abcdef":"ghijkl"}' :: jsonb :: mydict;
+ mydict
+----------------------
+ {"abcdef": "ghijkl"}
+(1 row)
+
+SELECT ('{"abcdef":"ghijkl"}' :: mydict) -> 'abcdef';
+ ?column?
+----------
+ "ghijkl"
+(1 row)
+
+DROP TYPE mydict;
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
+ dictentry
+-----------
+(0 rows)
+
diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out
index 215eb899be..91bc491a6c 100644
--- a/src/test/regress/expected/oidjoins.out
+++ b/src/test/regress/expected/oidjoins.out
@@ -182,6 +182,7 @@ NOTICE: checking pg_description {classoid} => pg_class {oid}
NOTICE: checking pg_cast {castsource} => pg_type {oid}
NOTICE: checking pg_cast {casttarget} => pg_type {oid}
NOTICE: checking pg_cast {castfunc} => pg_proc {oid}
+NOTICE: checking pg_dict {dicttypid} => pg_type {oid}
NOTICE: checking pg_enum {enumtypid} => pg_type {oid}
NOTICE: checking pg_namespace {nspowner} => pg_authid {oid}
NOTICE: checking pg_conversion {connamespace} => pg_namespace {oid}
diff --git a/src/test/regress/expected/opr_sanity.out b/src/test/regress/expected/opr_sanity.out
index 330eb0f765..f435feb108 100644
--- a/src/test/regress/expected/opr_sanity.out
+++ b/src/test/regress/expected/opr_sanity.out
@@ -388,11 +388,12 @@ WHERE 'cstring'::regtype = ANY (p1.proargtypes)
AND NOT EXISTS(SELECT 1 FROM pg_conversion WHERE conproc = p1.oid)
AND p1.oid != 'shell_in(cstring)'::regprocedure
ORDER BY 1;
- oid | proname
-------+--------------
+ oid | proname
+------+---------------
2293 | cstring_out
2501 | cstring_send
-(2 rows)
+ 9864 | dictionary_in
+(3 rows)
-- Likewise, look for functions that return cstring and aren't datatype output
-- functions nor typmod output functions.
@@ -405,11 +406,12 @@ WHERE p1.prorettype = 'cstring'::regtype
AND NOT EXISTS(SELECT 1 FROM pg_type WHERE typmodout = p1.oid)
AND p1.oid != 'shell_out(void)'::regprocedure
ORDER BY 1;
- oid | proname
-------+--------------
+ oid | proname
+------+----------------
2292 | cstring_in
2500 | cstring_recv
-(2 rows)
+ 9865 | dictionary_out
+(3 rows)
-- Check for length inconsistencies between the various argument-info arrays.
SELECT p1.oid, p1.proname
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 9a139f1e24..385398545b 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -111,7 +111,7 @@ test: select_views portals_p2 foreign_key cluster dependency guc bitmapops combo
# ----------
# Another group of parallel tests (JSON related)
# ----------
-test: json jsonb json_encoding jsonpath jsonpath_encoding jsonb_jsonpath
+test: dict json jsonb json_encoding jsonpath jsonpath_encoding jsonb_jsonpath
# ----------
# Another group of parallel tests
diff --git a/src/test/regress/sql/dict.sql b/src/test/regress/sql/dict.sql
new file mode 100644
index 0000000000..27ecd6a451
--- /dev/null
+++ b/src/test/regress/sql/dict.sql
@@ -0,0 +1,20 @@
+--
+-- Compression dictionaries tests
+--
+
+-- Dictionary types currently can be used only with JSONB
+\set ON_ERROR_STOP 0
+CREATE TYPE textdict AS DICTIONARY OF TEXT ('abcdef', 'ghijkl');
+\set ON_ERROR_STOP 1
+
+-- Simple happy path
+CREATE TYPE mydict AS DICTIONARY OF JSONB ('abcdef', 'ghijkl');
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
+
+SELECT '{"abcdef":"ghijkl"}' :: mydict;
+SELECT '{"abcdef":"ghijkl"}' :: mydict :: jsonb;
+SELECT '{"abcdef":"ghijkl"}' :: jsonb :: mydict;
+SELECT ('{"abcdef":"ghijkl"}' :: mydict) -> 'abcdef';
+
+DROP TYPE mydict;
+SELECT dictentry FROM pg_dict ORDER BY dictentry;
--
2.38.1
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v2 01/11] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/RgO6TscyR9fMvkEm1k5N=yu
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v2-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v3 01/11] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/.arW0LC=Yi8JO9BRih4YlyS
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v3-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v5 01/14] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_//gBZFZYGWm7urqUgDbu6PSe
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v5-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v6 01/15] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/74urtnrsBSymuH7bJczNOGS
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v6-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v7 01/16] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/VSGM3xNEmY7iJyL2wuWRCjV
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v7-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v2 01/11] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/RgO6TscyR9fMvkEm1k5N=yu
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v2-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v3 01/11] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/.arW0LC=Yi8JO9BRih4YlyS
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v3-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: [PATCH] Compression dictionaries for JSONB
@ 2024-01-17 10:50 Shubham Khanna <[email protected]>
parent: Aleksander Alekseev <[email protected]>
0 siblings, 1 reply; 88+ messages in thread
From: Shubham Khanna @ 2024-01-17 10:50 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>; Simon Riggs <[email protected]>; Nikita Malakhov <[email protected]>; Matthias van de Meent <[email protected]>; Jacob Champion <[email protected]>; Alvaro Herrera <[email protected]>; Zhihong Yu <[email protected]>
On Wed, Jan 17, 2024 at 4:16 PM Aleksander Alekseev
<[email protected]> wrote:
>
> Hi hackers,
>
> > 8272749e added a few more arguments to CastCreate(). Here is the rebased patch.
>
> After merging afbfc029 [1] the patch needed a rebase. PFA v10.
>
> The patch is still in a PoC state and this is exactly why comments and
> suggestions from the community are most welcome! Particularly I would
> like to know:
>
> 1. Would you call it a wanted feature considering the existence of
> Pluggable TOASTer patchset which (besides other things) tries to
> introduce type-aware TOASTers for EXTERNAL attributes? I know what
> Simon's [2] and Nikita's latest answers were, and I know my personal
> opinion on this [3][4], but I would like to hear from the rest of the
> community.
>
> 2. How should we make sure a dictionary will not consume all the
> available memory? Limiting the amount of dictionary entries to pow(2,
> 16) and having dictionary versions seems to work OK for ZSON. However
> it was pointed out that this may be an unwanted limitation for the
> in-core implementation.
>
> [1]: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=c727f511;hp=afbfc02983f86c4d71825...
> [2]: https://www.postgresql.org/message-id/CANbhV-HpCF852WcZuU0wyh1jMU4p6XLbV6rCRkZpnpeKQ9OenQ%40mail.gma...
> [3]: https://www.postgresql.org/message-id/CAJ7c6TN-N3%3DPSykmOjmW1EAf9YyyHFDHEznX-5VORsWUvVN-5w%40mail.g...
> [4]: https://www.postgresql.org/message-id/CAJ7c6TO2XTTk3cu5w6ePHfhYQkoNpw7u1jeqHf%3DGwn%2BoWci8eA%40mail...
I tried to apply the patch but it is failing at the Head. It is giving
the following error:
patching file src/test/regress/expected/dict.out
patching file src/test/regress/expected/oidjoins.out
patching file src/test/regress/expected/opr_sanity.out
patching file src/test/regress/parallel_schedule
Hunk #1 FAILED at 111.
1 out of 1 hunk FAILED -- saving rejects to file
src/test/regress/parallel_schedule.rej
patching file src/test/regress/sql/dict.sql
Please send the Re-base version of the patch.
Thanks and Regards,
Shubham Khanna.
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: [PATCH] Compression dictionaries for JSONB
@ 2024-01-17 14:21 Aleksander Alekseev <[email protected]>
parent: Shubham Khanna <[email protected]>
0 siblings, 2 replies; 88+ messages in thread
From: Aleksander Alekseev @ 2024-01-17 14:21 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>; +Cc: Shubham Khanna <[email protected]>; Simon Riggs <[email protected]>; Nikita Malakhov <[email protected]>; Matthias van de Meent <[email protected]>; Alvaro Herrera <[email protected]>; Zhihong Yu <[email protected]>
Hi Shubham,
> > > 8272749e added a few more arguments to CastCreate(). Here is the rebased patch.
> >
> > After merging afbfc029 [1] the patch needed a rebase. PFA v10.
> >
> > The patch is still in a PoC state and this is exactly why comments and
> > suggestions from the community are most welcome! Particularly I would
> > like to know:
> >
> > 1. Would you call it a wanted feature considering the existence of
> > Pluggable TOASTer patchset which (besides other things) tries to
> > introduce type-aware TOASTers for EXTERNAL attributes? I know what
> > Simon's [2] and Nikita's latest answers were, and I know my personal
> > opinion on this [3][4], but I would like to hear from the rest of the
> > community.
> >
> > 2. How should we make sure a dictionary will not consume all the
> > available memory? Limiting the amount of dictionary entries to pow(2,
> > 16) and having dictionary versions seems to work OK for ZSON. However
> > it was pointed out that this may be an unwanted limitation for the
> > in-core implementation.
> >
> > [1]: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=c727f511;hp=afbfc02983f86c4d71825...
> > [2]: https://www.postgresql.org/message-id/CANbhV-HpCF852WcZuU0wyh1jMU4p6XLbV6rCRkZpnpeKQ9OenQ%40mail.gma...
> > [3]: https://www.postgresql.org/message-id/CAJ7c6TN-N3%3DPSykmOjmW1EAf9YyyHFDHEznX-5VORsWUvVN-5w%40mail.g...
> > [4]: https://www.postgresql.org/message-id/CAJ7c6TO2XTTk3cu5w6ePHfhYQkoNpw7u1jeqHf%3DGwn%2BoWci8eA%40mail...
>
> I tried to apply the patch but it is failing at the Head. It is giving
> the following error:
Yes it does for a while now. Until we reach any agreement regarding
questions (1) and (2) personally I don't see the point in submitting
rebased patches. We can continue the discussion but mark the CF entry
as RwF for now it will be helpful.
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: [PATCH] Compression dictionaries for JSONB
@ 2024-01-17 14:27 Aleksander Alekseev <[email protected]>
parent: Aleksander Alekseev <[email protected]>
1 sibling, 1 reply; 88+ messages in thread
From: Aleksander Alekseev @ 2024-01-17 14:27 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>; +Cc: Shubham Khanna <[email protected]>; Nikita Malakhov <[email protected]>; Matthias van de Meent <[email protected]>; Alvaro Herrera <[email protected]>
Hi again,
> Yes it does for a while now. Until we reach any agreement regarding
> questions (1) and (2) personally I don't see the point in submitting
> rebased patches. We can continue the discussion but mark the CF entry
> as RwF for now it will be helpful.
Sorry, what I actually meant were the following questions:
"""
Several things occured to me:
- Does anyone believe that va_tag should be part of the utf8-like
bitmask in order to save a byte or two?
- The described approach means that compression dictionaries are not
going to be used when data is compressed in-place (i.e. within a
tuple), since no TOAST pointer is involved in this case. Also we will
be unable to add additional compression algorithms here. Does anyone
have problems with this? Should we use the reserved compression
algorithm id instead as a marker of an extended TOAST?
- It would be nice to decompose the feature in several independent
patches, e.g. modify TOAST first, then add compression dictionaries
without automatic update of the dictionaries, then add the automatic
update. I find it difficult to imagine however how to modify TOAST
pointers and test the code properly without a dependency on a larger
feature. Could anyone think of a trivial test case for extendable
TOAST? Maybe something we could add to src/test/modules similarly to
how we test SLRU, background workers, etc.
"""
Since there was not much activity since then (for 3 months) I don't
really see how to process further.
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: [PATCH] Compression dictionaries for JSONB
@ 2024-01-17 14:43 Nikita Malakhov <[email protected]>
parent: Aleksander Alekseev <[email protected]>
0 siblings, 0 replies; 88+ messages in thread
From: Nikita Malakhov @ 2024-01-17 14:43 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>; Shubham Khanna <[email protected]>; Matthias van de Meent <[email protected]>; Alvaro Herrera <[email protected]>
Hi,
Aleksander, there was a quite straightforward answer regarding Pluggable
TOAST
in other thread - the Pluggable TOAST feature is not desired by the
community,
and advanced TOAST mechanics would be accepted as parts of problematic
datatypes extended functionality, on a par with in and out functions, so
what I am
actually doing now - re-writing JSONb TOAST improvements to be called as
separate
functions without Pluggable TOAST API. This takes some time because there
is a large
and complex code base left by Nikita Glukhov who has lost interest in this
work due
to some reasons.
I, personally, think that these two features could benefit from each other,
but they could
be adapted to each other after I would introduce JSONb Toaster in v17
master.
If you don't mind please check the thread on extending the TOAST pointer -
it is important
for improving TOAST mechanics.
On Wed, Jan 17, 2024 at 5:27 PM Aleksander Alekseev <
[email protected]> wrote:
> Hi again,
>
> > Yes it does for a while now. Until we reach any agreement regarding
> > questions (1) and (2) personally I don't see the point in submitting
> > rebased patches. We can continue the discussion but mark the CF entry
> > as RwF for now it will be helpful.
>
> Sorry, what I actually meant were the following questions:
>
> """
> Several things occured to me:
>
> - Does anyone believe that va_tag should be part of the utf8-like
> bitmask in order to save a byte or two?
>
> - The described approach means that compression dictionaries are not
> going to be used when data is compressed in-place (i.e. within a
> tuple), since no TOAST pointer is involved in this case. Also we will
> be unable to add additional compression algorithms here. Does anyone
> have problems with this? Should we use the reserved compression
> algorithm id instead as a marker of an extended TOAST?
>
> - It would be nice to decompose the feature in several independent
> patches, e.g. modify TOAST first, then add compression dictionaries
> without automatic update of the dictionaries, then add the automatic
> update. I find it difficult to imagine however how to modify TOAST
> pointers and test the code properly without a dependency on a larger
> feature. Could anyone think of a trivial test case for extendable
> TOAST? Maybe something we could add to src/test/modules similarly to
> how we test SLRU, background workers, etc.
> """
>
> Since there was not much activity since then (for 3 months) I don't
> really see how to process further.
>
> --
> Best regards,
> Aleksander Alekseev
>
--
Regards,
Nikita Malakhov
Postgres Professional
The Russian Postgres Company
https://postgrespro.ru/
^ permalink raw reply [nested|flat] 88+ messages in thread
* Re: [PATCH] Compression dictionaries for JSONB
@ 2024-01-26 13:23 vignesh C <[email protected]>
parent: Aleksander Alekseev <[email protected]>
1 sibling, 0 replies; 88+ messages in thread
From: vignesh C @ 2024-01-26 13:23 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>; Shubham Khanna <[email protected]>; Simon Riggs <[email protected]>; Nikita Malakhov <[email protected]>; Matthias van de Meent <[email protected]>; Alvaro Herrera <[email protected]>; Zhihong Yu <[email protected]>
On Wed, 17 Jan 2024 at 19:52, Aleksander Alekseev
<[email protected]> wrote:
>
> Hi Shubham,
>
> > > > 8272749e added a few more arguments to CastCreate(). Here is the rebased patch.
> > >
> > > After merging afbfc029 [1] the patch needed a rebase. PFA v10.
> > >
> > > The patch is still in a PoC state and this is exactly why comments and
> > > suggestions from the community are most welcome! Particularly I would
> > > like to know:
> > >
> > > 1. Would you call it a wanted feature considering the existence of
> > > Pluggable TOASTer patchset which (besides other things) tries to
> > > introduce type-aware TOASTers for EXTERNAL attributes? I know what
> > > Simon's [2] and Nikita's latest answers were, and I know my personal
> > > opinion on this [3][4], but I would like to hear from the rest of the
> > > community.
> > >
> > > 2. How should we make sure a dictionary will not consume all the
> > > available memory? Limiting the amount of dictionary entries to pow(2,
> > > 16) and having dictionary versions seems to work OK for ZSON. However
> > > it was pointed out that this may be an unwanted limitation for the
> > > in-core implementation.
> > >
> > > [1]: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=c727f511;hp=afbfc02983f86c4d71825...
> > > [2]: https://www.postgresql.org/message-id/CANbhV-HpCF852WcZuU0wyh1jMU4p6XLbV6rCRkZpnpeKQ9OenQ%40mail.gma...
> > > [3]: https://www.postgresql.org/message-id/CAJ7c6TN-N3%3DPSykmOjmW1EAf9YyyHFDHEznX-5VORsWUvVN-5w%40mail.g...
> > > [4]: https://www.postgresql.org/message-id/CAJ7c6TO2XTTk3cu5w6ePHfhYQkoNpw7u1jeqHf%3DGwn%2BoWci8eA%40mail...
> >
> > I tried to apply the patch but it is failing at the Head. It is giving
> > the following error:
>
> Yes it does for a while now. Until we reach any agreement regarding
> questions (1) and (2) personally I don't see the point in submitting
> rebased patches. We can continue the discussion but mark the CF entry
> as RwF for now it will be helpful.
Thanks. I have updated the status to "Returned with feedback". Feel
free to create a new entry after we agree on the approach to take it
forward.
Regards,
Vignesh
^ permalink raw reply [nested|flat] 88+ messages in thread
end of thread, other threads:[~2024-01-26 13:23 UTC | newest]
Thread overview: 88+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2018-11-01 01:40 Re: Super PathKeys (Allowing sort order through precision loss functions) David Rowley <[email protected]>
2018-11-01 09:44 ` Tomas Vondra <[email protected]>
2018-11-29 17:21 Re: Changing the autovacuum launcher scheduling; oldest table first algorithm Dmitry Dolgov <[email protected]>
2018-11-30 01:48 ` Re: Changing the autovacuum launcher scheduling; oldest table first algorithm Michael Paquier <[email protected]>
2018-11-30 02:00 ` Re: Changing the autovacuum launcher scheduling; oldest table first algorithm Masahiko Sawada <[email protected]>
2018-11-30 09:40 ` Re: Changing the autovacuum launcher scheduling; oldest table first algorithm Dmitry Dolgov <[email protected]>
2020-03-06 23:12 [PATCH v12 03/11] Document historic behavior about hiding directories and special files Justin Pryzby <[email protected]>
2022-04-22 15:21 Re: [PATCH] Compression dictionaries for JSONB Zhihong Yu <[email protected]>
2022-04-25 13:15 ` Re: [PATCH] Compression dictionaries for JSONB Aleksander Alekseev <[email protected]>
2022-05-13 08:08 ` Re: [PATCH] Compression dictionaries for JSONB Aleksander Alekseev <[email protected]>
2022-06-05 19:51 ` Re: [PATCH] Compression dictionaries for JSONB Matthias van de Meent <[email protected]>
2022-06-15 12:38 ` Re: [PATCH] Compression dictionaries for JSONB Aleksander Alekseev <[email protected]>
2022-06-17 15:04 ` Re: [PATCH] Compression dictionaries for JSONB Aleksander Alekseev <[email protected]>
2022-07-04 12:45 ` Re: [PATCH] Compression dictionaries for JSONB Matthias van de Meent <[email protected]>
2022-07-04 14:00 ` Re: [PATCH] Compression dictionaries for JSONB Aleksander Alekseev <[email protected]>
2022-07-11 14:44 ` Re: [PATCH] Compression dictionaries for JSONB Aleksander Alekseev <[email protected]>
2022-07-11 15:41 ` Re: [PATCH] Compression dictionaries for JSONB Aleksander Alekseev <[email protected]>
2022-07-12 10:25 ` Re: [PATCH] Compression dictionaries for JSONB Nikita Malakhov <[email protected]>
2022-07-12 12:15 ` Re: [PATCH] Compression dictionaries for JSONB Aleksander Alekseev <[email protected]>
2022-07-17 18:15 ` Re: [PATCH] Compression dictionaries for JSONB Nikita Malakhov <[email protected]>
2022-07-18 12:26 ` Re: [PATCH] Compression dictionaries for JSONB Aleksander Alekseev <[email protected]>
2022-07-27 07:36 ` Re: [PATCH] Compression dictionaries for JSONB Simon Riggs <[email protected]>
2022-07-27 08:30 ` Re: [PATCH] Compression dictionaries for JSONB Matthias van de Meent <[email protected]>
2022-08-01 11:25 ` Re: [PATCH] Compression dictionaries for JSONB Aleksander Alekseev <[email protected]>
2022-08-19 07:57 ` Re: [PATCH] Compression dictionaries for JSONB Nikita Malakhov <[email protected]>
2022-09-02 10:50 ` Re: [PATCH] Compression dictionaries for JSONB Aleksander Alekseev <[email protected]>
2022-10-06 10:29 ` Re: [PATCH] Compression dictionaries for JSONB Aleksander Alekseev <[email protected]>
2022-11-04 08:37 ` Re: [PATCH] Compression dictionaries for JSONB Aleksander Alekseev <[email protected]>
2022-11-17 10:36 ` Re: [PATCH] Compression dictionaries for JSONB Aleksander Alekseev <[email protected]>
2024-01-17 10:50 ` Re: [PATCH] Compression dictionaries for JSONB Shubham Khanna <[email protected]>
2024-01-17 14:21 ` Re: [PATCH] Compression dictionaries for JSONB Aleksander Alekseev <[email protected]>
2024-01-17 14:27 ` Re: [PATCH] Compression dictionaries for JSONB Aleksander Alekseev <[email protected]>
2024-01-17 14:43 ` Re: [PATCH] Compression dictionaries for JSONB Nikita Malakhov <[email protected]>
2024-01-26 13:23 ` Re: [PATCH] Compression dictionaries for JSONB vignesh C <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v3 01/11] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v3 01/11] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v5 01/14] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v2 01/11] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v7 01/16] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v6 01/15] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v2 01/11] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox