agora inbox for [email protected]
help / color / mirror / Atom feedShortened URLs for commit messages
55+ messages / 6 participants
[nested] [flat]
* Shortened URLs for commit messages
@ 2017-05-24 03:25 Bruce Momjian <[email protected]>
2017-06-15 17:05 ` Re: Shortened URLs for commit messages Bruce Momjian <[email protected]>
0 siblings, 1 reply; 55+ messages in thread
From: Bruce Momjian @ 2017-05-24 03:25 UTC (permalink / raw)
To: pgsql-hackers
I have written the following sed script to convert regular Postgres
email message URLs to their shorter form for commit messages:
sed 's;http\(s\?\)://www\.postgresql\.org/message-id/;http\1://postgr.es/m/;gi'
in case this is helpful to anyone.
--
Bruce Momjian <[email protected]> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 55+ messages in thread
* Re: Shortened URLs for commit messages
2017-05-24 03:25 Shortened URLs for commit messages Bruce Momjian <[email protected]>
@ 2017-06-15 17:05 ` Bruce Momjian <[email protected]>
2017-06-16 13:51 ` Re: Shortened URLs for commit messages Bruce Momjian <[email protected]>
0 siblings, 1 reply; 55+ messages in thread
From: Bruce Momjian @ 2017-06-15 17:05 UTC (permalink / raw)
To: pgsql-hackers
On Tue, May 23, 2017 at 11:25:07PM -0400, Bruce Momjian wrote:
> I have written the following sed script to convert regular Postgres
> email message URLs to their shorter form for commit messages:
>
> sed 's;http\(s\?\)://www\.postgresql\.org/message-id/;http\1://postgr.es/m/;gi'
>
> in case this is helpful to anyone.
Oh, here's another one. I use an optional "Discussion:" tag in my
commit messages. This sed script converts a message-id into a proper
URL:
sed '/http/!s;^\(Discussion: *\)\(.*\)$;\1https://postgr.es/m/\2;'
For example:
-Discussion: [email protected]
+Discussion: https://postgr.es/m/[email protected]
Yeah!
--
Bruce Momjian <[email protected]> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 55+ messages in thread
* Re: Shortened URLs for commit messages
2017-05-24 03:25 Shortened URLs for commit messages Bruce Momjian <[email protected]>
2017-06-15 17:05 ` Re: Shortened URLs for commit messages Bruce Momjian <[email protected]>
@ 2017-06-16 13:51 ` Bruce Momjian <[email protected]>
2017-06-21 15:11 ` Re: Shortened URLs for commit messages Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 55+ messages in thread
From: Bruce Momjian @ 2017-06-16 13:51 UTC (permalink / raw)
To: pgsql-hackers
On Thu, Jun 15, 2017 at 01:05:19PM -0400, Bruce Momjian wrote:
> On Tue, May 23, 2017 at 11:25:07PM -0400, Bruce Momjian wrote:
> > I have written the following sed script to convert regular Postgres
> > email message URLs to their shorter form for commit messages:
> >
> > sed 's;http\(s\?\)://www\.postgresql\.org/message-id/;http\1://postgr.es/m/;gi'
> >
> > in case this is helpful to anyone.
>
> Oh, here's another one. I use an optional "Discussion:" tag in my
> commit messages. This sed script converts a message-id into a proper
> URL:
>
> sed '/http/!s;^\(Discussion: *\)\(.*\)$;\1https://postgr.es/m/\2;'
Oh, here is a fixed version that requires an @ sign, which all message
id's have:
sed '/http/!s;^\(Discussion: *\)\(.*@.*\)$;\1https://postgr.es/m/\2;'
--
Bruce Momjian <[email protected]> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 55+ messages in thread
* Re: Shortened URLs for commit messages
2017-05-24 03:25 Shortened URLs for commit messages Bruce Momjian <[email protected]>
2017-06-15 17:05 ` Re: Shortened URLs for commit messages Bruce Momjian <[email protected]>
2017-06-16 13:51 ` Re: Shortened URLs for commit messages Bruce Momjian <[email protected]>
@ 2017-06-21 15:11 ` Alvaro Herrera <[email protected]>
2017-06-21 16:27 ` Re: Shortened URLs for commit messages Bruce Momjian <[email protected]>
0 siblings, 1 reply; 55+ messages in thread
From: Alvaro Herrera @ 2017-06-21 15:11 UTC (permalink / raw)
To: Bruce Momjian <[email protected]>; +Cc: pgsql-hackers
Bruce Momjian wrote:
> Oh, here is a fixed version that requires an @ sign, which all message
> id's have:
>
> sed '/http/!s;^\(Discussion: *\)\(.*@.*\)$;\1https://postgr.es/m/\2;'
So how do you actually use this?
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 55+ messages in thread
* Re: Shortened URLs for commit messages
2017-05-24 03:25 Shortened URLs for commit messages Bruce Momjian <[email protected]>
2017-06-15 17:05 ` Re: Shortened URLs for commit messages Bruce Momjian <[email protected]>
2017-06-16 13:51 ` Re: Shortened URLs for commit messages Bruce Momjian <[email protected]>
2017-06-21 15:11 ` Re: Shortened URLs for commit messages Alvaro Herrera <[email protected]>
@ 2017-06-21 16:27 ` Bruce Momjian <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: Bruce Momjian @ 2017-06-21 16:27 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: pgsql-hackers
On Wed, Jun 21, 2017 at 11:11:57AM -0400, Alvaro Herrera wrote:
> Bruce Momjian wrote:
>
> > Oh, here is a fixed version that requires an @ sign, which all message
> > id's have:
> >
> > sed '/http/!s;^\(Discussion: *\)\(.*@.*\)$;\1https://postgr.es/m/\2;'
>
> So how do you actually use this?
My commit script is here:
https://momjian.us/main/writings/pgsql/src/pgcommit
It basically runs some checks and then creates a temp file with lines
labeled by their purpose. It edits the temp file, then runs the temp
file through a number of filters, and one of those does the URL
shortening via pgurl at:
https://momjian.us/main/writings/pgsql/src/pgurl
It also changes bare message-ids on the 'Discussion' line to shorted
URLs too. It also removes empty labeled lines, and exits if nothing has
been changed in the editor. It then does the commit (potentially to
multiple branches), and then the push.
The script calls many of other custom scripts but you can get an idea
how it works. I am happy to supply more tools as desired.
--
Bruce Momjian <[email protected]> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 55+ messages in thread
* move PartitionBoundInfo creation code
@ 2018-11-01 03:58 Amit Langote <[email protected]>
2018-11-01 04:02 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
0 siblings, 1 reply; 55+ messages in thread
From: Amit Langote @ 2018-11-01 03:58 UTC (permalink / raw)
To: pgsql-hackers
Hi,
Currently, the code that creates a PartitionBoundInfo struct from the
PartitionBoundSpec nodes of constituent partitions read from the catalog
is in RelationBuildPartitionDesc that's in partcache.c. I think that
da6f3e45dd that moved around the partitioning code [1] really missed the
opportunity to move the aforementioned code into partbounds.c. I think
there is quite a bit of logic contained in that code that can aptly be
said to belong in partbounds.c. Also, if factored out into a function of
its own with a proper interface, it could be useful to other callers that
may want to build it for, say, fake partitions [2] which are not real
relations.
Attached find a patch that does such refactoring, along with making some
functions in partbounds.c that are not needed outside static.
Thanks,
Amit
[1] https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=da6f3e45ddb
[2]
https://www.postgresql.org/message-id/009901d3f34c%2471e1bdc0%2455a53940%24%40lab.ntt.co.jp
From b03a6fee7624058197bdeb67c5da04f8e3492505 Mon Sep 17 00:00:00 2001
From: amit <[email protected]>
Date: Thu, 1 Nov 2018 11:32:35 +0900
Subject: [PATCH v1] Move PartitionBoundInfo creation code to partbounds.c
This Factors out the code that creates a PartitionBoundInfo struct
from a list of PartitionBoundSpec nodes into a function called
build_partition_boundinfo and moves it into partbounds.c, where it
aptly seems to belong.
Along with that movement, also make some functions in partbounds.c
static that are not used (or no longer used) outside of that file.
---
src/backend/partitioning/partbounds.c | 534 +++++++++++++++++++++++++++++++++-
src/backend/utils/cache/partcache.c | 526 +--------------------------------
src/include/partitioning/partbounds.h | 14 +-
3 files changed, 547 insertions(+), 527 deletions(-)
diff --git a/src/backend/partitioning/partbounds.c b/src/backend/partitioning/partbounds.c
index c94f73aadc..cd30bb2b25 100644
--- a/src/backend/partitioning/partbounds.c
+++ b/src/backend/partitioning/partbounds.c
@@ -36,6 +36,24 @@
#include "utils/ruleutils.h"
#include "utils/syscache.h"
+static PartitionRangeBound *make_one_partition_rbound(PartitionKey key, int index,
+ List *datums, bool lower);
+static int32 partition_hbound_cmp(int modulus1, int remainder1, int modulus2,
+ int remainder2);
+static int32 partition_rbound_cmp(int partnatts, FmgrInfo *partsupfunc,
+ Oid *partcollation, Datum *datums1,
+ PartitionRangeDatumKind *kind1, bool lower1,
+ PartitionRangeBound *b2);
+static int partition_range_bsearch(int partnatts, FmgrInfo *partsupfunc,
+ Oid *partcollation,
+ PartitionBoundInfo boundinfo,
+ PartitionRangeBound *probe, bool *is_equal);
+
+static int32 qsort_partition_hbound_cmp(const void *a, const void *b);
+static int32 qsort_partition_list_value_cmp(const void *a, const void *b,
+ void *arg);
+static int32 qsort_partition_rbound_cmp(const void *a, const void *b,
+ void *arg);
static int get_partition_bound_num_indexes(PartitionBoundInfo b);
static Expr *make_partition_op_expr(PartitionKey key, int keynum,
uint16 strategy, Expr *arg1, Expr *arg2);
@@ -93,6 +111,469 @@ get_qual_from_partbound(Relation rel, Relation parent,
}
/*
+ * build_partition_boundinfo
+ * Build a PartitionBoundInfo struct from a list of PartitionBoundSpec
+ * nodes
+ *
+ * partoids is the list of OIDs of partitions, which can be NIL if the
+ * caller doesn't have one. If non-NIL, oids should also be non-NULL and
+ * *oids should point to an array of OIDs containing space for
+ * list_length(partoids) elements.
+ */
+PartitionBoundInfo
+build_partition_boundinfo(PartitionKey key, List *boundspecs,
+ List *partoids, Oid **oids)
+{
+ ListCell *cell;
+ int i,
+ nparts;
+ int ndatums = 0;
+ int default_index = -1;
+
+ /* Hash partitioning specific */
+ PartitionHashBound **hbounds = NULL;
+
+ /* List partitioning specific */
+ PartitionListValue **all_values = NULL;
+ int null_index = -1;
+
+ /* Range partitioning specific */
+ PartitionRangeBound **rbounds = NULL;
+
+ PartitionBoundInfo boundinfo;
+ int *mapping;
+ int next_index = 0;
+
+ nparts = list_length(boundspecs);
+ Assert(nparts > 0);
+
+ /* Convert from node to the internal representation */
+ if (key->strategy == PARTITION_STRATEGY_HASH)
+ {
+ ndatums = nparts;
+ hbounds = (PartitionHashBound **)
+ palloc(nparts * sizeof(PartitionHashBound *));
+
+ i = 0;
+ foreach(cell, boundspecs)
+ {
+ PartitionBoundSpec *spec = castNode(PartitionBoundSpec,
+ lfirst(cell));
+
+ if (spec->strategy != PARTITION_STRATEGY_HASH)
+ elog(ERROR, "invalid strategy in partition bound spec");
+
+ hbounds[i] = (PartitionHashBound *)
+ palloc(sizeof(PartitionHashBound));
+
+ hbounds[i]->modulus = spec->modulus;
+ hbounds[i]->remainder = spec->remainder;
+ hbounds[i]->index = i;
+ i++;
+ }
+
+ /* Sort all the bounds in ascending order */
+ qsort(hbounds, nparts, sizeof(PartitionHashBound *),
+ qsort_partition_hbound_cmp);
+ }
+ else if (key->strategy == PARTITION_STRATEGY_LIST)
+ {
+ List *non_null_values = NIL;
+
+ /*
+ * Create a unified list of non-null values across all partitions.
+ */
+ i = 0;
+ null_index = -1;
+ foreach(cell, boundspecs)
+ {
+ PartitionBoundSpec *spec = castNode(PartitionBoundSpec,
+ lfirst(cell));
+ ListCell *c;
+
+ if (spec->strategy != PARTITION_STRATEGY_LIST)
+ elog(ERROR, "invalid strategy in partition bound spec");
+
+ /*
+ * Note the index of the partition bound spec for the default
+ * partition. There's no datum to add to the list of non-null
+ * datums for this partition.
+ */
+ if (spec->is_default)
+ {
+ default_index = i;
+ i++;
+ continue;
+ }
+
+ foreach(c, spec->listdatums)
+ {
+ Const *val = castNode(Const, lfirst(c));
+ PartitionListValue *list_value = NULL;
+
+ if (!val->constisnull)
+ {
+ list_value = (PartitionListValue *)
+ palloc0(sizeof(PartitionListValue));
+ list_value->index = i;
+ list_value->value = val->constvalue;
+ }
+ else
+ {
+ /*
+ * Never put a null into the values array, flag
+ * instead for the code further down below where we
+ * construct the actual relcache struct.
+ */
+ if (null_index != -1)
+ elog(ERROR, "found null more than once");
+ null_index = i;
+ }
+
+ if (list_value)
+ non_null_values = lappend(non_null_values, list_value);
+ }
+
+ i++;
+ }
+
+ ndatums = list_length(non_null_values);
+
+ /*
+ * Collect all list values in one array. Alongside the value, we
+ * also save the index of partition the value comes from.
+ */
+ all_values = (PartitionListValue **) palloc(ndatums *
+ sizeof(PartitionListValue *));
+ i = 0;
+ foreach(cell, non_null_values)
+ {
+ PartitionListValue *src = lfirst(cell);
+
+ all_values[i] = (PartitionListValue *)
+ palloc(sizeof(PartitionListValue));
+ all_values[i]->value = src->value;
+ all_values[i]->index = src->index;
+ i++;
+ }
+
+ qsort_arg(all_values, ndatums, sizeof(PartitionListValue *),
+ qsort_partition_list_value_cmp, (void *) key);
+ }
+ else if (key->strategy == PARTITION_STRATEGY_RANGE)
+ {
+ int k;
+ PartitionRangeBound **all_bounds,
+ *prev;
+
+ all_bounds = (PartitionRangeBound **) palloc0(2 * nparts *
+ sizeof(PartitionRangeBound *));
+
+ /* Create a unified list of range bounds across all the partitions. */
+ i = ndatums = 0;
+ foreach(cell, boundspecs)
+ {
+ PartitionBoundSpec *spec = castNode(PartitionBoundSpec,
+ lfirst(cell));
+ PartitionRangeBound *lower,
+ *upper;
+
+ if (spec->strategy != PARTITION_STRATEGY_RANGE)
+ elog(ERROR, "invalid strategy in partition bound spec");
+
+ /*
+ * Note the index of the partition bound spec for the default
+ * partition. There's no datum to add to the allbounds array
+ * for this partition.
+ */
+ if (spec->is_default)
+ {
+ default_index = i++;
+ continue;
+ }
+
+ lower = make_one_partition_rbound(key, i, spec->lowerdatums,
+ true);
+ upper = make_one_partition_rbound(key, i, spec->upperdatums,
+ false);
+ all_bounds[ndatums++] = lower;
+ all_bounds[ndatums++] = upper;
+ i++;
+ }
+
+ Assert(ndatums == nparts * 2 ||
+ (default_index != -1 && ndatums == (nparts - 1) * 2));
+
+ /* Sort all the bounds in ascending order */
+ qsort_arg(all_bounds, ndatums,
+ sizeof(PartitionRangeBound *),
+ qsort_partition_rbound_cmp,
+ (void *) key);
+
+ /* Save distinct bounds from all_bounds into rbounds. */
+ rbounds = (PartitionRangeBound **)
+ palloc(ndatums * sizeof(PartitionRangeBound *));
+ k = 0;
+ prev = NULL;
+ for (i = 0; i < ndatums; i++)
+ {
+ PartitionRangeBound *cur = all_bounds[i];
+ bool is_distinct = false;
+ int j;
+
+ /* Is the current bound distinct from the previous one? */
+ for (j = 0; j < key->partnatts; j++)
+ {
+ Datum cmpval;
+
+ if (prev == NULL || cur->kind[j] != prev->kind[j])
+ {
+ is_distinct = true;
+ break;
+ }
+
+ /*
+ * If the bounds are both MINVALUE or MAXVALUE, stop now
+ * and treat them as equal, since any values after this
+ * point must be ignored.
+ */
+ if (cur->kind[j] != PARTITION_RANGE_DATUM_VALUE)
+ break;
+
+ cmpval = FunctionCall2Coll(&key->partsupfunc[j],
+ key->partcollation[j],
+ cur->datums[j],
+ prev->datums[j]);
+ if (DatumGetInt32(cmpval) != 0)
+ {
+ is_distinct = true;
+ break;
+ }
+ }
+
+ /*
+ * Only if the bound is distinct save it into a temporary
+ * array i.e. rbounds which is later copied into boundinfo
+ * datums array.
+ */
+ if (is_distinct)
+ rbounds[k++] = all_bounds[i];
+
+ prev = cur;
+ }
+
+ /* Update ndatums to hold the count of distinct datums. */
+ ndatums = k;
+ }
+ else
+ elog(ERROR, "unexpected partition strategy: %d",
+ (int) key->strategy);
+
+ boundinfo = (PartitionBoundInfoData *)
+ palloc0(sizeof(PartitionBoundInfoData));
+ boundinfo->strategy = key->strategy;
+ boundinfo->default_index = -1;
+ boundinfo->ndatums = ndatums;
+ boundinfo->null_index = -1;
+ boundinfo->datums = (Datum **) palloc0(ndatums * sizeof(Datum *));
+
+ /* Initialize mapping array with invalid values */
+ mapping = (int *) palloc(sizeof(int) * nparts);
+ for (i = 0; i < nparts; i++)
+ mapping[i] = -1;
+
+ switch (key->strategy)
+ {
+ case PARTITION_STRATEGY_HASH:
+ {
+ /* Moduli are stored in ascending order */
+ int greatest_modulus = hbounds[ndatums - 1]->modulus;
+
+ boundinfo->indexes = (int *) palloc(greatest_modulus *
+ sizeof(int));
+
+ for (i = 0; i < greatest_modulus; i++)
+ boundinfo->indexes[i] = -1;
+
+ for (i = 0; i < nparts; i++)
+ {
+ int modulus = hbounds[i]->modulus;
+ int remainder = hbounds[i]->remainder;
+
+ boundinfo->datums[i] = (Datum *) palloc(2 *
+ sizeof(Datum));
+ boundinfo->datums[i][0] = Int32GetDatum(modulus);
+ boundinfo->datums[i][1] = Int32GetDatum(remainder);
+
+ while (remainder < greatest_modulus)
+ {
+ /* overlap? */
+ Assert(boundinfo->indexes[remainder] == -1);
+ boundinfo->indexes[remainder] = i;
+ remainder += modulus;
+ }
+
+ mapping[hbounds[i]->index] = i;
+ pfree(hbounds[i]);
+ }
+ pfree(hbounds);
+ break;
+ }
+
+ case PARTITION_STRATEGY_LIST:
+ {
+ boundinfo->indexes = (int *) palloc(ndatums * sizeof(int));
+
+ /*
+ * Copy values. Indexes of individual values are mapped
+ * to canonical values so that they match for any two list
+ * partitioned tables with same number of partitions and
+ * same lists per partition. One way to canonicalize is
+ * to assign the index in all_values[] of the smallest
+ * value of each partition, as the index of all of the
+ * partition's values.
+ */
+ for (i = 0; i < ndatums; i++)
+ {
+ boundinfo->datums[i] = (Datum *) palloc(sizeof(Datum));
+ boundinfo->datums[i][0] = datumCopy(all_values[i]->value,
+ key->parttypbyval[0],
+ key->parttyplen[0]);
+
+ /* If the old index has no mapping, assign one */
+ if (mapping[all_values[i]->index] == -1)
+ mapping[all_values[i]->index] = next_index++;
+
+ boundinfo->indexes[i] = mapping[all_values[i]->index];
+ }
+
+ /*
+ * If null-accepting partition has no mapped index yet,
+ * assign one. This could happen if such partition
+ * accepts only null and hence not covered in the above
+ * loop which only handled non-null values.
+ */
+ if (null_index != -1)
+ {
+ Assert(null_index >= 0);
+ if (mapping[null_index] == -1)
+ mapping[null_index] = next_index++;
+ boundinfo->null_index = mapping[null_index];
+ }
+
+ /* Assign mapped index for the default partition. */
+ if (default_index != -1)
+ {
+ /*
+ * The default partition accepts any value not
+ * specified in the lists of other partitions, hence
+ * it should not get mapped index while assigning
+ * those for non-null datums.
+ */
+ Assert(default_index >= 0 &&
+ mapping[default_index] == -1);
+ mapping[default_index] = next_index++;
+ boundinfo->default_index = mapping[default_index];
+ }
+
+ /* All partition must now have a valid mapping */
+ Assert(next_index == nparts);
+ break;
+ }
+
+ case PARTITION_STRATEGY_RANGE:
+ {
+ boundinfo->kind = (PartitionRangeDatumKind **)
+ palloc(ndatums *
+ sizeof(PartitionRangeDatumKind *));
+ boundinfo->indexes = (int *) palloc((ndatums + 1) *
+ sizeof(int));
+
+ for (i = 0; i < ndatums; i++)
+ {
+ int j;
+
+ boundinfo->datums[i] = (Datum *) palloc(key->partnatts *
+ sizeof(Datum));
+ boundinfo->kind[i] = (PartitionRangeDatumKind *)
+ palloc(key->partnatts *
+ sizeof(PartitionRangeDatumKind));
+ for (j = 0; j < key->partnatts; j++)
+ {
+ if (rbounds[i]->kind[j] == PARTITION_RANGE_DATUM_VALUE)
+ boundinfo->datums[i][j] =
+ datumCopy(rbounds[i]->datums[j],
+ key->parttypbyval[j],
+ key->parttyplen[j]);
+ boundinfo->kind[i][j] = rbounds[i]->kind[j];
+ }
+
+ /*
+ * There is no mapping for invalid indexes.
+ *
+ * Any lower bounds in the rbounds array have invalid
+ * indexes assigned, because the values between the
+ * previous bound (if there is one) and this (lower)
+ * bound are not part of the range of any existing
+ * partition.
+ */
+ if (rbounds[i]->lower)
+ boundinfo->indexes[i] = -1;
+ else
+ {
+ int orig_index = rbounds[i]->index;
+
+ /* If the old index has no mapping, assign one */
+ if (mapping[orig_index] == -1)
+ mapping[orig_index] = next_index++;
+
+ boundinfo->indexes[i] = mapping[orig_index];
+ }
+ }
+
+ /* Assign mapped index for the default partition. */
+ if (default_index != -1)
+ {
+ Assert(default_index >= 0 && mapping[default_index] == -1);
+ mapping[default_index] = next_index++;
+ boundinfo->default_index = mapping[default_index];
+ }
+ boundinfo->indexes[i] = -1;
+ break;
+ }
+
+ default:
+ elog(ERROR, "unexpected partition strategy: %d",
+ (int) key->strategy);
+ }
+
+ /*
+ * Now assign OIDs from the original array into mapped indexes of the
+ * result array. Order of OIDs in the former is defined by the
+ * catalog scan that retrieved them, whereas that in the latter is
+ * defined by canonicalized representation of the partition bounds.
+ */
+ if (partoids != NIL)
+ {
+ Oid *oids_orig_order = (Oid *) palloc(sizeof(Oid) * nparts);
+
+ Assert(oids != NULL);
+
+ i = 0;
+ foreach(cell, partoids)
+ oids_orig_order[i++] = lfirst_oid(cell);
+
+ /* Add the OIDs to result array in the bound order. */
+ for (i = 0; i < nparts; i++)
+ (*oids)[mapping[i]] = oids_orig_order[i];
+ pfree(oids_orig_order);
+ }
+ pfree(mapping);
+
+ return boundinfo;
+}
+
+/*
* Are two partition bound collections logically equal?
*
* Used in the keep logic of relcache.c (ie, in RelationClearRelation()).
@@ -763,7 +1244,7 @@ get_hash_partition_greatest_modulus(PartitionBoundInfo bound)
* and a flag telling whether the bound is lower or not. Made into a function
* because there are multiple sites that want to use this facility.
*/
-PartitionRangeBound *
+static PartitionRangeBound *
make_one_partition_rbound(PartitionKey key, int index, List *datums, bool lower)
{
PartitionRangeBound *bound;
@@ -819,7 +1300,7 @@ make_one_partition_rbound(PartitionKey key, int index, List *datums, bool lower)
* structure, which only stores the upper bound of a common boundary between
* two contiguous partitions.
*/
-int32
+static int32
partition_rbound_cmp(int partnatts, FmgrInfo *partsupfunc,
Oid *partcollation,
Datum *datums1, PartitionRangeDatumKind *kind1,
@@ -914,7 +1395,7 @@ partition_rbound_datum_cmp(FmgrInfo *partsupfunc, Oid *partcollation,
*
* Compares modulus first, then remainder if modulus is equal.
*/
-int32
+static int32
partition_hbound_cmp(int modulus1, int remainder1, int modulus2, int remainder2)
{
if (modulus1 < modulus2)
@@ -977,7 +1458,7 @@ partition_list_bsearch(FmgrInfo *partsupfunc, Oid *partcollation,
* *is_equal is set to true if the range bound at the returned index is equal
* to the input range bound
*/
-int
+static int
partition_range_bsearch(int partnatts, FmgrInfo *partsupfunc,
Oid *partcollation,
PartitionBoundInfo boundinfo,
@@ -1102,6 +1583,51 @@ partition_hash_bsearch(PartitionBoundInfo boundinfo,
}
/*
+ * qsort_partition_hbound_cmp
+ *
+ * We sort hash bounds by modulus, then by remainder.
+ */
+static int32
+qsort_partition_hbound_cmp(const void *a, const void *b)
+{
+ PartitionHashBound *h1 = (*(PartitionHashBound *const *) a);
+ PartitionHashBound *h2 = (*(PartitionHashBound *const *) b);
+
+ return partition_hbound_cmp(h1->modulus, h1->remainder,
+ h2->modulus, h2->remainder);
+}
+
+/*
+ * qsort_partition_list_value_cmp
+ *
+ * Compare two list partition bound datums
+ */
+static int32
+qsort_partition_list_value_cmp(const void *a, const void *b, void *arg)
+{
+ Datum val1 = (*(const PartitionListValue **) a)->value,
+ val2 = (*(const PartitionListValue **) b)->value;
+ PartitionKey key = (PartitionKey) arg;
+
+ return DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[0],
+ key->partcollation[0],
+ val1, val2));
+}
+
+/* Used when sorting range bounds across all range partitions */
+static int32
+qsort_partition_rbound_cmp(const void *a, const void *b, void *arg)
+{
+ PartitionRangeBound *b1 = (*(PartitionRangeBound *const *) a);
+ PartitionRangeBound *b2 = (*(PartitionRangeBound *const *) b);
+ PartitionKey key = (PartitionKey) arg;
+
+ return partition_rbound_cmp(key->partnatts, key->partsupfunc,
+ key->partcollation, b1->datums, b1->kind,
+ b1->lower, b2);
+}
+
+/*
* get_partition_bound_num_indexes
*
* Returns the number of the entries in the partition bound indexes array.
diff --git a/src/backend/utils/cache/partcache.c b/src/backend/utils/cache/partcache.c
index 5757301d05..954187dbd7 100644
--- a/src/backend/utils/cache/partcache.c
+++ b/src/backend/utils/cache/partcache.c
@@ -38,12 +38,6 @@
static List *generate_partition_qual(Relation rel);
-static int32 qsort_partition_hbound_cmp(const void *a, const void *b);
-static int32 qsort_partition_list_value_cmp(const void *a, const void *b,
- void *arg);
-static int32 qsort_partition_rbound_cmp(const void *a, const void *b,
- void *arg);
-
/*
* RelationBuildPartitionKey
@@ -260,36 +254,17 @@ RelationBuildPartitionKey(Relation relation)
void
RelationBuildPartitionDesc(Relation rel)
{
- List *inhoids,
- *partoids;
- Oid *oids = NULL;
- List *boundspecs = NIL;
- ListCell *cell;
- int i,
- nparts;
- PartitionKey key = RelationGetPartitionKey(rel);
- PartitionDesc result;
MemoryContext oldcxt;
-
- int ndatums = 0;
- int default_index = -1;
-
- /* Hash partitioning specific */
- PartitionHashBound **hbounds = NULL;
-
- /* List partitioning specific */
- PartitionListValue **all_values = NULL;
- int null_index = -1;
-
- /* Range partitioning specific */
- PartitionRangeBound **rbounds = NULL;
+ PartitionDesc partdesc;
+ List *inhoids,
+ *boundspecs;
+ ListCell *cell;
/* Get partition oids from pg_inherits */
inhoids = find_inheritance_children(RelationGetRelid(rel), NoLock);
/* Collect bound spec nodes in a list */
- i = 0;
- partoids = NIL;
+ boundspecs = NIL;
foreach(cell, inhoids)
{
Oid inhrelid = lfirst_oid(cell);
@@ -325,245 +300,10 @@ RelationBuildPartitionDesc(Relation rel)
}
boundspecs = lappend(boundspecs, boundspec);
- partoids = lappend_oid(partoids, inhrelid);
ReleaseSysCache(tuple);
}
- nparts = list_length(partoids);
-
- if (nparts > 0)
- {
- oids = (Oid *) palloc(nparts * sizeof(Oid));
- i = 0;
- foreach(cell, partoids)
- oids[i++] = lfirst_oid(cell);
-
- /* Convert from node to the internal representation */
- if (key->strategy == PARTITION_STRATEGY_HASH)
- {
- ndatums = nparts;
- hbounds = (PartitionHashBound **)
- palloc(nparts * sizeof(PartitionHashBound *));
-
- i = 0;
- foreach(cell, boundspecs)
- {
- PartitionBoundSpec *spec = castNode(PartitionBoundSpec,
- lfirst(cell));
-
- if (spec->strategy != PARTITION_STRATEGY_HASH)
- elog(ERROR, "invalid strategy in partition bound spec");
-
- hbounds[i] = (PartitionHashBound *)
- palloc(sizeof(PartitionHashBound));
-
- hbounds[i]->modulus = spec->modulus;
- hbounds[i]->remainder = spec->remainder;
- hbounds[i]->index = i;
- i++;
- }
-
- /* Sort all the bounds in ascending order */
- qsort(hbounds, nparts, sizeof(PartitionHashBound *),
- qsort_partition_hbound_cmp);
- }
- else if (key->strategy == PARTITION_STRATEGY_LIST)
- {
- List *non_null_values = NIL;
-
- /*
- * Create a unified list of non-null values across all partitions.
- */
- i = 0;
- null_index = -1;
- foreach(cell, boundspecs)
- {
- PartitionBoundSpec *spec = castNode(PartitionBoundSpec,
- lfirst(cell));
- ListCell *c;
-
- if (spec->strategy != PARTITION_STRATEGY_LIST)
- elog(ERROR, "invalid strategy in partition bound spec");
-
- /*
- * Note the index of the partition bound spec for the default
- * partition. There's no datum to add to the list of non-null
- * datums for this partition.
- */
- if (spec->is_default)
- {
- default_index = i;
- i++;
- continue;
- }
-
- foreach(c, spec->listdatums)
- {
- Const *val = castNode(Const, lfirst(c));
- PartitionListValue *list_value = NULL;
-
- if (!val->constisnull)
- {
- list_value = (PartitionListValue *)
- palloc0(sizeof(PartitionListValue));
- list_value->index = i;
- list_value->value = val->constvalue;
- }
- else
- {
- /*
- * Never put a null into the values array, flag
- * instead for the code further down below where we
- * construct the actual relcache struct.
- */
- if (null_index != -1)
- elog(ERROR, "found null more than once");
- null_index = i;
- }
-
- if (list_value)
- non_null_values = lappend(non_null_values,
- list_value);
- }
-
- i++;
- }
-
- ndatums = list_length(non_null_values);
-
- /*
- * Collect all list values in one array. Alongside the value, we
- * also save the index of partition the value comes from.
- */
- all_values = (PartitionListValue **) palloc(ndatums *
- sizeof(PartitionListValue *));
- i = 0;
- foreach(cell, non_null_values)
- {
- PartitionListValue *src = lfirst(cell);
-
- all_values[i] = (PartitionListValue *)
- palloc(sizeof(PartitionListValue));
- all_values[i]->value = src->value;
- all_values[i]->index = src->index;
- i++;
- }
-
- qsort_arg(all_values, ndatums, sizeof(PartitionListValue *),
- qsort_partition_list_value_cmp, (void *) key);
- }
- else if (key->strategy == PARTITION_STRATEGY_RANGE)
- {
- int k;
- PartitionRangeBound **all_bounds,
- *prev;
-
- all_bounds = (PartitionRangeBound **) palloc0(2 * nparts *
- sizeof(PartitionRangeBound *));
-
- /*
- * Create a unified list of range bounds across all the
- * partitions.
- */
- i = ndatums = 0;
- foreach(cell, boundspecs)
- {
- PartitionBoundSpec *spec = castNode(PartitionBoundSpec,
- lfirst(cell));
- PartitionRangeBound *lower,
- *upper;
-
- if (spec->strategy != PARTITION_STRATEGY_RANGE)
- elog(ERROR, "invalid strategy in partition bound spec");
-
- /*
- * Note the index of the partition bound spec for the default
- * partition. There's no datum to add to the allbounds array
- * for this partition.
- */
- if (spec->is_default)
- {
- default_index = i++;
- continue;
- }
-
- lower = make_one_partition_rbound(key, i, spec->lowerdatums,
- true);
- upper = make_one_partition_rbound(key, i, spec->upperdatums,
- false);
- all_bounds[ndatums++] = lower;
- all_bounds[ndatums++] = upper;
- i++;
- }
-
- Assert(ndatums == nparts * 2 ||
- (default_index != -1 && ndatums == (nparts - 1) * 2));
-
- /* Sort all the bounds in ascending order */
- qsort_arg(all_bounds, ndatums,
- sizeof(PartitionRangeBound *),
- qsort_partition_rbound_cmp,
- (void *) key);
-
- /* Save distinct bounds from all_bounds into rbounds. */
- rbounds = (PartitionRangeBound **)
- palloc(ndatums * sizeof(PartitionRangeBound *));
- k = 0;
- prev = NULL;
- for (i = 0; i < ndatums; i++)
- {
- PartitionRangeBound *cur = all_bounds[i];
- bool is_distinct = false;
- int j;
-
- /* Is the current bound distinct from the previous one? */
- for (j = 0; j < key->partnatts; j++)
- {
- Datum cmpval;
-
- if (prev == NULL || cur->kind[j] != prev->kind[j])
- {
- is_distinct = true;
- break;
- }
-
- /*
- * If the bounds are both MINVALUE or MAXVALUE, stop now
- * and treat them as equal, since any values after this
- * point must be ignored.
- */
- if (cur->kind[j] != PARTITION_RANGE_DATUM_VALUE)
- break;
-
- cmpval = FunctionCall2Coll(&key->partsupfunc[j],
- key->partcollation[j],
- cur->datums[j],
- prev->datums[j]);
- if (DatumGetInt32(cmpval) != 0)
- {
- is_distinct = true;
- break;
- }
- }
-
- /*
- * Only if the bound is distinct save it into a temporary
- * array i.e. rbounds which is later copied into boundinfo
- * datums array.
- */
- if (is_distinct)
- rbounds[k++] = all_bounds[i];
-
- prev = cur;
- }
-
- /* Update ndatums to hold the count of distinct datums. */
- ndatums = k;
- }
- else
- elog(ERROR, "unexpected partition strategy: %d",
- (int) key->strategy);
- }
+ Assert(list_length(inhoids) == list_length(boundspecs));
/* Now build the actual relcache partition descriptor */
rel->rd_pdcxt = AllocSetContextCreate(CacheMemoryContext,
@@ -572,210 +312,19 @@ RelationBuildPartitionDesc(Relation rel)
MemoryContextCopyAndSetIdentifier(rel->rd_pdcxt, RelationGetRelationName(rel));
oldcxt = MemoryContextSwitchTo(rel->rd_pdcxt);
-
- result = (PartitionDescData *) palloc0(sizeof(PartitionDescData));
- result->nparts = nparts;
- if (nparts > 0)
+ partdesc = (PartitionDescData *) palloc0(sizeof(PartitionDescData));
+ partdesc->nparts = list_length(inhoids);
+ if (partdesc->nparts > 0)
{
- PartitionBoundInfo boundinfo;
- int *mapping;
- int next_index = 0;
-
- result->oids = (Oid *) palloc0(nparts * sizeof(Oid));
-
- boundinfo = (PartitionBoundInfoData *)
- palloc0(sizeof(PartitionBoundInfoData));
- boundinfo->strategy = key->strategy;
- boundinfo->default_index = -1;
- boundinfo->ndatums = ndatums;
- boundinfo->null_index = -1;
- boundinfo->datums = (Datum **) palloc0(ndatums * sizeof(Datum *));
-
- /* Initialize mapping array with invalid values */
- mapping = (int *) palloc(sizeof(int) * nparts);
- for (i = 0; i < nparts; i++)
- mapping[i] = -1;
-
- switch (key->strategy)
- {
- case PARTITION_STRATEGY_HASH:
- {
- /* Moduli are stored in ascending order */
- int greatest_modulus = hbounds[ndatums - 1]->modulus;
-
- boundinfo->indexes = (int *) palloc(greatest_modulus *
- sizeof(int));
-
- for (i = 0; i < greatest_modulus; i++)
- boundinfo->indexes[i] = -1;
-
- for (i = 0; i < nparts; i++)
- {
- int modulus = hbounds[i]->modulus;
- int remainder = hbounds[i]->remainder;
-
- boundinfo->datums[i] = (Datum *) palloc(2 *
- sizeof(Datum));
- boundinfo->datums[i][0] = Int32GetDatum(modulus);
- boundinfo->datums[i][1] = Int32GetDatum(remainder);
-
- while (remainder < greatest_modulus)
- {
- /* overlap? */
- Assert(boundinfo->indexes[remainder] == -1);
- boundinfo->indexes[remainder] = i;
- remainder += modulus;
- }
-
- mapping[hbounds[i]->index] = i;
- pfree(hbounds[i]);
- }
- pfree(hbounds);
- break;
- }
-
- case PARTITION_STRATEGY_LIST:
- {
- boundinfo->indexes = (int *) palloc(ndatums * sizeof(int));
-
- /*
- * Copy values. Indexes of individual values are mapped
- * to canonical values so that they match for any two list
- * partitioned tables with same number of partitions and
- * same lists per partition. One way to canonicalize is
- * to assign the index in all_values[] of the smallest
- * value of each partition, as the index of all of the
- * partition's values.
- */
- for (i = 0; i < ndatums; i++)
- {
- boundinfo->datums[i] = (Datum *) palloc(sizeof(Datum));
- boundinfo->datums[i][0] = datumCopy(all_values[i]->value,
- key->parttypbyval[0],
- key->parttyplen[0]);
-
- /* If the old index has no mapping, assign one */
- if (mapping[all_values[i]->index] == -1)
- mapping[all_values[i]->index] = next_index++;
-
- boundinfo->indexes[i] = mapping[all_values[i]->index];
- }
-
- /*
- * If null-accepting partition has no mapped index yet,
- * assign one. This could happen if such partition
- * accepts only null and hence not covered in the above
- * loop which only handled non-null values.
- */
- if (null_index != -1)
- {
- Assert(null_index >= 0);
- if (mapping[null_index] == -1)
- mapping[null_index] = next_index++;
- boundinfo->null_index = mapping[null_index];
- }
-
- /* Assign mapped index for the default partition. */
- if (default_index != -1)
- {
- /*
- * The default partition accepts any value not
- * specified in the lists of other partitions, hence
- * it should not get mapped index while assigning
- * those for non-null datums.
- */
- Assert(default_index >= 0 &&
- mapping[default_index] == -1);
- mapping[default_index] = next_index++;
- boundinfo->default_index = mapping[default_index];
- }
-
- /* All partition must now have a valid mapping */
- Assert(next_index == nparts);
- break;
- }
-
- case PARTITION_STRATEGY_RANGE:
- {
- boundinfo->kind = (PartitionRangeDatumKind **)
- palloc(ndatums *
- sizeof(PartitionRangeDatumKind *));
- boundinfo->indexes = (int *) palloc((ndatums + 1) *
- sizeof(int));
-
- for (i = 0; i < ndatums; i++)
- {
- int j;
-
- boundinfo->datums[i] = (Datum *) palloc(key->partnatts *
- sizeof(Datum));
- boundinfo->kind[i] = (PartitionRangeDatumKind *)
- palloc(key->partnatts *
- sizeof(PartitionRangeDatumKind));
- for (j = 0; j < key->partnatts; j++)
- {
- if (rbounds[i]->kind[j] == PARTITION_RANGE_DATUM_VALUE)
- boundinfo->datums[i][j] =
- datumCopy(rbounds[i]->datums[j],
- key->parttypbyval[j],
- key->parttyplen[j]);
- boundinfo->kind[i][j] = rbounds[i]->kind[j];
- }
-
- /*
- * There is no mapping for invalid indexes.
- *
- * Any lower bounds in the rbounds array have invalid
- * indexes assigned, because the values between the
- * previous bound (if there is one) and this (lower)
- * bound are not part of the range of any existing
- * partition.
- */
- if (rbounds[i]->lower)
- boundinfo->indexes[i] = -1;
- else
- {
- int orig_index = rbounds[i]->index;
-
- /* If the old index has no mapping, assign one */
- if (mapping[orig_index] == -1)
- mapping[orig_index] = next_index++;
-
- boundinfo->indexes[i] = mapping[orig_index];
- }
- }
-
- /* Assign mapped index for the default partition. */
- if (default_index != -1)
- {
- Assert(default_index >= 0 && mapping[default_index] == -1);
- mapping[default_index] = next_index++;
- boundinfo->default_index = mapping[default_index];
- }
- boundinfo->indexes[i] = -1;
- break;
- }
-
- default:
- elog(ERROR, "unexpected partition strategy: %d",
- (int) key->strategy);
- }
-
- result->boundinfo = boundinfo;
-
- /*
- * Now assign OIDs from the original array into mapped indexes of the
- * result array. Order of OIDs in the former is defined by the
- * catalog scan that retrieved them, whereas that in the latter is
- * defined by canonicalized representation of the partition bounds.
- */
- for (i = 0; i < nparts; i++)
- result->oids[mapping[i]] = oids[i];
- pfree(mapping);
+ partdesc->oids = (Oid *) palloc(partdesc->nparts * sizeof(Oid));
+ partdesc->boundinfo =
+ build_partition_boundinfo(RelationGetPartitionKey(rel),
+ boundspecs,
+ inhoids, &partdesc->oids);
}
MemoryContextSwitchTo(oldcxt);
- rel->rd_partdesc = result;
+ rel->rd_partdesc = partdesc;
}
/*
@@ -917,48 +466,3 @@ generate_partition_qual(Relation rel)
return result;
}
-
-/*
- * qsort_partition_hbound_cmp
- *
- * We sort hash bounds by modulus, then by remainder.
- */
-static int32
-qsort_partition_hbound_cmp(const void *a, const void *b)
-{
- PartitionHashBound *h1 = (*(PartitionHashBound *const *) a);
- PartitionHashBound *h2 = (*(PartitionHashBound *const *) b);
-
- return partition_hbound_cmp(h1->modulus, h1->remainder,
- h2->modulus, h2->remainder);
-}
-
-/*
- * qsort_partition_list_value_cmp
- *
- * Compare two list partition bound datums
- */
-static int32
-qsort_partition_list_value_cmp(const void *a, const void *b, void *arg)
-{
- Datum val1 = (*(const PartitionListValue **) a)->value,
- val2 = (*(const PartitionListValue **) b)->value;
- PartitionKey key = (PartitionKey) arg;
-
- return DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[0],
- key->partcollation[0],
- val1, val2));
-}
-
-/* Used when sorting range bounds across all range partitions */
-static int32
-qsort_partition_rbound_cmp(const void *a, const void *b, void *arg)
-{
- PartitionRangeBound *b1 = (*(PartitionRangeBound *const *) a);
- PartitionRangeBound *b2 = (*(PartitionRangeBound *const *) b);
- PartitionKey key = (PartitionKey) arg;
-
- return partition_rbound_cmp(key->partnatts, key->partsupfunc,
- key->partcollation, b1->datums, b1->kind,
- b1->lower, b2);
-}
diff --git a/src/include/partitioning/partbounds.h b/src/include/partitioning/partbounds.h
index c7535e32fc..aebd222ccd 100644
--- a/src/include/partitioning/partbounds.h
+++ b/src/include/partitioning/partbounds.h
@@ -109,6 +109,8 @@ extern uint64 compute_partition_hash_value(int partnatts, FmgrInfo *partsupfunc,
Datum *values, bool *isnull);
extern List *get_qual_from_partbound(Relation rel, Relation parent,
PartitionBoundSpec *spec);
+extern PartitionBoundInfo build_partition_boundinfo(PartitionKey key,
+ List *boundspecs, List *partoids, Oid **oids);
extern bool partition_bounds_equal(int partnatts, int16 *parttyplen,
bool *parttypbyval, PartitionBoundInfo b1,
PartitionBoundInfo b2);
@@ -120,14 +122,6 @@ extern void check_default_partition_contents(Relation parent,
Relation defaultRel,
PartitionBoundSpec *new_spec);
-extern PartitionRangeBound *make_one_partition_rbound(PartitionKey key, int index,
- List *datums, bool lower);
-extern int32 partition_hbound_cmp(int modulus1, int remainder1, int modulus2,
- int remainder2);
-extern int32 partition_rbound_cmp(int partnatts, FmgrInfo *partsupfunc,
- Oid *partcollation, Datum *datums1,
- PartitionRangeDatumKind *kind1, bool lower1,
- PartitionRangeBound *b2);
extern int32 partition_rbound_datum_cmp(FmgrInfo *partsupfunc,
Oid *partcollation,
Datum *rb_datums, PartitionRangeDatumKind *rb_kind,
@@ -136,10 +130,6 @@ extern int partition_list_bsearch(FmgrInfo *partsupfunc,
Oid *partcollation,
PartitionBoundInfo boundinfo,
Datum value, bool *is_equal);
-extern int partition_range_bsearch(int partnatts, FmgrInfo *partsupfunc,
- Oid *partcollation,
- PartitionBoundInfo boundinfo,
- PartitionRangeBound *probe, bool *is_equal);
extern int partition_range_datum_bsearch(FmgrInfo *partsupfunc,
Oid *partcollation,
PartitionBoundInfo boundinfo,
--
2.11.0
Attachments:
[text/plain] v1-0001-Move-PartitionBoundInfo-creation-code-to-partboun.patch (36.1K, ../../[email protected]/2-v1-0001-Move-PartitionBoundInfo-creation-code-to-partboun.patch)
download | inline diff:
From b03a6fee7624058197bdeb67c5da04f8e3492505 Mon Sep 17 00:00:00 2001
From: amit <[email protected]>
Date: Thu, 1 Nov 2018 11:32:35 +0900
Subject: [PATCH v1] Move PartitionBoundInfo creation code to partbounds.c
This Factors out the code that creates a PartitionBoundInfo struct
from a list of PartitionBoundSpec nodes into a function called
build_partition_boundinfo and moves it into partbounds.c, where it
aptly seems to belong.
Along with that movement, also make some functions in partbounds.c
static that are not used (or no longer used) outside of that file.
---
src/backend/partitioning/partbounds.c | 534 +++++++++++++++++++++++++++++++++-
src/backend/utils/cache/partcache.c | 526 +--------------------------------
src/include/partitioning/partbounds.h | 14 +-
3 files changed, 547 insertions(+), 527 deletions(-)
diff --git a/src/backend/partitioning/partbounds.c b/src/backend/partitioning/partbounds.c
index c94f73aadc..cd30bb2b25 100644
--- a/src/backend/partitioning/partbounds.c
+++ b/src/backend/partitioning/partbounds.c
@@ -36,6 +36,24 @@
#include "utils/ruleutils.h"
#include "utils/syscache.h"
+static PartitionRangeBound *make_one_partition_rbound(PartitionKey key, int index,
+ List *datums, bool lower);
+static int32 partition_hbound_cmp(int modulus1, int remainder1, int modulus2,
+ int remainder2);
+static int32 partition_rbound_cmp(int partnatts, FmgrInfo *partsupfunc,
+ Oid *partcollation, Datum *datums1,
+ PartitionRangeDatumKind *kind1, bool lower1,
+ PartitionRangeBound *b2);
+static int partition_range_bsearch(int partnatts, FmgrInfo *partsupfunc,
+ Oid *partcollation,
+ PartitionBoundInfo boundinfo,
+ PartitionRangeBound *probe, bool *is_equal);
+
+static int32 qsort_partition_hbound_cmp(const void *a, const void *b);
+static int32 qsort_partition_list_value_cmp(const void *a, const void *b,
+ void *arg);
+static int32 qsort_partition_rbound_cmp(const void *a, const void *b,
+ void *arg);
static int get_partition_bound_num_indexes(PartitionBoundInfo b);
static Expr *make_partition_op_expr(PartitionKey key, int keynum,
uint16 strategy, Expr *arg1, Expr *arg2);
@@ -93,6 +111,469 @@ get_qual_from_partbound(Relation rel, Relation parent,
}
/*
+ * build_partition_boundinfo
+ * Build a PartitionBoundInfo struct from a list of PartitionBoundSpec
+ * nodes
+ *
+ * partoids is the list of OIDs of partitions, which can be NIL if the
+ * caller doesn't have one. If non-NIL, oids should also be non-NULL and
+ * *oids should point to an array of OIDs containing space for
+ * list_length(partoids) elements.
+ */
+PartitionBoundInfo
+build_partition_boundinfo(PartitionKey key, List *boundspecs,
+ List *partoids, Oid **oids)
+{
+ ListCell *cell;
+ int i,
+ nparts;
+ int ndatums = 0;
+ int default_index = -1;
+
+ /* Hash partitioning specific */
+ PartitionHashBound **hbounds = NULL;
+
+ /* List partitioning specific */
+ PartitionListValue **all_values = NULL;
+ int null_index = -1;
+
+ /* Range partitioning specific */
+ PartitionRangeBound **rbounds = NULL;
+
+ PartitionBoundInfo boundinfo;
+ int *mapping;
+ int next_index = 0;
+
+ nparts = list_length(boundspecs);
+ Assert(nparts > 0);
+
+ /* Convert from node to the internal representation */
+ if (key->strategy == PARTITION_STRATEGY_HASH)
+ {
+ ndatums = nparts;
+ hbounds = (PartitionHashBound **)
+ palloc(nparts * sizeof(PartitionHashBound *));
+
+ i = 0;
+ foreach(cell, boundspecs)
+ {
+ PartitionBoundSpec *spec = castNode(PartitionBoundSpec,
+ lfirst(cell));
+
+ if (spec->strategy != PARTITION_STRATEGY_HASH)
+ elog(ERROR, "invalid strategy in partition bound spec");
+
+ hbounds[i] = (PartitionHashBound *)
+ palloc(sizeof(PartitionHashBound));
+
+ hbounds[i]->modulus = spec->modulus;
+ hbounds[i]->remainder = spec->remainder;
+ hbounds[i]->index = i;
+ i++;
+ }
+
+ /* Sort all the bounds in ascending order */
+ qsort(hbounds, nparts, sizeof(PartitionHashBound *),
+ qsort_partition_hbound_cmp);
+ }
+ else if (key->strategy == PARTITION_STRATEGY_LIST)
+ {
+ List *non_null_values = NIL;
+
+ /*
+ * Create a unified list of non-null values across all partitions.
+ */
+ i = 0;
+ null_index = -1;
+ foreach(cell, boundspecs)
+ {
+ PartitionBoundSpec *spec = castNode(PartitionBoundSpec,
+ lfirst(cell));
+ ListCell *c;
+
+ if (spec->strategy != PARTITION_STRATEGY_LIST)
+ elog(ERROR, "invalid strategy in partition bound spec");
+
+ /*
+ * Note the index of the partition bound spec for the default
+ * partition. There's no datum to add to the list of non-null
+ * datums for this partition.
+ */
+ if (spec->is_default)
+ {
+ default_index = i;
+ i++;
+ continue;
+ }
+
+ foreach(c, spec->listdatums)
+ {
+ Const *val = castNode(Const, lfirst(c));
+ PartitionListValue *list_value = NULL;
+
+ if (!val->constisnull)
+ {
+ list_value = (PartitionListValue *)
+ palloc0(sizeof(PartitionListValue));
+ list_value->index = i;
+ list_value->value = val->constvalue;
+ }
+ else
+ {
+ /*
+ * Never put a null into the values array, flag
+ * instead for the code further down below where we
+ * construct the actual relcache struct.
+ */
+ if (null_index != -1)
+ elog(ERROR, "found null more than once");
+ null_index = i;
+ }
+
+ if (list_value)
+ non_null_values = lappend(non_null_values, list_value);
+ }
+
+ i++;
+ }
+
+ ndatums = list_length(non_null_values);
+
+ /*
+ * Collect all list values in one array. Alongside the value, we
+ * also save the index of partition the value comes from.
+ */
+ all_values = (PartitionListValue **) palloc(ndatums *
+ sizeof(PartitionListValue *));
+ i = 0;
+ foreach(cell, non_null_values)
+ {
+ PartitionListValue *src = lfirst(cell);
+
+ all_values[i] = (PartitionListValue *)
+ palloc(sizeof(PartitionListValue));
+ all_values[i]->value = src->value;
+ all_values[i]->index = src->index;
+ i++;
+ }
+
+ qsort_arg(all_values, ndatums, sizeof(PartitionListValue *),
+ qsort_partition_list_value_cmp, (void *) key);
+ }
+ else if (key->strategy == PARTITION_STRATEGY_RANGE)
+ {
+ int k;
+ PartitionRangeBound **all_bounds,
+ *prev;
+
+ all_bounds = (PartitionRangeBound **) palloc0(2 * nparts *
+ sizeof(PartitionRangeBound *));
+
+ /* Create a unified list of range bounds across all the partitions. */
+ i = ndatums = 0;
+ foreach(cell, boundspecs)
+ {
+ PartitionBoundSpec *spec = castNode(PartitionBoundSpec,
+ lfirst(cell));
+ PartitionRangeBound *lower,
+ *upper;
+
+ if (spec->strategy != PARTITION_STRATEGY_RANGE)
+ elog(ERROR, "invalid strategy in partition bound spec");
+
+ /*
+ * Note the index of the partition bound spec for the default
+ * partition. There's no datum to add to the allbounds array
+ * for this partition.
+ */
+ if (spec->is_default)
+ {
+ default_index = i++;
+ continue;
+ }
+
+ lower = make_one_partition_rbound(key, i, spec->lowerdatums,
+ true);
+ upper = make_one_partition_rbound(key, i, spec->upperdatums,
+ false);
+ all_bounds[ndatums++] = lower;
+ all_bounds[ndatums++] = upper;
+ i++;
+ }
+
+ Assert(ndatums == nparts * 2 ||
+ (default_index != -1 && ndatums == (nparts - 1) * 2));
+
+ /* Sort all the bounds in ascending order */
+ qsort_arg(all_bounds, ndatums,
+ sizeof(PartitionRangeBound *),
+ qsort_partition_rbound_cmp,
+ (void *) key);
+
+ /* Save distinct bounds from all_bounds into rbounds. */
+ rbounds = (PartitionRangeBound **)
+ palloc(ndatums * sizeof(PartitionRangeBound *));
+ k = 0;
+ prev = NULL;
+ for (i = 0; i < ndatums; i++)
+ {
+ PartitionRangeBound *cur = all_bounds[i];
+ bool is_distinct = false;
+ int j;
+
+ /* Is the current bound distinct from the previous one? */
+ for (j = 0; j < key->partnatts; j++)
+ {
+ Datum cmpval;
+
+ if (prev == NULL || cur->kind[j] != prev->kind[j])
+ {
+ is_distinct = true;
+ break;
+ }
+
+ /*
+ * If the bounds are both MINVALUE or MAXVALUE, stop now
+ * and treat them as equal, since any values after this
+ * point must be ignored.
+ */
+ if (cur->kind[j] != PARTITION_RANGE_DATUM_VALUE)
+ break;
+
+ cmpval = FunctionCall2Coll(&key->partsupfunc[j],
+ key->partcollation[j],
+ cur->datums[j],
+ prev->datums[j]);
+ if (DatumGetInt32(cmpval) != 0)
+ {
+ is_distinct = true;
+ break;
+ }
+ }
+
+ /*
+ * Only if the bound is distinct save it into a temporary
+ * array i.e. rbounds which is later copied into boundinfo
+ * datums array.
+ */
+ if (is_distinct)
+ rbounds[k++] = all_bounds[i];
+
+ prev = cur;
+ }
+
+ /* Update ndatums to hold the count of distinct datums. */
+ ndatums = k;
+ }
+ else
+ elog(ERROR, "unexpected partition strategy: %d",
+ (int) key->strategy);
+
+ boundinfo = (PartitionBoundInfoData *)
+ palloc0(sizeof(PartitionBoundInfoData));
+ boundinfo->strategy = key->strategy;
+ boundinfo->default_index = -1;
+ boundinfo->ndatums = ndatums;
+ boundinfo->null_index = -1;
+ boundinfo->datums = (Datum **) palloc0(ndatums * sizeof(Datum *));
+
+ /* Initialize mapping array with invalid values */
+ mapping = (int *) palloc(sizeof(int) * nparts);
+ for (i = 0; i < nparts; i++)
+ mapping[i] = -1;
+
+ switch (key->strategy)
+ {
+ case PARTITION_STRATEGY_HASH:
+ {
+ /* Moduli are stored in ascending order */
+ int greatest_modulus = hbounds[ndatums - 1]->modulus;
+
+ boundinfo->indexes = (int *) palloc(greatest_modulus *
+ sizeof(int));
+
+ for (i = 0; i < greatest_modulus; i++)
+ boundinfo->indexes[i] = -1;
+
+ for (i = 0; i < nparts; i++)
+ {
+ int modulus = hbounds[i]->modulus;
+ int remainder = hbounds[i]->remainder;
+
+ boundinfo->datums[i] = (Datum *) palloc(2 *
+ sizeof(Datum));
+ boundinfo->datums[i][0] = Int32GetDatum(modulus);
+ boundinfo->datums[i][1] = Int32GetDatum(remainder);
+
+ while (remainder < greatest_modulus)
+ {
+ /* overlap? */
+ Assert(boundinfo->indexes[remainder] == -1);
+ boundinfo->indexes[remainder] = i;
+ remainder += modulus;
+ }
+
+ mapping[hbounds[i]->index] = i;
+ pfree(hbounds[i]);
+ }
+ pfree(hbounds);
+ break;
+ }
+
+ case PARTITION_STRATEGY_LIST:
+ {
+ boundinfo->indexes = (int *) palloc(ndatums * sizeof(int));
+
+ /*
+ * Copy values. Indexes of individual values are mapped
+ * to canonical values so that they match for any two list
+ * partitioned tables with same number of partitions and
+ * same lists per partition. One way to canonicalize is
+ * to assign the index in all_values[] of the smallest
+ * value of each partition, as the index of all of the
+ * partition's values.
+ */
+ for (i = 0; i < ndatums; i++)
+ {
+ boundinfo->datums[i] = (Datum *) palloc(sizeof(Datum));
+ boundinfo->datums[i][0] = datumCopy(all_values[i]->value,
+ key->parttypbyval[0],
+ key->parttyplen[0]);
+
+ /* If the old index has no mapping, assign one */
+ if (mapping[all_values[i]->index] == -1)
+ mapping[all_values[i]->index] = next_index++;
+
+ boundinfo->indexes[i] = mapping[all_values[i]->index];
+ }
+
+ /*
+ * If null-accepting partition has no mapped index yet,
+ * assign one. This could happen if such partition
+ * accepts only null and hence not covered in the above
+ * loop which only handled non-null values.
+ */
+ if (null_index != -1)
+ {
+ Assert(null_index >= 0);
+ if (mapping[null_index] == -1)
+ mapping[null_index] = next_index++;
+ boundinfo->null_index = mapping[null_index];
+ }
+
+ /* Assign mapped index for the default partition. */
+ if (default_index != -1)
+ {
+ /*
+ * The default partition accepts any value not
+ * specified in the lists of other partitions, hence
+ * it should not get mapped index while assigning
+ * those for non-null datums.
+ */
+ Assert(default_index >= 0 &&
+ mapping[default_index] == -1);
+ mapping[default_index] = next_index++;
+ boundinfo->default_index = mapping[default_index];
+ }
+
+ /* All partition must now have a valid mapping */
+ Assert(next_index == nparts);
+ break;
+ }
+
+ case PARTITION_STRATEGY_RANGE:
+ {
+ boundinfo->kind = (PartitionRangeDatumKind **)
+ palloc(ndatums *
+ sizeof(PartitionRangeDatumKind *));
+ boundinfo->indexes = (int *) palloc((ndatums + 1) *
+ sizeof(int));
+
+ for (i = 0; i < ndatums; i++)
+ {
+ int j;
+
+ boundinfo->datums[i] = (Datum *) palloc(key->partnatts *
+ sizeof(Datum));
+ boundinfo->kind[i] = (PartitionRangeDatumKind *)
+ palloc(key->partnatts *
+ sizeof(PartitionRangeDatumKind));
+ for (j = 0; j < key->partnatts; j++)
+ {
+ if (rbounds[i]->kind[j] == PARTITION_RANGE_DATUM_VALUE)
+ boundinfo->datums[i][j] =
+ datumCopy(rbounds[i]->datums[j],
+ key->parttypbyval[j],
+ key->parttyplen[j]);
+ boundinfo->kind[i][j] = rbounds[i]->kind[j];
+ }
+
+ /*
+ * There is no mapping for invalid indexes.
+ *
+ * Any lower bounds in the rbounds array have invalid
+ * indexes assigned, because the values between the
+ * previous bound (if there is one) and this (lower)
+ * bound are not part of the range of any existing
+ * partition.
+ */
+ if (rbounds[i]->lower)
+ boundinfo->indexes[i] = -1;
+ else
+ {
+ int orig_index = rbounds[i]->index;
+
+ /* If the old index has no mapping, assign one */
+ if (mapping[orig_index] == -1)
+ mapping[orig_index] = next_index++;
+
+ boundinfo->indexes[i] = mapping[orig_index];
+ }
+ }
+
+ /* Assign mapped index for the default partition. */
+ if (default_index != -1)
+ {
+ Assert(default_index >= 0 && mapping[default_index] == -1);
+ mapping[default_index] = next_index++;
+ boundinfo->default_index = mapping[default_index];
+ }
+ boundinfo->indexes[i] = -1;
+ break;
+ }
+
+ default:
+ elog(ERROR, "unexpected partition strategy: %d",
+ (int) key->strategy);
+ }
+
+ /*
+ * Now assign OIDs from the original array into mapped indexes of the
+ * result array. Order of OIDs in the former is defined by the
+ * catalog scan that retrieved them, whereas that in the latter is
+ * defined by canonicalized representation of the partition bounds.
+ */
+ if (partoids != NIL)
+ {
+ Oid *oids_orig_order = (Oid *) palloc(sizeof(Oid) * nparts);
+
+ Assert(oids != NULL);
+
+ i = 0;
+ foreach(cell, partoids)
+ oids_orig_order[i++] = lfirst_oid(cell);
+
+ /* Add the OIDs to result array in the bound order. */
+ for (i = 0; i < nparts; i++)
+ (*oids)[mapping[i]] = oids_orig_order[i];
+ pfree(oids_orig_order);
+ }
+ pfree(mapping);
+
+ return boundinfo;
+}
+
+/*
* Are two partition bound collections logically equal?
*
* Used in the keep logic of relcache.c (ie, in RelationClearRelation()).
@@ -763,7 +1244,7 @@ get_hash_partition_greatest_modulus(PartitionBoundInfo bound)
* and a flag telling whether the bound is lower or not. Made into a function
* because there are multiple sites that want to use this facility.
*/
-PartitionRangeBound *
+static PartitionRangeBound *
make_one_partition_rbound(PartitionKey key, int index, List *datums, bool lower)
{
PartitionRangeBound *bound;
@@ -819,7 +1300,7 @@ make_one_partition_rbound(PartitionKey key, int index, List *datums, bool lower)
* structure, which only stores the upper bound of a common boundary between
* two contiguous partitions.
*/
-int32
+static int32
partition_rbound_cmp(int partnatts, FmgrInfo *partsupfunc,
Oid *partcollation,
Datum *datums1, PartitionRangeDatumKind *kind1,
@@ -914,7 +1395,7 @@ partition_rbound_datum_cmp(FmgrInfo *partsupfunc, Oid *partcollation,
*
* Compares modulus first, then remainder if modulus is equal.
*/
-int32
+static int32
partition_hbound_cmp(int modulus1, int remainder1, int modulus2, int remainder2)
{
if (modulus1 < modulus2)
@@ -977,7 +1458,7 @@ partition_list_bsearch(FmgrInfo *partsupfunc, Oid *partcollation,
* *is_equal is set to true if the range bound at the returned index is equal
* to the input range bound
*/
-int
+static int
partition_range_bsearch(int partnatts, FmgrInfo *partsupfunc,
Oid *partcollation,
PartitionBoundInfo boundinfo,
@@ -1102,6 +1583,51 @@ partition_hash_bsearch(PartitionBoundInfo boundinfo,
}
/*
+ * qsort_partition_hbound_cmp
+ *
+ * We sort hash bounds by modulus, then by remainder.
+ */
+static int32
+qsort_partition_hbound_cmp(const void *a, const void *b)
+{
+ PartitionHashBound *h1 = (*(PartitionHashBound *const *) a);
+ PartitionHashBound *h2 = (*(PartitionHashBound *const *) b);
+
+ return partition_hbound_cmp(h1->modulus, h1->remainder,
+ h2->modulus, h2->remainder);
+}
+
+/*
+ * qsort_partition_list_value_cmp
+ *
+ * Compare two list partition bound datums
+ */
+static int32
+qsort_partition_list_value_cmp(const void *a, const void *b, void *arg)
+{
+ Datum val1 = (*(const PartitionListValue **) a)->value,
+ val2 = (*(const PartitionListValue **) b)->value;
+ PartitionKey key = (PartitionKey) arg;
+
+ return DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[0],
+ key->partcollation[0],
+ val1, val2));
+}
+
+/* Used when sorting range bounds across all range partitions */
+static int32
+qsort_partition_rbound_cmp(const void *a, const void *b, void *arg)
+{
+ PartitionRangeBound *b1 = (*(PartitionRangeBound *const *) a);
+ PartitionRangeBound *b2 = (*(PartitionRangeBound *const *) b);
+ PartitionKey key = (PartitionKey) arg;
+
+ return partition_rbound_cmp(key->partnatts, key->partsupfunc,
+ key->partcollation, b1->datums, b1->kind,
+ b1->lower, b2);
+}
+
+/*
* get_partition_bound_num_indexes
*
* Returns the number of the entries in the partition bound indexes array.
diff --git a/src/backend/utils/cache/partcache.c b/src/backend/utils/cache/partcache.c
index 5757301d05..954187dbd7 100644
--- a/src/backend/utils/cache/partcache.c
+++ b/src/backend/utils/cache/partcache.c
@@ -38,12 +38,6 @@
static List *generate_partition_qual(Relation rel);
-static int32 qsort_partition_hbound_cmp(const void *a, const void *b);
-static int32 qsort_partition_list_value_cmp(const void *a, const void *b,
- void *arg);
-static int32 qsort_partition_rbound_cmp(const void *a, const void *b,
- void *arg);
-
/*
* RelationBuildPartitionKey
@@ -260,36 +254,17 @@ RelationBuildPartitionKey(Relation relation)
void
RelationBuildPartitionDesc(Relation rel)
{
- List *inhoids,
- *partoids;
- Oid *oids = NULL;
- List *boundspecs = NIL;
- ListCell *cell;
- int i,
- nparts;
- PartitionKey key = RelationGetPartitionKey(rel);
- PartitionDesc result;
MemoryContext oldcxt;
-
- int ndatums = 0;
- int default_index = -1;
-
- /* Hash partitioning specific */
- PartitionHashBound **hbounds = NULL;
-
- /* List partitioning specific */
- PartitionListValue **all_values = NULL;
- int null_index = -1;
-
- /* Range partitioning specific */
- PartitionRangeBound **rbounds = NULL;
+ PartitionDesc partdesc;
+ List *inhoids,
+ *boundspecs;
+ ListCell *cell;
/* Get partition oids from pg_inherits */
inhoids = find_inheritance_children(RelationGetRelid(rel), NoLock);
/* Collect bound spec nodes in a list */
- i = 0;
- partoids = NIL;
+ boundspecs = NIL;
foreach(cell, inhoids)
{
Oid inhrelid = lfirst_oid(cell);
@@ -325,245 +300,10 @@ RelationBuildPartitionDesc(Relation rel)
}
boundspecs = lappend(boundspecs, boundspec);
- partoids = lappend_oid(partoids, inhrelid);
ReleaseSysCache(tuple);
}
- nparts = list_length(partoids);
-
- if (nparts > 0)
- {
- oids = (Oid *) palloc(nparts * sizeof(Oid));
- i = 0;
- foreach(cell, partoids)
- oids[i++] = lfirst_oid(cell);
-
- /* Convert from node to the internal representation */
- if (key->strategy == PARTITION_STRATEGY_HASH)
- {
- ndatums = nparts;
- hbounds = (PartitionHashBound **)
- palloc(nparts * sizeof(PartitionHashBound *));
-
- i = 0;
- foreach(cell, boundspecs)
- {
- PartitionBoundSpec *spec = castNode(PartitionBoundSpec,
- lfirst(cell));
-
- if (spec->strategy != PARTITION_STRATEGY_HASH)
- elog(ERROR, "invalid strategy in partition bound spec");
-
- hbounds[i] = (PartitionHashBound *)
- palloc(sizeof(PartitionHashBound));
-
- hbounds[i]->modulus = spec->modulus;
- hbounds[i]->remainder = spec->remainder;
- hbounds[i]->index = i;
- i++;
- }
-
- /* Sort all the bounds in ascending order */
- qsort(hbounds, nparts, sizeof(PartitionHashBound *),
- qsort_partition_hbound_cmp);
- }
- else if (key->strategy == PARTITION_STRATEGY_LIST)
- {
- List *non_null_values = NIL;
-
- /*
- * Create a unified list of non-null values across all partitions.
- */
- i = 0;
- null_index = -1;
- foreach(cell, boundspecs)
- {
- PartitionBoundSpec *spec = castNode(PartitionBoundSpec,
- lfirst(cell));
- ListCell *c;
-
- if (spec->strategy != PARTITION_STRATEGY_LIST)
- elog(ERROR, "invalid strategy in partition bound spec");
-
- /*
- * Note the index of the partition bound spec for the default
- * partition. There's no datum to add to the list of non-null
- * datums for this partition.
- */
- if (spec->is_default)
- {
- default_index = i;
- i++;
- continue;
- }
-
- foreach(c, spec->listdatums)
- {
- Const *val = castNode(Const, lfirst(c));
- PartitionListValue *list_value = NULL;
-
- if (!val->constisnull)
- {
- list_value = (PartitionListValue *)
- palloc0(sizeof(PartitionListValue));
- list_value->index = i;
- list_value->value = val->constvalue;
- }
- else
- {
- /*
- * Never put a null into the values array, flag
- * instead for the code further down below where we
- * construct the actual relcache struct.
- */
- if (null_index != -1)
- elog(ERROR, "found null more than once");
- null_index = i;
- }
-
- if (list_value)
- non_null_values = lappend(non_null_values,
- list_value);
- }
-
- i++;
- }
-
- ndatums = list_length(non_null_values);
-
- /*
- * Collect all list values in one array. Alongside the value, we
- * also save the index of partition the value comes from.
- */
- all_values = (PartitionListValue **) palloc(ndatums *
- sizeof(PartitionListValue *));
- i = 0;
- foreach(cell, non_null_values)
- {
- PartitionListValue *src = lfirst(cell);
-
- all_values[i] = (PartitionListValue *)
- palloc(sizeof(PartitionListValue));
- all_values[i]->value = src->value;
- all_values[i]->index = src->index;
- i++;
- }
-
- qsort_arg(all_values, ndatums, sizeof(PartitionListValue *),
- qsort_partition_list_value_cmp, (void *) key);
- }
- else if (key->strategy == PARTITION_STRATEGY_RANGE)
- {
- int k;
- PartitionRangeBound **all_bounds,
- *prev;
-
- all_bounds = (PartitionRangeBound **) palloc0(2 * nparts *
- sizeof(PartitionRangeBound *));
-
- /*
- * Create a unified list of range bounds across all the
- * partitions.
- */
- i = ndatums = 0;
- foreach(cell, boundspecs)
- {
- PartitionBoundSpec *spec = castNode(PartitionBoundSpec,
- lfirst(cell));
- PartitionRangeBound *lower,
- *upper;
-
- if (spec->strategy != PARTITION_STRATEGY_RANGE)
- elog(ERROR, "invalid strategy in partition bound spec");
-
- /*
- * Note the index of the partition bound spec for the default
- * partition. There's no datum to add to the allbounds array
- * for this partition.
- */
- if (spec->is_default)
- {
- default_index = i++;
- continue;
- }
-
- lower = make_one_partition_rbound(key, i, spec->lowerdatums,
- true);
- upper = make_one_partition_rbound(key, i, spec->upperdatums,
- false);
- all_bounds[ndatums++] = lower;
- all_bounds[ndatums++] = upper;
- i++;
- }
-
- Assert(ndatums == nparts * 2 ||
- (default_index != -1 && ndatums == (nparts - 1) * 2));
-
- /* Sort all the bounds in ascending order */
- qsort_arg(all_bounds, ndatums,
- sizeof(PartitionRangeBound *),
- qsort_partition_rbound_cmp,
- (void *) key);
-
- /* Save distinct bounds from all_bounds into rbounds. */
- rbounds = (PartitionRangeBound **)
- palloc(ndatums * sizeof(PartitionRangeBound *));
- k = 0;
- prev = NULL;
- for (i = 0; i < ndatums; i++)
- {
- PartitionRangeBound *cur = all_bounds[i];
- bool is_distinct = false;
- int j;
-
- /* Is the current bound distinct from the previous one? */
- for (j = 0; j < key->partnatts; j++)
- {
- Datum cmpval;
-
- if (prev == NULL || cur->kind[j] != prev->kind[j])
- {
- is_distinct = true;
- break;
- }
-
- /*
- * If the bounds are both MINVALUE or MAXVALUE, stop now
- * and treat them as equal, since any values after this
- * point must be ignored.
- */
- if (cur->kind[j] != PARTITION_RANGE_DATUM_VALUE)
- break;
-
- cmpval = FunctionCall2Coll(&key->partsupfunc[j],
- key->partcollation[j],
- cur->datums[j],
- prev->datums[j]);
- if (DatumGetInt32(cmpval) != 0)
- {
- is_distinct = true;
- break;
- }
- }
-
- /*
- * Only if the bound is distinct save it into a temporary
- * array i.e. rbounds which is later copied into boundinfo
- * datums array.
- */
- if (is_distinct)
- rbounds[k++] = all_bounds[i];
-
- prev = cur;
- }
-
- /* Update ndatums to hold the count of distinct datums. */
- ndatums = k;
- }
- else
- elog(ERROR, "unexpected partition strategy: %d",
- (int) key->strategy);
- }
+ Assert(list_length(inhoids) == list_length(boundspecs));
/* Now build the actual relcache partition descriptor */
rel->rd_pdcxt = AllocSetContextCreate(CacheMemoryContext,
@@ -572,210 +312,19 @@ RelationBuildPartitionDesc(Relation rel)
MemoryContextCopyAndSetIdentifier(rel->rd_pdcxt, RelationGetRelationName(rel));
oldcxt = MemoryContextSwitchTo(rel->rd_pdcxt);
-
- result = (PartitionDescData *) palloc0(sizeof(PartitionDescData));
- result->nparts = nparts;
- if (nparts > 0)
+ partdesc = (PartitionDescData *) palloc0(sizeof(PartitionDescData));
+ partdesc->nparts = list_length(inhoids);
+ if (partdesc->nparts > 0)
{
- PartitionBoundInfo boundinfo;
- int *mapping;
- int next_index = 0;
-
- result->oids = (Oid *) palloc0(nparts * sizeof(Oid));
-
- boundinfo = (PartitionBoundInfoData *)
- palloc0(sizeof(PartitionBoundInfoData));
- boundinfo->strategy = key->strategy;
- boundinfo->default_index = -1;
- boundinfo->ndatums = ndatums;
- boundinfo->null_index = -1;
- boundinfo->datums = (Datum **) palloc0(ndatums * sizeof(Datum *));
-
- /* Initialize mapping array with invalid values */
- mapping = (int *) palloc(sizeof(int) * nparts);
- for (i = 0; i < nparts; i++)
- mapping[i] = -1;
-
- switch (key->strategy)
- {
- case PARTITION_STRATEGY_HASH:
- {
- /* Moduli are stored in ascending order */
- int greatest_modulus = hbounds[ndatums - 1]->modulus;
-
- boundinfo->indexes = (int *) palloc(greatest_modulus *
- sizeof(int));
-
- for (i = 0; i < greatest_modulus; i++)
- boundinfo->indexes[i] = -1;
-
- for (i = 0; i < nparts; i++)
- {
- int modulus = hbounds[i]->modulus;
- int remainder = hbounds[i]->remainder;
-
- boundinfo->datums[i] = (Datum *) palloc(2 *
- sizeof(Datum));
- boundinfo->datums[i][0] = Int32GetDatum(modulus);
- boundinfo->datums[i][1] = Int32GetDatum(remainder);
-
- while (remainder < greatest_modulus)
- {
- /* overlap? */
- Assert(boundinfo->indexes[remainder] == -1);
- boundinfo->indexes[remainder] = i;
- remainder += modulus;
- }
-
- mapping[hbounds[i]->index] = i;
- pfree(hbounds[i]);
- }
- pfree(hbounds);
- break;
- }
-
- case PARTITION_STRATEGY_LIST:
- {
- boundinfo->indexes = (int *) palloc(ndatums * sizeof(int));
-
- /*
- * Copy values. Indexes of individual values are mapped
- * to canonical values so that they match for any two list
- * partitioned tables with same number of partitions and
- * same lists per partition. One way to canonicalize is
- * to assign the index in all_values[] of the smallest
- * value of each partition, as the index of all of the
- * partition's values.
- */
- for (i = 0; i < ndatums; i++)
- {
- boundinfo->datums[i] = (Datum *) palloc(sizeof(Datum));
- boundinfo->datums[i][0] = datumCopy(all_values[i]->value,
- key->parttypbyval[0],
- key->parttyplen[0]);
-
- /* If the old index has no mapping, assign one */
- if (mapping[all_values[i]->index] == -1)
- mapping[all_values[i]->index] = next_index++;
-
- boundinfo->indexes[i] = mapping[all_values[i]->index];
- }
-
- /*
- * If null-accepting partition has no mapped index yet,
- * assign one. This could happen if such partition
- * accepts only null and hence not covered in the above
- * loop which only handled non-null values.
- */
- if (null_index != -1)
- {
- Assert(null_index >= 0);
- if (mapping[null_index] == -1)
- mapping[null_index] = next_index++;
- boundinfo->null_index = mapping[null_index];
- }
-
- /* Assign mapped index for the default partition. */
- if (default_index != -1)
- {
- /*
- * The default partition accepts any value not
- * specified in the lists of other partitions, hence
- * it should not get mapped index while assigning
- * those for non-null datums.
- */
- Assert(default_index >= 0 &&
- mapping[default_index] == -1);
- mapping[default_index] = next_index++;
- boundinfo->default_index = mapping[default_index];
- }
-
- /* All partition must now have a valid mapping */
- Assert(next_index == nparts);
- break;
- }
-
- case PARTITION_STRATEGY_RANGE:
- {
- boundinfo->kind = (PartitionRangeDatumKind **)
- palloc(ndatums *
- sizeof(PartitionRangeDatumKind *));
- boundinfo->indexes = (int *) palloc((ndatums + 1) *
- sizeof(int));
-
- for (i = 0; i < ndatums; i++)
- {
- int j;
-
- boundinfo->datums[i] = (Datum *) palloc(key->partnatts *
- sizeof(Datum));
- boundinfo->kind[i] = (PartitionRangeDatumKind *)
- palloc(key->partnatts *
- sizeof(PartitionRangeDatumKind));
- for (j = 0; j < key->partnatts; j++)
- {
- if (rbounds[i]->kind[j] == PARTITION_RANGE_DATUM_VALUE)
- boundinfo->datums[i][j] =
- datumCopy(rbounds[i]->datums[j],
- key->parttypbyval[j],
- key->parttyplen[j]);
- boundinfo->kind[i][j] = rbounds[i]->kind[j];
- }
-
- /*
- * There is no mapping for invalid indexes.
- *
- * Any lower bounds in the rbounds array have invalid
- * indexes assigned, because the values between the
- * previous bound (if there is one) and this (lower)
- * bound are not part of the range of any existing
- * partition.
- */
- if (rbounds[i]->lower)
- boundinfo->indexes[i] = -1;
- else
- {
- int orig_index = rbounds[i]->index;
-
- /* If the old index has no mapping, assign one */
- if (mapping[orig_index] == -1)
- mapping[orig_index] = next_index++;
-
- boundinfo->indexes[i] = mapping[orig_index];
- }
- }
-
- /* Assign mapped index for the default partition. */
- if (default_index != -1)
- {
- Assert(default_index >= 0 && mapping[default_index] == -1);
- mapping[default_index] = next_index++;
- boundinfo->default_index = mapping[default_index];
- }
- boundinfo->indexes[i] = -1;
- break;
- }
-
- default:
- elog(ERROR, "unexpected partition strategy: %d",
- (int) key->strategy);
- }
-
- result->boundinfo = boundinfo;
-
- /*
- * Now assign OIDs from the original array into mapped indexes of the
- * result array. Order of OIDs in the former is defined by the
- * catalog scan that retrieved them, whereas that in the latter is
- * defined by canonicalized representation of the partition bounds.
- */
- for (i = 0; i < nparts; i++)
- result->oids[mapping[i]] = oids[i];
- pfree(mapping);
+ partdesc->oids = (Oid *) palloc(partdesc->nparts * sizeof(Oid));
+ partdesc->boundinfo =
+ build_partition_boundinfo(RelationGetPartitionKey(rel),
+ boundspecs,
+ inhoids, &partdesc->oids);
}
MemoryContextSwitchTo(oldcxt);
- rel->rd_partdesc = result;
+ rel->rd_partdesc = partdesc;
}
/*
@@ -917,48 +466,3 @@ generate_partition_qual(Relation rel)
return result;
}
-
-/*
- * qsort_partition_hbound_cmp
- *
- * We sort hash bounds by modulus, then by remainder.
- */
-static int32
-qsort_partition_hbound_cmp(const void *a, const void *b)
-{
- PartitionHashBound *h1 = (*(PartitionHashBound *const *) a);
- PartitionHashBound *h2 = (*(PartitionHashBound *const *) b);
-
- return partition_hbound_cmp(h1->modulus, h1->remainder,
- h2->modulus, h2->remainder);
-}
-
-/*
- * qsort_partition_list_value_cmp
- *
- * Compare two list partition bound datums
- */
-static int32
-qsort_partition_list_value_cmp(const void *a, const void *b, void *arg)
-{
- Datum val1 = (*(const PartitionListValue **) a)->value,
- val2 = (*(const PartitionListValue **) b)->value;
- PartitionKey key = (PartitionKey) arg;
-
- return DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[0],
- key->partcollation[0],
- val1, val2));
-}
-
-/* Used when sorting range bounds across all range partitions */
-static int32
-qsort_partition_rbound_cmp(const void *a, const void *b, void *arg)
-{
- PartitionRangeBound *b1 = (*(PartitionRangeBound *const *) a);
- PartitionRangeBound *b2 = (*(PartitionRangeBound *const *) b);
- PartitionKey key = (PartitionKey) arg;
-
- return partition_rbound_cmp(key->partnatts, key->partsupfunc,
- key->partcollation, b1->datums, b1->kind,
- b1->lower, b2);
-}
diff --git a/src/include/partitioning/partbounds.h b/src/include/partitioning/partbounds.h
index c7535e32fc..aebd222ccd 100644
--- a/src/include/partitioning/partbounds.h
+++ b/src/include/partitioning/partbounds.h
@@ -109,6 +109,8 @@ extern uint64 compute_partition_hash_value(int partnatts, FmgrInfo *partsupfunc,
Datum *values, bool *isnull);
extern List *get_qual_from_partbound(Relation rel, Relation parent,
PartitionBoundSpec *spec);
+extern PartitionBoundInfo build_partition_boundinfo(PartitionKey key,
+ List *boundspecs, List *partoids, Oid **oids);
extern bool partition_bounds_equal(int partnatts, int16 *parttyplen,
bool *parttypbyval, PartitionBoundInfo b1,
PartitionBoundInfo b2);
@@ -120,14 +122,6 @@ extern void check_default_partition_contents(Relation parent,
Relation defaultRel,
PartitionBoundSpec *new_spec);
-extern PartitionRangeBound *make_one_partition_rbound(PartitionKey key, int index,
- List *datums, bool lower);
-extern int32 partition_hbound_cmp(int modulus1, int remainder1, int modulus2,
- int remainder2);
-extern int32 partition_rbound_cmp(int partnatts, FmgrInfo *partsupfunc,
- Oid *partcollation, Datum *datums1,
- PartitionRangeDatumKind *kind1, bool lower1,
- PartitionRangeBound *b2);
extern int32 partition_rbound_datum_cmp(FmgrInfo *partsupfunc,
Oid *partcollation,
Datum *rb_datums, PartitionRangeDatumKind *rb_kind,
@@ -136,10 +130,6 @@ extern int partition_list_bsearch(FmgrInfo *partsupfunc,
Oid *partcollation,
PartitionBoundInfo boundinfo,
Datum value, bool *is_equal);
-extern int partition_range_bsearch(int partnatts, FmgrInfo *partsupfunc,
- Oid *partcollation,
- PartitionBoundInfo boundinfo,
- PartitionRangeBound *probe, bool *is_equal);
extern int partition_range_datum_bsearch(FmgrInfo *partsupfunc,
Oid *partcollation,
PartitionBoundInfo boundinfo,
--
2.11.0
^ permalink raw reply [nested|flat] 55+ messages in thread
* Re: move PartitionBoundInfo creation code
2018-11-01 03:58 move PartitionBoundInfo creation code Amit Langote <[email protected]>
@ 2018-11-01 04:02 ` Michael Paquier <[email protected]>
2018-11-01 04:03 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
0 siblings, 1 reply; 55+ messages in thread
From: Michael Paquier @ 2018-11-01 04:02 UTC (permalink / raw)
To: Amit Langote <[email protected]>; +Cc: pgsql-hackers
On Thu, Nov 01, 2018 at 12:58:29PM +0900, Amit Langote wrote:
> Attached find a patch that does such refactoring, along with making some
> functions in partbounds.c that are not needed outside static.
This looks like a very good idea to me. Thanks for digging into that.
Please just make sure to register it to the next CF if not already
done.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 55+ messages in thread
* Re: move PartitionBoundInfo creation code
2018-11-01 03:58 move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-01 04:02 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
@ 2018-11-01 04:03 ` Amit Langote <[email protected]>
2018-11-05 07:21 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
0 siblings, 1 reply; 55+ messages in thread
From: Amit Langote @ 2018-11-01 04:03 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: pgsql-hackers
On 2018/11/01 13:02, Michael Paquier wrote:
> On Thu, Nov 01, 2018 at 12:58:29PM +0900, Amit Langote wrote:
>> Attached find a patch that does such refactoring, along with making some
>> functions in partbounds.c that are not needed outside static.
>
> This looks like a very good idea to me. Thanks for digging into that.
> Please just make sure to register it to the next CF if not already
> done.
Done a few moments ago. :)
Thanks,
Amit
^ permalink raw reply [nested|flat] 55+ messages in thread
* Re: move PartitionBoundInfo creation code
2018-11-01 03:58 move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-01 04:02 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-01 04:03 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
@ 2018-11-05 07:21 ` Michael Paquier <[email protected]>
2018-11-07 06:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
0 siblings, 1 reply; 55+ messages in thread
From: Michael Paquier @ 2018-11-05 07:21 UTC (permalink / raw)
To: Amit Langote <[email protected]>; +Cc: pgsql-hackers
On Thu, Nov 01, 2018 at 01:03:00PM +0900, Amit Langote wrote:
> Done a few moments ago. :)
From the file size this move is actually negative. From what I can see
partcache decreases to 400 lines, while partbounds increases to 3k
lines.
There are a couple of things that this patch is doing:
1) Move the functions comparing two bounds into partbounds.c.
2) Remove the chunk in charge of building PartitionBoundData into
partbounds.c for each method: list, hash and values.
From what I can see, this patch brings actually more confusion by doing
more things than just building all the PartitionBound structures as it
fills in each structure and then builds a mapping which is used to save
each partition OID into the correct mapping position. Wouldn't it move
on with a logic without this mapping so as the partition OIDs are
directly part of PartitionBound? It looks wrong to me to have
build_partition_boundinfo create not only partdesc->boundinfo but also
partdesc->oids, and the new routine is here to fill in data for the
former, not the latter.
The first phase building the bounds should switch to a switch/case like
the second phase.
PartitionHashBound & friends can become structures local to
partbounds.c as they are used only there.
To be more consistent with all the other routines, like
partition_bounds_equal/copy, wouldn't it be better to call the new
routine partition_bounds_build or partition_bounds_create?
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 55+ messages in thread
* Re: move PartitionBoundInfo creation code
2018-11-01 03:58 move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-01 04:02 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-01 04:03 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-05 07:21 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
@ 2018-11-07 06:34 ` Amit Langote <[email protected]>
2018-11-07 07:37 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
0 siblings, 1 reply; 55+ messages in thread
From: Amit Langote @ 2018-11-07 06:34 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: pgsql-hackers
Hi,
Thank your for taking a look.
On 2018/11/05 16:21, Michael Paquier wrote:
> On Thu, Nov 01, 2018 at 01:03:00PM +0900, Amit Langote wrote:
>> Done a few moments ago. :)
>
> From the file size this move is actually negative. From what I can see
> partcache decreases to 400 lines, while partbounds increases to 3k
> lines.
Hmm, is that because partbounds.c contains more functionality?
> There are a couple of things that this patch is doing:
> 1) Move the functions comparing two bounds into partbounds.c.
> 2) Remove the chunk in charge of building PartitionBoundData into
> partbounds.c for each method: list, hash and values.
>
> From what I can see, this patch brings actually more confusion by doing
> more things than just building all the PartitionBound structures as it
> fills in each structure and then builds a mapping which is used to save
> each partition OID into the correct mapping position. Wouldn't it move
> on with a logic without this mapping so as the partition OIDs are
> directly part of PartitionBound? It looks wrong to me to have
> build_partition_boundinfo create not only partdesc->boundinfo but also
> partdesc->oids, and the new routine is here to fill in data for the
> former, not the latter.
I think we can design the interface of partition_bounds_create such that
it returns information needed to re-arrange OIDs to be in the canonical
partition bound order, so the actual re-ordering of OIDs can be done by
the caller itself. (It may not always be OIDs, could be something else
like a struct to represent fake partitions.)
The canonical partition bound order is basically partition bounds sorted
in ascending order, or in some manner as defined by the qsort_partition_*
functions of individual partitioning methods. We need to map the new
canonical positions to the older ones by generating a map, which it only
makes sense to do in partition_bounds_create.
> The first phase building the bounds should switch to a switch/case like
> the second phase.
That bothered me too, so I looked at whether it'd be a good idea to have
just one switch () block and put all the code for different partitioning
methods into it. The resulting code seems more readable to me as one
doesn't have to shuffle between looking at the first block that generates
Datums from PartitionBoundSpecs and the second block that assigns them to
the Datum array in PartitionBoundInfo, for each partitioning method
respectively.
> PartitionHashBound & friends can become structures local to
> partbounds.c as they are used only there.
Ah, I missed that; moved in.
> To be more consistent with all the other routines, like
> partition_bounds_equal/copy, wouldn't it be better to call the new
> routine partition_bounds_build or partition_bounds_create?
Agree about naming consistency; I went with partition_bounds_create.
I've broken this down this into 3 patches for ease of review:
1. Local re-arrangement of the code in RelationBuildPartitionDesc such
that the code that creates PartitionBoundInfo is clearly separated. I've
also applied some of the comments above to that piece of code such as
unifying all of that code in one switch() block. I also found an
opportunity to simplify the code that maps canonical partition indexes to
original partition indexes.
2. Re-arrangement local to partcache.c that moves out the separated code
into partition_bounds_create() with the interface mentioned above.
3. Move partition_bounds_create to partbounds.c and make some functions
that are not needed outside partbounds.c static to that file.
Thanks,
Amit
^ permalink raw reply [nested|flat] 55+ messages in thread
* Re: move PartitionBoundInfo creation code
2018-11-01 03:58 move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-01 04:02 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-01 04:03 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-05 07:21 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 06:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
@ 2018-11-07 07:37 ` Michael Paquier <[email protected]>
2018-11-07 15:04 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 55+ messages in thread
From: Michael Paquier @ 2018-11-07 07:37 UTC (permalink / raw)
To: Amit Langote <[email protected]>; +Cc: pgsql-hackers
On Wed, Nov 07, 2018 at 03:34:38PM +0900, Amit Langote wrote:
> On 2018/11/05 16:21, Michael Paquier wrote:
>> On Thu, Nov 01, 2018 at 01:03:00PM +0900, Amit Langote wrote:
>>> Done a few moments ago. :)
>>
>> From the file size this move is actually negative. From what I can see
>> partcache decreases to 400 lines, while partbounds increases to 3k
>> lines.
>
> Hmm, is that because partbounds.c contains more functionality?
The move you are doing here makes sense, now refactoring is better if we
could avoid transforming a large file into a larger one and preserve
some more balance in the force.
> I think we can design the interface of partition_bounds_create such that
> it returns information needed to re-arrange OIDs to be in the canonical
> partition bound order, so the actual re-ordering of OIDs can be done by
> the caller itself. (It may not always be OIDs, could be something else
> like a struct to represent fake partitions.)
Interesting point. Do we have already in the code a new case for fake
partitions or a case where the partition OIDs are not used? I was
thinking why the partitions OIDs are not actually directly embedded in
each PartitionBoundInfo, removing at the same time the index numbers
used for the mapping. It looks sensible to return the mapping based
on your argument.
>> The first phase building the bounds should switch to a switch/case like
>> the second phase.
>
> That bothered me too, so I looked at whether it'd be a good idea to have
> just one switch () block and put all the code for different partitioning
> methods into it. The resulting code seems more readable to me as one
> doesn't have to shuffle between looking at the first block that generates
> Datums from PartitionBoundSpecs and the second block that assigns them to
> the Datum array in PartitionBoundInfo, for each partitioning method
> respectively.
Yeah, merging the first and second parts makes the most sense.
> I've broken this down this into 3 patches for ease of review:
>
> 1. Local re-arrangement of the code in RelationBuildPartitionDesc such
> that the code that creates PartitionBoundInfo is clearly separated. I've
> also applied some of the comments above to that piece of code such as
> unifying all of that code in one switch() block. I also found an
> opportunity to simplify the code that maps canonical partition indexes to
> original partition indexes.
>
> 2. Re-arrangement local to partcache.c that moves out the separated code
> into partition_bounds_create() with the interface mentioned above.
>
> 3. Move partition_bounds_create to partbounds.c and make some functions
> that are not needed outside partbounds.c static to that file.
That actually helps a lot in seeing how you handle the move, thanks.
What do you think about splitting the code in partition_bounds_create
even a bit more by having one routine per strategy?
Thinking crazy, we could also create a subfolder partitioning/bounds/
which includes three files with this refactoring:x
- range.c
- values.c
- hash.c
Then we keep partbounds.c which is the entry point used by partcache.c,
and partbounds.c relies on the stuff in each other sub-file when
building a strategy. This move could be interesting in the long term as
there are comparison routines and structures for each strategy if more
strategies are added.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 55+ messages in thread
* Re: move PartitionBoundInfo creation code
2018-11-01 03:58 move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-01 04:02 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-01 04:03 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-05 07:21 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 06:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-07 07:37 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
@ 2018-11-07 15:04 ` Alvaro Herrera <[email protected]>
2018-11-08 02:48 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-08 04:31 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
0 siblings, 2 replies; 55+ messages in thread
From: Alvaro Herrera @ 2018-11-07 15:04 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Amit Langote <[email protected]>; pgsql-hackers
On 2018-Nov-07, Michael Paquier wrote:
> On Wed, Nov 07, 2018 at 03:34:38PM +0900, Amit Langote wrote:
> > On 2018/11/05 16:21, Michael Paquier wrote:
> >> On Thu, Nov 01, 2018 at 01:03:00PM +0900, Amit Langote wrote:
> >>> Done a few moments ago. :)
> >>
> >> From the file size this move is actually negative. From what I can see
> >> partcache decreases to 400 lines, while partbounds increases to 3k
> >> lines.
> >
> > Hmm, is that because partbounds.c contains more functionality?
>
> The move you are doing here makes sense, now refactoring is better if we
> could avoid transforming a large file into a larger one and preserve
> some more balance in the force.
I think the code as it's structured in Amit's patch as a whole makes
sense -- partcache is smaller than partbounds, but why do we care? As
long as each is a reasonably well defined module, it seems better. In
the current situation we have a bit of a mess there. A 3k file is okay.
We're not talking about bloating, say, xlog.c which is almost 400kb now.
$ ls -lh src/backend/access/transam/xlog*
-rw-r--r-- 1 alvherre alvherre 23K oct 4 11:40 src/backend/access/transam/xlogarchive.c
-rw-r--r-- 1 alvherre alvherre 389K nov 7 11:49 src/backend/access/transam/xlog.c
-rw-r--r-- 1 alvherre alvherre 21K nov 3 12:15 src/backend/access/transam/xlogfuncs.c
-rw-r--r-- 1 alvherre alvherre 30K sep 3 12:58 src/backend/access/transam/xloginsert.c
-rw-r--r-- 1 alvherre alvherre 40K sep 3 12:58 src/backend/access/transam/xlogreader.c
-rw-r--r-- 1 alvherre alvherre 32K ago 5 21:34 src/backend/access/transam/xlogutils.c
> > I think we can design the interface of partition_bounds_create such that
> > it returns information needed to re-arrange OIDs to be in the canonical
> > partition bound order, so the actual re-ordering of OIDs can be done by
> > the caller itself. (It may not always be OIDs, could be something else
> > like a struct to represent fake partitions.)
This is interesting. I don't think the current interface is so bad that
we can't make a few tweaks later; IOW let's focus on the current patch
for now, and we can improve later. You (and David, and maybe someone
else) already have half a dozen patches touching this code, and I bet
some of these will have to be rebased over this one.
> Thinking crazy, we could also create a subfolder partitioning/bounds/
> which includes three files with this refactoring:x
> - range.c
> - values.c
> - hash.c
> Then we keep partbounds.c which is the entry point used by partcache.c,
> and partbounds.c relies on the stuff in each other sub-file when
> building a strategy. This move could be interesting in the long term as
> there are comparison routines and structures for each strategy if more
> strategies are added.
I'm not sure this is worthwhile. You'll create a bunch of independent
translation units in exchange for ...?
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 55+ messages in thread
* Re: move PartitionBoundInfo creation code
2018-11-01 03:58 move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-01 04:02 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-01 04:03 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-05 07:21 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 06:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-07 07:37 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 15:04 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
@ 2018-11-08 02:48 ` Michael Paquier <[email protected]>
2018-11-08 03:59 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
1 sibling, 1 reply; 55+ messages in thread
From: Michael Paquier @ 2018-11-08 02:48 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Amit Langote <[email protected]>; pgsql-hackers
On Wed, Nov 07, 2018 at 12:04:27PM -0300, Alvaro Herrera wrote:
> On 2018-Nov-07, Michael Paquier wrote:
> This is interesting. I don't think the current interface is so bad that
> we can't make a few tweaks later; IOW let's focus on the current patch
> for now, and we can improve later. You (and David, and maybe someone
> else) already have half a dozen patches touching this code, and I bet
> some of these will have to be rebased over this one.
Yeah, I am actually quite a fan of having the mapping being returned by
partition_bound_create and just let the caller handle the OID
assignment. That's neater. I think that the patch could gain a tad
more in clarity by splitting all sub-processing for each strategy into a
single routine though. The presented switch/case is a bit hard to
follow in the last patch set, and harder to review in details.
>> Thinking crazy, we could also create a subfolder partitioning/bounds/
>> which includes three files with this refactoring:x
>> - range.c
>> - values.c
>> - hash.c
>> Then we keep partbounds.c which is the entry point used by partcache.c,
>> and partbounds.c relies on the stuff in each other sub-file when
>> building a strategy. This move could be interesting in the long term as
>> there are comparison routines and structures for each strategy if more
>> strategies are added.
>
> I'm not sure this is worthwhile. You'll create a bunch of independent
> translation units in exchange for ...?
That's the part about going crazy and wild on this refactoring. Not all
crazy ideas are worth doing, still I like mentioning all possibilities
so as we miss nothing in the review process.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 55+ messages in thread
* Re: move PartitionBoundInfo creation code
2018-11-01 03:58 move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-01 04:02 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-01 04:03 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-05 07:21 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 06:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-07 07:37 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 15:04 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
2018-11-08 02:48 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
@ 2018-11-08 03:59 ` Amit Langote <[email protected]>
2018-11-08 06:11 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
0 siblings, 1 reply; 55+ messages in thread
From: Amit Langote @ 2018-11-08 03:59 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; Alvaro Herrera <[email protected]>; +Cc: pgsql-hackers
On 2018/11/08 11:48, Michael Paquier wrote:
>>> Thinking crazy, we could also create a subfolder partitioning/bounds/
>>> which includes three files with this refactoring:x
>>> - range.c
>>> - values.c
>>> - hash.c
>>> Then we keep partbounds.c which is the entry point used by partcache.c,
>>> and partbounds.c relies on the stuff in each other sub-file when
>>> building a strategy. This move could be interesting in the long term as
>>> there are comparison routines and structures for each strategy if more
>>> strategies are added.
>>
>> I'm not sure this is worthwhile. You'll create a bunch of independent
>> translation units in exchange for ...?
>
> That's the part about going crazy and wild on this refactoring. Not all
> crazy ideas are worth doing, still I like mentioning all possibilities
> so as we miss nothing in the review process.
It might be okay to split the big switch in partition_bounds_create() into
different functions for different partitioning methods for clarity as you
say, but let's avoid creating new files for range, list, and hash.
I will post an updated patch with that break down.
Thanks,
Amit
^ permalink raw reply [nested|flat] 55+ messages in thread
* Re: move PartitionBoundInfo creation code
2018-11-01 03:58 move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-01 04:02 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-01 04:03 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-05 07:21 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 06:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-07 07:37 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 15:04 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
2018-11-08 02:48 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-08 03:59 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
@ 2018-11-08 06:11 ` Amit Langote <[email protected]>
2018-11-12 13:55 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
0 siblings, 1 reply; 55+ messages in thread
From: Amit Langote @ 2018-11-08 06:11 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; Alvaro Herrera <[email protected]>; +Cc: pgsql-hackers
On 2018/11/08 12:59, Amit Langote wrote:
> It might be okay to split the big switch in partition_bounds_create() into
> different functions for different partitioning methods for clarity as you
> say, but let's avoid creating new files for range, list, and hash.
>
> I will post an updated patch with that break down.
And here is the new version. The break down into smaller local functions
for different partitioning methods is in patch 0002.
Thanks,
Amit
^ permalink raw reply [nested|flat] 55+ messages in thread
* Re: move PartitionBoundInfo creation code
2018-11-01 03:58 move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-01 04:02 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-01 04:03 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-05 07:21 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 06:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-07 07:37 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 15:04 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
2018-11-08 02:48 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-08 03:59 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-08 06:11 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
@ 2018-11-12 13:55 ` Michael Paquier <[email protected]>
2018-11-13 01:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
0 siblings, 1 reply; 55+ messages in thread
From: Michael Paquier @ 2018-11-12 13:55 UTC (permalink / raw)
To: Amit Langote <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; pgsql-hackers
On Thu, Nov 08, 2018 at 03:11:35PM +0900, Amit Langote wrote:
> And here is the new version. The break down into smaller local
> functions for different partitioning methods is in patch 0002.
Okay, here we go. I have spent a couple of hours on this patch,
checking the consistency of the new code and the new code, and the main
complain I have about the last version is that the code in charge of
building the mapping was made less robust. So my notes are:
- The initialization of the mapping should happen in
partition_bounds_create() before going through each routine. The
proposed patch was only doing the initialization for list bounds, using
an extra layer to track the canonical ordering of indexes associated
with a partition (listpart_canon_idx), and that's something that the
mapping can perfectly do, keeping the code more straight-forward (at
least it seems so to me) with the way null_index and default_index are
assigned.
- Some noise diffs were present, sometimes the indentation being wrong.
- Asserts could be used instead of elog(ERROR) if the strategy is not
the expected one. That looks cleaner to me.
The result is perhaps less smart than what you did, but that's more
robust. It may make sense to change the way the mapping is built but
I would suggest to do so after the actual refactoring. Attached is what
I finish with, and I have spent quite some time making sure that
all the new logic remains consistent with the previous one.
--
Michael
Attachments:
[text/x-diff] partition-bound-refactor.patch (40.2K, ../../[email protected]/2-partition-bound-refactor.patch)
download | inline diff:
diff --git a/src/backend/partitioning/partbounds.c b/src/backend/partitioning/partbounds.c
index c94f73aadc..0088e75000 100644
--- a/src/backend/partitioning/partbounds.c
+++ b/src/backend/partitioning/partbounds.c
@@ -36,6 +36,61 @@
#include "utils/ruleutils.h"
#include "utils/syscache.h"
+/*
+ * When qsort'ing partition bounds after reading from the catalog, each bound
+ * is represented with one of the following structs.
+ */
+
+/* One bound of a hash partition */
+typedef struct PartitionHashBound
+{
+ int modulus;
+ int remainder;
+ int index;
+} PartitionHashBound;
+
+/* One value coming from some (index'th) list partition */
+typedef struct PartitionListValue
+{
+ int index;
+ Datum value;
+} PartitionListValue;
+
+/* One bound of a range partition */
+typedef struct PartitionRangeBound
+{
+ int index;
+ Datum *datums; /* range bound datums */
+ PartitionRangeDatumKind *kind; /* the kind of each datum */
+ bool lower; /* this is the lower (vs upper) bound */
+} PartitionRangeBound;
+
+static int32 qsort_partition_hbound_cmp(const void *a, const void *b);
+static int32 qsort_partition_list_value_cmp(const void *a, const void *b,
+ void *arg);
+static int32 qsort_partition_rbound_cmp(const void *a, const void *b,
+ void *arg);
+static PartitionBoundInfo create_hash_bounds(List *boundspecs,
+ PartitionKey key,
+ int **mapping);
+static PartitionBoundInfo create_list_bounds(List *boundspecs,
+ PartitionKey key,
+ int **mapping);
+static PartitionBoundInfo create_range_bounds(List *boundspecs,
+ PartitionKey key,
+ int **mapping);
+static PartitionRangeBound *make_one_partition_rbound(PartitionKey key, int index,
+ List *datums, bool lower);
+static int32 partition_hbound_cmp(int modulus1, int remainder1, int modulus2,
+ int remainder2);
+static int32 partition_rbound_cmp(int partnatts, FmgrInfo *partsupfunc,
+ Oid *partcollation, Datum *datums1,
+ PartitionRangeDatumKind *kind1, bool lower1,
+ PartitionRangeBound *b2);
+static int partition_range_bsearch(int partnatts, FmgrInfo *partsupfunc,
+ Oid *partcollation,
+ PartitionBoundInfo boundinfo,
+ PartitionRangeBound *probe, bool *is_equal);
static int get_partition_bound_num_indexes(PartitionBoundInfo b);
static Expr *make_partition_op_expr(PartitionKey key, int keynum,
uint16 strategy, Expr *arg1, Expr *arg2);
@@ -92,6 +147,518 @@ get_qual_from_partbound(Relation rel, Relation parent,
return my_qual;
}
+/*
+ * partition_bounds_create
+ * Build a PartitionBoundInfo struct from a list of PartitionBoundSpec
+ * nodes
+ *
+ * This function creates a PartitionBoundInfo and fills the values of its
+ * various members based on the input list. Importantly, 'datums' array will
+ * contain Datum representation of individual bounds (possibly after
+ * de-duplication as in case of range bounds), sorted in a canonical order
+ * defined by qsort_partition_* functions of respective partitioning methods.
+ * 'indexes' array will contain as many elements as there are bounds (specific
+ * exceptions to this rule are listed in the function body), which represent
+ * the 0-based canonical positions of partitions.
+ *
+ * Upon return from this function, *mapping is set to an array of
+ * list_length(boundspecs) elements, each of which maps the canonical
+ * index of a given partition to its 0-based position in the original list.
+ *
+ * Note: All the memory allocated by this function, including that of the
+ * the returned PartitionBoundInfo and its members is allocated in the context
+ * that was active when the function was called.
+ */
+PartitionBoundInfo
+partition_bounds_create(List *boundspecs, PartitionKey key, int **mapping)
+{
+ int nparts = list_length(boundspecs);
+ int i;
+
+ Assert(nparts > 0);
+
+ /*
+ * For each partitioning method, we first convert the partition bounds
+ * from their parser node representation to the internal representation,
+ * along with any additional preprocessing (such performing de-duplication
+ * on range bounds). For each bound, we remember its partition's position
+ * (0-based) in the original list, so that we can add it to the *mapping
+ * array.
+ *
+ * Resulting bound datums are then added to the 'datums' array in
+ * PartitionBoundInfo. For each datum added, an integer indicating the
+ * canonical partition index is added to the 'indexes' array.
+ */
+
+ /*
+ * Initialize mapping array with invalid values, this is filled within
+ * each sub-routine below depending on the bound type.
+ */
+ *mapping = (int *) palloc(sizeof(int) * nparts);
+ for (i = 0; i < nparts; i++)
+ (*mapping)[i] = -1;
+
+ switch (key->strategy)
+ {
+ case PARTITION_STRATEGY_HASH:
+ return create_hash_bounds(boundspecs, key, mapping);
+
+ case PARTITION_STRATEGY_LIST:
+ return create_list_bounds(boundspecs, key, mapping);
+
+ case PARTITION_STRATEGY_RANGE:
+ return create_range_bounds(boundspecs, key, mapping);
+
+ default:
+ elog(ERROR, "unexpected partition strategy: %d",
+ (int) key->strategy);
+ break;
+ }
+
+ Assert(false);
+ return NULL; /* keep compiler quiet */
+}
+
+/*
+ * create_hash_bounds
+ * Create a PartitionBoundInfo for a hash partitioned table
+ */
+static PartitionBoundInfo
+create_hash_bounds(List *boundspecs, PartitionKey key, int **mapping)
+{
+ PartitionBoundInfo boundinfo;
+ PartitionHashBound **hbounds = NULL;
+ ListCell *cell;
+ int i,
+ nparts = list_length(boundspecs);
+ int ndatums = 0;
+ int greatest_modulus;
+
+ boundinfo = (PartitionBoundInfoData *)
+ palloc0(sizeof(PartitionBoundInfoData));
+ boundinfo->strategy = key->strategy;
+ /* No special hash partitions. */
+ boundinfo->null_index = -1;
+ boundinfo->default_index = -1;
+
+ ndatums = nparts;
+ hbounds = (PartitionHashBound **)
+ palloc(nparts * sizeof(PartitionHashBound *));
+
+ /* Convert from node to the internal representation */
+ i = 0;
+ foreach(cell, boundspecs)
+ {
+ PartitionBoundSpec *spec = castNode(PartitionBoundSpec, lfirst(cell));
+
+ Assert(spec->strategy == PARTITION_STRATEGY_HASH);
+
+ hbounds[i] = (PartitionHashBound *) palloc(sizeof(PartitionHashBound));
+ hbounds[i]->modulus = spec->modulus;
+ hbounds[i]->remainder = spec->remainder;
+ hbounds[i]->index = i;
+ i++;
+ }
+
+ /* Sort all the bounds in ascending order */
+ qsort(hbounds, nparts, sizeof(PartitionHashBound *),
+ qsort_partition_hbound_cmp);
+
+ /* After sorting, moduli are now stored in ascending order. */
+ greatest_modulus = hbounds[ndatums - 1]->modulus;
+
+ boundinfo->ndatums = ndatums;
+ boundinfo->datums = (Datum **) palloc0(ndatums * sizeof(Datum *));
+ boundinfo->indexes = (int *) palloc(greatest_modulus * sizeof(int));
+ for (i = 0; i < greatest_modulus; i++)
+ boundinfo->indexes[i] = -1;
+
+ /*
+ * For hash partitioning, there are as many datums (modulus and remainder
+ * pairs) as there are partitions. Indexes are simply values ranging from
+ * 0 to (nparts - 1).
+ */
+ for (i = 0; i < nparts; i++)
+ {
+ int modulus = hbounds[i]->modulus;
+ int remainder = hbounds[i]->remainder;
+
+ boundinfo->datums[i] = (Datum *) palloc(2 * sizeof(Datum));
+ boundinfo->datums[i][0] = Int32GetDatum(modulus);
+ boundinfo->datums[i][1] = Int32GetDatum(remainder);
+
+ while (remainder < greatest_modulus)
+ {
+ /* overlap? */
+ Assert(boundinfo->indexes[remainder] == -1);
+ boundinfo->indexes[remainder] = i;
+ remainder += modulus;
+ }
+
+ (*mapping)[hbounds[i]->index] = i;
+ pfree(hbounds[i]);
+ }
+ pfree(hbounds);
+
+ return boundinfo;
+}
+
+/*
+ * create_list_bounds
+ * Create a PartitionBoundInfo for a list partitioned table
+ */
+static PartitionBoundInfo
+create_list_bounds(List *boundspecs, PartitionKey key, int **mapping)
+{
+ PartitionBoundInfo boundinfo;
+ PartitionListValue **all_values = NULL;
+ ListCell *cell;
+ int i = 0;
+ int ndatums = 0;
+ int next_index = 0;
+ int default_index = -1;
+ int null_index = -1;
+ List *non_null_values = NIL;
+
+ boundinfo = (PartitionBoundInfoData *)
+ palloc0(sizeof(PartitionBoundInfoData));
+ boundinfo->strategy = key->strategy;
+ /* Will be set correctly below. */
+ boundinfo->null_index = -1;
+ boundinfo->default_index = -1;
+
+ /* Create a unified list of non-null values across all partitions. */
+ foreach(cell, boundspecs)
+ {
+ PartitionBoundSpec *spec = castNode(PartitionBoundSpec, lfirst(cell));
+ ListCell *c;
+
+ Assert(spec->strategy == PARTITION_STRATEGY_LIST);
+
+ /*
+ * Note the index of the partition bound spec for the default
+ * partition. There's no datum to add to the list on non-null datums
+ * for this partition.
+ */
+ if (spec->is_default)
+ {
+ default_index = i;
+ i++;
+ continue;
+ }
+
+ foreach(c, spec->listdatums)
+ {
+ Const *val = castNode(Const, lfirst(c));
+ PartitionListValue *list_value = NULL;
+
+ if (!val->constisnull)
+ {
+ list_value = (PartitionListValue *)
+ palloc0(sizeof(PartitionListValue));
+ list_value->index = i;
+ list_value->value = val->constvalue;
+ }
+ else
+ {
+ /*
+ * Never put a null into the values array, flag instead for
+ * the code further down below where we construct the actual
+ * relcache struct.
+ */
+ if (null_index != -1)
+ elog(ERROR, "found null more than once");
+ null_index = i;
+ }
+
+ if (list_value)
+ non_null_values = lappend(non_null_values, list_value);
+ }
+
+ i++;
+ }
+
+ ndatums = list_length(non_null_values);
+
+ /*
+ * Collect all list values in one array. Alongside the value, we also save
+ * the index of partition the value comes from.
+ */
+ all_values = (PartitionListValue **)
+ palloc(ndatums * sizeof(PartitionListValue *));
+ i = 0;
+ foreach(cell, non_null_values)
+ {
+ PartitionListValue *src = lfirst(cell);
+
+ all_values[i] = (PartitionListValue *)
+ palloc(sizeof(PartitionListValue));
+ all_values[i]->value = src->value;
+ all_values[i]->index = src->index;
+ i++;
+ }
+
+ qsort_arg(all_values, ndatums, sizeof(PartitionListValue *),
+ qsort_partition_list_value_cmp, (void *) key);
+
+ boundinfo->ndatums = ndatums;
+ boundinfo->datums = (Datum **) palloc0(ndatums * sizeof(Datum *));
+ boundinfo->indexes = (int *) palloc(ndatums * sizeof(int));
+
+ /*
+ * Copy values. Indexes of individual values are mapped to canonical
+ * values so that they match for any two list partitioned tables with same
+ * number of partitions and same lists per partition. One way to
+ * canonicalize is to assign the index in all_values[] of the smallest
+ * value of each partition, as the index of all of the partition's values.
+ */
+ for (i = 0; i < ndatums; i++)
+ {
+ int orig_index = all_values[i]->index;
+
+ boundinfo->datums[i] = (Datum *) palloc(sizeof(Datum));
+ boundinfo->datums[i][0] = datumCopy(all_values[i]->value,
+ key->parttypbyval[0],
+ key->parttyplen[0]);
+
+ /* If the old index has no mapping, assign one */
+ if ((*mapping)[orig_index] == -1)
+ (*mapping)[orig_index] = next_index++;
+
+ boundinfo->indexes[i] = (*mapping)[orig_index];
+ }
+
+ /*
+ * If null-accepting partition has no mapped index yet, assign one. This
+ * could happen if such partition accepts only null and hence not covered
+ * in the above loop which only handled non-null values.
+ */
+ if (null_index != -1)
+ {
+ Assert(null_index >= 0);
+ if ((*mapping)[null_index] == -1)
+ (*mapping)[null_index] = next_index++;
+ boundinfo->null_index = (*mapping)[null_index];
+ }
+
+ /* Assign mapped index for the default partition. */
+ if (default_index != -1)
+ {
+ /*
+ * The default partition accepts any value not specified in the lists
+ * of other partitions, hence it should not get mapped index while
+ * assigning those for non-null datums.
+ */
+ Assert(default_index >= 0);
+ Assert((*mapping)[default_index] == -1);
+ (*mapping)[default_index] = next_index++;
+ boundinfo->default_index = (*mapping)[default_index];
+ }
+
+ /* All partition must now have been assigned canonical indexes. */
+ Assert(next_index == list_length(boundspecs));
+ return boundinfo;
+}
+
+/*
+ * create_range_bounds
+ * Create a PartitionBoundInfo for a range partitioned table
+ */
+static PartitionBoundInfo
+create_range_bounds(List *boundspecs, PartitionKey key, int **mapping)
+{
+ PartitionBoundInfo boundinfo;
+ PartitionRangeBound **rbounds = NULL;
+ PartitionRangeBound **all_bounds,
+ *prev;
+ ListCell *cell;
+ int i,
+ k,
+ nparts = list_length(boundspecs);
+ int ndatums = 0;
+ int default_index = -1;
+ int next_index = 0;
+
+ boundinfo = (PartitionBoundInfoData *)
+ palloc0(sizeof(PartitionBoundInfoData));
+ boundinfo->strategy = key->strategy;
+ /* There is no special null-accepting range partition. */
+ boundinfo->null_index = -1;
+ /* Will be set correctly below. */
+ boundinfo->default_index = -1;
+
+ all_bounds = (PartitionRangeBound **)
+ palloc0(2 * nparts * sizeof(PartitionRangeBound *));
+
+ /* Create a unified list of range bounds across all the partitions. */
+ i = ndatums = 0;
+ foreach(cell, boundspecs)
+ {
+ PartitionBoundSpec *spec = castNode(PartitionBoundSpec, lfirst(cell));
+ PartitionRangeBound *lower,
+ *upper;
+
+ Assert(spec->strategy == PARTITION_STRATEGY_RANGE);
+
+ /*
+ * Note the index of the partition bound spec for the default
+ * partition. There's no datum to add to the all_bounds array for
+ * this partition.
+ */
+ if (spec->is_default)
+ {
+ default_index = i++;
+ continue;
+ }
+
+ lower = make_one_partition_rbound(key, i, spec->lowerdatums, true);
+ upper = make_one_partition_rbound(key, i, spec->upperdatums, false);
+ all_bounds[ndatums++] = lower;
+ all_bounds[ndatums++] = upper;
+ i++;
+ }
+
+ Assert(ndatums == nparts * 2 ||
+ (default_index != -1 && ndatums == (nparts - 1) * 2));
+
+ /* Sort all the bounds in ascending order */
+ qsort_arg(all_bounds, ndatums,
+ sizeof(PartitionRangeBound *),
+ qsort_partition_rbound_cmp,
+ (void *) key);
+
+ /* Save distinct bounds from all_bounds into rbounds. */
+ rbounds = (PartitionRangeBound **)
+ palloc(ndatums * sizeof(PartitionRangeBound *));
+ k = 0;
+ prev = NULL;
+ for (i = 0; i < ndatums; i++)
+ {
+ PartitionRangeBound *cur = all_bounds[i];
+ bool is_distinct = false;
+ int j;
+
+ /* Is the current bound distinct from the previous one? */
+ for (j = 0; j < key->partnatts; j++)
+ {
+ Datum cmpval;
+
+ if (prev == NULL || cur->kind[j] != prev->kind[j])
+ {
+ is_distinct = true;
+ break;
+ }
+
+ /*
+ * If the bounds are both MINVALUE or MAXVALUE, stop now and treat
+ * them as equal, since any values after this point must be
+ * ignored.
+ */
+ if (cur->kind[j] != PARTITION_RANGE_DATUM_VALUE)
+ break;
+
+ cmpval = FunctionCall2Coll(&key->partsupfunc[j],
+ key->partcollation[j],
+ cur->datums[j],
+ prev->datums[j]);
+ if (DatumGetInt32(cmpval) != 0)
+ {
+ is_distinct = true;
+ break;
+ }
+ }
+
+ /*
+ * Only if the bound is distinct save it into a temporary array, i.e,
+ * rbounds which is later copied into boundinfo datums array.
+ */
+ if (is_distinct)
+ rbounds[k++] = all_bounds[i];
+
+ prev = cur;
+ }
+
+ /* Update ndatums to hold the count of distinct datums. */
+ ndatums = k;
+
+ /*
+ * Add datums to boundinfo. Canonical indexes are values ranging from 0
+ * to nparts - 1, assigned in that order to each partition's upper bound.
+ * For 'datums' elements that are lower bounds, there is -1 in the
+ * 'indexes' array to signify that no partition exists for the values less
+ * than such a bound and greater than or equal to the previous upper
+ * bound.
+ */
+ boundinfo->ndatums = ndatums;
+ boundinfo->datums = (Datum **) palloc0(ndatums * sizeof(Datum *));
+ boundinfo->kind = (PartitionRangeDatumKind **)
+ palloc(ndatums *
+ sizeof(PartitionRangeDatumKind *));
+
+ /*
+ * For range partitioning, an additional value of -1 is stored as the last
+ * element.
+ */
+ boundinfo->indexes = (int *) palloc((ndatums + 1) * sizeof(int));
+
+ for (i = 0; i < ndatums; i++)
+ {
+ int j;
+
+ boundinfo->datums[i] = (Datum *) palloc(key->partnatts *
+ sizeof(Datum));
+ boundinfo->kind[i] = (PartitionRangeDatumKind *)
+ palloc(key->partnatts *
+ sizeof(PartitionRangeDatumKind));
+ for (j = 0; j < key->partnatts; j++)
+ {
+ if (rbounds[i]->kind[j] == PARTITION_RANGE_DATUM_VALUE)
+ boundinfo->datums[i][j] =
+ datumCopy(rbounds[i]->datums[j],
+ key->parttypbyval[j],
+ key->parttyplen[j]);
+ boundinfo->kind[i][j] = rbounds[i]->kind[j];
+ }
+
+ /*
+ * There is no mapping for invalid indexes.
+ *
+ * Any lower bounds in the rbounds array have invalid indexes
+ * assigned, because the values between the previous bound (if there
+ * is one) and this (lower) bound are not part of the range of any
+ * existing partition.
+ */
+ if (rbounds[i]->lower)
+ boundinfo->indexes[i] = -1;
+ else
+ {
+ int orig_index = rbounds[i]->index;
+
+ /* If the old index has no mapping, assign one */
+ if ((*mapping)[orig_index] == -1)
+ (*mapping)[orig_index] = next_index++;
+
+ boundinfo->indexes[i] = (*mapping)[orig_index];
+ }
+ }
+
+ /* Assign mapped index for the default partition. */
+ if (default_index != -1)
+ {
+ Assert(default_index >= 0 && (*mapping)[default_index] == -1);
+ (*mapping)[default_index] = next_index++;
+ boundinfo->default_index = (*mapping)[default_index];
+ }
+
+ /* The extra -1 element. */
+ Assert(i == ndatums);
+ boundinfo->indexes[i] = -1;
+
+ /* All partition must now have been assigned canonical indexes. */
+ Assert(next_index == nparts);
+ return boundinfo;
+}
+
/*
* Are two partition bound collections logically equal?
*
@@ -763,7 +1330,7 @@ get_hash_partition_greatest_modulus(PartitionBoundInfo bound)
* and a flag telling whether the bound is lower or not. Made into a function
* because there are multiple sites that want to use this facility.
*/
-PartitionRangeBound *
+static PartitionRangeBound *
make_one_partition_rbound(PartitionKey key, int index, List *datums, bool lower)
{
PartitionRangeBound *bound;
@@ -819,7 +1386,7 @@ make_one_partition_rbound(PartitionKey key, int index, List *datums, bool lower)
* structure, which only stores the upper bound of a common boundary between
* two contiguous partitions.
*/
-int32
+static int32
partition_rbound_cmp(int partnatts, FmgrInfo *partsupfunc,
Oid *partcollation,
Datum *datums1, PartitionRangeDatumKind *kind1,
@@ -914,7 +1481,7 @@ partition_rbound_datum_cmp(FmgrInfo *partsupfunc, Oid *partcollation,
*
* Compares modulus first, then remainder if modulus is equal.
*/
-int32
+static int32
partition_hbound_cmp(int modulus1, int remainder1, int modulus2, int remainder2)
{
if (modulus1 < modulus2)
@@ -977,7 +1544,7 @@ partition_list_bsearch(FmgrInfo *partsupfunc, Oid *partcollation,
* *is_equal is set to true if the range bound at the returned index is equal
* to the input range bound
*/
-int
+static int
partition_range_bsearch(int partnatts, FmgrInfo *partsupfunc,
Oid *partcollation,
PartitionBoundInfo boundinfo,
@@ -1101,6 +1668,55 @@ partition_hash_bsearch(PartitionBoundInfo boundinfo,
return lo;
}
+/*
+ * qsort_partition_hbound_cmp
+ *
+ * Hash bounds are sorted by modulus, then by remainder.
+ */
+static int32
+qsort_partition_hbound_cmp(const void *a, const void *b)
+{
+ PartitionHashBound *h1 = (*(PartitionHashBound *const *) a);
+ PartitionHashBound *h2 = (*(PartitionHashBound *const *) b);
+
+ return partition_hbound_cmp(h1->modulus, h1->remainder,
+ h2->modulus, h2->remainder);
+}
+
+/*
+ * qsort_partition_list_value_cmp
+ *
+ * Compare two list partition bound datums.
+ */
+static int32
+qsort_partition_list_value_cmp(const void *a, const void *b, void *arg)
+{
+ Datum val1 = (*(const PartitionListValue **) a)->value,
+ val2 = (*(const PartitionListValue **) b)->value;
+ PartitionKey key = (PartitionKey) arg;
+
+ return DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[0],
+ key->partcollation[0],
+ val1, val2));
+}
+
+/*
+ * qsort_partition_rbound_cmp
+ *
+ * Used when sorting range bounds across all range partitions.
+ */
+static int32
+qsort_partition_rbound_cmp(const void *a, const void *b, void *arg)
+{
+ PartitionRangeBound *b1 = (*(PartitionRangeBound *const *) a);
+ PartitionRangeBound *b2 = (*(PartitionRangeBound *const *) b);
+ PartitionKey key = (PartitionKey) arg;
+
+ return partition_rbound_cmp(key->partnatts, key->partsupfunc,
+ key->partcollation, b1->datums, b1->kind,
+ b1->lower, b2);
+}
+
/*
* get_partition_bound_num_indexes
*
diff --git a/src/backend/utils/cache/partcache.c b/src/backend/utils/cache/partcache.c
index 5757301d05..020bca1f54 100644
--- a/src/backend/utils/cache/partcache.c
+++ b/src/backend/utils/cache/partcache.c
@@ -38,12 +38,6 @@
static List *generate_partition_qual(Relation rel);
-static int32 qsort_partition_hbound_cmp(const void *a, const void *b);
-static int32 qsort_partition_list_value_cmp(const void *a, const void *b,
- void *arg);
-static int32 qsort_partition_rbound_cmp(const void *a, const void *b,
- void *arg);
-
/*
* RelationBuildPartitionKey
@@ -260,36 +254,22 @@ RelationBuildPartitionKey(Relation relation)
void
RelationBuildPartitionDesc(Relation rel)
{
- List *inhoids,
- *partoids;
- Oid *oids = NULL;
+ PartitionDesc partdesc;
+ PartitionBoundInfo boundinfo;
+ List *inhoids;
List *boundspecs = NIL;
ListCell *cell;
int i,
nparts;
PartitionKey key = RelationGetPartitionKey(rel);
- PartitionDesc result;
MemoryContext oldcxt;
-
- int ndatums = 0;
- int default_index = -1;
-
- /* Hash partitioning specific */
- PartitionHashBound **hbounds = NULL;
-
- /* List partitioning specific */
- PartitionListValue **all_values = NULL;
- int null_index = -1;
-
- /* Range partitioning specific */
- PartitionRangeBound **rbounds = NULL;
+ Oid *oids_orig;
+ int *mapping;
/* Get partition oids from pg_inherits */
inhoids = find_inheritance_children(RelationGetRelid(rel), NoLock);
/* Collect bound spec nodes in a list */
- i = 0;
- partoids = NIL;
foreach(cell, inhoids)
{
Oid inhrelid = lfirst_oid(cell);
@@ -325,245 +305,10 @@ RelationBuildPartitionDesc(Relation rel)
}
boundspecs = lappend(boundspecs, boundspec);
- partoids = lappend_oid(partoids, inhrelid);
ReleaseSysCache(tuple);
}
- nparts = list_length(partoids);
-
- if (nparts > 0)
- {
- oids = (Oid *) palloc(nparts * sizeof(Oid));
- i = 0;
- foreach(cell, partoids)
- oids[i++] = lfirst_oid(cell);
-
- /* Convert from node to the internal representation */
- if (key->strategy == PARTITION_STRATEGY_HASH)
- {
- ndatums = nparts;
- hbounds = (PartitionHashBound **)
- palloc(nparts * sizeof(PartitionHashBound *));
-
- i = 0;
- foreach(cell, boundspecs)
- {
- PartitionBoundSpec *spec = castNode(PartitionBoundSpec,
- lfirst(cell));
-
- if (spec->strategy != PARTITION_STRATEGY_HASH)
- elog(ERROR, "invalid strategy in partition bound spec");
-
- hbounds[i] = (PartitionHashBound *)
- palloc(sizeof(PartitionHashBound));
-
- hbounds[i]->modulus = spec->modulus;
- hbounds[i]->remainder = spec->remainder;
- hbounds[i]->index = i;
- i++;
- }
-
- /* Sort all the bounds in ascending order */
- qsort(hbounds, nparts, sizeof(PartitionHashBound *),
- qsort_partition_hbound_cmp);
- }
- else if (key->strategy == PARTITION_STRATEGY_LIST)
- {
- List *non_null_values = NIL;
-
- /*
- * Create a unified list of non-null values across all partitions.
- */
- i = 0;
- null_index = -1;
- foreach(cell, boundspecs)
- {
- PartitionBoundSpec *spec = castNode(PartitionBoundSpec,
- lfirst(cell));
- ListCell *c;
-
- if (spec->strategy != PARTITION_STRATEGY_LIST)
- elog(ERROR, "invalid strategy in partition bound spec");
-
- /*
- * Note the index of the partition bound spec for the default
- * partition. There's no datum to add to the list of non-null
- * datums for this partition.
- */
- if (spec->is_default)
- {
- default_index = i;
- i++;
- continue;
- }
-
- foreach(c, spec->listdatums)
- {
- Const *val = castNode(Const, lfirst(c));
- PartitionListValue *list_value = NULL;
-
- if (!val->constisnull)
- {
- list_value = (PartitionListValue *)
- palloc0(sizeof(PartitionListValue));
- list_value->index = i;
- list_value->value = val->constvalue;
- }
- else
- {
- /*
- * Never put a null into the values array, flag
- * instead for the code further down below where we
- * construct the actual relcache struct.
- */
- if (null_index != -1)
- elog(ERROR, "found null more than once");
- null_index = i;
- }
-
- if (list_value)
- non_null_values = lappend(non_null_values,
- list_value);
- }
-
- i++;
- }
-
- ndatums = list_length(non_null_values);
-
- /*
- * Collect all list values in one array. Alongside the value, we
- * also save the index of partition the value comes from.
- */
- all_values = (PartitionListValue **) palloc(ndatums *
- sizeof(PartitionListValue *));
- i = 0;
- foreach(cell, non_null_values)
- {
- PartitionListValue *src = lfirst(cell);
-
- all_values[i] = (PartitionListValue *)
- palloc(sizeof(PartitionListValue));
- all_values[i]->value = src->value;
- all_values[i]->index = src->index;
- i++;
- }
-
- qsort_arg(all_values, ndatums, sizeof(PartitionListValue *),
- qsort_partition_list_value_cmp, (void *) key);
- }
- else if (key->strategy == PARTITION_STRATEGY_RANGE)
- {
- int k;
- PartitionRangeBound **all_bounds,
- *prev;
-
- all_bounds = (PartitionRangeBound **) palloc0(2 * nparts *
- sizeof(PartitionRangeBound *));
-
- /*
- * Create a unified list of range bounds across all the
- * partitions.
- */
- i = ndatums = 0;
- foreach(cell, boundspecs)
- {
- PartitionBoundSpec *spec = castNode(PartitionBoundSpec,
- lfirst(cell));
- PartitionRangeBound *lower,
- *upper;
-
- if (spec->strategy != PARTITION_STRATEGY_RANGE)
- elog(ERROR, "invalid strategy in partition bound spec");
-
- /*
- * Note the index of the partition bound spec for the default
- * partition. There's no datum to add to the allbounds array
- * for this partition.
- */
- if (spec->is_default)
- {
- default_index = i++;
- continue;
- }
-
- lower = make_one_partition_rbound(key, i, spec->lowerdatums,
- true);
- upper = make_one_partition_rbound(key, i, spec->upperdatums,
- false);
- all_bounds[ndatums++] = lower;
- all_bounds[ndatums++] = upper;
- i++;
- }
-
- Assert(ndatums == nparts * 2 ||
- (default_index != -1 && ndatums == (nparts - 1) * 2));
-
- /* Sort all the bounds in ascending order */
- qsort_arg(all_bounds, ndatums,
- sizeof(PartitionRangeBound *),
- qsort_partition_rbound_cmp,
- (void *) key);
-
- /* Save distinct bounds from all_bounds into rbounds. */
- rbounds = (PartitionRangeBound **)
- palloc(ndatums * sizeof(PartitionRangeBound *));
- k = 0;
- prev = NULL;
- for (i = 0; i < ndatums; i++)
- {
- PartitionRangeBound *cur = all_bounds[i];
- bool is_distinct = false;
- int j;
-
- /* Is the current bound distinct from the previous one? */
- for (j = 0; j < key->partnatts; j++)
- {
- Datum cmpval;
-
- if (prev == NULL || cur->kind[j] != prev->kind[j])
- {
- is_distinct = true;
- break;
- }
-
- /*
- * If the bounds are both MINVALUE or MAXVALUE, stop now
- * and treat them as equal, since any values after this
- * point must be ignored.
- */
- if (cur->kind[j] != PARTITION_RANGE_DATUM_VALUE)
- break;
-
- cmpval = FunctionCall2Coll(&key->partsupfunc[j],
- key->partcollation[j],
- cur->datums[j],
- prev->datums[j]);
- if (DatumGetInt32(cmpval) != 0)
- {
- is_distinct = true;
- break;
- }
- }
-
- /*
- * Only if the bound is distinct save it into a temporary
- * array i.e. rbounds which is later copied into boundinfo
- * datums array.
- */
- if (is_distinct)
- rbounds[k++] = all_bounds[i];
-
- prev = cur;
- }
-
- /* Update ndatums to hold the count of distinct datums. */
- ndatums = k;
- }
- else
- elog(ERROR, "unexpected partition strategy: %d",
- (int) key->strategy);
- }
+ nparts = list_length(boundspecs);
/* Now build the actual relcache partition descriptor */
rel->rd_pdcxt = AllocSetContextCreate(CacheMemoryContext,
@@ -572,210 +317,41 @@ RelationBuildPartitionDesc(Relation rel)
MemoryContextCopyAndSetIdentifier(rel->rd_pdcxt, RelationGetRelationName(rel));
oldcxt = MemoryContextSwitchTo(rel->rd_pdcxt);
-
- result = (PartitionDescData *) palloc0(sizeof(PartitionDescData));
- result->nparts = nparts;
- if (nparts > 0)
- {
- PartitionBoundInfo boundinfo;
- int *mapping;
- int next_index = 0;
-
- result->oids = (Oid *) palloc0(nparts * sizeof(Oid));
-
- boundinfo = (PartitionBoundInfoData *)
- palloc0(sizeof(PartitionBoundInfoData));
- boundinfo->strategy = key->strategy;
- boundinfo->default_index = -1;
- boundinfo->ndatums = ndatums;
- boundinfo->null_index = -1;
- boundinfo->datums = (Datum **) palloc0(ndatums * sizeof(Datum *));
-
- /* Initialize mapping array with invalid values */
- mapping = (int *) palloc(sizeof(int) * nparts);
- for (i = 0; i < nparts; i++)
- mapping[i] = -1;
-
- switch (key->strategy)
- {
- case PARTITION_STRATEGY_HASH:
- {
- /* Moduli are stored in ascending order */
- int greatest_modulus = hbounds[ndatums - 1]->modulus;
-
- boundinfo->indexes = (int *) palloc(greatest_modulus *
- sizeof(int));
-
- for (i = 0; i < greatest_modulus; i++)
- boundinfo->indexes[i] = -1;
-
- for (i = 0; i < nparts; i++)
- {
- int modulus = hbounds[i]->modulus;
- int remainder = hbounds[i]->remainder;
-
- boundinfo->datums[i] = (Datum *) palloc(2 *
- sizeof(Datum));
- boundinfo->datums[i][0] = Int32GetDatum(modulus);
- boundinfo->datums[i][1] = Int32GetDatum(remainder);
-
- while (remainder < greatest_modulus)
- {
- /* overlap? */
- Assert(boundinfo->indexes[remainder] == -1);
- boundinfo->indexes[remainder] = i;
- remainder += modulus;
- }
-
- mapping[hbounds[i]->index] = i;
- pfree(hbounds[i]);
- }
- pfree(hbounds);
- break;
- }
-
- case PARTITION_STRATEGY_LIST:
- {
- boundinfo->indexes = (int *) palloc(ndatums * sizeof(int));
-
- /*
- * Copy values. Indexes of individual values are mapped
- * to canonical values so that they match for any two list
- * partitioned tables with same number of partitions and
- * same lists per partition. One way to canonicalize is
- * to assign the index in all_values[] of the smallest
- * value of each partition, as the index of all of the
- * partition's values.
- */
- for (i = 0; i < ndatums; i++)
- {
- boundinfo->datums[i] = (Datum *) palloc(sizeof(Datum));
- boundinfo->datums[i][0] = datumCopy(all_values[i]->value,
- key->parttypbyval[0],
- key->parttyplen[0]);
-
- /* If the old index has no mapping, assign one */
- if (mapping[all_values[i]->index] == -1)
- mapping[all_values[i]->index] = next_index++;
-
- boundinfo->indexes[i] = mapping[all_values[i]->index];
- }
-
- /*
- * If null-accepting partition has no mapped index yet,
- * assign one. This could happen if such partition
- * accepts only null and hence not covered in the above
- * loop which only handled non-null values.
- */
- if (null_index != -1)
- {
- Assert(null_index >= 0);
- if (mapping[null_index] == -1)
- mapping[null_index] = next_index++;
- boundinfo->null_index = mapping[null_index];
- }
-
- /* Assign mapped index for the default partition. */
- if (default_index != -1)
- {
- /*
- * The default partition accepts any value not
- * specified in the lists of other partitions, hence
- * it should not get mapped index while assigning
- * those for non-null datums.
- */
- Assert(default_index >= 0 &&
- mapping[default_index] == -1);
- mapping[default_index] = next_index++;
- boundinfo->default_index = mapping[default_index];
- }
-
- /* All partition must now have a valid mapping */
- Assert(next_index == nparts);
- break;
- }
-
- case PARTITION_STRATEGY_RANGE:
- {
- boundinfo->kind = (PartitionRangeDatumKind **)
- palloc(ndatums *
- sizeof(PartitionRangeDatumKind *));
- boundinfo->indexes = (int *) palloc((ndatums + 1) *
- sizeof(int));
-
- for (i = 0; i < ndatums; i++)
- {
- int j;
-
- boundinfo->datums[i] = (Datum *) palloc(key->partnatts *
- sizeof(Datum));
- boundinfo->kind[i] = (PartitionRangeDatumKind *)
- palloc(key->partnatts *
- sizeof(PartitionRangeDatumKind));
- for (j = 0; j < key->partnatts; j++)
- {
- if (rbounds[i]->kind[j] == PARTITION_RANGE_DATUM_VALUE)
- boundinfo->datums[i][j] =
- datumCopy(rbounds[i]->datums[j],
- key->parttypbyval[j],
- key->parttyplen[j]);
- boundinfo->kind[i][j] = rbounds[i]->kind[j];
- }
-
- /*
- * There is no mapping for invalid indexes.
- *
- * Any lower bounds in the rbounds array have invalid
- * indexes assigned, because the values between the
- * previous bound (if there is one) and this (lower)
- * bound are not part of the range of any existing
- * partition.
- */
- if (rbounds[i]->lower)
- boundinfo->indexes[i] = -1;
- else
- {
- int orig_index = rbounds[i]->index;
-
- /* If the old index has no mapping, assign one */
- if (mapping[orig_index] == -1)
- mapping[orig_index] = next_index++;
-
- boundinfo->indexes[i] = mapping[orig_index];
- }
- }
-
- /* Assign mapped index for the default partition. */
- if (default_index != -1)
- {
- Assert(default_index >= 0 && mapping[default_index] == -1);
- mapping[default_index] = next_index++;
- boundinfo->default_index = mapping[default_index];
- }
- boundinfo->indexes[i] = -1;
- break;
- }
-
- default:
- elog(ERROR, "unexpected partition strategy: %d",
- (int) key->strategy);
- }
-
- result->boundinfo = boundinfo;
-
- /*
- * Now assign OIDs from the original array into mapped indexes of the
- * result array. Order of OIDs in the former is defined by the
- * catalog scan that retrieved them, whereas that in the latter is
- * defined by canonicalized representation of the partition bounds.
- */
- for (i = 0; i < nparts; i++)
- result->oids[mapping[i]] = oids[i];
- pfree(mapping);
- }
+ partdesc = (PartitionDescData *) palloc0(sizeof(PartitionDescData));
+ partdesc->nparts = nparts;
+ /* oids and boundinfo are allocated below. */
MemoryContextSwitchTo(oldcxt);
- rel->rd_partdesc = result;
+
+ if (nparts == 0)
+ {
+ rel->rd_partdesc = partdesc;
+ return;
+ }
+
+ /* First create the PartitionBoundInfo in caller's context. */
+ boundinfo = partition_bounds_create(boundspecs, key, &mapping);
+ oids_orig = (Oid *) palloc(sizeof(Oid) * partdesc->nparts);
+ i = 0;
+ foreach(cell, inhoids)
+ oids_orig[i++] = lfirst_oid(cell);
+
+ /* Now copy boundinfo and oids into partdesc. */
+ oldcxt = MemoryContextSwitchTo(rel->rd_pdcxt);
+ partdesc->boundinfo = partition_bounds_copy(boundinfo, key);
+ partdesc->oids = (Oid *) palloc(partdesc->nparts * sizeof(Oid));
+
+ /*
+ * Now assign OIDs from the original array into mapped indexes of the
+ * result array. Order of OIDs in the former is defined by the catalog
+ * scan that retrieved them, whereas that in the latter is defined by
+ * canonicalized representation of the partition bounds.
+ */
+ for (i = 0; i < partdesc->nparts; i++)
+ partdesc->oids[mapping[i]] = oids_orig[i];
+ MemoryContextSwitchTo(oldcxt);
+
+ rel->rd_partdesc = partdesc;
}
/*
@@ -917,48 +493,3 @@ generate_partition_qual(Relation rel)
return result;
}
-
-/*
- * qsort_partition_hbound_cmp
- *
- * We sort hash bounds by modulus, then by remainder.
- */
-static int32
-qsort_partition_hbound_cmp(const void *a, const void *b)
-{
- PartitionHashBound *h1 = (*(PartitionHashBound *const *) a);
- PartitionHashBound *h2 = (*(PartitionHashBound *const *) b);
-
- return partition_hbound_cmp(h1->modulus, h1->remainder,
- h2->modulus, h2->remainder);
-}
-
-/*
- * qsort_partition_list_value_cmp
- *
- * Compare two list partition bound datums
- */
-static int32
-qsort_partition_list_value_cmp(const void *a, const void *b, void *arg)
-{
- Datum val1 = (*(const PartitionListValue **) a)->value,
- val2 = (*(const PartitionListValue **) b)->value;
- PartitionKey key = (PartitionKey) arg;
-
- return DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[0],
- key->partcollation[0],
- val1, val2));
-}
-
-/* Used when sorting range bounds across all range partitions */
-static int32
-qsort_partition_rbound_cmp(const void *a, const void *b, void *arg)
-{
- PartitionRangeBound *b1 = (*(PartitionRangeBound *const *) a);
- PartitionRangeBound *b2 = (*(PartitionRangeBound *const *) b);
- PartitionKey key = (PartitionKey) arg;
-
- return partition_rbound_cmp(key->partnatts, key->partsupfunc,
- key->partcollation, b1->datums, b1->kind,
- b1->lower, b2);
-}
diff --git a/src/include/partitioning/partbounds.h b/src/include/partitioning/partbounds.h
index c7535e32fc..7a697d1c0a 100644
--- a/src/include/partitioning/partbounds.h
+++ b/src/include/partitioning/partbounds.h
@@ -75,40 +75,14 @@ typedef struct PartitionBoundInfoData
#define partition_bound_accepts_nulls(bi) ((bi)->null_index != -1)
#define partition_bound_has_default(bi) ((bi)->default_index != -1)
-/*
- * When qsort'ing partition bounds after reading from the catalog, each bound
- * is represented with one of the following structs.
- */
-
-/* One bound of a hash partition */
-typedef struct PartitionHashBound
-{
- int modulus;
- int remainder;
- int index;
-} PartitionHashBound;
-
-/* One value coming from some (index'th) list partition */
-typedef struct PartitionListValue
-{
- int index;
- Datum value;
-} PartitionListValue;
-
-/* One bound of a range partition */
-typedef struct PartitionRangeBound
-{
- int index;
- Datum *datums; /* range bound datums */
- PartitionRangeDatumKind *kind; /* the kind of each datum */
- bool lower; /* this is the lower (vs upper) bound */
-} PartitionRangeBound;
-
extern int get_hash_partition_greatest_modulus(PartitionBoundInfo b);
extern uint64 compute_partition_hash_value(int partnatts, FmgrInfo *partsupfunc,
Datum *values, bool *isnull);
extern List *get_qual_from_partbound(Relation rel, Relation parent,
PartitionBoundSpec *spec);
+extern PartitionBoundInfo partition_bounds_create(List *boundspecs,
+ PartitionKey key,
+ int **mapping);
extern bool partition_bounds_equal(int partnatts, int16 *parttyplen,
bool *parttypbyval, PartitionBoundInfo b1,
PartitionBoundInfo b2);
@@ -120,14 +94,6 @@ extern void check_default_partition_contents(Relation parent,
Relation defaultRel,
PartitionBoundSpec *new_spec);
-extern PartitionRangeBound *make_one_partition_rbound(PartitionKey key, int index,
- List *datums, bool lower);
-extern int32 partition_hbound_cmp(int modulus1, int remainder1, int modulus2,
- int remainder2);
-extern int32 partition_rbound_cmp(int partnatts, FmgrInfo *partsupfunc,
- Oid *partcollation, Datum *datums1,
- PartitionRangeDatumKind *kind1, bool lower1,
- PartitionRangeBound *b2);
extern int32 partition_rbound_datum_cmp(FmgrInfo *partsupfunc,
Oid *partcollation,
Datum *rb_datums, PartitionRangeDatumKind *rb_kind,
@@ -136,10 +102,6 @@ extern int partition_list_bsearch(FmgrInfo *partsupfunc,
Oid *partcollation,
PartitionBoundInfo boundinfo,
Datum value, bool *is_equal);
-extern int partition_range_bsearch(int partnatts, FmgrInfo *partsupfunc,
- Oid *partcollation,
- PartitionBoundInfo boundinfo,
- PartitionRangeBound *probe, bool *is_equal);
extern int partition_range_datum_bsearch(FmgrInfo *partsupfunc,
Oid *partcollation,
PartitionBoundInfo boundinfo,
[application/pgp-signature] signature.asc (833B, ../../[email protected]/3-signature.asc)
download
^ permalink raw reply [nested|flat] 55+ messages in thread
* Re: move PartitionBoundInfo creation code
2018-11-01 03:58 move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-01 04:02 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-01 04:03 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-05 07:21 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 06:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-07 07:37 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 15:04 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
2018-11-08 02:48 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-08 03:59 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-08 06:11 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-12 13:55 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
@ 2018-11-13 01:34 ` Amit Langote <[email protected]>
2018-11-13 02:34 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
0 siblings, 1 reply; 55+ messages in thread
From: Amit Langote @ 2018-11-13 01:34 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; pgsql-hackers
Hi,
On 2018/11/12 22:55, Michael Paquier wrote:
> On Thu, Nov 08, 2018 at 03:11:35PM +0900, Amit Langote wrote:
>> And here is the new version. The break down into smaller local
>> functions for different partitioning methods is in patch 0002.
>
> Okay, here we go.
Thank you for reviewing.
> I have spent a couple of hours on this patch,
> checking the consistency of the new code and the new code, and the main
> complain I have about the last version is that the code in charge of
> building the mapping was made less robust.
> So my notes are:
> - The initialization of the mapping should happen in
> partition_bounds_create() before going through each routine. The
> proposed patch was only doing the initialization for list bounds, using
> an extra layer to track the canonical ordering of indexes associated
> with a partition (listpart_canon_idx), and that's something that the
> mapping can perfectly do, keeping the code more straight-forward (at
> least it seems so to me) with the way null_index and default_index are
> assigned.
I looked at this and the new implementation seems safe and is a bit faster
due to removing listpart_canon_idx stuff.
> - Some noise diffs were present, sometimes the indentation being wrong.
> - Asserts could be used instead of elog(ERROR) if the strategy is not
> the expected one. That looks cleaner to me.
Hmm, Assert or elog is your call, but I'd learned that we prefer elog when
checking a value read from the disk?
> The result is perhaps less smart than what you did, but that's more
> robust. It may make sense to change the way the mapping is built but
> I would suggest to do so after the actual refactoring. Attached is what
> I finish with, and I have spent quite some time making sure that
> all the new logic remains consistent with the previous one.
Thank you, the new logic seems sane to me.
I noticed a few things while going over the new patch.
I'd accidentally ended up changing the mapping elements to mean that they
map canonical indexes of partitions to their original indexes, but your
rewrite has restored the original meaning, so the following comment that I
wrote is obsolete:
+ * Upon return from this function, *mapping is set to an array of
+ * list_length(boundspecs) elements, each of which maps the canonical
+ * index of a given partition to its 0-based position in the original list.
It should be: "each of which maps the original index of a partition to its
canonical index."
+ /*
+ * For each partitioning method, we first convert the partition bounds
+ * from their parser node representation to the internal representation,
+ * along with any additional preprocessing (such performing de-duplication
+ * on range bounds). For each bound, we remember its partition's position
+ * (0-based) in the original list, so that we can add it to the *mapping
+ * array.
+ *
+ * Resulting bound datums are then added to the 'datums' array in
+ * PartitionBoundInfo. For each datum added, an integer indicating the
+ * canonical partition index is added to the 'indexes' array.
+ */
Maybe we can slightly rearrange this comment (along with fixing typos and
obsolete facts) as:
/*
* For each partitioning method, we first convert the partition bounds f
* from their parser node representation to the internal representation,
* along with any additional preprocessing (such as de-duplicating range
* bounds). Resulting bound datums are then added to the 'datums' array
* in PartitionBoundInfo. For each datum added, an integer indicating the
* canonical partition index is added to the 'indexes' array.
*
* For each bound, we remember its partition's position (0-based) in the
* original list to later map it to the canonical index.
*/
I'd proposed to rewrite the following comments in create_list_bounds, but
maybe you thought it's related to listmap_canon_idx stuff which no longer
exists.
+ /*
+ * Copy values. Indexes of individual values are mapped to canonical
+ * values so that they match for any two list partitioned tables with same
+ * number of partitions and same lists per partition. One way to
+ * canonicalize is to assign the index in all_values[] of the smallest
+ * value of each partition, as the index of all of the partition's values.
+ */
Here's what I'd proposed (fixing typos and removing the sentence that
mentioned listmap_canon_idx).
/*
* Copy values. Canonical indexes are values ranging from 0 to nparts - 1
* assigned to each partition such that all datums of a given partition
* receive the same value. The value for a given partition is the index
* of that partition's smallest datum in the all_values[] array.
*/
Similarly, for:
+ /*
+ * If null-accepting partition has no mapped index yet, assign one. This
+ * could happen if such partition accepts only null and hence not covered
+ * in the above loop which only handled non-null values.
+ */
I'd proposed:
/*
* Set the canonical value for null_index, if any.
*
* It's possible that the null-accepting partition has not been
* assigned an index yet, which could happen if such partition
* accepts only null and hence not handled in the above loop
* which only looked at non-null values.
*/
For:
+ /* Assign mapped index for the default partition. */
I'd proposed:
/* Set the canonical value for default_index, if any. */
Thanks,
Amit
^ permalink raw reply [nested|flat] 55+ messages in thread
* Re: move PartitionBoundInfo creation code
2018-11-01 03:58 move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-01 04:02 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-01 04:03 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-05 07:21 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 06:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-07 07:37 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 15:04 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
2018-11-08 02:48 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-08 03:59 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-08 06:11 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-12 13:55 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-13 01:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
@ 2018-11-13 02:34 ` Michael Paquier <[email protected]>
2018-11-13 03:40 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-13 12:58 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
0 siblings, 2 replies; 55+ messages in thread
From: Michael Paquier @ 2018-11-13 02:34 UTC (permalink / raw)
To: Amit Langote <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; pgsql-hackers
On Tue, Nov 13, 2018 at 10:34:50AM +0900, Amit Langote wrote:
> Hmm, Assert or elog is your call, but I'd learned that we prefer elog when
> checking a value read from the disk?
Let's keep elog() then, which is consistent with the previous code. If
there is any meaning in switching to an assert(), it could always be
changed later, if that proves necessary.
> Thank you, the new logic seems sane to me.
>
> I noticed a few things while going over the new patch.
Thanks for going through it.
> I'd accidentally ended up changing the mapping elements to mean that they
> map canonical indexes of partitions to their original indexes, but your
> rewrite has restored the original meaning, so the following comment that I
> wrote is obsolete:
>
> + * Upon return from this function, *mapping is set to an array of
> + * list_length(boundspecs) elements, each of which maps the canonical
> + * index of a given partition to its 0-based position in the original list.
>
> It should be: "each of which maps the original index of a partition to its
> canonical index."
Indeed. Good point. I have missed that bit.
> Maybe we can slightly rearrange this comment (along with fixing typos and
> obsolete facts) as:
>
> /*
> * For each partitioning method, we first convert the partition bounds f
> * from their parser node representation to the internal representation,
> * along with any additional preprocessing (such as de-duplicating range
> * bounds). Resulting bound datums are then added to the 'datums' array
> * in PartitionBoundInfo. For each datum added, an integer indicating the
> * canonical partition index is added to the 'indexes' array.
> *
> * For each bound, we remember its partition's position (0-based) in the
> * original list to later map it to the canonical index.
> */
OK, that looks cleaner.
> I'd proposed:
>
> /*
> * Set the canonical value for null_index, if any.
> *
> * It's possible that the null-accepting partition has not been
> * assigned an index yet, which could happen if such partition
> * accepts only null and hence not handled in the above loop
> * which only looked at non-null values.
> */
No issues with that. So canonical index is used as a more common term,
which seems fine.
> I'd proposed:
>
> /* Set the canonical value for default_index, if any. */
And this one either, using a conditional at the end is more adapted
anyway. And there are two occurences of that.
Attached is an updated patch. Perhaps you are spotting something else?
--
Michael
Attachments:
[text/x-diff] partition-bound-refactor-v2.patch (40.4K, ../../[email protected]/2-partition-bound-refactor-v2.patch)
download | inline diff:
diff --git a/src/backend/partitioning/partbounds.c b/src/backend/partitioning/partbounds.c
index c94f73aadc..70cee6862b 100644
--- a/src/backend/partitioning/partbounds.c
+++ b/src/backend/partitioning/partbounds.c
@@ -36,6 +36,61 @@
#include "utils/ruleutils.h"
#include "utils/syscache.h"
+/*
+ * When qsort'ing partition bounds after reading from the catalog, each bound
+ * is represented with one of the following structs.
+ */
+
+/* One bound of a hash partition */
+typedef struct PartitionHashBound
+{
+ int modulus;
+ int remainder;
+ int index;
+} PartitionHashBound;
+
+/* One value coming from some (index'th) list partition */
+typedef struct PartitionListValue
+{
+ int index;
+ Datum value;
+} PartitionListValue;
+
+/* One bound of a range partition */
+typedef struct PartitionRangeBound
+{
+ int index;
+ Datum *datums; /* range bound datums */
+ PartitionRangeDatumKind *kind; /* the kind of each datum */
+ bool lower; /* this is the lower (vs upper) bound */
+} PartitionRangeBound;
+
+static int32 qsort_partition_hbound_cmp(const void *a, const void *b);
+static int32 qsort_partition_list_value_cmp(const void *a, const void *b,
+ void *arg);
+static int32 qsort_partition_rbound_cmp(const void *a, const void *b,
+ void *arg);
+static PartitionBoundInfo create_hash_bounds(List *boundspecs,
+ PartitionKey key,
+ int **mapping);
+static PartitionBoundInfo create_list_bounds(List *boundspecs,
+ PartitionKey key,
+ int **mapping);
+static PartitionBoundInfo create_range_bounds(List *boundspecs,
+ PartitionKey key,
+ int **mapping);
+static PartitionRangeBound *make_one_partition_rbound(PartitionKey key, int index,
+ List *datums, bool lower);
+static int32 partition_hbound_cmp(int modulus1, int remainder1, int modulus2,
+ int remainder2);
+static int32 partition_rbound_cmp(int partnatts, FmgrInfo *partsupfunc,
+ Oid *partcollation, Datum *datums1,
+ PartitionRangeDatumKind *kind1, bool lower1,
+ PartitionRangeBound *b2);
+static int partition_range_bsearch(int partnatts, FmgrInfo *partsupfunc,
+ Oid *partcollation,
+ PartitionBoundInfo boundinfo,
+ PartitionRangeBound *probe, bool *is_equal);
static int get_partition_bound_num_indexes(PartitionBoundInfo b);
static Expr *make_partition_op_expr(PartitionKey key, int keynum,
uint16 strategy, Expr *arg1, Expr *arg2);
@@ -92,6 +147,522 @@ get_qual_from_partbound(Relation rel, Relation parent,
return my_qual;
}
+/*
+ * partition_bounds_create
+ * Build a PartitionBoundInfo struct from a list of PartitionBoundSpec
+ * nodes
+ *
+ * This function creates a PartitionBoundInfo and fills the values of its
+ * various members based on the input list. Importantly, 'datums' array will
+ * contain Datum representation of individual bounds (possibly after
+ * de-duplication as in case of range bounds), sorted in a canonical order
+ * defined by qsort_partition_* functions of respective partitioning methods.
+ * 'indexes' array will contain as many elements as there are bounds (specific
+ * exceptions to this rule are listed in the function body), which represent
+ * the 0-based canonical positions of partitions.
+ *
+ * Upon return from this function, *mapping is set to an array of
+ * list_length(boundspecs) elements, each of which maps the original index of
+ * a partition to its canonical index.
+ *
+ * Note: All the memory allocated by this function, including that of the
+ * the returned PartitionBoundInfo and its members is allocated in the context
+ * that was active when the function was called.
+ */
+PartitionBoundInfo
+partition_bounds_create(List *boundspecs, PartitionKey key, int **mapping)
+{
+ int nparts = list_length(boundspecs);
+ int i;
+
+ Assert(nparts > 0);
+
+ /*
+ * For each partitioning method, we first convert the partition bounds
+ * from their parser node representation to the internal representation,
+ * along with any additional preprocessing (such as de-duplicating range
+ * bounds). Resulting bound datums are then added to the 'datums' array
+ * in PartitionBoundInfo. For each datum added, an integer indicating the
+ * canonical partition index is added to the 'indexes' array.
+ *
+ * For each bound, we remember its partition's position (0-based) in the
+ * original list to later map it to the canonical index.
+ */
+
+ /*
+ * Initialize mapping array with invalid values, this is filled within
+ * each sub-routine below depending on the bound type.
+ */
+ *mapping = (int *) palloc(sizeof(int) * nparts);
+ for (i = 0; i < nparts; i++)
+ (*mapping)[i] = -1;
+
+ switch (key->strategy)
+ {
+ case PARTITION_STRATEGY_HASH:
+ return create_hash_bounds(boundspecs, key, mapping);
+
+ case PARTITION_STRATEGY_LIST:
+ return create_list_bounds(boundspecs, key, mapping);
+
+ case PARTITION_STRATEGY_RANGE:
+ return create_range_bounds(boundspecs, key, mapping);
+
+ default:
+ elog(ERROR, "unexpected partition strategy: %d",
+ (int) key->strategy);
+ break;
+ }
+
+ Assert(false);
+ return NULL; /* keep compiler quiet */
+}
+
+/*
+ * create_hash_bounds
+ * Create a PartitionBoundInfo for a hash partitioned table
+ */
+static PartitionBoundInfo
+create_hash_bounds(List *boundspecs, PartitionKey key, int **mapping)
+{
+ PartitionBoundInfo boundinfo;
+ PartitionHashBound **hbounds = NULL;
+ ListCell *cell;
+ int i,
+ nparts = list_length(boundspecs);
+ int ndatums = 0;
+ int greatest_modulus;
+
+ boundinfo = (PartitionBoundInfoData *)
+ palloc0(sizeof(PartitionBoundInfoData));
+ boundinfo->strategy = key->strategy;
+ /* No special hash partitions. */
+ boundinfo->null_index = -1;
+ boundinfo->default_index = -1;
+
+ ndatums = nparts;
+ hbounds = (PartitionHashBound **)
+ palloc(nparts * sizeof(PartitionHashBound *));
+
+ /* Convert from node to the internal representation */
+ i = 0;
+ foreach(cell, boundspecs)
+ {
+ PartitionBoundSpec *spec = castNode(PartitionBoundSpec, lfirst(cell));
+
+ if (spec->strategy != PARTITION_STRATEGY_HASH)
+ elog(ERROR, "invalid strategy in partition bound spec");
+
+ hbounds[i] = (PartitionHashBound *) palloc(sizeof(PartitionHashBound));
+ hbounds[i]->modulus = spec->modulus;
+ hbounds[i]->remainder = spec->remainder;
+ hbounds[i]->index = i;
+ i++;
+ }
+
+ /* Sort all the bounds in ascending order */
+ qsort(hbounds, nparts, sizeof(PartitionHashBound *),
+ qsort_partition_hbound_cmp);
+
+ /* After sorting, moduli are now stored in ascending order. */
+ greatest_modulus = hbounds[ndatums - 1]->modulus;
+
+ boundinfo->ndatums = ndatums;
+ boundinfo->datums = (Datum **) palloc0(ndatums * sizeof(Datum *));
+ boundinfo->indexes = (int *) palloc(greatest_modulus * sizeof(int));
+ for (i = 0; i < greatest_modulus; i++)
+ boundinfo->indexes[i] = -1;
+
+ /*
+ * For hash partitioning, there are as many datums (modulus and remainder
+ * pairs) as there are partitions. Indexes are simply values ranging from
+ * 0 to (nparts - 1).
+ */
+ for (i = 0; i < nparts; i++)
+ {
+ int modulus = hbounds[i]->modulus;
+ int remainder = hbounds[i]->remainder;
+
+ boundinfo->datums[i] = (Datum *) palloc(2 * sizeof(Datum));
+ boundinfo->datums[i][0] = Int32GetDatum(modulus);
+ boundinfo->datums[i][1] = Int32GetDatum(remainder);
+
+ while (remainder < greatest_modulus)
+ {
+ /* overlap? */
+ Assert(boundinfo->indexes[remainder] == -1);
+ boundinfo->indexes[remainder] = i;
+ remainder += modulus;
+ }
+
+ (*mapping)[hbounds[i]->index] = i;
+ pfree(hbounds[i]);
+ }
+ pfree(hbounds);
+
+ return boundinfo;
+}
+
+/*
+ * create_list_bounds
+ * Create a PartitionBoundInfo for a list partitioned table
+ */
+static PartitionBoundInfo
+create_list_bounds(List *boundspecs, PartitionKey key, int **mapping)
+{
+ PartitionBoundInfo boundinfo;
+ PartitionListValue **all_values = NULL;
+ ListCell *cell;
+ int i = 0;
+ int ndatums = 0;
+ int next_index = 0;
+ int default_index = -1;
+ int null_index = -1;
+ List *non_null_values = NIL;
+
+ boundinfo = (PartitionBoundInfoData *)
+ palloc0(sizeof(PartitionBoundInfoData));
+ boundinfo->strategy = key->strategy;
+ /* Will be set correctly below. */
+ boundinfo->null_index = -1;
+ boundinfo->default_index = -1;
+
+ /* Create a unified list of non-null values across all partitions. */
+ foreach(cell, boundspecs)
+ {
+ PartitionBoundSpec *spec = castNode(PartitionBoundSpec, lfirst(cell));
+ ListCell *c;
+
+ if (spec->strategy != PARTITION_STRATEGY_LIST)
+ elog(ERROR, "invalid strategy in partition bound spec");
+
+ /*
+ * Note the index of the partition bound spec for the default
+ * partition. There's no datum to add to the list on non-null datums
+ * for this partition.
+ */
+ if (spec->is_default)
+ {
+ default_index = i;
+ i++;
+ continue;
+ }
+
+ foreach(c, spec->listdatums)
+ {
+ Const *val = castNode(Const, lfirst(c));
+ PartitionListValue *list_value = NULL;
+
+ if (!val->constisnull)
+ {
+ list_value = (PartitionListValue *)
+ palloc0(sizeof(PartitionListValue));
+ list_value->index = i;
+ list_value->value = val->constvalue;
+ }
+ else
+ {
+ /*
+ * Never put a null into the values array, flag instead for
+ * the code further down below where we construct the actual
+ * relcache struct.
+ */
+ if (null_index != -1)
+ elog(ERROR, "found null more than once");
+ null_index = i;
+ }
+
+ if (list_value)
+ non_null_values = lappend(non_null_values, list_value);
+ }
+
+ i++;
+ }
+
+ ndatums = list_length(non_null_values);
+
+ /*
+ * Collect all list values in one array. Alongside the value, we also save
+ * the index of partition the value comes from.
+ */
+ all_values = (PartitionListValue **)
+ palloc(ndatums * sizeof(PartitionListValue *));
+ i = 0;
+ foreach(cell, non_null_values)
+ {
+ PartitionListValue *src = lfirst(cell);
+
+ all_values[i] = (PartitionListValue *)
+ palloc(sizeof(PartitionListValue));
+ all_values[i]->value = src->value;
+ all_values[i]->index = src->index;
+ i++;
+ }
+
+ qsort_arg(all_values, ndatums, sizeof(PartitionListValue *),
+ qsort_partition_list_value_cmp, (void *) key);
+
+ boundinfo->ndatums = ndatums;
+ boundinfo->datums = (Datum **) palloc0(ndatums * sizeof(Datum *));
+ boundinfo->indexes = (int *) palloc(ndatums * sizeof(int));
+
+ /*
+ * Copy values. Canonical indexes are values ranging from 0 to (nparts -
+ * 1) assigned to each partition such that all datums of a given partition
+ * receive the same value. The value for a given partition is the index of
+ * that partition's smallest datum in the all_values[] array.
+ */
+ for (i = 0; i < ndatums; i++)
+ {
+ int orig_index = all_values[i]->index;
+
+ boundinfo->datums[i] = (Datum *) palloc(sizeof(Datum));
+ boundinfo->datums[i][0] = datumCopy(all_values[i]->value,
+ key->parttypbyval[0],
+ key->parttyplen[0]);
+
+ /* If the old index has no mapping, assign one */
+ if ((*mapping)[orig_index] == -1)
+ (*mapping)[orig_index] = next_index++;
+
+ boundinfo->indexes[i] = (*mapping)[orig_index];
+ }
+
+ /*
+ * Set the canonical value for null_index, if any.
+ *
+ * It is possible that the null-accepting partition has not been assigned
+ * an index yet, which could happen if such partition accepts only null
+ * and hence not handled in the above loop which only looked at non-null
+ * values.
+ */
+ if (null_index != -1)
+ {
+ Assert(null_index >= 0);
+ if ((*mapping)[null_index] == -1)
+ (*mapping)[null_index] = next_index++;
+ boundinfo->null_index = (*mapping)[null_index];
+ }
+
+ /* Set the canonical value for default_index, if any. */
+ if (default_index != -1)
+ {
+ /*
+ * The default partition accepts any value not specified in the lists
+ * of other partitions, hence it should not get mapped index while
+ * assigning those for non-null datums.
+ */
+ Assert(default_index >= 0);
+ Assert((*mapping)[default_index] == -1);
+ (*mapping)[default_index] = next_index++;
+ boundinfo->default_index = (*mapping)[default_index];
+ }
+
+ /* All partition must now have been assigned canonical indexes. */
+ Assert(next_index == list_length(boundspecs));
+ return boundinfo;
+}
+
+/*
+ * create_range_bounds
+ * Create a PartitionBoundInfo for a range partitioned table
+ */
+static PartitionBoundInfo
+create_range_bounds(List *boundspecs, PartitionKey key, int **mapping)
+{
+ PartitionBoundInfo boundinfo;
+ PartitionRangeBound **rbounds = NULL;
+ PartitionRangeBound **all_bounds,
+ *prev;
+ ListCell *cell;
+ int i,
+ k,
+ nparts = list_length(boundspecs);
+ int ndatums = 0;
+ int default_index = -1;
+ int next_index = 0;
+
+ boundinfo = (PartitionBoundInfoData *)
+ palloc0(sizeof(PartitionBoundInfoData));
+ boundinfo->strategy = key->strategy;
+ /* There is no special null-accepting range partition. */
+ boundinfo->null_index = -1;
+ /* Will be set correctly below. */
+ boundinfo->default_index = -1;
+
+ all_bounds = (PartitionRangeBound **)
+ palloc0(2 * nparts * sizeof(PartitionRangeBound *));
+
+ /* Create a unified list of range bounds across all the partitions. */
+ i = ndatums = 0;
+ foreach(cell, boundspecs)
+ {
+ PartitionBoundSpec *spec = castNode(PartitionBoundSpec, lfirst(cell));
+ PartitionRangeBound *lower,
+ *upper;
+
+ if (spec->strategy != PARTITION_STRATEGY_RANGE)
+ elog(ERROR, "invalid strategy in partition bound spec");
+
+ /*
+ * Note the index of the partition bound spec for the default
+ * partition. There's no datum to add to the all_bounds array for
+ * this partition.
+ */
+ if (spec->is_default)
+ {
+ default_index = i++;
+ continue;
+ }
+
+ lower = make_one_partition_rbound(key, i, spec->lowerdatums, true);
+ upper = make_one_partition_rbound(key, i, spec->upperdatums, false);
+ all_bounds[ndatums++] = lower;
+ all_bounds[ndatums++] = upper;
+ i++;
+ }
+
+ Assert(ndatums == nparts * 2 ||
+ (default_index != -1 && ndatums == (nparts - 1) * 2));
+
+ /* Sort all the bounds in ascending order */
+ qsort_arg(all_bounds, ndatums,
+ sizeof(PartitionRangeBound *),
+ qsort_partition_rbound_cmp,
+ (void *) key);
+
+ /* Save distinct bounds from all_bounds into rbounds. */
+ rbounds = (PartitionRangeBound **)
+ palloc(ndatums * sizeof(PartitionRangeBound *));
+ k = 0;
+ prev = NULL;
+ for (i = 0; i < ndatums; i++)
+ {
+ PartitionRangeBound *cur = all_bounds[i];
+ bool is_distinct = false;
+ int j;
+
+ /* Is the current bound distinct from the previous one? */
+ for (j = 0; j < key->partnatts; j++)
+ {
+ Datum cmpval;
+
+ if (prev == NULL || cur->kind[j] != prev->kind[j])
+ {
+ is_distinct = true;
+ break;
+ }
+
+ /*
+ * If the bounds are both MINVALUE or MAXVALUE, stop now and treat
+ * them as equal, since any values after this point must be
+ * ignored.
+ */
+ if (cur->kind[j] != PARTITION_RANGE_DATUM_VALUE)
+ break;
+
+ cmpval = FunctionCall2Coll(&key->partsupfunc[j],
+ key->partcollation[j],
+ cur->datums[j],
+ prev->datums[j]);
+ if (DatumGetInt32(cmpval) != 0)
+ {
+ is_distinct = true;
+ break;
+ }
+ }
+
+ /*
+ * Only if the bound is distinct save it into a temporary array, i.e,
+ * rbounds which is later copied into boundinfo datums array.
+ */
+ if (is_distinct)
+ rbounds[k++] = all_bounds[i];
+
+ prev = cur;
+ }
+
+ /* Update ndatums to hold the count of distinct datums. */
+ ndatums = k;
+
+ /*
+ * Add datums to boundinfo. Canonical indexes are values ranging from 0
+ * to nparts - 1, assigned in that order to each partition's upper bound.
+ * For 'datums' elements that are lower bounds, there is -1 in the
+ * 'indexes' array to signify that no partition exists for the values less
+ * than such a bound and greater than or equal to the previous upper
+ * bound.
+ */
+ boundinfo->ndatums = ndatums;
+ boundinfo->datums = (Datum **) palloc0(ndatums * sizeof(Datum *));
+ boundinfo->kind = (PartitionRangeDatumKind **)
+ palloc(ndatums *
+ sizeof(PartitionRangeDatumKind *));
+
+ /*
+ * For range partitioning, an additional value of -1 is stored as the last
+ * element.
+ */
+ boundinfo->indexes = (int *) palloc((ndatums + 1) * sizeof(int));
+
+ for (i = 0; i < ndatums; i++)
+ {
+ int j;
+
+ boundinfo->datums[i] = (Datum *) palloc(key->partnatts *
+ sizeof(Datum));
+ boundinfo->kind[i] = (PartitionRangeDatumKind *)
+ palloc(key->partnatts *
+ sizeof(PartitionRangeDatumKind));
+ for (j = 0; j < key->partnatts; j++)
+ {
+ if (rbounds[i]->kind[j] == PARTITION_RANGE_DATUM_VALUE)
+ boundinfo->datums[i][j] =
+ datumCopy(rbounds[i]->datums[j],
+ key->parttypbyval[j],
+ key->parttyplen[j]);
+ boundinfo->kind[i][j] = rbounds[i]->kind[j];
+ }
+
+ /*
+ * There is no mapping for invalid indexes.
+ *
+ * Any lower bounds in the rbounds array have invalid indexes
+ * assigned, because the values between the previous bound (if there
+ * is one) and this (lower) bound are not part of the range of any
+ * existing partition.
+ */
+ if (rbounds[i]->lower)
+ boundinfo->indexes[i] = -1;
+ else
+ {
+ int orig_index = rbounds[i]->index;
+
+ /* If the old index has no mapping, assign one */
+ if ((*mapping)[orig_index] == -1)
+ (*mapping)[orig_index] = next_index++;
+
+ boundinfo->indexes[i] = (*mapping)[orig_index];
+ }
+ }
+
+ /* Set the canonical value for default_index, if any. */
+ if (default_index != -1)
+ {
+ Assert(default_index >= 0 && (*mapping)[default_index] == -1);
+ (*mapping)[default_index] = next_index++;
+ boundinfo->default_index = (*mapping)[default_index];
+ }
+
+ /* The extra -1 element. */
+ Assert(i == ndatums);
+ boundinfo->indexes[i] = -1;
+
+ /* All partition must now have been assigned canonical indexes. */
+ Assert(next_index == nparts);
+ return boundinfo;
+}
+
/*
* Are two partition bound collections logically equal?
*
@@ -763,7 +1334,7 @@ get_hash_partition_greatest_modulus(PartitionBoundInfo bound)
* and a flag telling whether the bound is lower or not. Made into a function
* because there are multiple sites that want to use this facility.
*/
-PartitionRangeBound *
+static PartitionRangeBound *
make_one_partition_rbound(PartitionKey key, int index, List *datums, bool lower)
{
PartitionRangeBound *bound;
@@ -819,7 +1390,7 @@ make_one_partition_rbound(PartitionKey key, int index, List *datums, bool lower)
* structure, which only stores the upper bound of a common boundary between
* two contiguous partitions.
*/
-int32
+static int32
partition_rbound_cmp(int partnatts, FmgrInfo *partsupfunc,
Oid *partcollation,
Datum *datums1, PartitionRangeDatumKind *kind1,
@@ -914,7 +1485,7 @@ partition_rbound_datum_cmp(FmgrInfo *partsupfunc, Oid *partcollation,
*
* Compares modulus first, then remainder if modulus is equal.
*/
-int32
+static int32
partition_hbound_cmp(int modulus1, int remainder1, int modulus2, int remainder2)
{
if (modulus1 < modulus2)
@@ -977,7 +1548,7 @@ partition_list_bsearch(FmgrInfo *partsupfunc, Oid *partcollation,
* *is_equal is set to true if the range bound at the returned index is equal
* to the input range bound
*/
-int
+static int
partition_range_bsearch(int partnatts, FmgrInfo *partsupfunc,
Oid *partcollation,
PartitionBoundInfo boundinfo,
@@ -1101,6 +1672,55 @@ partition_hash_bsearch(PartitionBoundInfo boundinfo,
return lo;
}
+/*
+ * qsort_partition_hbound_cmp
+ *
+ * Hash bounds are sorted by modulus, then by remainder.
+ */
+static int32
+qsort_partition_hbound_cmp(const void *a, const void *b)
+{
+ PartitionHashBound *h1 = (*(PartitionHashBound *const *) a);
+ PartitionHashBound *h2 = (*(PartitionHashBound *const *) b);
+
+ return partition_hbound_cmp(h1->modulus, h1->remainder,
+ h2->modulus, h2->remainder);
+}
+
+/*
+ * qsort_partition_list_value_cmp
+ *
+ * Compare two list partition bound datums.
+ */
+static int32
+qsort_partition_list_value_cmp(const void *a, const void *b, void *arg)
+{
+ Datum val1 = (*(const PartitionListValue **) a)->value,
+ val2 = (*(const PartitionListValue **) b)->value;
+ PartitionKey key = (PartitionKey) arg;
+
+ return DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[0],
+ key->partcollation[0],
+ val1, val2));
+}
+
+/*
+ * qsort_partition_rbound_cmp
+ *
+ * Used when sorting range bounds across all range partitions.
+ */
+static int32
+qsort_partition_rbound_cmp(const void *a, const void *b, void *arg)
+{
+ PartitionRangeBound *b1 = (*(PartitionRangeBound *const *) a);
+ PartitionRangeBound *b2 = (*(PartitionRangeBound *const *) b);
+ PartitionKey key = (PartitionKey) arg;
+
+ return partition_rbound_cmp(key->partnatts, key->partsupfunc,
+ key->partcollation, b1->datums, b1->kind,
+ b1->lower, b2);
+}
+
/*
* get_partition_bound_num_indexes
*
diff --git a/src/backend/utils/cache/partcache.c b/src/backend/utils/cache/partcache.c
index 5757301d05..020bca1f54 100644
--- a/src/backend/utils/cache/partcache.c
+++ b/src/backend/utils/cache/partcache.c
@@ -38,12 +38,6 @@
static List *generate_partition_qual(Relation rel);
-static int32 qsort_partition_hbound_cmp(const void *a, const void *b);
-static int32 qsort_partition_list_value_cmp(const void *a, const void *b,
- void *arg);
-static int32 qsort_partition_rbound_cmp(const void *a, const void *b,
- void *arg);
-
/*
* RelationBuildPartitionKey
@@ -260,36 +254,22 @@ RelationBuildPartitionKey(Relation relation)
void
RelationBuildPartitionDesc(Relation rel)
{
- List *inhoids,
- *partoids;
- Oid *oids = NULL;
+ PartitionDesc partdesc;
+ PartitionBoundInfo boundinfo;
+ List *inhoids;
List *boundspecs = NIL;
ListCell *cell;
int i,
nparts;
PartitionKey key = RelationGetPartitionKey(rel);
- PartitionDesc result;
MemoryContext oldcxt;
-
- int ndatums = 0;
- int default_index = -1;
-
- /* Hash partitioning specific */
- PartitionHashBound **hbounds = NULL;
-
- /* List partitioning specific */
- PartitionListValue **all_values = NULL;
- int null_index = -1;
-
- /* Range partitioning specific */
- PartitionRangeBound **rbounds = NULL;
+ Oid *oids_orig;
+ int *mapping;
/* Get partition oids from pg_inherits */
inhoids = find_inheritance_children(RelationGetRelid(rel), NoLock);
/* Collect bound spec nodes in a list */
- i = 0;
- partoids = NIL;
foreach(cell, inhoids)
{
Oid inhrelid = lfirst_oid(cell);
@@ -325,245 +305,10 @@ RelationBuildPartitionDesc(Relation rel)
}
boundspecs = lappend(boundspecs, boundspec);
- partoids = lappend_oid(partoids, inhrelid);
ReleaseSysCache(tuple);
}
- nparts = list_length(partoids);
-
- if (nparts > 0)
- {
- oids = (Oid *) palloc(nparts * sizeof(Oid));
- i = 0;
- foreach(cell, partoids)
- oids[i++] = lfirst_oid(cell);
-
- /* Convert from node to the internal representation */
- if (key->strategy == PARTITION_STRATEGY_HASH)
- {
- ndatums = nparts;
- hbounds = (PartitionHashBound **)
- palloc(nparts * sizeof(PartitionHashBound *));
-
- i = 0;
- foreach(cell, boundspecs)
- {
- PartitionBoundSpec *spec = castNode(PartitionBoundSpec,
- lfirst(cell));
-
- if (spec->strategy != PARTITION_STRATEGY_HASH)
- elog(ERROR, "invalid strategy in partition bound spec");
-
- hbounds[i] = (PartitionHashBound *)
- palloc(sizeof(PartitionHashBound));
-
- hbounds[i]->modulus = spec->modulus;
- hbounds[i]->remainder = spec->remainder;
- hbounds[i]->index = i;
- i++;
- }
-
- /* Sort all the bounds in ascending order */
- qsort(hbounds, nparts, sizeof(PartitionHashBound *),
- qsort_partition_hbound_cmp);
- }
- else if (key->strategy == PARTITION_STRATEGY_LIST)
- {
- List *non_null_values = NIL;
-
- /*
- * Create a unified list of non-null values across all partitions.
- */
- i = 0;
- null_index = -1;
- foreach(cell, boundspecs)
- {
- PartitionBoundSpec *spec = castNode(PartitionBoundSpec,
- lfirst(cell));
- ListCell *c;
-
- if (spec->strategy != PARTITION_STRATEGY_LIST)
- elog(ERROR, "invalid strategy in partition bound spec");
-
- /*
- * Note the index of the partition bound spec for the default
- * partition. There's no datum to add to the list of non-null
- * datums for this partition.
- */
- if (spec->is_default)
- {
- default_index = i;
- i++;
- continue;
- }
-
- foreach(c, spec->listdatums)
- {
- Const *val = castNode(Const, lfirst(c));
- PartitionListValue *list_value = NULL;
-
- if (!val->constisnull)
- {
- list_value = (PartitionListValue *)
- palloc0(sizeof(PartitionListValue));
- list_value->index = i;
- list_value->value = val->constvalue;
- }
- else
- {
- /*
- * Never put a null into the values array, flag
- * instead for the code further down below where we
- * construct the actual relcache struct.
- */
- if (null_index != -1)
- elog(ERROR, "found null more than once");
- null_index = i;
- }
-
- if (list_value)
- non_null_values = lappend(non_null_values,
- list_value);
- }
-
- i++;
- }
-
- ndatums = list_length(non_null_values);
-
- /*
- * Collect all list values in one array. Alongside the value, we
- * also save the index of partition the value comes from.
- */
- all_values = (PartitionListValue **) palloc(ndatums *
- sizeof(PartitionListValue *));
- i = 0;
- foreach(cell, non_null_values)
- {
- PartitionListValue *src = lfirst(cell);
-
- all_values[i] = (PartitionListValue *)
- palloc(sizeof(PartitionListValue));
- all_values[i]->value = src->value;
- all_values[i]->index = src->index;
- i++;
- }
-
- qsort_arg(all_values, ndatums, sizeof(PartitionListValue *),
- qsort_partition_list_value_cmp, (void *) key);
- }
- else if (key->strategy == PARTITION_STRATEGY_RANGE)
- {
- int k;
- PartitionRangeBound **all_bounds,
- *prev;
-
- all_bounds = (PartitionRangeBound **) palloc0(2 * nparts *
- sizeof(PartitionRangeBound *));
-
- /*
- * Create a unified list of range bounds across all the
- * partitions.
- */
- i = ndatums = 0;
- foreach(cell, boundspecs)
- {
- PartitionBoundSpec *spec = castNode(PartitionBoundSpec,
- lfirst(cell));
- PartitionRangeBound *lower,
- *upper;
-
- if (spec->strategy != PARTITION_STRATEGY_RANGE)
- elog(ERROR, "invalid strategy in partition bound spec");
-
- /*
- * Note the index of the partition bound spec for the default
- * partition. There's no datum to add to the allbounds array
- * for this partition.
- */
- if (spec->is_default)
- {
- default_index = i++;
- continue;
- }
-
- lower = make_one_partition_rbound(key, i, spec->lowerdatums,
- true);
- upper = make_one_partition_rbound(key, i, spec->upperdatums,
- false);
- all_bounds[ndatums++] = lower;
- all_bounds[ndatums++] = upper;
- i++;
- }
-
- Assert(ndatums == nparts * 2 ||
- (default_index != -1 && ndatums == (nparts - 1) * 2));
-
- /* Sort all the bounds in ascending order */
- qsort_arg(all_bounds, ndatums,
- sizeof(PartitionRangeBound *),
- qsort_partition_rbound_cmp,
- (void *) key);
-
- /* Save distinct bounds from all_bounds into rbounds. */
- rbounds = (PartitionRangeBound **)
- palloc(ndatums * sizeof(PartitionRangeBound *));
- k = 0;
- prev = NULL;
- for (i = 0; i < ndatums; i++)
- {
- PartitionRangeBound *cur = all_bounds[i];
- bool is_distinct = false;
- int j;
-
- /* Is the current bound distinct from the previous one? */
- for (j = 0; j < key->partnatts; j++)
- {
- Datum cmpval;
-
- if (prev == NULL || cur->kind[j] != prev->kind[j])
- {
- is_distinct = true;
- break;
- }
-
- /*
- * If the bounds are both MINVALUE or MAXVALUE, stop now
- * and treat them as equal, since any values after this
- * point must be ignored.
- */
- if (cur->kind[j] != PARTITION_RANGE_DATUM_VALUE)
- break;
-
- cmpval = FunctionCall2Coll(&key->partsupfunc[j],
- key->partcollation[j],
- cur->datums[j],
- prev->datums[j]);
- if (DatumGetInt32(cmpval) != 0)
- {
- is_distinct = true;
- break;
- }
- }
-
- /*
- * Only if the bound is distinct save it into a temporary
- * array i.e. rbounds which is later copied into boundinfo
- * datums array.
- */
- if (is_distinct)
- rbounds[k++] = all_bounds[i];
-
- prev = cur;
- }
-
- /* Update ndatums to hold the count of distinct datums. */
- ndatums = k;
- }
- else
- elog(ERROR, "unexpected partition strategy: %d",
- (int) key->strategy);
- }
+ nparts = list_length(boundspecs);
/* Now build the actual relcache partition descriptor */
rel->rd_pdcxt = AllocSetContextCreate(CacheMemoryContext,
@@ -572,210 +317,41 @@ RelationBuildPartitionDesc(Relation rel)
MemoryContextCopyAndSetIdentifier(rel->rd_pdcxt, RelationGetRelationName(rel));
oldcxt = MemoryContextSwitchTo(rel->rd_pdcxt);
-
- result = (PartitionDescData *) palloc0(sizeof(PartitionDescData));
- result->nparts = nparts;
- if (nparts > 0)
- {
- PartitionBoundInfo boundinfo;
- int *mapping;
- int next_index = 0;
-
- result->oids = (Oid *) palloc0(nparts * sizeof(Oid));
-
- boundinfo = (PartitionBoundInfoData *)
- palloc0(sizeof(PartitionBoundInfoData));
- boundinfo->strategy = key->strategy;
- boundinfo->default_index = -1;
- boundinfo->ndatums = ndatums;
- boundinfo->null_index = -1;
- boundinfo->datums = (Datum **) palloc0(ndatums * sizeof(Datum *));
-
- /* Initialize mapping array with invalid values */
- mapping = (int *) palloc(sizeof(int) * nparts);
- for (i = 0; i < nparts; i++)
- mapping[i] = -1;
-
- switch (key->strategy)
- {
- case PARTITION_STRATEGY_HASH:
- {
- /* Moduli are stored in ascending order */
- int greatest_modulus = hbounds[ndatums - 1]->modulus;
-
- boundinfo->indexes = (int *) palloc(greatest_modulus *
- sizeof(int));
-
- for (i = 0; i < greatest_modulus; i++)
- boundinfo->indexes[i] = -1;
-
- for (i = 0; i < nparts; i++)
- {
- int modulus = hbounds[i]->modulus;
- int remainder = hbounds[i]->remainder;
-
- boundinfo->datums[i] = (Datum *) palloc(2 *
- sizeof(Datum));
- boundinfo->datums[i][0] = Int32GetDatum(modulus);
- boundinfo->datums[i][1] = Int32GetDatum(remainder);
-
- while (remainder < greatest_modulus)
- {
- /* overlap? */
- Assert(boundinfo->indexes[remainder] == -1);
- boundinfo->indexes[remainder] = i;
- remainder += modulus;
- }
-
- mapping[hbounds[i]->index] = i;
- pfree(hbounds[i]);
- }
- pfree(hbounds);
- break;
- }
-
- case PARTITION_STRATEGY_LIST:
- {
- boundinfo->indexes = (int *) palloc(ndatums * sizeof(int));
-
- /*
- * Copy values. Indexes of individual values are mapped
- * to canonical values so that they match for any two list
- * partitioned tables with same number of partitions and
- * same lists per partition. One way to canonicalize is
- * to assign the index in all_values[] of the smallest
- * value of each partition, as the index of all of the
- * partition's values.
- */
- for (i = 0; i < ndatums; i++)
- {
- boundinfo->datums[i] = (Datum *) palloc(sizeof(Datum));
- boundinfo->datums[i][0] = datumCopy(all_values[i]->value,
- key->parttypbyval[0],
- key->parttyplen[0]);
-
- /* If the old index has no mapping, assign one */
- if (mapping[all_values[i]->index] == -1)
- mapping[all_values[i]->index] = next_index++;
-
- boundinfo->indexes[i] = mapping[all_values[i]->index];
- }
-
- /*
- * If null-accepting partition has no mapped index yet,
- * assign one. This could happen if such partition
- * accepts only null and hence not covered in the above
- * loop which only handled non-null values.
- */
- if (null_index != -1)
- {
- Assert(null_index >= 0);
- if (mapping[null_index] == -1)
- mapping[null_index] = next_index++;
- boundinfo->null_index = mapping[null_index];
- }
-
- /* Assign mapped index for the default partition. */
- if (default_index != -1)
- {
- /*
- * The default partition accepts any value not
- * specified in the lists of other partitions, hence
- * it should not get mapped index while assigning
- * those for non-null datums.
- */
- Assert(default_index >= 0 &&
- mapping[default_index] == -1);
- mapping[default_index] = next_index++;
- boundinfo->default_index = mapping[default_index];
- }
-
- /* All partition must now have a valid mapping */
- Assert(next_index == nparts);
- break;
- }
-
- case PARTITION_STRATEGY_RANGE:
- {
- boundinfo->kind = (PartitionRangeDatumKind **)
- palloc(ndatums *
- sizeof(PartitionRangeDatumKind *));
- boundinfo->indexes = (int *) palloc((ndatums + 1) *
- sizeof(int));
-
- for (i = 0; i < ndatums; i++)
- {
- int j;
-
- boundinfo->datums[i] = (Datum *) palloc(key->partnatts *
- sizeof(Datum));
- boundinfo->kind[i] = (PartitionRangeDatumKind *)
- palloc(key->partnatts *
- sizeof(PartitionRangeDatumKind));
- for (j = 0; j < key->partnatts; j++)
- {
- if (rbounds[i]->kind[j] == PARTITION_RANGE_DATUM_VALUE)
- boundinfo->datums[i][j] =
- datumCopy(rbounds[i]->datums[j],
- key->parttypbyval[j],
- key->parttyplen[j]);
- boundinfo->kind[i][j] = rbounds[i]->kind[j];
- }
-
- /*
- * There is no mapping for invalid indexes.
- *
- * Any lower bounds in the rbounds array have invalid
- * indexes assigned, because the values between the
- * previous bound (if there is one) and this (lower)
- * bound are not part of the range of any existing
- * partition.
- */
- if (rbounds[i]->lower)
- boundinfo->indexes[i] = -1;
- else
- {
- int orig_index = rbounds[i]->index;
-
- /* If the old index has no mapping, assign one */
- if (mapping[orig_index] == -1)
- mapping[orig_index] = next_index++;
-
- boundinfo->indexes[i] = mapping[orig_index];
- }
- }
-
- /* Assign mapped index for the default partition. */
- if (default_index != -1)
- {
- Assert(default_index >= 0 && mapping[default_index] == -1);
- mapping[default_index] = next_index++;
- boundinfo->default_index = mapping[default_index];
- }
- boundinfo->indexes[i] = -1;
- break;
- }
-
- default:
- elog(ERROR, "unexpected partition strategy: %d",
- (int) key->strategy);
- }
-
- result->boundinfo = boundinfo;
-
- /*
- * Now assign OIDs from the original array into mapped indexes of the
- * result array. Order of OIDs in the former is defined by the
- * catalog scan that retrieved them, whereas that in the latter is
- * defined by canonicalized representation of the partition bounds.
- */
- for (i = 0; i < nparts; i++)
- result->oids[mapping[i]] = oids[i];
- pfree(mapping);
- }
+ partdesc = (PartitionDescData *) palloc0(sizeof(PartitionDescData));
+ partdesc->nparts = nparts;
+ /* oids and boundinfo are allocated below. */
MemoryContextSwitchTo(oldcxt);
- rel->rd_partdesc = result;
+
+ if (nparts == 0)
+ {
+ rel->rd_partdesc = partdesc;
+ return;
+ }
+
+ /* First create the PartitionBoundInfo in caller's context. */
+ boundinfo = partition_bounds_create(boundspecs, key, &mapping);
+ oids_orig = (Oid *) palloc(sizeof(Oid) * partdesc->nparts);
+ i = 0;
+ foreach(cell, inhoids)
+ oids_orig[i++] = lfirst_oid(cell);
+
+ /* Now copy boundinfo and oids into partdesc. */
+ oldcxt = MemoryContextSwitchTo(rel->rd_pdcxt);
+ partdesc->boundinfo = partition_bounds_copy(boundinfo, key);
+ partdesc->oids = (Oid *) palloc(partdesc->nparts * sizeof(Oid));
+
+ /*
+ * Now assign OIDs from the original array into mapped indexes of the
+ * result array. Order of OIDs in the former is defined by the catalog
+ * scan that retrieved them, whereas that in the latter is defined by
+ * canonicalized representation of the partition bounds.
+ */
+ for (i = 0; i < partdesc->nparts; i++)
+ partdesc->oids[mapping[i]] = oids_orig[i];
+ MemoryContextSwitchTo(oldcxt);
+
+ rel->rd_partdesc = partdesc;
}
/*
@@ -917,48 +493,3 @@ generate_partition_qual(Relation rel)
return result;
}
-
-/*
- * qsort_partition_hbound_cmp
- *
- * We sort hash bounds by modulus, then by remainder.
- */
-static int32
-qsort_partition_hbound_cmp(const void *a, const void *b)
-{
- PartitionHashBound *h1 = (*(PartitionHashBound *const *) a);
- PartitionHashBound *h2 = (*(PartitionHashBound *const *) b);
-
- return partition_hbound_cmp(h1->modulus, h1->remainder,
- h2->modulus, h2->remainder);
-}
-
-/*
- * qsort_partition_list_value_cmp
- *
- * Compare two list partition bound datums
- */
-static int32
-qsort_partition_list_value_cmp(const void *a, const void *b, void *arg)
-{
- Datum val1 = (*(const PartitionListValue **) a)->value,
- val2 = (*(const PartitionListValue **) b)->value;
- PartitionKey key = (PartitionKey) arg;
-
- return DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[0],
- key->partcollation[0],
- val1, val2));
-}
-
-/* Used when sorting range bounds across all range partitions */
-static int32
-qsort_partition_rbound_cmp(const void *a, const void *b, void *arg)
-{
- PartitionRangeBound *b1 = (*(PartitionRangeBound *const *) a);
- PartitionRangeBound *b2 = (*(PartitionRangeBound *const *) b);
- PartitionKey key = (PartitionKey) arg;
-
- return partition_rbound_cmp(key->partnatts, key->partsupfunc,
- key->partcollation, b1->datums, b1->kind,
- b1->lower, b2);
-}
diff --git a/src/include/partitioning/partbounds.h b/src/include/partitioning/partbounds.h
index c7535e32fc..7a697d1c0a 100644
--- a/src/include/partitioning/partbounds.h
+++ b/src/include/partitioning/partbounds.h
@@ -75,40 +75,14 @@ typedef struct PartitionBoundInfoData
#define partition_bound_accepts_nulls(bi) ((bi)->null_index != -1)
#define partition_bound_has_default(bi) ((bi)->default_index != -1)
-/*
- * When qsort'ing partition bounds after reading from the catalog, each bound
- * is represented with one of the following structs.
- */
-
-/* One bound of a hash partition */
-typedef struct PartitionHashBound
-{
- int modulus;
- int remainder;
- int index;
-} PartitionHashBound;
-
-/* One value coming from some (index'th) list partition */
-typedef struct PartitionListValue
-{
- int index;
- Datum value;
-} PartitionListValue;
-
-/* One bound of a range partition */
-typedef struct PartitionRangeBound
-{
- int index;
- Datum *datums; /* range bound datums */
- PartitionRangeDatumKind *kind; /* the kind of each datum */
- bool lower; /* this is the lower (vs upper) bound */
-} PartitionRangeBound;
-
extern int get_hash_partition_greatest_modulus(PartitionBoundInfo b);
extern uint64 compute_partition_hash_value(int partnatts, FmgrInfo *partsupfunc,
Datum *values, bool *isnull);
extern List *get_qual_from_partbound(Relation rel, Relation parent,
PartitionBoundSpec *spec);
+extern PartitionBoundInfo partition_bounds_create(List *boundspecs,
+ PartitionKey key,
+ int **mapping);
extern bool partition_bounds_equal(int partnatts, int16 *parttyplen,
bool *parttypbyval, PartitionBoundInfo b1,
PartitionBoundInfo b2);
@@ -120,14 +94,6 @@ extern void check_default_partition_contents(Relation parent,
Relation defaultRel,
PartitionBoundSpec *new_spec);
-extern PartitionRangeBound *make_one_partition_rbound(PartitionKey key, int index,
- List *datums, bool lower);
-extern int32 partition_hbound_cmp(int modulus1, int remainder1, int modulus2,
- int remainder2);
-extern int32 partition_rbound_cmp(int partnatts, FmgrInfo *partsupfunc,
- Oid *partcollation, Datum *datums1,
- PartitionRangeDatumKind *kind1, bool lower1,
- PartitionRangeBound *b2);
extern int32 partition_rbound_datum_cmp(FmgrInfo *partsupfunc,
Oid *partcollation,
Datum *rb_datums, PartitionRangeDatumKind *rb_kind,
@@ -136,10 +102,6 @@ extern int partition_list_bsearch(FmgrInfo *partsupfunc,
Oid *partcollation,
PartitionBoundInfo boundinfo,
Datum value, bool *is_equal);
-extern int partition_range_bsearch(int partnatts, FmgrInfo *partsupfunc,
- Oid *partcollation,
- PartitionBoundInfo boundinfo,
- PartitionRangeBound *probe, bool *is_equal);
extern int partition_range_datum_bsearch(FmgrInfo *partsupfunc,
Oid *partcollation,
PartitionBoundInfo boundinfo,
[application/pgp-signature] signature.asc (833B, ../../[email protected]/3-signature.asc)
download
^ permalink raw reply [nested|flat] 55+ messages in thread
* Re: move PartitionBoundInfo creation code
2018-11-01 03:58 move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-01 04:02 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-01 04:03 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-05 07:21 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 06:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-07 07:37 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 15:04 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
2018-11-08 02:48 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-08 03:59 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-08 06:11 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-12 13:55 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-13 01:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-13 02:34 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
@ 2018-11-13 03:40 ` Amit Langote <[email protected]>
2018-11-13 04:04 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
1 sibling, 1 reply; 55+ messages in thread
From: Amit Langote @ 2018-11-13 03:40 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; pgsql-hackers
On 2018/11/13 11:34, Michael Paquier wrote:
> Attached is an updated patch. Perhaps you are spotting something else?
Looks good to me.
Thanks,
Amit
^ permalink raw reply [nested|flat] 55+ messages in thread
* Re: move PartitionBoundInfo creation code
2018-11-01 03:58 move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-01 04:02 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-01 04:03 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-05 07:21 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 06:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-07 07:37 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 15:04 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
2018-11-08 02:48 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-08 03:59 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-08 06:11 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-12 13:55 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-13 01:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-13 02:34 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-13 03:40 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
@ 2018-11-13 04:04 ` Amit Langote <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: Amit Langote @ 2018-11-13 04:04 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; pgsql-hackers
On 2018/11/13 12:40, Amit Langote wrote:
> On 2018/11/13 11:34, Michael Paquier wrote:
>> Attached is an updated patch. Perhaps you are spotting something else?
>
> Looks good to me.
Marked the CF entry as Ready for Committer.
Thanks,
Amit
^ permalink raw reply [nested|flat] 55+ messages in thread
* Re: move PartitionBoundInfo creation code
2018-11-01 03:58 move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-01 04:02 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-01 04:03 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-05 07:21 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 06:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-07 07:37 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 15:04 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
2018-11-08 02:48 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-08 03:59 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-08 06:11 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-12 13:55 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-13 01:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-13 02:34 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
@ 2018-11-13 12:58 ` Alvaro Herrera <[email protected]>
2018-11-13 13:35 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
1 sibling, 1 reply; 55+ messages in thread
From: Alvaro Herrera @ 2018-11-13 12:58 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Amit Langote <[email protected]>; pgsql-hackers
"the context that was active when the function was called" is typically
expressed simply as "the current memory context". Perhaps the whole
phrase can be reduced to "The object returned by this function is wholly
allocated in the current memory context"?
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 55+ messages in thread
* Re: move PartitionBoundInfo creation code
2018-11-01 03:58 move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-01 04:02 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-01 04:03 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-05 07:21 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 06:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-07 07:37 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 15:04 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
2018-11-08 02:48 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-08 03:59 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-08 06:11 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-12 13:55 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-13 01:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-13 02:34 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-13 12:58 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
@ 2018-11-13 13:35 ` Michael Paquier <[email protected]>
2018-11-13 13:59 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 55+ messages in thread
From: Michael Paquier @ 2018-11-13 13:35 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Amit Langote <[email protected]>; pgsql-hackers
On Tue, Nov 13, 2018 at 09:58:08AM -0300, Alvaro Herrera wrote:
> "the context that was active when the function was called" is typically
> expressed simply as "the current memory context". Perhaps the whole
> phrase can be reduced to "The object returned by this function is wholly
> allocated in the current memory context"?
Yes, that looks cleaner. I am planning to do a last lookup round on
tomorrow morning my time before committing, so I may still tweak a
couple of other words...
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 55+ messages in thread
* Re: move PartitionBoundInfo creation code
2018-11-01 03:58 move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-01 04:02 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-01 04:03 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-05 07:21 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 06:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-07 07:37 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 15:04 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
2018-11-08 02:48 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-08 03:59 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-08 06:11 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-12 13:55 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-13 01:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-13 02:34 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-13 12:58 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
2018-11-13 13:35 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
@ 2018-11-13 13:59 ` Alvaro Herrera <[email protected]>
2018-11-14 01:04 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-14 01:09 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
0 siblings, 2 replies; 55+ messages in thread
From: Alvaro Herrera @ 2018-11-13 13:59 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Amit Langote <[email protected]>; pgsql-hackers
On 2018-Nov-13, Michael Paquier wrote:
> On Tue, Nov 13, 2018 at 09:58:08AM -0300, Alvaro Herrera wrote:
> > "the context that was active when the function was called" is typically
> > expressed simply as "the current memory context". Perhaps the whole
> > phrase can be reduced to "The object returned by this function is wholly
> > allocated in the current memory context"?
>
> Yes, that looks cleaner. I am planning to do a last lookup round on
> tomorrow morning my time before committing, so I may still tweak a
> couple of other words...
Cool.
I gave the patch a read and it looks reasonable to me.
Memory management in RelationBuildPartitionDesc is crummy, but I don't
think it's this patch's fault.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 55+ messages in thread
* Re: move PartitionBoundInfo creation code
2018-11-01 03:58 move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-01 04:02 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-01 04:03 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-05 07:21 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 06:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-07 07:37 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 15:04 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
2018-11-08 02:48 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-08 03:59 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-08 06:11 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-12 13:55 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-13 01:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-13 02:34 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-13 12:58 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
2018-11-13 13:35 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-13 13:59 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
@ 2018-11-14 01:04 ` Michael Paquier <[email protected]>
1 sibling, 0 replies; 55+ messages in thread
From: Michael Paquier @ 2018-11-14 01:04 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Amit Langote <[email protected]>; pgsql-hackers
On Tue, Nov 13, 2018 at 10:59:15AM -0300, Alvaro Herrera wrote:
> I gave the patch a read and it looks reasonable to me.
>
> Memory management in RelationBuildPartitionDesc is crummy, but I don't
> think it's this patch's fault.
I agree, and that shows up pretty clearly after the refactoring is done.
The mess is partially caused by the handling around the case where there
is no partition data to attach to PartitionDescData. I would personally
much prefer if we could also avoid using partition_bounds_copy.
It does not prevent the first refactoring step, so I have committed the
patch as it is already doing a lot.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 55+ messages in thread
* Re: move PartitionBoundInfo creation code
2018-11-01 03:58 move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-01 04:02 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-01 04:03 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-05 07:21 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 06:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-07 07:37 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 15:04 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
2018-11-08 02:48 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-08 03:59 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-08 06:11 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-12 13:55 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-13 01:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-13 02:34 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-13 12:58 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
2018-11-13 13:35 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-13 13:59 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
@ 2018-11-14 01:09 ` Amit Langote <[email protected]>
1 sibling, 0 replies; 55+ messages in thread
From: Amit Langote @ 2018-11-14 01:09 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; Michael Paquier <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2018/11/13 22:59, Alvaro Herrera wrote:
> On 2018-Nov-13, Michael Paquier wrote:
>
>> On Tue, Nov 13, 2018 at 09:58:08AM -0300, Alvaro Herrera wrote:
>>> "the context that was active when the function was called" is typically
>>> expressed simply as "the current memory context". Perhaps the whole
>>> phrase can be reduced to "The object returned by this function is wholly
>>> allocated in the current memory context"?
Yeah, don't know why I had to put it in such convoluted manner.
>> Yes, that looks cleaner. I am planning to do a last lookup round on
>> tomorrow morning my time before committing, so I may still tweak a
>> couple of other words...
>
> Cool.
Thanks Michael.
> I gave the patch a read and it looks reasonable to me.
>
> Memory management in RelationBuildPartitionDesc is crummy, but I don't
> think it's this patch's fault.
Are you perhaps referring to the discussion about partitioning cache
management we had a few months back, but decided to postpone any
development work to PG 12? The following thread, maybe:
https://www.postgresql.org/message-id/143ed9a4-6038-76d4-9a55-502035815e68%40lab.ntt.co.jp
Thanks,
Amit
^ permalink raw reply [nested|flat] 55+ messages in thread
* Re: move PartitionBoundInfo creation code
2018-11-01 03:58 move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-01 04:02 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-01 04:03 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-05 07:21 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 06:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-07 07:37 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 15:04 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
@ 2018-11-08 04:31 ` Amit Langote <[email protected]>
2018-11-08 13:30 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
1 sibling, 1 reply; 55+ messages in thread
From: Amit Langote @ 2018-11-08 04:31 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; Michael Paquier <[email protected]>; +Cc: pgsql-hackers
On 2018/11/08 0:04, Alvaro Herrera wrote:
>> On Wed, Nov 07, 2018 at 03:34:38PM +0900, Amit Langote wrote:
>>> I think we can design the interface of partition_bounds_create such that
>>> it returns information needed to re-arrange OIDs to be in the canonical
>>> partition bound order, so the actual re-ordering of OIDs can be done by
>>> the caller itself. (It may not always be OIDs, could be something else
>>> like a struct to represent fake partitions.)
>
> This is interesting. I don't think the current interface is so bad that
> we can't make a few tweaks later; IOW let's focus on the current patch
> for now, and we can improve later. You (and David, and maybe someone
> else) already have half a dozen patches touching this code, and I bet
> some of these will have to be rebased over this one.
The patch on ATTACH/DETACH concurrently thread is perhaps touching close
to here, but I'm not sure if any of it should be concerned about how we
generate the PartitionBoundInfo which the patch here trying to make into
its own function?
Thanks,
Amit
^ permalink raw reply [nested|flat] 55+ messages in thread
* Re: move PartitionBoundInfo creation code
2018-11-01 03:58 move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-01 04:02 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-01 04:03 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-05 07:21 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 06:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-07 07:37 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 15:04 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
2018-11-08 04:31 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
@ 2018-11-08 13:30 ` Alvaro Herrera <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: Alvaro Herrera @ 2018-11-08 13:30 UTC (permalink / raw)
To: Amit Langote <[email protected]>; +Cc: Michael Paquier <[email protected]>; pgsql-hackers
On 2018-Nov-08, Amit Langote wrote:
> On 2018/11/08 0:04, Alvaro Herrera wrote:
> >> On Wed, Nov 07, 2018 at 03:34:38PM +0900, Amit Langote wrote:
> >>> I think we can design the interface of partition_bounds_create such that
> >>> it returns information needed to re-arrange OIDs to be in the canonical
> >>> partition bound order, so the actual re-ordering of OIDs can be done by
> >>> the caller itself. (It may not always be OIDs, could be something else
> >>> like a struct to represent fake partitions.)
> >
> > This is interesting. I don't think the current interface is so bad that
> > we can't make a few tweaks later; IOW let's focus on the current patch
> > for now, and we can improve later. You (and David, and maybe someone
> > else) already have half a dozen patches touching this code, and I bet
> > some of these will have to be rebased over this one.
>
> The patch on ATTACH/DETACH concurrently thread is perhaps touching close
> to here, but I'm not sure if any of it should be concerned about how we
> generate the PartitionBoundInfo which the patch here trying to make into
> its own function?
Yeah, definitely not.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 55+ messages in thread
* [PATCH v33 4/5] Subscripting documentation
@ 2019-02-01 10:47 erthalion <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: erthalion @ 2019-02-01 10:47 UTC (permalink / raw)
Supporting documentation for generalized subscripting. It includes the
description of a new field in pg_type, the new section for jsonb
documentation about subscripting feature on this data type, and also the
tutorial about how to write subscripting operator for a custom data type.
Reviewed-by: Tom Lane, Arthur Zakirov, Oleksandr Shulgin
---
doc/src/sgml/catalogs.sgml | 8 ++
doc/src/sgml/extend.sgml | 6 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/json.sgml | 39 ++++++
doc/src/sgml/ref/create_type.sgml | 33 ++++-
doc/src/sgml/xsubscripting.sgml | 111 +++++++++++++++++
src/tutorial/Makefile | 4 +-
src/tutorial/subscripting.c | 201 ++++++++++++++++++++++++++++++
src/tutorial/subscripting.source | 71 +++++++++++
9 files changed, 470 insertions(+), 4 deletions(-)
create mode 100644 doc/src/sgml/xsubscripting.sgml
create mode 100644 src/tutorial/subscripting.c
create mode 100644 src/tutorial/subscripting.source
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 048ff284f7..b17e9ee382 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8971,6 +8971,14 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>typsubshandler</structfield></entry>
+ <entry><type>regproc</type></entry>
+ <entry><literal><link linkend="catalog-pg-proc"><structname>pg_proc</structname></link>.oid</literal></entry>
+ <entry>Custom subscripting function with type-specific logic for parsing
+ and validation, or 0 if this type doesn't support subscripting.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>typdefault</structfield> <type>text</type>
diff --git a/doc/src/sgml/extend.sgml b/doc/src/sgml/extend.sgml
index 890ff97b7a..65cf5706b6 100644
--- a/doc/src/sgml/extend.sgml
+++ b/doc/src/sgml/extend.sgml
@@ -33,6 +33,11 @@
operators (starting in <xref linkend="xoper"/>)
</para>
</listitem>
+ <listitem>
+ <para>
+ subscripting procedure (starting in <xref linkend="xsubscripting"/>)
+ </para>
+ </listitem>
<listitem>
<para>
operator classes for indexes (starting in <xref linkend="xindex"/>)
@@ -481,6 +486,7 @@ RETURNS anycompatible AS ...
&xaggr;
&xtypes;
&xoper;
+ &xsubscripting;
&xindex;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 64b5da0070..4c3fd37fc1 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -69,6 +69,7 @@
<!ENTITY xplang SYSTEM "xplang.sgml">
<!ENTITY xoper SYSTEM "xoper.sgml">
<!ENTITY xtypes SYSTEM "xtypes.sgml">
+<!ENTITY xsubscripting SYSTEM "xsubscripting.sgml">
<!ENTITY plperl SYSTEM "plperl.sgml">
<!ENTITY plpython SYSTEM "plpython.sgml">
<!ENTITY plsql SYSTEM "plpgsql.sgml">
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index c0a6554d4d..5c538dca05 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -602,6 +602,45 @@ SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qu
</para>
</sect2>
+ <sect2 id="jsonb-subscripting">
+ <title><type>jsonb</type> Subscripting</title>
+ <para>
+ <type>jsonb</type> data type supports array-style subscripting expressions
+ to extract or update particular elements. It's possible to use multiple
+ subscripting expressions to extract nested values. In this case, a chain of
+ subscripting expressions follows the same rules as the
+ <literal>path</literal> argument in <literal>jsonb_set</literal> function,
+ e.g. in case of arrays it is a 0-based operation or that negative integers
+ that appear in <literal>path</literal> count from the end of JSON arrays.
+ The result of subscripting expressions is always jsonb data type. An
+ example of subscripting syntax:
+<programlisting>
+-- Extract value by key
+SELECT ('{"a": 1}'::jsonb)['a'];
+
+-- Extract nested value by key path
+SELECT ('{"a": {"b": {"c": 1}}}'::jsonb)['a']['b']['c'];
+
+-- Extract element by index
+SELECT ('[1, "2", null]'::jsonb)[1];
+
+-- Update value by key
+UPDATE table_name SET jsonb_field['key'] = 1;
+
+-- Select records using where clause with subscripting. Since the result of
+-- subscripting is jsonb and we basically want to compare two jsonb objects, we
+-- need to put the value in double quotes to be able to convert it to jsonb.
+SELECT * FROM table_name WHERE jsonb_field['key'] = '"value"';
+</programlisting>
+
+ There is no special indexing support for such kind of expressions, but you
+ can always create a functional index that includes it
+<programlisting>
+CREATE INDEX idx ON table_name ((jsonb_field['key']));
+</programlisting>
+ </para>
+ </sect2>
+
<sect2>
<title>Transforms</title>
diff --git a/doc/src/sgml/ref/create_type.sgml b/doc/src/sgml/ref/create_type.sgml
index 111f8e65d2..a34df4d247 100644
--- a/doc/src/sgml/ref/create_type.sgml
+++ b/doc/src/sgml/ref/create_type.sgml
@@ -54,6 +54,7 @@ CREATE TYPE <replaceable class="parameter">name</replaceable> (
[ , ELEMENT = <replaceable class="parameter">element</replaceable> ]
[ , DELIMITER = <replaceable class="parameter">delimiter</replaceable> ]
[ , COLLATABLE = <replaceable class="parameter">collatable</replaceable> ]
+ [ , SUBSCRIPTING_HANDLER = <replaceable class="parameter">subscripting_handler_function</replaceable> ]
)
CREATE TYPE <replaceable class="parameter">name</replaceable>
@@ -196,8 +197,9 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
<replaceable class="parameter">receive_function</replaceable>,
<replaceable class="parameter">send_function</replaceable>,
<replaceable class="parameter">type_modifier_input_function</replaceable>,
- <replaceable class="parameter">type_modifier_output_function</replaceable> and
- <replaceable class="parameter">analyze_function</replaceable>
+ <replaceable class="parameter">type_modifier_output_function</replaceable>,
+ <replaceable class="parameter">analyze_function</replaceable>,
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
are optional. Generally these functions have to be coded in C
or another low-level language.
</para>
@@ -454,6 +456,22 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
make use of the collation information; this does not happen
automatically merely by marking the type collatable.
</para>
+
+ <para>
+ The optional
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
+ contains type-specific logic for subscripting of the data type.
+ By default, there is no such function provided, which means that the data
+ type doesn't support subscripting. The subscripting function must be
+ declared to take a single argument of type <type>internal</type>, and return
+ a <type>internal</type> result. There are two examples of implementation for
+ subscripting functions in case of array
+ (<replaceable class="parameter">array_subscripting_handler</replaceable>)
+ and jsonb
+ (<replaceable class="parameter">jsonb_subscripting_handler</replaceable>)
+ types in <filename>src/backend/utils/adt/arrayfuncs.c</filename> and
+ <filename>src/backend/utils/adt/jsonfuncs.c</filename>, respectively.
+ </para>
</refsect2>
<refsect2>
@@ -769,6 +787,17 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">subscripting_handler_function</replaceable></term>
+ <listitem>
+ <para>
+ The name of a function that returns list of type-specific callback functions to
+ support subscripting logic for the data type.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
diff --git a/doc/src/sgml/xsubscripting.sgml b/doc/src/sgml/xsubscripting.sgml
new file mode 100644
index 0000000000..7224e81fa2
--- /dev/null
+++ b/doc/src/sgml/xsubscripting.sgml
@@ -0,0 +1,111 @@
+<!-- doc/src/sgml/xsubscripting.sgml -->
+
+ <sect1 id="xsubscripting">
+ <title>User-defined subscripting procedure</title>
+
+ <indexterm zone="xsubscripting">
+ <primary>custom subscripting</primary>
+ </indexterm>
+ <para>
+ When you define a new base type, you can also specify a custom procedure to
+ handle subscripting expressions. It must contain logic for verification and
+ evaluation of this expression, i.e. fetching or updating some data in this
+ data type. For instance:
+</para>
+<programlisting><![CDATA[
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = someType;
+ sbsref->refassgntype = someType;
+
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ // some validation and coercion logic
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some assignment logic
+
+ return newContainer;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some fetch logic based on sbstate
+}]]>
+</programlisting>
+
+<para>
+ Then you can define a subscripting procedure and a custom data type:
+</para>
+<programlisting>
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '<replaceable>filename</replaceable>'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 4,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler,
+);
+</programlisting>
+
+<para>
+ and use it as usual:
+</para>
+<programlisting>
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
+</programlisting>
+
+
+ <para>
+ The examples of custom subscripting implementation can be found in
+ <filename>subscripting.sql</filename> and <filename>subscripting.c</filename>
+ in the <filename>src/tutorial</filename> directory of the source distribution.
+ See the <filename>README</filename> file in that directory for instructions
+ about running the examples.
+ </para>
+
+</sect1>
diff --git a/src/tutorial/Makefile b/src/tutorial/Makefile
index 16dc390f71..0ead60c2d4 100644
--- a/src/tutorial/Makefile
+++ b/src/tutorial/Makefile
@@ -13,8 +13,8 @@
#
#-------------------------------------------------------------------------
-MODULES = complex funcs
-DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql
+MODULES = complex funcs subscripting
+DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql subscripting.sql
ifdef NO_PGXS
subdir = src/tutorial
diff --git a/src/tutorial/subscripting.c b/src/tutorial/subscripting.c
new file mode 100644
index 0000000000..1eb8c45652
--- /dev/null
+++ b/src/tutorial/subscripting.c
@@ -0,0 +1,201 @@
+/*
+ * src/tutorial/subscripting.c
+ *
+ ******************************************************************************
+ This file contains routines that can be bound to a Postgres backend and
+ called by the backend in the process of processing queries. The calling
+ format for these routines is dictated by Postgres architecture.
+******************************************************************************/
+
+#include "postgres.h"
+
+#include "catalog/pg_type.h"
+#include "executor/executor.h"
+#include "executor/execExpr.h"
+#include "nodes/nodeFuncs.h"
+#include "parser/parse_coerce.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+SubscriptingRef * custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref);
+SubscriptingRef * custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate);
+Datum custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate);
+Datum custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate);
+
+PG_FUNCTION_INFO_V1(custom_in);
+PG_FUNCTION_INFO_V1(custom_out);
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+/*****************************************************************************
+ * Input/Output functions
+ *****************************************************************************/
+
+Datum
+custom_in(PG_FUNCTION_ARGS)
+{
+ char *str = PG_GETARG_CSTRING(0);
+ int firstValue,
+ secondValue;
+ Custom *result;
+
+ if (sscanf(str, " ( %d , %d )", &firstValue, &secondValue) != 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
+ errmsg("invalid input syntax for complex: \"%s\"",
+ str)));
+
+
+ result = (Custom *) palloc(sizeof(Custom));
+ result->first = firstValue;
+ result->second = secondValue;
+ PG_RETURN_POINTER(result);
+}
+
+Datum
+custom_out(PG_FUNCTION_ARGS)
+{
+ Custom *custom = (Custom *) PG_GETARG_POINTER(0);
+ char *result;
+
+ result = psprintf("(%d, %d)", custom->first, custom->second);
+ PG_RETURN_CSTRING(result);
+}
+
+/*****************************************************************************
+ * Custom subscripting logic functions
+ *****************************************************************************/
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = INT4OID;
+ sbsref->refassgntype = INT4OID;
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ List *upperIndexpr = NIL;
+ ListCell *l;
+
+ if (sbsref->reflowerindexpr != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *)lfirst(sbsref->reflowerindexpr->head))))));
+
+ foreach(l, sbsref->refupperindexpr)
+ {
+ Node *subexpr = (Node *) lfirst(l);
+
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *) lfirst(sbsref->refupperindexpr->head))))));
+
+ subexpr = coerce_to_target_type(pstate,
+ subexpr, exprType(subexpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript must have integer type"),
+ parser_errposition(pstate, exprLocation(subexpr))));
+
+ upperIndexpr = lappend(upperIndexpr, subexpr);
+
+ if (isAssignment)
+ {
+ Node *assignExpr = (Node *) sbsref->refassgnexpr;
+ Node *new_from;
+
+ new_from = coerce_to_target_type(pstate,
+ assignExpr, exprType(assignExpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (new_from == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom assignment requires int type"),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation(assignExpr))));
+ sbsref->refassgnexpr = (Expr *)new_from;
+ }
+ }
+
+ sbsref->refupperindexpr = upperIndexpr;
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ Custom *container= (Custom *) containerSource;
+ int index;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ return (Datum) container->first;
+ else
+ return (Datum) container->second;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ int index;
+ Custom *container = (Custom *) containerSource;
+
+ if (sbstate->resnull)
+ return containerSource;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ container->first = DatumGetInt32(sbstate->replacevalue);
+ else
+ container->second = DatumGetInt32(sbstate->replacevalue);
+
+ return (Datum) container;
+}
diff --git a/src/tutorial/subscripting.source b/src/tutorial/subscripting.source
new file mode 100644
index 0000000000..837cf30612
--- /dev/null
+++ b/src/tutorial/subscripting.source
@@ -0,0 +1,71 @@
+---------------------------------------------------------------------------
+--
+-- subscripting.sql-
+-- This file shows how to create a new subscripting procedure for
+-- user-defined type.
+--
+--
+-- Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+-- Portions Copyright (c) 1994, Regents of the University of California
+--
+-- src/tutorial/subscripting.source
+--
+---------------------------------------------------------------------------
+
+-----------------------------
+-- Creating a new type:
+-- We are going to create a new type called 'complex' which represents
+-- complex numbers.
+-- A user-defined type must have an input and an output function, and
+-- optionally can have binary input and output functions. All of these
+-- are usually user-defined C functions.
+-----------------------------
+
+-- Assume the user defined functions are in /home/erthalion/programms/postgresql-master/src/tutorial/complex$DLSUFFIX
+-- (we do not want to assume this is in the dynamic loader search path).
+-- Look at $PWD/complex.c for the source. Note that we declare all of
+-- them as STRICT, so we do not need to cope with NULL inputs in the
+-- C code. We also mark them IMMUTABLE, since they always return the
+-- same outputs given the same inputs.
+
+-- the input function 'complex_in' takes a null-terminated string (the
+-- textual representation of the type) and turns it into the internal
+-- (in memory) representation. You will get a message telling you 'complex'
+-- does not exist yet but that's okay.
+
+CREATE FUNCTION custom_in(cstring)
+ RETURNS custom
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+-- the output function 'complex_out' takes the internal representation and
+-- converts it into the textual representation.
+
+CREATE FUNCTION custom_out(custom)
+ RETURNS cstring
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 8,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler
+);
+
+-- we can use it in a table
+
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
--
2.21.0
--u66dlpg6gaevsofr
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v33-0005-Filling-gaps-in-jsonb-arrays.patch"
^ permalink raw reply [nested|flat] 55+ messages in thread
* [PATCH v33 4/5] Subscripting documentation
@ 2019-02-01 10:47 erthalion <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: erthalion @ 2019-02-01 10:47 UTC (permalink / raw)
Supporting documentation for generalized subscripting. It includes the
description of a new field in pg_type, the new section for jsonb
documentation about subscripting feature on this data type, and also the
tutorial about how to write subscripting operator for a custom data type.
Reviewed-by: Tom Lane, Arthur Zakirov, Oleksandr Shulgin
---
doc/src/sgml/catalogs.sgml | 8 ++
doc/src/sgml/extend.sgml | 6 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/json.sgml | 39 ++++++
doc/src/sgml/ref/create_type.sgml | 33 ++++-
doc/src/sgml/xsubscripting.sgml | 111 +++++++++++++++++
src/tutorial/Makefile | 4 +-
src/tutorial/subscripting.c | 201 ++++++++++++++++++++++++++++++
src/tutorial/subscripting.source | 71 +++++++++++
9 files changed, 470 insertions(+), 4 deletions(-)
create mode 100644 doc/src/sgml/xsubscripting.sgml
create mode 100644 src/tutorial/subscripting.c
create mode 100644 src/tutorial/subscripting.source
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 048ff284f7..b17e9ee382 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8971,6 +8971,14 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>typsubshandler</structfield></entry>
+ <entry><type>regproc</type></entry>
+ <entry><literal><link linkend="catalog-pg-proc"><structname>pg_proc</structname></link>.oid</literal></entry>
+ <entry>Custom subscripting function with type-specific logic for parsing
+ and validation, or 0 if this type doesn't support subscripting.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>typdefault</structfield> <type>text</type>
diff --git a/doc/src/sgml/extend.sgml b/doc/src/sgml/extend.sgml
index 890ff97b7a..65cf5706b6 100644
--- a/doc/src/sgml/extend.sgml
+++ b/doc/src/sgml/extend.sgml
@@ -33,6 +33,11 @@
operators (starting in <xref linkend="xoper"/>)
</para>
</listitem>
+ <listitem>
+ <para>
+ subscripting procedure (starting in <xref linkend="xsubscripting"/>)
+ </para>
+ </listitem>
<listitem>
<para>
operator classes for indexes (starting in <xref linkend="xindex"/>)
@@ -481,6 +486,7 @@ RETURNS anycompatible AS ...
&xaggr;
&xtypes;
&xoper;
+ &xsubscripting;
&xindex;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 64b5da0070..4c3fd37fc1 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -69,6 +69,7 @@
<!ENTITY xplang SYSTEM "xplang.sgml">
<!ENTITY xoper SYSTEM "xoper.sgml">
<!ENTITY xtypes SYSTEM "xtypes.sgml">
+<!ENTITY xsubscripting SYSTEM "xsubscripting.sgml">
<!ENTITY plperl SYSTEM "plperl.sgml">
<!ENTITY plpython SYSTEM "plpython.sgml">
<!ENTITY plsql SYSTEM "plpgsql.sgml">
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index c0a6554d4d..5c538dca05 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -602,6 +602,45 @@ SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qu
</para>
</sect2>
+ <sect2 id="jsonb-subscripting">
+ <title><type>jsonb</type> Subscripting</title>
+ <para>
+ <type>jsonb</type> data type supports array-style subscripting expressions
+ to extract or update particular elements. It's possible to use multiple
+ subscripting expressions to extract nested values. In this case, a chain of
+ subscripting expressions follows the same rules as the
+ <literal>path</literal> argument in <literal>jsonb_set</literal> function,
+ e.g. in case of arrays it is a 0-based operation or that negative integers
+ that appear in <literal>path</literal> count from the end of JSON arrays.
+ The result of subscripting expressions is always jsonb data type. An
+ example of subscripting syntax:
+<programlisting>
+-- Extract value by key
+SELECT ('{"a": 1}'::jsonb)['a'];
+
+-- Extract nested value by key path
+SELECT ('{"a": {"b": {"c": 1}}}'::jsonb)['a']['b']['c'];
+
+-- Extract element by index
+SELECT ('[1, "2", null]'::jsonb)[1];
+
+-- Update value by key
+UPDATE table_name SET jsonb_field['key'] = 1;
+
+-- Select records using where clause with subscripting. Since the result of
+-- subscripting is jsonb and we basically want to compare two jsonb objects, we
+-- need to put the value in double quotes to be able to convert it to jsonb.
+SELECT * FROM table_name WHERE jsonb_field['key'] = '"value"';
+</programlisting>
+
+ There is no special indexing support for such kind of expressions, but you
+ can always create a functional index that includes it
+<programlisting>
+CREATE INDEX idx ON table_name ((jsonb_field['key']));
+</programlisting>
+ </para>
+ </sect2>
+
<sect2>
<title>Transforms</title>
diff --git a/doc/src/sgml/ref/create_type.sgml b/doc/src/sgml/ref/create_type.sgml
index 111f8e65d2..a34df4d247 100644
--- a/doc/src/sgml/ref/create_type.sgml
+++ b/doc/src/sgml/ref/create_type.sgml
@@ -54,6 +54,7 @@ CREATE TYPE <replaceable class="parameter">name</replaceable> (
[ , ELEMENT = <replaceable class="parameter">element</replaceable> ]
[ , DELIMITER = <replaceable class="parameter">delimiter</replaceable> ]
[ , COLLATABLE = <replaceable class="parameter">collatable</replaceable> ]
+ [ , SUBSCRIPTING_HANDLER = <replaceable class="parameter">subscripting_handler_function</replaceable> ]
)
CREATE TYPE <replaceable class="parameter">name</replaceable>
@@ -196,8 +197,9 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
<replaceable class="parameter">receive_function</replaceable>,
<replaceable class="parameter">send_function</replaceable>,
<replaceable class="parameter">type_modifier_input_function</replaceable>,
- <replaceable class="parameter">type_modifier_output_function</replaceable> and
- <replaceable class="parameter">analyze_function</replaceable>
+ <replaceable class="parameter">type_modifier_output_function</replaceable>,
+ <replaceable class="parameter">analyze_function</replaceable>,
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
are optional. Generally these functions have to be coded in C
or another low-level language.
</para>
@@ -454,6 +456,22 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
make use of the collation information; this does not happen
automatically merely by marking the type collatable.
</para>
+
+ <para>
+ The optional
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
+ contains type-specific logic for subscripting of the data type.
+ By default, there is no such function provided, which means that the data
+ type doesn't support subscripting. The subscripting function must be
+ declared to take a single argument of type <type>internal</type>, and return
+ a <type>internal</type> result. There are two examples of implementation for
+ subscripting functions in case of array
+ (<replaceable class="parameter">array_subscripting_handler</replaceable>)
+ and jsonb
+ (<replaceable class="parameter">jsonb_subscripting_handler</replaceable>)
+ types in <filename>src/backend/utils/adt/arrayfuncs.c</filename> and
+ <filename>src/backend/utils/adt/jsonfuncs.c</filename>, respectively.
+ </para>
</refsect2>
<refsect2>
@@ -769,6 +787,17 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">subscripting_handler_function</replaceable></term>
+ <listitem>
+ <para>
+ The name of a function that returns list of type-specific callback functions to
+ support subscripting logic for the data type.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
diff --git a/doc/src/sgml/xsubscripting.sgml b/doc/src/sgml/xsubscripting.sgml
new file mode 100644
index 0000000000..7224e81fa2
--- /dev/null
+++ b/doc/src/sgml/xsubscripting.sgml
@@ -0,0 +1,111 @@
+<!-- doc/src/sgml/xsubscripting.sgml -->
+
+ <sect1 id="xsubscripting">
+ <title>User-defined subscripting procedure</title>
+
+ <indexterm zone="xsubscripting">
+ <primary>custom subscripting</primary>
+ </indexterm>
+ <para>
+ When you define a new base type, you can also specify a custom procedure to
+ handle subscripting expressions. It must contain logic for verification and
+ evaluation of this expression, i.e. fetching or updating some data in this
+ data type. For instance:
+</para>
+<programlisting><![CDATA[
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = someType;
+ sbsref->refassgntype = someType;
+
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ // some validation and coercion logic
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some assignment logic
+
+ return newContainer;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some fetch logic based on sbstate
+}]]>
+</programlisting>
+
+<para>
+ Then you can define a subscripting procedure and a custom data type:
+</para>
+<programlisting>
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '<replaceable>filename</replaceable>'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 4,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler,
+);
+</programlisting>
+
+<para>
+ and use it as usual:
+</para>
+<programlisting>
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
+</programlisting>
+
+
+ <para>
+ The examples of custom subscripting implementation can be found in
+ <filename>subscripting.sql</filename> and <filename>subscripting.c</filename>
+ in the <filename>src/tutorial</filename> directory of the source distribution.
+ See the <filename>README</filename> file in that directory for instructions
+ about running the examples.
+ </para>
+
+</sect1>
diff --git a/src/tutorial/Makefile b/src/tutorial/Makefile
index 16dc390f71..0ead60c2d4 100644
--- a/src/tutorial/Makefile
+++ b/src/tutorial/Makefile
@@ -13,8 +13,8 @@
#
#-------------------------------------------------------------------------
-MODULES = complex funcs
-DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql
+MODULES = complex funcs subscripting
+DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql subscripting.sql
ifdef NO_PGXS
subdir = src/tutorial
diff --git a/src/tutorial/subscripting.c b/src/tutorial/subscripting.c
new file mode 100644
index 0000000000..1eb8c45652
--- /dev/null
+++ b/src/tutorial/subscripting.c
@@ -0,0 +1,201 @@
+/*
+ * src/tutorial/subscripting.c
+ *
+ ******************************************************************************
+ This file contains routines that can be bound to a Postgres backend and
+ called by the backend in the process of processing queries. The calling
+ format for these routines is dictated by Postgres architecture.
+******************************************************************************/
+
+#include "postgres.h"
+
+#include "catalog/pg_type.h"
+#include "executor/executor.h"
+#include "executor/execExpr.h"
+#include "nodes/nodeFuncs.h"
+#include "parser/parse_coerce.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+SubscriptingRef * custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref);
+SubscriptingRef * custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate);
+Datum custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate);
+Datum custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate);
+
+PG_FUNCTION_INFO_V1(custom_in);
+PG_FUNCTION_INFO_V1(custom_out);
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+/*****************************************************************************
+ * Input/Output functions
+ *****************************************************************************/
+
+Datum
+custom_in(PG_FUNCTION_ARGS)
+{
+ char *str = PG_GETARG_CSTRING(0);
+ int firstValue,
+ secondValue;
+ Custom *result;
+
+ if (sscanf(str, " ( %d , %d )", &firstValue, &secondValue) != 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
+ errmsg("invalid input syntax for complex: \"%s\"",
+ str)));
+
+
+ result = (Custom *) palloc(sizeof(Custom));
+ result->first = firstValue;
+ result->second = secondValue;
+ PG_RETURN_POINTER(result);
+}
+
+Datum
+custom_out(PG_FUNCTION_ARGS)
+{
+ Custom *custom = (Custom *) PG_GETARG_POINTER(0);
+ char *result;
+
+ result = psprintf("(%d, %d)", custom->first, custom->second);
+ PG_RETURN_CSTRING(result);
+}
+
+/*****************************************************************************
+ * Custom subscripting logic functions
+ *****************************************************************************/
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = INT4OID;
+ sbsref->refassgntype = INT4OID;
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ List *upperIndexpr = NIL;
+ ListCell *l;
+
+ if (sbsref->reflowerindexpr != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *)lfirst(sbsref->reflowerindexpr->head))))));
+
+ foreach(l, sbsref->refupperindexpr)
+ {
+ Node *subexpr = (Node *) lfirst(l);
+
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *) lfirst(sbsref->refupperindexpr->head))))));
+
+ subexpr = coerce_to_target_type(pstate,
+ subexpr, exprType(subexpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript must have integer type"),
+ parser_errposition(pstate, exprLocation(subexpr))));
+
+ upperIndexpr = lappend(upperIndexpr, subexpr);
+
+ if (isAssignment)
+ {
+ Node *assignExpr = (Node *) sbsref->refassgnexpr;
+ Node *new_from;
+
+ new_from = coerce_to_target_type(pstate,
+ assignExpr, exprType(assignExpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (new_from == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom assignment requires int type"),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation(assignExpr))));
+ sbsref->refassgnexpr = (Expr *)new_from;
+ }
+ }
+
+ sbsref->refupperindexpr = upperIndexpr;
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ Custom *container= (Custom *) containerSource;
+ int index;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ return (Datum) container->first;
+ else
+ return (Datum) container->second;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ int index;
+ Custom *container = (Custom *) containerSource;
+
+ if (sbstate->resnull)
+ return containerSource;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ container->first = DatumGetInt32(sbstate->replacevalue);
+ else
+ container->second = DatumGetInt32(sbstate->replacevalue);
+
+ return (Datum) container;
+}
diff --git a/src/tutorial/subscripting.source b/src/tutorial/subscripting.source
new file mode 100644
index 0000000000..837cf30612
--- /dev/null
+++ b/src/tutorial/subscripting.source
@@ -0,0 +1,71 @@
+---------------------------------------------------------------------------
+--
+-- subscripting.sql-
+-- This file shows how to create a new subscripting procedure for
+-- user-defined type.
+--
+--
+-- Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+-- Portions Copyright (c) 1994, Regents of the University of California
+--
+-- src/tutorial/subscripting.source
+--
+---------------------------------------------------------------------------
+
+-----------------------------
+-- Creating a new type:
+-- We are going to create a new type called 'complex' which represents
+-- complex numbers.
+-- A user-defined type must have an input and an output function, and
+-- optionally can have binary input and output functions. All of these
+-- are usually user-defined C functions.
+-----------------------------
+
+-- Assume the user defined functions are in /home/erthalion/programms/postgresql-master/src/tutorial/complex$DLSUFFIX
+-- (we do not want to assume this is in the dynamic loader search path).
+-- Look at $PWD/complex.c for the source. Note that we declare all of
+-- them as STRICT, so we do not need to cope with NULL inputs in the
+-- C code. We also mark them IMMUTABLE, since they always return the
+-- same outputs given the same inputs.
+
+-- the input function 'complex_in' takes a null-terminated string (the
+-- textual representation of the type) and turns it into the internal
+-- (in memory) representation. You will get a message telling you 'complex'
+-- does not exist yet but that's okay.
+
+CREATE FUNCTION custom_in(cstring)
+ RETURNS custom
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+-- the output function 'complex_out' takes the internal representation and
+-- converts it into the textual representation.
+
+CREATE FUNCTION custom_out(custom)
+ RETURNS cstring
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 8,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler
+);
+
+-- we can use it in a table
+
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
--
2.21.0
--u66dlpg6gaevsofr
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v33-0005-Filling-gaps-in-jsonb-arrays.patch"
^ permalink raw reply [nested|flat] 55+ messages in thread
* [PATCH v33 4/5] Subscripting documentation
@ 2019-02-01 10:47 erthalion <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: erthalion @ 2019-02-01 10:47 UTC (permalink / raw)
Supporting documentation for generalized subscripting. It includes the
description of a new field in pg_type, the new section for jsonb
documentation about subscripting feature on this data type, and also the
tutorial about how to write subscripting operator for a custom data type.
Reviewed-by: Tom Lane, Arthur Zakirov, Oleksandr Shulgin
---
doc/src/sgml/catalogs.sgml | 8 ++
doc/src/sgml/extend.sgml | 6 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/json.sgml | 39 ++++++
doc/src/sgml/ref/create_type.sgml | 33 ++++-
doc/src/sgml/xsubscripting.sgml | 111 +++++++++++++++++
src/tutorial/Makefile | 4 +-
src/tutorial/subscripting.c | 201 ++++++++++++++++++++++++++++++
src/tutorial/subscripting.source | 71 +++++++++++
9 files changed, 470 insertions(+), 4 deletions(-)
create mode 100644 doc/src/sgml/xsubscripting.sgml
create mode 100644 src/tutorial/subscripting.c
create mode 100644 src/tutorial/subscripting.source
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 048ff284f7..b17e9ee382 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8971,6 +8971,14 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>typsubshandler</structfield></entry>
+ <entry><type>regproc</type></entry>
+ <entry><literal><link linkend="catalog-pg-proc"><structname>pg_proc</structname></link>.oid</literal></entry>
+ <entry>Custom subscripting function with type-specific logic for parsing
+ and validation, or 0 if this type doesn't support subscripting.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>typdefault</structfield> <type>text</type>
diff --git a/doc/src/sgml/extend.sgml b/doc/src/sgml/extend.sgml
index 890ff97b7a..65cf5706b6 100644
--- a/doc/src/sgml/extend.sgml
+++ b/doc/src/sgml/extend.sgml
@@ -33,6 +33,11 @@
operators (starting in <xref linkend="xoper"/>)
</para>
</listitem>
+ <listitem>
+ <para>
+ subscripting procedure (starting in <xref linkend="xsubscripting"/>)
+ </para>
+ </listitem>
<listitem>
<para>
operator classes for indexes (starting in <xref linkend="xindex"/>)
@@ -481,6 +486,7 @@ RETURNS anycompatible AS ...
&xaggr;
&xtypes;
&xoper;
+ &xsubscripting;
&xindex;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 64b5da0070..4c3fd37fc1 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -69,6 +69,7 @@
<!ENTITY xplang SYSTEM "xplang.sgml">
<!ENTITY xoper SYSTEM "xoper.sgml">
<!ENTITY xtypes SYSTEM "xtypes.sgml">
+<!ENTITY xsubscripting SYSTEM "xsubscripting.sgml">
<!ENTITY plperl SYSTEM "plperl.sgml">
<!ENTITY plpython SYSTEM "plpython.sgml">
<!ENTITY plsql SYSTEM "plpgsql.sgml">
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index c0a6554d4d..5c538dca05 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -602,6 +602,45 @@ SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qu
</para>
</sect2>
+ <sect2 id="jsonb-subscripting">
+ <title><type>jsonb</type> Subscripting</title>
+ <para>
+ <type>jsonb</type> data type supports array-style subscripting expressions
+ to extract or update particular elements. It's possible to use multiple
+ subscripting expressions to extract nested values. In this case, a chain of
+ subscripting expressions follows the same rules as the
+ <literal>path</literal> argument in <literal>jsonb_set</literal> function,
+ e.g. in case of arrays it is a 0-based operation or that negative integers
+ that appear in <literal>path</literal> count from the end of JSON arrays.
+ The result of subscripting expressions is always jsonb data type. An
+ example of subscripting syntax:
+<programlisting>
+-- Extract value by key
+SELECT ('{"a": 1}'::jsonb)['a'];
+
+-- Extract nested value by key path
+SELECT ('{"a": {"b": {"c": 1}}}'::jsonb)['a']['b']['c'];
+
+-- Extract element by index
+SELECT ('[1, "2", null]'::jsonb)[1];
+
+-- Update value by key
+UPDATE table_name SET jsonb_field['key'] = 1;
+
+-- Select records using where clause with subscripting. Since the result of
+-- subscripting is jsonb and we basically want to compare two jsonb objects, we
+-- need to put the value in double quotes to be able to convert it to jsonb.
+SELECT * FROM table_name WHERE jsonb_field['key'] = '"value"';
+</programlisting>
+
+ There is no special indexing support for such kind of expressions, but you
+ can always create a functional index that includes it
+<programlisting>
+CREATE INDEX idx ON table_name ((jsonb_field['key']));
+</programlisting>
+ </para>
+ </sect2>
+
<sect2>
<title>Transforms</title>
diff --git a/doc/src/sgml/ref/create_type.sgml b/doc/src/sgml/ref/create_type.sgml
index 111f8e65d2..a34df4d247 100644
--- a/doc/src/sgml/ref/create_type.sgml
+++ b/doc/src/sgml/ref/create_type.sgml
@@ -54,6 +54,7 @@ CREATE TYPE <replaceable class="parameter">name</replaceable> (
[ , ELEMENT = <replaceable class="parameter">element</replaceable> ]
[ , DELIMITER = <replaceable class="parameter">delimiter</replaceable> ]
[ , COLLATABLE = <replaceable class="parameter">collatable</replaceable> ]
+ [ , SUBSCRIPTING_HANDLER = <replaceable class="parameter">subscripting_handler_function</replaceable> ]
)
CREATE TYPE <replaceable class="parameter">name</replaceable>
@@ -196,8 +197,9 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
<replaceable class="parameter">receive_function</replaceable>,
<replaceable class="parameter">send_function</replaceable>,
<replaceable class="parameter">type_modifier_input_function</replaceable>,
- <replaceable class="parameter">type_modifier_output_function</replaceable> and
- <replaceable class="parameter">analyze_function</replaceable>
+ <replaceable class="parameter">type_modifier_output_function</replaceable>,
+ <replaceable class="parameter">analyze_function</replaceable>,
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
are optional. Generally these functions have to be coded in C
or another low-level language.
</para>
@@ -454,6 +456,22 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
make use of the collation information; this does not happen
automatically merely by marking the type collatable.
</para>
+
+ <para>
+ The optional
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
+ contains type-specific logic for subscripting of the data type.
+ By default, there is no such function provided, which means that the data
+ type doesn't support subscripting. The subscripting function must be
+ declared to take a single argument of type <type>internal</type>, and return
+ a <type>internal</type> result. There are two examples of implementation for
+ subscripting functions in case of array
+ (<replaceable class="parameter">array_subscripting_handler</replaceable>)
+ and jsonb
+ (<replaceable class="parameter">jsonb_subscripting_handler</replaceable>)
+ types in <filename>src/backend/utils/adt/arrayfuncs.c</filename> and
+ <filename>src/backend/utils/adt/jsonfuncs.c</filename>, respectively.
+ </para>
</refsect2>
<refsect2>
@@ -769,6 +787,17 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">subscripting_handler_function</replaceable></term>
+ <listitem>
+ <para>
+ The name of a function that returns list of type-specific callback functions to
+ support subscripting logic for the data type.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
diff --git a/doc/src/sgml/xsubscripting.sgml b/doc/src/sgml/xsubscripting.sgml
new file mode 100644
index 0000000000..7224e81fa2
--- /dev/null
+++ b/doc/src/sgml/xsubscripting.sgml
@@ -0,0 +1,111 @@
+<!-- doc/src/sgml/xsubscripting.sgml -->
+
+ <sect1 id="xsubscripting">
+ <title>User-defined subscripting procedure</title>
+
+ <indexterm zone="xsubscripting">
+ <primary>custom subscripting</primary>
+ </indexterm>
+ <para>
+ When you define a new base type, you can also specify a custom procedure to
+ handle subscripting expressions. It must contain logic for verification and
+ evaluation of this expression, i.e. fetching or updating some data in this
+ data type. For instance:
+</para>
+<programlisting><![CDATA[
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = someType;
+ sbsref->refassgntype = someType;
+
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ // some validation and coercion logic
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some assignment logic
+
+ return newContainer;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some fetch logic based on sbstate
+}]]>
+</programlisting>
+
+<para>
+ Then you can define a subscripting procedure and a custom data type:
+</para>
+<programlisting>
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '<replaceable>filename</replaceable>'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 4,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler,
+);
+</programlisting>
+
+<para>
+ and use it as usual:
+</para>
+<programlisting>
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
+</programlisting>
+
+
+ <para>
+ The examples of custom subscripting implementation can be found in
+ <filename>subscripting.sql</filename> and <filename>subscripting.c</filename>
+ in the <filename>src/tutorial</filename> directory of the source distribution.
+ See the <filename>README</filename> file in that directory for instructions
+ about running the examples.
+ </para>
+
+</sect1>
diff --git a/src/tutorial/Makefile b/src/tutorial/Makefile
index 16dc390f71..0ead60c2d4 100644
--- a/src/tutorial/Makefile
+++ b/src/tutorial/Makefile
@@ -13,8 +13,8 @@
#
#-------------------------------------------------------------------------
-MODULES = complex funcs
-DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql
+MODULES = complex funcs subscripting
+DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql subscripting.sql
ifdef NO_PGXS
subdir = src/tutorial
diff --git a/src/tutorial/subscripting.c b/src/tutorial/subscripting.c
new file mode 100644
index 0000000000..1eb8c45652
--- /dev/null
+++ b/src/tutorial/subscripting.c
@@ -0,0 +1,201 @@
+/*
+ * src/tutorial/subscripting.c
+ *
+ ******************************************************************************
+ This file contains routines that can be bound to a Postgres backend and
+ called by the backend in the process of processing queries. The calling
+ format for these routines is dictated by Postgres architecture.
+******************************************************************************/
+
+#include "postgres.h"
+
+#include "catalog/pg_type.h"
+#include "executor/executor.h"
+#include "executor/execExpr.h"
+#include "nodes/nodeFuncs.h"
+#include "parser/parse_coerce.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+SubscriptingRef * custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref);
+SubscriptingRef * custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate);
+Datum custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate);
+Datum custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate);
+
+PG_FUNCTION_INFO_V1(custom_in);
+PG_FUNCTION_INFO_V1(custom_out);
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+/*****************************************************************************
+ * Input/Output functions
+ *****************************************************************************/
+
+Datum
+custom_in(PG_FUNCTION_ARGS)
+{
+ char *str = PG_GETARG_CSTRING(0);
+ int firstValue,
+ secondValue;
+ Custom *result;
+
+ if (sscanf(str, " ( %d , %d )", &firstValue, &secondValue) != 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
+ errmsg("invalid input syntax for complex: \"%s\"",
+ str)));
+
+
+ result = (Custom *) palloc(sizeof(Custom));
+ result->first = firstValue;
+ result->second = secondValue;
+ PG_RETURN_POINTER(result);
+}
+
+Datum
+custom_out(PG_FUNCTION_ARGS)
+{
+ Custom *custom = (Custom *) PG_GETARG_POINTER(0);
+ char *result;
+
+ result = psprintf("(%d, %d)", custom->first, custom->second);
+ PG_RETURN_CSTRING(result);
+}
+
+/*****************************************************************************
+ * Custom subscripting logic functions
+ *****************************************************************************/
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = INT4OID;
+ sbsref->refassgntype = INT4OID;
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ List *upperIndexpr = NIL;
+ ListCell *l;
+
+ if (sbsref->reflowerindexpr != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *)lfirst(sbsref->reflowerindexpr->head))))));
+
+ foreach(l, sbsref->refupperindexpr)
+ {
+ Node *subexpr = (Node *) lfirst(l);
+
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *) lfirst(sbsref->refupperindexpr->head))))));
+
+ subexpr = coerce_to_target_type(pstate,
+ subexpr, exprType(subexpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript must have integer type"),
+ parser_errposition(pstate, exprLocation(subexpr))));
+
+ upperIndexpr = lappend(upperIndexpr, subexpr);
+
+ if (isAssignment)
+ {
+ Node *assignExpr = (Node *) sbsref->refassgnexpr;
+ Node *new_from;
+
+ new_from = coerce_to_target_type(pstate,
+ assignExpr, exprType(assignExpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (new_from == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom assignment requires int type"),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation(assignExpr))));
+ sbsref->refassgnexpr = (Expr *)new_from;
+ }
+ }
+
+ sbsref->refupperindexpr = upperIndexpr;
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ Custom *container= (Custom *) containerSource;
+ int index;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ return (Datum) container->first;
+ else
+ return (Datum) container->second;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ int index;
+ Custom *container = (Custom *) containerSource;
+
+ if (sbstate->resnull)
+ return containerSource;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ container->first = DatumGetInt32(sbstate->replacevalue);
+ else
+ container->second = DatumGetInt32(sbstate->replacevalue);
+
+ return (Datum) container;
+}
diff --git a/src/tutorial/subscripting.source b/src/tutorial/subscripting.source
new file mode 100644
index 0000000000..837cf30612
--- /dev/null
+++ b/src/tutorial/subscripting.source
@@ -0,0 +1,71 @@
+---------------------------------------------------------------------------
+--
+-- subscripting.sql-
+-- This file shows how to create a new subscripting procedure for
+-- user-defined type.
+--
+--
+-- Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+-- Portions Copyright (c) 1994, Regents of the University of California
+--
+-- src/tutorial/subscripting.source
+--
+---------------------------------------------------------------------------
+
+-----------------------------
+-- Creating a new type:
+-- We are going to create a new type called 'complex' which represents
+-- complex numbers.
+-- A user-defined type must have an input and an output function, and
+-- optionally can have binary input and output functions. All of these
+-- are usually user-defined C functions.
+-----------------------------
+
+-- Assume the user defined functions are in /home/erthalion/programms/postgresql-master/src/tutorial/complex$DLSUFFIX
+-- (we do not want to assume this is in the dynamic loader search path).
+-- Look at $PWD/complex.c for the source. Note that we declare all of
+-- them as STRICT, so we do not need to cope with NULL inputs in the
+-- C code. We also mark them IMMUTABLE, since they always return the
+-- same outputs given the same inputs.
+
+-- the input function 'complex_in' takes a null-terminated string (the
+-- textual representation of the type) and turns it into the internal
+-- (in memory) representation. You will get a message telling you 'complex'
+-- does not exist yet but that's okay.
+
+CREATE FUNCTION custom_in(cstring)
+ RETURNS custom
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+-- the output function 'complex_out' takes the internal representation and
+-- converts it into the textual representation.
+
+CREATE FUNCTION custom_out(custom)
+ RETURNS cstring
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 8,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler
+);
+
+-- we can use it in a table
+
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
--
2.21.0
--u66dlpg6gaevsofr
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v33-0005-Filling-gaps-in-jsonb-arrays.patch"
^ permalink raw reply [nested|flat] 55+ messages in thread
* [PATCH v33 4/5] Subscripting documentation
@ 2019-02-01 10:47 erthalion <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: erthalion @ 2019-02-01 10:47 UTC (permalink / raw)
Supporting documentation for generalized subscripting. It includes the
description of a new field in pg_type, the new section for jsonb
documentation about subscripting feature on this data type, and also the
tutorial about how to write subscripting operator for a custom data type.
Reviewed-by: Tom Lane, Arthur Zakirov, Oleksandr Shulgin
---
doc/src/sgml/catalogs.sgml | 8 ++
doc/src/sgml/extend.sgml | 6 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/json.sgml | 39 ++++++
doc/src/sgml/ref/create_type.sgml | 33 ++++-
doc/src/sgml/xsubscripting.sgml | 111 +++++++++++++++++
src/tutorial/Makefile | 4 +-
src/tutorial/subscripting.c | 201 ++++++++++++++++++++++++++++++
src/tutorial/subscripting.source | 71 +++++++++++
9 files changed, 470 insertions(+), 4 deletions(-)
create mode 100644 doc/src/sgml/xsubscripting.sgml
create mode 100644 src/tutorial/subscripting.c
create mode 100644 src/tutorial/subscripting.source
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 048ff284f7..b17e9ee382 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8971,6 +8971,14 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>typsubshandler</structfield></entry>
+ <entry><type>regproc</type></entry>
+ <entry><literal><link linkend="catalog-pg-proc"><structname>pg_proc</structname></link>.oid</literal></entry>
+ <entry>Custom subscripting function with type-specific logic for parsing
+ and validation, or 0 if this type doesn't support subscripting.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>typdefault</structfield> <type>text</type>
diff --git a/doc/src/sgml/extend.sgml b/doc/src/sgml/extend.sgml
index 890ff97b7a..65cf5706b6 100644
--- a/doc/src/sgml/extend.sgml
+++ b/doc/src/sgml/extend.sgml
@@ -33,6 +33,11 @@
operators (starting in <xref linkend="xoper"/>)
</para>
</listitem>
+ <listitem>
+ <para>
+ subscripting procedure (starting in <xref linkend="xsubscripting"/>)
+ </para>
+ </listitem>
<listitem>
<para>
operator classes for indexes (starting in <xref linkend="xindex"/>)
@@ -481,6 +486,7 @@ RETURNS anycompatible AS ...
&xaggr;
&xtypes;
&xoper;
+ &xsubscripting;
&xindex;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 64b5da0070..4c3fd37fc1 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -69,6 +69,7 @@
<!ENTITY xplang SYSTEM "xplang.sgml">
<!ENTITY xoper SYSTEM "xoper.sgml">
<!ENTITY xtypes SYSTEM "xtypes.sgml">
+<!ENTITY xsubscripting SYSTEM "xsubscripting.sgml">
<!ENTITY plperl SYSTEM "plperl.sgml">
<!ENTITY plpython SYSTEM "plpython.sgml">
<!ENTITY plsql SYSTEM "plpgsql.sgml">
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index c0a6554d4d..5c538dca05 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -602,6 +602,45 @@ SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qu
</para>
</sect2>
+ <sect2 id="jsonb-subscripting">
+ <title><type>jsonb</type> Subscripting</title>
+ <para>
+ <type>jsonb</type> data type supports array-style subscripting expressions
+ to extract or update particular elements. It's possible to use multiple
+ subscripting expressions to extract nested values. In this case, a chain of
+ subscripting expressions follows the same rules as the
+ <literal>path</literal> argument in <literal>jsonb_set</literal> function,
+ e.g. in case of arrays it is a 0-based operation or that negative integers
+ that appear in <literal>path</literal> count from the end of JSON arrays.
+ The result of subscripting expressions is always jsonb data type. An
+ example of subscripting syntax:
+<programlisting>
+-- Extract value by key
+SELECT ('{"a": 1}'::jsonb)['a'];
+
+-- Extract nested value by key path
+SELECT ('{"a": {"b": {"c": 1}}}'::jsonb)['a']['b']['c'];
+
+-- Extract element by index
+SELECT ('[1, "2", null]'::jsonb)[1];
+
+-- Update value by key
+UPDATE table_name SET jsonb_field['key'] = 1;
+
+-- Select records using where clause with subscripting. Since the result of
+-- subscripting is jsonb and we basically want to compare two jsonb objects, we
+-- need to put the value in double quotes to be able to convert it to jsonb.
+SELECT * FROM table_name WHERE jsonb_field['key'] = '"value"';
+</programlisting>
+
+ There is no special indexing support for such kind of expressions, but you
+ can always create a functional index that includes it
+<programlisting>
+CREATE INDEX idx ON table_name ((jsonb_field['key']));
+</programlisting>
+ </para>
+ </sect2>
+
<sect2>
<title>Transforms</title>
diff --git a/doc/src/sgml/ref/create_type.sgml b/doc/src/sgml/ref/create_type.sgml
index 111f8e65d2..a34df4d247 100644
--- a/doc/src/sgml/ref/create_type.sgml
+++ b/doc/src/sgml/ref/create_type.sgml
@@ -54,6 +54,7 @@ CREATE TYPE <replaceable class="parameter">name</replaceable> (
[ , ELEMENT = <replaceable class="parameter">element</replaceable> ]
[ , DELIMITER = <replaceable class="parameter">delimiter</replaceable> ]
[ , COLLATABLE = <replaceable class="parameter">collatable</replaceable> ]
+ [ , SUBSCRIPTING_HANDLER = <replaceable class="parameter">subscripting_handler_function</replaceable> ]
)
CREATE TYPE <replaceable class="parameter">name</replaceable>
@@ -196,8 +197,9 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
<replaceable class="parameter">receive_function</replaceable>,
<replaceable class="parameter">send_function</replaceable>,
<replaceable class="parameter">type_modifier_input_function</replaceable>,
- <replaceable class="parameter">type_modifier_output_function</replaceable> and
- <replaceable class="parameter">analyze_function</replaceable>
+ <replaceable class="parameter">type_modifier_output_function</replaceable>,
+ <replaceable class="parameter">analyze_function</replaceable>,
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
are optional. Generally these functions have to be coded in C
or another low-level language.
</para>
@@ -454,6 +456,22 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
make use of the collation information; this does not happen
automatically merely by marking the type collatable.
</para>
+
+ <para>
+ The optional
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
+ contains type-specific logic for subscripting of the data type.
+ By default, there is no such function provided, which means that the data
+ type doesn't support subscripting. The subscripting function must be
+ declared to take a single argument of type <type>internal</type>, and return
+ a <type>internal</type> result. There are two examples of implementation for
+ subscripting functions in case of array
+ (<replaceable class="parameter">array_subscripting_handler</replaceable>)
+ and jsonb
+ (<replaceable class="parameter">jsonb_subscripting_handler</replaceable>)
+ types in <filename>src/backend/utils/adt/arrayfuncs.c</filename> and
+ <filename>src/backend/utils/adt/jsonfuncs.c</filename>, respectively.
+ </para>
</refsect2>
<refsect2>
@@ -769,6 +787,17 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">subscripting_handler_function</replaceable></term>
+ <listitem>
+ <para>
+ The name of a function that returns list of type-specific callback functions to
+ support subscripting logic for the data type.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
diff --git a/doc/src/sgml/xsubscripting.sgml b/doc/src/sgml/xsubscripting.sgml
new file mode 100644
index 0000000000..7224e81fa2
--- /dev/null
+++ b/doc/src/sgml/xsubscripting.sgml
@@ -0,0 +1,111 @@
+<!-- doc/src/sgml/xsubscripting.sgml -->
+
+ <sect1 id="xsubscripting">
+ <title>User-defined subscripting procedure</title>
+
+ <indexterm zone="xsubscripting">
+ <primary>custom subscripting</primary>
+ </indexterm>
+ <para>
+ When you define a new base type, you can also specify a custom procedure to
+ handle subscripting expressions. It must contain logic for verification and
+ evaluation of this expression, i.e. fetching or updating some data in this
+ data type. For instance:
+</para>
+<programlisting><![CDATA[
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = someType;
+ sbsref->refassgntype = someType;
+
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ // some validation and coercion logic
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some assignment logic
+
+ return newContainer;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some fetch logic based on sbstate
+}]]>
+</programlisting>
+
+<para>
+ Then you can define a subscripting procedure and a custom data type:
+</para>
+<programlisting>
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '<replaceable>filename</replaceable>'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 4,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler,
+);
+</programlisting>
+
+<para>
+ and use it as usual:
+</para>
+<programlisting>
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
+</programlisting>
+
+
+ <para>
+ The examples of custom subscripting implementation can be found in
+ <filename>subscripting.sql</filename> and <filename>subscripting.c</filename>
+ in the <filename>src/tutorial</filename> directory of the source distribution.
+ See the <filename>README</filename> file in that directory for instructions
+ about running the examples.
+ </para>
+
+</sect1>
diff --git a/src/tutorial/Makefile b/src/tutorial/Makefile
index 16dc390f71..0ead60c2d4 100644
--- a/src/tutorial/Makefile
+++ b/src/tutorial/Makefile
@@ -13,8 +13,8 @@
#
#-------------------------------------------------------------------------
-MODULES = complex funcs
-DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql
+MODULES = complex funcs subscripting
+DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql subscripting.sql
ifdef NO_PGXS
subdir = src/tutorial
diff --git a/src/tutorial/subscripting.c b/src/tutorial/subscripting.c
new file mode 100644
index 0000000000..1eb8c45652
--- /dev/null
+++ b/src/tutorial/subscripting.c
@@ -0,0 +1,201 @@
+/*
+ * src/tutorial/subscripting.c
+ *
+ ******************************************************************************
+ This file contains routines that can be bound to a Postgres backend and
+ called by the backend in the process of processing queries. The calling
+ format for these routines is dictated by Postgres architecture.
+******************************************************************************/
+
+#include "postgres.h"
+
+#include "catalog/pg_type.h"
+#include "executor/executor.h"
+#include "executor/execExpr.h"
+#include "nodes/nodeFuncs.h"
+#include "parser/parse_coerce.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+SubscriptingRef * custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref);
+SubscriptingRef * custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate);
+Datum custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate);
+Datum custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate);
+
+PG_FUNCTION_INFO_V1(custom_in);
+PG_FUNCTION_INFO_V1(custom_out);
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+/*****************************************************************************
+ * Input/Output functions
+ *****************************************************************************/
+
+Datum
+custom_in(PG_FUNCTION_ARGS)
+{
+ char *str = PG_GETARG_CSTRING(0);
+ int firstValue,
+ secondValue;
+ Custom *result;
+
+ if (sscanf(str, " ( %d , %d )", &firstValue, &secondValue) != 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
+ errmsg("invalid input syntax for complex: \"%s\"",
+ str)));
+
+
+ result = (Custom *) palloc(sizeof(Custom));
+ result->first = firstValue;
+ result->second = secondValue;
+ PG_RETURN_POINTER(result);
+}
+
+Datum
+custom_out(PG_FUNCTION_ARGS)
+{
+ Custom *custom = (Custom *) PG_GETARG_POINTER(0);
+ char *result;
+
+ result = psprintf("(%d, %d)", custom->first, custom->second);
+ PG_RETURN_CSTRING(result);
+}
+
+/*****************************************************************************
+ * Custom subscripting logic functions
+ *****************************************************************************/
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = INT4OID;
+ sbsref->refassgntype = INT4OID;
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ List *upperIndexpr = NIL;
+ ListCell *l;
+
+ if (sbsref->reflowerindexpr != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *)lfirst(sbsref->reflowerindexpr->head))))));
+
+ foreach(l, sbsref->refupperindexpr)
+ {
+ Node *subexpr = (Node *) lfirst(l);
+
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *) lfirst(sbsref->refupperindexpr->head))))));
+
+ subexpr = coerce_to_target_type(pstate,
+ subexpr, exprType(subexpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript must have integer type"),
+ parser_errposition(pstate, exprLocation(subexpr))));
+
+ upperIndexpr = lappend(upperIndexpr, subexpr);
+
+ if (isAssignment)
+ {
+ Node *assignExpr = (Node *) sbsref->refassgnexpr;
+ Node *new_from;
+
+ new_from = coerce_to_target_type(pstate,
+ assignExpr, exprType(assignExpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (new_from == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom assignment requires int type"),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation(assignExpr))));
+ sbsref->refassgnexpr = (Expr *)new_from;
+ }
+ }
+
+ sbsref->refupperindexpr = upperIndexpr;
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ Custom *container= (Custom *) containerSource;
+ int index;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ return (Datum) container->first;
+ else
+ return (Datum) container->second;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ int index;
+ Custom *container = (Custom *) containerSource;
+
+ if (sbstate->resnull)
+ return containerSource;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ container->first = DatumGetInt32(sbstate->replacevalue);
+ else
+ container->second = DatumGetInt32(sbstate->replacevalue);
+
+ return (Datum) container;
+}
diff --git a/src/tutorial/subscripting.source b/src/tutorial/subscripting.source
new file mode 100644
index 0000000000..837cf30612
--- /dev/null
+++ b/src/tutorial/subscripting.source
@@ -0,0 +1,71 @@
+---------------------------------------------------------------------------
+--
+-- subscripting.sql-
+-- This file shows how to create a new subscripting procedure for
+-- user-defined type.
+--
+--
+-- Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+-- Portions Copyright (c) 1994, Regents of the University of California
+--
+-- src/tutorial/subscripting.source
+--
+---------------------------------------------------------------------------
+
+-----------------------------
+-- Creating a new type:
+-- We are going to create a new type called 'complex' which represents
+-- complex numbers.
+-- A user-defined type must have an input and an output function, and
+-- optionally can have binary input and output functions. All of these
+-- are usually user-defined C functions.
+-----------------------------
+
+-- Assume the user defined functions are in /home/erthalion/programms/postgresql-master/src/tutorial/complex$DLSUFFIX
+-- (we do not want to assume this is in the dynamic loader search path).
+-- Look at $PWD/complex.c for the source. Note that we declare all of
+-- them as STRICT, so we do not need to cope with NULL inputs in the
+-- C code. We also mark them IMMUTABLE, since they always return the
+-- same outputs given the same inputs.
+
+-- the input function 'complex_in' takes a null-terminated string (the
+-- textual representation of the type) and turns it into the internal
+-- (in memory) representation. You will get a message telling you 'complex'
+-- does not exist yet but that's okay.
+
+CREATE FUNCTION custom_in(cstring)
+ RETURNS custom
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+-- the output function 'complex_out' takes the internal representation and
+-- converts it into the textual representation.
+
+CREATE FUNCTION custom_out(custom)
+ RETURNS cstring
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 8,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler
+);
+
+-- we can use it in a table
+
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
--
2.21.0
--u66dlpg6gaevsofr
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v33-0005-Filling-gaps-in-jsonb-arrays.patch"
^ permalink raw reply [nested|flat] 55+ messages in thread
* [PATCH v33 4/5] Subscripting documentation
@ 2019-02-01 10:47 erthalion <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: erthalion @ 2019-02-01 10:47 UTC (permalink / raw)
Supporting documentation for generalized subscripting. It includes the
description of a new field in pg_type, the new section for jsonb
documentation about subscripting feature on this data type, and also the
tutorial about how to write subscripting operator for a custom data type.
Reviewed-by: Tom Lane, Arthur Zakirov, Oleksandr Shulgin
---
doc/src/sgml/catalogs.sgml | 8 ++
doc/src/sgml/extend.sgml | 6 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/json.sgml | 39 ++++++
doc/src/sgml/ref/create_type.sgml | 33 ++++-
doc/src/sgml/xsubscripting.sgml | 111 +++++++++++++++++
src/tutorial/Makefile | 4 +-
src/tutorial/subscripting.c | 201 ++++++++++++++++++++++++++++++
src/tutorial/subscripting.source | 71 +++++++++++
9 files changed, 470 insertions(+), 4 deletions(-)
create mode 100644 doc/src/sgml/xsubscripting.sgml
create mode 100644 src/tutorial/subscripting.c
create mode 100644 src/tutorial/subscripting.source
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 048ff284f7..b17e9ee382 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8971,6 +8971,14 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>typsubshandler</structfield></entry>
+ <entry><type>regproc</type></entry>
+ <entry><literal><link linkend="catalog-pg-proc"><structname>pg_proc</structname></link>.oid</literal></entry>
+ <entry>Custom subscripting function with type-specific logic for parsing
+ and validation, or 0 if this type doesn't support subscripting.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>typdefault</structfield> <type>text</type>
diff --git a/doc/src/sgml/extend.sgml b/doc/src/sgml/extend.sgml
index 890ff97b7a..65cf5706b6 100644
--- a/doc/src/sgml/extend.sgml
+++ b/doc/src/sgml/extend.sgml
@@ -33,6 +33,11 @@
operators (starting in <xref linkend="xoper"/>)
</para>
</listitem>
+ <listitem>
+ <para>
+ subscripting procedure (starting in <xref linkend="xsubscripting"/>)
+ </para>
+ </listitem>
<listitem>
<para>
operator classes for indexes (starting in <xref linkend="xindex"/>)
@@ -481,6 +486,7 @@ RETURNS anycompatible AS ...
&xaggr;
&xtypes;
&xoper;
+ &xsubscripting;
&xindex;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 64b5da0070..4c3fd37fc1 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -69,6 +69,7 @@
<!ENTITY xplang SYSTEM "xplang.sgml">
<!ENTITY xoper SYSTEM "xoper.sgml">
<!ENTITY xtypes SYSTEM "xtypes.sgml">
+<!ENTITY xsubscripting SYSTEM "xsubscripting.sgml">
<!ENTITY plperl SYSTEM "plperl.sgml">
<!ENTITY plpython SYSTEM "plpython.sgml">
<!ENTITY plsql SYSTEM "plpgsql.sgml">
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index c0a6554d4d..5c538dca05 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -602,6 +602,45 @@ SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qu
</para>
</sect2>
+ <sect2 id="jsonb-subscripting">
+ <title><type>jsonb</type> Subscripting</title>
+ <para>
+ <type>jsonb</type> data type supports array-style subscripting expressions
+ to extract or update particular elements. It's possible to use multiple
+ subscripting expressions to extract nested values. In this case, a chain of
+ subscripting expressions follows the same rules as the
+ <literal>path</literal> argument in <literal>jsonb_set</literal> function,
+ e.g. in case of arrays it is a 0-based operation or that negative integers
+ that appear in <literal>path</literal> count from the end of JSON arrays.
+ The result of subscripting expressions is always jsonb data type. An
+ example of subscripting syntax:
+<programlisting>
+-- Extract value by key
+SELECT ('{"a": 1}'::jsonb)['a'];
+
+-- Extract nested value by key path
+SELECT ('{"a": {"b": {"c": 1}}}'::jsonb)['a']['b']['c'];
+
+-- Extract element by index
+SELECT ('[1, "2", null]'::jsonb)[1];
+
+-- Update value by key
+UPDATE table_name SET jsonb_field['key'] = 1;
+
+-- Select records using where clause with subscripting. Since the result of
+-- subscripting is jsonb and we basically want to compare two jsonb objects, we
+-- need to put the value in double quotes to be able to convert it to jsonb.
+SELECT * FROM table_name WHERE jsonb_field['key'] = '"value"';
+</programlisting>
+
+ There is no special indexing support for such kind of expressions, but you
+ can always create a functional index that includes it
+<programlisting>
+CREATE INDEX idx ON table_name ((jsonb_field['key']));
+</programlisting>
+ </para>
+ </sect2>
+
<sect2>
<title>Transforms</title>
diff --git a/doc/src/sgml/ref/create_type.sgml b/doc/src/sgml/ref/create_type.sgml
index 111f8e65d2..a34df4d247 100644
--- a/doc/src/sgml/ref/create_type.sgml
+++ b/doc/src/sgml/ref/create_type.sgml
@@ -54,6 +54,7 @@ CREATE TYPE <replaceable class="parameter">name</replaceable> (
[ , ELEMENT = <replaceable class="parameter">element</replaceable> ]
[ , DELIMITER = <replaceable class="parameter">delimiter</replaceable> ]
[ , COLLATABLE = <replaceable class="parameter">collatable</replaceable> ]
+ [ , SUBSCRIPTING_HANDLER = <replaceable class="parameter">subscripting_handler_function</replaceable> ]
)
CREATE TYPE <replaceable class="parameter">name</replaceable>
@@ -196,8 +197,9 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
<replaceable class="parameter">receive_function</replaceable>,
<replaceable class="parameter">send_function</replaceable>,
<replaceable class="parameter">type_modifier_input_function</replaceable>,
- <replaceable class="parameter">type_modifier_output_function</replaceable> and
- <replaceable class="parameter">analyze_function</replaceable>
+ <replaceable class="parameter">type_modifier_output_function</replaceable>,
+ <replaceable class="parameter">analyze_function</replaceable>,
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
are optional. Generally these functions have to be coded in C
or another low-level language.
</para>
@@ -454,6 +456,22 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
make use of the collation information; this does not happen
automatically merely by marking the type collatable.
</para>
+
+ <para>
+ The optional
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
+ contains type-specific logic for subscripting of the data type.
+ By default, there is no such function provided, which means that the data
+ type doesn't support subscripting. The subscripting function must be
+ declared to take a single argument of type <type>internal</type>, and return
+ a <type>internal</type> result. There are two examples of implementation for
+ subscripting functions in case of array
+ (<replaceable class="parameter">array_subscripting_handler</replaceable>)
+ and jsonb
+ (<replaceable class="parameter">jsonb_subscripting_handler</replaceable>)
+ types in <filename>src/backend/utils/adt/arrayfuncs.c</filename> and
+ <filename>src/backend/utils/adt/jsonfuncs.c</filename>, respectively.
+ </para>
</refsect2>
<refsect2>
@@ -769,6 +787,17 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">subscripting_handler_function</replaceable></term>
+ <listitem>
+ <para>
+ The name of a function that returns list of type-specific callback functions to
+ support subscripting logic for the data type.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
diff --git a/doc/src/sgml/xsubscripting.sgml b/doc/src/sgml/xsubscripting.sgml
new file mode 100644
index 0000000000..7224e81fa2
--- /dev/null
+++ b/doc/src/sgml/xsubscripting.sgml
@@ -0,0 +1,111 @@
+<!-- doc/src/sgml/xsubscripting.sgml -->
+
+ <sect1 id="xsubscripting">
+ <title>User-defined subscripting procedure</title>
+
+ <indexterm zone="xsubscripting">
+ <primary>custom subscripting</primary>
+ </indexterm>
+ <para>
+ When you define a new base type, you can also specify a custom procedure to
+ handle subscripting expressions. It must contain logic for verification and
+ evaluation of this expression, i.e. fetching or updating some data in this
+ data type. For instance:
+</para>
+<programlisting><![CDATA[
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = someType;
+ sbsref->refassgntype = someType;
+
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ // some validation and coercion logic
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some assignment logic
+
+ return newContainer;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some fetch logic based on sbstate
+}]]>
+</programlisting>
+
+<para>
+ Then you can define a subscripting procedure and a custom data type:
+</para>
+<programlisting>
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '<replaceable>filename</replaceable>'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 4,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler,
+);
+</programlisting>
+
+<para>
+ and use it as usual:
+</para>
+<programlisting>
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
+</programlisting>
+
+
+ <para>
+ The examples of custom subscripting implementation can be found in
+ <filename>subscripting.sql</filename> and <filename>subscripting.c</filename>
+ in the <filename>src/tutorial</filename> directory of the source distribution.
+ See the <filename>README</filename> file in that directory for instructions
+ about running the examples.
+ </para>
+
+</sect1>
diff --git a/src/tutorial/Makefile b/src/tutorial/Makefile
index 16dc390f71..0ead60c2d4 100644
--- a/src/tutorial/Makefile
+++ b/src/tutorial/Makefile
@@ -13,8 +13,8 @@
#
#-------------------------------------------------------------------------
-MODULES = complex funcs
-DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql
+MODULES = complex funcs subscripting
+DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql subscripting.sql
ifdef NO_PGXS
subdir = src/tutorial
diff --git a/src/tutorial/subscripting.c b/src/tutorial/subscripting.c
new file mode 100644
index 0000000000..1eb8c45652
--- /dev/null
+++ b/src/tutorial/subscripting.c
@@ -0,0 +1,201 @@
+/*
+ * src/tutorial/subscripting.c
+ *
+ ******************************************************************************
+ This file contains routines that can be bound to a Postgres backend and
+ called by the backend in the process of processing queries. The calling
+ format for these routines is dictated by Postgres architecture.
+******************************************************************************/
+
+#include "postgres.h"
+
+#include "catalog/pg_type.h"
+#include "executor/executor.h"
+#include "executor/execExpr.h"
+#include "nodes/nodeFuncs.h"
+#include "parser/parse_coerce.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+SubscriptingRef * custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref);
+SubscriptingRef * custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate);
+Datum custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate);
+Datum custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate);
+
+PG_FUNCTION_INFO_V1(custom_in);
+PG_FUNCTION_INFO_V1(custom_out);
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+/*****************************************************************************
+ * Input/Output functions
+ *****************************************************************************/
+
+Datum
+custom_in(PG_FUNCTION_ARGS)
+{
+ char *str = PG_GETARG_CSTRING(0);
+ int firstValue,
+ secondValue;
+ Custom *result;
+
+ if (sscanf(str, " ( %d , %d )", &firstValue, &secondValue) != 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
+ errmsg("invalid input syntax for complex: \"%s\"",
+ str)));
+
+
+ result = (Custom *) palloc(sizeof(Custom));
+ result->first = firstValue;
+ result->second = secondValue;
+ PG_RETURN_POINTER(result);
+}
+
+Datum
+custom_out(PG_FUNCTION_ARGS)
+{
+ Custom *custom = (Custom *) PG_GETARG_POINTER(0);
+ char *result;
+
+ result = psprintf("(%d, %d)", custom->first, custom->second);
+ PG_RETURN_CSTRING(result);
+}
+
+/*****************************************************************************
+ * Custom subscripting logic functions
+ *****************************************************************************/
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = INT4OID;
+ sbsref->refassgntype = INT4OID;
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ List *upperIndexpr = NIL;
+ ListCell *l;
+
+ if (sbsref->reflowerindexpr != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *)lfirst(sbsref->reflowerindexpr->head))))));
+
+ foreach(l, sbsref->refupperindexpr)
+ {
+ Node *subexpr = (Node *) lfirst(l);
+
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *) lfirst(sbsref->refupperindexpr->head))))));
+
+ subexpr = coerce_to_target_type(pstate,
+ subexpr, exprType(subexpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript must have integer type"),
+ parser_errposition(pstate, exprLocation(subexpr))));
+
+ upperIndexpr = lappend(upperIndexpr, subexpr);
+
+ if (isAssignment)
+ {
+ Node *assignExpr = (Node *) sbsref->refassgnexpr;
+ Node *new_from;
+
+ new_from = coerce_to_target_type(pstate,
+ assignExpr, exprType(assignExpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (new_from == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom assignment requires int type"),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation(assignExpr))));
+ sbsref->refassgnexpr = (Expr *)new_from;
+ }
+ }
+
+ sbsref->refupperindexpr = upperIndexpr;
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ Custom *container= (Custom *) containerSource;
+ int index;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ return (Datum) container->first;
+ else
+ return (Datum) container->second;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ int index;
+ Custom *container = (Custom *) containerSource;
+
+ if (sbstate->resnull)
+ return containerSource;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ container->first = DatumGetInt32(sbstate->replacevalue);
+ else
+ container->second = DatumGetInt32(sbstate->replacevalue);
+
+ return (Datum) container;
+}
diff --git a/src/tutorial/subscripting.source b/src/tutorial/subscripting.source
new file mode 100644
index 0000000000..837cf30612
--- /dev/null
+++ b/src/tutorial/subscripting.source
@@ -0,0 +1,71 @@
+---------------------------------------------------------------------------
+--
+-- subscripting.sql-
+-- This file shows how to create a new subscripting procedure for
+-- user-defined type.
+--
+--
+-- Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+-- Portions Copyright (c) 1994, Regents of the University of California
+--
+-- src/tutorial/subscripting.source
+--
+---------------------------------------------------------------------------
+
+-----------------------------
+-- Creating a new type:
+-- We are going to create a new type called 'complex' which represents
+-- complex numbers.
+-- A user-defined type must have an input and an output function, and
+-- optionally can have binary input and output functions. All of these
+-- are usually user-defined C functions.
+-----------------------------
+
+-- Assume the user defined functions are in /home/erthalion/programms/postgresql-master/src/tutorial/complex$DLSUFFIX
+-- (we do not want to assume this is in the dynamic loader search path).
+-- Look at $PWD/complex.c for the source. Note that we declare all of
+-- them as STRICT, so we do not need to cope with NULL inputs in the
+-- C code. We also mark them IMMUTABLE, since they always return the
+-- same outputs given the same inputs.
+
+-- the input function 'complex_in' takes a null-terminated string (the
+-- textual representation of the type) and turns it into the internal
+-- (in memory) representation. You will get a message telling you 'complex'
+-- does not exist yet but that's okay.
+
+CREATE FUNCTION custom_in(cstring)
+ RETURNS custom
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+-- the output function 'complex_out' takes the internal representation and
+-- converts it into the textual representation.
+
+CREATE FUNCTION custom_out(custom)
+ RETURNS cstring
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 8,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler
+);
+
+-- we can use it in a table
+
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
--
2.21.0
--u66dlpg6gaevsofr
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v33-0005-Filling-gaps-in-jsonb-arrays.patch"
^ permalink raw reply [nested|flat] 55+ messages in thread
* [PATCH v33 4/5] Subscripting documentation
@ 2019-02-01 10:47 erthalion <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: erthalion @ 2019-02-01 10:47 UTC (permalink / raw)
Supporting documentation for generalized subscripting. It includes the
description of a new field in pg_type, the new section for jsonb
documentation about subscripting feature on this data type, and also the
tutorial about how to write subscripting operator for a custom data type.
Reviewed-by: Tom Lane, Arthur Zakirov, Oleksandr Shulgin
---
doc/src/sgml/catalogs.sgml | 8 ++
doc/src/sgml/extend.sgml | 6 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/json.sgml | 39 ++++++
doc/src/sgml/ref/create_type.sgml | 33 ++++-
doc/src/sgml/xsubscripting.sgml | 111 +++++++++++++++++
src/tutorial/Makefile | 4 +-
src/tutorial/subscripting.c | 201 ++++++++++++++++++++++++++++++
src/tutorial/subscripting.source | 71 +++++++++++
9 files changed, 470 insertions(+), 4 deletions(-)
create mode 100644 doc/src/sgml/xsubscripting.sgml
create mode 100644 src/tutorial/subscripting.c
create mode 100644 src/tutorial/subscripting.source
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 048ff284f7..b17e9ee382 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8971,6 +8971,14 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>typsubshandler</structfield></entry>
+ <entry><type>regproc</type></entry>
+ <entry><literal><link linkend="catalog-pg-proc"><structname>pg_proc</structname></link>.oid</literal></entry>
+ <entry>Custom subscripting function with type-specific logic for parsing
+ and validation, or 0 if this type doesn't support subscripting.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>typdefault</structfield> <type>text</type>
diff --git a/doc/src/sgml/extend.sgml b/doc/src/sgml/extend.sgml
index 890ff97b7a..65cf5706b6 100644
--- a/doc/src/sgml/extend.sgml
+++ b/doc/src/sgml/extend.sgml
@@ -33,6 +33,11 @@
operators (starting in <xref linkend="xoper"/>)
</para>
</listitem>
+ <listitem>
+ <para>
+ subscripting procedure (starting in <xref linkend="xsubscripting"/>)
+ </para>
+ </listitem>
<listitem>
<para>
operator classes for indexes (starting in <xref linkend="xindex"/>)
@@ -481,6 +486,7 @@ RETURNS anycompatible AS ...
&xaggr;
&xtypes;
&xoper;
+ &xsubscripting;
&xindex;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 64b5da0070..4c3fd37fc1 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -69,6 +69,7 @@
<!ENTITY xplang SYSTEM "xplang.sgml">
<!ENTITY xoper SYSTEM "xoper.sgml">
<!ENTITY xtypes SYSTEM "xtypes.sgml">
+<!ENTITY xsubscripting SYSTEM "xsubscripting.sgml">
<!ENTITY plperl SYSTEM "plperl.sgml">
<!ENTITY plpython SYSTEM "plpython.sgml">
<!ENTITY plsql SYSTEM "plpgsql.sgml">
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index c0a6554d4d..5c538dca05 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -602,6 +602,45 @@ SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qu
</para>
</sect2>
+ <sect2 id="jsonb-subscripting">
+ <title><type>jsonb</type> Subscripting</title>
+ <para>
+ <type>jsonb</type> data type supports array-style subscripting expressions
+ to extract or update particular elements. It's possible to use multiple
+ subscripting expressions to extract nested values. In this case, a chain of
+ subscripting expressions follows the same rules as the
+ <literal>path</literal> argument in <literal>jsonb_set</literal> function,
+ e.g. in case of arrays it is a 0-based operation or that negative integers
+ that appear in <literal>path</literal> count from the end of JSON arrays.
+ The result of subscripting expressions is always jsonb data type. An
+ example of subscripting syntax:
+<programlisting>
+-- Extract value by key
+SELECT ('{"a": 1}'::jsonb)['a'];
+
+-- Extract nested value by key path
+SELECT ('{"a": {"b": {"c": 1}}}'::jsonb)['a']['b']['c'];
+
+-- Extract element by index
+SELECT ('[1, "2", null]'::jsonb)[1];
+
+-- Update value by key
+UPDATE table_name SET jsonb_field['key'] = 1;
+
+-- Select records using where clause with subscripting. Since the result of
+-- subscripting is jsonb and we basically want to compare two jsonb objects, we
+-- need to put the value in double quotes to be able to convert it to jsonb.
+SELECT * FROM table_name WHERE jsonb_field['key'] = '"value"';
+</programlisting>
+
+ There is no special indexing support for such kind of expressions, but you
+ can always create a functional index that includes it
+<programlisting>
+CREATE INDEX idx ON table_name ((jsonb_field['key']));
+</programlisting>
+ </para>
+ </sect2>
+
<sect2>
<title>Transforms</title>
diff --git a/doc/src/sgml/ref/create_type.sgml b/doc/src/sgml/ref/create_type.sgml
index 111f8e65d2..a34df4d247 100644
--- a/doc/src/sgml/ref/create_type.sgml
+++ b/doc/src/sgml/ref/create_type.sgml
@@ -54,6 +54,7 @@ CREATE TYPE <replaceable class="parameter">name</replaceable> (
[ , ELEMENT = <replaceable class="parameter">element</replaceable> ]
[ , DELIMITER = <replaceable class="parameter">delimiter</replaceable> ]
[ , COLLATABLE = <replaceable class="parameter">collatable</replaceable> ]
+ [ , SUBSCRIPTING_HANDLER = <replaceable class="parameter">subscripting_handler_function</replaceable> ]
)
CREATE TYPE <replaceable class="parameter">name</replaceable>
@@ -196,8 +197,9 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
<replaceable class="parameter">receive_function</replaceable>,
<replaceable class="parameter">send_function</replaceable>,
<replaceable class="parameter">type_modifier_input_function</replaceable>,
- <replaceable class="parameter">type_modifier_output_function</replaceable> and
- <replaceable class="parameter">analyze_function</replaceable>
+ <replaceable class="parameter">type_modifier_output_function</replaceable>,
+ <replaceable class="parameter">analyze_function</replaceable>,
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
are optional. Generally these functions have to be coded in C
or another low-level language.
</para>
@@ -454,6 +456,22 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
make use of the collation information; this does not happen
automatically merely by marking the type collatable.
</para>
+
+ <para>
+ The optional
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
+ contains type-specific logic for subscripting of the data type.
+ By default, there is no such function provided, which means that the data
+ type doesn't support subscripting. The subscripting function must be
+ declared to take a single argument of type <type>internal</type>, and return
+ a <type>internal</type> result. There are two examples of implementation for
+ subscripting functions in case of array
+ (<replaceable class="parameter">array_subscripting_handler</replaceable>)
+ and jsonb
+ (<replaceable class="parameter">jsonb_subscripting_handler</replaceable>)
+ types in <filename>src/backend/utils/adt/arrayfuncs.c</filename> and
+ <filename>src/backend/utils/adt/jsonfuncs.c</filename>, respectively.
+ </para>
</refsect2>
<refsect2>
@@ -769,6 +787,17 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">subscripting_handler_function</replaceable></term>
+ <listitem>
+ <para>
+ The name of a function that returns list of type-specific callback functions to
+ support subscripting logic for the data type.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
diff --git a/doc/src/sgml/xsubscripting.sgml b/doc/src/sgml/xsubscripting.sgml
new file mode 100644
index 0000000000..7224e81fa2
--- /dev/null
+++ b/doc/src/sgml/xsubscripting.sgml
@@ -0,0 +1,111 @@
+<!-- doc/src/sgml/xsubscripting.sgml -->
+
+ <sect1 id="xsubscripting">
+ <title>User-defined subscripting procedure</title>
+
+ <indexterm zone="xsubscripting">
+ <primary>custom subscripting</primary>
+ </indexterm>
+ <para>
+ When you define a new base type, you can also specify a custom procedure to
+ handle subscripting expressions. It must contain logic for verification and
+ evaluation of this expression, i.e. fetching or updating some data in this
+ data type. For instance:
+</para>
+<programlisting><![CDATA[
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = someType;
+ sbsref->refassgntype = someType;
+
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ // some validation and coercion logic
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some assignment logic
+
+ return newContainer;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some fetch logic based on sbstate
+}]]>
+</programlisting>
+
+<para>
+ Then you can define a subscripting procedure and a custom data type:
+</para>
+<programlisting>
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '<replaceable>filename</replaceable>'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 4,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler,
+);
+</programlisting>
+
+<para>
+ and use it as usual:
+</para>
+<programlisting>
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
+</programlisting>
+
+
+ <para>
+ The examples of custom subscripting implementation can be found in
+ <filename>subscripting.sql</filename> and <filename>subscripting.c</filename>
+ in the <filename>src/tutorial</filename> directory of the source distribution.
+ See the <filename>README</filename> file in that directory for instructions
+ about running the examples.
+ </para>
+
+</sect1>
diff --git a/src/tutorial/Makefile b/src/tutorial/Makefile
index 16dc390f71..0ead60c2d4 100644
--- a/src/tutorial/Makefile
+++ b/src/tutorial/Makefile
@@ -13,8 +13,8 @@
#
#-------------------------------------------------------------------------
-MODULES = complex funcs
-DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql
+MODULES = complex funcs subscripting
+DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql subscripting.sql
ifdef NO_PGXS
subdir = src/tutorial
diff --git a/src/tutorial/subscripting.c b/src/tutorial/subscripting.c
new file mode 100644
index 0000000000..1eb8c45652
--- /dev/null
+++ b/src/tutorial/subscripting.c
@@ -0,0 +1,201 @@
+/*
+ * src/tutorial/subscripting.c
+ *
+ ******************************************************************************
+ This file contains routines that can be bound to a Postgres backend and
+ called by the backend in the process of processing queries. The calling
+ format for these routines is dictated by Postgres architecture.
+******************************************************************************/
+
+#include "postgres.h"
+
+#include "catalog/pg_type.h"
+#include "executor/executor.h"
+#include "executor/execExpr.h"
+#include "nodes/nodeFuncs.h"
+#include "parser/parse_coerce.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+SubscriptingRef * custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref);
+SubscriptingRef * custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate);
+Datum custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate);
+Datum custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate);
+
+PG_FUNCTION_INFO_V1(custom_in);
+PG_FUNCTION_INFO_V1(custom_out);
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+/*****************************************************************************
+ * Input/Output functions
+ *****************************************************************************/
+
+Datum
+custom_in(PG_FUNCTION_ARGS)
+{
+ char *str = PG_GETARG_CSTRING(0);
+ int firstValue,
+ secondValue;
+ Custom *result;
+
+ if (sscanf(str, " ( %d , %d )", &firstValue, &secondValue) != 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
+ errmsg("invalid input syntax for complex: \"%s\"",
+ str)));
+
+
+ result = (Custom *) palloc(sizeof(Custom));
+ result->first = firstValue;
+ result->second = secondValue;
+ PG_RETURN_POINTER(result);
+}
+
+Datum
+custom_out(PG_FUNCTION_ARGS)
+{
+ Custom *custom = (Custom *) PG_GETARG_POINTER(0);
+ char *result;
+
+ result = psprintf("(%d, %d)", custom->first, custom->second);
+ PG_RETURN_CSTRING(result);
+}
+
+/*****************************************************************************
+ * Custom subscripting logic functions
+ *****************************************************************************/
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = INT4OID;
+ sbsref->refassgntype = INT4OID;
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ List *upperIndexpr = NIL;
+ ListCell *l;
+
+ if (sbsref->reflowerindexpr != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *)lfirst(sbsref->reflowerindexpr->head))))));
+
+ foreach(l, sbsref->refupperindexpr)
+ {
+ Node *subexpr = (Node *) lfirst(l);
+
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *) lfirst(sbsref->refupperindexpr->head))))));
+
+ subexpr = coerce_to_target_type(pstate,
+ subexpr, exprType(subexpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript must have integer type"),
+ parser_errposition(pstate, exprLocation(subexpr))));
+
+ upperIndexpr = lappend(upperIndexpr, subexpr);
+
+ if (isAssignment)
+ {
+ Node *assignExpr = (Node *) sbsref->refassgnexpr;
+ Node *new_from;
+
+ new_from = coerce_to_target_type(pstate,
+ assignExpr, exprType(assignExpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (new_from == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom assignment requires int type"),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation(assignExpr))));
+ sbsref->refassgnexpr = (Expr *)new_from;
+ }
+ }
+
+ sbsref->refupperindexpr = upperIndexpr;
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ Custom *container= (Custom *) containerSource;
+ int index;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ return (Datum) container->first;
+ else
+ return (Datum) container->second;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ int index;
+ Custom *container = (Custom *) containerSource;
+
+ if (sbstate->resnull)
+ return containerSource;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ container->first = DatumGetInt32(sbstate->replacevalue);
+ else
+ container->second = DatumGetInt32(sbstate->replacevalue);
+
+ return (Datum) container;
+}
diff --git a/src/tutorial/subscripting.source b/src/tutorial/subscripting.source
new file mode 100644
index 0000000000..837cf30612
--- /dev/null
+++ b/src/tutorial/subscripting.source
@@ -0,0 +1,71 @@
+---------------------------------------------------------------------------
+--
+-- subscripting.sql-
+-- This file shows how to create a new subscripting procedure for
+-- user-defined type.
+--
+--
+-- Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+-- Portions Copyright (c) 1994, Regents of the University of California
+--
+-- src/tutorial/subscripting.source
+--
+---------------------------------------------------------------------------
+
+-----------------------------
+-- Creating a new type:
+-- We are going to create a new type called 'complex' which represents
+-- complex numbers.
+-- A user-defined type must have an input and an output function, and
+-- optionally can have binary input and output functions. All of these
+-- are usually user-defined C functions.
+-----------------------------
+
+-- Assume the user defined functions are in /home/erthalion/programms/postgresql-master/src/tutorial/complex$DLSUFFIX
+-- (we do not want to assume this is in the dynamic loader search path).
+-- Look at $PWD/complex.c for the source. Note that we declare all of
+-- them as STRICT, so we do not need to cope with NULL inputs in the
+-- C code. We also mark them IMMUTABLE, since they always return the
+-- same outputs given the same inputs.
+
+-- the input function 'complex_in' takes a null-terminated string (the
+-- textual representation of the type) and turns it into the internal
+-- (in memory) representation. You will get a message telling you 'complex'
+-- does not exist yet but that's okay.
+
+CREATE FUNCTION custom_in(cstring)
+ RETURNS custom
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+-- the output function 'complex_out' takes the internal representation and
+-- converts it into the textual representation.
+
+CREATE FUNCTION custom_out(custom)
+ RETURNS cstring
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 8,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler
+);
+
+-- we can use it in a table
+
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
--
2.21.0
--u66dlpg6gaevsofr
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v33-0005-Filling-gaps-in-jsonb-arrays.patch"
^ permalink raw reply [nested|flat] 55+ messages in thread
* [PATCH v33 4/5] Subscripting documentation
@ 2019-02-01 10:47 erthalion <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: erthalion @ 2019-02-01 10:47 UTC (permalink / raw)
Supporting documentation for generalized subscripting. It includes the
description of a new field in pg_type, the new section for jsonb
documentation about subscripting feature on this data type, and also the
tutorial about how to write subscripting operator for a custom data type.
Reviewed-by: Tom Lane, Arthur Zakirov, Oleksandr Shulgin
---
doc/src/sgml/catalogs.sgml | 8 ++
doc/src/sgml/extend.sgml | 6 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/json.sgml | 39 ++++++
doc/src/sgml/ref/create_type.sgml | 33 ++++-
doc/src/sgml/xsubscripting.sgml | 111 +++++++++++++++++
src/tutorial/Makefile | 4 +-
src/tutorial/subscripting.c | 201 ++++++++++++++++++++++++++++++
src/tutorial/subscripting.source | 71 +++++++++++
9 files changed, 470 insertions(+), 4 deletions(-)
create mode 100644 doc/src/sgml/xsubscripting.sgml
create mode 100644 src/tutorial/subscripting.c
create mode 100644 src/tutorial/subscripting.source
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 048ff284f7..b17e9ee382 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8971,6 +8971,14 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>typsubshandler</structfield></entry>
+ <entry><type>regproc</type></entry>
+ <entry><literal><link linkend="catalog-pg-proc"><structname>pg_proc</structname></link>.oid</literal></entry>
+ <entry>Custom subscripting function with type-specific logic for parsing
+ and validation, or 0 if this type doesn't support subscripting.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>typdefault</structfield> <type>text</type>
diff --git a/doc/src/sgml/extend.sgml b/doc/src/sgml/extend.sgml
index 890ff97b7a..65cf5706b6 100644
--- a/doc/src/sgml/extend.sgml
+++ b/doc/src/sgml/extend.sgml
@@ -33,6 +33,11 @@
operators (starting in <xref linkend="xoper"/>)
</para>
</listitem>
+ <listitem>
+ <para>
+ subscripting procedure (starting in <xref linkend="xsubscripting"/>)
+ </para>
+ </listitem>
<listitem>
<para>
operator classes for indexes (starting in <xref linkend="xindex"/>)
@@ -481,6 +486,7 @@ RETURNS anycompatible AS ...
&xaggr;
&xtypes;
&xoper;
+ &xsubscripting;
&xindex;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 64b5da0070..4c3fd37fc1 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -69,6 +69,7 @@
<!ENTITY xplang SYSTEM "xplang.sgml">
<!ENTITY xoper SYSTEM "xoper.sgml">
<!ENTITY xtypes SYSTEM "xtypes.sgml">
+<!ENTITY xsubscripting SYSTEM "xsubscripting.sgml">
<!ENTITY plperl SYSTEM "plperl.sgml">
<!ENTITY plpython SYSTEM "plpython.sgml">
<!ENTITY plsql SYSTEM "plpgsql.sgml">
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index c0a6554d4d..5c538dca05 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -602,6 +602,45 @@ SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qu
</para>
</sect2>
+ <sect2 id="jsonb-subscripting">
+ <title><type>jsonb</type> Subscripting</title>
+ <para>
+ <type>jsonb</type> data type supports array-style subscripting expressions
+ to extract or update particular elements. It's possible to use multiple
+ subscripting expressions to extract nested values. In this case, a chain of
+ subscripting expressions follows the same rules as the
+ <literal>path</literal> argument in <literal>jsonb_set</literal> function,
+ e.g. in case of arrays it is a 0-based operation or that negative integers
+ that appear in <literal>path</literal> count from the end of JSON arrays.
+ The result of subscripting expressions is always jsonb data type. An
+ example of subscripting syntax:
+<programlisting>
+-- Extract value by key
+SELECT ('{"a": 1}'::jsonb)['a'];
+
+-- Extract nested value by key path
+SELECT ('{"a": {"b": {"c": 1}}}'::jsonb)['a']['b']['c'];
+
+-- Extract element by index
+SELECT ('[1, "2", null]'::jsonb)[1];
+
+-- Update value by key
+UPDATE table_name SET jsonb_field['key'] = 1;
+
+-- Select records using where clause with subscripting. Since the result of
+-- subscripting is jsonb and we basically want to compare two jsonb objects, we
+-- need to put the value in double quotes to be able to convert it to jsonb.
+SELECT * FROM table_name WHERE jsonb_field['key'] = '"value"';
+</programlisting>
+
+ There is no special indexing support for such kind of expressions, but you
+ can always create a functional index that includes it
+<programlisting>
+CREATE INDEX idx ON table_name ((jsonb_field['key']));
+</programlisting>
+ </para>
+ </sect2>
+
<sect2>
<title>Transforms</title>
diff --git a/doc/src/sgml/ref/create_type.sgml b/doc/src/sgml/ref/create_type.sgml
index 111f8e65d2..a34df4d247 100644
--- a/doc/src/sgml/ref/create_type.sgml
+++ b/doc/src/sgml/ref/create_type.sgml
@@ -54,6 +54,7 @@ CREATE TYPE <replaceable class="parameter">name</replaceable> (
[ , ELEMENT = <replaceable class="parameter">element</replaceable> ]
[ , DELIMITER = <replaceable class="parameter">delimiter</replaceable> ]
[ , COLLATABLE = <replaceable class="parameter">collatable</replaceable> ]
+ [ , SUBSCRIPTING_HANDLER = <replaceable class="parameter">subscripting_handler_function</replaceable> ]
)
CREATE TYPE <replaceable class="parameter">name</replaceable>
@@ -196,8 +197,9 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
<replaceable class="parameter">receive_function</replaceable>,
<replaceable class="parameter">send_function</replaceable>,
<replaceable class="parameter">type_modifier_input_function</replaceable>,
- <replaceable class="parameter">type_modifier_output_function</replaceable> and
- <replaceable class="parameter">analyze_function</replaceable>
+ <replaceable class="parameter">type_modifier_output_function</replaceable>,
+ <replaceable class="parameter">analyze_function</replaceable>,
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
are optional. Generally these functions have to be coded in C
or another low-level language.
</para>
@@ -454,6 +456,22 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
make use of the collation information; this does not happen
automatically merely by marking the type collatable.
</para>
+
+ <para>
+ The optional
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
+ contains type-specific logic for subscripting of the data type.
+ By default, there is no such function provided, which means that the data
+ type doesn't support subscripting. The subscripting function must be
+ declared to take a single argument of type <type>internal</type>, and return
+ a <type>internal</type> result. There are two examples of implementation for
+ subscripting functions in case of array
+ (<replaceable class="parameter">array_subscripting_handler</replaceable>)
+ and jsonb
+ (<replaceable class="parameter">jsonb_subscripting_handler</replaceable>)
+ types in <filename>src/backend/utils/adt/arrayfuncs.c</filename> and
+ <filename>src/backend/utils/adt/jsonfuncs.c</filename>, respectively.
+ </para>
</refsect2>
<refsect2>
@@ -769,6 +787,17 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">subscripting_handler_function</replaceable></term>
+ <listitem>
+ <para>
+ The name of a function that returns list of type-specific callback functions to
+ support subscripting logic for the data type.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
diff --git a/doc/src/sgml/xsubscripting.sgml b/doc/src/sgml/xsubscripting.sgml
new file mode 100644
index 0000000000..7224e81fa2
--- /dev/null
+++ b/doc/src/sgml/xsubscripting.sgml
@@ -0,0 +1,111 @@
+<!-- doc/src/sgml/xsubscripting.sgml -->
+
+ <sect1 id="xsubscripting">
+ <title>User-defined subscripting procedure</title>
+
+ <indexterm zone="xsubscripting">
+ <primary>custom subscripting</primary>
+ </indexterm>
+ <para>
+ When you define a new base type, you can also specify a custom procedure to
+ handle subscripting expressions. It must contain logic for verification and
+ evaluation of this expression, i.e. fetching or updating some data in this
+ data type. For instance:
+</para>
+<programlisting><![CDATA[
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = someType;
+ sbsref->refassgntype = someType;
+
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ // some validation and coercion logic
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some assignment logic
+
+ return newContainer;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some fetch logic based on sbstate
+}]]>
+</programlisting>
+
+<para>
+ Then you can define a subscripting procedure and a custom data type:
+</para>
+<programlisting>
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '<replaceable>filename</replaceable>'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 4,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler,
+);
+</programlisting>
+
+<para>
+ and use it as usual:
+</para>
+<programlisting>
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
+</programlisting>
+
+
+ <para>
+ The examples of custom subscripting implementation can be found in
+ <filename>subscripting.sql</filename> and <filename>subscripting.c</filename>
+ in the <filename>src/tutorial</filename> directory of the source distribution.
+ See the <filename>README</filename> file in that directory for instructions
+ about running the examples.
+ </para>
+
+</sect1>
diff --git a/src/tutorial/Makefile b/src/tutorial/Makefile
index 16dc390f71..0ead60c2d4 100644
--- a/src/tutorial/Makefile
+++ b/src/tutorial/Makefile
@@ -13,8 +13,8 @@
#
#-------------------------------------------------------------------------
-MODULES = complex funcs
-DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql
+MODULES = complex funcs subscripting
+DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql subscripting.sql
ifdef NO_PGXS
subdir = src/tutorial
diff --git a/src/tutorial/subscripting.c b/src/tutorial/subscripting.c
new file mode 100644
index 0000000000..1eb8c45652
--- /dev/null
+++ b/src/tutorial/subscripting.c
@@ -0,0 +1,201 @@
+/*
+ * src/tutorial/subscripting.c
+ *
+ ******************************************************************************
+ This file contains routines that can be bound to a Postgres backend and
+ called by the backend in the process of processing queries. The calling
+ format for these routines is dictated by Postgres architecture.
+******************************************************************************/
+
+#include "postgres.h"
+
+#include "catalog/pg_type.h"
+#include "executor/executor.h"
+#include "executor/execExpr.h"
+#include "nodes/nodeFuncs.h"
+#include "parser/parse_coerce.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+SubscriptingRef * custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref);
+SubscriptingRef * custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate);
+Datum custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate);
+Datum custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate);
+
+PG_FUNCTION_INFO_V1(custom_in);
+PG_FUNCTION_INFO_V1(custom_out);
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+/*****************************************************************************
+ * Input/Output functions
+ *****************************************************************************/
+
+Datum
+custom_in(PG_FUNCTION_ARGS)
+{
+ char *str = PG_GETARG_CSTRING(0);
+ int firstValue,
+ secondValue;
+ Custom *result;
+
+ if (sscanf(str, " ( %d , %d )", &firstValue, &secondValue) != 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
+ errmsg("invalid input syntax for complex: \"%s\"",
+ str)));
+
+
+ result = (Custom *) palloc(sizeof(Custom));
+ result->first = firstValue;
+ result->second = secondValue;
+ PG_RETURN_POINTER(result);
+}
+
+Datum
+custom_out(PG_FUNCTION_ARGS)
+{
+ Custom *custom = (Custom *) PG_GETARG_POINTER(0);
+ char *result;
+
+ result = psprintf("(%d, %d)", custom->first, custom->second);
+ PG_RETURN_CSTRING(result);
+}
+
+/*****************************************************************************
+ * Custom subscripting logic functions
+ *****************************************************************************/
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = INT4OID;
+ sbsref->refassgntype = INT4OID;
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ List *upperIndexpr = NIL;
+ ListCell *l;
+
+ if (sbsref->reflowerindexpr != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *)lfirst(sbsref->reflowerindexpr->head))))));
+
+ foreach(l, sbsref->refupperindexpr)
+ {
+ Node *subexpr = (Node *) lfirst(l);
+
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *) lfirst(sbsref->refupperindexpr->head))))));
+
+ subexpr = coerce_to_target_type(pstate,
+ subexpr, exprType(subexpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript must have integer type"),
+ parser_errposition(pstate, exprLocation(subexpr))));
+
+ upperIndexpr = lappend(upperIndexpr, subexpr);
+
+ if (isAssignment)
+ {
+ Node *assignExpr = (Node *) sbsref->refassgnexpr;
+ Node *new_from;
+
+ new_from = coerce_to_target_type(pstate,
+ assignExpr, exprType(assignExpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (new_from == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom assignment requires int type"),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation(assignExpr))));
+ sbsref->refassgnexpr = (Expr *)new_from;
+ }
+ }
+
+ sbsref->refupperindexpr = upperIndexpr;
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ Custom *container= (Custom *) containerSource;
+ int index;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ return (Datum) container->first;
+ else
+ return (Datum) container->second;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ int index;
+ Custom *container = (Custom *) containerSource;
+
+ if (sbstate->resnull)
+ return containerSource;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ container->first = DatumGetInt32(sbstate->replacevalue);
+ else
+ container->second = DatumGetInt32(sbstate->replacevalue);
+
+ return (Datum) container;
+}
diff --git a/src/tutorial/subscripting.source b/src/tutorial/subscripting.source
new file mode 100644
index 0000000000..837cf30612
--- /dev/null
+++ b/src/tutorial/subscripting.source
@@ -0,0 +1,71 @@
+---------------------------------------------------------------------------
+--
+-- subscripting.sql-
+-- This file shows how to create a new subscripting procedure for
+-- user-defined type.
+--
+--
+-- Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+-- Portions Copyright (c) 1994, Regents of the University of California
+--
+-- src/tutorial/subscripting.source
+--
+---------------------------------------------------------------------------
+
+-----------------------------
+-- Creating a new type:
+-- We are going to create a new type called 'complex' which represents
+-- complex numbers.
+-- A user-defined type must have an input and an output function, and
+-- optionally can have binary input and output functions. All of these
+-- are usually user-defined C functions.
+-----------------------------
+
+-- Assume the user defined functions are in /home/erthalion/programms/postgresql-master/src/tutorial/complex$DLSUFFIX
+-- (we do not want to assume this is in the dynamic loader search path).
+-- Look at $PWD/complex.c for the source. Note that we declare all of
+-- them as STRICT, so we do not need to cope with NULL inputs in the
+-- C code. We also mark them IMMUTABLE, since they always return the
+-- same outputs given the same inputs.
+
+-- the input function 'complex_in' takes a null-terminated string (the
+-- textual representation of the type) and turns it into the internal
+-- (in memory) representation. You will get a message telling you 'complex'
+-- does not exist yet but that's okay.
+
+CREATE FUNCTION custom_in(cstring)
+ RETURNS custom
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+-- the output function 'complex_out' takes the internal representation and
+-- converts it into the textual representation.
+
+CREATE FUNCTION custom_out(custom)
+ RETURNS cstring
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 8,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler
+);
+
+-- we can use it in a table
+
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
--
2.21.0
--u66dlpg6gaevsofr
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v33-0005-Filling-gaps-in-jsonb-arrays.patch"
^ permalink raw reply [nested|flat] 55+ messages in thread
* [PATCH v33 4/5] Subscripting documentation
@ 2019-02-01 10:47 erthalion <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: erthalion @ 2019-02-01 10:47 UTC (permalink / raw)
Supporting documentation for generalized subscripting. It includes the
description of a new field in pg_type, the new section for jsonb
documentation about subscripting feature on this data type, and also the
tutorial about how to write subscripting operator for a custom data type.
Reviewed-by: Tom Lane, Arthur Zakirov, Oleksandr Shulgin
---
doc/src/sgml/catalogs.sgml | 8 ++
doc/src/sgml/extend.sgml | 6 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/json.sgml | 39 ++++++
doc/src/sgml/ref/create_type.sgml | 33 ++++-
doc/src/sgml/xsubscripting.sgml | 111 +++++++++++++++++
src/tutorial/Makefile | 4 +-
src/tutorial/subscripting.c | 201 ++++++++++++++++++++++++++++++
src/tutorial/subscripting.source | 71 +++++++++++
9 files changed, 470 insertions(+), 4 deletions(-)
create mode 100644 doc/src/sgml/xsubscripting.sgml
create mode 100644 src/tutorial/subscripting.c
create mode 100644 src/tutorial/subscripting.source
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 048ff284f7..b17e9ee382 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8971,6 +8971,14 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>typsubshandler</structfield></entry>
+ <entry><type>regproc</type></entry>
+ <entry><literal><link linkend="catalog-pg-proc"><structname>pg_proc</structname></link>.oid</literal></entry>
+ <entry>Custom subscripting function with type-specific logic for parsing
+ and validation, or 0 if this type doesn't support subscripting.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>typdefault</structfield> <type>text</type>
diff --git a/doc/src/sgml/extend.sgml b/doc/src/sgml/extend.sgml
index 890ff97b7a..65cf5706b6 100644
--- a/doc/src/sgml/extend.sgml
+++ b/doc/src/sgml/extend.sgml
@@ -33,6 +33,11 @@
operators (starting in <xref linkend="xoper"/>)
</para>
</listitem>
+ <listitem>
+ <para>
+ subscripting procedure (starting in <xref linkend="xsubscripting"/>)
+ </para>
+ </listitem>
<listitem>
<para>
operator classes for indexes (starting in <xref linkend="xindex"/>)
@@ -481,6 +486,7 @@ RETURNS anycompatible AS ...
&xaggr;
&xtypes;
&xoper;
+ &xsubscripting;
&xindex;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 64b5da0070..4c3fd37fc1 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -69,6 +69,7 @@
<!ENTITY xplang SYSTEM "xplang.sgml">
<!ENTITY xoper SYSTEM "xoper.sgml">
<!ENTITY xtypes SYSTEM "xtypes.sgml">
+<!ENTITY xsubscripting SYSTEM "xsubscripting.sgml">
<!ENTITY plperl SYSTEM "plperl.sgml">
<!ENTITY plpython SYSTEM "plpython.sgml">
<!ENTITY plsql SYSTEM "plpgsql.sgml">
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index c0a6554d4d..5c538dca05 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -602,6 +602,45 @@ SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qu
</para>
</sect2>
+ <sect2 id="jsonb-subscripting">
+ <title><type>jsonb</type> Subscripting</title>
+ <para>
+ <type>jsonb</type> data type supports array-style subscripting expressions
+ to extract or update particular elements. It's possible to use multiple
+ subscripting expressions to extract nested values. In this case, a chain of
+ subscripting expressions follows the same rules as the
+ <literal>path</literal> argument in <literal>jsonb_set</literal> function,
+ e.g. in case of arrays it is a 0-based operation or that negative integers
+ that appear in <literal>path</literal> count from the end of JSON arrays.
+ The result of subscripting expressions is always jsonb data type. An
+ example of subscripting syntax:
+<programlisting>
+-- Extract value by key
+SELECT ('{"a": 1}'::jsonb)['a'];
+
+-- Extract nested value by key path
+SELECT ('{"a": {"b": {"c": 1}}}'::jsonb)['a']['b']['c'];
+
+-- Extract element by index
+SELECT ('[1, "2", null]'::jsonb)[1];
+
+-- Update value by key
+UPDATE table_name SET jsonb_field['key'] = 1;
+
+-- Select records using where clause with subscripting. Since the result of
+-- subscripting is jsonb and we basically want to compare two jsonb objects, we
+-- need to put the value in double quotes to be able to convert it to jsonb.
+SELECT * FROM table_name WHERE jsonb_field['key'] = '"value"';
+</programlisting>
+
+ There is no special indexing support for such kind of expressions, but you
+ can always create a functional index that includes it
+<programlisting>
+CREATE INDEX idx ON table_name ((jsonb_field['key']));
+</programlisting>
+ </para>
+ </sect2>
+
<sect2>
<title>Transforms</title>
diff --git a/doc/src/sgml/ref/create_type.sgml b/doc/src/sgml/ref/create_type.sgml
index 111f8e65d2..a34df4d247 100644
--- a/doc/src/sgml/ref/create_type.sgml
+++ b/doc/src/sgml/ref/create_type.sgml
@@ -54,6 +54,7 @@ CREATE TYPE <replaceable class="parameter">name</replaceable> (
[ , ELEMENT = <replaceable class="parameter">element</replaceable> ]
[ , DELIMITER = <replaceable class="parameter">delimiter</replaceable> ]
[ , COLLATABLE = <replaceable class="parameter">collatable</replaceable> ]
+ [ , SUBSCRIPTING_HANDLER = <replaceable class="parameter">subscripting_handler_function</replaceable> ]
)
CREATE TYPE <replaceable class="parameter">name</replaceable>
@@ -196,8 +197,9 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
<replaceable class="parameter">receive_function</replaceable>,
<replaceable class="parameter">send_function</replaceable>,
<replaceable class="parameter">type_modifier_input_function</replaceable>,
- <replaceable class="parameter">type_modifier_output_function</replaceable> and
- <replaceable class="parameter">analyze_function</replaceable>
+ <replaceable class="parameter">type_modifier_output_function</replaceable>,
+ <replaceable class="parameter">analyze_function</replaceable>,
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
are optional. Generally these functions have to be coded in C
or another low-level language.
</para>
@@ -454,6 +456,22 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
make use of the collation information; this does not happen
automatically merely by marking the type collatable.
</para>
+
+ <para>
+ The optional
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
+ contains type-specific logic for subscripting of the data type.
+ By default, there is no such function provided, which means that the data
+ type doesn't support subscripting. The subscripting function must be
+ declared to take a single argument of type <type>internal</type>, and return
+ a <type>internal</type> result. There are two examples of implementation for
+ subscripting functions in case of array
+ (<replaceable class="parameter">array_subscripting_handler</replaceable>)
+ and jsonb
+ (<replaceable class="parameter">jsonb_subscripting_handler</replaceable>)
+ types in <filename>src/backend/utils/adt/arrayfuncs.c</filename> and
+ <filename>src/backend/utils/adt/jsonfuncs.c</filename>, respectively.
+ </para>
</refsect2>
<refsect2>
@@ -769,6 +787,17 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">subscripting_handler_function</replaceable></term>
+ <listitem>
+ <para>
+ The name of a function that returns list of type-specific callback functions to
+ support subscripting logic for the data type.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
diff --git a/doc/src/sgml/xsubscripting.sgml b/doc/src/sgml/xsubscripting.sgml
new file mode 100644
index 0000000000..7224e81fa2
--- /dev/null
+++ b/doc/src/sgml/xsubscripting.sgml
@@ -0,0 +1,111 @@
+<!-- doc/src/sgml/xsubscripting.sgml -->
+
+ <sect1 id="xsubscripting">
+ <title>User-defined subscripting procedure</title>
+
+ <indexterm zone="xsubscripting">
+ <primary>custom subscripting</primary>
+ </indexterm>
+ <para>
+ When you define a new base type, you can also specify a custom procedure to
+ handle subscripting expressions. It must contain logic for verification and
+ evaluation of this expression, i.e. fetching or updating some data in this
+ data type. For instance:
+</para>
+<programlisting><![CDATA[
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = someType;
+ sbsref->refassgntype = someType;
+
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ // some validation and coercion logic
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some assignment logic
+
+ return newContainer;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some fetch logic based on sbstate
+}]]>
+</programlisting>
+
+<para>
+ Then you can define a subscripting procedure and a custom data type:
+</para>
+<programlisting>
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '<replaceable>filename</replaceable>'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 4,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler,
+);
+</programlisting>
+
+<para>
+ and use it as usual:
+</para>
+<programlisting>
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
+</programlisting>
+
+
+ <para>
+ The examples of custom subscripting implementation can be found in
+ <filename>subscripting.sql</filename> and <filename>subscripting.c</filename>
+ in the <filename>src/tutorial</filename> directory of the source distribution.
+ See the <filename>README</filename> file in that directory for instructions
+ about running the examples.
+ </para>
+
+</sect1>
diff --git a/src/tutorial/Makefile b/src/tutorial/Makefile
index 16dc390f71..0ead60c2d4 100644
--- a/src/tutorial/Makefile
+++ b/src/tutorial/Makefile
@@ -13,8 +13,8 @@
#
#-------------------------------------------------------------------------
-MODULES = complex funcs
-DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql
+MODULES = complex funcs subscripting
+DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql subscripting.sql
ifdef NO_PGXS
subdir = src/tutorial
diff --git a/src/tutorial/subscripting.c b/src/tutorial/subscripting.c
new file mode 100644
index 0000000000..1eb8c45652
--- /dev/null
+++ b/src/tutorial/subscripting.c
@@ -0,0 +1,201 @@
+/*
+ * src/tutorial/subscripting.c
+ *
+ ******************************************************************************
+ This file contains routines that can be bound to a Postgres backend and
+ called by the backend in the process of processing queries. The calling
+ format for these routines is dictated by Postgres architecture.
+******************************************************************************/
+
+#include "postgres.h"
+
+#include "catalog/pg_type.h"
+#include "executor/executor.h"
+#include "executor/execExpr.h"
+#include "nodes/nodeFuncs.h"
+#include "parser/parse_coerce.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+SubscriptingRef * custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref);
+SubscriptingRef * custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate);
+Datum custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate);
+Datum custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate);
+
+PG_FUNCTION_INFO_V1(custom_in);
+PG_FUNCTION_INFO_V1(custom_out);
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+/*****************************************************************************
+ * Input/Output functions
+ *****************************************************************************/
+
+Datum
+custom_in(PG_FUNCTION_ARGS)
+{
+ char *str = PG_GETARG_CSTRING(0);
+ int firstValue,
+ secondValue;
+ Custom *result;
+
+ if (sscanf(str, " ( %d , %d )", &firstValue, &secondValue) != 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
+ errmsg("invalid input syntax for complex: \"%s\"",
+ str)));
+
+
+ result = (Custom *) palloc(sizeof(Custom));
+ result->first = firstValue;
+ result->second = secondValue;
+ PG_RETURN_POINTER(result);
+}
+
+Datum
+custom_out(PG_FUNCTION_ARGS)
+{
+ Custom *custom = (Custom *) PG_GETARG_POINTER(0);
+ char *result;
+
+ result = psprintf("(%d, %d)", custom->first, custom->second);
+ PG_RETURN_CSTRING(result);
+}
+
+/*****************************************************************************
+ * Custom subscripting logic functions
+ *****************************************************************************/
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = INT4OID;
+ sbsref->refassgntype = INT4OID;
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ List *upperIndexpr = NIL;
+ ListCell *l;
+
+ if (sbsref->reflowerindexpr != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *)lfirst(sbsref->reflowerindexpr->head))))));
+
+ foreach(l, sbsref->refupperindexpr)
+ {
+ Node *subexpr = (Node *) lfirst(l);
+
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *) lfirst(sbsref->refupperindexpr->head))))));
+
+ subexpr = coerce_to_target_type(pstate,
+ subexpr, exprType(subexpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript must have integer type"),
+ parser_errposition(pstate, exprLocation(subexpr))));
+
+ upperIndexpr = lappend(upperIndexpr, subexpr);
+
+ if (isAssignment)
+ {
+ Node *assignExpr = (Node *) sbsref->refassgnexpr;
+ Node *new_from;
+
+ new_from = coerce_to_target_type(pstate,
+ assignExpr, exprType(assignExpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (new_from == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom assignment requires int type"),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation(assignExpr))));
+ sbsref->refassgnexpr = (Expr *)new_from;
+ }
+ }
+
+ sbsref->refupperindexpr = upperIndexpr;
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ Custom *container= (Custom *) containerSource;
+ int index;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ return (Datum) container->first;
+ else
+ return (Datum) container->second;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ int index;
+ Custom *container = (Custom *) containerSource;
+
+ if (sbstate->resnull)
+ return containerSource;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ container->first = DatumGetInt32(sbstate->replacevalue);
+ else
+ container->second = DatumGetInt32(sbstate->replacevalue);
+
+ return (Datum) container;
+}
diff --git a/src/tutorial/subscripting.source b/src/tutorial/subscripting.source
new file mode 100644
index 0000000000..837cf30612
--- /dev/null
+++ b/src/tutorial/subscripting.source
@@ -0,0 +1,71 @@
+---------------------------------------------------------------------------
+--
+-- subscripting.sql-
+-- This file shows how to create a new subscripting procedure for
+-- user-defined type.
+--
+--
+-- Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+-- Portions Copyright (c) 1994, Regents of the University of California
+--
+-- src/tutorial/subscripting.source
+--
+---------------------------------------------------------------------------
+
+-----------------------------
+-- Creating a new type:
+-- We are going to create a new type called 'complex' which represents
+-- complex numbers.
+-- A user-defined type must have an input and an output function, and
+-- optionally can have binary input and output functions. All of these
+-- are usually user-defined C functions.
+-----------------------------
+
+-- Assume the user defined functions are in /home/erthalion/programms/postgresql-master/src/tutorial/complex$DLSUFFIX
+-- (we do not want to assume this is in the dynamic loader search path).
+-- Look at $PWD/complex.c for the source. Note that we declare all of
+-- them as STRICT, so we do not need to cope with NULL inputs in the
+-- C code. We also mark them IMMUTABLE, since they always return the
+-- same outputs given the same inputs.
+
+-- the input function 'complex_in' takes a null-terminated string (the
+-- textual representation of the type) and turns it into the internal
+-- (in memory) representation. You will get a message telling you 'complex'
+-- does not exist yet but that's okay.
+
+CREATE FUNCTION custom_in(cstring)
+ RETURNS custom
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+-- the output function 'complex_out' takes the internal representation and
+-- converts it into the textual representation.
+
+CREATE FUNCTION custom_out(custom)
+ RETURNS cstring
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 8,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler
+);
+
+-- we can use it in a table
+
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
--
2.21.0
--u66dlpg6gaevsofr
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v33-0005-Filling-gaps-in-jsonb-arrays.patch"
^ permalink raw reply [nested|flat] 55+ messages in thread
* [PATCH v31 4/6] Subscripting documentation
@ 2019-02-01 10:47 erthalion <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: erthalion @ 2019-02-01 10:47 UTC (permalink / raw)
Supporting documentation for generalized subscripting. It includes the
description of a new field in pg_type, the new section for jsonb
documentation about subscripting feature on this data type, and also the
tutorial about how to write subscripting operator for a custom data type.
Reviewed-by: Tom Lane, Arthur Zakirov, Oleksandr Shulgin
---
doc/src/sgml/catalogs.sgml | 8 ++
doc/src/sgml/extend.sgml | 6 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/json.sgml | 39 ++++++
doc/src/sgml/ref/create_type.sgml | 33 ++++-
doc/src/sgml/xsubscripting.sgml | 111 +++++++++++++++++
src/tutorial/Makefile | 4 +-
src/tutorial/subscripting.c | 201 ++++++++++++++++++++++++++++++
src/tutorial/subscripting.source | 71 +++++++++++
9 files changed, 470 insertions(+), 4 deletions(-)
create mode 100644 doc/src/sgml/xsubscripting.sgml
create mode 100644 src/tutorial/subscripting.c
create mode 100644 src/tutorial/subscripting.source
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 9d8fa0bec3..439bee0678 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8957,6 +8957,14 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>typsubshandler</structfield></entry>
+ <entry><type>regproc</type></entry>
+ <entry><literal><link linkend="catalog-pg-proc"><structname>pg_proc</structname></link>.oid</literal></entry>
+ <entry>Custom subscripting function with type-specific logic for parsing
+ and validation, or 0 if this type doesn't support subscripting.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>typdefault</structfield> <type>text</type>
diff --git a/doc/src/sgml/extend.sgml b/doc/src/sgml/extend.sgml
index c1ffb14571..5d3cf479b4 100644
--- a/doc/src/sgml/extend.sgml
+++ b/doc/src/sgml/extend.sgml
@@ -33,6 +33,11 @@
operators (starting in <xref linkend="xoper"/>)
</para>
</listitem>
+ <listitem>
+ <para>
+ subscripting procedure (starting in <xref linkend="xsubscripting"/>)
+ </para>
+ </listitem>
<listitem>
<para>
operator classes for indexes (starting in <xref linkend="xindex"/>)
@@ -481,6 +486,7 @@ RETURNS anycompatible AS ...
&xaggr;
&xtypes;
&xoper;
+ &xsubscripting;
&xindex;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 68179f71cd..34249f8c60 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -69,6 +69,7 @@
<!ENTITY xplang SYSTEM "xplang.sgml">
<!ENTITY xoper SYSTEM "xoper.sgml">
<!ENTITY xtypes SYSTEM "xtypes.sgml">
+<!ENTITY xsubscripting SYSTEM "xsubscripting.sgml">
<!ENTITY plperl SYSTEM "plperl.sgml">
<!ENTITY plpython SYSTEM "plpython.sgml">
<!ENTITY plsql SYSTEM "plpgsql.sgml">
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index c0a6554d4d..3bffe8049b 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -602,6 +602,45 @@ SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qu
</para>
</sect2>
+ <sect2 id="jsonb-subscripting">
+ <title><type>jsonb</type> Subscripting</title>
+ <para>
+ <type>jsonb</type> data type supports array-style subscripting expressions
+ to extract or update particular element. It's possible to use multiple
+ subscripting expressions to extract nested values. In this case a chain of
+ subscripting expressions follows the same rules as the
+ <literal>path</literal> argument in <literal>jsonb_set</literal> function,
+ e.g. in case of arrays it is a 0-based operation or that negative integers
+ that appear in <literal>path</literal> count from the end of JSON arrays.
+ The result of subscripting expressions is always jsonb data type. An
+ example of subscripting syntax:
+<programlisting>
+-- Extract value by key
+SELECT ('{"a": 1}'::jsonb)['a'];
+
+-- Extract nested value by key path
+SELECT ('{"a": {"b": {"c": 1}}}'::jsonb)['a']['b']['c'];
+
+-- Extract element by index
+SELECT ('[1, "2", null]'::jsonb)[1];
+
+-- Update value by key
+UPDATE table_name SET jsonb_field['key'] = 1;
+
+-- Select records using where clause with subscripting. Since the result of
+-- subscripting is jsonb and we basically want to compare two jsonb objects, we
+-- need to put the value in double quotes to be able to convert it to jsonb.
+SELECT * FROM table_name WHERE jsonb_field['key'] = '"value"';
+</programlisting>
+
+ There is no special indexing support for such kind of expressions, but you
+ always can create a functional index that includes it
+<programlisting>
+CREATE INDEX idx ON table_name ((jsonb_field['key']));
+</programlisting>
+ </para>
+ </sect2>
+
<sect2>
<title>Transforms</title>
diff --git a/doc/src/sgml/ref/create_type.sgml b/doc/src/sgml/ref/create_type.sgml
index 111f8e65d2..ec67761c66 100644
--- a/doc/src/sgml/ref/create_type.sgml
+++ b/doc/src/sgml/ref/create_type.sgml
@@ -54,6 +54,7 @@ CREATE TYPE <replaceable class="parameter">name</replaceable> (
[ , ELEMENT = <replaceable class="parameter">element</replaceable> ]
[ , DELIMITER = <replaceable class="parameter">delimiter</replaceable> ]
[ , COLLATABLE = <replaceable class="parameter">collatable</replaceable> ]
+ [ , SUBSCRIPTING_HANDLER = <replaceable class="parameter">subscripting_handler_function</replaceable> ]
)
CREATE TYPE <replaceable class="parameter">name</replaceable>
@@ -196,8 +197,9 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
<replaceable class="parameter">receive_function</replaceable>,
<replaceable class="parameter">send_function</replaceable>,
<replaceable class="parameter">type_modifier_input_function</replaceable>,
- <replaceable class="parameter">type_modifier_output_function</replaceable> and
- <replaceable class="parameter">analyze_function</replaceable>
+ <replaceable class="parameter">type_modifier_output_function</replaceable>,
+ <replaceable class="parameter">analyze_function</replaceable>,
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
are optional. Generally these functions have to be coded in C
or another low-level language.
</para>
@@ -454,6 +456,22 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
make use of the collation information; this does not happen
automatically merely by marking the type collatable.
</para>
+
+ <para>
+ The optional
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
+ contains type-specific logic for subscripting of the data type.
+ By default, there is no such function provided, which means that the data
+ type doesn't support subscripting. The subscripting function must be
+ declared to take a single argument of type <type>internal</type>, and return
+ a <type>internal</type> result. There are two examples of implementation for
+ subscripting functions in case of array
+ (<replaceable class="parameter">array_subscripting_handler</replaceable>)
+ and jsonb
+ (<replaceable class="parameter">jsonb_subscripting_handler</replaceable>)
+ types in <filename>src/backend/utils/adt/arrayfuncs.c</filename> and
+ <filename>src/backend/utils/adt/jsonfuncs.c</filename> corresponding.
+ </para>
</refsect2>
<refsect2>
@@ -769,6 +787,17 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">subscripting_handler_function</replaceable></term>
+ <listitem>
+ <para>
+ The name of a function that returns list of type-specific callback functions to
+ support subscripting logic for the data type.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
diff --git a/doc/src/sgml/xsubscripting.sgml b/doc/src/sgml/xsubscripting.sgml
new file mode 100644
index 0000000000..d701631223
--- /dev/null
+++ b/doc/src/sgml/xsubscripting.sgml
@@ -0,0 +1,111 @@
+<!-- doc/src/sgml/xsubscripting.sgml -->
+
+ <sect1 id="xsubscripting">
+ <title>User-defined subscripting procedure</title>
+
+ <indexterm zone="xsubscripting">
+ <primary>custom subscripting</primary>
+ </indexterm>
+ <para>
+ When you define a new base type, you can also specify a custom procedures to
+ handle subscripting expressions. They must contain logic for verification and
+ evaluation of this expression, i.e. fetching or updating some data in this
+ data type. For instance:
+</para>
+<programlisting><![CDATA[
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = someType;
+ sbsref->refassgntype = someType;
+
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ // some validation and coercion logic
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some assignment logic
+
+ return newContainer;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some fetch logic based on sbsdata
+}]]>
+</programlisting>
+
+<para>
+ Then you can define a subscripting procedures and a custom data type:
+</para>
+<programlisting>
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '<replaceable>filename</replaceable>'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 4,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler,
+);
+</programlisting>
+
+<para>
+ and use it as usual:
+</para>
+<programlisting>
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
+</programlisting>
+
+
+ <para>
+ The examples of custom subscripting implementation can be found in
+ <filename>subscripting.sql</filename> and <filename>subscripting.c</filename>
+ in the <filename>src/tutorial</filename> directory of the source distribution.
+ See the <filename>README</filename> file in that directory for instructions
+ about running the examples.
+ </para>
+
+</sect1>
diff --git a/src/tutorial/Makefile b/src/tutorial/Makefile
index 16dc390f71..0ead60c2d4 100644
--- a/src/tutorial/Makefile
+++ b/src/tutorial/Makefile
@@ -13,8 +13,8 @@
#
#-------------------------------------------------------------------------
-MODULES = complex funcs
-DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql
+MODULES = complex funcs subscripting
+DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql subscripting.sql
ifdef NO_PGXS
subdir = src/tutorial
diff --git a/src/tutorial/subscripting.c b/src/tutorial/subscripting.c
new file mode 100644
index 0000000000..1eb8c45652
--- /dev/null
+++ b/src/tutorial/subscripting.c
@@ -0,0 +1,201 @@
+/*
+ * src/tutorial/subscripting.c
+ *
+ ******************************************************************************
+ This file contains routines that can be bound to a Postgres backend and
+ called by the backend in the process of processing queries. The calling
+ format for these routines is dictated by Postgres architecture.
+******************************************************************************/
+
+#include "postgres.h"
+
+#include "catalog/pg_type.h"
+#include "executor/executor.h"
+#include "executor/execExpr.h"
+#include "nodes/nodeFuncs.h"
+#include "parser/parse_coerce.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+SubscriptingRef * custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref);
+SubscriptingRef * custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate);
+Datum custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate);
+Datum custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate);
+
+PG_FUNCTION_INFO_V1(custom_in);
+PG_FUNCTION_INFO_V1(custom_out);
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+/*****************************************************************************
+ * Input/Output functions
+ *****************************************************************************/
+
+Datum
+custom_in(PG_FUNCTION_ARGS)
+{
+ char *str = PG_GETARG_CSTRING(0);
+ int firstValue,
+ secondValue;
+ Custom *result;
+
+ if (sscanf(str, " ( %d , %d )", &firstValue, &secondValue) != 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
+ errmsg("invalid input syntax for complex: \"%s\"",
+ str)));
+
+
+ result = (Custom *) palloc(sizeof(Custom));
+ result->first = firstValue;
+ result->second = secondValue;
+ PG_RETURN_POINTER(result);
+}
+
+Datum
+custom_out(PG_FUNCTION_ARGS)
+{
+ Custom *custom = (Custom *) PG_GETARG_POINTER(0);
+ char *result;
+
+ result = psprintf("(%d, %d)", custom->first, custom->second);
+ PG_RETURN_CSTRING(result);
+}
+
+/*****************************************************************************
+ * Custom subscripting logic functions
+ *****************************************************************************/
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = INT4OID;
+ sbsref->refassgntype = INT4OID;
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ List *upperIndexpr = NIL;
+ ListCell *l;
+
+ if (sbsref->reflowerindexpr != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *)lfirst(sbsref->reflowerindexpr->head))))));
+
+ foreach(l, sbsref->refupperindexpr)
+ {
+ Node *subexpr = (Node *) lfirst(l);
+
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *) lfirst(sbsref->refupperindexpr->head))))));
+
+ subexpr = coerce_to_target_type(pstate,
+ subexpr, exprType(subexpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript must have integer type"),
+ parser_errposition(pstate, exprLocation(subexpr))));
+
+ upperIndexpr = lappend(upperIndexpr, subexpr);
+
+ if (isAssignment)
+ {
+ Node *assignExpr = (Node *) sbsref->refassgnexpr;
+ Node *new_from;
+
+ new_from = coerce_to_target_type(pstate,
+ assignExpr, exprType(assignExpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (new_from == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom assignment requires int type"),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation(assignExpr))));
+ sbsref->refassgnexpr = (Expr *)new_from;
+ }
+ }
+
+ sbsref->refupperindexpr = upperIndexpr;
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ Custom *container= (Custom *) containerSource;
+ int index;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ return (Datum) container->first;
+ else
+ return (Datum) container->second;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ int index;
+ Custom *container = (Custom *) containerSource;
+
+ if (sbstate->resnull)
+ return containerSource;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ container->first = DatumGetInt32(sbstate->replacevalue);
+ else
+ container->second = DatumGetInt32(sbstate->replacevalue);
+
+ return (Datum) container;
+}
diff --git a/src/tutorial/subscripting.source b/src/tutorial/subscripting.source
new file mode 100644
index 0000000000..837cf30612
--- /dev/null
+++ b/src/tutorial/subscripting.source
@@ -0,0 +1,71 @@
+---------------------------------------------------------------------------
+--
+-- subscripting.sql-
+-- This file shows how to create a new subscripting procedure for
+-- user-defined type.
+--
+--
+-- Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+-- Portions Copyright (c) 1994, Regents of the University of California
+--
+-- src/tutorial/subscripting.source
+--
+---------------------------------------------------------------------------
+
+-----------------------------
+-- Creating a new type:
+-- We are going to create a new type called 'complex' which represents
+-- complex numbers.
+-- A user-defined type must have an input and an output function, and
+-- optionally can have binary input and output functions. All of these
+-- are usually user-defined C functions.
+-----------------------------
+
+-- Assume the user defined functions are in /home/erthalion/programms/postgresql-master/src/tutorial/complex$DLSUFFIX
+-- (we do not want to assume this is in the dynamic loader search path).
+-- Look at $PWD/complex.c for the source. Note that we declare all of
+-- them as STRICT, so we do not need to cope with NULL inputs in the
+-- C code. We also mark them IMMUTABLE, since they always return the
+-- same outputs given the same inputs.
+
+-- the input function 'complex_in' takes a null-terminated string (the
+-- textual representation of the type) and turns it into the internal
+-- (in memory) representation. You will get a message telling you 'complex'
+-- does not exist yet but that's okay.
+
+CREATE FUNCTION custom_in(cstring)
+ RETURNS custom
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+-- the output function 'complex_out' takes the internal representation and
+-- converts it into the textual representation.
+
+CREATE FUNCTION custom_out(custom)
+ RETURNS cstring
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 8,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler
+);
+
+-- we can use it in a table
+
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
--
2.21.0
--bvn7grlx53cj2mie
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v31-0005-Polymorphic-subscripting.patch"
^ permalink raw reply [nested|flat] 55+ messages in thread
* [PATCH v33 4/5] Subscripting documentation
@ 2019-02-01 10:47 erthalion <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: erthalion @ 2019-02-01 10:47 UTC (permalink / raw)
Supporting documentation for generalized subscripting. It includes the
description of a new field in pg_type, the new section for jsonb
documentation about subscripting feature on this data type, and also the
tutorial about how to write subscripting operator for a custom data type.
Reviewed-by: Tom Lane, Arthur Zakirov, Oleksandr Shulgin
---
doc/src/sgml/catalogs.sgml | 8 ++
doc/src/sgml/extend.sgml | 6 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/json.sgml | 39 ++++++
doc/src/sgml/ref/create_type.sgml | 33 ++++-
doc/src/sgml/xsubscripting.sgml | 111 +++++++++++++++++
src/tutorial/Makefile | 4 +-
src/tutorial/subscripting.c | 201 ++++++++++++++++++++++++++++++
src/tutorial/subscripting.source | 71 +++++++++++
9 files changed, 470 insertions(+), 4 deletions(-)
create mode 100644 doc/src/sgml/xsubscripting.sgml
create mode 100644 src/tutorial/subscripting.c
create mode 100644 src/tutorial/subscripting.source
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 048ff284f7..b17e9ee382 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8971,6 +8971,14 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>typsubshandler</structfield></entry>
+ <entry><type>regproc</type></entry>
+ <entry><literal><link linkend="catalog-pg-proc"><structname>pg_proc</structname></link>.oid</literal></entry>
+ <entry>Custom subscripting function with type-specific logic for parsing
+ and validation, or 0 if this type doesn't support subscripting.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>typdefault</structfield> <type>text</type>
diff --git a/doc/src/sgml/extend.sgml b/doc/src/sgml/extend.sgml
index 890ff97b7a..65cf5706b6 100644
--- a/doc/src/sgml/extend.sgml
+++ b/doc/src/sgml/extend.sgml
@@ -33,6 +33,11 @@
operators (starting in <xref linkend="xoper"/>)
</para>
</listitem>
+ <listitem>
+ <para>
+ subscripting procedure (starting in <xref linkend="xsubscripting"/>)
+ </para>
+ </listitem>
<listitem>
<para>
operator classes for indexes (starting in <xref linkend="xindex"/>)
@@ -481,6 +486,7 @@ RETURNS anycompatible AS ...
&xaggr;
&xtypes;
&xoper;
+ &xsubscripting;
&xindex;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 64b5da0070..4c3fd37fc1 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -69,6 +69,7 @@
<!ENTITY xplang SYSTEM "xplang.sgml">
<!ENTITY xoper SYSTEM "xoper.sgml">
<!ENTITY xtypes SYSTEM "xtypes.sgml">
+<!ENTITY xsubscripting SYSTEM "xsubscripting.sgml">
<!ENTITY plperl SYSTEM "plperl.sgml">
<!ENTITY plpython SYSTEM "plpython.sgml">
<!ENTITY plsql SYSTEM "plpgsql.sgml">
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index c0a6554d4d..5c538dca05 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -602,6 +602,45 @@ SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qu
</para>
</sect2>
+ <sect2 id="jsonb-subscripting">
+ <title><type>jsonb</type> Subscripting</title>
+ <para>
+ <type>jsonb</type> data type supports array-style subscripting expressions
+ to extract or update particular elements. It's possible to use multiple
+ subscripting expressions to extract nested values. In this case, a chain of
+ subscripting expressions follows the same rules as the
+ <literal>path</literal> argument in <literal>jsonb_set</literal> function,
+ e.g. in case of arrays it is a 0-based operation or that negative integers
+ that appear in <literal>path</literal> count from the end of JSON arrays.
+ The result of subscripting expressions is always jsonb data type. An
+ example of subscripting syntax:
+<programlisting>
+-- Extract value by key
+SELECT ('{"a": 1}'::jsonb)['a'];
+
+-- Extract nested value by key path
+SELECT ('{"a": {"b": {"c": 1}}}'::jsonb)['a']['b']['c'];
+
+-- Extract element by index
+SELECT ('[1, "2", null]'::jsonb)[1];
+
+-- Update value by key
+UPDATE table_name SET jsonb_field['key'] = 1;
+
+-- Select records using where clause with subscripting. Since the result of
+-- subscripting is jsonb and we basically want to compare two jsonb objects, we
+-- need to put the value in double quotes to be able to convert it to jsonb.
+SELECT * FROM table_name WHERE jsonb_field['key'] = '"value"';
+</programlisting>
+
+ There is no special indexing support for such kind of expressions, but you
+ can always create a functional index that includes it
+<programlisting>
+CREATE INDEX idx ON table_name ((jsonb_field['key']));
+</programlisting>
+ </para>
+ </sect2>
+
<sect2>
<title>Transforms</title>
diff --git a/doc/src/sgml/ref/create_type.sgml b/doc/src/sgml/ref/create_type.sgml
index 111f8e65d2..a34df4d247 100644
--- a/doc/src/sgml/ref/create_type.sgml
+++ b/doc/src/sgml/ref/create_type.sgml
@@ -54,6 +54,7 @@ CREATE TYPE <replaceable class="parameter">name</replaceable> (
[ , ELEMENT = <replaceable class="parameter">element</replaceable> ]
[ , DELIMITER = <replaceable class="parameter">delimiter</replaceable> ]
[ , COLLATABLE = <replaceable class="parameter">collatable</replaceable> ]
+ [ , SUBSCRIPTING_HANDLER = <replaceable class="parameter">subscripting_handler_function</replaceable> ]
)
CREATE TYPE <replaceable class="parameter">name</replaceable>
@@ -196,8 +197,9 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
<replaceable class="parameter">receive_function</replaceable>,
<replaceable class="parameter">send_function</replaceable>,
<replaceable class="parameter">type_modifier_input_function</replaceable>,
- <replaceable class="parameter">type_modifier_output_function</replaceable> and
- <replaceable class="parameter">analyze_function</replaceable>
+ <replaceable class="parameter">type_modifier_output_function</replaceable>,
+ <replaceable class="parameter">analyze_function</replaceable>,
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
are optional. Generally these functions have to be coded in C
or another low-level language.
</para>
@@ -454,6 +456,22 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
make use of the collation information; this does not happen
automatically merely by marking the type collatable.
</para>
+
+ <para>
+ The optional
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
+ contains type-specific logic for subscripting of the data type.
+ By default, there is no such function provided, which means that the data
+ type doesn't support subscripting. The subscripting function must be
+ declared to take a single argument of type <type>internal</type>, and return
+ a <type>internal</type> result. There are two examples of implementation for
+ subscripting functions in case of array
+ (<replaceable class="parameter">array_subscripting_handler</replaceable>)
+ and jsonb
+ (<replaceable class="parameter">jsonb_subscripting_handler</replaceable>)
+ types in <filename>src/backend/utils/adt/arrayfuncs.c</filename> and
+ <filename>src/backend/utils/adt/jsonfuncs.c</filename>, respectively.
+ </para>
</refsect2>
<refsect2>
@@ -769,6 +787,17 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">subscripting_handler_function</replaceable></term>
+ <listitem>
+ <para>
+ The name of a function that returns list of type-specific callback functions to
+ support subscripting logic for the data type.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
diff --git a/doc/src/sgml/xsubscripting.sgml b/doc/src/sgml/xsubscripting.sgml
new file mode 100644
index 0000000000..7224e81fa2
--- /dev/null
+++ b/doc/src/sgml/xsubscripting.sgml
@@ -0,0 +1,111 @@
+<!-- doc/src/sgml/xsubscripting.sgml -->
+
+ <sect1 id="xsubscripting">
+ <title>User-defined subscripting procedure</title>
+
+ <indexterm zone="xsubscripting">
+ <primary>custom subscripting</primary>
+ </indexterm>
+ <para>
+ When you define a new base type, you can also specify a custom procedure to
+ handle subscripting expressions. It must contain logic for verification and
+ evaluation of this expression, i.e. fetching or updating some data in this
+ data type. For instance:
+</para>
+<programlisting><![CDATA[
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = someType;
+ sbsref->refassgntype = someType;
+
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ // some validation and coercion logic
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some assignment logic
+
+ return newContainer;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some fetch logic based on sbstate
+}]]>
+</programlisting>
+
+<para>
+ Then you can define a subscripting procedure and a custom data type:
+</para>
+<programlisting>
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '<replaceable>filename</replaceable>'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 4,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler,
+);
+</programlisting>
+
+<para>
+ and use it as usual:
+</para>
+<programlisting>
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
+</programlisting>
+
+
+ <para>
+ The examples of custom subscripting implementation can be found in
+ <filename>subscripting.sql</filename> and <filename>subscripting.c</filename>
+ in the <filename>src/tutorial</filename> directory of the source distribution.
+ See the <filename>README</filename> file in that directory for instructions
+ about running the examples.
+ </para>
+
+</sect1>
diff --git a/src/tutorial/Makefile b/src/tutorial/Makefile
index 16dc390f71..0ead60c2d4 100644
--- a/src/tutorial/Makefile
+++ b/src/tutorial/Makefile
@@ -13,8 +13,8 @@
#
#-------------------------------------------------------------------------
-MODULES = complex funcs
-DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql
+MODULES = complex funcs subscripting
+DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql subscripting.sql
ifdef NO_PGXS
subdir = src/tutorial
diff --git a/src/tutorial/subscripting.c b/src/tutorial/subscripting.c
new file mode 100644
index 0000000000..1eb8c45652
--- /dev/null
+++ b/src/tutorial/subscripting.c
@@ -0,0 +1,201 @@
+/*
+ * src/tutorial/subscripting.c
+ *
+ ******************************************************************************
+ This file contains routines that can be bound to a Postgres backend and
+ called by the backend in the process of processing queries. The calling
+ format for these routines is dictated by Postgres architecture.
+******************************************************************************/
+
+#include "postgres.h"
+
+#include "catalog/pg_type.h"
+#include "executor/executor.h"
+#include "executor/execExpr.h"
+#include "nodes/nodeFuncs.h"
+#include "parser/parse_coerce.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+SubscriptingRef * custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref);
+SubscriptingRef * custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate);
+Datum custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate);
+Datum custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate);
+
+PG_FUNCTION_INFO_V1(custom_in);
+PG_FUNCTION_INFO_V1(custom_out);
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+/*****************************************************************************
+ * Input/Output functions
+ *****************************************************************************/
+
+Datum
+custom_in(PG_FUNCTION_ARGS)
+{
+ char *str = PG_GETARG_CSTRING(0);
+ int firstValue,
+ secondValue;
+ Custom *result;
+
+ if (sscanf(str, " ( %d , %d )", &firstValue, &secondValue) != 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
+ errmsg("invalid input syntax for complex: \"%s\"",
+ str)));
+
+
+ result = (Custom *) palloc(sizeof(Custom));
+ result->first = firstValue;
+ result->second = secondValue;
+ PG_RETURN_POINTER(result);
+}
+
+Datum
+custom_out(PG_FUNCTION_ARGS)
+{
+ Custom *custom = (Custom *) PG_GETARG_POINTER(0);
+ char *result;
+
+ result = psprintf("(%d, %d)", custom->first, custom->second);
+ PG_RETURN_CSTRING(result);
+}
+
+/*****************************************************************************
+ * Custom subscripting logic functions
+ *****************************************************************************/
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = INT4OID;
+ sbsref->refassgntype = INT4OID;
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ List *upperIndexpr = NIL;
+ ListCell *l;
+
+ if (sbsref->reflowerindexpr != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *)lfirst(sbsref->reflowerindexpr->head))))));
+
+ foreach(l, sbsref->refupperindexpr)
+ {
+ Node *subexpr = (Node *) lfirst(l);
+
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *) lfirst(sbsref->refupperindexpr->head))))));
+
+ subexpr = coerce_to_target_type(pstate,
+ subexpr, exprType(subexpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript must have integer type"),
+ parser_errposition(pstate, exprLocation(subexpr))));
+
+ upperIndexpr = lappend(upperIndexpr, subexpr);
+
+ if (isAssignment)
+ {
+ Node *assignExpr = (Node *) sbsref->refassgnexpr;
+ Node *new_from;
+
+ new_from = coerce_to_target_type(pstate,
+ assignExpr, exprType(assignExpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (new_from == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom assignment requires int type"),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation(assignExpr))));
+ sbsref->refassgnexpr = (Expr *)new_from;
+ }
+ }
+
+ sbsref->refupperindexpr = upperIndexpr;
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ Custom *container= (Custom *) containerSource;
+ int index;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ return (Datum) container->first;
+ else
+ return (Datum) container->second;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ int index;
+ Custom *container = (Custom *) containerSource;
+
+ if (sbstate->resnull)
+ return containerSource;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ container->first = DatumGetInt32(sbstate->replacevalue);
+ else
+ container->second = DatumGetInt32(sbstate->replacevalue);
+
+ return (Datum) container;
+}
diff --git a/src/tutorial/subscripting.source b/src/tutorial/subscripting.source
new file mode 100644
index 0000000000..837cf30612
--- /dev/null
+++ b/src/tutorial/subscripting.source
@@ -0,0 +1,71 @@
+---------------------------------------------------------------------------
+--
+-- subscripting.sql-
+-- This file shows how to create a new subscripting procedure for
+-- user-defined type.
+--
+--
+-- Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+-- Portions Copyright (c) 1994, Regents of the University of California
+--
+-- src/tutorial/subscripting.source
+--
+---------------------------------------------------------------------------
+
+-----------------------------
+-- Creating a new type:
+-- We are going to create a new type called 'complex' which represents
+-- complex numbers.
+-- A user-defined type must have an input and an output function, and
+-- optionally can have binary input and output functions. All of these
+-- are usually user-defined C functions.
+-----------------------------
+
+-- Assume the user defined functions are in /home/erthalion/programms/postgresql-master/src/tutorial/complex$DLSUFFIX
+-- (we do not want to assume this is in the dynamic loader search path).
+-- Look at $PWD/complex.c for the source. Note that we declare all of
+-- them as STRICT, so we do not need to cope with NULL inputs in the
+-- C code. We also mark them IMMUTABLE, since they always return the
+-- same outputs given the same inputs.
+
+-- the input function 'complex_in' takes a null-terminated string (the
+-- textual representation of the type) and turns it into the internal
+-- (in memory) representation. You will get a message telling you 'complex'
+-- does not exist yet but that's okay.
+
+CREATE FUNCTION custom_in(cstring)
+ RETURNS custom
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+-- the output function 'complex_out' takes the internal representation and
+-- converts it into the textual representation.
+
+CREATE FUNCTION custom_out(custom)
+ RETURNS cstring
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 8,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler
+);
+
+-- we can use it in a table
+
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
--
2.21.0
--u66dlpg6gaevsofr
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v33-0005-Filling-gaps-in-jsonb-arrays.patch"
^ permalink raw reply [nested|flat] 55+ messages in thread
* [PATCH v34 4/5] Subscripting documentation
@ 2019-02-01 10:47 erthalion <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: erthalion @ 2019-02-01 10:47 UTC (permalink / raw)
Supporting documentation for generalized subscripting. It includes the
description of a new field in pg_type, the new section for jsonb
documentation about subscripting feature on this data type, and also the
tutorial about how to write subscripting operator for a custom data type.
Reviewed-by: Tom Lane, Arthur Zakirov, Oleksandr Shulgin
---
doc/src/sgml/catalogs.sgml | 8 ++
doc/src/sgml/extend.sgml | 6 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/json.sgml | 39 ++++++
doc/src/sgml/ref/create_type.sgml | 33 ++++-
doc/src/sgml/xsubscripting.sgml | 111 +++++++++++++++++
src/tutorial/Makefile | 4 +-
src/tutorial/subscripting.c | 201 ++++++++++++++++++++++++++++++
src/tutorial/subscripting.source | 71 +++++++++++
9 files changed, 470 insertions(+), 4 deletions(-)
create mode 100644 doc/src/sgml/xsubscripting.sgml
create mode 100644 src/tutorial/subscripting.c
create mode 100644 src/tutorial/subscripting.source
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 7e99928d0c..104cea3584 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8990,6 +8990,14 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>typsubshandler</structfield></entry>
+ <entry><type>regproc</type></entry>
+ <entry><literal><link linkend="catalog-pg-proc"><structname>pg_proc</structname></link>.oid</literal></entry>
+ <entry>Custom subscripting function with type-specific logic for parsing
+ and validation, or 0 if this type doesn't support subscripting.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>typdefault</structfield> <type>text</type>
diff --git a/doc/src/sgml/extend.sgml b/doc/src/sgml/extend.sgml
index e486006224..23c1764571 100644
--- a/doc/src/sgml/extend.sgml
+++ b/doc/src/sgml/extend.sgml
@@ -33,6 +33,11 @@
operators (starting in <xref linkend="xoper"/>)
</para>
</listitem>
+ <listitem>
+ <para>
+ subscripting procedure (starting in <xref linkend="xsubscripting"/>)
+ </para>
+ </listitem>
<listitem>
<para>
operator classes for indexes (starting in <xref linkend="xindex"/>)
@@ -481,6 +486,7 @@ RETURNS anycompatible AS ...
&xaggr;
&xtypes;
&xoper;
+ &xsubscripting;
&xindex;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 828396d4a9..3bebb34c1a 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -69,6 +69,7 @@
<!ENTITY xplang SYSTEM "xplang.sgml">
<!ENTITY xoper SYSTEM "xoper.sgml">
<!ENTITY xtypes SYSTEM "xtypes.sgml">
+<!ENTITY xsubscripting SYSTEM "xsubscripting.sgml">
<!ENTITY plperl SYSTEM "plperl.sgml">
<!ENTITY plpython SYSTEM "plpython.sgml">
<!ENTITY plsql SYSTEM "plpgsql.sgml">
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index c0a6554d4d..5c538dca05 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -602,6 +602,45 @@ SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qu
</para>
</sect2>
+ <sect2 id="jsonb-subscripting">
+ <title><type>jsonb</type> Subscripting</title>
+ <para>
+ <type>jsonb</type> data type supports array-style subscripting expressions
+ to extract or update particular elements. It's possible to use multiple
+ subscripting expressions to extract nested values. In this case, a chain of
+ subscripting expressions follows the same rules as the
+ <literal>path</literal> argument in <literal>jsonb_set</literal> function,
+ e.g. in case of arrays it is a 0-based operation or that negative integers
+ that appear in <literal>path</literal> count from the end of JSON arrays.
+ The result of subscripting expressions is always jsonb data type. An
+ example of subscripting syntax:
+<programlisting>
+-- Extract value by key
+SELECT ('{"a": 1}'::jsonb)['a'];
+
+-- Extract nested value by key path
+SELECT ('{"a": {"b": {"c": 1}}}'::jsonb)['a']['b']['c'];
+
+-- Extract element by index
+SELECT ('[1, "2", null]'::jsonb)[1];
+
+-- Update value by key
+UPDATE table_name SET jsonb_field['key'] = 1;
+
+-- Select records using where clause with subscripting. Since the result of
+-- subscripting is jsonb and we basically want to compare two jsonb objects, we
+-- need to put the value in double quotes to be able to convert it to jsonb.
+SELECT * FROM table_name WHERE jsonb_field['key'] = '"value"';
+</programlisting>
+
+ There is no special indexing support for such kind of expressions, but you
+ can always create a functional index that includes it
+<programlisting>
+CREATE INDEX idx ON table_name ((jsonb_field['key']));
+</programlisting>
+ </para>
+ </sect2>
+
<sect2>
<title>Transforms</title>
diff --git a/doc/src/sgml/ref/create_type.sgml b/doc/src/sgml/ref/create_type.sgml
index 111f8e65d2..a34df4d247 100644
--- a/doc/src/sgml/ref/create_type.sgml
+++ b/doc/src/sgml/ref/create_type.sgml
@@ -54,6 +54,7 @@ CREATE TYPE <replaceable class="parameter">name</replaceable> (
[ , ELEMENT = <replaceable class="parameter">element</replaceable> ]
[ , DELIMITER = <replaceable class="parameter">delimiter</replaceable> ]
[ , COLLATABLE = <replaceable class="parameter">collatable</replaceable> ]
+ [ , SUBSCRIPTING_HANDLER = <replaceable class="parameter">subscripting_handler_function</replaceable> ]
)
CREATE TYPE <replaceable class="parameter">name</replaceable>
@@ -196,8 +197,9 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
<replaceable class="parameter">receive_function</replaceable>,
<replaceable class="parameter">send_function</replaceable>,
<replaceable class="parameter">type_modifier_input_function</replaceable>,
- <replaceable class="parameter">type_modifier_output_function</replaceable> and
- <replaceable class="parameter">analyze_function</replaceable>
+ <replaceable class="parameter">type_modifier_output_function</replaceable>,
+ <replaceable class="parameter">analyze_function</replaceable>,
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
are optional. Generally these functions have to be coded in C
or another low-level language.
</para>
@@ -454,6 +456,22 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
make use of the collation information; this does not happen
automatically merely by marking the type collatable.
</para>
+
+ <para>
+ The optional
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
+ contains type-specific logic for subscripting of the data type.
+ By default, there is no such function provided, which means that the data
+ type doesn't support subscripting. The subscripting function must be
+ declared to take a single argument of type <type>internal</type>, and return
+ a <type>internal</type> result. There are two examples of implementation for
+ subscripting functions in case of array
+ (<replaceable class="parameter">array_subscripting_handler</replaceable>)
+ and jsonb
+ (<replaceable class="parameter">jsonb_subscripting_handler</replaceable>)
+ types in <filename>src/backend/utils/adt/arrayfuncs.c</filename> and
+ <filename>src/backend/utils/adt/jsonfuncs.c</filename>, respectively.
+ </para>
</refsect2>
<refsect2>
@@ -769,6 +787,17 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">subscripting_handler_function</replaceable></term>
+ <listitem>
+ <para>
+ The name of a function that returns list of type-specific callback functions to
+ support subscripting logic for the data type.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
diff --git a/doc/src/sgml/xsubscripting.sgml b/doc/src/sgml/xsubscripting.sgml
new file mode 100644
index 0000000000..7224e81fa2
--- /dev/null
+++ b/doc/src/sgml/xsubscripting.sgml
@@ -0,0 +1,111 @@
+<!-- doc/src/sgml/xsubscripting.sgml -->
+
+ <sect1 id="xsubscripting">
+ <title>User-defined subscripting procedure</title>
+
+ <indexterm zone="xsubscripting">
+ <primary>custom subscripting</primary>
+ </indexterm>
+ <para>
+ When you define a new base type, you can also specify a custom procedure to
+ handle subscripting expressions. It must contain logic for verification and
+ evaluation of this expression, i.e. fetching or updating some data in this
+ data type. For instance:
+</para>
+<programlisting><![CDATA[
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = someType;
+ sbsref->refassgntype = someType;
+
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ // some validation and coercion logic
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some assignment logic
+
+ return newContainer;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some fetch logic based on sbstate
+}]]>
+</programlisting>
+
+<para>
+ Then you can define a subscripting procedure and a custom data type:
+</para>
+<programlisting>
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '<replaceable>filename</replaceable>'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 4,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler,
+);
+</programlisting>
+
+<para>
+ and use it as usual:
+</para>
+<programlisting>
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
+</programlisting>
+
+
+ <para>
+ The examples of custom subscripting implementation can be found in
+ <filename>subscripting.sql</filename> and <filename>subscripting.c</filename>
+ in the <filename>src/tutorial</filename> directory of the source distribution.
+ See the <filename>README</filename> file in that directory for instructions
+ about running the examples.
+ </para>
+
+</sect1>
diff --git a/src/tutorial/Makefile b/src/tutorial/Makefile
index 16dc390f71..0ead60c2d4 100644
--- a/src/tutorial/Makefile
+++ b/src/tutorial/Makefile
@@ -13,8 +13,8 @@
#
#-------------------------------------------------------------------------
-MODULES = complex funcs
-DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql
+MODULES = complex funcs subscripting
+DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql subscripting.sql
ifdef NO_PGXS
subdir = src/tutorial
diff --git a/src/tutorial/subscripting.c b/src/tutorial/subscripting.c
new file mode 100644
index 0000000000..1eb8c45652
--- /dev/null
+++ b/src/tutorial/subscripting.c
@@ -0,0 +1,201 @@
+/*
+ * src/tutorial/subscripting.c
+ *
+ ******************************************************************************
+ This file contains routines that can be bound to a Postgres backend and
+ called by the backend in the process of processing queries. The calling
+ format for these routines is dictated by Postgres architecture.
+******************************************************************************/
+
+#include "postgres.h"
+
+#include "catalog/pg_type.h"
+#include "executor/executor.h"
+#include "executor/execExpr.h"
+#include "nodes/nodeFuncs.h"
+#include "parser/parse_coerce.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+SubscriptingRef * custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref);
+SubscriptingRef * custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate);
+Datum custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate);
+Datum custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate);
+
+PG_FUNCTION_INFO_V1(custom_in);
+PG_FUNCTION_INFO_V1(custom_out);
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+/*****************************************************************************
+ * Input/Output functions
+ *****************************************************************************/
+
+Datum
+custom_in(PG_FUNCTION_ARGS)
+{
+ char *str = PG_GETARG_CSTRING(0);
+ int firstValue,
+ secondValue;
+ Custom *result;
+
+ if (sscanf(str, " ( %d , %d )", &firstValue, &secondValue) != 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
+ errmsg("invalid input syntax for complex: \"%s\"",
+ str)));
+
+
+ result = (Custom *) palloc(sizeof(Custom));
+ result->first = firstValue;
+ result->second = secondValue;
+ PG_RETURN_POINTER(result);
+}
+
+Datum
+custom_out(PG_FUNCTION_ARGS)
+{
+ Custom *custom = (Custom *) PG_GETARG_POINTER(0);
+ char *result;
+
+ result = psprintf("(%d, %d)", custom->first, custom->second);
+ PG_RETURN_CSTRING(result);
+}
+
+/*****************************************************************************
+ * Custom subscripting logic functions
+ *****************************************************************************/
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = INT4OID;
+ sbsref->refassgntype = INT4OID;
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ List *upperIndexpr = NIL;
+ ListCell *l;
+
+ if (sbsref->reflowerindexpr != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *)lfirst(sbsref->reflowerindexpr->head))))));
+
+ foreach(l, sbsref->refupperindexpr)
+ {
+ Node *subexpr = (Node *) lfirst(l);
+
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *) lfirst(sbsref->refupperindexpr->head))))));
+
+ subexpr = coerce_to_target_type(pstate,
+ subexpr, exprType(subexpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript must have integer type"),
+ parser_errposition(pstate, exprLocation(subexpr))));
+
+ upperIndexpr = lappend(upperIndexpr, subexpr);
+
+ if (isAssignment)
+ {
+ Node *assignExpr = (Node *) sbsref->refassgnexpr;
+ Node *new_from;
+
+ new_from = coerce_to_target_type(pstate,
+ assignExpr, exprType(assignExpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (new_from == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom assignment requires int type"),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation(assignExpr))));
+ sbsref->refassgnexpr = (Expr *)new_from;
+ }
+ }
+
+ sbsref->refupperindexpr = upperIndexpr;
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ Custom *container= (Custom *) containerSource;
+ int index;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ return (Datum) container->first;
+ else
+ return (Datum) container->second;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ int index;
+ Custom *container = (Custom *) containerSource;
+
+ if (sbstate->resnull)
+ return containerSource;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ container->first = DatumGetInt32(sbstate->replacevalue);
+ else
+ container->second = DatumGetInt32(sbstate->replacevalue);
+
+ return (Datum) container;
+}
diff --git a/src/tutorial/subscripting.source b/src/tutorial/subscripting.source
new file mode 100644
index 0000000000..837cf30612
--- /dev/null
+++ b/src/tutorial/subscripting.source
@@ -0,0 +1,71 @@
+---------------------------------------------------------------------------
+--
+-- subscripting.sql-
+-- This file shows how to create a new subscripting procedure for
+-- user-defined type.
+--
+--
+-- Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+-- Portions Copyright (c) 1994, Regents of the University of California
+--
+-- src/tutorial/subscripting.source
+--
+---------------------------------------------------------------------------
+
+-----------------------------
+-- Creating a new type:
+-- We are going to create a new type called 'complex' which represents
+-- complex numbers.
+-- A user-defined type must have an input and an output function, and
+-- optionally can have binary input and output functions. All of these
+-- are usually user-defined C functions.
+-----------------------------
+
+-- Assume the user defined functions are in /home/erthalion/programms/postgresql-master/src/tutorial/complex$DLSUFFIX
+-- (we do not want to assume this is in the dynamic loader search path).
+-- Look at $PWD/complex.c for the source. Note that we declare all of
+-- them as STRICT, so we do not need to cope with NULL inputs in the
+-- C code. We also mark them IMMUTABLE, since they always return the
+-- same outputs given the same inputs.
+
+-- the input function 'complex_in' takes a null-terminated string (the
+-- textual representation of the type) and turns it into the internal
+-- (in memory) representation. You will get a message telling you 'complex'
+-- does not exist yet but that's okay.
+
+CREATE FUNCTION custom_in(cstring)
+ RETURNS custom
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+-- the output function 'complex_out' takes the internal representation and
+-- converts it into the textual representation.
+
+CREATE FUNCTION custom_out(custom)
+ RETURNS cstring
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 8,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler
+);
+
+-- we can use it in a table
+
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
--
2.21.0
--j77abfbqdsj2lq6d
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v34-0005-Filling-gaps-in-jsonb-arrays.patch"
^ permalink raw reply [nested|flat] 55+ messages in thread
* [PATCH v33 4/5] Subscripting documentation
@ 2019-02-01 10:47 erthalion <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: erthalion @ 2019-02-01 10:47 UTC (permalink / raw)
Supporting documentation for generalized subscripting. It includes the
description of a new field in pg_type, the new section for jsonb
documentation about subscripting feature on this data type, and also the
tutorial about how to write subscripting operator for a custom data type.
Reviewed-by: Tom Lane, Arthur Zakirov, Oleksandr Shulgin
---
doc/src/sgml/catalogs.sgml | 8 ++
doc/src/sgml/extend.sgml | 6 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/json.sgml | 39 ++++++
doc/src/sgml/ref/create_type.sgml | 33 ++++-
doc/src/sgml/xsubscripting.sgml | 111 +++++++++++++++++
src/tutorial/Makefile | 4 +-
src/tutorial/subscripting.c | 201 ++++++++++++++++++++++++++++++
src/tutorial/subscripting.source | 71 +++++++++++
9 files changed, 470 insertions(+), 4 deletions(-)
create mode 100644 doc/src/sgml/xsubscripting.sgml
create mode 100644 src/tutorial/subscripting.c
create mode 100644 src/tutorial/subscripting.source
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 048ff284f7..b17e9ee382 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8971,6 +8971,14 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>typsubshandler</structfield></entry>
+ <entry><type>regproc</type></entry>
+ <entry><literal><link linkend="catalog-pg-proc"><structname>pg_proc</structname></link>.oid</literal></entry>
+ <entry>Custom subscripting function with type-specific logic for parsing
+ and validation, or 0 if this type doesn't support subscripting.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>typdefault</structfield> <type>text</type>
diff --git a/doc/src/sgml/extend.sgml b/doc/src/sgml/extend.sgml
index 890ff97b7a..65cf5706b6 100644
--- a/doc/src/sgml/extend.sgml
+++ b/doc/src/sgml/extend.sgml
@@ -33,6 +33,11 @@
operators (starting in <xref linkend="xoper"/>)
</para>
</listitem>
+ <listitem>
+ <para>
+ subscripting procedure (starting in <xref linkend="xsubscripting"/>)
+ </para>
+ </listitem>
<listitem>
<para>
operator classes for indexes (starting in <xref linkend="xindex"/>)
@@ -481,6 +486,7 @@ RETURNS anycompatible AS ...
&xaggr;
&xtypes;
&xoper;
+ &xsubscripting;
&xindex;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 64b5da0070..4c3fd37fc1 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -69,6 +69,7 @@
<!ENTITY xplang SYSTEM "xplang.sgml">
<!ENTITY xoper SYSTEM "xoper.sgml">
<!ENTITY xtypes SYSTEM "xtypes.sgml">
+<!ENTITY xsubscripting SYSTEM "xsubscripting.sgml">
<!ENTITY plperl SYSTEM "plperl.sgml">
<!ENTITY plpython SYSTEM "plpython.sgml">
<!ENTITY plsql SYSTEM "plpgsql.sgml">
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index c0a6554d4d..5c538dca05 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -602,6 +602,45 @@ SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qu
</para>
</sect2>
+ <sect2 id="jsonb-subscripting">
+ <title><type>jsonb</type> Subscripting</title>
+ <para>
+ <type>jsonb</type> data type supports array-style subscripting expressions
+ to extract or update particular elements. It's possible to use multiple
+ subscripting expressions to extract nested values. In this case, a chain of
+ subscripting expressions follows the same rules as the
+ <literal>path</literal> argument in <literal>jsonb_set</literal> function,
+ e.g. in case of arrays it is a 0-based operation or that negative integers
+ that appear in <literal>path</literal> count from the end of JSON arrays.
+ The result of subscripting expressions is always jsonb data type. An
+ example of subscripting syntax:
+<programlisting>
+-- Extract value by key
+SELECT ('{"a": 1}'::jsonb)['a'];
+
+-- Extract nested value by key path
+SELECT ('{"a": {"b": {"c": 1}}}'::jsonb)['a']['b']['c'];
+
+-- Extract element by index
+SELECT ('[1, "2", null]'::jsonb)[1];
+
+-- Update value by key
+UPDATE table_name SET jsonb_field['key'] = 1;
+
+-- Select records using where clause with subscripting. Since the result of
+-- subscripting is jsonb and we basically want to compare two jsonb objects, we
+-- need to put the value in double quotes to be able to convert it to jsonb.
+SELECT * FROM table_name WHERE jsonb_field['key'] = '"value"';
+</programlisting>
+
+ There is no special indexing support for such kind of expressions, but you
+ can always create a functional index that includes it
+<programlisting>
+CREATE INDEX idx ON table_name ((jsonb_field['key']));
+</programlisting>
+ </para>
+ </sect2>
+
<sect2>
<title>Transforms</title>
diff --git a/doc/src/sgml/ref/create_type.sgml b/doc/src/sgml/ref/create_type.sgml
index 111f8e65d2..a34df4d247 100644
--- a/doc/src/sgml/ref/create_type.sgml
+++ b/doc/src/sgml/ref/create_type.sgml
@@ -54,6 +54,7 @@ CREATE TYPE <replaceable class="parameter">name</replaceable> (
[ , ELEMENT = <replaceable class="parameter">element</replaceable> ]
[ , DELIMITER = <replaceable class="parameter">delimiter</replaceable> ]
[ , COLLATABLE = <replaceable class="parameter">collatable</replaceable> ]
+ [ , SUBSCRIPTING_HANDLER = <replaceable class="parameter">subscripting_handler_function</replaceable> ]
)
CREATE TYPE <replaceable class="parameter">name</replaceable>
@@ -196,8 +197,9 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
<replaceable class="parameter">receive_function</replaceable>,
<replaceable class="parameter">send_function</replaceable>,
<replaceable class="parameter">type_modifier_input_function</replaceable>,
- <replaceable class="parameter">type_modifier_output_function</replaceable> and
- <replaceable class="parameter">analyze_function</replaceable>
+ <replaceable class="parameter">type_modifier_output_function</replaceable>,
+ <replaceable class="parameter">analyze_function</replaceable>,
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
are optional. Generally these functions have to be coded in C
or another low-level language.
</para>
@@ -454,6 +456,22 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
make use of the collation information; this does not happen
automatically merely by marking the type collatable.
</para>
+
+ <para>
+ The optional
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
+ contains type-specific logic for subscripting of the data type.
+ By default, there is no such function provided, which means that the data
+ type doesn't support subscripting. The subscripting function must be
+ declared to take a single argument of type <type>internal</type>, and return
+ a <type>internal</type> result. There are two examples of implementation for
+ subscripting functions in case of array
+ (<replaceable class="parameter">array_subscripting_handler</replaceable>)
+ and jsonb
+ (<replaceable class="parameter">jsonb_subscripting_handler</replaceable>)
+ types in <filename>src/backend/utils/adt/arrayfuncs.c</filename> and
+ <filename>src/backend/utils/adt/jsonfuncs.c</filename>, respectively.
+ </para>
</refsect2>
<refsect2>
@@ -769,6 +787,17 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">subscripting_handler_function</replaceable></term>
+ <listitem>
+ <para>
+ The name of a function that returns list of type-specific callback functions to
+ support subscripting logic for the data type.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
diff --git a/doc/src/sgml/xsubscripting.sgml b/doc/src/sgml/xsubscripting.sgml
new file mode 100644
index 0000000000..7224e81fa2
--- /dev/null
+++ b/doc/src/sgml/xsubscripting.sgml
@@ -0,0 +1,111 @@
+<!-- doc/src/sgml/xsubscripting.sgml -->
+
+ <sect1 id="xsubscripting">
+ <title>User-defined subscripting procedure</title>
+
+ <indexterm zone="xsubscripting">
+ <primary>custom subscripting</primary>
+ </indexterm>
+ <para>
+ When you define a new base type, you can also specify a custom procedure to
+ handle subscripting expressions. It must contain logic for verification and
+ evaluation of this expression, i.e. fetching or updating some data in this
+ data type. For instance:
+</para>
+<programlisting><![CDATA[
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = someType;
+ sbsref->refassgntype = someType;
+
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ // some validation and coercion logic
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some assignment logic
+
+ return newContainer;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some fetch logic based on sbstate
+}]]>
+</programlisting>
+
+<para>
+ Then you can define a subscripting procedure and a custom data type:
+</para>
+<programlisting>
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '<replaceable>filename</replaceable>'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 4,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler,
+);
+</programlisting>
+
+<para>
+ and use it as usual:
+</para>
+<programlisting>
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
+</programlisting>
+
+
+ <para>
+ The examples of custom subscripting implementation can be found in
+ <filename>subscripting.sql</filename> and <filename>subscripting.c</filename>
+ in the <filename>src/tutorial</filename> directory of the source distribution.
+ See the <filename>README</filename> file in that directory for instructions
+ about running the examples.
+ </para>
+
+</sect1>
diff --git a/src/tutorial/Makefile b/src/tutorial/Makefile
index 16dc390f71..0ead60c2d4 100644
--- a/src/tutorial/Makefile
+++ b/src/tutorial/Makefile
@@ -13,8 +13,8 @@
#
#-------------------------------------------------------------------------
-MODULES = complex funcs
-DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql
+MODULES = complex funcs subscripting
+DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql subscripting.sql
ifdef NO_PGXS
subdir = src/tutorial
diff --git a/src/tutorial/subscripting.c b/src/tutorial/subscripting.c
new file mode 100644
index 0000000000..1eb8c45652
--- /dev/null
+++ b/src/tutorial/subscripting.c
@@ -0,0 +1,201 @@
+/*
+ * src/tutorial/subscripting.c
+ *
+ ******************************************************************************
+ This file contains routines that can be bound to a Postgres backend and
+ called by the backend in the process of processing queries. The calling
+ format for these routines is dictated by Postgres architecture.
+******************************************************************************/
+
+#include "postgres.h"
+
+#include "catalog/pg_type.h"
+#include "executor/executor.h"
+#include "executor/execExpr.h"
+#include "nodes/nodeFuncs.h"
+#include "parser/parse_coerce.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+SubscriptingRef * custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref);
+SubscriptingRef * custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate);
+Datum custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate);
+Datum custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate);
+
+PG_FUNCTION_INFO_V1(custom_in);
+PG_FUNCTION_INFO_V1(custom_out);
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+/*****************************************************************************
+ * Input/Output functions
+ *****************************************************************************/
+
+Datum
+custom_in(PG_FUNCTION_ARGS)
+{
+ char *str = PG_GETARG_CSTRING(0);
+ int firstValue,
+ secondValue;
+ Custom *result;
+
+ if (sscanf(str, " ( %d , %d )", &firstValue, &secondValue) != 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
+ errmsg("invalid input syntax for complex: \"%s\"",
+ str)));
+
+
+ result = (Custom *) palloc(sizeof(Custom));
+ result->first = firstValue;
+ result->second = secondValue;
+ PG_RETURN_POINTER(result);
+}
+
+Datum
+custom_out(PG_FUNCTION_ARGS)
+{
+ Custom *custom = (Custom *) PG_GETARG_POINTER(0);
+ char *result;
+
+ result = psprintf("(%d, %d)", custom->first, custom->second);
+ PG_RETURN_CSTRING(result);
+}
+
+/*****************************************************************************
+ * Custom subscripting logic functions
+ *****************************************************************************/
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = INT4OID;
+ sbsref->refassgntype = INT4OID;
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ List *upperIndexpr = NIL;
+ ListCell *l;
+
+ if (sbsref->reflowerindexpr != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *)lfirst(sbsref->reflowerindexpr->head))))));
+
+ foreach(l, sbsref->refupperindexpr)
+ {
+ Node *subexpr = (Node *) lfirst(l);
+
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *) lfirst(sbsref->refupperindexpr->head))))));
+
+ subexpr = coerce_to_target_type(pstate,
+ subexpr, exprType(subexpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript must have integer type"),
+ parser_errposition(pstate, exprLocation(subexpr))));
+
+ upperIndexpr = lappend(upperIndexpr, subexpr);
+
+ if (isAssignment)
+ {
+ Node *assignExpr = (Node *) sbsref->refassgnexpr;
+ Node *new_from;
+
+ new_from = coerce_to_target_type(pstate,
+ assignExpr, exprType(assignExpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (new_from == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom assignment requires int type"),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation(assignExpr))));
+ sbsref->refassgnexpr = (Expr *)new_from;
+ }
+ }
+
+ sbsref->refupperindexpr = upperIndexpr;
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ Custom *container= (Custom *) containerSource;
+ int index;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ return (Datum) container->first;
+ else
+ return (Datum) container->second;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ int index;
+ Custom *container = (Custom *) containerSource;
+
+ if (sbstate->resnull)
+ return containerSource;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ container->first = DatumGetInt32(sbstate->replacevalue);
+ else
+ container->second = DatumGetInt32(sbstate->replacevalue);
+
+ return (Datum) container;
+}
diff --git a/src/tutorial/subscripting.source b/src/tutorial/subscripting.source
new file mode 100644
index 0000000000..837cf30612
--- /dev/null
+++ b/src/tutorial/subscripting.source
@@ -0,0 +1,71 @@
+---------------------------------------------------------------------------
+--
+-- subscripting.sql-
+-- This file shows how to create a new subscripting procedure for
+-- user-defined type.
+--
+--
+-- Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+-- Portions Copyright (c) 1994, Regents of the University of California
+--
+-- src/tutorial/subscripting.source
+--
+---------------------------------------------------------------------------
+
+-----------------------------
+-- Creating a new type:
+-- We are going to create a new type called 'complex' which represents
+-- complex numbers.
+-- A user-defined type must have an input and an output function, and
+-- optionally can have binary input and output functions. All of these
+-- are usually user-defined C functions.
+-----------------------------
+
+-- Assume the user defined functions are in /home/erthalion/programms/postgresql-master/src/tutorial/complex$DLSUFFIX
+-- (we do not want to assume this is in the dynamic loader search path).
+-- Look at $PWD/complex.c for the source. Note that we declare all of
+-- them as STRICT, so we do not need to cope with NULL inputs in the
+-- C code. We also mark them IMMUTABLE, since they always return the
+-- same outputs given the same inputs.
+
+-- the input function 'complex_in' takes a null-terminated string (the
+-- textual representation of the type) and turns it into the internal
+-- (in memory) representation. You will get a message telling you 'complex'
+-- does not exist yet but that's okay.
+
+CREATE FUNCTION custom_in(cstring)
+ RETURNS custom
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+-- the output function 'complex_out' takes the internal representation and
+-- converts it into the textual representation.
+
+CREATE FUNCTION custom_out(custom)
+ RETURNS cstring
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 8,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler
+);
+
+-- we can use it in a table
+
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
--
2.21.0
--u66dlpg6gaevsofr
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v33-0005-Filling-gaps-in-jsonb-arrays.patch"
^ permalink raw reply [nested|flat] 55+ messages in thread
* [PATCH v33 4/5] Subscripting documentation
@ 2019-02-01 10:47 erthalion <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: erthalion @ 2019-02-01 10:47 UTC (permalink / raw)
Supporting documentation for generalized subscripting. It includes the
description of a new field in pg_type, the new section for jsonb
documentation about subscripting feature on this data type, and also the
tutorial about how to write subscripting operator for a custom data type.
Reviewed-by: Tom Lane, Arthur Zakirov, Oleksandr Shulgin
---
doc/src/sgml/catalogs.sgml | 8 ++
doc/src/sgml/extend.sgml | 6 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/json.sgml | 39 ++++++
doc/src/sgml/ref/create_type.sgml | 33 ++++-
doc/src/sgml/xsubscripting.sgml | 111 +++++++++++++++++
src/tutorial/Makefile | 4 +-
src/tutorial/subscripting.c | 201 ++++++++++++++++++++++++++++++
src/tutorial/subscripting.source | 71 +++++++++++
9 files changed, 470 insertions(+), 4 deletions(-)
create mode 100644 doc/src/sgml/xsubscripting.sgml
create mode 100644 src/tutorial/subscripting.c
create mode 100644 src/tutorial/subscripting.source
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 048ff284f7..b17e9ee382 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8971,6 +8971,14 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>typsubshandler</structfield></entry>
+ <entry><type>regproc</type></entry>
+ <entry><literal><link linkend="catalog-pg-proc"><structname>pg_proc</structname></link>.oid</literal></entry>
+ <entry>Custom subscripting function with type-specific logic for parsing
+ and validation, or 0 if this type doesn't support subscripting.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>typdefault</structfield> <type>text</type>
diff --git a/doc/src/sgml/extend.sgml b/doc/src/sgml/extend.sgml
index 890ff97b7a..65cf5706b6 100644
--- a/doc/src/sgml/extend.sgml
+++ b/doc/src/sgml/extend.sgml
@@ -33,6 +33,11 @@
operators (starting in <xref linkend="xoper"/>)
</para>
</listitem>
+ <listitem>
+ <para>
+ subscripting procedure (starting in <xref linkend="xsubscripting"/>)
+ </para>
+ </listitem>
<listitem>
<para>
operator classes for indexes (starting in <xref linkend="xindex"/>)
@@ -481,6 +486,7 @@ RETURNS anycompatible AS ...
&xaggr;
&xtypes;
&xoper;
+ &xsubscripting;
&xindex;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 64b5da0070..4c3fd37fc1 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -69,6 +69,7 @@
<!ENTITY xplang SYSTEM "xplang.sgml">
<!ENTITY xoper SYSTEM "xoper.sgml">
<!ENTITY xtypes SYSTEM "xtypes.sgml">
+<!ENTITY xsubscripting SYSTEM "xsubscripting.sgml">
<!ENTITY plperl SYSTEM "plperl.sgml">
<!ENTITY plpython SYSTEM "plpython.sgml">
<!ENTITY plsql SYSTEM "plpgsql.sgml">
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index c0a6554d4d..5c538dca05 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -602,6 +602,45 @@ SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qu
</para>
</sect2>
+ <sect2 id="jsonb-subscripting">
+ <title><type>jsonb</type> Subscripting</title>
+ <para>
+ <type>jsonb</type> data type supports array-style subscripting expressions
+ to extract or update particular elements. It's possible to use multiple
+ subscripting expressions to extract nested values. In this case, a chain of
+ subscripting expressions follows the same rules as the
+ <literal>path</literal> argument in <literal>jsonb_set</literal> function,
+ e.g. in case of arrays it is a 0-based operation or that negative integers
+ that appear in <literal>path</literal> count from the end of JSON arrays.
+ The result of subscripting expressions is always jsonb data type. An
+ example of subscripting syntax:
+<programlisting>
+-- Extract value by key
+SELECT ('{"a": 1}'::jsonb)['a'];
+
+-- Extract nested value by key path
+SELECT ('{"a": {"b": {"c": 1}}}'::jsonb)['a']['b']['c'];
+
+-- Extract element by index
+SELECT ('[1, "2", null]'::jsonb)[1];
+
+-- Update value by key
+UPDATE table_name SET jsonb_field['key'] = 1;
+
+-- Select records using where clause with subscripting. Since the result of
+-- subscripting is jsonb and we basically want to compare two jsonb objects, we
+-- need to put the value in double quotes to be able to convert it to jsonb.
+SELECT * FROM table_name WHERE jsonb_field['key'] = '"value"';
+</programlisting>
+
+ There is no special indexing support for such kind of expressions, but you
+ can always create a functional index that includes it
+<programlisting>
+CREATE INDEX idx ON table_name ((jsonb_field['key']));
+</programlisting>
+ </para>
+ </sect2>
+
<sect2>
<title>Transforms</title>
diff --git a/doc/src/sgml/ref/create_type.sgml b/doc/src/sgml/ref/create_type.sgml
index 111f8e65d2..a34df4d247 100644
--- a/doc/src/sgml/ref/create_type.sgml
+++ b/doc/src/sgml/ref/create_type.sgml
@@ -54,6 +54,7 @@ CREATE TYPE <replaceable class="parameter">name</replaceable> (
[ , ELEMENT = <replaceable class="parameter">element</replaceable> ]
[ , DELIMITER = <replaceable class="parameter">delimiter</replaceable> ]
[ , COLLATABLE = <replaceable class="parameter">collatable</replaceable> ]
+ [ , SUBSCRIPTING_HANDLER = <replaceable class="parameter">subscripting_handler_function</replaceable> ]
)
CREATE TYPE <replaceable class="parameter">name</replaceable>
@@ -196,8 +197,9 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
<replaceable class="parameter">receive_function</replaceable>,
<replaceable class="parameter">send_function</replaceable>,
<replaceable class="parameter">type_modifier_input_function</replaceable>,
- <replaceable class="parameter">type_modifier_output_function</replaceable> and
- <replaceable class="parameter">analyze_function</replaceable>
+ <replaceable class="parameter">type_modifier_output_function</replaceable>,
+ <replaceable class="parameter">analyze_function</replaceable>,
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
are optional. Generally these functions have to be coded in C
or another low-level language.
</para>
@@ -454,6 +456,22 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
make use of the collation information; this does not happen
automatically merely by marking the type collatable.
</para>
+
+ <para>
+ The optional
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
+ contains type-specific logic for subscripting of the data type.
+ By default, there is no such function provided, which means that the data
+ type doesn't support subscripting. The subscripting function must be
+ declared to take a single argument of type <type>internal</type>, and return
+ a <type>internal</type> result. There are two examples of implementation for
+ subscripting functions in case of array
+ (<replaceable class="parameter">array_subscripting_handler</replaceable>)
+ and jsonb
+ (<replaceable class="parameter">jsonb_subscripting_handler</replaceable>)
+ types in <filename>src/backend/utils/adt/arrayfuncs.c</filename> and
+ <filename>src/backend/utils/adt/jsonfuncs.c</filename>, respectively.
+ </para>
</refsect2>
<refsect2>
@@ -769,6 +787,17 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">subscripting_handler_function</replaceable></term>
+ <listitem>
+ <para>
+ The name of a function that returns list of type-specific callback functions to
+ support subscripting logic for the data type.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
diff --git a/doc/src/sgml/xsubscripting.sgml b/doc/src/sgml/xsubscripting.sgml
new file mode 100644
index 0000000000..7224e81fa2
--- /dev/null
+++ b/doc/src/sgml/xsubscripting.sgml
@@ -0,0 +1,111 @@
+<!-- doc/src/sgml/xsubscripting.sgml -->
+
+ <sect1 id="xsubscripting">
+ <title>User-defined subscripting procedure</title>
+
+ <indexterm zone="xsubscripting">
+ <primary>custom subscripting</primary>
+ </indexterm>
+ <para>
+ When you define a new base type, you can also specify a custom procedure to
+ handle subscripting expressions. It must contain logic for verification and
+ evaluation of this expression, i.e. fetching or updating some data in this
+ data type. For instance:
+</para>
+<programlisting><![CDATA[
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = someType;
+ sbsref->refassgntype = someType;
+
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ // some validation and coercion logic
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some assignment logic
+
+ return newContainer;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some fetch logic based on sbstate
+}]]>
+</programlisting>
+
+<para>
+ Then you can define a subscripting procedure and a custom data type:
+</para>
+<programlisting>
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '<replaceable>filename</replaceable>'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 4,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler,
+);
+</programlisting>
+
+<para>
+ and use it as usual:
+</para>
+<programlisting>
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
+</programlisting>
+
+
+ <para>
+ The examples of custom subscripting implementation can be found in
+ <filename>subscripting.sql</filename> and <filename>subscripting.c</filename>
+ in the <filename>src/tutorial</filename> directory of the source distribution.
+ See the <filename>README</filename> file in that directory for instructions
+ about running the examples.
+ </para>
+
+</sect1>
diff --git a/src/tutorial/Makefile b/src/tutorial/Makefile
index 16dc390f71..0ead60c2d4 100644
--- a/src/tutorial/Makefile
+++ b/src/tutorial/Makefile
@@ -13,8 +13,8 @@
#
#-------------------------------------------------------------------------
-MODULES = complex funcs
-DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql
+MODULES = complex funcs subscripting
+DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql subscripting.sql
ifdef NO_PGXS
subdir = src/tutorial
diff --git a/src/tutorial/subscripting.c b/src/tutorial/subscripting.c
new file mode 100644
index 0000000000..1eb8c45652
--- /dev/null
+++ b/src/tutorial/subscripting.c
@@ -0,0 +1,201 @@
+/*
+ * src/tutorial/subscripting.c
+ *
+ ******************************************************************************
+ This file contains routines that can be bound to a Postgres backend and
+ called by the backend in the process of processing queries. The calling
+ format for these routines is dictated by Postgres architecture.
+******************************************************************************/
+
+#include "postgres.h"
+
+#include "catalog/pg_type.h"
+#include "executor/executor.h"
+#include "executor/execExpr.h"
+#include "nodes/nodeFuncs.h"
+#include "parser/parse_coerce.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+SubscriptingRef * custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref);
+SubscriptingRef * custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate);
+Datum custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate);
+Datum custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate);
+
+PG_FUNCTION_INFO_V1(custom_in);
+PG_FUNCTION_INFO_V1(custom_out);
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+/*****************************************************************************
+ * Input/Output functions
+ *****************************************************************************/
+
+Datum
+custom_in(PG_FUNCTION_ARGS)
+{
+ char *str = PG_GETARG_CSTRING(0);
+ int firstValue,
+ secondValue;
+ Custom *result;
+
+ if (sscanf(str, " ( %d , %d )", &firstValue, &secondValue) != 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
+ errmsg("invalid input syntax for complex: \"%s\"",
+ str)));
+
+
+ result = (Custom *) palloc(sizeof(Custom));
+ result->first = firstValue;
+ result->second = secondValue;
+ PG_RETURN_POINTER(result);
+}
+
+Datum
+custom_out(PG_FUNCTION_ARGS)
+{
+ Custom *custom = (Custom *) PG_GETARG_POINTER(0);
+ char *result;
+
+ result = psprintf("(%d, %d)", custom->first, custom->second);
+ PG_RETURN_CSTRING(result);
+}
+
+/*****************************************************************************
+ * Custom subscripting logic functions
+ *****************************************************************************/
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = INT4OID;
+ sbsref->refassgntype = INT4OID;
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ List *upperIndexpr = NIL;
+ ListCell *l;
+
+ if (sbsref->reflowerindexpr != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *)lfirst(sbsref->reflowerindexpr->head))))));
+
+ foreach(l, sbsref->refupperindexpr)
+ {
+ Node *subexpr = (Node *) lfirst(l);
+
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *) lfirst(sbsref->refupperindexpr->head))))));
+
+ subexpr = coerce_to_target_type(pstate,
+ subexpr, exprType(subexpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript must have integer type"),
+ parser_errposition(pstate, exprLocation(subexpr))));
+
+ upperIndexpr = lappend(upperIndexpr, subexpr);
+
+ if (isAssignment)
+ {
+ Node *assignExpr = (Node *) sbsref->refassgnexpr;
+ Node *new_from;
+
+ new_from = coerce_to_target_type(pstate,
+ assignExpr, exprType(assignExpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (new_from == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom assignment requires int type"),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation(assignExpr))));
+ sbsref->refassgnexpr = (Expr *)new_from;
+ }
+ }
+
+ sbsref->refupperindexpr = upperIndexpr;
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ Custom *container= (Custom *) containerSource;
+ int index;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ return (Datum) container->first;
+ else
+ return (Datum) container->second;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ int index;
+ Custom *container = (Custom *) containerSource;
+
+ if (sbstate->resnull)
+ return containerSource;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ container->first = DatumGetInt32(sbstate->replacevalue);
+ else
+ container->second = DatumGetInt32(sbstate->replacevalue);
+
+ return (Datum) container;
+}
diff --git a/src/tutorial/subscripting.source b/src/tutorial/subscripting.source
new file mode 100644
index 0000000000..837cf30612
--- /dev/null
+++ b/src/tutorial/subscripting.source
@@ -0,0 +1,71 @@
+---------------------------------------------------------------------------
+--
+-- subscripting.sql-
+-- This file shows how to create a new subscripting procedure for
+-- user-defined type.
+--
+--
+-- Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+-- Portions Copyright (c) 1994, Regents of the University of California
+--
+-- src/tutorial/subscripting.source
+--
+---------------------------------------------------------------------------
+
+-----------------------------
+-- Creating a new type:
+-- We are going to create a new type called 'complex' which represents
+-- complex numbers.
+-- A user-defined type must have an input and an output function, and
+-- optionally can have binary input and output functions. All of these
+-- are usually user-defined C functions.
+-----------------------------
+
+-- Assume the user defined functions are in /home/erthalion/programms/postgresql-master/src/tutorial/complex$DLSUFFIX
+-- (we do not want to assume this is in the dynamic loader search path).
+-- Look at $PWD/complex.c for the source. Note that we declare all of
+-- them as STRICT, so we do not need to cope with NULL inputs in the
+-- C code. We also mark them IMMUTABLE, since they always return the
+-- same outputs given the same inputs.
+
+-- the input function 'complex_in' takes a null-terminated string (the
+-- textual representation of the type) and turns it into the internal
+-- (in memory) representation. You will get a message telling you 'complex'
+-- does not exist yet but that's okay.
+
+CREATE FUNCTION custom_in(cstring)
+ RETURNS custom
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+-- the output function 'complex_out' takes the internal representation and
+-- converts it into the textual representation.
+
+CREATE FUNCTION custom_out(custom)
+ RETURNS cstring
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 8,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler
+);
+
+-- we can use it in a table
+
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
--
2.21.0
--u66dlpg6gaevsofr
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v33-0005-Filling-gaps-in-jsonb-arrays.patch"
^ permalink raw reply [nested|flat] 55+ messages in thread
* [PATCH v33 4/5] Subscripting documentation
@ 2019-02-01 10:47 erthalion <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: erthalion @ 2019-02-01 10:47 UTC (permalink / raw)
Supporting documentation for generalized subscripting. It includes the
description of a new field in pg_type, the new section for jsonb
documentation about subscripting feature on this data type, and also the
tutorial about how to write subscripting operator for a custom data type.
Reviewed-by: Tom Lane, Arthur Zakirov, Oleksandr Shulgin
---
doc/src/sgml/catalogs.sgml | 8 ++
doc/src/sgml/extend.sgml | 6 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/json.sgml | 39 ++++++
doc/src/sgml/ref/create_type.sgml | 33 ++++-
doc/src/sgml/xsubscripting.sgml | 111 +++++++++++++++++
src/tutorial/Makefile | 4 +-
src/tutorial/subscripting.c | 201 ++++++++++++++++++++++++++++++
src/tutorial/subscripting.source | 71 +++++++++++
9 files changed, 470 insertions(+), 4 deletions(-)
create mode 100644 doc/src/sgml/xsubscripting.sgml
create mode 100644 src/tutorial/subscripting.c
create mode 100644 src/tutorial/subscripting.source
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 048ff284f7..b17e9ee382 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8971,6 +8971,14 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>typsubshandler</structfield></entry>
+ <entry><type>regproc</type></entry>
+ <entry><literal><link linkend="catalog-pg-proc"><structname>pg_proc</structname></link>.oid</literal></entry>
+ <entry>Custom subscripting function with type-specific logic for parsing
+ and validation, or 0 if this type doesn't support subscripting.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>typdefault</structfield> <type>text</type>
diff --git a/doc/src/sgml/extend.sgml b/doc/src/sgml/extend.sgml
index 890ff97b7a..65cf5706b6 100644
--- a/doc/src/sgml/extend.sgml
+++ b/doc/src/sgml/extend.sgml
@@ -33,6 +33,11 @@
operators (starting in <xref linkend="xoper"/>)
</para>
</listitem>
+ <listitem>
+ <para>
+ subscripting procedure (starting in <xref linkend="xsubscripting"/>)
+ </para>
+ </listitem>
<listitem>
<para>
operator classes for indexes (starting in <xref linkend="xindex"/>)
@@ -481,6 +486,7 @@ RETURNS anycompatible AS ...
&xaggr;
&xtypes;
&xoper;
+ &xsubscripting;
&xindex;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 64b5da0070..4c3fd37fc1 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -69,6 +69,7 @@
<!ENTITY xplang SYSTEM "xplang.sgml">
<!ENTITY xoper SYSTEM "xoper.sgml">
<!ENTITY xtypes SYSTEM "xtypes.sgml">
+<!ENTITY xsubscripting SYSTEM "xsubscripting.sgml">
<!ENTITY plperl SYSTEM "plperl.sgml">
<!ENTITY plpython SYSTEM "plpython.sgml">
<!ENTITY plsql SYSTEM "plpgsql.sgml">
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index c0a6554d4d..5c538dca05 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -602,6 +602,45 @@ SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qu
</para>
</sect2>
+ <sect2 id="jsonb-subscripting">
+ <title><type>jsonb</type> Subscripting</title>
+ <para>
+ <type>jsonb</type> data type supports array-style subscripting expressions
+ to extract or update particular elements. It's possible to use multiple
+ subscripting expressions to extract nested values. In this case, a chain of
+ subscripting expressions follows the same rules as the
+ <literal>path</literal> argument in <literal>jsonb_set</literal> function,
+ e.g. in case of arrays it is a 0-based operation or that negative integers
+ that appear in <literal>path</literal> count from the end of JSON arrays.
+ The result of subscripting expressions is always jsonb data type. An
+ example of subscripting syntax:
+<programlisting>
+-- Extract value by key
+SELECT ('{"a": 1}'::jsonb)['a'];
+
+-- Extract nested value by key path
+SELECT ('{"a": {"b": {"c": 1}}}'::jsonb)['a']['b']['c'];
+
+-- Extract element by index
+SELECT ('[1, "2", null]'::jsonb)[1];
+
+-- Update value by key
+UPDATE table_name SET jsonb_field['key'] = 1;
+
+-- Select records using where clause with subscripting. Since the result of
+-- subscripting is jsonb and we basically want to compare two jsonb objects, we
+-- need to put the value in double quotes to be able to convert it to jsonb.
+SELECT * FROM table_name WHERE jsonb_field['key'] = '"value"';
+</programlisting>
+
+ There is no special indexing support for such kind of expressions, but you
+ can always create a functional index that includes it
+<programlisting>
+CREATE INDEX idx ON table_name ((jsonb_field['key']));
+</programlisting>
+ </para>
+ </sect2>
+
<sect2>
<title>Transforms</title>
diff --git a/doc/src/sgml/ref/create_type.sgml b/doc/src/sgml/ref/create_type.sgml
index 111f8e65d2..a34df4d247 100644
--- a/doc/src/sgml/ref/create_type.sgml
+++ b/doc/src/sgml/ref/create_type.sgml
@@ -54,6 +54,7 @@ CREATE TYPE <replaceable class="parameter">name</replaceable> (
[ , ELEMENT = <replaceable class="parameter">element</replaceable> ]
[ , DELIMITER = <replaceable class="parameter">delimiter</replaceable> ]
[ , COLLATABLE = <replaceable class="parameter">collatable</replaceable> ]
+ [ , SUBSCRIPTING_HANDLER = <replaceable class="parameter">subscripting_handler_function</replaceable> ]
)
CREATE TYPE <replaceable class="parameter">name</replaceable>
@@ -196,8 +197,9 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
<replaceable class="parameter">receive_function</replaceable>,
<replaceable class="parameter">send_function</replaceable>,
<replaceable class="parameter">type_modifier_input_function</replaceable>,
- <replaceable class="parameter">type_modifier_output_function</replaceable> and
- <replaceable class="parameter">analyze_function</replaceable>
+ <replaceable class="parameter">type_modifier_output_function</replaceable>,
+ <replaceable class="parameter">analyze_function</replaceable>,
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
are optional. Generally these functions have to be coded in C
or another low-level language.
</para>
@@ -454,6 +456,22 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
make use of the collation information; this does not happen
automatically merely by marking the type collatable.
</para>
+
+ <para>
+ The optional
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
+ contains type-specific logic for subscripting of the data type.
+ By default, there is no such function provided, which means that the data
+ type doesn't support subscripting. The subscripting function must be
+ declared to take a single argument of type <type>internal</type>, and return
+ a <type>internal</type> result. There are two examples of implementation for
+ subscripting functions in case of array
+ (<replaceable class="parameter">array_subscripting_handler</replaceable>)
+ and jsonb
+ (<replaceable class="parameter">jsonb_subscripting_handler</replaceable>)
+ types in <filename>src/backend/utils/adt/arrayfuncs.c</filename> and
+ <filename>src/backend/utils/adt/jsonfuncs.c</filename>, respectively.
+ </para>
</refsect2>
<refsect2>
@@ -769,6 +787,17 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">subscripting_handler_function</replaceable></term>
+ <listitem>
+ <para>
+ The name of a function that returns list of type-specific callback functions to
+ support subscripting logic for the data type.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
diff --git a/doc/src/sgml/xsubscripting.sgml b/doc/src/sgml/xsubscripting.sgml
new file mode 100644
index 0000000000..7224e81fa2
--- /dev/null
+++ b/doc/src/sgml/xsubscripting.sgml
@@ -0,0 +1,111 @@
+<!-- doc/src/sgml/xsubscripting.sgml -->
+
+ <sect1 id="xsubscripting">
+ <title>User-defined subscripting procedure</title>
+
+ <indexterm zone="xsubscripting">
+ <primary>custom subscripting</primary>
+ </indexterm>
+ <para>
+ When you define a new base type, you can also specify a custom procedure to
+ handle subscripting expressions. It must contain logic for verification and
+ evaluation of this expression, i.e. fetching or updating some data in this
+ data type. For instance:
+</para>
+<programlisting><![CDATA[
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = someType;
+ sbsref->refassgntype = someType;
+
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ // some validation and coercion logic
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some assignment logic
+
+ return newContainer;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some fetch logic based on sbstate
+}]]>
+</programlisting>
+
+<para>
+ Then you can define a subscripting procedure and a custom data type:
+</para>
+<programlisting>
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '<replaceable>filename</replaceable>'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 4,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler,
+);
+</programlisting>
+
+<para>
+ and use it as usual:
+</para>
+<programlisting>
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
+</programlisting>
+
+
+ <para>
+ The examples of custom subscripting implementation can be found in
+ <filename>subscripting.sql</filename> and <filename>subscripting.c</filename>
+ in the <filename>src/tutorial</filename> directory of the source distribution.
+ See the <filename>README</filename> file in that directory for instructions
+ about running the examples.
+ </para>
+
+</sect1>
diff --git a/src/tutorial/Makefile b/src/tutorial/Makefile
index 16dc390f71..0ead60c2d4 100644
--- a/src/tutorial/Makefile
+++ b/src/tutorial/Makefile
@@ -13,8 +13,8 @@
#
#-------------------------------------------------------------------------
-MODULES = complex funcs
-DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql
+MODULES = complex funcs subscripting
+DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql subscripting.sql
ifdef NO_PGXS
subdir = src/tutorial
diff --git a/src/tutorial/subscripting.c b/src/tutorial/subscripting.c
new file mode 100644
index 0000000000..1eb8c45652
--- /dev/null
+++ b/src/tutorial/subscripting.c
@@ -0,0 +1,201 @@
+/*
+ * src/tutorial/subscripting.c
+ *
+ ******************************************************************************
+ This file contains routines that can be bound to a Postgres backend and
+ called by the backend in the process of processing queries. The calling
+ format for these routines is dictated by Postgres architecture.
+******************************************************************************/
+
+#include "postgres.h"
+
+#include "catalog/pg_type.h"
+#include "executor/executor.h"
+#include "executor/execExpr.h"
+#include "nodes/nodeFuncs.h"
+#include "parser/parse_coerce.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+SubscriptingRef * custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref);
+SubscriptingRef * custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate);
+Datum custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate);
+Datum custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate);
+
+PG_FUNCTION_INFO_V1(custom_in);
+PG_FUNCTION_INFO_V1(custom_out);
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+/*****************************************************************************
+ * Input/Output functions
+ *****************************************************************************/
+
+Datum
+custom_in(PG_FUNCTION_ARGS)
+{
+ char *str = PG_GETARG_CSTRING(0);
+ int firstValue,
+ secondValue;
+ Custom *result;
+
+ if (sscanf(str, " ( %d , %d )", &firstValue, &secondValue) != 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
+ errmsg("invalid input syntax for complex: \"%s\"",
+ str)));
+
+
+ result = (Custom *) palloc(sizeof(Custom));
+ result->first = firstValue;
+ result->second = secondValue;
+ PG_RETURN_POINTER(result);
+}
+
+Datum
+custom_out(PG_FUNCTION_ARGS)
+{
+ Custom *custom = (Custom *) PG_GETARG_POINTER(0);
+ char *result;
+
+ result = psprintf("(%d, %d)", custom->first, custom->second);
+ PG_RETURN_CSTRING(result);
+}
+
+/*****************************************************************************
+ * Custom subscripting logic functions
+ *****************************************************************************/
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = INT4OID;
+ sbsref->refassgntype = INT4OID;
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ List *upperIndexpr = NIL;
+ ListCell *l;
+
+ if (sbsref->reflowerindexpr != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *)lfirst(sbsref->reflowerindexpr->head))))));
+
+ foreach(l, sbsref->refupperindexpr)
+ {
+ Node *subexpr = (Node *) lfirst(l);
+
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *) lfirst(sbsref->refupperindexpr->head))))));
+
+ subexpr = coerce_to_target_type(pstate,
+ subexpr, exprType(subexpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript must have integer type"),
+ parser_errposition(pstate, exprLocation(subexpr))));
+
+ upperIndexpr = lappend(upperIndexpr, subexpr);
+
+ if (isAssignment)
+ {
+ Node *assignExpr = (Node *) sbsref->refassgnexpr;
+ Node *new_from;
+
+ new_from = coerce_to_target_type(pstate,
+ assignExpr, exprType(assignExpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (new_from == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom assignment requires int type"),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation(assignExpr))));
+ sbsref->refassgnexpr = (Expr *)new_from;
+ }
+ }
+
+ sbsref->refupperindexpr = upperIndexpr;
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ Custom *container= (Custom *) containerSource;
+ int index;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ return (Datum) container->first;
+ else
+ return (Datum) container->second;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ int index;
+ Custom *container = (Custom *) containerSource;
+
+ if (sbstate->resnull)
+ return containerSource;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ container->first = DatumGetInt32(sbstate->replacevalue);
+ else
+ container->second = DatumGetInt32(sbstate->replacevalue);
+
+ return (Datum) container;
+}
diff --git a/src/tutorial/subscripting.source b/src/tutorial/subscripting.source
new file mode 100644
index 0000000000..837cf30612
--- /dev/null
+++ b/src/tutorial/subscripting.source
@@ -0,0 +1,71 @@
+---------------------------------------------------------------------------
+--
+-- subscripting.sql-
+-- This file shows how to create a new subscripting procedure for
+-- user-defined type.
+--
+--
+-- Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+-- Portions Copyright (c) 1994, Regents of the University of California
+--
+-- src/tutorial/subscripting.source
+--
+---------------------------------------------------------------------------
+
+-----------------------------
+-- Creating a new type:
+-- We are going to create a new type called 'complex' which represents
+-- complex numbers.
+-- A user-defined type must have an input and an output function, and
+-- optionally can have binary input and output functions. All of these
+-- are usually user-defined C functions.
+-----------------------------
+
+-- Assume the user defined functions are in /home/erthalion/programms/postgresql-master/src/tutorial/complex$DLSUFFIX
+-- (we do not want to assume this is in the dynamic loader search path).
+-- Look at $PWD/complex.c for the source. Note that we declare all of
+-- them as STRICT, so we do not need to cope with NULL inputs in the
+-- C code. We also mark them IMMUTABLE, since they always return the
+-- same outputs given the same inputs.
+
+-- the input function 'complex_in' takes a null-terminated string (the
+-- textual representation of the type) and turns it into the internal
+-- (in memory) representation. You will get a message telling you 'complex'
+-- does not exist yet but that's okay.
+
+CREATE FUNCTION custom_in(cstring)
+ RETURNS custom
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+-- the output function 'complex_out' takes the internal representation and
+-- converts it into the textual representation.
+
+CREATE FUNCTION custom_out(custom)
+ RETURNS cstring
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 8,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler
+);
+
+-- we can use it in a table
+
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
--
2.21.0
--u66dlpg6gaevsofr
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v33-0005-Filling-gaps-in-jsonb-arrays.patch"
^ permalink raw reply [nested|flat] 55+ messages in thread
* [PATCH v29 4/6] Subscripting documentation
@ 2019-02-01 10:47 erthalion <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: erthalion @ 2019-02-01 10:47 UTC (permalink / raw)
Supporting documentation for generalized subscripting. It includes the
description of a new field in pg_type, the new section for jsonb
documentation about subscripting feature on this data type, and also the
tutorial about how to write subscripting operator for a custom data type.
Reviewed-by: Tom Lane, Arthur Zakirov, Oleksandr Shulgin
---
doc/src/sgml/catalogs.sgml | 8 ++
doc/src/sgml/extend.sgml | 6 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/json.sgml | 39 ++++++
doc/src/sgml/ref/create_type.sgml | 33 ++++-
doc/src/sgml/xsubscripting.sgml | 111 +++++++++++++++++
src/tutorial/Makefile | 4 +-
src/tutorial/subscripting.c | 201 ++++++++++++++++++++++++++++++
src/tutorial/subscripting.source | 71 +++++++++++
9 files changed, 470 insertions(+), 4 deletions(-)
create mode 100644 doc/src/sgml/xsubscripting.sgml
create mode 100644 src/tutorial/subscripting.c
create mode 100644 src/tutorial/subscripting.source
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 55694c4368..b8fde6af12 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8012,6 +8012,14 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>typsubshandler</structfield></entry>
+ <entry><type>regproc</type></entry>
+ <entry><literal><link linkend="catalog-pg-proc"><structname>pg_proc</structname></link>.oid</literal></entry>
+ <entry>Custom subscripting function with type-specific logic for parsing
+ and validation, or 0 if this type doesn't support subscripting.</entry>
+ </row>
+
<row>
<entry><structfield>typdefaultbin</structfield></entry>
<entry><type>pg_node_tree</type></entry>
diff --git a/doc/src/sgml/extend.sgml b/doc/src/sgml/extend.sgml
index a3046f22d0..9a4ce542ba 100644
--- a/doc/src/sgml/extend.sgml
+++ b/doc/src/sgml/extend.sgml
@@ -33,6 +33,11 @@
operators (starting in <xref linkend="xoper"/>)
</para>
</listitem>
+ <listitem>
+ <para>
+ subscripting procedure (starting in <xref linkend="xsubscripting"/>)
+ </para>
+ </listitem>
<listitem>
<para>
operator classes for indexes (starting in <xref linkend="xindex"/>)
@@ -314,6 +319,7 @@
&xaggr;
&xtypes;
&xoper;
+ &xsubscripting;
&xindex;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 3da2365ea9..650e21b7e1 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -69,6 +69,7 @@
<!ENTITY xplang SYSTEM "xplang.sgml">
<!ENTITY xoper SYSTEM "xoper.sgml">
<!ENTITY xtypes SYSTEM "xtypes.sgml">
+<!ENTITY xsubscripting SYSTEM "xsubscripting.sgml">
<!ENTITY plperl SYSTEM "plperl.sgml">
<!ENTITY plpython SYSTEM "plpython.sgml">
<!ENTITY plsql SYSTEM "plpgsql.sgml">
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index 6ff8751870..d6f2f1d55f 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -600,6 +600,45 @@ SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qu
</para>
</sect2>
+ <sect2 id="jsonb-subscripting">
+ <title><type>jsonb</type> Subscripting</title>
+ <para>
+ <type>jsonb</type> data type supports array-style subscripting expressions
+ to extract or update particular element. It's possible to use multiple
+ subscripting expressions to extract nested values. In this case a chain of
+ subscripting expressions follows the same rules as the
+ <literal>path</literal> argument in <literal>jsonb_set</literal> function,
+ e.g. in case of arrays it is a 0-based operation or that negative integers
+ that appear in <literal>path</literal> count from the end of JSON arrays.
+ The result of subscripting expressions is always jsonb data type. An
+ example of subscripting syntax:
+<programlisting>
+-- Extract value by key
+SELECT ('{"a": 1}'::jsonb)['a'];
+
+-- Extract nested value by key path
+SELECT ('{"a": {"b": {"c": 1}}}'::jsonb)['a']['b']['c'];
+
+-- Extract element by index
+SELECT ('[1, "2", null]'::jsonb)[1];
+
+-- Update value by key
+UPDATE table_name SET jsonb_field['key'] = 1;
+
+-- Select records using where clause with subscripting. Since the result of
+-- subscripting is jsonb and we basically want to compare two jsonb objects, we
+-- need to put the value in double quotes to be able to convert it to jsonb.
+SELECT * FROM table_name WHERE jsonb_field['key'] = '"value"';
+</programlisting>
+
+ There is no special indexing support for such kind of expressions, but you
+ always can create a functional index that includes it
+<programlisting>
+CREATE INDEX idx ON table_name ((jsonb_field['key']));
+</programlisting>
+ </para>
+ </sect2>
+
<sect2>
<title>Transforms</title>
diff --git a/doc/src/sgml/ref/create_type.sgml b/doc/src/sgml/ref/create_type.sgml
index 175315f3d7..bd622f5ef3 100644
--- a/doc/src/sgml/ref/create_type.sgml
+++ b/doc/src/sgml/ref/create_type.sgml
@@ -54,6 +54,7 @@ CREATE TYPE <replaceable class="parameter">name</replaceable> (
[ , ELEMENT = <replaceable class="parameter">element</replaceable> ]
[ , DELIMITER = <replaceable class="parameter">delimiter</replaceable> ]
[ , COLLATABLE = <replaceable class="parameter">collatable</replaceable> ]
+ [ , SUBSCRIPTING_HANDLER = <replaceable class="parameter">subscripting_handler_function</replaceable> ]
)
CREATE TYPE <replaceable class="parameter">name</replaceable>
@@ -196,8 +197,9 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
<replaceable class="parameter">receive_function</replaceable>,
<replaceable class="parameter">send_function</replaceable>,
<replaceable class="parameter">type_modifier_input_function</replaceable>,
- <replaceable class="parameter">type_modifier_output_function</replaceable> and
- <replaceable class="parameter">analyze_function</replaceable>
+ <replaceable class="parameter">type_modifier_output_function</replaceable>,
+ <replaceable class="parameter">analyze_function</replaceable>,
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
are optional. Generally these functions have to be coded in C
or another low-level language.
</para>
@@ -454,6 +456,22 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
make use of the collation information; this does not happen
automatically merely by marking the type collatable.
</para>
+
+ <para>
+ The optional
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
+ contains type-specific logic for subscripting of the data type.
+ By default, there is no such function provided, which means that the data
+ type doesn't support subscripting. The subscripting function must be
+ declared to take a single argument of type <type>internal</type>, and return
+ a <type>internal</type> result. There are two examples of implementation for
+ subscripting functions in case of array
+ (<replaceable class="parameter">array_subscripting_handler</replaceable>)
+ and jsonb
+ (<replaceable class="parameter">jsonb_subscripting_handler</replaceable>)
+ types in <filename>src/backend/utils/adt/arrayfuncs.c</filename> and
+ <filename>src/backend/utils/adt/jsonfuncs.c</filename> corresponding.
+ </para>
</refsect2>
<refsect2>
@@ -769,6 +787,17 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">subscripting_handler_function</replaceable></term>
+ <listitem>
+ <para>
+ The name of a function that returns list of type-specific callback functions to
+ support subscripting logic for the data type.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
diff --git a/doc/src/sgml/xsubscripting.sgml b/doc/src/sgml/xsubscripting.sgml
new file mode 100644
index 0000000000..d701631223
--- /dev/null
+++ b/doc/src/sgml/xsubscripting.sgml
@@ -0,0 +1,111 @@
+<!-- doc/src/sgml/xsubscripting.sgml -->
+
+ <sect1 id="xsubscripting">
+ <title>User-defined subscripting procedure</title>
+
+ <indexterm zone="xsubscripting">
+ <primary>custom subscripting</primary>
+ </indexterm>
+ <para>
+ When you define a new base type, you can also specify a custom procedures to
+ handle subscripting expressions. They must contain logic for verification and
+ evaluation of this expression, i.e. fetching or updating some data in this
+ data type. For instance:
+</para>
+<programlisting><![CDATA[
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = someType;
+ sbsref->refassgntype = someType;
+
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ // some validation and coercion logic
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some assignment logic
+
+ return newContainer;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some fetch logic based on sbsdata
+}]]>
+</programlisting>
+
+<para>
+ Then you can define a subscripting procedures and a custom data type:
+</para>
+<programlisting>
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '<replaceable>filename</replaceable>'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 4,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler,
+);
+</programlisting>
+
+<para>
+ and use it as usual:
+</para>
+<programlisting>
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
+</programlisting>
+
+
+ <para>
+ The examples of custom subscripting implementation can be found in
+ <filename>subscripting.sql</filename> and <filename>subscripting.c</filename>
+ in the <filename>src/tutorial</filename> directory of the source distribution.
+ See the <filename>README</filename> file in that directory for instructions
+ about running the examples.
+ </para>
+
+</sect1>
diff --git a/src/tutorial/Makefile b/src/tutorial/Makefile
index 16dc390f71..0ead60c2d4 100644
--- a/src/tutorial/Makefile
+++ b/src/tutorial/Makefile
@@ -13,8 +13,8 @@
#
#-------------------------------------------------------------------------
-MODULES = complex funcs
-DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql
+MODULES = complex funcs subscripting
+DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql subscripting.sql
ifdef NO_PGXS
subdir = src/tutorial
diff --git a/src/tutorial/subscripting.c b/src/tutorial/subscripting.c
new file mode 100644
index 0000000000..1eb8c45652
--- /dev/null
+++ b/src/tutorial/subscripting.c
@@ -0,0 +1,201 @@
+/*
+ * src/tutorial/subscripting.c
+ *
+ ******************************************************************************
+ This file contains routines that can be bound to a Postgres backend and
+ called by the backend in the process of processing queries. The calling
+ format for these routines is dictated by Postgres architecture.
+******************************************************************************/
+
+#include "postgres.h"
+
+#include "catalog/pg_type.h"
+#include "executor/executor.h"
+#include "executor/execExpr.h"
+#include "nodes/nodeFuncs.h"
+#include "parser/parse_coerce.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+SubscriptingRef * custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref);
+SubscriptingRef * custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate);
+Datum custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate);
+Datum custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate);
+
+PG_FUNCTION_INFO_V1(custom_in);
+PG_FUNCTION_INFO_V1(custom_out);
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+/*****************************************************************************
+ * Input/Output functions
+ *****************************************************************************/
+
+Datum
+custom_in(PG_FUNCTION_ARGS)
+{
+ char *str = PG_GETARG_CSTRING(0);
+ int firstValue,
+ secondValue;
+ Custom *result;
+
+ if (sscanf(str, " ( %d , %d )", &firstValue, &secondValue) != 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
+ errmsg("invalid input syntax for complex: \"%s\"",
+ str)));
+
+
+ result = (Custom *) palloc(sizeof(Custom));
+ result->first = firstValue;
+ result->second = secondValue;
+ PG_RETURN_POINTER(result);
+}
+
+Datum
+custom_out(PG_FUNCTION_ARGS)
+{
+ Custom *custom = (Custom *) PG_GETARG_POINTER(0);
+ char *result;
+
+ result = psprintf("(%d, %d)", custom->first, custom->second);
+ PG_RETURN_CSTRING(result);
+}
+
+/*****************************************************************************
+ * Custom subscripting logic functions
+ *****************************************************************************/
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = INT4OID;
+ sbsref->refassgntype = INT4OID;
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ List *upperIndexpr = NIL;
+ ListCell *l;
+
+ if (sbsref->reflowerindexpr != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *)lfirst(sbsref->reflowerindexpr->head))))));
+
+ foreach(l, sbsref->refupperindexpr)
+ {
+ Node *subexpr = (Node *) lfirst(l);
+
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *) lfirst(sbsref->refupperindexpr->head))))));
+
+ subexpr = coerce_to_target_type(pstate,
+ subexpr, exprType(subexpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript must have integer type"),
+ parser_errposition(pstate, exprLocation(subexpr))));
+
+ upperIndexpr = lappend(upperIndexpr, subexpr);
+
+ if (isAssignment)
+ {
+ Node *assignExpr = (Node *) sbsref->refassgnexpr;
+ Node *new_from;
+
+ new_from = coerce_to_target_type(pstate,
+ assignExpr, exprType(assignExpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (new_from == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom assignment requires int type"),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation(assignExpr))));
+ sbsref->refassgnexpr = (Expr *)new_from;
+ }
+ }
+
+ sbsref->refupperindexpr = upperIndexpr;
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ Custom *container= (Custom *) containerSource;
+ int index;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ return (Datum) container->first;
+ else
+ return (Datum) container->second;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ int index;
+ Custom *container = (Custom *) containerSource;
+
+ if (sbstate->resnull)
+ return containerSource;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ container->first = DatumGetInt32(sbstate->replacevalue);
+ else
+ container->second = DatumGetInt32(sbstate->replacevalue);
+
+ return (Datum) container;
+}
diff --git a/src/tutorial/subscripting.source b/src/tutorial/subscripting.source
new file mode 100644
index 0000000000..837cf30612
--- /dev/null
+++ b/src/tutorial/subscripting.source
@@ -0,0 +1,71 @@
+---------------------------------------------------------------------------
+--
+-- subscripting.sql-
+-- This file shows how to create a new subscripting procedure for
+-- user-defined type.
+--
+--
+-- Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+-- Portions Copyright (c) 1994, Regents of the University of California
+--
+-- src/tutorial/subscripting.source
+--
+---------------------------------------------------------------------------
+
+-----------------------------
+-- Creating a new type:
+-- We are going to create a new type called 'complex' which represents
+-- complex numbers.
+-- A user-defined type must have an input and an output function, and
+-- optionally can have binary input and output functions. All of these
+-- are usually user-defined C functions.
+-----------------------------
+
+-- Assume the user defined functions are in /home/erthalion/programms/postgresql-master/src/tutorial/complex$DLSUFFIX
+-- (we do not want to assume this is in the dynamic loader search path).
+-- Look at $PWD/complex.c for the source. Note that we declare all of
+-- them as STRICT, so we do not need to cope with NULL inputs in the
+-- C code. We also mark them IMMUTABLE, since they always return the
+-- same outputs given the same inputs.
+
+-- the input function 'complex_in' takes a null-terminated string (the
+-- textual representation of the type) and turns it into the internal
+-- (in memory) representation. You will get a message telling you 'complex'
+-- does not exist yet but that's okay.
+
+CREATE FUNCTION custom_in(cstring)
+ RETURNS custom
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+-- the output function 'complex_out' takes the internal representation and
+-- converts it into the textual representation.
+
+CREATE FUNCTION custom_out(custom)
+ RETURNS cstring
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 8,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler
+);
+
+-- we can use it in a table
+
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
--
2.21.0
--adquyx6kg5n26q6j
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v29-0005-Polymorphic-subscripting.patch"
^ permalink raw reply [nested|flat] 55+ messages in thread
* [PATCH v33 4/5] Subscripting documentation
@ 2019-02-01 10:47 erthalion <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: erthalion @ 2019-02-01 10:47 UTC (permalink / raw)
Supporting documentation for generalized subscripting. It includes the
description of a new field in pg_type, the new section for jsonb
documentation about subscripting feature on this data type, and also the
tutorial about how to write subscripting operator for a custom data type.
Reviewed-by: Tom Lane, Arthur Zakirov, Oleksandr Shulgin
---
doc/src/sgml/catalogs.sgml | 8 ++
doc/src/sgml/extend.sgml | 6 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/json.sgml | 39 ++++++
doc/src/sgml/ref/create_type.sgml | 33 ++++-
doc/src/sgml/xsubscripting.sgml | 111 +++++++++++++++++
src/tutorial/Makefile | 4 +-
src/tutorial/subscripting.c | 201 ++++++++++++++++++++++++++++++
src/tutorial/subscripting.source | 71 +++++++++++
9 files changed, 470 insertions(+), 4 deletions(-)
create mode 100644 doc/src/sgml/xsubscripting.sgml
create mode 100644 src/tutorial/subscripting.c
create mode 100644 src/tutorial/subscripting.source
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 048ff284f7..b17e9ee382 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8971,6 +8971,14 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>typsubshandler</structfield></entry>
+ <entry><type>regproc</type></entry>
+ <entry><literal><link linkend="catalog-pg-proc"><structname>pg_proc</structname></link>.oid</literal></entry>
+ <entry>Custom subscripting function with type-specific logic for parsing
+ and validation, or 0 if this type doesn't support subscripting.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>typdefault</structfield> <type>text</type>
diff --git a/doc/src/sgml/extend.sgml b/doc/src/sgml/extend.sgml
index 890ff97b7a..65cf5706b6 100644
--- a/doc/src/sgml/extend.sgml
+++ b/doc/src/sgml/extend.sgml
@@ -33,6 +33,11 @@
operators (starting in <xref linkend="xoper"/>)
</para>
</listitem>
+ <listitem>
+ <para>
+ subscripting procedure (starting in <xref linkend="xsubscripting"/>)
+ </para>
+ </listitem>
<listitem>
<para>
operator classes for indexes (starting in <xref linkend="xindex"/>)
@@ -481,6 +486,7 @@ RETURNS anycompatible AS ...
&xaggr;
&xtypes;
&xoper;
+ &xsubscripting;
&xindex;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 64b5da0070..4c3fd37fc1 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -69,6 +69,7 @@
<!ENTITY xplang SYSTEM "xplang.sgml">
<!ENTITY xoper SYSTEM "xoper.sgml">
<!ENTITY xtypes SYSTEM "xtypes.sgml">
+<!ENTITY xsubscripting SYSTEM "xsubscripting.sgml">
<!ENTITY plperl SYSTEM "plperl.sgml">
<!ENTITY plpython SYSTEM "plpython.sgml">
<!ENTITY plsql SYSTEM "plpgsql.sgml">
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index c0a6554d4d..5c538dca05 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -602,6 +602,45 @@ SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qu
</para>
</sect2>
+ <sect2 id="jsonb-subscripting">
+ <title><type>jsonb</type> Subscripting</title>
+ <para>
+ <type>jsonb</type> data type supports array-style subscripting expressions
+ to extract or update particular elements. It's possible to use multiple
+ subscripting expressions to extract nested values. In this case, a chain of
+ subscripting expressions follows the same rules as the
+ <literal>path</literal> argument in <literal>jsonb_set</literal> function,
+ e.g. in case of arrays it is a 0-based operation or that negative integers
+ that appear in <literal>path</literal> count from the end of JSON arrays.
+ The result of subscripting expressions is always jsonb data type. An
+ example of subscripting syntax:
+<programlisting>
+-- Extract value by key
+SELECT ('{"a": 1}'::jsonb)['a'];
+
+-- Extract nested value by key path
+SELECT ('{"a": {"b": {"c": 1}}}'::jsonb)['a']['b']['c'];
+
+-- Extract element by index
+SELECT ('[1, "2", null]'::jsonb)[1];
+
+-- Update value by key
+UPDATE table_name SET jsonb_field['key'] = 1;
+
+-- Select records using where clause with subscripting. Since the result of
+-- subscripting is jsonb and we basically want to compare two jsonb objects, we
+-- need to put the value in double quotes to be able to convert it to jsonb.
+SELECT * FROM table_name WHERE jsonb_field['key'] = '"value"';
+</programlisting>
+
+ There is no special indexing support for such kind of expressions, but you
+ can always create a functional index that includes it
+<programlisting>
+CREATE INDEX idx ON table_name ((jsonb_field['key']));
+</programlisting>
+ </para>
+ </sect2>
+
<sect2>
<title>Transforms</title>
diff --git a/doc/src/sgml/ref/create_type.sgml b/doc/src/sgml/ref/create_type.sgml
index 111f8e65d2..a34df4d247 100644
--- a/doc/src/sgml/ref/create_type.sgml
+++ b/doc/src/sgml/ref/create_type.sgml
@@ -54,6 +54,7 @@ CREATE TYPE <replaceable class="parameter">name</replaceable> (
[ , ELEMENT = <replaceable class="parameter">element</replaceable> ]
[ , DELIMITER = <replaceable class="parameter">delimiter</replaceable> ]
[ , COLLATABLE = <replaceable class="parameter">collatable</replaceable> ]
+ [ , SUBSCRIPTING_HANDLER = <replaceable class="parameter">subscripting_handler_function</replaceable> ]
)
CREATE TYPE <replaceable class="parameter">name</replaceable>
@@ -196,8 +197,9 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
<replaceable class="parameter">receive_function</replaceable>,
<replaceable class="parameter">send_function</replaceable>,
<replaceable class="parameter">type_modifier_input_function</replaceable>,
- <replaceable class="parameter">type_modifier_output_function</replaceable> and
- <replaceable class="parameter">analyze_function</replaceable>
+ <replaceable class="parameter">type_modifier_output_function</replaceable>,
+ <replaceable class="parameter">analyze_function</replaceable>,
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
are optional. Generally these functions have to be coded in C
or another low-level language.
</para>
@@ -454,6 +456,22 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
make use of the collation information; this does not happen
automatically merely by marking the type collatable.
</para>
+
+ <para>
+ The optional
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
+ contains type-specific logic for subscripting of the data type.
+ By default, there is no such function provided, which means that the data
+ type doesn't support subscripting. The subscripting function must be
+ declared to take a single argument of type <type>internal</type>, and return
+ a <type>internal</type> result. There are two examples of implementation for
+ subscripting functions in case of array
+ (<replaceable class="parameter">array_subscripting_handler</replaceable>)
+ and jsonb
+ (<replaceable class="parameter">jsonb_subscripting_handler</replaceable>)
+ types in <filename>src/backend/utils/adt/arrayfuncs.c</filename> and
+ <filename>src/backend/utils/adt/jsonfuncs.c</filename>, respectively.
+ </para>
</refsect2>
<refsect2>
@@ -769,6 +787,17 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">subscripting_handler_function</replaceable></term>
+ <listitem>
+ <para>
+ The name of a function that returns list of type-specific callback functions to
+ support subscripting logic for the data type.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
diff --git a/doc/src/sgml/xsubscripting.sgml b/doc/src/sgml/xsubscripting.sgml
new file mode 100644
index 0000000000..7224e81fa2
--- /dev/null
+++ b/doc/src/sgml/xsubscripting.sgml
@@ -0,0 +1,111 @@
+<!-- doc/src/sgml/xsubscripting.sgml -->
+
+ <sect1 id="xsubscripting">
+ <title>User-defined subscripting procedure</title>
+
+ <indexterm zone="xsubscripting">
+ <primary>custom subscripting</primary>
+ </indexterm>
+ <para>
+ When you define a new base type, you can also specify a custom procedure to
+ handle subscripting expressions. It must contain logic for verification and
+ evaluation of this expression, i.e. fetching or updating some data in this
+ data type. For instance:
+</para>
+<programlisting><![CDATA[
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = someType;
+ sbsref->refassgntype = someType;
+
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ // some validation and coercion logic
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some assignment logic
+
+ return newContainer;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some fetch logic based on sbstate
+}]]>
+</programlisting>
+
+<para>
+ Then you can define a subscripting procedure and a custom data type:
+</para>
+<programlisting>
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '<replaceable>filename</replaceable>'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 4,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler,
+);
+</programlisting>
+
+<para>
+ and use it as usual:
+</para>
+<programlisting>
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
+</programlisting>
+
+
+ <para>
+ The examples of custom subscripting implementation can be found in
+ <filename>subscripting.sql</filename> and <filename>subscripting.c</filename>
+ in the <filename>src/tutorial</filename> directory of the source distribution.
+ See the <filename>README</filename> file in that directory for instructions
+ about running the examples.
+ </para>
+
+</sect1>
diff --git a/src/tutorial/Makefile b/src/tutorial/Makefile
index 16dc390f71..0ead60c2d4 100644
--- a/src/tutorial/Makefile
+++ b/src/tutorial/Makefile
@@ -13,8 +13,8 @@
#
#-------------------------------------------------------------------------
-MODULES = complex funcs
-DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql
+MODULES = complex funcs subscripting
+DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql subscripting.sql
ifdef NO_PGXS
subdir = src/tutorial
diff --git a/src/tutorial/subscripting.c b/src/tutorial/subscripting.c
new file mode 100644
index 0000000000..1eb8c45652
--- /dev/null
+++ b/src/tutorial/subscripting.c
@@ -0,0 +1,201 @@
+/*
+ * src/tutorial/subscripting.c
+ *
+ ******************************************************************************
+ This file contains routines that can be bound to a Postgres backend and
+ called by the backend in the process of processing queries. The calling
+ format for these routines is dictated by Postgres architecture.
+******************************************************************************/
+
+#include "postgres.h"
+
+#include "catalog/pg_type.h"
+#include "executor/executor.h"
+#include "executor/execExpr.h"
+#include "nodes/nodeFuncs.h"
+#include "parser/parse_coerce.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+SubscriptingRef * custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref);
+SubscriptingRef * custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate);
+Datum custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate);
+Datum custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate);
+
+PG_FUNCTION_INFO_V1(custom_in);
+PG_FUNCTION_INFO_V1(custom_out);
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+/*****************************************************************************
+ * Input/Output functions
+ *****************************************************************************/
+
+Datum
+custom_in(PG_FUNCTION_ARGS)
+{
+ char *str = PG_GETARG_CSTRING(0);
+ int firstValue,
+ secondValue;
+ Custom *result;
+
+ if (sscanf(str, " ( %d , %d )", &firstValue, &secondValue) != 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
+ errmsg("invalid input syntax for complex: \"%s\"",
+ str)));
+
+
+ result = (Custom *) palloc(sizeof(Custom));
+ result->first = firstValue;
+ result->second = secondValue;
+ PG_RETURN_POINTER(result);
+}
+
+Datum
+custom_out(PG_FUNCTION_ARGS)
+{
+ Custom *custom = (Custom *) PG_GETARG_POINTER(0);
+ char *result;
+
+ result = psprintf("(%d, %d)", custom->first, custom->second);
+ PG_RETURN_CSTRING(result);
+}
+
+/*****************************************************************************
+ * Custom subscripting logic functions
+ *****************************************************************************/
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = INT4OID;
+ sbsref->refassgntype = INT4OID;
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ List *upperIndexpr = NIL;
+ ListCell *l;
+
+ if (sbsref->reflowerindexpr != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *)lfirst(sbsref->reflowerindexpr->head))))));
+
+ foreach(l, sbsref->refupperindexpr)
+ {
+ Node *subexpr = (Node *) lfirst(l);
+
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *) lfirst(sbsref->refupperindexpr->head))))));
+
+ subexpr = coerce_to_target_type(pstate,
+ subexpr, exprType(subexpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript must have integer type"),
+ parser_errposition(pstate, exprLocation(subexpr))));
+
+ upperIndexpr = lappend(upperIndexpr, subexpr);
+
+ if (isAssignment)
+ {
+ Node *assignExpr = (Node *) sbsref->refassgnexpr;
+ Node *new_from;
+
+ new_from = coerce_to_target_type(pstate,
+ assignExpr, exprType(assignExpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (new_from == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom assignment requires int type"),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation(assignExpr))));
+ sbsref->refassgnexpr = (Expr *)new_from;
+ }
+ }
+
+ sbsref->refupperindexpr = upperIndexpr;
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ Custom *container= (Custom *) containerSource;
+ int index;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ return (Datum) container->first;
+ else
+ return (Datum) container->second;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ int index;
+ Custom *container = (Custom *) containerSource;
+
+ if (sbstate->resnull)
+ return containerSource;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ container->first = DatumGetInt32(sbstate->replacevalue);
+ else
+ container->second = DatumGetInt32(sbstate->replacevalue);
+
+ return (Datum) container;
+}
diff --git a/src/tutorial/subscripting.source b/src/tutorial/subscripting.source
new file mode 100644
index 0000000000..837cf30612
--- /dev/null
+++ b/src/tutorial/subscripting.source
@@ -0,0 +1,71 @@
+---------------------------------------------------------------------------
+--
+-- subscripting.sql-
+-- This file shows how to create a new subscripting procedure for
+-- user-defined type.
+--
+--
+-- Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+-- Portions Copyright (c) 1994, Regents of the University of California
+--
+-- src/tutorial/subscripting.source
+--
+---------------------------------------------------------------------------
+
+-----------------------------
+-- Creating a new type:
+-- We are going to create a new type called 'complex' which represents
+-- complex numbers.
+-- A user-defined type must have an input and an output function, and
+-- optionally can have binary input and output functions. All of these
+-- are usually user-defined C functions.
+-----------------------------
+
+-- Assume the user defined functions are in /home/erthalion/programms/postgresql-master/src/tutorial/complex$DLSUFFIX
+-- (we do not want to assume this is in the dynamic loader search path).
+-- Look at $PWD/complex.c for the source. Note that we declare all of
+-- them as STRICT, so we do not need to cope with NULL inputs in the
+-- C code. We also mark them IMMUTABLE, since they always return the
+-- same outputs given the same inputs.
+
+-- the input function 'complex_in' takes a null-terminated string (the
+-- textual representation of the type) and turns it into the internal
+-- (in memory) representation. You will get a message telling you 'complex'
+-- does not exist yet but that's okay.
+
+CREATE FUNCTION custom_in(cstring)
+ RETURNS custom
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+-- the output function 'complex_out' takes the internal representation and
+-- converts it into the textual representation.
+
+CREATE FUNCTION custom_out(custom)
+ RETURNS cstring
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 8,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler
+);
+
+-- we can use it in a table
+
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
--
2.21.0
--u66dlpg6gaevsofr
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v33-0005-Filling-gaps-in-jsonb-arrays.patch"
^ permalink raw reply [nested|flat] 55+ messages in thread
* [PATCH v33 4/5] Subscripting documentation
@ 2019-02-01 10:47 erthalion <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: erthalion @ 2019-02-01 10:47 UTC (permalink / raw)
Supporting documentation for generalized subscripting. It includes the
description of a new field in pg_type, the new section for jsonb
documentation about subscripting feature on this data type, and also the
tutorial about how to write subscripting operator for a custom data type.
Reviewed-by: Tom Lane, Arthur Zakirov, Oleksandr Shulgin
---
doc/src/sgml/catalogs.sgml | 8 ++
doc/src/sgml/extend.sgml | 6 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/json.sgml | 39 ++++++
doc/src/sgml/ref/create_type.sgml | 33 ++++-
doc/src/sgml/xsubscripting.sgml | 111 +++++++++++++++++
src/tutorial/Makefile | 4 +-
src/tutorial/subscripting.c | 201 ++++++++++++++++++++++++++++++
src/tutorial/subscripting.source | 71 +++++++++++
9 files changed, 470 insertions(+), 4 deletions(-)
create mode 100644 doc/src/sgml/xsubscripting.sgml
create mode 100644 src/tutorial/subscripting.c
create mode 100644 src/tutorial/subscripting.source
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 048ff284f7..b17e9ee382 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8971,6 +8971,14 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>typsubshandler</structfield></entry>
+ <entry><type>regproc</type></entry>
+ <entry><literal><link linkend="catalog-pg-proc"><structname>pg_proc</structname></link>.oid</literal></entry>
+ <entry>Custom subscripting function with type-specific logic for parsing
+ and validation, or 0 if this type doesn't support subscripting.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>typdefault</structfield> <type>text</type>
diff --git a/doc/src/sgml/extend.sgml b/doc/src/sgml/extend.sgml
index 890ff97b7a..65cf5706b6 100644
--- a/doc/src/sgml/extend.sgml
+++ b/doc/src/sgml/extend.sgml
@@ -33,6 +33,11 @@
operators (starting in <xref linkend="xoper"/>)
</para>
</listitem>
+ <listitem>
+ <para>
+ subscripting procedure (starting in <xref linkend="xsubscripting"/>)
+ </para>
+ </listitem>
<listitem>
<para>
operator classes for indexes (starting in <xref linkend="xindex"/>)
@@ -481,6 +486,7 @@ RETURNS anycompatible AS ...
&xaggr;
&xtypes;
&xoper;
+ &xsubscripting;
&xindex;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 64b5da0070..4c3fd37fc1 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -69,6 +69,7 @@
<!ENTITY xplang SYSTEM "xplang.sgml">
<!ENTITY xoper SYSTEM "xoper.sgml">
<!ENTITY xtypes SYSTEM "xtypes.sgml">
+<!ENTITY xsubscripting SYSTEM "xsubscripting.sgml">
<!ENTITY plperl SYSTEM "plperl.sgml">
<!ENTITY plpython SYSTEM "plpython.sgml">
<!ENTITY plsql SYSTEM "plpgsql.sgml">
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index c0a6554d4d..5c538dca05 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -602,6 +602,45 @@ SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qu
</para>
</sect2>
+ <sect2 id="jsonb-subscripting">
+ <title><type>jsonb</type> Subscripting</title>
+ <para>
+ <type>jsonb</type> data type supports array-style subscripting expressions
+ to extract or update particular elements. It's possible to use multiple
+ subscripting expressions to extract nested values. In this case, a chain of
+ subscripting expressions follows the same rules as the
+ <literal>path</literal> argument in <literal>jsonb_set</literal> function,
+ e.g. in case of arrays it is a 0-based operation or that negative integers
+ that appear in <literal>path</literal> count from the end of JSON arrays.
+ The result of subscripting expressions is always jsonb data type. An
+ example of subscripting syntax:
+<programlisting>
+-- Extract value by key
+SELECT ('{"a": 1}'::jsonb)['a'];
+
+-- Extract nested value by key path
+SELECT ('{"a": {"b": {"c": 1}}}'::jsonb)['a']['b']['c'];
+
+-- Extract element by index
+SELECT ('[1, "2", null]'::jsonb)[1];
+
+-- Update value by key
+UPDATE table_name SET jsonb_field['key'] = 1;
+
+-- Select records using where clause with subscripting. Since the result of
+-- subscripting is jsonb and we basically want to compare two jsonb objects, we
+-- need to put the value in double quotes to be able to convert it to jsonb.
+SELECT * FROM table_name WHERE jsonb_field['key'] = '"value"';
+</programlisting>
+
+ There is no special indexing support for such kind of expressions, but you
+ can always create a functional index that includes it
+<programlisting>
+CREATE INDEX idx ON table_name ((jsonb_field['key']));
+</programlisting>
+ </para>
+ </sect2>
+
<sect2>
<title>Transforms</title>
diff --git a/doc/src/sgml/ref/create_type.sgml b/doc/src/sgml/ref/create_type.sgml
index 111f8e65d2..a34df4d247 100644
--- a/doc/src/sgml/ref/create_type.sgml
+++ b/doc/src/sgml/ref/create_type.sgml
@@ -54,6 +54,7 @@ CREATE TYPE <replaceable class="parameter">name</replaceable> (
[ , ELEMENT = <replaceable class="parameter">element</replaceable> ]
[ , DELIMITER = <replaceable class="parameter">delimiter</replaceable> ]
[ , COLLATABLE = <replaceable class="parameter">collatable</replaceable> ]
+ [ , SUBSCRIPTING_HANDLER = <replaceable class="parameter">subscripting_handler_function</replaceable> ]
)
CREATE TYPE <replaceable class="parameter">name</replaceable>
@@ -196,8 +197,9 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
<replaceable class="parameter">receive_function</replaceable>,
<replaceable class="parameter">send_function</replaceable>,
<replaceable class="parameter">type_modifier_input_function</replaceable>,
- <replaceable class="parameter">type_modifier_output_function</replaceable> and
- <replaceable class="parameter">analyze_function</replaceable>
+ <replaceable class="parameter">type_modifier_output_function</replaceable>,
+ <replaceable class="parameter">analyze_function</replaceable>,
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
are optional. Generally these functions have to be coded in C
or another low-level language.
</para>
@@ -454,6 +456,22 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
make use of the collation information; this does not happen
automatically merely by marking the type collatable.
</para>
+
+ <para>
+ The optional
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
+ contains type-specific logic for subscripting of the data type.
+ By default, there is no such function provided, which means that the data
+ type doesn't support subscripting. The subscripting function must be
+ declared to take a single argument of type <type>internal</type>, and return
+ a <type>internal</type> result. There are two examples of implementation for
+ subscripting functions in case of array
+ (<replaceable class="parameter">array_subscripting_handler</replaceable>)
+ and jsonb
+ (<replaceable class="parameter">jsonb_subscripting_handler</replaceable>)
+ types in <filename>src/backend/utils/adt/arrayfuncs.c</filename> and
+ <filename>src/backend/utils/adt/jsonfuncs.c</filename>, respectively.
+ </para>
</refsect2>
<refsect2>
@@ -769,6 +787,17 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">subscripting_handler_function</replaceable></term>
+ <listitem>
+ <para>
+ The name of a function that returns list of type-specific callback functions to
+ support subscripting logic for the data type.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
diff --git a/doc/src/sgml/xsubscripting.sgml b/doc/src/sgml/xsubscripting.sgml
new file mode 100644
index 0000000000..7224e81fa2
--- /dev/null
+++ b/doc/src/sgml/xsubscripting.sgml
@@ -0,0 +1,111 @@
+<!-- doc/src/sgml/xsubscripting.sgml -->
+
+ <sect1 id="xsubscripting">
+ <title>User-defined subscripting procedure</title>
+
+ <indexterm zone="xsubscripting">
+ <primary>custom subscripting</primary>
+ </indexterm>
+ <para>
+ When you define a new base type, you can also specify a custom procedure to
+ handle subscripting expressions. It must contain logic for verification and
+ evaluation of this expression, i.e. fetching or updating some data in this
+ data type. For instance:
+</para>
+<programlisting><![CDATA[
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = someType;
+ sbsref->refassgntype = someType;
+
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ // some validation and coercion logic
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some assignment logic
+
+ return newContainer;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some fetch logic based on sbstate
+}]]>
+</programlisting>
+
+<para>
+ Then you can define a subscripting procedure and a custom data type:
+</para>
+<programlisting>
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '<replaceable>filename</replaceable>'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 4,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler,
+);
+</programlisting>
+
+<para>
+ and use it as usual:
+</para>
+<programlisting>
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
+</programlisting>
+
+
+ <para>
+ The examples of custom subscripting implementation can be found in
+ <filename>subscripting.sql</filename> and <filename>subscripting.c</filename>
+ in the <filename>src/tutorial</filename> directory of the source distribution.
+ See the <filename>README</filename> file in that directory for instructions
+ about running the examples.
+ </para>
+
+</sect1>
diff --git a/src/tutorial/Makefile b/src/tutorial/Makefile
index 16dc390f71..0ead60c2d4 100644
--- a/src/tutorial/Makefile
+++ b/src/tutorial/Makefile
@@ -13,8 +13,8 @@
#
#-------------------------------------------------------------------------
-MODULES = complex funcs
-DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql
+MODULES = complex funcs subscripting
+DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql subscripting.sql
ifdef NO_PGXS
subdir = src/tutorial
diff --git a/src/tutorial/subscripting.c b/src/tutorial/subscripting.c
new file mode 100644
index 0000000000..1eb8c45652
--- /dev/null
+++ b/src/tutorial/subscripting.c
@@ -0,0 +1,201 @@
+/*
+ * src/tutorial/subscripting.c
+ *
+ ******************************************************************************
+ This file contains routines that can be bound to a Postgres backend and
+ called by the backend in the process of processing queries. The calling
+ format for these routines is dictated by Postgres architecture.
+******************************************************************************/
+
+#include "postgres.h"
+
+#include "catalog/pg_type.h"
+#include "executor/executor.h"
+#include "executor/execExpr.h"
+#include "nodes/nodeFuncs.h"
+#include "parser/parse_coerce.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+SubscriptingRef * custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref);
+SubscriptingRef * custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate);
+Datum custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate);
+Datum custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate);
+
+PG_FUNCTION_INFO_V1(custom_in);
+PG_FUNCTION_INFO_V1(custom_out);
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+/*****************************************************************************
+ * Input/Output functions
+ *****************************************************************************/
+
+Datum
+custom_in(PG_FUNCTION_ARGS)
+{
+ char *str = PG_GETARG_CSTRING(0);
+ int firstValue,
+ secondValue;
+ Custom *result;
+
+ if (sscanf(str, " ( %d , %d )", &firstValue, &secondValue) != 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
+ errmsg("invalid input syntax for complex: \"%s\"",
+ str)));
+
+
+ result = (Custom *) palloc(sizeof(Custom));
+ result->first = firstValue;
+ result->second = secondValue;
+ PG_RETURN_POINTER(result);
+}
+
+Datum
+custom_out(PG_FUNCTION_ARGS)
+{
+ Custom *custom = (Custom *) PG_GETARG_POINTER(0);
+ char *result;
+
+ result = psprintf("(%d, %d)", custom->first, custom->second);
+ PG_RETURN_CSTRING(result);
+}
+
+/*****************************************************************************
+ * Custom subscripting logic functions
+ *****************************************************************************/
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = INT4OID;
+ sbsref->refassgntype = INT4OID;
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ List *upperIndexpr = NIL;
+ ListCell *l;
+
+ if (sbsref->reflowerindexpr != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *)lfirst(sbsref->reflowerindexpr->head))))));
+
+ foreach(l, sbsref->refupperindexpr)
+ {
+ Node *subexpr = (Node *) lfirst(l);
+
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *) lfirst(sbsref->refupperindexpr->head))))));
+
+ subexpr = coerce_to_target_type(pstate,
+ subexpr, exprType(subexpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript must have integer type"),
+ parser_errposition(pstate, exprLocation(subexpr))));
+
+ upperIndexpr = lappend(upperIndexpr, subexpr);
+
+ if (isAssignment)
+ {
+ Node *assignExpr = (Node *) sbsref->refassgnexpr;
+ Node *new_from;
+
+ new_from = coerce_to_target_type(pstate,
+ assignExpr, exprType(assignExpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (new_from == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom assignment requires int type"),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation(assignExpr))));
+ sbsref->refassgnexpr = (Expr *)new_from;
+ }
+ }
+
+ sbsref->refupperindexpr = upperIndexpr;
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ Custom *container= (Custom *) containerSource;
+ int index;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ return (Datum) container->first;
+ else
+ return (Datum) container->second;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ int index;
+ Custom *container = (Custom *) containerSource;
+
+ if (sbstate->resnull)
+ return containerSource;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ container->first = DatumGetInt32(sbstate->replacevalue);
+ else
+ container->second = DatumGetInt32(sbstate->replacevalue);
+
+ return (Datum) container;
+}
diff --git a/src/tutorial/subscripting.source b/src/tutorial/subscripting.source
new file mode 100644
index 0000000000..837cf30612
--- /dev/null
+++ b/src/tutorial/subscripting.source
@@ -0,0 +1,71 @@
+---------------------------------------------------------------------------
+--
+-- subscripting.sql-
+-- This file shows how to create a new subscripting procedure for
+-- user-defined type.
+--
+--
+-- Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+-- Portions Copyright (c) 1994, Regents of the University of California
+--
+-- src/tutorial/subscripting.source
+--
+---------------------------------------------------------------------------
+
+-----------------------------
+-- Creating a new type:
+-- We are going to create a new type called 'complex' which represents
+-- complex numbers.
+-- A user-defined type must have an input and an output function, and
+-- optionally can have binary input and output functions. All of these
+-- are usually user-defined C functions.
+-----------------------------
+
+-- Assume the user defined functions are in /home/erthalion/programms/postgresql-master/src/tutorial/complex$DLSUFFIX
+-- (we do not want to assume this is in the dynamic loader search path).
+-- Look at $PWD/complex.c for the source. Note that we declare all of
+-- them as STRICT, so we do not need to cope with NULL inputs in the
+-- C code. We also mark them IMMUTABLE, since they always return the
+-- same outputs given the same inputs.
+
+-- the input function 'complex_in' takes a null-terminated string (the
+-- textual representation of the type) and turns it into the internal
+-- (in memory) representation. You will get a message telling you 'complex'
+-- does not exist yet but that's okay.
+
+CREATE FUNCTION custom_in(cstring)
+ RETURNS custom
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+-- the output function 'complex_out' takes the internal representation and
+-- converts it into the textual representation.
+
+CREATE FUNCTION custom_out(custom)
+ RETURNS cstring
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 8,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler
+);
+
+-- we can use it in a table
+
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
--
2.21.0
--u66dlpg6gaevsofr
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v33-0005-Filling-gaps-in-jsonb-arrays.patch"
^ permalink raw reply [nested|flat] 55+ messages in thread
* [PATCH v35 4/5] Subscripting documentation
@ 2019-02-01 10:47 erthalion <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: erthalion @ 2019-02-01 10:47 UTC (permalink / raw)
Supporting documentation for generalized subscripting. It includes the
description of a new field in pg_type, the new section for jsonb
documentation about subscripting feature on this data type, and also the
tutorial about how to write subscripting operator for a custom data type.
Reviewed-by: Tom Lane, Arthur Zakirov, Oleksandr Shulgin
---
doc/src/sgml/catalogs.sgml | 8 ++
doc/src/sgml/extend.sgml | 6 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/json.sgml | 39 ++++++
doc/src/sgml/ref/create_type.sgml | 33 ++++-
doc/src/sgml/xsubscripting.sgml | 113 +++++++++++++++++
src/tutorial/Makefile | 4 +-
src/tutorial/subscripting.c | 204 ++++++++++++++++++++++++++++++
src/tutorial/subscripting.source | 71 +++++++++++
9 files changed, 475 insertions(+), 4 deletions(-)
create mode 100644 doc/src/sgml/xsubscripting.sgml
create mode 100644 src/tutorial/subscripting.c
create mode 100644 src/tutorial/subscripting.source
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 7e99928d0c..104cea3584 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8990,6 +8990,14 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>typsubshandler</structfield></entry>
+ <entry><type>regproc</type></entry>
+ <entry><literal><link linkend="catalog-pg-proc"><structname>pg_proc</structname></link>.oid</literal></entry>
+ <entry>Custom subscripting function with type-specific logic for parsing
+ and validation, or 0 if this type doesn't support subscripting.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>typdefault</structfield> <type>text</type>
diff --git a/doc/src/sgml/extend.sgml b/doc/src/sgml/extend.sgml
index e486006224..23c1764571 100644
--- a/doc/src/sgml/extend.sgml
+++ b/doc/src/sgml/extend.sgml
@@ -33,6 +33,11 @@
operators (starting in <xref linkend="xoper"/>)
</para>
</listitem>
+ <listitem>
+ <para>
+ subscripting procedure (starting in <xref linkend="xsubscripting"/>)
+ </para>
+ </listitem>
<listitem>
<para>
operator classes for indexes (starting in <xref linkend="xindex"/>)
@@ -481,6 +486,7 @@ RETURNS anycompatible AS ...
&xaggr;
&xtypes;
&xoper;
+ &xsubscripting;
&xindex;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 828396d4a9..3bebb34c1a 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -69,6 +69,7 @@
<!ENTITY xplang SYSTEM "xplang.sgml">
<!ENTITY xoper SYSTEM "xoper.sgml">
<!ENTITY xtypes SYSTEM "xtypes.sgml">
+<!ENTITY xsubscripting SYSTEM "xsubscripting.sgml">
<!ENTITY plperl SYSTEM "plperl.sgml">
<!ENTITY plpython SYSTEM "plpython.sgml">
<!ENTITY plsql SYSTEM "plpgsql.sgml">
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index c0a6554d4d..5c538dca05 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -602,6 +602,45 @@ SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qu
</para>
</sect2>
+ <sect2 id="jsonb-subscripting">
+ <title><type>jsonb</type> Subscripting</title>
+ <para>
+ <type>jsonb</type> data type supports array-style subscripting expressions
+ to extract or update particular elements. It's possible to use multiple
+ subscripting expressions to extract nested values. In this case, a chain of
+ subscripting expressions follows the same rules as the
+ <literal>path</literal> argument in <literal>jsonb_set</literal> function,
+ e.g. in case of arrays it is a 0-based operation or that negative integers
+ that appear in <literal>path</literal> count from the end of JSON arrays.
+ The result of subscripting expressions is always jsonb data type. An
+ example of subscripting syntax:
+<programlisting>
+-- Extract value by key
+SELECT ('{"a": 1}'::jsonb)['a'];
+
+-- Extract nested value by key path
+SELECT ('{"a": {"b": {"c": 1}}}'::jsonb)['a']['b']['c'];
+
+-- Extract element by index
+SELECT ('[1, "2", null]'::jsonb)[1];
+
+-- Update value by key
+UPDATE table_name SET jsonb_field['key'] = 1;
+
+-- Select records using where clause with subscripting. Since the result of
+-- subscripting is jsonb and we basically want to compare two jsonb objects, we
+-- need to put the value in double quotes to be able to convert it to jsonb.
+SELECT * FROM table_name WHERE jsonb_field['key'] = '"value"';
+</programlisting>
+
+ There is no special indexing support for such kind of expressions, but you
+ can always create a functional index that includes it
+<programlisting>
+CREATE INDEX idx ON table_name ((jsonb_field['key']));
+</programlisting>
+ </para>
+ </sect2>
+
<sect2>
<title>Transforms</title>
diff --git a/doc/src/sgml/ref/create_type.sgml b/doc/src/sgml/ref/create_type.sgml
index 111f8e65d2..a34df4d247 100644
--- a/doc/src/sgml/ref/create_type.sgml
+++ b/doc/src/sgml/ref/create_type.sgml
@@ -54,6 +54,7 @@ CREATE TYPE <replaceable class="parameter">name</replaceable> (
[ , ELEMENT = <replaceable class="parameter">element</replaceable> ]
[ , DELIMITER = <replaceable class="parameter">delimiter</replaceable> ]
[ , COLLATABLE = <replaceable class="parameter">collatable</replaceable> ]
+ [ , SUBSCRIPTING_HANDLER = <replaceable class="parameter">subscripting_handler_function</replaceable> ]
)
CREATE TYPE <replaceable class="parameter">name</replaceable>
@@ -196,8 +197,9 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
<replaceable class="parameter">receive_function</replaceable>,
<replaceable class="parameter">send_function</replaceable>,
<replaceable class="parameter">type_modifier_input_function</replaceable>,
- <replaceable class="parameter">type_modifier_output_function</replaceable> and
- <replaceable class="parameter">analyze_function</replaceable>
+ <replaceable class="parameter">type_modifier_output_function</replaceable>,
+ <replaceable class="parameter">analyze_function</replaceable>,
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
are optional. Generally these functions have to be coded in C
or another low-level language.
</para>
@@ -454,6 +456,22 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
make use of the collation information; this does not happen
automatically merely by marking the type collatable.
</para>
+
+ <para>
+ The optional
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
+ contains type-specific logic for subscripting of the data type.
+ By default, there is no such function provided, which means that the data
+ type doesn't support subscripting. The subscripting function must be
+ declared to take a single argument of type <type>internal</type>, and return
+ a <type>internal</type> result. There are two examples of implementation for
+ subscripting functions in case of array
+ (<replaceable class="parameter">array_subscripting_handler</replaceable>)
+ and jsonb
+ (<replaceable class="parameter">jsonb_subscripting_handler</replaceable>)
+ types in <filename>src/backend/utils/adt/arrayfuncs.c</filename> and
+ <filename>src/backend/utils/adt/jsonfuncs.c</filename>, respectively.
+ </para>
</refsect2>
<refsect2>
@@ -769,6 +787,17 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">subscripting_handler_function</replaceable></term>
+ <listitem>
+ <para>
+ The name of a function that returns list of type-specific callback functions to
+ support subscripting logic for the data type.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
diff --git a/doc/src/sgml/xsubscripting.sgml b/doc/src/sgml/xsubscripting.sgml
new file mode 100644
index 0000000000..5282df0361
--- /dev/null
+++ b/doc/src/sgml/xsubscripting.sgml
@@ -0,0 +1,113 @@
+<!-- doc/src/sgml/xsubscripting.sgml -->
+
+ <sect1 id="xsubscripting">
+ <title>User-defined subscripting procedure</title>
+
+ <indexterm zone="xsubscripting">
+ <primary>custom subscripting</primary>
+ </indexterm>
+ <para>
+ When you define a new base type, you can also specify a custom procedure to
+ handle subscripting expressions. It must contain logic for verification and
+ evaluation of this expression, i.e. fetching or updating some data in this
+ data type. For instance:
+</para>
+<programlisting><![CDATA[
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = someType;
+ sbsref->refassgntype = someType;
+
+ // You can also store any necessary information into sbsref->refopaque
+
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ // some validation and coercion logic
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some assignment logic
+
+ return newContainer;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some fetch logic based on sbstate
+}]]>
+</programlisting>
+
+<para>
+ Then you can define a subscripting procedure and a custom data type:
+</para>
+<programlisting>
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '<replaceable>filename</replaceable>'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 4,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler,
+);
+</programlisting>
+
+<para>
+ and use it as usual:
+</para>
+<programlisting>
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
+</programlisting>
+
+
+ <para>
+ The examples of custom subscripting implementation can be found in
+ <filename>subscripting.sql</filename> and <filename>subscripting.c</filename>
+ in the <filename>src/tutorial</filename> directory of the source distribution.
+ See the <filename>README</filename> file in that directory for instructions
+ about running the examples.
+ </para>
+
+</sect1>
diff --git a/src/tutorial/Makefile b/src/tutorial/Makefile
index 16dc390f71..0ead60c2d4 100644
--- a/src/tutorial/Makefile
+++ b/src/tutorial/Makefile
@@ -13,8 +13,8 @@
#
#-------------------------------------------------------------------------
-MODULES = complex funcs
-DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql
+MODULES = complex funcs subscripting
+DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql subscripting.sql
ifdef NO_PGXS
subdir = src/tutorial
diff --git a/src/tutorial/subscripting.c b/src/tutorial/subscripting.c
new file mode 100644
index 0000000000..aaac7927c6
--- /dev/null
+++ b/src/tutorial/subscripting.c
@@ -0,0 +1,204 @@
+/*
+ * src/tutorial/subscripting.c
+ *
+ ******************************************************************************
+ This file contains routines that can be bound to a Postgres backend and
+ called by the backend in the process of processing queries. The calling
+ format for these routines is dictated by Postgres architecture.
+******************************************************************************/
+
+#include "postgres.h"
+
+#include "catalog/pg_type.h"
+#include "executor/executor.h"
+#include "executor/execExpr.h"
+#include "nodes/nodeFuncs.h"
+#include "parser/parse_coerce.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+SubscriptingRef * custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref);
+SubscriptingRef * custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate);
+Datum custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate);
+Datum custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate);
+
+PG_FUNCTION_INFO_V1(custom_in);
+PG_FUNCTION_INFO_V1(custom_out);
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+/*****************************************************************************
+ * Input/Output functions
+ *****************************************************************************/
+
+Datum
+custom_in(PG_FUNCTION_ARGS)
+{
+ char *str = PG_GETARG_CSTRING(0);
+ int firstValue,
+ secondValue;
+ Custom *result;
+
+ if (sscanf(str, " ( %d , %d )", &firstValue, &secondValue) != 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
+ errmsg("invalid input syntax for complex: \"%s\"",
+ str)));
+
+
+ result = (Custom *) palloc(sizeof(Custom));
+ result->first = firstValue;
+ result->second = secondValue;
+ PG_RETURN_POINTER(result);
+}
+
+Datum
+custom_out(PG_FUNCTION_ARGS)
+{
+ Custom *custom = (Custom *) PG_GETARG_POINTER(0);
+ char *result;
+
+ result = psprintf("(%d, %d)", custom->first, custom->second);
+ PG_RETURN_CSTRING(result);
+}
+
+/*****************************************************************************
+ * Custom subscripting logic functions
+ *****************************************************************************/
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = INT4OID;
+ sbsref->refassgntype = INT4OID;
+
+ /* You can also store any necessary information into sbsref->refopaque */
+
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ List *upperIndexpr = NIL;
+ ListCell *l;
+
+ if (sbsref->reflowerindexpr != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *)lfirst(sbsref->reflowerindexpr->head))))));
+
+ foreach(l, sbsref->refupperindexpr)
+ {
+ Node *subexpr = (Node *) lfirst(l);
+
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *) lfirst(sbsref->refupperindexpr->head))))));
+
+ subexpr = coerce_to_target_type(pstate,
+ subexpr, exprType(subexpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript must have integer type"),
+ parser_errposition(pstate, exprLocation(subexpr))));
+
+ upperIndexpr = lappend(upperIndexpr, subexpr);
+
+ if (isAssignment)
+ {
+ Node *assignExpr = (Node *) sbsref->refassgnexpr;
+ Node *new_from;
+
+ new_from = coerce_to_target_type(pstate,
+ assignExpr, exprType(assignExpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (new_from == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom assignment requires int type"),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation(assignExpr))));
+ sbsref->refassgnexpr = (Expr *)new_from;
+ }
+ }
+
+ sbsref->refupperindexpr = upperIndexpr;
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ Custom *container= (Custom *) containerSource;
+ int index;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ return (Datum) container->first;
+ else
+ return (Datum) container->second;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ int index;
+ Custom *container = (Custom *) containerSource;
+
+ if (sbstate->resnull)
+ return containerSource;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ container->first = DatumGetInt32(sbstate->replacevalue);
+ else
+ container->second = DatumGetInt32(sbstate->replacevalue);
+
+ return (Datum) container;
+}
diff --git a/src/tutorial/subscripting.source b/src/tutorial/subscripting.source
new file mode 100644
index 0000000000..837cf30612
--- /dev/null
+++ b/src/tutorial/subscripting.source
@@ -0,0 +1,71 @@
+---------------------------------------------------------------------------
+--
+-- subscripting.sql-
+-- This file shows how to create a new subscripting procedure for
+-- user-defined type.
+--
+--
+-- Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+-- Portions Copyright (c) 1994, Regents of the University of California
+--
+-- src/tutorial/subscripting.source
+--
+---------------------------------------------------------------------------
+
+-----------------------------
+-- Creating a new type:
+-- We are going to create a new type called 'complex' which represents
+-- complex numbers.
+-- A user-defined type must have an input and an output function, and
+-- optionally can have binary input and output functions. All of these
+-- are usually user-defined C functions.
+-----------------------------
+
+-- Assume the user defined functions are in /home/erthalion/programms/postgresql-master/src/tutorial/complex$DLSUFFIX
+-- (we do not want to assume this is in the dynamic loader search path).
+-- Look at $PWD/complex.c for the source. Note that we declare all of
+-- them as STRICT, so we do not need to cope with NULL inputs in the
+-- C code. We also mark them IMMUTABLE, since they always return the
+-- same outputs given the same inputs.
+
+-- the input function 'complex_in' takes a null-terminated string (the
+-- textual representation of the type) and turns it into the internal
+-- (in memory) representation. You will get a message telling you 'complex'
+-- does not exist yet but that's okay.
+
+CREATE FUNCTION custom_in(cstring)
+ RETURNS custom
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+-- the output function 'complex_out' takes the internal representation and
+-- converts it into the textual representation.
+
+CREATE FUNCTION custom_out(custom)
+ RETURNS cstring
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 8,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler
+);
+
+-- we can use it in a table
+
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
--
2.21.0
--evj76j7ekwacmcsz
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v35-0005-Filling-gaps-in-jsonb-arrays.patch"
^ permalink raw reply [nested|flat] 55+ messages in thread
* [PATCH v30 4/6] Subscripting documentation
@ 2019-02-01 10:47 erthalion <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: erthalion @ 2019-02-01 10:47 UTC (permalink / raw)
Supporting documentation for generalized subscripting. It includes the
description of a new field in pg_type, the new section for jsonb
documentation about subscripting feature on this data type, and also the
tutorial about how to write subscripting operator for a custom data type.
Reviewed-by: Tom Lane, Arthur Zakirov, Oleksandr Shulgin
---
doc/src/sgml/catalogs.sgml | 8 ++
doc/src/sgml/extend.sgml | 6 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/json.sgml | 39 ++++++
doc/src/sgml/ref/create_type.sgml | 33 ++++-
doc/src/sgml/xsubscripting.sgml | 111 +++++++++++++++++
src/tutorial/Makefile | 4 +-
src/tutorial/subscripting.c | 201 ++++++++++++++++++++++++++++++
src/tutorial/subscripting.source | 71 +++++++++++
9 files changed, 470 insertions(+), 4 deletions(-)
create mode 100644 doc/src/sgml/xsubscripting.sgml
create mode 100644 src/tutorial/subscripting.c
create mode 100644 src/tutorial/subscripting.source
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 34bc0d0526..328a1da6fe 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -7911,6 +7911,14 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>typsubshandler</structfield></entry>
+ <entry><type>regproc</type></entry>
+ <entry><literal><link linkend="catalog-pg-proc"><structname>pg_proc</structname></link>.oid</literal></entry>
+ <entry>Custom subscripting function with type-specific logic for parsing
+ and validation, or 0 if this type doesn't support subscripting.</entry>
+ </row>
+
<row>
<entry><structfield>typdefaultbin</structfield></entry>
<entry><type>pg_node_tree</type></entry>
diff --git a/doc/src/sgml/extend.sgml b/doc/src/sgml/extend.sgml
index 9ec1af780b..057010157e 100644
--- a/doc/src/sgml/extend.sgml
+++ b/doc/src/sgml/extend.sgml
@@ -33,6 +33,11 @@
operators (starting in <xref linkend="xoper"/>)
</para>
</listitem>
+ <listitem>
+ <para>
+ subscripting procedure (starting in <xref linkend="xsubscripting"/>)
+ </para>
+ </listitem>
<listitem>
<para>
operator classes for indexes (starting in <xref linkend="xindex"/>)
@@ -314,6 +319,7 @@
&xaggr;
&xtypes;
&xoper;
+ &xsubscripting;
&xindex;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 3da2365ea9..650e21b7e1 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -69,6 +69,7 @@
<!ENTITY xplang SYSTEM "xplang.sgml">
<!ENTITY xoper SYSTEM "xoper.sgml">
<!ENTITY xtypes SYSTEM "xtypes.sgml">
+<!ENTITY xsubscripting SYSTEM "xsubscripting.sgml">
<!ENTITY plperl SYSTEM "plperl.sgml">
<!ENTITY plpython SYSTEM "plpython.sgml">
<!ENTITY plsql SYSTEM "plpgsql.sgml">
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index 1b6aaf0a55..189f03b41e 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -600,6 +600,45 @@ SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qu
</para>
</sect2>
+ <sect2 id="jsonb-subscripting">
+ <title><type>jsonb</type> Subscripting</title>
+ <para>
+ <type>jsonb</type> data type supports array-style subscripting expressions
+ to extract or update particular element. It's possible to use multiple
+ subscripting expressions to extract nested values. In this case a chain of
+ subscripting expressions follows the same rules as the
+ <literal>path</literal> argument in <literal>jsonb_set</literal> function,
+ e.g. in case of arrays it is a 0-based operation or that negative integers
+ that appear in <literal>path</literal> count from the end of JSON arrays.
+ The result of subscripting expressions is always jsonb data type. An
+ example of subscripting syntax:
+<programlisting>
+-- Extract value by key
+SELECT ('{"a": 1}'::jsonb)['a'];
+
+-- Extract nested value by key path
+SELECT ('{"a": {"b": {"c": 1}}}'::jsonb)['a']['b']['c'];
+
+-- Extract element by index
+SELECT ('[1, "2", null]'::jsonb)[1];
+
+-- Update value by key
+UPDATE table_name SET jsonb_field['key'] = 1;
+
+-- Select records using where clause with subscripting. Since the result of
+-- subscripting is jsonb and we basically want to compare two jsonb objects, we
+-- need to put the value in double quotes to be able to convert it to jsonb.
+SELECT * FROM table_name WHERE jsonb_field['key'] = '"value"';
+</programlisting>
+
+ There is no special indexing support for such kind of expressions, but you
+ always can create a functional index that includes it
+<programlisting>
+CREATE INDEX idx ON table_name ((jsonb_field['key']));
+</programlisting>
+ </para>
+ </sect2>
+
<sect2>
<title>Transforms</title>
diff --git a/doc/src/sgml/ref/create_type.sgml b/doc/src/sgml/ref/create_type.sgml
index 175315f3d7..bd622f5ef3 100644
--- a/doc/src/sgml/ref/create_type.sgml
+++ b/doc/src/sgml/ref/create_type.sgml
@@ -54,6 +54,7 @@ CREATE TYPE <replaceable class="parameter">name</replaceable> (
[ , ELEMENT = <replaceable class="parameter">element</replaceable> ]
[ , DELIMITER = <replaceable class="parameter">delimiter</replaceable> ]
[ , COLLATABLE = <replaceable class="parameter">collatable</replaceable> ]
+ [ , SUBSCRIPTING_HANDLER = <replaceable class="parameter">subscripting_handler_function</replaceable> ]
)
CREATE TYPE <replaceable class="parameter">name</replaceable>
@@ -196,8 +197,9 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
<replaceable class="parameter">receive_function</replaceable>,
<replaceable class="parameter">send_function</replaceable>,
<replaceable class="parameter">type_modifier_input_function</replaceable>,
- <replaceable class="parameter">type_modifier_output_function</replaceable> and
- <replaceable class="parameter">analyze_function</replaceable>
+ <replaceable class="parameter">type_modifier_output_function</replaceable>,
+ <replaceable class="parameter">analyze_function</replaceable>,
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
are optional. Generally these functions have to be coded in C
or another low-level language.
</para>
@@ -454,6 +456,22 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
make use of the collation information; this does not happen
automatically merely by marking the type collatable.
</para>
+
+ <para>
+ The optional
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
+ contains type-specific logic for subscripting of the data type.
+ By default, there is no such function provided, which means that the data
+ type doesn't support subscripting. The subscripting function must be
+ declared to take a single argument of type <type>internal</type>, and return
+ a <type>internal</type> result. There are two examples of implementation for
+ subscripting functions in case of array
+ (<replaceable class="parameter">array_subscripting_handler</replaceable>)
+ and jsonb
+ (<replaceable class="parameter">jsonb_subscripting_handler</replaceable>)
+ types in <filename>src/backend/utils/adt/arrayfuncs.c</filename> and
+ <filename>src/backend/utils/adt/jsonfuncs.c</filename> corresponding.
+ </para>
</refsect2>
<refsect2>
@@ -769,6 +787,17 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">subscripting_handler_function</replaceable></term>
+ <listitem>
+ <para>
+ The name of a function that returns list of type-specific callback functions to
+ support subscripting logic for the data type.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
diff --git a/doc/src/sgml/xsubscripting.sgml b/doc/src/sgml/xsubscripting.sgml
new file mode 100644
index 0000000000..d701631223
--- /dev/null
+++ b/doc/src/sgml/xsubscripting.sgml
@@ -0,0 +1,111 @@
+<!-- doc/src/sgml/xsubscripting.sgml -->
+
+ <sect1 id="xsubscripting">
+ <title>User-defined subscripting procedure</title>
+
+ <indexterm zone="xsubscripting">
+ <primary>custom subscripting</primary>
+ </indexterm>
+ <para>
+ When you define a new base type, you can also specify a custom procedures to
+ handle subscripting expressions. They must contain logic for verification and
+ evaluation of this expression, i.e. fetching or updating some data in this
+ data type. For instance:
+</para>
+<programlisting><![CDATA[
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = someType;
+ sbsref->refassgntype = someType;
+
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ // some validation and coercion logic
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some assignment logic
+
+ return newContainer;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some fetch logic based on sbsdata
+}]]>
+</programlisting>
+
+<para>
+ Then you can define a subscripting procedures and a custom data type:
+</para>
+<programlisting>
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '<replaceable>filename</replaceable>'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 4,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler,
+);
+</programlisting>
+
+<para>
+ and use it as usual:
+</para>
+<programlisting>
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
+</programlisting>
+
+
+ <para>
+ The examples of custom subscripting implementation can be found in
+ <filename>subscripting.sql</filename> and <filename>subscripting.c</filename>
+ in the <filename>src/tutorial</filename> directory of the source distribution.
+ See the <filename>README</filename> file in that directory for instructions
+ about running the examples.
+ </para>
+
+</sect1>
diff --git a/src/tutorial/Makefile b/src/tutorial/Makefile
index 16dc390f71..0ead60c2d4 100644
--- a/src/tutorial/Makefile
+++ b/src/tutorial/Makefile
@@ -13,8 +13,8 @@
#
#-------------------------------------------------------------------------
-MODULES = complex funcs
-DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql
+MODULES = complex funcs subscripting
+DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql subscripting.sql
ifdef NO_PGXS
subdir = src/tutorial
diff --git a/src/tutorial/subscripting.c b/src/tutorial/subscripting.c
new file mode 100644
index 0000000000..1eb8c45652
--- /dev/null
+++ b/src/tutorial/subscripting.c
@@ -0,0 +1,201 @@
+/*
+ * src/tutorial/subscripting.c
+ *
+ ******************************************************************************
+ This file contains routines that can be bound to a Postgres backend and
+ called by the backend in the process of processing queries. The calling
+ format for these routines is dictated by Postgres architecture.
+******************************************************************************/
+
+#include "postgres.h"
+
+#include "catalog/pg_type.h"
+#include "executor/executor.h"
+#include "executor/execExpr.h"
+#include "nodes/nodeFuncs.h"
+#include "parser/parse_coerce.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+SubscriptingRef * custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref);
+SubscriptingRef * custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate);
+Datum custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate);
+Datum custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate);
+
+PG_FUNCTION_INFO_V1(custom_in);
+PG_FUNCTION_INFO_V1(custom_out);
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+/*****************************************************************************
+ * Input/Output functions
+ *****************************************************************************/
+
+Datum
+custom_in(PG_FUNCTION_ARGS)
+{
+ char *str = PG_GETARG_CSTRING(0);
+ int firstValue,
+ secondValue;
+ Custom *result;
+
+ if (sscanf(str, " ( %d , %d )", &firstValue, &secondValue) != 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
+ errmsg("invalid input syntax for complex: \"%s\"",
+ str)));
+
+
+ result = (Custom *) palloc(sizeof(Custom));
+ result->first = firstValue;
+ result->second = secondValue;
+ PG_RETURN_POINTER(result);
+}
+
+Datum
+custom_out(PG_FUNCTION_ARGS)
+{
+ Custom *custom = (Custom *) PG_GETARG_POINTER(0);
+ char *result;
+
+ result = psprintf("(%d, %d)", custom->first, custom->second);
+ PG_RETURN_CSTRING(result);
+}
+
+/*****************************************************************************
+ * Custom subscripting logic functions
+ *****************************************************************************/
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = INT4OID;
+ sbsref->refassgntype = INT4OID;
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ List *upperIndexpr = NIL;
+ ListCell *l;
+
+ if (sbsref->reflowerindexpr != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *)lfirst(sbsref->reflowerindexpr->head))))));
+
+ foreach(l, sbsref->refupperindexpr)
+ {
+ Node *subexpr = (Node *) lfirst(l);
+
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *) lfirst(sbsref->refupperindexpr->head))))));
+
+ subexpr = coerce_to_target_type(pstate,
+ subexpr, exprType(subexpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript must have integer type"),
+ parser_errposition(pstate, exprLocation(subexpr))));
+
+ upperIndexpr = lappend(upperIndexpr, subexpr);
+
+ if (isAssignment)
+ {
+ Node *assignExpr = (Node *) sbsref->refassgnexpr;
+ Node *new_from;
+
+ new_from = coerce_to_target_type(pstate,
+ assignExpr, exprType(assignExpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (new_from == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom assignment requires int type"),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation(assignExpr))));
+ sbsref->refassgnexpr = (Expr *)new_from;
+ }
+ }
+
+ sbsref->refupperindexpr = upperIndexpr;
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ Custom *container= (Custom *) containerSource;
+ int index;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ return (Datum) container->first;
+ else
+ return (Datum) container->second;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ int index;
+ Custom *container = (Custom *) containerSource;
+
+ if (sbstate->resnull)
+ return containerSource;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ container->first = DatumGetInt32(sbstate->replacevalue);
+ else
+ container->second = DatumGetInt32(sbstate->replacevalue);
+
+ return (Datum) container;
+}
diff --git a/src/tutorial/subscripting.source b/src/tutorial/subscripting.source
new file mode 100644
index 0000000000..837cf30612
--- /dev/null
+++ b/src/tutorial/subscripting.source
@@ -0,0 +1,71 @@
+---------------------------------------------------------------------------
+--
+-- subscripting.sql-
+-- This file shows how to create a new subscripting procedure for
+-- user-defined type.
+--
+--
+-- Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+-- Portions Copyright (c) 1994, Regents of the University of California
+--
+-- src/tutorial/subscripting.source
+--
+---------------------------------------------------------------------------
+
+-----------------------------
+-- Creating a new type:
+-- We are going to create a new type called 'complex' which represents
+-- complex numbers.
+-- A user-defined type must have an input and an output function, and
+-- optionally can have binary input and output functions. All of these
+-- are usually user-defined C functions.
+-----------------------------
+
+-- Assume the user defined functions are in /home/erthalion/programms/postgresql-master/src/tutorial/complex$DLSUFFIX
+-- (we do not want to assume this is in the dynamic loader search path).
+-- Look at $PWD/complex.c for the source. Note that we declare all of
+-- them as STRICT, so we do not need to cope with NULL inputs in the
+-- C code. We also mark them IMMUTABLE, since they always return the
+-- same outputs given the same inputs.
+
+-- the input function 'complex_in' takes a null-terminated string (the
+-- textual representation of the type) and turns it into the internal
+-- (in memory) representation. You will get a message telling you 'complex'
+-- does not exist yet but that's okay.
+
+CREATE FUNCTION custom_in(cstring)
+ RETURNS custom
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+-- the output function 'complex_out' takes the internal representation and
+-- converts it into the textual representation.
+
+CREATE FUNCTION custom_out(custom)
+ RETURNS cstring
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 8,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler
+);
+
+-- we can use it in a table
+
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
--
2.21.0
--g5st7yssq7xavhfb
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v30-0005-Polymorphic-subscripting.patch"
^ permalink raw reply [nested|flat] 55+ messages in thread
* [PATCH v33 4/5] Subscripting documentation
@ 2019-02-01 10:47 erthalion <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: erthalion @ 2019-02-01 10:47 UTC (permalink / raw)
Supporting documentation for generalized subscripting. It includes the
description of a new field in pg_type, the new section for jsonb
documentation about subscripting feature on this data type, and also the
tutorial about how to write subscripting operator for a custom data type.
Reviewed-by: Tom Lane, Arthur Zakirov, Oleksandr Shulgin
---
doc/src/sgml/catalogs.sgml | 8 ++
doc/src/sgml/extend.sgml | 6 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/json.sgml | 39 ++++++
doc/src/sgml/ref/create_type.sgml | 33 ++++-
doc/src/sgml/xsubscripting.sgml | 111 +++++++++++++++++
src/tutorial/Makefile | 4 +-
src/tutorial/subscripting.c | 201 ++++++++++++++++++++++++++++++
src/tutorial/subscripting.source | 71 +++++++++++
9 files changed, 470 insertions(+), 4 deletions(-)
create mode 100644 doc/src/sgml/xsubscripting.sgml
create mode 100644 src/tutorial/subscripting.c
create mode 100644 src/tutorial/subscripting.source
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 048ff284f7..b17e9ee382 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8971,6 +8971,14 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>typsubshandler</structfield></entry>
+ <entry><type>regproc</type></entry>
+ <entry><literal><link linkend="catalog-pg-proc"><structname>pg_proc</structname></link>.oid</literal></entry>
+ <entry>Custom subscripting function with type-specific logic for parsing
+ and validation, or 0 if this type doesn't support subscripting.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>typdefault</structfield> <type>text</type>
diff --git a/doc/src/sgml/extend.sgml b/doc/src/sgml/extend.sgml
index 890ff97b7a..65cf5706b6 100644
--- a/doc/src/sgml/extend.sgml
+++ b/doc/src/sgml/extend.sgml
@@ -33,6 +33,11 @@
operators (starting in <xref linkend="xoper"/>)
</para>
</listitem>
+ <listitem>
+ <para>
+ subscripting procedure (starting in <xref linkend="xsubscripting"/>)
+ </para>
+ </listitem>
<listitem>
<para>
operator classes for indexes (starting in <xref linkend="xindex"/>)
@@ -481,6 +486,7 @@ RETURNS anycompatible AS ...
&xaggr;
&xtypes;
&xoper;
+ &xsubscripting;
&xindex;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 64b5da0070..4c3fd37fc1 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -69,6 +69,7 @@
<!ENTITY xplang SYSTEM "xplang.sgml">
<!ENTITY xoper SYSTEM "xoper.sgml">
<!ENTITY xtypes SYSTEM "xtypes.sgml">
+<!ENTITY xsubscripting SYSTEM "xsubscripting.sgml">
<!ENTITY plperl SYSTEM "plperl.sgml">
<!ENTITY plpython SYSTEM "plpython.sgml">
<!ENTITY plsql SYSTEM "plpgsql.sgml">
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index c0a6554d4d..5c538dca05 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -602,6 +602,45 @@ SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qu
</para>
</sect2>
+ <sect2 id="jsonb-subscripting">
+ <title><type>jsonb</type> Subscripting</title>
+ <para>
+ <type>jsonb</type> data type supports array-style subscripting expressions
+ to extract or update particular elements. It's possible to use multiple
+ subscripting expressions to extract nested values. In this case, a chain of
+ subscripting expressions follows the same rules as the
+ <literal>path</literal> argument in <literal>jsonb_set</literal> function,
+ e.g. in case of arrays it is a 0-based operation or that negative integers
+ that appear in <literal>path</literal> count from the end of JSON arrays.
+ The result of subscripting expressions is always jsonb data type. An
+ example of subscripting syntax:
+<programlisting>
+-- Extract value by key
+SELECT ('{"a": 1}'::jsonb)['a'];
+
+-- Extract nested value by key path
+SELECT ('{"a": {"b": {"c": 1}}}'::jsonb)['a']['b']['c'];
+
+-- Extract element by index
+SELECT ('[1, "2", null]'::jsonb)[1];
+
+-- Update value by key
+UPDATE table_name SET jsonb_field['key'] = 1;
+
+-- Select records using where clause with subscripting. Since the result of
+-- subscripting is jsonb and we basically want to compare two jsonb objects, we
+-- need to put the value in double quotes to be able to convert it to jsonb.
+SELECT * FROM table_name WHERE jsonb_field['key'] = '"value"';
+</programlisting>
+
+ There is no special indexing support for such kind of expressions, but you
+ can always create a functional index that includes it
+<programlisting>
+CREATE INDEX idx ON table_name ((jsonb_field['key']));
+</programlisting>
+ </para>
+ </sect2>
+
<sect2>
<title>Transforms</title>
diff --git a/doc/src/sgml/ref/create_type.sgml b/doc/src/sgml/ref/create_type.sgml
index 111f8e65d2..a34df4d247 100644
--- a/doc/src/sgml/ref/create_type.sgml
+++ b/doc/src/sgml/ref/create_type.sgml
@@ -54,6 +54,7 @@ CREATE TYPE <replaceable class="parameter">name</replaceable> (
[ , ELEMENT = <replaceable class="parameter">element</replaceable> ]
[ , DELIMITER = <replaceable class="parameter">delimiter</replaceable> ]
[ , COLLATABLE = <replaceable class="parameter">collatable</replaceable> ]
+ [ , SUBSCRIPTING_HANDLER = <replaceable class="parameter">subscripting_handler_function</replaceable> ]
)
CREATE TYPE <replaceable class="parameter">name</replaceable>
@@ -196,8 +197,9 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
<replaceable class="parameter">receive_function</replaceable>,
<replaceable class="parameter">send_function</replaceable>,
<replaceable class="parameter">type_modifier_input_function</replaceable>,
- <replaceable class="parameter">type_modifier_output_function</replaceable> and
- <replaceable class="parameter">analyze_function</replaceable>
+ <replaceable class="parameter">type_modifier_output_function</replaceable>,
+ <replaceable class="parameter">analyze_function</replaceable>,
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
are optional. Generally these functions have to be coded in C
or another low-level language.
</para>
@@ -454,6 +456,22 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
make use of the collation information; this does not happen
automatically merely by marking the type collatable.
</para>
+
+ <para>
+ The optional
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
+ contains type-specific logic for subscripting of the data type.
+ By default, there is no such function provided, which means that the data
+ type doesn't support subscripting. The subscripting function must be
+ declared to take a single argument of type <type>internal</type>, and return
+ a <type>internal</type> result. There are two examples of implementation for
+ subscripting functions in case of array
+ (<replaceable class="parameter">array_subscripting_handler</replaceable>)
+ and jsonb
+ (<replaceable class="parameter">jsonb_subscripting_handler</replaceable>)
+ types in <filename>src/backend/utils/adt/arrayfuncs.c</filename> and
+ <filename>src/backend/utils/adt/jsonfuncs.c</filename>, respectively.
+ </para>
</refsect2>
<refsect2>
@@ -769,6 +787,17 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">subscripting_handler_function</replaceable></term>
+ <listitem>
+ <para>
+ The name of a function that returns list of type-specific callback functions to
+ support subscripting logic for the data type.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
diff --git a/doc/src/sgml/xsubscripting.sgml b/doc/src/sgml/xsubscripting.sgml
new file mode 100644
index 0000000000..7224e81fa2
--- /dev/null
+++ b/doc/src/sgml/xsubscripting.sgml
@@ -0,0 +1,111 @@
+<!-- doc/src/sgml/xsubscripting.sgml -->
+
+ <sect1 id="xsubscripting">
+ <title>User-defined subscripting procedure</title>
+
+ <indexterm zone="xsubscripting">
+ <primary>custom subscripting</primary>
+ </indexterm>
+ <para>
+ When you define a new base type, you can also specify a custom procedure to
+ handle subscripting expressions. It must contain logic for verification and
+ evaluation of this expression, i.e. fetching or updating some data in this
+ data type. For instance:
+</para>
+<programlisting><![CDATA[
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = someType;
+ sbsref->refassgntype = someType;
+
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ // some validation and coercion logic
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some assignment logic
+
+ return newContainer;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some fetch logic based on sbstate
+}]]>
+</programlisting>
+
+<para>
+ Then you can define a subscripting procedure and a custom data type:
+</para>
+<programlisting>
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '<replaceable>filename</replaceable>'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 4,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler,
+);
+</programlisting>
+
+<para>
+ and use it as usual:
+</para>
+<programlisting>
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
+</programlisting>
+
+
+ <para>
+ The examples of custom subscripting implementation can be found in
+ <filename>subscripting.sql</filename> and <filename>subscripting.c</filename>
+ in the <filename>src/tutorial</filename> directory of the source distribution.
+ See the <filename>README</filename> file in that directory for instructions
+ about running the examples.
+ </para>
+
+</sect1>
diff --git a/src/tutorial/Makefile b/src/tutorial/Makefile
index 16dc390f71..0ead60c2d4 100644
--- a/src/tutorial/Makefile
+++ b/src/tutorial/Makefile
@@ -13,8 +13,8 @@
#
#-------------------------------------------------------------------------
-MODULES = complex funcs
-DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql
+MODULES = complex funcs subscripting
+DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql subscripting.sql
ifdef NO_PGXS
subdir = src/tutorial
diff --git a/src/tutorial/subscripting.c b/src/tutorial/subscripting.c
new file mode 100644
index 0000000000..1eb8c45652
--- /dev/null
+++ b/src/tutorial/subscripting.c
@@ -0,0 +1,201 @@
+/*
+ * src/tutorial/subscripting.c
+ *
+ ******************************************************************************
+ This file contains routines that can be bound to a Postgres backend and
+ called by the backend in the process of processing queries. The calling
+ format for these routines is dictated by Postgres architecture.
+******************************************************************************/
+
+#include "postgres.h"
+
+#include "catalog/pg_type.h"
+#include "executor/executor.h"
+#include "executor/execExpr.h"
+#include "nodes/nodeFuncs.h"
+#include "parser/parse_coerce.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+SubscriptingRef * custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref);
+SubscriptingRef * custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate);
+Datum custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate);
+Datum custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate);
+
+PG_FUNCTION_INFO_V1(custom_in);
+PG_FUNCTION_INFO_V1(custom_out);
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+/*****************************************************************************
+ * Input/Output functions
+ *****************************************************************************/
+
+Datum
+custom_in(PG_FUNCTION_ARGS)
+{
+ char *str = PG_GETARG_CSTRING(0);
+ int firstValue,
+ secondValue;
+ Custom *result;
+
+ if (sscanf(str, " ( %d , %d )", &firstValue, &secondValue) != 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
+ errmsg("invalid input syntax for complex: \"%s\"",
+ str)));
+
+
+ result = (Custom *) palloc(sizeof(Custom));
+ result->first = firstValue;
+ result->second = secondValue;
+ PG_RETURN_POINTER(result);
+}
+
+Datum
+custom_out(PG_FUNCTION_ARGS)
+{
+ Custom *custom = (Custom *) PG_GETARG_POINTER(0);
+ char *result;
+
+ result = psprintf("(%d, %d)", custom->first, custom->second);
+ PG_RETURN_CSTRING(result);
+}
+
+/*****************************************************************************
+ * Custom subscripting logic functions
+ *****************************************************************************/
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = INT4OID;
+ sbsref->refassgntype = INT4OID;
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ List *upperIndexpr = NIL;
+ ListCell *l;
+
+ if (sbsref->reflowerindexpr != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *)lfirst(sbsref->reflowerindexpr->head))))));
+
+ foreach(l, sbsref->refupperindexpr)
+ {
+ Node *subexpr = (Node *) lfirst(l);
+
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *) lfirst(sbsref->refupperindexpr->head))))));
+
+ subexpr = coerce_to_target_type(pstate,
+ subexpr, exprType(subexpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript must have integer type"),
+ parser_errposition(pstate, exprLocation(subexpr))));
+
+ upperIndexpr = lappend(upperIndexpr, subexpr);
+
+ if (isAssignment)
+ {
+ Node *assignExpr = (Node *) sbsref->refassgnexpr;
+ Node *new_from;
+
+ new_from = coerce_to_target_type(pstate,
+ assignExpr, exprType(assignExpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (new_from == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom assignment requires int type"),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation(assignExpr))));
+ sbsref->refassgnexpr = (Expr *)new_from;
+ }
+ }
+
+ sbsref->refupperindexpr = upperIndexpr;
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ Custom *container= (Custom *) containerSource;
+ int index;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ return (Datum) container->first;
+ else
+ return (Datum) container->second;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ int index;
+ Custom *container = (Custom *) containerSource;
+
+ if (sbstate->resnull)
+ return containerSource;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ container->first = DatumGetInt32(sbstate->replacevalue);
+ else
+ container->second = DatumGetInt32(sbstate->replacevalue);
+
+ return (Datum) container;
+}
diff --git a/src/tutorial/subscripting.source b/src/tutorial/subscripting.source
new file mode 100644
index 0000000000..837cf30612
--- /dev/null
+++ b/src/tutorial/subscripting.source
@@ -0,0 +1,71 @@
+---------------------------------------------------------------------------
+--
+-- subscripting.sql-
+-- This file shows how to create a new subscripting procedure for
+-- user-defined type.
+--
+--
+-- Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+-- Portions Copyright (c) 1994, Regents of the University of California
+--
+-- src/tutorial/subscripting.source
+--
+---------------------------------------------------------------------------
+
+-----------------------------
+-- Creating a new type:
+-- We are going to create a new type called 'complex' which represents
+-- complex numbers.
+-- A user-defined type must have an input and an output function, and
+-- optionally can have binary input and output functions. All of these
+-- are usually user-defined C functions.
+-----------------------------
+
+-- Assume the user defined functions are in /home/erthalion/programms/postgresql-master/src/tutorial/complex$DLSUFFIX
+-- (we do not want to assume this is in the dynamic loader search path).
+-- Look at $PWD/complex.c for the source. Note that we declare all of
+-- them as STRICT, so we do not need to cope with NULL inputs in the
+-- C code. We also mark them IMMUTABLE, since they always return the
+-- same outputs given the same inputs.
+
+-- the input function 'complex_in' takes a null-terminated string (the
+-- textual representation of the type) and turns it into the internal
+-- (in memory) representation. You will get a message telling you 'complex'
+-- does not exist yet but that's okay.
+
+CREATE FUNCTION custom_in(cstring)
+ RETURNS custom
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+-- the output function 'complex_out' takes the internal representation and
+-- converts it into the textual representation.
+
+CREATE FUNCTION custom_out(custom)
+ RETURNS cstring
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 8,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler
+);
+
+-- we can use it in a table
+
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
--
2.21.0
--u66dlpg6gaevsofr
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v33-0005-Filling-gaps-in-jsonb-arrays.patch"
^ permalink raw reply [nested|flat] 55+ messages in thread
* [PATCH v33 4/5] Subscripting documentation
@ 2019-02-01 10:47 erthalion <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: erthalion @ 2019-02-01 10:47 UTC (permalink / raw)
Supporting documentation for generalized subscripting. It includes the
description of a new field in pg_type, the new section for jsonb
documentation about subscripting feature on this data type, and also the
tutorial about how to write subscripting operator for a custom data type.
Reviewed-by: Tom Lane, Arthur Zakirov, Oleksandr Shulgin
---
doc/src/sgml/catalogs.sgml | 8 ++
doc/src/sgml/extend.sgml | 6 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/json.sgml | 39 ++++++
doc/src/sgml/ref/create_type.sgml | 33 ++++-
doc/src/sgml/xsubscripting.sgml | 111 +++++++++++++++++
src/tutorial/Makefile | 4 +-
src/tutorial/subscripting.c | 201 ++++++++++++++++++++++++++++++
src/tutorial/subscripting.source | 71 +++++++++++
9 files changed, 470 insertions(+), 4 deletions(-)
create mode 100644 doc/src/sgml/xsubscripting.sgml
create mode 100644 src/tutorial/subscripting.c
create mode 100644 src/tutorial/subscripting.source
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 048ff284f7..b17e9ee382 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8971,6 +8971,14 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>typsubshandler</structfield></entry>
+ <entry><type>regproc</type></entry>
+ <entry><literal><link linkend="catalog-pg-proc"><structname>pg_proc</structname></link>.oid</literal></entry>
+ <entry>Custom subscripting function with type-specific logic for parsing
+ and validation, or 0 if this type doesn't support subscripting.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>typdefault</structfield> <type>text</type>
diff --git a/doc/src/sgml/extend.sgml b/doc/src/sgml/extend.sgml
index 890ff97b7a..65cf5706b6 100644
--- a/doc/src/sgml/extend.sgml
+++ b/doc/src/sgml/extend.sgml
@@ -33,6 +33,11 @@
operators (starting in <xref linkend="xoper"/>)
</para>
</listitem>
+ <listitem>
+ <para>
+ subscripting procedure (starting in <xref linkend="xsubscripting"/>)
+ </para>
+ </listitem>
<listitem>
<para>
operator classes for indexes (starting in <xref linkend="xindex"/>)
@@ -481,6 +486,7 @@ RETURNS anycompatible AS ...
&xaggr;
&xtypes;
&xoper;
+ &xsubscripting;
&xindex;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 64b5da0070..4c3fd37fc1 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -69,6 +69,7 @@
<!ENTITY xplang SYSTEM "xplang.sgml">
<!ENTITY xoper SYSTEM "xoper.sgml">
<!ENTITY xtypes SYSTEM "xtypes.sgml">
+<!ENTITY xsubscripting SYSTEM "xsubscripting.sgml">
<!ENTITY plperl SYSTEM "plperl.sgml">
<!ENTITY plpython SYSTEM "plpython.sgml">
<!ENTITY plsql SYSTEM "plpgsql.sgml">
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index c0a6554d4d..5c538dca05 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -602,6 +602,45 @@ SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qu
</para>
</sect2>
+ <sect2 id="jsonb-subscripting">
+ <title><type>jsonb</type> Subscripting</title>
+ <para>
+ <type>jsonb</type> data type supports array-style subscripting expressions
+ to extract or update particular elements. It's possible to use multiple
+ subscripting expressions to extract nested values. In this case, a chain of
+ subscripting expressions follows the same rules as the
+ <literal>path</literal> argument in <literal>jsonb_set</literal> function,
+ e.g. in case of arrays it is a 0-based operation or that negative integers
+ that appear in <literal>path</literal> count from the end of JSON arrays.
+ The result of subscripting expressions is always jsonb data type. An
+ example of subscripting syntax:
+<programlisting>
+-- Extract value by key
+SELECT ('{"a": 1}'::jsonb)['a'];
+
+-- Extract nested value by key path
+SELECT ('{"a": {"b": {"c": 1}}}'::jsonb)['a']['b']['c'];
+
+-- Extract element by index
+SELECT ('[1, "2", null]'::jsonb)[1];
+
+-- Update value by key
+UPDATE table_name SET jsonb_field['key'] = 1;
+
+-- Select records using where clause with subscripting. Since the result of
+-- subscripting is jsonb and we basically want to compare two jsonb objects, we
+-- need to put the value in double quotes to be able to convert it to jsonb.
+SELECT * FROM table_name WHERE jsonb_field['key'] = '"value"';
+</programlisting>
+
+ There is no special indexing support for such kind of expressions, but you
+ can always create a functional index that includes it
+<programlisting>
+CREATE INDEX idx ON table_name ((jsonb_field['key']));
+</programlisting>
+ </para>
+ </sect2>
+
<sect2>
<title>Transforms</title>
diff --git a/doc/src/sgml/ref/create_type.sgml b/doc/src/sgml/ref/create_type.sgml
index 111f8e65d2..a34df4d247 100644
--- a/doc/src/sgml/ref/create_type.sgml
+++ b/doc/src/sgml/ref/create_type.sgml
@@ -54,6 +54,7 @@ CREATE TYPE <replaceable class="parameter">name</replaceable> (
[ , ELEMENT = <replaceable class="parameter">element</replaceable> ]
[ , DELIMITER = <replaceable class="parameter">delimiter</replaceable> ]
[ , COLLATABLE = <replaceable class="parameter">collatable</replaceable> ]
+ [ , SUBSCRIPTING_HANDLER = <replaceable class="parameter">subscripting_handler_function</replaceable> ]
)
CREATE TYPE <replaceable class="parameter">name</replaceable>
@@ -196,8 +197,9 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
<replaceable class="parameter">receive_function</replaceable>,
<replaceable class="parameter">send_function</replaceable>,
<replaceable class="parameter">type_modifier_input_function</replaceable>,
- <replaceable class="parameter">type_modifier_output_function</replaceable> and
- <replaceable class="parameter">analyze_function</replaceable>
+ <replaceable class="parameter">type_modifier_output_function</replaceable>,
+ <replaceable class="parameter">analyze_function</replaceable>,
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
are optional. Generally these functions have to be coded in C
or another low-level language.
</para>
@@ -454,6 +456,22 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
make use of the collation information; this does not happen
automatically merely by marking the type collatable.
</para>
+
+ <para>
+ The optional
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
+ contains type-specific logic for subscripting of the data type.
+ By default, there is no such function provided, which means that the data
+ type doesn't support subscripting. The subscripting function must be
+ declared to take a single argument of type <type>internal</type>, and return
+ a <type>internal</type> result. There are two examples of implementation for
+ subscripting functions in case of array
+ (<replaceable class="parameter">array_subscripting_handler</replaceable>)
+ and jsonb
+ (<replaceable class="parameter">jsonb_subscripting_handler</replaceable>)
+ types in <filename>src/backend/utils/adt/arrayfuncs.c</filename> and
+ <filename>src/backend/utils/adt/jsonfuncs.c</filename>, respectively.
+ </para>
</refsect2>
<refsect2>
@@ -769,6 +787,17 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">subscripting_handler_function</replaceable></term>
+ <listitem>
+ <para>
+ The name of a function that returns list of type-specific callback functions to
+ support subscripting logic for the data type.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
diff --git a/doc/src/sgml/xsubscripting.sgml b/doc/src/sgml/xsubscripting.sgml
new file mode 100644
index 0000000000..7224e81fa2
--- /dev/null
+++ b/doc/src/sgml/xsubscripting.sgml
@@ -0,0 +1,111 @@
+<!-- doc/src/sgml/xsubscripting.sgml -->
+
+ <sect1 id="xsubscripting">
+ <title>User-defined subscripting procedure</title>
+
+ <indexterm zone="xsubscripting">
+ <primary>custom subscripting</primary>
+ </indexterm>
+ <para>
+ When you define a new base type, you can also specify a custom procedure to
+ handle subscripting expressions. It must contain logic for verification and
+ evaluation of this expression, i.e. fetching or updating some data in this
+ data type. For instance:
+</para>
+<programlisting><![CDATA[
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = someType;
+ sbsref->refassgntype = someType;
+
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ // some validation and coercion logic
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some assignment logic
+
+ return newContainer;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some fetch logic based on sbstate
+}]]>
+</programlisting>
+
+<para>
+ Then you can define a subscripting procedure and a custom data type:
+</para>
+<programlisting>
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '<replaceable>filename</replaceable>'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 4,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler,
+);
+</programlisting>
+
+<para>
+ and use it as usual:
+</para>
+<programlisting>
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
+</programlisting>
+
+
+ <para>
+ The examples of custom subscripting implementation can be found in
+ <filename>subscripting.sql</filename> and <filename>subscripting.c</filename>
+ in the <filename>src/tutorial</filename> directory of the source distribution.
+ See the <filename>README</filename> file in that directory for instructions
+ about running the examples.
+ </para>
+
+</sect1>
diff --git a/src/tutorial/Makefile b/src/tutorial/Makefile
index 16dc390f71..0ead60c2d4 100644
--- a/src/tutorial/Makefile
+++ b/src/tutorial/Makefile
@@ -13,8 +13,8 @@
#
#-------------------------------------------------------------------------
-MODULES = complex funcs
-DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql
+MODULES = complex funcs subscripting
+DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql subscripting.sql
ifdef NO_PGXS
subdir = src/tutorial
diff --git a/src/tutorial/subscripting.c b/src/tutorial/subscripting.c
new file mode 100644
index 0000000000..1eb8c45652
--- /dev/null
+++ b/src/tutorial/subscripting.c
@@ -0,0 +1,201 @@
+/*
+ * src/tutorial/subscripting.c
+ *
+ ******************************************************************************
+ This file contains routines that can be bound to a Postgres backend and
+ called by the backend in the process of processing queries. The calling
+ format for these routines is dictated by Postgres architecture.
+******************************************************************************/
+
+#include "postgres.h"
+
+#include "catalog/pg_type.h"
+#include "executor/executor.h"
+#include "executor/execExpr.h"
+#include "nodes/nodeFuncs.h"
+#include "parser/parse_coerce.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+SubscriptingRef * custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref);
+SubscriptingRef * custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate);
+Datum custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate);
+Datum custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate);
+
+PG_FUNCTION_INFO_V1(custom_in);
+PG_FUNCTION_INFO_V1(custom_out);
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+/*****************************************************************************
+ * Input/Output functions
+ *****************************************************************************/
+
+Datum
+custom_in(PG_FUNCTION_ARGS)
+{
+ char *str = PG_GETARG_CSTRING(0);
+ int firstValue,
+ secondValue;
+ Custom *result;
+
+ if (sscanf(str, " ( %d , %d )", &firstValue, &secondValue) != 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
+ errmsg("invalid input syntax for complex: \"%s\"",
+ str)));
+
+
+ result = (Custom *) palloc(sizeof(Custom));
+ result->first = firstValue;
+ result->second = secondValue;
+ PG_RETURN_POINTER(result);
+}
+
+Datum
+custom_out(PG_FUNCTION_ARGS)
+{
+ Custom *custom = (Custom *) PG_GETARG_POINTER(0);
+ char *result;
+
+ result = psprintf("(%d, %d)", custom->first, custom->second);
+ PG_RETURN_CSTRING(result);
+}
+
+/*****************************************************************************
+ * Custom subscripting logic functions
+ *****************************************************************************/
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = INT4OID;
+ sbsref->refassgntype = INT4OID;
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ List *upperIndexpr = NIL;
+ ListCell *l;
+
+ if (sbsref->reflowerindexpr != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *)lfirst(sbsref->reflowerindexpr->head))))));
+
+ foreach(l, sbsref->refupperindexpr)
+ {
+ Node *subexpr = (Node *) lfirst(l);
+
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *) lfirst(sbsref->refupperindexpr->head))))));
+
+ subexpr = coerce_to_target_type(pstate,
+ subexpr, exprType(subexpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript must have integer type"),
+ parser_errposition(pstate, exprLocation(subexpr))));
+
+ upperIndexpr = lappend(upperIndexpr, subexpr);
+
+ if (isAssignment)
+ {
+ Node *assignExpr = (Node *) sbsref->refassgnexpr;
+ Node *new_from;
+
+ new_from = coerce_to_target_type(pstate,
+ assignExpr, exprType(assignExpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (new_from == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom assignment requires int type"),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation(assignExpr))));
+ sbsref->refassgnexpr = (Expr *)new_from;
+ }
+ }
+
+ sbsref->refupperindexpr = upperIndexpr;
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ Custom *container= (Custom *) containerSource;
+ int index;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ return (Datum) container->first;
+ else
+ return (Datum) container->second;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ int index;
+ Custom *container = (Custom *) containerSource;
+
+ if (sbstate->resnull)
+ return containerSource;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ container->first = DatumGetInt32(sbstate->replacevalue);
+ else
+ container->second = DatumGetInt32(sbstate->replacevalue);
+
+ return (Datum) container;
+}
diff --git a/src/tutorial/subscripting.source b/src/tutorial/subscripting.source
new file mode 100644
index 0000000000..837cf30612
--- /dev/null
+++ b/src/tutorial/subscripting.source
@@ -0,0 +1,71 @@
+---------------------------------------------------------------------------
+--
+-- subscripting.sql-
+-- This file shows how to create a new subscripting procedure for
+-- user-defined type.
+--
+--
+-- Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+-- Portions Copyright (c) 1994, Regents of the University of California
+--
+-- src/tutorial/subscripting.source
+--
+---------------------------------------------------------------------------
+
+-----------------------------
+-- Creating a new type:
+-- We are going to create a new type called 'complex' which represents
+-- complex numbers.
+-- A user-defined type must have an input and an output function, and
+-- optionally can have binary input and output functions. All of these
+-- are usually user-defined C functions.
+-----------------------------
+
+-- Assume the user defined functions are in /home/erthalion/programms/postgresql-master/src/tutorial/complex$DLSUFFIX
+-- (we do not want to assume this is in the dynamic loader search path).
+-- Look at $PWD/complex.c for the source. Note that we declare all of
+-- them as STRICT, so we do not need to cope with NULL inputs in the
+-- C code. We also mark them IMMUTABLE, since they always return the
+-- same outputs given the same inputs.
+
+-- the input function 'complex_in' takes a null-terminated string (the
+-- textual representation of the type) and turns it into the internal
+-- (in memory) representation. You will get a message telling you 'complex'
+-- does not exist yet but that's okay.
+
+CREATE FUNCTION custom_in(cstring)
+ RETURNS custom
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+-- the output function 'complex_out' takes the internal representation and
+-- converts it into the textual representation.
+
+CREATE FUNCTION custom_out(custom)
+ RETURNS cstring
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 8,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler
+);
+
+-- we can use it in a table
+
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
--
2.21.0
--u66dlpg6gaevsofr
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v33-0005-Filling-gaps-in-jsonb-arrays.patch"
^ permalink raw reply [nested|flat] 55+ messages in thread
* [PATCH v32 4/6] Subscripting documentation
@ 2019-02-01 10:47 erthalion <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: erthalion @ 2019-02-01 10:47 UTC (permalink / raw)
Supporting documentation for generalized subscripting. It includes the
description of a new field in pg_type, the new section for jsonb
documentation about subscripting feature on this data type, and also the
tutorial about how to write subscripting operator for a custom data type.
Reviewed-by: Tom Lane, Arthur Zakirov, Oleksandr Shulgin
---
doc/src/sgml/catalogs.sgml | 8 ++
doc/src/sgml/extend.sgml | 6 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/json.sgml | 39 ++++++
doc/src/sgml/ref/create_type.sgml | 33 ++++-
doc/src/sgml/xsubscripting.sgml | 111 +++++++++++++++++
src/tutorial/Makefile | 4 +-
src/tutorial/subscripting.c | 201 ++++++++++++++++++++++++++++++
src/tutorial/subscripting.source | 71 +++++++++++
9 files changed, 470 insertions(+), 4 deletions(-)
create mode 100644 doc/src/sgml/xsubscripting.sgml
create mode 100644 src/tutorial/subscripting.c
create mode 100644 src/tutorial/subscripting.source
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 048ff284f7..b17e9ee382 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8971,6 +8971,14 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>typsubshandler</structfield></entry>
+ <entry><type>regproc</type></entry>
+ <entry><literal><link linkend="catalog-pg-proc"><structname>pg_proc</structname></link>.oid</literal></entry>
+ <entry>Custom subscripting function with type-specific logic for parsing
+ and validation, or 0 if this type doesn't support subscripting.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>typdefault</structfield> <type>text</type>
diff --git a/doc/src/sgml/extend.sgml b/doc/src/sgml/extend.sgml
index 890ff97b7a..65cf5706b6 100644
--- a/doc/src/sgml/extend.sgml
+++ b/doc/src/sgml/extend.sgml
@@ -33,6 +33,11 @@
operators (starting in <xref linkend="xoper"/>)
</para>
</listitem>
+ <listitem>
+ <para>
+ subscripting procedure (starting in <xref linkend="xsubscripting"/>)
+ </para>
+ </listitem>
<listitem>
<para>
operator classes for indexes (starting in <xref linkend="xindex"/>)
@@ -481,6 +486,7 @@ RETURNS anycompatible AS ...
&xaggr;
&xtypes;
&xoper;
+ &xsubscripting;
&xindex;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 64b5da0070..4c3fd37fc1 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -69,6 +69,7 @@
<!ENTITY xplang SYSTEM "xplang.sgml">
<!ENTITY xoper SYSTEM "xoper.sgml">
<!ENTITY xtypes SYSTEM "xtypes.sgml">
+<!ENTITY xsubscripting SYSTEM "xsubscripting.sgml">
<!ENTITY plperl SYSTEM "plperl.sgml">
<!ENTITY plpython SYSTEM "plpython.sgml">
<!ENTITY plsql SYSTEM "plpgsql.sgml">
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index c0a6554d4d..5c538dca05 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -602,6 +602,45 @@ SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qu
</para>
</sect2>
+ <sect2 id="jsonb-subscripting">
+ <title><type>jsonb</type> Subscripting</title>
+ <para>
+ <type>jsonb</type> data type supports array-style subscripting expressions
+ to extract or update particular elements. It's possible to use multiple
+ subscripting expressions to extract nested values. In this case, a chain of
+ subscripting expressions follows the same rules as the
+ <literal>path</literal> argument in <literal>jsonb_set</literal> function,
+ e.g. in case of arrays it is a 0-based operation or that negative integers
+ that appear in <literal>path</literal> count from the end of JSON arrays.
+ The result of subscripting expressions is always jsonb data type. An
+ example of subscripting syntax:
+<programlisting>
+-- Extract value by key
+SELECT ('{"a": 1}'::jsonb)['a'];
+
+-- Extract nested value by key path
+SELECT ('{"a": {"b": {"c": 1}}}'::jsonb)['a']['b']['c'];
+
+-- Extract element by index
+SELECT ('[1, "2", null]'::jsonb)[1];
+
+-- Update value by key
+UPDATE table_name SET jsonb_field['key'] = 1;
+
+-- Select records using where clause with subscripting. Since the result of
+-- subscripting is jsonb and we basically want to compare two jsonb objects, we
+-- need to put the value in double quotes to be able to convert it to jsonb.
+SELECT * FROM table_name WHERE jsonb_field['key'] = '"value"';
+</programlisting>
+
+ There is no special indexing support for such kind of expressions, but you
+ can always create a functional index that includes it
+<programlisting>
+CREATE INDEX idx ON table_name ((jsonb_field['key']));
+</programlisting>
+ </para>
+ </sect2>
+
<sect2>
<title>Transforms</title>
diff --git a/doc/src/sgml/ref/create_type.sgml b/doc/src/sgml/ref/create_type.sgml
index 111f8e65d2..a34df4d247 100644
--- a/doc/src/sgml/ref/create_type.sgml
+++ b/doc/src/sgml/ref/create_type.sgml
@@ -54,6 +54,7 @@ CREATE TYPE <replaceable class="parameter">name</replaceable> (
[ , ELEMENT = <replaceable class="parameter">element</replaceable> ]
[ , DELIMITER = <replaceable class="parameter">delimiter</replaceable> ]
[ , COLLATABLE = <replaceable class="parameter">collatable</replaceable> ]
+ [ , SUBSCRIPTING_HANDLER = <replaceable class="parameter">subscripting_handler_function</replaceable> ]
)
CREATE TYPE <replaceable class="parameter">name</replaceable>
@@ -196,8 +197,9 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
<replaceable class="parameter">receive_function</replaceable>,
<replaceable class="parameter">send_function</replaceable>,
<replaceable class="parameter">type_modifier_input_function</replaceable>,
- <replaceable class="parameter">type_modifier_output_function</replaceable> and
- <replaceable class="parameter">analyze_function</replaceable>
+ <replaceable class="parameter">type_modifier_output_function</replaceable>,
+ <replaceable class="parameter">analyze_function</replaceable>,
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
are optional. Generally these functions have to be coded in C
or another low-level language.
</para>
@@ -454,6 +456,22 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
make use of the collation information; this does not happen
automatically merely by marking the type collatable.
</para>
+
+ <para>
+ The optional
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
+ contains type-specific logic for subscripting of the data type.
+ By default, there is no such function provided, which means that the data
+ type doesn't support subscripting. The subscripting function must be
+ declared to take a single argument of type <type>internal</type>, and return
+ a <type>internal</type> result. There are two examples of implementation for
+ subscripting functions in case of array
+ (<replaceable class="parameter">array_subscripting_handler</replaceable>)
+ and jsonb
+ (<replaceable class="parameter">jsonb_subscripting_handler</replaceable>)
+ types in <filename>src/backend/utils/adt/arrayfuncs.c</filename> and
+ <filename>src/backend/utils/adt/jsonfuncs.c</filename>, respectively.
+ </para>
</refsect2>
<refsect2>
@@ -769,6 +787,17 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">subscripting_handler_function</replaceable></term>
+ <listitem>
+ <para>
+ The name of a function that returns list of type-specific callback functions to
+ support subscripting logic for the data type.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
diff --git a/doc/src/sgml/xsubscripting.sgml b/doc/src/sgml/xsubscripting.sgml
new file mode 100644
index 0000000000..7224e81fa2
--- /dev/null
+++ b/doc/src/sgml/xsubscripting.sgml
@@ -0,0 +1,111 @@
+<!-- doc/src/sgml/xsubscripting.sgml -->
+
+ <sect1 id="xsubscripting">
+ <title>User-defined subscripting procedure</title>
+
+ <indexterm zone="xsubscripting">
+ <primary>custom subscripting</primary>
+ </indexterm>
+ <para>
+ When you define a new base type, you can also specify a custom procedure to
+ handle subscripting expressions. It must contain logic for verification and
+ evaluation of this expression, i.e. fetching or updating some data in this
+ data type. For instance:
+</para>
+<programlisting><![CDATA[
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = someType;
+ sbsref->refassgntype = someType;
+
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ // some validation and coercion logic
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some assignment logic
+
+ return newContainer;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some fetch logic based on sbstate
+}]]>
+</programlisting>
+
+<para>
+ Then you can define a subscripting procedure and a custom data type:
+</para>
+<programlisting>
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '<replaceable>filename</replaceable>'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 4,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler,
+);
+</programlisting>
+
+<para>
+ and use it as usual:
+</para>
+<programlisting>
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
+</programlisting>
+
+
+ <para>
+ The examples of custom subscripting implementation can be found in
+ <filename>subscripting.sql</filename> and <filename>subscripting.c</filename>
+ in the <filename>src/tutorial</filename> directory of the source distribution.
+ See the <filename>README</filename> file in that directory for instructions
+ about running the examples.
+ </para>
+
+</sect1>
diff --git a/src/tutorial/Makefile b/src/tutorial/Makefile
index 16dc390f71..0ead60c2d4 100644
--- a/src/tutorial/Makefile
+++ b/src/tutorial/Makefile
@@ -13,8 +13,8 @@
#
#-------------------------------------------------------------------------
-MODULES = complex funcs
-DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql
+MODULES = complex funcs subscripting
+DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql subscripting.sql
ifdef NO_PGXS
subdir = src/tutorial
diff --git a/src/tutorial/subscripting.c b/src/tutorial/subscripting.c
new file mode 100644
index 0000000000..1eb8c45652
--- /dev/null
+++ b/src/tutorial/subscripting.c
@@ -0,0 +1,201 @@
+/*
+ * src/tutorial/subscripting.c
+ *
+ ******************************************************************************
+ This file contains routines that can be bound to a Postgres backend and
+ called by the backend in the process of processing queries. The calling
+ format for these routines is dictated by Postgres architecture.
+******************************************************************************/
+
+#include "postgres.h"
+
+#include "catalog/pg_type.h"
+#include "executor/executor.h"
+#include "executor/execExpr.h"
+#include "nodes/nodeFuncs.h"
+#include "parser/parse_coerce.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+SubscriptingRef * custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref);
+SubscriptingRef * custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate);
+Datum custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate);
+Datum custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate);
+
+PG_FUNCTION_INFO_V1(custom_in);
+PG_FUNCTION_INFO_V1(custom_out);
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+/*****************************************************************************
+ * Input/Output functions
+ *****************************************************************************/
+
+Datum
+custom_in(PG_FUNCTION_ARGS)
+{
+ char *str = PG_GETARG_CSTRING(0);
+ int firstValue,
+ secondValue;
+ Custom *result;
+
+ if (sscanf(str, " ( %d , %d )", &firstValue, &secondValue) != 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
+ errmsg("invalid input syntax for complex: \"%s\"",
+ str)));
+
+
+ result = (Custom *) palloc(sizeof(Custom));
+ result->first = firstValue;
+ result->second = secondValue;
+ PG_RETURN_POINTER(result);
+}
+
+Datum
+custom_out(PG_FUNCTION_ARGS)
+{
+ Custom *custom = (Custom *) PG_GETARG_POINTER(0);
+ char *result;
+
+ result = psprintf("(%d, %d)", custom->first, custom->second);
+ PG_RETURN_CSTRING(result);
+}
+
+/*****************************************************************************
+ * Custom subscripting logic functions
+ *****************************************************************************/
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = INT4OID;
+ sbsref->refassgntype = INT4OID;
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ List *upperIndexpr = NIL;
+ ListCell *l;
+
+ if (sbsref->reflowerindexpr != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *)lfirst(sbsref->reflowerindexpr->head))))));
+
+ foreach(l, sbsref->refupperindexpr)
+ {
+ Node *subexpr = (Node *) lfirst(l);
+
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *) lfirst(sbsref->refupperindexpr->head))))));
+
+ subexpr = coerce_to_target_type(pstate,
+ subexpr, exprType(subexpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript must have integer type"),
+ parser_errposition(pstate, exprLocation(subexpr))));
+
+ upperIndexpr = lappend(upperIndexpr, subexpr);
+
+ if (isAssignment)
+ {
+ Node *assignExpr = (Node *) sbsref->refassgnexpr;
+ Node *new_from;
+
+ new_from = coerce_to_target_type(pstate,
+ assignExpr, exprType(assignExpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (new_from == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom assignment requires int type"),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation(assignExpr))));
+ sbsref->refassgnexpr = (Expr *)new_from;
+ }
+ }
+
+ sbsref->refupperindexpr = upperIndexpr;
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ Custom *container= (Custom *) containerSource;
+ int index;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ return (Datum) container->first;
+ else
+ return (Datum) container->second;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ int index;
+ Custom *container = (Custom *) containerSource;
+
+ if (sbstate->resnull)
+ return containerSource;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ container->first = DatumGetInt32(sbstate->replacevalue);
+ else
+ container->second = DatumGetInt32(sbstate->replacevalue);
+
+ return (Datum) container;
+}
diff --git a/src/tutorial/subscripting.source b/src/tutorial/subscripting.source
new file mode 100644
index 0000000000..837cf30612
--- /dev/null
+++ b/src/tutorial/subscripting.source
@@ -0,0 +1,71 @@
+---------------------------------------------------------------------------
+--
+-- subscripting.sql-
+-- This file shows how to create a new subscripting procedure for
+-- user-defined type.
+--
+--
+-- Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+-- Portions Copyright (c) 1994, Regents of the University of California
+--
+-- src/tutorial/subscripting.source
+--
+---------------------------------------------------------------------------
+
+-----------------------------
+-- Creating a new type:
+-- We are going to create a new type called 'complex' which represents
+-- complex numbers.
+-- A user-defined type must have an input and an output function, and
+-- optionally can have binary input and output functions. All of these
+-- are usually user-defined C functions.
+-----------------------------
+
+-- Assume the user defined functions are in /home/erthalion/programms/postgresql-master/src/tutorial/complex$DLSUFFIX
+-- (we do not want to assume this is in the dynamic loader search path).
+-- Look at $PWD/complex.c for the source. Note that we declare all of
+-- them as STRICT, so we do not need to cope with NULL inputs in the
+-- C code. We also mark them IMMUTABLE, since they always return the
+-- same outputs given the same inputs.
+
+-- the input function 'complex_in' takes a null-terminated string (the
+-- textual representation of the type) and turns it into the internal
+-- (in memory) representation. You will get a message telling you 'complex'
+-- does not exist yet but that's okay.
+
+CREATE FUNCTION custom_in(cstring)
+ RETURNS custom
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+-- the output function 'complex_out' takes the internal representation and
+-- converts it into the textual representation.
+
+CREATE FUNCTION custom_out(custom)
+ RETURNS cstring
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 8,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler
+);
+
+-- we can use it in a table
+
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
--
2.21.0
--xuxpg5p32zizz4zy
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v32-0005-Polymorphic-subscripting.patch"
^ permalink raw reply [nested|flat] 55+ messages in thread
* [PATCH v33 4/5] Subscripting documentation
@ 2019-02-01 10:47 erthalion <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: erthalion @ 2019-02-01 10:47 UTC (permalink / raw)
Supporting documentation for generalized subscripting. It includes the
description of a new field in pg_type, the new section for jsonb
documentation about subscripting feature on this data type, and also the
tutorial about how to write subscripting operator for a custom data type.
Reviewed-by: Tom Lane, Arthur Zakirov, Oleksandr Shulgin
---
doc/src/sgml/catalogs.sgml | 8 ++
doc/src/sgml/extend.sgml | 6 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/json.sgml | 39 ++++++
doc/src/sgml/ref/create_type.sgml | 33 ++++-
doc/src/sgml/xsubscripting.sgml | 111 +++++++++++++++++
src/tutorial/Makefile | 4 +-
src/tutorial/subscripting.c | 201 ++++++++++++++++++++++++++++++
src/tutorial/subscripting.source | 71 +++++++++++
9 files changed, 470 insertions(+), 4 deletions(-)
create mode 100644 doc/src/sgml/xsubscripting.sgml
create mode 100644 src/tutorial/subscripting.c
create mode 100644 src/tutorial/subscripting.source
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 048ff284f7..b17e9ee382 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8971,6 +8971,14 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>typsubshandler</structfield></entry>
+ <entry><type>regproc</type></entry>
+ <entry><literal><link linkend="catalog-pg-proc"><structname>pg_proc</structname></link>.oid</literal></entry>
+ <entry>Custom subscripting function with type-specific logic for parsing
+ and validation, or 0 if this type doesn't support subscripting.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>typdefault</structfield> <type>text</type>
diff --git a/doc/src/sgml/extend.sgml b/doc/src/sgml/extend.sgml
index 890ff97b7a..65cf5706b6 100644
--- a/doc/src/sgml/extend.sgml
+++ b/doc/src/sgml/extend.sgml
@@ -33,6 +33,11 @@
operators (starting in <xref linkend="xoper"/>)
</para>
</listitem>
+ <listitem>
+ <para>
+ subscripting procedure (starting in <xref linkend="xsubscripting"/>)
+ </para>
+ </listitem>
<listitem>
<para>
operator classes for indexes (starting in <xref linkend="xindex"/>)
@@ -481,6 +486,7 @@ RETURNS anycompatible AS ...
&xaggr;
&xtypes;
&xoper;
+ &xsubscripting;
&xindex;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 64b5da0070..4c3fd37fc1 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -69,6 +69,7 @@
<!ENTITY xplang SYSTEM "xplang.sgml">
<!ENTITY xoper SYSTEM "xoper.sgml">
<!ENTITY xtypes SYSTEM "xtypes.sgml">
+<!ENTITY xsubscripting SYSTEM "xsubscripting.sgml">
<!ENTITY plperl SYSTEM "plperl.sgml">
<!ENTITY plpython SYSTEM "plpython.sgml">
<!ENTITY plsql SYSTEM "plpgsql.sgml">
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index c0a6554d4d..5c538dca05 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -602,6 +602,45 @@ SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qu
</para>
</sect2>
+ <sect2 id="jsonb-subscripting">
+ <title><type>jsonb</type> Subscripting</title>
+ <para>
+ <type>jsonb</type> data type supports array-style subscripting expressions
+ to extract or update particular elements. It's possible to use multiple
+ subscripting expressions to extract nested values. In this case, a chain of
+ subscripting expressions follows the same rules as the
+ <literal>path</literal> argument in <literal>jsonb_set</literal> function,
+ e.g. in case of arrays it is a 0-based operation or that negative integers
+ that appear in <literal>path</literal> count from the end of JSON arrays.
+ The result of subscripting expressions is always jsonb data type. An
+ example of subscripting syntax:
+<programlisting>
+-- Extract value by key
+SELECT ('{"a": 1}'::jsonb)['a'];
+
+-- Extract nested value by key path
+SELECT ('{"a": {"b": {"c": 1}}}'::jsonb)['a']['b']['c'];
+
+-- Extract element by index
+SELECT ('[1, "2", null]'::jsonb)[1];
+
+-- Update value by key
+UPDATE table_name SET jsonb_field['key'] = 1;
+
+-- Select records using where clause with subscripting. Since the result of
+-- subscripting is jsonb and we basically want to compare two jsonb objects, we
+-- need to put the value in double quotes to be able to convert it to jsonb.
+SELECT * FROM table_name WHERE jsonb_field['key'] = '"value"';
+</programlisting>
+
+ There is no special indexing support for such kind of expressions, but you
+ can always create a functional index that includes it
+<programlisting>
+CREATE INDEX idx ON table_name ((jsonb_field['key']));
+</programlisting>
+ </para>
+ </sect2>
+
<sect2>
<title>Transforms</title>
diff --git a/doc/src/sgml/ref/create_type.sgml b/doc/src/sgml/ref/create_type.sgml
index 111f8e65d2..a34df4d247 100644
--- a/doc/src/sgml/ref/create_type.sgml
+++ b/doc/src/sgml/ref/create_type.sgml
@@ -54,6 +54,7 @@ CREATE TYPE <replaceable class="parameter">name</replaceable> (
[ , ELEMENT = <replaceable class="parameter">element</replaceable> ]
[ , DELIMITER = <replaceable class="parameter">delimiter</replaceable> ]
[ , COLLATABLE = <replaceable class="parameter">collatable</replaceable> ]
+ [ , SUBSCRIPTING_HANDLER = <replaceable class="parameter">subscripting_handler_function</replaceable> ]
)
CREATE TYPE <replaceable class="parameter">name</replaceable>
@@ -196,8 +197,9 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
<replaceable class="parameter">receive_function</replaceable>,
<replaceable class="parameter">send_function</replaceable>,
<replaceable class="parameter">type_modifier_input_function</replaceable>,
- <replaceable class="parameter">type_modifier_output_function</replaceable> and
- <replaceable class="parameter">analyze_function</replaceable>
+ <replaceable class="parameter">type_modifier_output_function</replaceable>,
+ <replaceable class="parameter">analyze_function</replaceable>,
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
are optional. Generally these functions have to be coded in C
or another low-level language.
</para>
@@ -454,6 +456,22 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
make use of the collation information; this does not happen
automatically merely by marking the type collatable.
</para>
+
+ <para>
+ The optional
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
+ contains type-specific logic for subscripting of the data type.
+ By default, there is no such function provided, which means that the data
+ type doesn't support subscripting. The subscripting function must be
+ declared to take a single argument of type <type>internal</type>, and return
+ a <type>internal</type> result. There are two examples of implementation for
+ subscripting functions in case of array
+ (<replaceable class="parameter">array_subscripting_handler</replaceable>)
+ and jsonb
+ (<replaceable class="parameter">jsonb_subscripting_handler</replaceable>)
+ types in <filename>src/backend/utils/adt/arrayfuncs.c</filename> and
+ <filename>src/backend/utils/adt/jsonfuncs.c</filename>, respectively.
+ </para>
</refsect2>
<refsect2>
@@ -769,6 +787,17 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">subscripting_handler_function</replaceable></term>
+ <listitem>
+ <para>
+ The name of a function that returns list of type-specific callback functions to
+ support subscripting logic for the data type.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
diff --git a/doc/src/sgml/xsubscripting.sgml b/doc/src/sgml/xsubscripting.sgml
new file mode 100644
index 0000000000..7224e81fa2
--- /dev/null
+++ b/doc/src/sgml/xsubscripting.sgml
@@ -0,0 +1,111 @@
+<!-- doc/src/sgml/xsubscripting.sgml -->
+
+ <sect1 id="xsubscripting">
+ <title>User-defined subscripting procedure</title>
+
+ <indexterm zone="xsubscripting">
+ <primary>custom subscripting</primary>
+ </indexterm>
+ <para>
+ When you define a new base type, you can also specify a custom procedure to
+ handle subscripting expressions. It must contain logic for verification and
+ evaluation of this expression, i.e. fetching or updating some data in this
+ data type. For instance:
+</para>
+<programlisting><![CDATA[
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = someType;
+ sbsref->refassgntype = someType;
+
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ // some validation and coercion logic
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some assignment logic
+
+ return newContainer;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some fetch logic based on sbstate
+}]]>
+</programlisting>
+
+<para>
+ Then you can define a subscripting procedure and a custom data type:
+</para>
+<programlisting>
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '<replaceable>filename</replaceable>'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 4,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler,
+);
+</programlisting>
+
+<para>
+ and use it as usual:
+</para>
+<programlisting>
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
+</programlisting>
+
+
+ <para>
+ The examples of custom subscripting implementation can be found in
+ <filename>subscripting.sql</filename> and <filename>subscripting.c</filename>
+ in the <filename>src/tutorial</filename> directory of the source distribution.
+ See the <filename>README</filename> file in that directory for instructions
+ about running the examples.
+ </para>
+
+</sect1>
diff --git a/src/tutorial/Makefile b/src/tutorial/Makefile
index 16dc390f71..0ead60c2d4 100644
--- a/src/tutorial/Makefile
+++ b/src/tutorial/Makefile
@@ -13,8 +13,8 @@
#
#-------------------------------------------------------------------------
-MODULES = complex funcs
-DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql
+MODULES = complex funcs subscripting
+DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql subscripting.sql
ifdef NO_PGXS
subdir = src/tutorial
diff --git a/src/tutorial/subscripting.c b/src/tutorial/subscripting.c
new file mode 100644
index 0000000000..1eb8c45652
--- /dev/null
+++ b/src/tutorial/subscripting.c
@@ -0,0 +1,201 @@
+/*
+ * src/tutorial/subscripting.c
+ *
+ ******************************************************************************
+ This file contains routines that can be bound to a Postgres backend and
+ called by the backend in the process of processing queries. The calling
+ format for these routines is dictated by Postgres architecture.
+******************************************************************************/
+
+#include "postgres.h"
+
+#include "catalog/pg_type.h"
+#include "executor/executor.h"
+#include "executor/execExpr.h"
+#include "nodes/nodeFuncs.h"
+#include "parser/parse_coerce.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+SubscriptingRef * custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref);
+SubscriptingRef * custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate);
+Datum custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate);
+Datum custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate);
+
+PG_FUNCTION_INFO_V1(custom_in);
+PG_FUNCTION_INFO_V1(custom_out);
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+/*****************************************************************************
+ * Input/Output functions
+ *****************************************************************************/
+
+Datum
+custom_in(PG_FUNCTION_ARGS)
+{
+ char *str = PG_GETARG_CSTRING(0);
+ int firstValue,
+ secondValue;
+ Custom *result;
+
+ if (sscanf(str, " ( %d , %d )", &firstValue, &secondValue) != 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
+ errmsg("invalid input syntax for complex: \"%s\"",
+ str)));
+
+
+ result = (Custom *) palloc(sizeof(Custom));
+ result->first = firstValue;
+ result->second = secondValue;
+ PG_RETURN_POINTER(result);
+}
+
+Datum
+custom_out(PG_FUNCTION_ARGS)
+{
+ Custom *custom = (Custom *) PG_GETARG_POINTER(0);
+ char *result;
+
+ result = psprintf("(%d, %d)", custom->first, custom->second);
+ PG_RETURN_CSTRING(result);
+}
+
+/*****************************************************************************
+ * Custom subscripting logic functions
+ *****************************************************************************/
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = INT4OID;
+ sbsref->refassgntype = INT4OID;
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ List *upperIndexpr = NIL;
+ ListCell *l;
+
+ if (sbsref->reflowerindexpr != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *)lfirst(sbsref->reflowerindexpr->head))))));
+
+ foreach(l, sbsref->refupperindexpr)
+ {
+ Node *subexpr = (Node *) lfirst(l);
+
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *) lfirst(sbsref->refupperindexpr->head))))));
+
+ subexpr = coerce_to_target_type(pstate,
+ subexpr, exprType(subexpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript must have integer type"),
+ parser_errposition(pstate, exprLocation(subexpr))));
+
+ upperIndexpr = lappend(upperIndexpr, subexpr);
+
+ if (isAssignment)
+ {
+ Node *assignExpr = (Node *) sbsref->refassgnexpr;
+ Node *new_from;
+
+ new_from = coerce_to_target_type(pstate,
+ assignExpr, exprType(assignExpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (new_from == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom assignment requires int type"),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation(assignExpr))));
+ sbsref->refassgnexpr = (Expr *)new_from;
+ }
+ }
+
+ sbsref->refupperindexpr = upperIndexpr;
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ Custom *container= (Custom *) containerSource;
+ int index;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ return (Datum) container->first;
+ else
+ return (Datum) container->second;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ int index;
+ Custom *container = (Custom *) containerSource;
+
+ if (sbstate->resnull)
+ return containerSource;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ container->first = DatumGetInt32(sbstate->replacevalue);
+ else
+ container->second = DatumGetInt32(sbstate->replacevalue);
+
+ return (Datum) container;
+}
diff --git a/src/tutorial/subscripting.source b/src/tutorial/subscripting.source
new file mode 100644
index 0000000000..837cf30612
--- /dev/null
+++ b/src/tutorial/subscripting.source
@@ -0,0 +1,71 @@
+---------------------------------------------------------------------------
+--
+-- subscripting.sql-
+-- This file shows how to create a new subscripting procedure for
+-- user-defined type.
+--
+--
+-- Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+-- Portions Copyright (c) 1994, Regents of the University of California
+--
+-- src/tutorial/subscripting.source
+--
+---------------------------------------------------------------------------
+
+-----------------------------
+-- Creating a new type:
+-- We are going to create a new type called 'complex' which represents
+-- complex numbers.
+-- A user-defined type must have an input and an output function, and
+-- optionally can have binary input and output functions. All of these
+-- are usually user-defined C functions.
+-----------------------------
+
+-- Assume the user defined functions are in /home/erthalion/programms/postgresql-master/src/tutorial/complex$DLSUFFIX
+-- (we do not want to assume this is in the dynamic loader search path).
+-- Look at $PWD/complex.c for the source. Note that we declare all of
+-- them as STRICT, so we do not need to cope with NULL inputs in the
+-- C code. We also mark them IMMUTABLE, since they always return the
+-- same outputs given the same inputs.
+
+-- the input function 'complex_in' takes a null-terminated string (the
+-- textual representation of the type) and turns it into the internal
+-- (in memory) representation. You will get a message telling you 'complex'
+-- does not exist yet but that's okay.
+
+CREATE FUNCTION custom_in(cstring)
+ RETURNS custom
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+-- the output function 'complex_out' takes the internal representation and
+-- converts it into the textual representation.
+
+CREATE FUNCTION custom_out(custom)
+ RETURNS cstring
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 8,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler
+);
+
+-- we can use it in a table
+
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
--
2.21.0
--u66dlpg6gaevsofr
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v33-0005-Filling-gaps-in-jsonb-arrays.patch"
^ permalink raw reply [nested|flat] 55+ messages in thread
* [PATCH v33 4/5] Subscripting documentation
@ 2019-02-01 10:47 erthalion <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: erthalion @ 2019-02-01 10:47 UTC (permalink / raw)
Supporting documentation for generalized subscripting. It includes the
description of a new field in pg_type, the new section for jsonb
documentation about subscripting feature on this data type, and also the
tutorial about how to write subscripting operator for a custom data type.
Reviewed-by: Tom Lane, Arthur Zakirov, Oleksandr Shulgin
---
doc/src/sgml/catalogs.sgml | 8 ++
doc/src/sgml/extend.sgml | 6 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/json.sgml | 39 ++++++
doc/src/sgml/ref/create_type.sgml | 33 ++++-
doc/src/sgml/xsubscripting.sgml | 111 +++++++++++++++++
src/tutorial/Makefile | 4 +-
src/tutorial/subscripting.c | 201 ++++++++++++++++++++++++++++++
src/tutorial/subscripting.source | 71 +++++++++++
9 files changed, 470 insertions(+), 4 deletions(-)
create mode 100644 doc/src/sgml/xsubscripting.sgml
create mode 100644 src/tutorial/subscripting.c
create mode 100644 src/tutorial/subscripting.source
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 048ff284f7..b17e9ee382 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8971,6 +8971,14 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>typsubshandler</structfield></entry>
+ <entry><type>regproc</type></entry>
+ <entry><literal><link linkend="catalog-pg-proc"><structname>pg_proc</structname></link>.oid</literal></entry>
+ <entry>Custom subscripting function with type-specific logic for parsing
+ and validation, or 0 if this type doesn't support subscripting.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>typdefault</structfield> <type>text</type>
diff --git a/doc/src/sgml/extend.sgml b/doc/src/sgml/extend.sgml
index 890ff97b7a..65cf5706b6 100644
--- a/doc/src/sgml/extend.sgml
+++ b/doc/src/sgml/extend.sgml
@@ -33,6 +33,11 @@
operators (starting in <xref linkend="xoper"/>)
</para>
</listitem>
+ <listitem>
+ <para>
+ subscripting procedure (starting in <xref linkend="xsubscripting"/>)
+ </para>
+ </listitem>
<listitem>
<para>
operator classes for indexes (starting in <xref linkend="xindex"/>)
@@ -481,6 +486,7 @@ RETURNS anycompatible AS ...
&xaggr;
&xtypes;
&xoper;
+ &xsubscripting;
&xindex;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 64b5da0070..4c3fd37fc1 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -69,6 +69,7 @@
<!ENTITY xplang SYSTEM "xplang.sgml">
<!ENTITY xoper SYSTEM "xoper.sgml">
<!ENTITY xtypes SYSTEM "xtypes.sgml">
+<!ENTITY xsubscripting SYSTEM "xsubscripting.sgml">
<!ENTITY plperl SYSTEM "plperl.sgml">
<!ENTITY plpython SYSTEM "plpython.sgml">
<!ENTITY plsql SYSTEM "plpgsql.sgml">
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index c0a6554d4d..5c538dca05 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -602,6 +602,45 @@ SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qu
</para>
</sect2>
+ <sect2 id="jsonb-subscripting">
+ <title><type>jsonb</type> Subscripting</title>
+ <para>
+ <type>jsonb</type> data type supports array-style subscripting expressions
+ to extract or update particular elements. It's possible to use multiple
+ subscripting expressions to extract nested values. In this case, a chain of
+ subscripting expressions follows the same rules as the
+ <literal>path</literal> argument in <literal>jsonb_set</literal> function,
+ e.g. in case of arrays it is a 0-based operation or that negative integers
+ that appear in <literal>path</literal> count from the end of JSON arrays.
+ The result of subscripting expressions is always jsonb data type. An
+ example of subscripting syntax:
+<programlisting>
+-- Extract value by key
+SELECT ('{"a": 1}'::jsonb)['a'];
+
+-- Extract nested value by key path
+SELECT ('{"a": {"b": {"c": 1}}}'::jsonb)['a']['b']['c'];
+
+-- Extract element by index
+SELECT ('[1, "2", null]'::jsonb)[1];
+
+-- Update value by key
+UPDATE table_name SET jsonb_field['key'] = 1;
+
+-- Select records using where clause with subscripting. Since the result of
+-- subscripting is jsonb and we basically want to compare two jsonb objects, we
+-- need to put the value in double quotes to be able to convert it to jsonb.
+SELECT * FROM table_name WHERE jsonb_field['key'] = '"value"';
+</programlisting>
+
+ There is no special indexing support for such kind of expressions, but you
+ can always create a functional index that includes it
+<programlisting>
+CREATE INDEX idx ON table_name ((jsonb_field['key']));
+</programlisting>
+ </para>
+ </sect2>
+
<sect2>
<title>Transforms</title>
diff --git a/doc/src/sgml/ref/create_type.sgml b/doc/src/sgml/ref/create_type.sgml
index 111f8e65d2..a34df4d247 100644
--- a/doc/src/sgml/ref/create_type.sgml
+++ b/doc/src/sgml/ref/create_type.sgml
@@ -54,6 +54,7 @@ CREATE TYPE <replaceable class="parameter">name</replaceable> (
[ , ELEMENT = <replaceable class="parameter">element</replaceable> ]
[ , DELIMITER = <replaceable class="parameter">delimiter</replaceable> ]
[ , COLLATABLE = <replaceable class="parameter">collatable</replaceable> ]
+ [ , SUBSCRIPTING_HANDLER = <replaceable class="parameter">subscripting_handler_function</replaceable> ]
)
CREATE TYPE <replaceable class="parameter">name</replaceable>
@@ -196,8 +197,9 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
<replaceable class="parameter">receive_function</replaceable>,
<replaceable class="parameter">send_function</replaceable>,
<replaceable class="parameter">type_modifier_input_function</replaceable>,
- <replaceable class="parameter">type_modifier_output_function</replaceable> and
- <replaceable class="parameter">analyze_function</replaceable>
+ <replaceable class="parameter">type_modifier_output_function</replaceable>,
+ <replaceable class="parameter">analyze_function</replaceable>,
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
are optional. Generally these functions have to be coded in C
or another low-level language.
</para>
@@ -454,6 +456,22 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
make use of the collation information; this does not happen
automatically merely by marking the type collatable.
</para>
+
+ <para>
+ The optional
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
+ contains type-specific logic for subscripting of the data type.
+ By default, there is no such function provided, which means that the data
+ type doesn't support subscripting. The subscripting function must be
+ declared to take a single argument of type <type>internal</type>, and return
+ a <type>internal</type> result. There are two examples of implementation for
+ subscripting functions in case of array
+ (<replaceable class="parameter">array_subscripting_handler</replaceable>)
+ and jsonb
+ (<replaceable class="parameter">jsonb_subscripting_handler</replaceable>)
+ types in <filename>src/backend/utils/adt/arrayfuncs.c</filename> and
+ <filename>src/backend/utils/adt/jsonfuncs.c</filename>, respectively.
+ </para>
</refsect2>
<refsect2>
@@ -769,6 +787,17 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">subscripting_handler_function</replaceable></term>
+ <listitem>
+ <para>
+ The name of a function that returns list of type-specific callback functions to
+ support subscripting logic for the data type.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
diff --git a/doc/src/sgml/xsubscripting.sgml b/doc/src/sgml/xsubscripting.sgml
new file mode 100644
index 0000000000..7224e81fa2
--- /dev/null
+++ b/doc/src/sgml/xsubscripting.sgml
@@ -0,0 +1,111 @@
+<!-- doc/src/sgml/xsubscripting.sgml -->
+
+ <sect1 id="xsubscripting">
+ <title>User-defined subscripting procedure</title>
+
+ <indexterm zone="xsubscripting">
+ <primary>custom subscripting</primary>
+ </indexterm>
+ <para>
+ When you define a new base type, you can also specify a custom procedure to
+ handle subscripting expressions. It must contain logic for verification and
+ evaluation of this expression, i.e. fetching or updating some data in this
+ data type. For instance:
+</para>
+<programlisting><![CDATA[
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = someType;
+ sbsref->refassgntype = someType;
+
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ // some validation and coercion logic
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some assignment logic
+
+ return newContainer;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some fetch logic based on sbstate
+}]]>
+</programlisting>
+
+<para>
+ Then you can define a subscripting procedure and a custom data type:
+</para>
+<programlisting>
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '<replaceable>filename</replaceable>'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 4,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler,
+);
+</programlisting>
+
+<para>
+ and use it as usual:
+</para>
+<programlisting>
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
+</programlisting>
+
+
+ <para>
+ The examples of custom subscripting implementation can be found in
+ <filename>subscripting.sql</filename> and <filename>subscripting.c</filename>
+ in the <filename>src/tutorial</filename> directory of the source distribution.
+ See the <filename>README</filename> file in that directory for instructions
+ about running the examples.
+ </para>
+
+</sect1>
diff --git a/src/tutorial/Makefile b/src/tutorial/Makefile
index 16dc390f71..0ead60c2d4 100644
--- a/src/tutorial/Makefile
+++ b/src/tutorial/Makefile
@@ -13,8 +13,8 @@
#
#-------------------------------------------------------------------------
-MODULES = complex funcs
-DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql
+MODULES = complex funcs subscripting
+DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql subscripting.sql
ifdef NO_PGXS
subdir = src/tutorial
diff --git a/src/tutorial/subscripting.c b/src/tutorial/subscripting.c
new file mode 100644
index 0000000000..1eb8c45652
--- /dev/null
+++ b/src/tutorial/subscripting.c
@@ -0,0 +1,201 @@
+/*
+ * src/tutorial/subscripting.c
+ *
+ ******************************************************************************
+ This file contains routines that can be bound to a Postgres backend and
+ called by the backend in the process of processing queries. The calling
+ format for these routines is dictated by Postgres architecture.
+******************************************************************************/
+
+#include "postgres.h"
+
+#include "catalog/pg_type.h"
+#include "executor/executor.h"
+#include "executor/execExpr.h"
+#include "nodes/nodeFuncs.h"
+#include "parser/parse_coerce.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+SubscriptingRef * custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref);
+SubscriptingRef * custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate);
+Datum custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate);
+Datum custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate);
+
+PG_FUNCTION_INFO_V1(custom_in);
+PG_FUNCTION_INFO_V1(custom_out);
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+/*****************************************************************************
+ * Input/Output functions
+ *****************************************************************************/
+
+Datum
+custom_in(PG_FUNCTION_ARGS)
+{
+ char *str = PG_GETARG_CSTRING(0);
+ int firstValue,
+ secondValue;
+ Custom *result;
+
+ if (sscanf(str, " ( %d , %d )", &firstValue, &secondValue) != 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
+ errmsg("invalid input syntax for complex: \"%s\"",
+ str)));
+
+
+ result = (Custom *) palloc(sizeof(Custom));
+ result->first = firstValue;
+ result->second = secondValue;
+ PG_RETURN_POINTER(result);
+}
+
+Datum
+custom_out(PG_FUNCTION_ARGS)
+{
+ Custom *custom = (Custom *) PG_GETARG_POINTER(0);
+ char *result;
+
+ result = psprintf("(%d, %d)", custom->first, custom->second);
+ PG_RETURN_CSTRING(result);
+}
+
+/*****************************************************************************
+ * Custom subscripting logic functions
+ *****************************************************************************/
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = INT4OID;
+ sbsref->refassgntype = INT4OID;
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ List *upperIndexpr = NIL;
+ ListCell *l;
+
+ if (sbsref->reflowerindexpr != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *)lfirst(sbsref->reflowerindexpr->head))))));
+
+ foreach(l, sbsref->refupperindexpr)
+ {
+ Node *subexpr = (Node *) lfirst(l);
+
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *) lfirst(sbsref->refupperindexpr->head))))));
+
+ subexpr = coerce_to_target_type(pstate,
+ subexpr, exprType(subexpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript must have integer type"),
+ parser_errposition(pstate, exprLocation(subexpr))));
+
+ upperIndexpr = lappend(upperIndexpr, subexpr);
+
+ if (isAssignment)
+ {
+ Node *assignExpr = (Node *) sbsref->refassgnexpr;
+ Node *new_from;
+
+ new_from = coerce_to_target_type(pstate,
+ assignExpr, exprType(assignExpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (new_from == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom assignment requires int type"),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation(assignExpr))));
+ sbsref->refassgnexpr = (Expr *)new_from;
+ }
+ }
+
+ sbsref->refupperindexpr = upperIndexpr;
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ Custom *container= (Custom *) containerSource;
+ int index;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ return (Datum) container->first;
+ else
+ return (Datum) container->second;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ int index;
+ Custom *container = (Custom *) containerSource;
+
+ if (sbstate->resnull)
+ return containerSource;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ container->first = DatumGetInt32(sbstate->replacevalue);
+ else
+ container->second = DatumGetInt32(sbstate->replacevalue);
+
+ return (Datum) container;
+}
diff --git a/src/tutorial/subscripting.source b/src/tutorial/subscripting.source
new file mode 100644
index 0000000000..837cf30612
--- /dev/null
+++ b/src/tutorial/subscripting.source
@@ -0,0 +1,71 @@
+---------------------------------------------------------------------------
+--
+-- subscripting.sql-
+-- This file shows how to create a new subscripting procedure for
+-- user-defined type.
+--
+--
+-- Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+-- Portions Copyright (c) 1994, Regents of the University of California
+--
+-- src/tutorial/subscripting.source
+--
+---------------------------------------------------------------------------
+
+-----------------------------
+-- Creating a new type:
+-- We are going to create a new type called 'complex' which represents
+-- complex numbers.
+-- A user-defined type must have an input and an output function, and
+-- optionally can have binary input and output functions. All of these
+-- are usually user-defined C functions.
+-----------------------------
+
+-- Assume the user defined functions are in /home/erthalion/programms/postgresql-master/src/tutorial/complex$DLSUFFIX
+-- (we do not want to assume this is in the dynamic loader search path).
+-- Look at $PWD/complex.c for the source. Note that we declare all of
+-- them as STRICT, so we do not need to cope with NULL inputs in the
+-- C code. We also mark them IMMUTABLE, since they always return the
+-- same outputs given the same inputs.
+
+-- the input function 'complex_in' takes a null-terminated string (the
+-- textual representation of the type) and turns it into the internal
+-- (in memory) representation. You will get a message telling you 'complex'
+-- does not exist yet but that's okay.
+
+CREATE FUNCTION custom_in(cstring)
+ RETURNS custom
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+-- the output function 'complex_out' takes the internal representation and
+-- converts it into the textual representation.
+
+CREATE FUNCTION custom_out(custom)
+ RETURNS cstring
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 8,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler
+);
+
+-- we can use it in a table
+
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
--
2.21.0
--u66dlpg6gaevsofr
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v33-0005-Filling-gaps-in-jsonb-arrays.patch"
^ permalink raw reply [nested|flat] 55+ messages in thread
* [PATCH v33 4/5] Subscripting documentation
@ 2019-02-01 10:47 erthalion <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: erthalion @ 2019-02-01 10:47 UTC (permalink / raw)
Supporting documentation for generalized subscripting. It includes the
description of a new field in pg_type, the new section for jsonb
documentation about subscripting feature on this data type, and also the
tutorial about how to write subscripting operator for a custom data type.
Reviewed-by: Tom Lane, Arthur Zakirov, Oleksandr Shulgin
---
doc/src/sgml/catalogs.sgml | 8 ++
doc/src/sgml/extend.sgml | 6 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/json.sgml | 39 ++++++
doc/src/sgml/ref/create_type.sgml | 33 ++++-
doc/src/sgml/xsubscripting.sgml | 111 +++++++++++++++++
src/tutorial/Makefile | 4 +-
src/tutorial/subscripting.c | 201 ++++++++++++++++++++++++++++++
src/tutorial/subscripting.source | 71 +++++++++++
9 files changed, 470 insertions(+), 4 deletions(-)
create mode 100644 doc/src/sgml/xsubscripting.sgml
create mode 100644 src/tutorial/subscripting.c
create mode 100644 src/tutorial/subscripting.source
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 048ff284f7..b17e9ee382 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8971,6 +8971,14 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>typsubshandler</structfield></entry>
+ <entry><type>regproc</type></entry>
+ <entry><literal><link linkend="catalog-pg-proc"><structname>pg_proc</structname></link>.oid</literal></entry>
+ <entry>Custom subscripting function with type-specific logic for parsing
+ and validation, or 0 if this type doesn't support subscripting.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>typdefault</structfield> <type>text</type>
diff --git a/doc/src/sgml/extend.sgml b/doc/src/sgml/extend.sgml
index 890ff97b7a..65cf5706b6 100644
--- a/doc/src/sgml/extend.sgml
+++ b/doc/src/sgml/extend.sgml
@@ -33,6 +33,11 @@
operators (starting in <xref linkend="xoper"/>)
</para>
</listitem>
+ <listitem>
+ <para>
+ subscripting procedure (starting in <xref linkend="xsubscripting"/>)
+ </para>
+ </listitem>
<listitem>
<para>
operator classes for indexes (starting in <xref linkend="xindex"/>)
@@ -481,6 +486,7 @@ RETURNS anycompatible AS ...
&xaggr;
&xtypes;
&xoper;
+ &xsubscripting;
&xindex;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 64b5da0070..4c3fd37fc1 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -69,6 +69,7 @@
<!ENTITY xplang SYSTEM "xplang.sgml">
<!ENTITY xoper SYSTEM "xoper.sgml">
<!ENTITY xtypes SYSTEM "xtypes.sgml">
+<!ENTITY xsubscripting SYSTEM "xsubscripting.sgml">
<!ENTITY plperl SYSTEM "plperl.sgml">
<!ENTITY plpython SYSTEM "plpython.sgml">
<!ENTITY plsql SYSTEM "plpgsql.sgml">
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index c0a6554d4d..5c538dca05 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -602,6 +602,45 @@ SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qu
</para>
</sect2>
+ <sect2 id="jsonb-subscripting">
+ <title><type>jsonb</type> Subscripting</title>
+ <para>
+ <type>jsonb</type> data type supports array-style subscripting expressions
+ to extract or update particular elements. It's possible to use multiple
+ subscripting expressions to extract nested values. In this case, a chain of
+ subscripting expressions follows the same rules as the
+ <literal>path</literal> argument in <literal>jsonb_set</literal> function,
+ e.g. in case of arrays it is a 0-based operation or that negative integers
+ that appear in <literal>path</literal> count from the end of JSON arrays.
+ The result of subscripting expressions is always jsonb data type. An
+ example of subscripting syntax:
+<programlisting>
+-- Extract value by key
+SELECT ('{"a": 1}'::jsonb)['a'];
+
+-- Extract nested value by key path
+SELECT ('{"a": {"b": {"c": 1}}}'::jsonb)['a']['b']['c'];
+
+-- Extract element by index
+SELECT ('[1, "2", null]'::jsonb)[1];
+
+-- Update value by key
+UPDATE table_name SET jsonb_field['key'] = 1;
+
+-- Select records using where clause with subscripting. Since the result of
+-- subscripting is jsonb and we basically want to compare two jsonb objects, we
+-- need to put the value in double quotes to be able to convert it to jsonb.
+SELECT * FROM table_name WHERE jsonb_field['key'] = '"value"';
+</programlisting>
+
+ There is no special indexing support for such kind of expressions, but you
+ can always create a functional index that includes it
+<programlisting>
+CREATE INDEX idx ON table_name ((jsonb_field['key']));
+</programlisting>
+ </para>
+ </sect2>
+
<sect2>
<title>Transforms</title>
diff --git a/doc/src/sgml/ref/create_type.sgml b/doc/src/sgml/ref/create_type.sgml
index 111f8e65d2..a34df4d247 100644
--- a/doc/src/sgml/ref/create_type.sgml
+++ b/doc/src/sgml/ref/create_type.sgml
@@ -54,6 +54,7 @@ CREATE TYPE <replaceable class="parameter">name</replaceable> (
[ , ELEMENT = <replaceable class="parameter">element</replaceable> ]
[ , DELIMITER = <replaceable class="parameter">delimiter</replaceable> ]
[ , COLLATABLE = <replaceable class="parameter">collatable</replaceable> ]
+ [ , SUBSCRIPTING_HANDLER = <replaceable class="parameter">subscripting_handler_function</replaceable> ]
)
CREATE TYPE <replaceable class="parameter">name</replaceable>
@@ -196,8 +197,9 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
<replaceable class="parameter">receive_function</replaceable>,
<replaceable class="parameter">send_function</replaceable>,
<replaceable class="parameter">type_modifier_input_function</replaceable>,
- <replaceable class="parameter">type_modifier_output_function</replaceable> and
- <replaceable class="parameter">analyze_function</replaceable>
+ <replaceable class="parameter">type_modifier_output_function</replaceable>,
+ <replaceable class="parameter">analyze_function</replaceable>,
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
are optional. Generally these functions have to be coded in C
or another low-level language.
</para>
@@ -454,6 +456,22 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
make use of the collation information; this does not happen
automatically merely by marking the type collatable.
</para>
+
+ <para>
+ The optional
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
+ contains type-specific logic for subscripting of the data type.
+ By default, there is no such function provided, which means that the data
+ type doesn't support subscripting. The subscripting function must be
+ declared to take a single argument of type <type>internal</type>, and return
+ a <type>internal</type> result. There are two examples of implementation for
+ subscripting functions in case of array
+ (<replaceable class="parameter">array_subscripting_handler</replaceable>)
+ and jsonb
+ (<replaceable class="parameter">jsonb_subscripting_handler</replaceable>)
+ types in <filename>src/backend/utils/adt/arrayfuncs.c</filename> and
+ <filename>src/backend/utils/adt/jsonfuncs.c</filename>, respectively.
+ </para>
</refsect2>
<refsect2>
@@ -769,6 +787,17 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">subscripting_handler_function</replaceable></term>
+ <listitem>
+ <para>
+ The name of a function that returns list of type-specific callback functions to
+ support subscripting logic for the data type.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
diff --git a/doc/src/sgml/xsubscripting.sgml b/doc/src/sgml/xsubscripting.sgml
new file mode 100644
index 0000000000..7224e81fa2
--- /dev/null
+++ b/doc/src/sgml/xsubscripting.sgml
@@ -0,0 +1,111 @@
+<!-- doc/src/sgml/xsubscripting.sgml -->
+
+ <sect1 id="xsubscripting">
+ <title>User-defined subscripting procedure</title>
+
+ <indexterm zone="xsubscripting">
+ <primary>custom subscripting</primary>
+ </indexterm>
+ <para>
+ When you define a new base type, you can also specify a custom procedure to
+ handle subscripting expressions. It must contain logic for verification and
+ evaluation of this expression, i.e. fetching or updating some data in this
+ data type. For instance:
+</para>
+<programlisting><![CDATA[
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = someType;
+ sbsref->refassgntype = someType;
+
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ // some validation and coercion logic
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some assignment logic
+
+ return newContainer;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some fetch logic based on sbstate
+}]]>
+</programlisting>
+
+<para>
+ Then you can define a subscripting procedure and a custom data type:
+</para>
+<programlisting>
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '<replaceable>filename</replaceable>'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 4,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler,
+);
+</programlisting>
+
+<para>
+ and use it as usual:
+</para>
+<programlisting>
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
+</programlisting>
+
+
+ <para>
+ The examples of custom subscripting implementation can be found in
+ <filename>subscripting.sql</filename> and <filename>subscripting.c</filename>
+ in the <filename>src/tutorial</filename> directory of the source distribution.
+ See the <filename>README</filename> file in that directory for instructions
+ about running the examples.
+ </para>
+
+</sect1>
diff --git a/src/tutorial/Makefile b/src/tutorial/Makefile
index 16dc390f71..0ead60c2d4 100644
--- a/src/tutorial/Makefile
+++ b/src/tutorial/Makefile
@@ -13,8 +13,8 @@
#
#-------------------------------------------------------------------------
-MODULES = complex funcs
-DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql
+MODULES = complex funcs subscripting
+DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql subscripting.sql
ifdef NO_PGXS
subdir = src/tutorial
diff --git a/src/tutorial/subscripting.c b/src/tutorial/subscripting.c
new file mode 100644
index 0000000000..1eb8c45652
--- /dev/null
+++ b/src/tutorial/subscripting.c
@@ -0,0 +1,201 @@
+/*
+ * src/tutorial/subscripting.c
+ *
+ ******************************************************************************
+ This file contains routines that can be bound to a Postgres backend and
+ called by the backend in the process of processing queries. The calling
+ format for these routines is dictated by Postgres architecture.
+******************************************************************************/
+
+#include "postgres.h"
+
+#include "catalog/pg_type.h"
+#include "executor/executor.h"
+#include "executor/execExpr.h"
+#include "nodes/nodeFuncs.h"
+#include "parser/parse_coerce.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+SubscriptingRef * custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref);
+SubscriptingRef * custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate);
+Datum custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate);
+Datum custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate);
+
+PG_FUNCTION_INFO_V1(custom_in);
+PG_FUNCTION_INFO_V1(custom_out);
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+/*****************************************************************************
+ * Input/Output functions
+ *****************************************************************************/
+
+Datum
+custom_in(PG_FUNCTION_ARGS)
+{
+ char *str = PG_GETARG_CSTRING(0);
+ int firstValue,
+ secondValue;
+ Custom *result;
+
+ if (sscanf(str, " ( %d , %d )", &firstValue, &secondValue) != 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
+ errmsg("invalid input syntax for complex: \"%s\"",
+ str)));
+
+
+ result = (Custom *) palloc(sizeof(Custom));
+ result->first = firstValue;
+ result->second = secondValue;
+ PG_RETURN_POINTER(result);
+}
+
+Datum
+custom_out(PG_FUNCTION_ARGS)
+{
+ Custom *custom = (Custom *) PG_GETARG_POINTER(0);
+ char *result;
+
+ result = psprintf("(%d, %d)", custom->first, custom->second);
+ PG_RETURN_CSTRING(result);
+}
+
+/*****************************************************************************
+ * Custom subscripting logic functions
+ *****************************************************************************/
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = INT4OID;
+ sbsref->refassgntype = INT4OID;
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ List *upperIndexpr = NIL;
+ ListCell *l;
+
+ if (sbsref->reflowerindexpr != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *)lfirst(sbsref->reflowerindexpr->head))))));
+
+ foreach(l, sbsref->refupperindexpr)
+ {
+ Node *subexpr = (Node *) lfirst(l);
+
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *) lfirst(sbsref->refupperindexpr->head))))));
+
+ subexpr = coerce_to_target_type(pstate,
+ subexpr, exprType(subexpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript must have integer type"),
+ parser_errposition(pstate, exprLocation(subexpr))));
+
+ upperIndexpr = lappend(upperIndexpr, subexpr);
+
+ if (isAssignment)
+ {
+ Node *assignExpr = (Node *) sbsref->refassgnexpr;
+ Node *new_from;
+
+ new_from = coerce_to_target_type(pstate,
+ assignExpr, exprType(assignExpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (new_from == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom assignment requires int type"),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation(assignExpr))));
+ sbsref->refassgnexpr = (Expr *)new_from;
+ }
+ }
+
+ sbsref->refupperindexpr = upperIndexpr;
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ Custom *container= (Custom *) containerSource;
+ int index;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ return (Datum) container->first;
+ else
+ return (Datum) container->second;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ int index;
+ Custom *container = (Custom *) containerSource;
+
+ if (sbstate->resnull)
+ return containerSource;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ container->first = DatumGetInt32(sbstate->replacevalue);
+ else
+ container->second = DatumGetInt32(sbstate->replacevalue);
+
+ return (Datum) container;
+}
diff --git a/src/tutorial/subscripting.source b/src/tutorial/subscripting.source
new file mode 100644
index 0000000000..837cf30612
--- /dev/null
+++ b/src/tutorial/subscripting.source
@@ -0,0 +1,71 @@
+---------------------------------------------------------------------------
+--
+-- subscripting.sql-
+-- This file shows how to create a new subscripting procedure for
+-- user-defined type.
+--
+--
+-- Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+-- Portions Copyright (c) 1994, Regents of the University of California
+--
+-- src/tutorial/subscripting.source
+--
+---------------------------------------------------------------------------
+
+-----------------------------
+-- Creating a new type:
+-- We are going to create a new type called 'complex' which represents
+-- complex numbers.
+-- A user-defined type must have an input and an output function, and
+-- optionally can have binary input and output functions. All of these
+-- are usually user-defined C functions.
+-----------------------------
+
+-- Assume the user defined functions are in /home/erthalion/programms/postgresql-master/src/tutorial/complex$DLSUFFIX
+-- (we do not want to assume this is in the dynamic loader search path).
+-- Look at $PWD/complex.c for the source. Note that we declare all of
+-- them as STRICT, so we do not need to cope with NULL inputs in the
+-- C code. We also mark them IMMUTABLE, since they always return the
+-- same outputs given the same inputs.
+
+-- the input function 'complex_in' takes a null-terminated string (the
+-- textual representation of the type) and turns it into the internal
+-- (in memory) representation. You will get a message telling you 'complex'
+-- does not exist yet but that's okay.
+
+CREATE FUNCTION custom_in(cstring)
+ RETURNS custom
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+-- the output function 'complex_out' takes the internal representation and
+-- converts it into the textual representation.
+
+CREATE FUNCTION custom_out(custom)
+ RETURNS cstring
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 8,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler
+);
+
+-- we can use it in a table
+
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
--
2.21.0
--u66dlpg6gaevsofr
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v33-0005-Filling-gaps-in-jsonb-arrays.patch"
^ permalink raw reply [nested|flat] 55+ messages in thread
* [PATCH v32 4/6] Subscripting documentation
@ 2019-02-01 10:47 erthalion <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: erthalion @ 2019-02-01 10:47 UTC (permalink / raw)
Supporting documentation for generalized subscripting. It includes the
description of a new field in pg_type, the new section for jsonb
documentation about subscripting feature on this data type, and also the
tutorial about how to write subscripting operator for a custom data type.
Reviewed-by: Tom Lane, Arthur Zakirov, Oleksandr Shulgin
---
doc/src/sgml/catalogs.sgml | 8 ++
doc/src/sgml/extend.sgml | 6 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/json.sgml | 39 ++++++
doc/src/sgml/ref/create_type.sgml | 33 ++++-
doc/src/sgml/xsubscripting.sgml | 111 +++++++++++++++++
src/tutorial/Makefile | 4 +-
src/tutorial/subscripting.c | 201 ++++++++++++++++++++++++++++++
src/tutorial/subscripting.source | 71 +++++++++++
9 files changed, 470 insertions(+), 4 deletions(-)
create mode 100644 doc/src/sgml/xsubscripting.sgml
create mode 100644 src/tutorial/subscripting.c
create mode 100644 src/tutorial/subscripting.source
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 048ff284f7..b17e9ee382 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8971,6 +8971,14 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>typsubshandler</structfield></entry>
+ <entry><type>regproc</type></entry>
+ <entry><literal><link linkend="catalog-pg-proc"><structname>pg_proc</structname></link>.oid</literal></entry>
+ <entry>Custom subscripting function with type-specific logic for parsing
+ and validation, or 0 if this type doesn't support subscripting.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>typdefault</structfield> <type>text</type>
diff --git a/doc/src/sgml/extend.sgml b/doc/src/sgml/extend.sgml
index 890ff97b7a..65cf5706b6 100644
--- a/doc/src/sgml/extend.sgml
+++ b/doc/src/sgml/extend.sgml
@@ -33,6 +33,11 @@
operators (starting in <xref linkend="xoper"/>)
</para>
</listitem>
+ <listitem>
+ <para>
+ subscripting procedure (starting in <xref linkend="xsubscripting"/>)
+ </para>
+ </listitem>
<listitem>
<para>
operator classes for indexes (starting in <xref linkend="xindex"/>)
@@ -481,6 +486,7 @@ RETURNS anycompatible AS ...
&xaggr;
&xtypes;
&xoper;
+ &xsubscripting;
&xindex;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 64b5da0070..4c3fd37fc1 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -69,6 +69,7 @@
<!ENTITY xplang SYSTEM "xplang.sgml">
<!ENTITY xoper SYSTEM "xoper.sgml">
<!ENTITY xtypes SYSTEM "xtypes.sgml">
+<!ENTITY xsubscripting SYSTEM "xsubscripting.sgml">
<!ENTITY plperl SYSTEM "plperl.sgml">
<!ENTITY plpython SYSTEM "plpython.sgml">
<!ENTITY plsql SYSTEM "plpgsql.sgml">
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index c0a6554d4d..5c538dca05 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -602,6 +602,45 @@ SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qu
</para>
</sect2>
+ <sect2 id="jsonb-subscripting">
+ <title><type>jsonb</type> Subscripting</title>
+ <para>
+ <type>jsonb</type> data type supports array-style subscripting expressions
+ to extract or update particular elements. It's possible to use multiple
+ subscripting expressions to extract nested values. In this case, a chain of
+ subscripting expressions follows the same rules as the
+ <literal>path</literal> argument in <literal>jsonb_set</literal> function,
+ e.g. in case of arrays it is a 0-based operation or that negative integers
+ that appear in <literal>path</literal> count from the end of JSON arrays.
+ The result of subscripting expressions is always jsonb data type. An
+ example of subscripting syntax:
+<programlisting>
+-- Extract value by key
+SELECT ('{"a": 1}'::jsonb)['a'];
+
+-- Extract nested value by key path
+SELECT ('{"a": {"b": {"c": 1}}}'::jsonb)['a']['b']['c'];
+
+-- Extract element by index
+SELECT ('[1, "2", null]'::jsonb)[1];
+
+-- Update value by key
+UPDATE table_name SET jsonb_field['key'] = 1;
+
+-- Select records using where clause with subscripting. Since the result of
+-- subscripting is jsonb and we basically want to compare two jsonb objects, we
+-- need to put the value in double quotes to be able to convert it to jsonb.
+SELECT * FROM table_name WHERE jsonb_field['key'] = '"value"';
+</programlisting>
+
+ There is no special indexing support for such kind of expressions, but you
+ can always create a functional index that includes it
+<programlisting>
+CREATE INDEX idx ON table_name ((jsonb_field['key']));
+</programlisting>
+ </para>
+ </sect2>
+
<sect2>
<title>Transforms</title>
diff --git a/doc/src/sgml/ref/create_type.sgml b/doc/src/sgml/ref/create_type.sgml
index 111f8e65d2..a34df4d247 100644
--- a/doc/src/sgml/ref/create_type.sgml
+++ b/doc/src/sgml/ref/create_type.sgml
@@ -54,6 +54,7 @@ CREATE TYPE <replaceable class="parameter">name</replaceable> (
[ , ELEMENT = <replaceable class="parameter">element</replaceable> ]
[ , DELIMITER = <replaceable class="parameter">delimiter</replaceable> ]
[ , COLLATABLE = <replaceable class="parameter">collatable</replaceable> ]
+ [ , SUBSCRIPTING_HANDLER = <replaceable class="parameter">subscripting_handler_function</replaceable> ]
)
CREATE TYPE <replaceable class="parameter">name</replaceable>
@@ -196,8 +197,9 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
<replaceable class="parameter">receive_function</replaceable>,
<replaceable class="parameter">send_function</replaceable>,
<replaceable class="parameter">type_modifier_input_function</replaceable>,
- <replaceable class="parameter">type_modifier_output_function</replaceable> and
- <replaceable class="parameter">analyze_function</replaceable>
+ <replaceable class="parameter">type_modifier_output_function</replaceable>,
+ <replaceable class="parameter">analyze_function</replaceable>,
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
are optional. Generally these functions have to be coded in C
or another low-level language.
</para>
@@ -454,6 +456,22 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
make use of the collation information; this does not happen
automatically merely by marking the type collatable.
</para>
+
+ <para>
+ The optional
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
+ contains type-specific logic for subscripting of the data type.
+ By default, there is no such function provided, which means that the data
+ type doesn't support subscripting. The subscripting function must be
+ declared to take a single argument of type <type>internal</type>, and return
+ a <type>internal</type> result. There are two examples of implementation for
+ subscripting functions in case of array
+ (<replaceable class="parameter">array_subscripting_handler</replaceable>)
+ and jsonb
+ (<replaceable class="parameter">jsonb_subscripting_handler</replaceable>)
+ types in <filename>src/backend/utils/adt/arrayfuncs.c</filename> and
+ <filename>src/backend/utils/adt/jsonfuncs.c</filename>, respectively.
+ </para>
</refsect2>
<refsect2>
@@ -769,6 +787,17 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">subscripting_handler_function</replaceable></term>
+ <listitem>
+ <para>
+ The name of a function that returns list of type-specific callback functions to
+ support subscripting logic for the data type.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
diff --git a/doc/src/sgml/xsubscripting.sgml b/doc/src/sgml/xsubscripting.sgml
new file mode 100644
index 0000000000..7224e81fa2
--- /dev/null
+++ b/doc/src/sgml/xsubscripting.sgml
@@ -0,0 +1,111 @@
+<!-- doc/src/sgml/xsubscripting.sgml -->
+
+ <sect1 id="xsubscripting">
+ <title>User-defined subscripting procedure</title>
+
+ <indexterm zone="xsubscripting">
+ <primary>custom subscripting</primary>
+ </indexterm>
+ <para>
+ When you define a new base type, you can also specify a custom procedure to
+ handle subscripting expressions. It must contain logic for verification and
+ evaluation of this expression, i.e. fetching or updating some data in this
+ data type. For instance:
+</para>
+<programlisting><![CDATA[
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = someType;
+ sbsref->refassgntype = someType;
+
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ // some validation and coercion logic
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some assignment logic
+
+ return newContainer;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some fetch logic based on sbstate
+}]]>
+</programlisting>
+
+<para>
+ Then you can define a subscripting procedure and a custom data type:
+</para>
+<programlisting>
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '<replaceable>filename</replaceable>'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 4,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler,
+);
+</programlisting>
+
+<para>
+ and use it as usual:
+</para>
+<programlisting>
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
+</programlisting>
+
+
+ <para>
+ The examples of custom subscripting implementation can be found in
+ <filename>subscripting.sql</filename> and <filename>subscripting.c</filename>
+ in the <filename>src/tutorial</filename> directory of the source distribution.
+ See the <filename>README</filename> file in that directory for instructions
+ about running the examples.
+ </para>
+
+</sect1>
diff --git a/src/tutorial/Makefile b/src/tutorial/Makefile
index 16dc390f71..0ead60c2d4 100644
--- a/src/tutorial/Makefile
+++ b/src/tutorial/Makefile
@@ -13,8 +13,8 @@
#
#-------------------------------------------------------------------------
-MODULES = complex funcs
-DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql
+MODULES = complex funcs subscripting
+DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql subscripting.sql
ifdef NO_PGXS
subdir = src/tutorial
diff --git a/src/tutorial/subscripting.c b/src/tutorial/subscripting.c
new file mode 100644
index 0000000000..1eb8c45652
--- /dev/null
+++ b/src/tutorial/subscripting.c
@@ -0,0 +1,201 @@
+/*
+ * src/tutorial/subscripting.c
+ *
+ ******************************************************************************
+ This file contains routines that can be bound to a Postgres backend and
+ called by the backend in the process of processing queries. The calling
+ format for these routines is dictated by Postgres architecture.
+******************************************************************************/
+
+#include "postgres.h"
+
+#include "catalog/pg_type.h"
+#include "executor/executor.h"
+#include "executor/execExpr.h"
+#include "nodes/nodeFuncs.h"
+#include "parser/parse_coerce.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+SubscriptingRef * custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref);
+SubscriptingRef * custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate);
+Datum custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate);
+Datum custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate);
+
+PG_FUNCTION_INFO_V1(custom_in);
+PG_FUNCTION_INFO_V1(custom_out);
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+/*****************************************************************************
+ * Input/Output functions
+ *****************************************************************************/
+
+Datum
+custom_in(PG_FUNCTION_ARGS)
+{
+ char *str = PG_GETARG_CSTRING(0);
+ int firstValue,
+ secondValue;
+ Custom *result;
+
+ if (sscanf(str, " ( %d , %d )", &firstValue, &secondValue) != 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
+ errmsg("invalid input syntax for complex: \"%s\"",
+ str)));
+
+
+ result = (Custom *) palloc(sizeof(Custom));
+ result->first = firstValue;
+ result->second = secondValue;
+ PG_RETURN_POINTER(result);
+}
+
+Datum
+custom_out(PG_FUNCTION_ARGS)
+{
+ Custom *custom = (Custom *) PG_GETARG_POINTER(0);
+ char *result;
+
+ result = psprintf("(%d, %d)", custom->first, custom->second);
+ PG_RETURN_CSTRING(result);
+}
+
+/*****************************************************************************
+ * Custom subscripting logic functions
+ *****************************************************************************/
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = INT4OID;
+ sbsref->refassgntype = INT4OID;
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ List *upperIndexpr = NIL;
+ ListCell *l;
+
+ if (sbsref->reflowerindexpr != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *)lfirst(sbsref->reflowerindexpr->head))))));
+
+ foreach(l, sbsref->refupperindexpr)
+ {
+ Node *subexpr = (Node *) lfirst(l);
+
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *) lfirst(sbsref->refupperindexpr->head))))));
+
+ subexpr = coerce_to_target_type(pstate,
+ subexpr, exprType(subexpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript must have integer type"),
+ parser_errposition(pstate, exprLocation(subexpr))));
+
+ upperIndexpr = lappend(upperIndexpr, subexpr);
+
+ if (isAssignment)
+ {
+ Node *assignExpr = (Node *) sbsref->refassgnexpr;
+ Node *new_from;
+
+ new_from = coerce_to_target_type(pstate,
+ assignExpr, exprType(assignExpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (new_from == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom assignment requires int type"),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation(assignExpr))));
+ sbsref->refassgnexpr = (Expr *)new_from;
+ }
+ }
+
+ sbsref->refupperindexpr = upperIndexpr;
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ Custom *container= (Custom *) containerSource;
+ int index;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ return (Datum) container->first;
+ else
+ return (Datum) container->second;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ int index;
+ Custom *container = (Custom *) containerSource;
+
+ if (sbstate->resnull)
+ return containerSource;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ container->first = DatumGetInt32(sbstate->replacevalue);
+ else
+ container->second = DatumGetInt32(sbstate->replacevalue);
+
+ return (Datum) container;
+}
diff --git a/src/tutorial/subscripting.source b/src/tutorial/subscripting.source
new file mode 100644
index 0000000000..837cf30612
--- /dev/null
+++ b/src/tutorial/subscripting.source
@@ -0,0 +1,71 @@
+---------------------------------------------------------------------------
+--
+-- subscripting.sql-
+-- This file shows how to create a new subscripting procedure for
+-- user-defined type.
+--
+--
+-- Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+-- Portions Copyright (c) 1994, Regents of the University of California
+--
+-- src/tutorial/subscripting.source
+--
+---------------------------------------------------------------------------
+
+-----------------------------
+-- Creating a new type:
+-- We are going to create a new type called 'complex' which represents
+-- complex numbers.
+-- A user-defined type must have an input and an output function, and
+-- optionally can have binary input and output functions. All of these
+-- are usually user-defined C functions.
+-----------------------------
+
+-- Assume the user defined functions are in /home/erthalion/programms/postgresql-master/src/tutorial/complex$DLSUFFIX
+-- (we do not want to assume this is in the dynamic loader search path).
+-- Look at $PWD/complex.c for the source. Note that we declare all of
+-- them as STRICT, so we do not need to cope with NULL inputs in the
+-- C code. We also mark them IMMUTABLE, since they always return the
+-- same outputs given the same inputs.
+
+-- the input function 'complex_in' takes a null-terminated string (the
+-- textual representation of the type) and turns it into the internal
+-- (in memory) representation. You will get a message telling you 'complex'
+-- does not exist yet but that's okay.
+
+CREATE FUNCTION custom_in(cstring)
+ RETURNS custom
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+-- the output function 'complex_out' takes the internal representation and
+-- converts it into the textual representation.
+
+CREATE FUNCTION custom_out(custom)
+ RETURNS cstring
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 8,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler
+);
+
+-- we can use it in a table
+
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
--
2.21.0
--ques2rolqe435p6o
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v32-0005-Polymorphic-subscripting.patch"
^ permalink raw reply [nested|flat] 55+ messages in thread
* [PATCH v33 4/5] Subscripting documentation
@ 2019-02-01 10:47 erthalion <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: erthalion @ 2019-02-01 10:47 UTC (permalink / raw)
Supporting documentation for generalized subscripting. It includes the
description of a new field in pg_type, the new section for jsonb
documentation about subscripting feature on this data type, and also the
tutorial about how to write subscripting operator for a custom data type.
Reviewed-by: Tom Lane, Arthur Zakirov, Oleksandr Shulgin
---
doc/src/sgml/catalogs.sgml | 8 ++
doc/src/sgml/extend.sgml | 6 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/json.sgml | 39 ++++++
doc/src/sgml/ref/create_type.sgml | 33 ++++-
doc/src/sgml/xsubscripting.sgml | 111 +++++++++++++++++
src/tutorial/Makefile | 4 +-
src/tutorial/subscripting.c | 201 ++++++++++++++++++++++++++++++
src/tutorial/subscripting.source | 71 +++++++++++
9 files changed, 470 insertions(+), 4 deletions(-)
create mode 100644 doc/src/sgml/xsubscripting.sgml
create mode 100644 src/tutorial/subscripting.c
create mode 100644 src/tutorial/subscripting.source
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 048ff284f7..b17e9ee382 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8971,6 +8971,14 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>typsubshandler</structfield></entry>
+ <entry><type>regproc</type></entry>
+ <entry><literal><link linkend="catalog-pg-proc"><structname>pg_proc</structname></link>.oid</literal></entry>
+ <entry>Custom subscripting function with type-specific logic for parsing
+ and validation, or 0 if this type doesn't support subscripting.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>typdefault</structfield> <type>text</type>
diff --git a/doc/src/sgml/extend.sgml b/doc/src/sgml/extend.sgml
index 890ff97b7a..65cf5706b6 100644
--- a/doc/src/sgml/extend.sgml
+++ b/doc/src/sgml/extend.sgml
@@ -33,6 +33,11 @@
operators (starting in <xref linkend="xoper"/>)
</para>
</listitem>
+ <listitem>
+ <para>
+ subscripting procedure (starting in <xref linkend="xsubscripting"/>)
+ </para>
+ </listitem>
<listitem>
<para>
operator classes for indexes (starting in <xref linkend="xindex"/>)
@@ -481,6 +486,7 @@ RETURNS anycompatible AS ...
&xaggr;
&xtypes;
&xoper;
+ &xsubscripting;
&xindex;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 64b5da0070..4c3fd37fc1 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -69,6 +69,7 @@
<!ENTITY xplang SYSTEM "xplang.sgml">
<!ENTITY xoper SYSTEM "xoper.sgml">
<!ENTITY xtypes SYSTEM "xtypes.sgml">
+<!ENTITY xsubscripting SYSTEM "xsubscripting.sgml">
<!ENTITY plperl SYSTEM "plperl.sgml">
<!ENTITY plpython SYSTEM "plpython.sgml">
<!ENTITY plsql SYSTEM "plpgsql.sgml">
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index c0a6554d4d..5c538dca05 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -602,6 +602,45 @@ SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qu
</para>
</sect2>
+ <sect2 id="jsonb-subscripting">
+ <title><type>jsonb</type> Subscripting</title>
+ <para>
+ <type>jsonb</type> data type supports array-style subscripting expressions
+ to extract or update particular elements. It's possible to use multiple
+ subscripting expressions to extract nested values. In this case, a chain of
+ subscripting expressions follows the same rules as the
+ <literal>path</literal> argument in <literal>jsonb_set</literal> function,
+ e.g. in case of arrays it is a 0-based operation or that negative integers
+ that appear in <literal>path</literal> count from the end of JSON arrays.
+ The result of subscripting expressions is always jsonb data type. An
+ example of subscripting syntax:
+<programlisting>
+-- Extract value by key
+SELECT ('{"a": 1}'::jsonb)['a'];
+
+-- Extract nested value by key path
+SELECT ('{"a": {"b": {"c": 1}}}'::jsonb)['a']['b']['c'];
+
+-- Extract element by index
+SELECT ('[1, "2", null]'::jsonb)[1];
+
+-- Update value by key
+UPDATE table_name SET jsonb_field['key'] = 1;
+
+-- Select records using where clause with subscripting. Since the result of
+-- subscripting is jsonb and we basically want to compare two jsonb objects, we
+-- need to put the value in double quotes to be able to convert it to jsonb.
+SELECT * FROM table_name WHERE jsonb_field['key'] = '"value"';
+</programlisting>
+
+ There is no special indexing support for such kind of expressions, but you
+ can always create a functional index that includes it
+<programlisting>
+CREATE INDEX idx ON table_name ((jsonb_field['key']));
+</programlisting>
+ </para>
+ </sect2>
+
<sect2>
<title>Transforms</title>
diff --git a/doc/src/sgml/ref/create_type.sgml b/doc/src/sgml/ref/create_type.sgml
index 111f8e65d2..a34df4d247 100644
--- a/doc/src/sgml/ref/create_type.sgml
+++ b/doc/src/sgml/ref/create_type.sgml
@@ -54,6 +54,7 @@ CREATE TYPE <replaceable class="parameter">name</replaceable> (
[ , ELEMENT = <replaceable class="parameter">element</replaceable> ]
[ , DELIMITER = <replaceable class="parameter">delimiter</replaceable> ]
[ , COLLATABLE = <replaceable class="parameter">collatable</replaceable> ]
+ [ , SUBSCRIPTING_HANDLER = <replaceable class="parameter">subscripting_handler_function</replaceable> ]
)
CREATE TYPE <replaceable class="parameter">name</replaceable>
@@ -196,8 +197,9 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
<replaceable class="parameter">receive_function</replaceable>,
<replaceable class="parameter">send_function</replaceable>,
<replaceable class="parameter">type_modifier_input_function</replaceable>,
- <replaceable class="parameter">type_modifier_output_function</replaceable> and
- <replaceable class="parameter">analyze_function</replaceable>
+ <replaceable class="parameter">type_modifier_output_function</replaceable>,
+ <replaceable class="parameter">analyze_function</replaceable>,
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
are optional. Generally these functions have to be coded in C
or another low-level language.
</para>
@@ -454,6 +456,22 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
make use of the collation information; this does not happen
automatically merely by marking the type collatable.
</para>
+
+ <para>
+ The optional
+ <replaceable class="parameter">subscripting_handler_function</replaceable>
+ contains type-specific logic for subscripting of the data type.
+ By default, there is no such function provided, which means that the data
+ type doesn't support subscripting. The subscripting function must be
+ declared to take a single argument of type <type>internal</type>, and return
+ a <type>internal</type> result. There are two examples of implementation for
+ subscripting functions in case of array
+ (<replaceable class="parameter">array_subscripting_handler</replaceable>)
+ and jsonb
+ (<replaceable class="parameter">jsonb_subscripting_handler</replaceable>)
+ types in <filename>src/backend/utils/adt/arrayfuncs.c</filename> and
+ <filename>src/backend/utils/adt/jsonfuncs.c</filename>, respectively.
+ </para>
</refsect2>
<refsect2>
@@ -769,6 +787,17 @@ CREATE TYPE <replaceable class="parameter">name</replaceable>
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">subscripting_handler_function</replaceable></term>
+ <listitem>
+ <para>
+ The name of a function that returns list of type-specific callback functions to
+ support subscripting logic for the data type.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
diff --git a/doc/src/sgml/xsubscripting.sgml b/doc/src/sgml/xsubscripting.sgml
new file mode 100644
index 0000000000..7224e81fa2
--- /dev/null
+++ b/doc/src/sgml/xsubscripting.sgml
@@ -0,0 +1,111 @@
+<!-- doc/src/sgml/xsubscripting.sgml -->
+
+ <sect1 id="xsubscripting">
+ <title>User-defined subscripting procedure</title>
+
+ <indexterm zone="xsubscripting">
+ <primary>custom subscripting</primary>
+ </indexterm>
+ <para>
+ When you define a new base type, you can also specify a custom procedure to
+ handle subscripting expressions. It must contain logic for verification and
+ evaluation of this expression, i.e. fetching or updating some data in this
+ data type. For instance:
+</para>
+<programlisting><![CDATA[
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = someType;
+ sbsref->refassgntype = someType;
+
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ // some validation and coercion logic
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some assignment logic
+
+ return newContainer;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ // Some fetch logic based on sbstate
+}]]>
+</programlisting>
+
+<para>
+ Then you can define a subscripting procedure and a custom data type:
+</para>
+<programlisting>
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '<replaceable>filename</replaceable>'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 4,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler,
+);
+</programlisting>
+
+<para>
+ and use it as usual:
+</para>
+<programlisting>
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
+</programlisting>
+
+
+ <para>
+ The examples of custom subscripting implementation can be found in
+ <filename>subscripting.sql</filename> and <filename>subscripting.c</filename>
+ in the <filename>src/tutorial</filename> directory of the source distribution.
+ See the <filename>README</filename> file in that directory for instructions
+ about running the examples.
+ </para>
+
+</sect1>
diff --git a/src/tutorial/Makefile b/src/tutorial/Makefile
index 16dc390f71..0ead60c2d4 100644
--- a/src/tutorial/Makefile
+++ b/src/tutorial/Makefile
@@ -13,8 +13,8 @@
#
#-------------------------------------------------------------------------
-MODULES = complex funcs
-DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql
+MODULES = complex funcs subscripting
+DATA_built = advanced.sql basics.sql complex.sql funcs.sql syscat.sql subscripting.sql
ifdef NO_PGXS
subdir = src/tutorial
diff --git a/src/tutorial/subscripting.c b/src/tutorial/subscripting.c
new file mode 100644
index 0000000000..1eb8c45652
--- /dev/null
+++ b/src/tutorial/subscripting.c
@@ -0,0 +1,201 @@
+/*
+ * src/tutorial/subscripting.c
+ *
+ ******************************************************************************
+ This file contains routines that can be bound to a Postgres backend and
+ called by the backend in the process of processing queries. The calling
+ format for these routines is dictated by Postgres architecture.
+******************************************************************************/
+
+#include "postgres.h"
+
+#include "catalog/pg_type.h"
+#include "executor/executor.h"
+#include "executor/execExpr.h"
+#include "nodes/nodeFuncs.h"
+#include "parser/parse_coerce.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+typedef struct Custom
+{
+ int first;
+ int second;
+} Custom;
+
+SubscriptingRef * custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref);
+SubscriptingRef * custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate);
+Datum custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate);
+Datum custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate);
+
+PG_FUNCTION_INFO_V1(custom_in);
+PG_FUNCTION_INFO_V1(custom_out);
+PG_FUNCTION_INFO_V1(custom_subscripting_handler);
+
+/*****************************************************************************
+ * Input/Output functions
+ *****************************************************************************/
+
+Datum
+custom_in(PG_FUNCTION_ARGS)
+{
+ char *str = PG_GETARG_CSTRING(0);
+ int firstValue,
+ secondValue;
+ Custom *result;
+
+ if (sscanf(str, " ( %d , %d )", &firstValue, &secondValue) != 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
+ errmsg("invalid input syntax for complex: \"%s\"",
+ str)));
+
+
+ result = (Custom *) palloc(sizeof(Custom));
+ result->first = firstValue;
+ result->second = secondValue;
+ PG_RETURN_POINTER(result);
+}
+
+Datum
+custom_out(PG_FUNCTION_ARGS)
+{
+ Custom *custom = (Custom *) PG_GETARG_POINTER(0);
+ char *result;
+
+ result = psprintf("(%d, %d)", custom->first, custom->second);
+ PG_RETURN_CSTRING(result);
+}
+
+/*****************************************************************************
+ * Custom subscripting logic functions
+ *****************************************************************************/
+
+Datum
+custom_subscripting_handler(PG_FUNCTION_ARGS)
+{
+ SubscriptRoutines *sbsroutines = (SubscriptRoutines *)
+ palloc(sizeof(SubscriptRoutines));
+
+ sbsroutines->prepare = custom_subscript_prepare;
+ sbsroutines->validate = custom_subscript_validate;
+ sbsroutines->fetch = custom_subscript_fetch;
+ sbsroutines->assign = custom_subscript_assign;
+
+ PG_RETURN_POINTER(sbsroutines);
+}
+
+SubscriptingRef *
+custom_subscript_prepare(bool isAssignment, SubscriptingRef *sbsref)
+{
+ sbsref->refelemtype = INT4OID;
+ sbsref->refassgntype = INT4OID;
+ return sbsref;
+}
+
+SubscriptingRef *
+custom_subscript_validate(bool isAssignment, SubscriptingRef *sbsref,
+ ParseState *pstate)
+{
+ List *upperIndexpr = NIL;
+ ListCell *l;
+
+ if (sbsref->reflowerindexpr != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *)lfirst(sbsref->reflowerindexpr->head))))));
+
+ foreach(l, sbsref->refupperindexpr)
+ {
+ Node *subexpr = (Node *) lfirst(l);
+
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript does not support slices"),
+ parser_errposition(pstate, exprLocation(
+ ((Node *) lfirst(sbsref->refupperindexpr->head))))));
+
+ subexpr = coerce_to_target_type(pstate,
+ subexpr, exprType(subexpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (subexpr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom subscript must have integer type"),
+ parser_errposition(pstate, exprLocation(subexpr))));
+
+ upperIndexpr = lappend(upperIndexpr, subexpr);
+
+ if (isAssignment)
+ {
+ Node *assignExpr = (Node *) sbsref->refassgnexpr;
+ Node *new_from;
+
+ new_from = coerce_to_target_type(pstate,
+ assignExpr, exprType(assignExpr),
+ INT4OID, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+ if (new_from == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("custom assignment requires int type"),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation(assignExpr))));
+ sbsref->refassgnexpr = (Expr *)new_from;
+ }
+ }
+
+ sbsref->refupperindexpr = upperIndexpr;
+
+ return sbsref;
+}
+
+Datum
+custom_subscript_fetch(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ Custom *container= (Custom *) containerSource;
+ int index;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ return (Datum) container->first;
+ else
+ return (Datum) container->second;
+}
+
+Datum
+custom_subscript_assign(Datum containerSource, SubscriptingRefState *sbstate)
+{
+ int index;
+ Custom *container = (Custom *) containerSource;
+
+ if (sbstate->resnull)
+ return containerSource;
+
+ if (sbstate->numupper != 1)
+ ereport(ERROR, (errmsg("custom does not support nested subscripting")));
+
+ index = DatumGetInt32(sbstate->upperindex[0]);
+
+ if (index == 1)
+ container->first = DatumGetInt32(sbstate->replacevalue);
+ else
+ container->second = DatumGetInt32(sbstate->replacevalue);
+
+ return (Datum) container;
+}
diff --git a/src/tutorial/subscripting.source b/src/tutorial/subscripting.source
new file mode 100644
index 0000000000..837cf30612
--- /dev/null
+++ b/src/tutorial/subscripting.source
@@ -0,0 +1,71 @@
+---------------------------------------------------------------------------
+--
+-- subscripting.sql-
+-- This file shows how to create a new subscripting procedure for
+-- user-defined type.
+--
+--
+-- Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+-- Portions Copyright (c) 1994, Regents of the University of California
+--
+-- src/tutorial/subscripting.source
+--
+---------------------------------------------------------------------------
+
+-----------------------------
+-- Creating a new type:
+-- We are going to create a new type called 'complex' which represents
+-- complex numbers.
+-- A user-defined type must have an input and an output function, and
+-- optionally can have binary input and output functions. All of these
+-- are usually user-defined C functions.
+-----------------------------
+
+-- Assume the user defined functions are in /home/erthalion/programms/postgresql-master/src/tutorial/complex$DLSUFFIX
+-- (we do not want to assume this is in the dynamic loader search path).
+-- Look at $PWD/complex.c for the source. Note that we declare all of
+-- them as STRICT, so we do not need to cope with NULL inputs in the
+-- C code. We also mark them IMMUTABLE, since they always return the
+-- same outputs given the same inputs.
+
+-- the input function 'complex_in' takes a null-terminated string (the
+-- textual representation of the type) and turns it into the internal
+-- (in memory) representation. You will get a message telling you 'complex'
+-- does not exist yet but that's okay.
+
+CREATE FUNCTION custom_in(cstring)
+ RETURNS custom
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+-- the output function 'complex_out' takes the internal representation and
+-- converts it into the textual representation.
+
+CREATE FUNCTION custom_out(custom)
+ RETURNS cstring
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION custom_subscripting_handler(internal)
+ RETURNS internal
+ AS '_OBJWD_/subscripting'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE TYPE custom (
+ internallength = 8,
+ input = custom_in,
+ output = custom_out,
+ subscripting_handler = custom_subscripting_handler
+);
+
+-- we can use it in a table
+
+CREATE TABLE test_subscripting (
+ data custom
+);
+
+INSERT INTO test_subscripting VALUES ('(1, 2)');
+
+SELECT data[0] from test_subscripting;
+
+UPDATE test_subscripting SET data[1] = 3;
--
2.21.0
--u66dlpg6gaevsofr
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v33-0005-Filling-gaps-in-jsonb-arrays.patch"
^ permalink raw reply [nested|flat] 55+ messages in thread
* Re: Combine Prune and Freeze records emitted by vacuum
@ 2024-03-20 01:36 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 55+ messages in thread
From: Melanie Plageman @ 2024-03-20 01:36 UTC (permalink / raw)
To: Heikki Linnakangas <[email protected]>; +Cc: pgsql-hackers; Andres Freund <[email protected]>; Robert Haas <[email protected]>; Peter Geoghegan <[email protected]>
On Mon, Mar 18, 2024 at 01:15:21AM +0200, Heikki Linnakangas wrote:
> On 15/03/2024 02:56, Melanie Plageman wrote:
> > Okay, so I was going to start using xl_heap_prune for vacuum here too,
> > but I realized it would be bigger because of the
> > snapshotConflictHorizon. Do you think there is a non-terrible way to
> > make the snapshotConflictHorizon optional? Like with a flag?
>
> Yeah, another flag would do the trick.
Okay, I've done this in attached v4 (including removing
XLOG_HEAP2_VACUUM). I had to put the snapshot conflict horizon in the
"main chunk" of data available at replay regardless of whether or not
the record ended up including an FPI.
I made it its own sub-record (xlhp_conflict_horizon) less to help with
alignment (though we can use all the help we can get there) and more to
keep it from getting lost. When you look at heapam_xlog.h, you can see
what a XLOG_HEAP2_PRUNE record will contain starting with the
xl_heap_prune struct and then all the sub-record types.
> > > I introduced a few sub-record types similar to what you suggested --
> > > they help a bit with alignment, so I think they are worth keeping. There
> > > are comments around them, but perhaps a larger diagram of the layout of
> > > a the new XLOG_HEAP2_PRUNE record would be helpful.
> >
> > I started doing this, but I can't find a way of laying out the diagram
> > that pgindent doesn't destroy. I thought I remember other diagrams in
> > the source code showing the layout of something (something with pages
> > somewhere?) that don't get messed up by pgindent, but I couldn't find
> > them.
>
> See src/tools/pgindent/README, section "Cleaning up in case of failure or
> ugly output":
>
> /*----------
> * Text here will not be touched by pgindent.
> */
>
Cool. This version doesn't include the spiffy drawing I promised yet.
> Note that we don't provide WAL compatibility across major versions. You can
> fully remove the old xl_heap_freeze_page format. (We should bump
> XLOG_PAGE_MAGIC when this is committed though)
I've removed the xl_heap_freeze (and xl_heap_prune). I didn't bump
XLOG_PAGE_MAGIC.
> > > On the point of removing the freeze-only code path from
> > > heap_page_prune() (now heap_page_prune_and_freeze()): while doing this,
> > > I realized that heap_pre_freeze_checks() was not being called in the
> > > case that we decide to freeze because we emitted an FPI while setting
> > > the hint bit. I've fixed that, however, I've done so by moving
> > > heap_pre_freeze_checks() into the critical section. I think that is not
> > > okay? I could move it earlier and not do call it when the hint bit FPI
> > > leads us to freeze tuples. But, I think that would lead to us doing a
> > > lot less validation of tuples being frozen when checksums are enabled.
> > > Or, I could make two critical sections?
> >
> > I found another approach and just do the pre-freeze checks if we are
> > considering freezing except for the FPI criteria.
>
> Hmm, I think you can make this simpler if you use
> XLogCheckBufferNeedsBackup() to check if the hint-bit update will generate a
> full-page-image before entering the critical section. Like you did to check
> if pruning will generate a full-page-image. I included that change in the
> attached patches.
I used your proposed structure. You had XLogCheckBufferNeedsBackup()
twice in your proposed version a few lines apart. I don't think there is
any point in checking it twice. If we are going to rely on
XLogCheckBufferNeedsBackup() to tell us whether or not setting the hint
bit is *likely* to emit an FPI, then we might as well just call
XLogCheckBufferNeedsBackup() once.
> I don't fully understand this:
>
> > /*
> > * If we will freeze tuples on the page or they were all already frozen
> > * on the page, if we will set the page all-frozen in the visibility map,
> > * we can advance relfrozenxid and relminmxid to the values in
> > * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid.
> > */
> > if (presult->all_frozen || presult->nfrozen > 0)
> > {
> > presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
> > presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
> > }
> > else
> > {
> > presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
> > presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
> > }
>
> Firstly, the comment is outdated, because we have already done the freezing
> at this point. But more importantly, I don't understand the logic here.
> Could we check just presult->nfrozen > 0 here and get the same result?
I've updated the comment. The reason I had
if (presult->all_frozen || presult->nfrozen > 0) was because of this
comment in heapam.h in the HeapPageFreeze struct
* "Freeze" NewRelfrozenXid/NewRelminMxid trackers.
*
* Trackers used when heap_freeze_execute_prepared freezes, or when there
* are zero freeze plans for a page. It is always valid for vacuumlazy.c
* to freeze any page, by definition. This even includes pages that have
* no tuples with storage to consider in the first place. That way the
* 'totally_frozen' results from heap_prepare_freeze_tuple can always be
* used in the same way, even when no freeze plans need to be executed to
* "freeze the page". Only the "freeze" path needs to consider the need
* to set pages all-frozen in the visibility map under this scheme.
*
* When we freeze a page, we generally freeze all XIDs < OldestXmin, only
* leaving behind XIDs that are ineligible for freezing, if any. And so
* you might wonder why these trackers are necessary at all; why should
* _any_ page that VACUUM freezes _ever_ be left with XIDs/MXIDs that
* ratchet back the top-level NewRelfrozenXid/NewRelminMxid trackers?
*
* It is useful to use a definition of "freeze the page" that does not
* overspecify how MultiXacts are affected. heap_prepare_freeze_tuple
* generally prefers to remove Multis eagerly, but lazy processing is used
* in cases where laziness allows VACUUM to avoid allocating a new Multi.
* The "freeze the page" trackers enable this flexibility.
*/
So, I don't really know if it is right to just check presult->nfrozen >
0 when updating relminmxid. I have changed it to the way you suggested.
But we can change it back.
> Here are some patches on top of your patches for some further refactorings.
> Some notable changes in heap_page_prune_and_freeze():
>
> - I moved the heap_prepare_freeze_tuple() work from the 2nd loop to the 1st
> one. It seems more clear and efficient that way.
cool. I kept this.
> - I extracted the code to generate the WAL record to a separate function.
cool. kept this.
> These patches are in a very rough WIP state, but I wanted to share early. I
> haven't done much testing, and I'm not wedded to these changes, but I think
> they make it more readable. Please include / squash in the patch set if you
> agree with them.
I've squashed the changes into and across my nineteen patches :)
I cleaned up your sugestions a bit and made a few stylistic choices.
In this version, I also broke up the last couple commits which
streamlined the WAL record and eliminated XLOG_HEAP2_FREEZE/VACUUM and
redistributed those changes in a way that I thought made sense.
Now, the progression is that in one commit we merge the prune and freeze
record, eliminating the XLOG_HEAP2_FREEZE record. Then, in another
commit, we eliminate the XLOG_HEAP2_VACUUM record. Then a later commit
streamlines the new mega xl_heap_prune struct into the variable size
structure based on which modifications it includes.
> From 622620a7875ae8c1626e9cd118156e0c734d44ed Mon Sep 17 00:00:00 2001
> From: Heikki Linnakangas <[email protected]>
> Date: Sun, 17 Mar 2024 22:52:28 +0200
> Subject: [PATCH v3heikki 1/4] Inline heap_frz_conflict_horizon() to the
> caller.
>
> FIXME: This frz_conflict_horizon business looks fishy to me. We have:
> - local frz_conflict_horizon variable,
> - presult->frz_conflict_horizon, and
> - prstate.snapshotConflictHorizon
Yea, this is a mistake I made when I was rebasing some changes in. The
local variable is gone now.
> should we really have all three, and what are the differences?
We do need both the prstate.snapshotConflictHorizon and the
presult->frz_conflict_horizon because the youngest freezable xmin will
often be different than the oldest removable xmax, so we have to track
both and take the most conservative one if we are both freezing and
pruning.
The third (local variable) one was an oops.
> From 0219842487931f899abcf183c863c43270c098f0 Mon Sep 17 00:00:00 2001
> From: Heikki Linnakangas <[email protected]>
> Date: Sun, 17 Mar 2024 23:05:31 +0200
> Subject: [PATCH v3heikki 2/4] Misc cleanup
> ---
> src/backend/access/heap/pruneheap.c | 16 +++++++---------
> src/backend/access/heap/vacuumlazy.c | 7 ++++++-
> 2 files changed, 13 insertions(+), 10 deletions(-)
> diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
> index 8d3723faf3a..d3df7029667 100644
> --- a/src/backend/access/heap/vacuumlazy.c
> +++ b/src/backend/access/heap/vacuumlazy.c
> @@ -433,7 +433,9 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
> * heap_page_prune_and_freeze(). We expect vistest will always make
> * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is <
> * OldestXmin. lazy_scan_prune must never become confused about whether a
> - * tuple should be frozen or removed. (In the future we might want to
> + * tuple should be frozen or removed.
> + * HEIKKI: Is such confusion possible anymore?
> + * (In the future we might want to
> * teach lazy_scan_prune to recompute vistest from time to time, to
> * increase the number of dead tuples it can prune away.)
TBH, I don't really know what this comment is even saying. But
lazy_scan_prune() doesn't do any freezing anymore, so I removed this
sentence.
> */
> @@ -1387,6 +1389,9 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
> * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
> * left with storage after pruning.
> *
> + * HEIKKI: does the above paragraph still make sense? We don't call
> + * HeapTupleSatisfiesVacuum() here anymore
> + *
Yea this whole comment definitely doesn't belong here anymore. Even
though we are calling HeapTupleSatisfiesVacuum() (from inside
heap_prune_satisifes_vacuum()) inside heap_page_prune_and_freeze(), the
comment really doesn't fit anywhere in there either. The comment is
describing a situation that is no longer possible. So describing a
situation that is no longer possible in a part of the code that it never
could have been possible does not make sense to me. I've removed the
comment.
> * As of Postgres 17, we circumvent this problem altogether by reusing the
> * result of heap_page_prune_and_freeze()'s visibility check. Without the
> * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
> --
> 2.39.2
>
> From d72cebf13f9866112309883f72a382fc2cb57e17 Mon Sep 17 00:00:00 2001
> From: Heikki Linnakangas <[email protected]>
> Date: Sun, 17 Mar 2024 23:08:42 +0200
> Subject: [PATCH v3heikki 3/4] Move work to the first loop
> src/backend/access/heap/pruneheap.c | 141 ++++++++++------------------
> 1 file changed, 52 insertions(+), 89 deletions(-)
>
> diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
> index b3573bb628b..3541628799a 100644
> --- a/src/backend/access/heap/pruneheap.c
> +++ b/src/backend/access/heap/pruneheap.c
> @@ -78,9 +78,6 @@ static int heap_prune_chain(Buffer buffer,
> PruneState *prstate, PruneFreezeResult *presult);
>
> static inline HTSV_Result htsv_get_valid_status(int status);
> -static void prune_prepare_freeze_tuple(Page page, OffsetNumber offnum, PruneState *prstate,
> - HeapPageFreeze *pagefrz, HeapTupleFreeze *frozen,
> - PruneFreezeResult *presult);
> static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
> static void heap_prune_record_redirect(PruneState *prstate,
> OffsetNumber offnum, OffsetNumber rdoffnum,
> @@ -322,6 +319,16 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
> /* for recovery conflicts */
> presult->frz_conflict_horizon = InvalidTransactionId;
>
> + /*
> + * We will update the VM after pruning, collecting LP_DEAD items, and
> + * freezing tuples. Keep track of whether or not the page is all_visible
> + * and all_frozen and use this information to update the VM. all_visible
> + * implies lpdead_items == 0, but don't trust all_frozen result unless
> + * all_visible is also set to true.
> + */
> + /* HEIKKI: the caller updates the VM? not us */
I've updated this comment.
Other questions and notes on v4:
xl_heap_prune->flags is a uint8, but we are already using 7 of the bits.
Should we make it a unit16?
Eventually, I would like to avoid emitting a separate XLOG_HEAP2_VISIBLE
record for vacuum's first and second passes and just include the VM
update flags in the xl_heap_prune record. xl_heap_visible->flags is a
uint8. If we made xl_heap_prune->flags uint16, we could probably combine
them (though maybe we want other bits available). Also vacuum's second
pass doesn't set a snapshotConflictHorizon, so if we combined
xl_heap_visible and xl_heap_prune for vacuum we would end up saving even
more space (since vacuum sets xl_heap_visible->snapshotConflictHorizon
to InvalidXLogRecPtr).
A note on sub-record naming: I kept xl_heap_freeze_plan's name but
prefixed the other sub-records with xlhp. Do you think it is worth
renaming it (to xlhp_freeze_plan)? Also, should I change xlhp_freeze to
xlhp_freeze_page?
- Melanie
Attachments:
[text/x-diff] v4-0001-Reorganize-heap_page_prune-function-comment.patch (3.6K, ../../20240320013602.6sypr4cx6sefpemg@liskov/2-v4-0001-Reorganize-heap_page_prune-function-comment.patch)
download | inline diff:
From d1b1ce71c88a04e3c5a2480827bd3af3ffbe1372 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Mon, 18 Mar 2024 19:01:17 -0400
Subject: [PATCH v4 01/19] Reorganize heap_page_prune() function comment
heap_page_prune()'s function header comment didn't explain the
parameters in the same order they appear in the function. Fix that.
While we are at it, move some parts of the initial function body
comments around so they are in more relevant locations.
---
src/backend/access/heap/pruneheap.c | 30 ++++++++++++++---------------
1 file changed, 15 insertions(+), 15 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 4f12413b8b1..b5895406ec2 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -193,22 +193,26 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*
* Caller must have pin and buffer cleanup lock on the page. Note that we
* don't update the FSM information for page on caller's behalf. Caller might
- * also need to account for a reduction in the length of the line pointer
- * array following array truncation by us.
+ * also need to account for a reduction in the length of the line pointer array
+ * following array truncation by us.
*
- * vistest is used to distinguish whether tuples are DEAD or RECENTLY_DEAD
- * (see heap_prune_satisfies_vacuum and
- * HeapTupleSatisfiesVacuum).
+ * Our strategy is to scan the page and make lists of items to change, then
+ * apply the changes within a critical section. This keeps as much logic as
+ * possible out of the critical section, and also ensures that WAL replay will
+ * work the same as the normal case.
*
- * mark_unused_now indicates whether or not dead items can be set LP_UNUSED during
- * pruning.
+ * vistest is used to distinguish whether tuples are DEAD or RECENTLY_DEAD (see
+ * heap_prune_satisfies_vacuum and HeapTupleSatisfiesVacuum).
*
- * off_loc is the offset location required by the caller to use in error
- * callback.
+ * mark_unused_now indicates whether or not dead items can be set LP_UNUSED
+ * during pruning.
*
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
* heap_page_prune() is responsible for initializing it.
+ *
+ * off_loc is the offset location required by the caller to use in error
+ * callback.
*/
void
heap_page_prune(Relation relation, Buffer buffer,
@@ -225,11 +229,6 @@ heap_page_prune(Relation relation, Buffer buffer,
HeapTupleData tup;
/*
- * Our strategy is to scan the page and make lists of items to change,
- * then apply the changes within a critical section. This keeps as much
- * logic as possible out of the critical section, and also ensures that
- * WAL replay will work the same as the normal case.
- *
* First, initialize the new pd_prune_xid value to zero (indicating no
* prunable tuples). If we find any tuples which may soon become
* prunable, we will save the lowest relevant XID in new_prune_xid. Also
@@ -241,12 +240,13 @@ heap_page_prune(Relation relation, Buffer buffer,
prstate.mark_unused_now = mark_unused_now;
prstate.snapshotConflictHorizon = InvalidTransactionId;
prstate.nredirected = prstate.ndead = prstate.nunused = 0;
- memset(prstate.marked, 0, sizeof(prstate.marked));
/*
* presult->htsv is not initialized here because all ntuple spots in the
* array will be set either to a valid HTSV_Result value or -1.
*/
+ memset(prstate.marked, 0, sizeof(prstate.marked));
+
presult->ndeleted = 0;
presult->nnewlpdead = 0;
--
2.40.1
[text/x-diff] v4-0002-Remove-unused-PruneState-member-rel.patch (1.4K, ../../20240320013602.6sypr4cx6sefpemg@liskov/3-v4-0002-Remove-unused-PruneState-member-rel.patch)
download | inline diff:
From f7c3a7680e144e55f8c1c3eccaa5a7a1af47a47e Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Mon, 18 Mar 2024 18:59:09 -0400
Subject: [PATCH v4 02/19] Remove unused PruneState member rel
PruneState->rel is no longer being used, so just remove it.
---
src/backend/access/heap/pruneheap.c | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index b5895406ec2..08cb2a6f533 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -29,8 +29,6 @@
/* Working data for heap_page_prune and subroutines */
typedef struct
{
- Relation rel;
-
/* tuple visibility test, initialized for the relation */
GlobalVisState *vistest;
/* whether or not dead items can be set LP_UNUSED during pruning */
@@ -235,7 +233,6 @@ heap_page_prune(Relation relation, Buffer buffer,
* initialize the rest of our working state.
*/
prstate.new_prune_xid = InvalidTransactionId;
- prstate.rel = relation;
prstate.vistest = vistest;
prstate.mark_unused_now = mark_unused_now;
prstate.snapshotConflictHorizon = InvalidTransactionId;
@@ -251,7 +248,7 @@ heap_page_prune(Relation relation, Buffer buffer,
presult->nnewlpdead = 0;
maxoff = PageGetMaxOffsetNumber(page);
- tup.t_tableOid = RelationGetRelid(prstate.rel);
+ tup.t_tableOid = RelationGetRelid(relation);
/*
* Determine HTSV for all tuples.
--
2.40.1
[text/x-diff] v4-0003-lazy_scan_prune-tests-tuple-vis-with-GlobalVisTes.patch (2.1K, ../../20240320013602.6sypr4cx6sefpemg@liskov/4-v4-0003-lazy_scan_prune-tests-tuple-vis-with-GlobalVisTes.patch)
download | inline diff:
From 416650477aa7a2da4db57070fcb2c3524093bf09 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Sat, 6 Jan 2024 13:14:47 -0500
Subject: [PATCH v4 03/19] lazy_scan_prune tests tuple vis with GlobalVisTest
One requirement for eventually combining the prune and freeze records,
is that we must check during pruning if live tuples on the page are
visible to everyone and thus, whether or not the page is all visible. We
only consider opportunistically freezing tuples if the whole page is all
visible and could be set all frozen.
During pruning (in heap_page_prune()), we do not have access to
VacuumCutoffs -- as on access pruning also calls heap_page_prune(). We
do, however, have access to a GlobalVisState. This can be used to
determine whether or not the tuple is visible to everyone. It also has
the potential of being more up-to-date than VacuumCutoffs->OldestXmin.
This commit simply modifies lazy_scan_prune() to use GlobalVisState
instead of OldestXmin. Future commits will move the
all_visible/all_frozen calculation into heap_page_prune().
---
src/backend/access/heap/vacuumlazy.c | 10 +++++++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 18004907750..3a991f0ea71 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -1579,11 +1579,15 @@ lazy_scan_prune(LVRelState *vacrel,
/*
* The inserter definitely committed. But is it old enough
- * that everyone sees it as committed?
+ * that everyone sees it as committed? A
+ * FrozenTransactionId is seen as committed to everyone.
+ * Otherwise, we check if there is a snapshot that
+ * considers this xid to still be running, and if so, we
+ * don't consider the page all-visible.
*/
xmin = HeapTupleHeaderGetXmin(htup);
- if (!TransactionIdPrecedes(xmin,
- vacrel->cutoffs.OldestXmin))
+ if (xmin != FrozenTransactionId &&
+ !GlobalVisTestIsRemovableXid(vacrel->vistest, xmin))
{
all_visible = false;
break;
--
2.40.1
[text/x-diff] v4-0004-Pass-heap_prune_chain-PruneResult-output-paramete.patch (3.3K, ../../20240320013602.6sypr4cx6sefpemg@liskov/5-v4-0004-Pass-heap_prune_chain-PruneResult-output-paramete.patch)
download | inline diff:
From 94c0256a9c7e11001fcf4dec8998943e0b326f50 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Sat, 6 Jan 2024 13:39:59 -0500
Subject: [PATCH v4 04/19] Pass heap_prune_chain() PruneResult output parameter
Future commits will set other members of PruneResult in
heap_prune_chain(), so start passing it as an output parameter now. This
eliminates the output parameter htsv -- the array of HTSV_Results --
since that is a member of the PruneResult.
---
src/backend/access/heap/pruneheap.c | 15 +++++++--------
1 file changed, 7 insertions(+), 8 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 08cb2a6f533..7eb21b603ba 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -59,8 +59,7 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- int8 *htsv,
- PruneState *prstate);
+ PruneState *prstate, PruneResult *presult);
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
@@ -322,7 +321,7 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Process this item or chain of items */
presult->ndeleted += heap_prune_chain(buffer, offnum,
- presult->htsv, &prstate);
+ &prstate, presult);
}
/* Clear the offset information once we have processed the given page. */
@@ -451,7 +450,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
/*
* Prune specified line pointer or a HOT chain originating at line pointer.
*
- * Tuple visibility information is provided in htsv.
+ * Tuple visibility information is provided in presult->htsv.
*
* If the item is an index-referenced tuple (i.e. not a heap-only tuple),
* the HOT chain is pruned by removing all DEAD tuples at the start of the HOT
@@ -481,7 +480,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
*/
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
- int8 *htsv, PruneState *prstate)
+ PruneState *prstate, PruneResult *presult)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -502,7 +501,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
*/
if (ItemIdIsNormal(rootlp))
{
- Assert(htsv[rootoffnum] != -1);
+ Assert(presult->htsv[rootoffnum] != -1);
htup = (HeapTupleHeader) PageGetItem(dp, rootlp);
if (HeapTupleHeaderIsHeapOnly(htup))
@@ -525,7 +524,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
* either here or while following a chain below. Whichever path
* gets there first will mark the tuple unused.
*/
- if (htsv[rootoffnum] == HEAPTUPLE_DEAD &&
+ if (presult->htsv[rootoffnum] == HEAPTUPLE_DEAD &&
!HeapTupleHeaderIsHotUpdated(htup))
{
heap_prune_record_unused(prstate, rootoffnum);
@@ -622,7 +621,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
*/
tupdead = recent_dead = false;
- switch (htsv_get_valid_status(htsv[offnum]))
+ switch (htsv_get_valid_status(presult->htsv[offnum]))
{
case HEAPTUPLE_DEAD:
tupdead = true;
--
2.40.1
[text/x-diff] v4-0005-heap_page_prune-sets-all_visible-and-frz_conflict.patch (18.8K, ../../20240320013602.6sypr4cx6sefpemg@liskov/6-v4-0005-heap_page_prune-sets-all_visible-and-frz_conflict.patch)
download | inline diff:
From caecf591376392a5518cb42923542f380e1c327c Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Sat, 6 Jan 2024 14:01:37 -0500
Subject: [PATCH v4 05/19] heap_page_prune sets all_visible and
frz_conflict_horizon
In order to combine the prune and freeze records, we must know if the
page is eligible to be opportunistically frozen before finishing
pruning. Save all_visible in the PruneResult and set it to false when we
see non-removable tuples which are not visible to everyone.
We will also need to ensure that the snapshotConflictHorizon for the combined
prune + freeze record is the more conservative of that calculated for each of
pruning and freezing. Calculate the visibility_cutoff_xid for the purposes of
freezing -- the newest xmin on the page -- in heap_page_prune() and save it in
PruneResult.frz_conflict_horizon.
---
src/backend/access/heap/pruneheap.c | 127 +++++++++++++++++++++++++--
src/backend/access/heap/vacuumlazy.c | 121 ++++++-------------------
src/include/access/heapam.h | 3 +
3 files changed, 151 insertions(+), 100 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 7eb21b603ba..bd30296ef1a 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -63,8 +63,10 @@ static int heap_prune_chain(Buffer buffer,
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
-static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum);
-static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum);
+static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
+ PruneResult *presult);
+static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
+ PruneResult *presult);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
@@ -246,6 +248,14 @@ heap_page_prune(Relation relation, Buffer buffer,
presult->ndeleted = 0;
presult->nnewlpdead = 0;
+ /*
+ * Keep track of whether or not the page is all_visible in case the caller
+ * wants to use this information to update the VM.
+ */
+ presult->all_visible = true;
+ /* for recovery conflicts */
+ presult->frz_conflict_horizon = InvalidTransactionId;
+
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(relation);
@@ -297,8 +307,97 @@ heap_page_prune(Relation relation, Buffer buffer,
presult->htsv[offnum] = heap_prune_satisfies_vacuum(&prstate, &tup,
buffer);
+ switch (presult->htsv[offnum])
+ {
+ case HEAPTUPLE_DEAD:
+
+ /*
+ * Deliberately delay unsetting all_visible until later during
+ * pruning. Removable dead tuples shouldn't preclude freezing
+ * the page. After finishing this first pass of tuple
+ * visibility checks, initialize all_visible_except_removable
+ * with the current value of all_visible to indicate whether
+ * or not the page is all visible except for dead tuples. This
+ * will allow us to attempt to freeze the page after pruning.
+ * Later during pruning, if we encounter an LP_DEAD item or
+ * are setting an item LP_DEAD, we will unset all_visible. As
+ * long as we unset it before updating the visibility map,
+ * this will be correct.
+ */
+ break;
+ case HEAPTUPLE_LIVE:
+
+ /*
+ * Is the tuple definitely visible to all transactions?
+ *
+ * NB: Like with per-tuple hint bits, we can't set the
+ * PD_ALL_VISIBLE flag if the inserter committed
+ * asynchronously. See SetHintBits for more info. Check that
+ * the tuple is hinted xmin-committed because of that.
+ */
+ if (presult->all_visible)
+ {
+ TransactionId xmin;
+
+ if (!HeapTupleHeaderXminCommitted(htup))
+ {
+ presult->all_visible = false;
+ break;
+ }
+
+ /*
+ * The inserter definitely committed. But is it old enough
+ * that everyone sees it as committed? A
+ * FrozenTransactionId is seen as committed to everyone.
+ * Otherwise, we check if there is a snapshot that
+ * considers this xid to still be running, and if so, we
+ * don't consider the page all-visible.
+ */
+ xmin = HeapTupleHeaderGetXmin(htup);
+ if (xmin != FrozenTransactionId &&
+ !GlobalVisTestIsRemovableXid(vistest, xmin))
+ {
+ presult->all_visible = false;
+ break;
+ }
+
+ /* Track newest xmin on page. */
+ if (TransactionIdFollows(xmin, presult->frz_conflict_horizon) &&
+ TransactionIdIsNormal(xmin))
+ presult->frz_conflict_horizon = xmin;
+ }
+ break;
+ case HEAPTUPLE_RECENTLY_DEAD:
+ presult->all_visible = false;
+ break;
+ case HEAPTUPLE_INSERT_IN_PROGRESS:
+ presult->all_visible = false;
+ break;
+ case HEAPTUPLE_DELETE_IN_PROGRESS:
+ /* This is an expected case during concurrent vacuum */
+ presult->all_visible = false;
+ break;
+ default:
+ elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result");
+ break;
+ }
}
+ /*
+ * For vacuum, if the whole page will become frozen, we consider
+ * opportunistically freezing tuples. Dead tuples which will be removed by
+ * the end of vacuuming should not preclude us from opportunistically
+ * freezing. We will not be able to freeze the whole page if there are
+ * tuples present which are not visible to everyone or if there are dead
+ * tuples which are not yet removable. We need all_visible to be false if
+ * LP_DEAD tuples remain after pruning so that we do not incorrectly
+ * update the visibility map or page hint bit. So, we will update
+ * presult->all_visible to reflect the presence of LP_DEAD items while
+ * pruning and keep all_visible_except_removable to permit freezing if the
+ * whole page will eventually become all visible after removing tuples.
+ */
+ presult->all_visible_except_removable = presult->all_visible;
+
/* Scan the page */
for (offnum = FirstOffsetNumber;
offnum <= maxoff;
@@ -593,10 +692,14 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
/*
* If the caller set mark_unused_now true, we can set dead line
* pointers LP_UNUSED now. We don't increment ndeleted here since
- * the LP was already marked dead.
+ * the LP was already marked dead. If it will not be marked
+ * LP_UNUSED, it will remain LP_DEAD, making the page not
+ * all_visible.
*/
if (unlikely(prstate->mark_unused_now))
heap_prune_record_unused(prstate, offnum);
+ else
+ presult->all_visible = false;
break;
}
@@ -733,7 +836,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
* redirect the root to the correct chain member.
*/
if (i >= nchain)
- heap_prune_record_dead_or_unused(prstate, rootoffnum);
+ heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
else
heap_prune_record_redirect(prstate, rootoffnum, chainitems[i]);
}
@@ -746,7 +849,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
* redirect item. We can clean up by setting the redirect item to
* DEAD state or LP_UNUSED if the caller indicated.
*/
- heap_prune_record_dead_or_unused(prstate, rootoffnum);
+ heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
}
return ndeleted;
@@ -783,13 +886,20 @@ heap_prune_record_redirect(PruneState *prstate,
/* Record line pointer to be marked dead */
static void
-heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum)
+heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
+ PruneResult *presult)
{
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
prstate->ndead++;
Assert(!prstate->marked[offnum]);
prstate->marked[offnum] = true;
+
+ /*
+ * Setting the line pointer LP_DEAD means the page will definitely not be
+ * all_visible.
+ */
+ presult->all_visible = false;
}
/*
@@ -799,7 +909,8 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum)
* pointers LP_DEAD if mark_unused_now is true.
*/
static void
-heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum)
+heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
+ PruneResult *presult)
{
/*
* If the caller set mark_unused_now to true, we can remove dead tuples
@@ -810,7 +921,7 @@ heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum)
if (unlikely(prstate->mark_unused_now))
heap_prune_record_unused(prstate, offnum);
else
- heap_prune_record_dead(prstate, offnum);
+ heap_prune_record_dead(prstate, offnum, presult);
}
/* Record line pointer to be marked unused */
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 3a991f0ea71..f9892f4cd08 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -1422,9 +1422,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool all_visible,
- all_frozen;
- TransactionId visibility_cutoff_xid;
+ bool all_frozen;
int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
@@ -1465,17 +1463,16 @@ lazy_scan_prune(LVRelState *vacrel,
&presult, &vacrel->offnum);
/*
- * We will update the VM after collecting LP_DEAD items and freezing
- * tuples. Keep track of whether or not the page is all_visible and
- * all_frozen and use this information to update the VM. all_visible
- * implies 0 lpdead_items, but don't trust all_frozen result unless
- * all_visible is also set to true.
+ * Now scan the page to collect LP_DEAD items and check for tuples
+ * requiring freezing among remaining tuples with storage. We will update
+ * the VM after collecting LP_DEAD items and freezing tuples. Pruning will
+ * have determined whether or not the page is all_visible. Keep track of
+ * whether or not the page is all_frozen and use this information to
+ * update the VM. all_visible implies lpdead_items == 0, but don't trust
+ * all_frozen result unless all_visible is also set to true.
*
- * Also keep track of the visibility cutoff xid for recovery conflicts.
*/
- all_visible = true;
all_frozen = true;
- visibility_cutoff_xid = InvalidTransactionId;
/*
* Now scan the page to collect LP_DEAD items and update the variables set
@@ -1516,11 +1513,6 @@ lazy_scan_prune(LVRelState *vacrel,
* will only happen every other VACUUM, at most. Besides, VACUUM
* must treat hastup/nonempty_pages as provisional no matter how
* LP_DEAD items are handled (handled here, or handled later on).
- *
- * Also deliberately delay unsetting all_visible until just before
- * we return to lazy_scan_heap caller, as explained in full below.
- * (This is another case where it's useful to anticipate that any
- * LP_DEAD items will become LP_UNUSED during the ongoing VACUUM.)
*/
deadoffsets[lpdead_items++] = offnum;
continue;
@@ -1558,46 +1550,6 @@ lazy_scan_prune(LVRelState *vacrel,
* what acquire_sample_rows() does.
*/
live_tuples++;
-
- /*
- * Is the tuple definitely visible to all transactions?
- *
- * NB: Like with per-tuple hint bits, we can't set the
- * PD_ALL_VISIBLE flag if the inserter committed
- * asynchronously. See SetHintBits for more info. Check that
- * the tuple is hinted xmin-committed because of that.
- */
- if (all_visible)
- {
- TransactionId xmin;
-
- if (!HeapTupleHeaderXminCommitted(htup))
- {
- all_visible = false;
- break;
- }
-
- /*
- * The inserter definitely committed. But is it old enough
- * that everyone sees it as committed? A
- * FrozenTransactionId is seen as committed to everyone.
- * Otherwise, we check if there is a snapshot that
- * considers this xid to still be running, and if so, we
- * don't consider the page all-visible.
- */
- xmin = HeapTupleHeaderGetXmin(htup);
- if (xmin != FrozenTransactionId &&
- !GlobalVisTestIsRemovableXid(vacrel->vistest, xmin))
- {
- all_visible = false;
- break;
- }
-
- /* Track newest xmin on page. */
- if (TransactionIdFollows(xmin, visibility_cutoff_xid) &&
- TransactionIdIsNormal(xmin))
- visibility_cutoff_xid = xmin;
- }
break;
case HEAPTUPLE_RECENTLY_DEAD:
@@ -1607,7 +1559,6 @@ lazy_scan_prune(LVRelState *vacrel,
* pruning.)
*/
recently_dead_tuples++;
- all_visible = false;
break;
case HEAPTUPLE_INSERT_IN_PROGRESS:
@@ -1618,16 +1569,13 @@ lazy_scan_prune(LVRelState *vacrel,
* results. This assumption is a bit shaky, but it is what
* acquire_sample_rows() does, so be consistent.
*/
- all_visible = false;
break;
case HEAPTUPLE_DELETE_IN_PROGRESS:
- /* This is an expected case during concurrent vacuum */
- all_visible = false;
/*
- * Count such rows as live. As above, we assume the deleting
- * transaction will commit and update the counters after we
- * report.
+ * This an expected case during concurrent vacuum. Count such
+ * rows as live. As above, we assume the deleting transaction
+ * will commit and update the counters after we report.
*/
live_tuples++;
break;
@@ -1670,7 +1618,7 @@ lazy_scan_prune(LVRelState *vacrel,
* page all-frozen afterwards (might not happen until final heap pass).
*/
if (pagefrz.freeze_required || tuples_frozen == 0 ||
- (all_visible && all_frozen &&
+ (presult.all_visible_except_removable && all_frozen &&
fpi_before != pgWalUsage.wal_fpi))
{
/*
@@ -1703,16 +1651,16 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->frozen_pages++;
/*
- * We can use visibility_cutoff_xid as our cutoff for conflicts
+ * We can use frz_conflict_horizon as our cutoff for conflicts
* when the whole page is eligible to become all-frozen in the VM
* once we're done with it. Otherwise we generate a conservative
* cutoff by stepping back from OldestXmin.
*/
- if (all_visible && all_frozen)
+ if (presult.all_visible_except_removable && all_frozen)
{
/* Using same cutoff when setting VM is now unnecessary */
- snapshotConflictHorizon = visibility_cutoff_xid;
- visibility_cutoff_xid = InvalidTransactionId;
+ snapshotConflictHorizon = presult.frz_conflict_horizon;
+ presult.frz_conflict_horizon = InvalidTransactionId;
}
else
{
@@ -1748,17 +1696,19 @@ lazy_scan_prune(LVRelState *vacrel,
*/
#ifdef USE_ASSERT_CHECKING
/* Note that all_frozen value does not matter when !all_visible */
- if (all_visible && lpdead_items == 0)
+ if (presult.all_visible)
{
TransactionId debug_cutoff;
bool debug_all_frozen;
+ Assert(lpdead_items == 0);
+
if (!heap_page_is_all_visible(vacrel, buf,
&debug_cutoff, &debug_all_frozen))
Assert(false);
Assert(!TransactionIdIsValid(debug_cutoff) ||
- debug_cutoff == visibility_cutoff_xid);
+ debug_cutoff == presult.frz_conflict_horizon);
}
#endif
@@ -1783,19 +1733,6 @@ lazy_scan_prune(LVRelState *vacrel,
Assert(dead_items->num_items <= dead_items->max_items);
pgstat_progress_update_param(PROGRESS_VACUUM_NUM_DEAD_TUPLES,
dead_items->num_items);
-
- /*
- * It was convenient to ignore LP_DEAD items in all_visible earlier on
- * to make the choice of whether or not to freeze the page unaffected
- * by the short-term presence of LP_DEAD items. These LP_DEAD items
- * were effectively assumed to be LP_UNUSED items in the making. It
- * doesn't matter which heap pass (initial pass or final pass) ends up
- * setting the page all-frozen, as long as the ongoing VACUUM does it.
- *
- * Now that freezing has been finalized, unset all_visible. It needs
- * to reflect the present state of things, as expected by our caller.
- */
- all_visible = false;
}
/* Finally, add page-local counts to whole-VACUUM counts */
@@ -1812,20 +1749,20 @@ lazy_scan_prune(LVRelState *vacrel,
/* Did we find LP_DEAD items? */
*has_lpdead_items = (lpdead_items > 0);
- Assert(!all_visible || !(*has_lpdead_items));
+ Assert(!presult.all_visible || !(*has_lpdead_items));
/*
* Handle setting visibility map bit based on information from the VM (as
* of last heap_vac_scan_next_block() call), and from all_visible and
* all_frozen variables
*/
- if (!all_visible_according_to_vm && all_visible)
+ if (!all_visible_according_to_vm && presult.all_visible)
{
uint8 flags = VISIBILITYMAP_ALL_VISIBLE;
if (all_frozen)
{
- Assert(!TransactionIdIsValid(visibility_cutoff_xid));
+ Assert(!TransactionIdIsValid(presult.frz_conflict_horizon));
flags |= VISIBILITYMAP_ALL_FROZEN;
}
@@ -1845,7 +1782,7 @@ lazy_scan_prune(LVRelState *vacrel,
PageSetAllVisible(page);
MarkBufferDirty(buf);
visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
- vmbuffer, visibility_cutoff_xid,
+ vmbuffer, presult.frz_conflict_horizon,
flags);
}
@@ -1893,7 +1830,7 @@ lazy_scan_prune(LVRelState *vacrel,
* it as all-frozen. Note that all_frozen is only valid if all_visible is
* true, so we must check both all_visible and all_frozen.
*/
- else if (all_visible_according_to_vm && all_visible &&
+ else if (all_visible_according_to_vm && presult.all_visible &&
all_frozen && !VM_ALL_FROZEN(vacrel->rel, blkno, &vmbuffer))
{
/*
@@ -1910,11 +1847,11 @@ lazy_scan_prune(LVRelState *vacrel,
/*
* Set the page all-frozen (and all-visible) in the VM.
*
- * We can pass InvalidTransactionId as our visibility_cutoff_xid,
- * since a snapshotConflictHorizon sufficient to make everything safe
- * for REDO was logged when the page's tuples were frozen.
+ * We can pass InvalidTransactionId as our frz_conflict_horizon, since
+ * a snapshotConflictHorizon sufficient to make everything safe for
+ * REDO was logged when the page's tuples were frozen.
*/
- Assert(!TransactionIdIsValid(visibility_cutoff_xid));
+ Assert(!TransactionIdIsValid(presult.frz_conflict_horizon));
visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
vmbuffer, InvalidTransactionId,
VISIBILITYMAP_ALL_VISIBLE |
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 4b133f68593..d8e65ae7e35 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -198,6 +198,9 @@ typedef struct PruneResult
{
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
+ bool all_visible; /* Whether or not the page is all visible */
+ bool all_visible_except_removable;
+ TransactionId frz_conflict_horizon; /* Newest xmin on the page */
/*
* Tuple visibility is only computed once for each tuple, for correctness
--
2.40.1
[text/x-diff] v4-0006-Add-reference-to-VacuumCutoffs-in-HeapPageFreeze.patch (12.4K, ../../20240320013602.6sypr4cx6sefpemg@liskov/7-v4-0006-Add-reference-to-VacuumCutoffs-in-HeapPageFreeze.patch)
download | inline diff:
From 23036ff6bad6a50574dac7f398fcdf9f171f7120 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Sat, 6 Jan 2024 16:22:17 -0500
Subject: [PATCH v4 06/19] Add reference to VacuumCutoffs in HeapPageFreeze
Future commits will move opportunistic freezing into the main path of
pruning in heap_page_prune(). Because on-access pruning will not do
opportunistic freezing, it is cleaner to keep the visibility information
required for calling heap_prepare_freeze_tuple() inside of the
HeapPageFreeze structure itself by saving a reference to VacuumCutoffs.
---
src/backend/access/heap/heapam.c | 67 ++++++++++++++--------------
src/backend/access/heap/vacuumlazy.c | 3 +-
src/include/access/heapam.h | 2 +-
3 files changed, 36 insertions(+), 36 deletions(-)
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 34bc60f625f..7261c4988d7 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -6023,7 +6023,7 @@ heap_inplace_update(Relation relation, HeapTuple tuple)
*/
static TransactionId
FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
- const struct VacuumCutoffs *cutoffs, uint16 *flags,
+ uint16 *flags,
HeapPageFreeze *pagefrz)
{
TransactionId newxmax;
@@ -6049,12 +6049,12 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
pagefrz->freeze_required = true;
return InvalidTransactionId;
}
- else if (MultiXactIdPrecedes(multi, cutoffs->relminmxid))
+ else if (MultiXactIdPrecedes(multi, pagefrz->cutoffs->relminmxid))
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
errmsg_internal("found multixact %u from before relminmxid %u",
- multi, cutoffs->relminmxid)));
- else if (MultiXactIdPrecedes(multi, cutoffs->OldestMxact))
+ multi, pagefrz->cutoffs->relminmxid)));
+ else if (MultiXactIdPrecedes(multi, pagefrz->cutoffs->OldestMxact))
{
TransactionId update_xact;
@@ -6069,7 +6069,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
errmsg_internal("multixact %u from before multi freeze cutoff %u found to be still running",
- multi, cutoffs->OldestMxact)));
+ multi, pagefrz->cutoffs->OldestMxact)));
if (HEAP_XMAX_IS_LOCKED_ONLY(t_infomask))
{
@@ -6080,13 +6080,13 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
/* replace multi with single XID for its updater? */
update_xact = MultiXactIdGetUpdateXid(multi, t_infomask);
- if (TransactionIdPrecedes(update_xact, cutoffs->relfrozenxid))
+ if (TransactionIdPrecedes(update_xact, pagefrz->cutoffs->relfrozenxid))
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
errmsg_internal("multixact %u contains update XID %u from before relfrozenxid %u",
multi, update_xact,
- cutoffs->relfrozenxid)));
- else if (TransactionIdPrecedes(update_xact, cutoffs->OldestXmin))
+ pagefrz->cutoffs->relfrozenxid)));
+ else if (TransactionIdPrecedes(update_xact, pagefrz->cutoffs->OldestXmin))
{
/*
* Updater XID has to have aborted (otherwise the tuple would have
@@ -6098,7 +6098,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
(errcode(ERRCODE_DATA_CORRUPTED),
errmsg_internal("multixact %u contains committed update XID %u from before removable cutoff %u",
multi, update_xact,
- cutoffs->OldestXmin)));
+ pagefrz->cutoffs->OldestXmin)));
*flags |= FRM_INVALIDATE_XMAX;
pagefrz->freeze_required = true;
return InvalidTransactionId;
@@ -6150,9 +6150,9 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
{
TransactionId xid = members[i].xid;
- Assert(!TransactionIdPrecedes(xid, cutoffs->relfrozenxid));
+ Assert(!TransactionIdPrecedes(xid, pagefrz->cutoffs->relfrozenxid));
- if (TransactionIdPrecedes(xid, cutoffs->FreezeLimit))
+ if (TransactionIdPrecedes(xid, pagefrz->cutoffs->FreezeLimit))
{
/* Can't violate the FreezeLimit postcondition */
need_replace = true;
@@ -6164,7 +6164,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
/* Can't violate the MultiXactCutoff postcondition, either */
if (!need_replace)
- need_replace = MultiXactIdPrecedes(multi, cutoffs->MultiXactCutoff);
+ need_replace = MultiXactIdPrecedes(multi, pagefrz->cutoffs->MultiXactCutoff);
if (!need_replace)
{
@@ -6203,7 +6203,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
TransactionId xid = members[i].xid;
MultiXactStatus mstatus = members[i].status;
- Assert(!TransactionIdPrecedes(xid, cutoffs->relfrozenxid));
+ Assert(!TransactionIdPrecedes(xid, pagefrz->cutoffs->relfrozenxid));
if (!ISUPDATE_from_mxstatus(mstatus))
{
@@ -6214,12 +6214,12 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
if (TransactionIdIsCurrentTransactionId(xid) ||
TransactionIdIsInProgress(xid))
{
- if (TransactionIdPrecedes(xid, cutoffs->OldestXmin))
+ if (TransactionIdPrecedes(xid, pagefrz->cutoffs->OldestXmin))
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
errmsg_internal("multixact %u contains running locker XID %u from before removable cutoff %u",
multi, xid,
- cutoffs->OldestXmin)));
+ pagefrz->cutoffs->OldestXmin)));
newmembers[nnewmembers++] = members[i];
has_lockers = true;
}
@@ -6277,11 +6277,11 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
* We determined that updater must be kept -- add it to pending new
* members list
*/
- if (TransactionIdPrecedes(xid, cutoffs->OldestXmin))
+ if (TransactionIdPrecedes(xid, pagefrz->cutoffs->OldestXmin))
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
errmsg_internal("multixact %u contains committed update XID %u from before removable cutoff %u",
- multi, xid, cutoffs->OldestXmin)));
+ multi, xid, pagefrz->cutoffs->OldestXmin)));
newmembers[nnewmembers++] = members[i];
}
@@ -6373,7 +6373,6 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
*/
bool
heap_prepare_freeze_tuple(HeapTupleHeader tuple,
- const struct VacuumCutoffs *cutoffs,
HeapPageFreeze *pagefrz,
HeapTupleFreeze *frz, bool *totally_frozen)
{
@@ -6401,14 +6400,14 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
xmin_already_frozen = true;
else
{
- if (TransactionIdPrecedes(xid, cutoffs->relfrozenxid))
+ if (TransactionIdPrecedes(xid, pagefrz->cutoffs->relfrozenxid))
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
errmsg_internal("found xmin %u from before relfrozenxid %u",
- xid, cutoffs->relfrozenxid)));
+ xid, pagefrz->cutoffs->relfrozenxid)));
/* Will set freeze_xmin flags in freeze plan below */
- freeze_xmin = TransactionIdPrecedes(xid, cutoffs->OldestXmin);
+ freeze_xmin = TransactionIdPrecedes(xid, pagefrz->cutoffs->OldestXmin);
/* Verify that xmin committed if and when freeze plan is executed */
if (freeze_xmin)
@@ -6422,8 +6421,8 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
xid = HeapTupleHeaderGetXvac(tuple);
if (TransactionIdIsNormal(xid))
{
- Assert(TransactionIdPrecedesOrEquals(cutoffs->relfrozenxid, xid));
- Assert(TransactionIdPrecedes(xid, cutoffs->OldestXmin));
+ Assert(TransactionIdPrecedesOrEquals(pagefrz->cutoffs->relfrozenxid, xid));
+ Assert(TransactionIdPrecedes(xid, pagefrz->cutoffs->OldestXmin));
/*
* For Xvac, we always freeze proactively. This allows totally_frozen
@@ -6448,8 +6447,7 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
* perform no-op xmax processing. The only constraint is that the
* FreezeLimit/MultiXactCutoff postcondition must never be violated.
*/
- newxmax = FreezeMultiXactId(xid, tuple->t_infomask, cutoffs,
- &flags, pagefrz);
+ newxmax = FreezeMultiXactId(xid, tuple->t_infomask, &flags, pagefrz);
if (flags & FRM_NOOP)
{
@@ -6472,7 +6470,7 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
* (This repeats work from FreezeMultiXactId, but allows "no
* freeze" tracker maintenance to happen in only one place.)
*/
- Assert(!MultiXactIdPrecedes(newxmax, cutoffs->MultiXactCutoff));
+ Assert(!MultiXactIdPrecedes(newxmax, pagefrz->cutoffs->MultiXactCutoff));
Assert(MultiXactIdIsValid(newxmax) && xid == newxmax);
}
else if (flags & FRM_RETURN_IS_XID)
@@ -6481,7 +6479,7 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
* xmax will become an updater Xid (original MultiXact's updater
* member Xid will be carried forward as a simple Xid in Xmax).
*/
- Assert(!TransactionIdPrecedes(newxmax, cutoffs->OldestXmin));
+ Assert(!TransactionIdPrecedes(newxmax, pagefrz->cutoffs->OldestXmin));
/*
* NB -- some of these transformations are only valid because we
@@ -6505,7 +6503,7 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
* xmax is an old MultiXactId that we have to replace with a new
* MultiXactId, to carry forward two or more original member XIDs.
*/
- Assert(!MultiXactIdPrecedes(newxmax, cutoffs->OldestMxact));
+ Assert(!MultiXactIdPrecedes(newxmax, pagefrz->cutoffs->OldestMxact));
/*
* We can't use GetMultiXactIdHintBits directly on the new multi
@@ -6540,14 +6538,14 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
else if (TransactionIdIsNormal(xid))
{
/* Raw xmax is normal XID */
- if (TransactionIdPrecedes(xid, cutoffs->relfrozenxid))
+ if (TransactionIdPrecedes(xid, pagefrz->cutoffs->relfrozenxid))
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
errmsg_internal("found xmax %u from before relfrozenxid %u",
- xid, cutoffs->relfrozenxid)));
+ xid, pagefrz->cutoffs->relfrozenxid)));
/* Will set freeze_xmax flags in freeze plan below */
- freeze_xmax = TransactionIdPrecedes(xid, cutoffs->OldestXmin);
+ freeze_xmax = TransactionIdPrecedes(xid, pagefrz->cutoffs->OldestXmin);
/*
* Verify that xmax aborted if and when freeze plan is executed,
@@ -6627,7 +6625,7 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
* Does this tuple force caller to freeze the entire page?
*/
pagefrz->freeze_required =
- heap_tuple_should_freeze(tuple, cutoffs,
+ heap_tuple_should_freeze(tuple, pagefrz->cutoffs,
&pagefrz->NoFreezePageRelfrozenXid,
&pagefrz->NoFreezePageRelminMxid);
}
@@ -6949,8 +6947,9 @@ heap_freeze_tuple(HeapTupleHeader tuple,
pagefrz.NoFreezePageRelfrozenXid = FreezeLimit;
pagefrz.NoFreezePageRelminMxid = MultiXactCutoff;
- do_freeze = heap_prepare_freeze_tuple(tuple, &cutoffs,
- &pagefrz, &frz, &totally_frozen);
+ pagefrz.cutoffs = &cutoffs;
+
+ do_freeze = heap_prepare_freeze_tuple(tuple, &pagefrz, &frz, &totally_frozen);
/*
* Note that because this is not a WAL-logged operation, we don't need to
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index f9892f4cd08..06e0e841582 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -1442,6 +1442,7 @@ lazy_scan_prune(LVRelState *vacrel,
pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid;
pagefrz.NoFreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.NoFreezePageRelminMxid = vacrel->NewRelminMxid;
+ pagefrz.cutoffs = &vacrel->cutoffs;
tuples_frozen = 0;
lpdead_items = 0;
live_tuples = 0;
@@ -1587,7 +1588,7 @@ lazy_scan_prune(LVRelState *vacrel,
hastup = true; /* page makes rel truncation unsafe */
/* Tuple with storage -- consider need to freeze */
- if (heap_prepare_freeze_tuple(htup, &vacrel->cutoffs, &pagefrz,
+ if (heap_prepare_freeze_tuple(htup, &pagefrz,
&frozen[tuples_frozen], &totally_frozen))
{
/* Save prepared freeze plan for later */
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index d8e65ae7e35..297ba03bf09 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -189,6 +189,7 @@ typedef struct HeapPageFreeze
TransactionId NoFreezePageRelfrozenXid;
MultiXactId NoFreezePageRelminMxid;
+ struct VacuumCutoffs *cutoffs;
} HeapPageFreeze;
/*
@@ -295,7 +296,6 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple,
extern void heap_inplace_update(Relation relation, HeapTuple tuple);
extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
- const struct VacuumCutoffs *cutoffs,
HeapPageFreeze *pagefrz,
HeapTupleFreeze *frz, bool *totally_frozen);
extern void heap_freeze_execute_prepared(Relation rel, Buffer buffer,
--
2.40.1
[text/x-diff] v4-0007-Prepare-freeze-tuples-in-heap_page_prune.patch (11.8K, ../../20240320013602.6sypr4cx6sefpemg@liskov/8-v4-0007-Prepare-freeze-tuples-in-heap_page_prune.patch)
download | inline diff:
From 470e695363d881d48d1fe839ea6770938f20d078 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Mon, 18 Mar 2024 20:01:38 -0400
Subject: [PATCH v4 07/19] Prepare freeze tuples in heap_page_prune()
In order to combine the freeze and prune records, we must determine
which tuples are freezable before actually executing pruning. All of the
page modifications should be made in the same critical section along
with emitting the combined WAL. Determine whether or not tuples should
or must be frozen and whether or not the page will be all frozen as a
consequence during pruning.
---
src/backend/access/heap/pruneheap.c | 41 +++++++++++++++--
src/backend/access/heap/vacuumlazy.c | 68 +++++++---------------------
src/include/access/heapam.h | 12 +++++
3 files changed, 66 insertions(+), 55 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index bd30296ef1a..afc5ea5e0e7 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -153,7 +153,7 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* not the relation has indexes, since we cannot safely determine
* that during on-access pruning with the current implementation.
*/
- heap_page_prune(relation, buffer, vistest, false,
+ heap_page_prune(relation, buffer, vistest, false, NULL,
&presult, NULL);
/*
@@ -206,6 +206,9 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* mark_unused_now indicates whether or not dead items can be set LP_UNUSED
* during pruning.
*
+ * pagefrz contains both input and output parameters used if the caller is
+ * interested in potentially freezing tuples on the page.
+ *
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
* heap_page_prune() is responsible for initializing it.
@@ -217,6 +220,7 @@ void
heap_page_prune(Relation relation, Buffer buffer,
GlobalVisState *vistest,
bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
PruneResult *presult,
OffsetNumber *off_loc)
{
@@ -247,11 +251,16 @@ heap_page_prune(Relation relation, Buffer buffer,
presult->ndeleted = 0;
presult->nnewlpdead = 0;
+ presult->nfrozen = 0;
/*
- * Keep track of whether or not the page is all_visible in case the caller
- * wants to use this information to update the VM.
+ * Caller will update the VM after pruning, collecting LP_DEAD items, and
+ * freezing tuples. Keep track of whether or not the page is all_visible
+ * and all_frozen and use this information to update the VM. all_visible
+ * implies lpdead_items == 0, but don't trust all_frozen result unless
+ * all_visible is also set to true.
*/
+ presult->all_frozen = true;
presult->all_visible = true;
/* for recovery conflicts */
presult->frz_conflict_horizon = InvalidTransactionId;
@@ -381,6 +390,32 @@ heap_page_prune(Relation relation, Buffer buffer,
elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result");
break;
}
+
+ /*
+ * Consider freezing any normal tuples which will not be removed
+ */
+ if (presult->htsv[offnum] != HEAPTUPLE_DEAD && pagefrz)
+ {
+ bool totally_frozen;
+
+ /* Tuple with storage -- consider need to freeze */
+ if ((heap_prepare_freeze_tuple(htup, pagefrz,
+ &presult->frozen[presult->nfrozen],
+ &totally_frozen)))
+ {
+ /* Save prepared freeze plan for later */
+ presult->frozen[presult->nfrozen++].offset = offnum;
+ }
+
+ /*
+ * If any tuple isn't either totally frozen already or eligible to
+ * become totally frozen (according to its freeze plan), then the
+ * page definitely cannot be set all-frozen in the visibility map
+ * later on
+ */
+ if (!totally_frozen)
+ presult->all_frozen = false;
+ }
}
/*
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 06e0e841582..4187c998d25 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -1416,16 +1416,13 @@ lazy_scan_prune(LVRelState *vacrel,
maxoff;
ItemId itemid;
PruneResult presult;
- int tuples_frozen,
- lpdead_items,
+ int lpdead_items,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool all_frozen;
int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
- HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
@@ -1443,7 +1440,6 @@ lazy_scan_prune(LVRelState *vacrel,
pagefrz.NoFreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.NoFreezePageRelminMxid = vacrel->NewRelminMxid;
pagefrz.cutoffs = &vacrel->cutoffs;
- tuples_frozen = 0;
lpdead_items = 0;
live_tuples = 0;
recently_dead_tuples = 0;
@@ -1461,31 +1457,20 @@ lazy_scan_prune(LVRelState *vacrel,
* false otherwise.
*/
heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
- &presult, &vacrel->offnum);
+ &pagefrz, &presult, &vacrel->offnum);
/*
* Now scan the page to collect LP_DEAD items and check for tuples
* requiring freezing among remaining tuples with storage. We will update
* the VM after collecting LP_DEAD items and freezing tuples. Pruning will
- * have determined whether or not the page is all_visible. Keep track of
- * whether or not the page is all_frozen and use this information to
- * update the VM. all_visible implies lpdead_items == 0, but don't trust
- * all_frozen result unless all_visible is also set to true.
+ * have determined whether or not the page is all_visible and able to
+ * become all_frozen.
*
*/
- all_frozen = true;
-
- /*
- * Now scan the page to collect LP_DEAD items and update the variables set
- * just above.
- */
for (offnum = FirstOffsetNumber;
offnum <= maxoff;
offnum = OffsetNumberNext(offnum))
{
- HeapTupleHeader htup;
- bool totally_frozen;
-
/*
* Set the offset number so that we can display it along with any
* error that occurred while processing this tuple.
@@ -1521,8 +1506,6 @@ lazy_scan_prune(LVRelState *vacrel,
Assert(ItemIdIsNormal(itemid));
- htup = (HeapTupleHeader) PageGetItem(page, itemid);
-
/*
* The criteria for counting a tuple as live in this block need to
* match what analyze.c's acquire_sample_rows() does, otherwise VACUUM
@@ -1587,29 +1570,8 @@ lazy_scan_prune(LVRelState *vacrel,
hastup = true; /* page makes rel truncation unsafe */
- /* Tuple with storage -- consider need to freeze */
- if (heap_prepare_freeze_tuple(htup, &pagefrz,
- &frozen[tuples_frozen], &totally_frozen))
- {
- /* Save prepared freeze plan for later */
- frozen[tuples_frozen++].offset = offnum;
- }
-
- /*
- * If any tuple isn't either totally frozen already or eligible to
- * become totally frozen (according to its freeze plan), then the page
- * definitely cannot be set all-frozen in the visibility map later on
- */
- if (!totally_frozen)
- all_frozen = false;
}
- /*
- * We have now divided every item on the page into either an LP_DEAD item
- * that will need to be vacuumed in indexes later, or a LP_NORMAL tuple
- * that remains and needs to be considered for freezing now (LP_UNUSED and
- * LP_REDIRECT items also remain, but are of no further interest to us).
- */
vacrel->offnum = InvalidOffsetNumber;
/*
@@ -1618,8 +1580,8 @@ lazy_scan_prune(LVRelState *vacrel,
* freeze when pruning generated an FPI, if doing so means that we set the
* page all-frozen afterwards (might not happen until final heap pass).
*/
- if (pagefrz.freeze_required || tuples_frozen == 0 ||
- (presult.all_visible_except_removable && all_frozen &&
+ if (pagefrz.freeze_required || presult.nfrozen == 0 ||
+ (presult.all_visible_except_removable && presult.all_frozen &&
fpi_before != pgWalUsage.wal_fpi))
{
/*
@@ -1629,7 +1591,7 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- if (tuples_frozen == 0)
+ if (presult.nfrozen == 0)
{
/*
* We have no freeze plans to execute, so there's no added cost
@@ -1657,7 +1619,7 @@ lazy_scan_prune(LVRelState *vacrel,
* once we're done with it. Otherwise we generate a conservative
* cutoff by stepping back from OldestXmin.
*/
- if (presult.all_visible_except_removable && all_frozen)
+ if (presult.all_visible_except_removable && presult.all_frozen)
{
/* Using same cutoff when setting VM is now unnecessary */
snapshotConflictHorizon = presult.frz_conflict_horizon;
@@ -1673,7 +1635,7 @@ lazy_scan_prune(LVRelState *vacrel,
/* Execute all freeze plans for page as a single atomic action */
heap_freeze_execute_prepared(vacrel->rel, buf,
snapshotConflictHorizon,
- frozen, tuples_frozen);
+ presult.frozen, presult.nfrozen);
}
}
else
@@ -1684,8 +1646,8 @@ lazy_scan_prune(LVRelState *vacrel,
*/
vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid;
vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid;
- all_frozen = false;
- tuples_frozen = 0; /* avoid miscounts in instrumentation */
+ presult.all_frozen = false;
+ presult.nfrozen = 0; /* avoid miscounts in instrumentation */
}
/*
@@ -1708,6 +1670,8 @@ lazy_scan_prune(LVRelState *vacrel,
&debug_cutoff, &debug_all_frozen))
Assert(false);
+ Assert(presult.all_frozen == debug_all_frozen);
+
Assert(!TransactionIdIsValid(debug_cutoff) ||
debug_cutoff == presult.frz_conflict_horizon);
}
@@ -1738,7 +1702,7 @@ lazy_scan_prune(LVRelState *vacrel,
/* Finally, add page-local counts to whole-VACUUM counts */
vacrel->tuples_deleted += presult.ndeleted;
- vacrel->tuples_frozen += tuples_frozen;
+ vacrel->tuples_frozen += presult.nfrozen;
vacrel->lpdead_items += lpdead_items;
vacrel->live_tuples += live_tuples;
vacrel->recently_dead_tuples += recently_dead_tuples;
@@ -1761,7 +1725,7 @@ lazy_scan_prune(LVRelState *vacrel,
{
uint8 flags = VISIBILITYMAP_ALL_VISIBLE;
- if (all_frozen)
+ if (presult.all_frozen)
{
Assert(!TransactionIdIsValid(presult.frz_conflict_horizon));
flags |= VISIBILITYMAP_ALL_FROZEN;
@@ -1832,7 +1796,7 @@ lazy_scan_prune(LVRelState *vacrel,
* true, so we must check both all_visible and all_frozen.
*/
else if (all_visible_according_to_vm && presult.all_visible &&
- all_frozen && !VM_ALL_FROZEN(vacrel->rel, blkno, &vmbuffer))
+ presult.all_frozen && !VM_ALL_FROZEN(vacrel->rel, blkno, &vmbuffer))
{
/*
* Avoid relying on all_visible_according_to_vm as a proxy for the
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 297ba03bf09..2339abfd28a 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -201,6 +201,11 @@ typedef struct PruneResult
int nnewlpdead; /* Number of newly LP_DEAD items */
bool all_visible; /* Whether or not the page is all visible */
bool all_visible_except_removable;
+ /* Whether or not the page can be set all frozen in the VM */
+ bool all_frozen;
+
+ /* Number of newly frozen tuples */
+ int nfrozen;
TransactionId frz_conflict_horizon; /* Newest xmin on the page */
/*
@@ -213,6 +218,12 @@ typedef struct PruneResult
* 1. Otherwise every access would need to subtract 1.
*/
int8 htsv[MaxHeapTuplesPerPage + 1];
+
+
+ /*
+ * One entry for every tuple that we may freeze.
+ */
+ HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
} PruneResult;
/*
@@ -324,6 +335,7 @@ extern void heap_page_prune_opt(Relation relation, Buffer buffer);
extern void heap_page_prune(Relation relation, Buffer buffer,
struct GlobalVisState *vistest,
bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
PruneResult *presult,
OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer,
--
2.40.1
[text/x-diff] v4-0008-lazy_scan_prune-reorder-freeze-execution-logic.patch (5.9K, ../../20240320013602.6sypr4cx6sefpemg@liskov/9-v4-0008-lazy_scan_prune-reorder-freeze-execution-logic.patch)
download | inline diff:
From 3c208039193ce94111e8ddc1b03828cf820e11e3 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Tue, 19 Mar 2024 19:30:16 -0400
Subject: [PATCH v4 08/19] lazy_scan_prune reorder freeze execution logic
To combine the prune and freeze records, freezing must be done before a
pruning WAL record is emitted. We will move the freeze execution into
heap_page_prune() in future commits. lazy_scan_prune() currently
executes freezing, updates vacrel->NewRelfrozenXid and
vacrel->NewRelminMxid, and resets the snapshotConflictHorizon that the
visibility map update record may use in the same block of if statements.
This commit starts reordering that logic so that the freeze execution
can be separated from the other updates which should not be done in
pruning.
---
src/backend/access/heap/vacuumlazy.c | 92 +++++++++++++++-------------
1 file changed, 49 insertions(+), 43 deletions(-)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 4187c998d25..74ebab25a95 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -1421,6 +1421,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
+ bool do_freeze;
int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
@@ -1580,10 +1581,15 @@ lazy_scan_prune(LVRelState *vacrel,
* freeze when pruning generated an FPI, if doing so means that we set the
* page all-frozen afterwards (might not happen until final heap pass).
*/
- if (pagefrz.freeze_required || presult.nfrozen == 0 ||
+ do_freeze = pagefrz.freeze_required ||
(presult.all_visible_except_removable && presult.all_frozen &&
- fpi_before != pgWalUsage.wal_fpi))
+ presult.nfrozen > 0 &&
+ fpi_before != pgWalUsage.wal_fpi);
+
+ if (do_freeze)
{
+ TransactionId snapshotConflictHorizon;
+
/*
* We're freezing the page. Our final NewRelfrozenXid doesn't need to
* be affected by the XIDs that are just about to be frozen anyway.
@@ -1591,52 +1597,52 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- if (presult.nfrozen == 0)
- {
- /*
- * We have no freeze plans to execute, so there's no added cost
- * from following the freeze path. That's why it was chosen. This
- * is important in the case where the page only contains totally
- * frozen tuples at this point (perhaps only following pruning).
- * Such pages can be marked all-frozen in the VM by our caller,
- * even though none of its tuples were newly frozen here (note
- * that the "no freeze" path never sets pages all-frozen).
- *
- * We never increment the frozen_pages instrumentation counter
- * here, since it only counts pages with newly frozen tuples
- * (don't confuse that with pages newly set all-frozen in VM).
- */
- }
+ vacrel->frozen_pages++;
+
+ /*
+ * We can use frz_conflict_horizon as our cutoff for conflicts when
+ * the whole page is eligible to become all-frozen in the VM once
+ * we're done with it. Otherwise we generate a conservative cutoff by
+ * stepping back from OldestXmin.
+ */
+ if (presult.all_visible_except_removable && presult.all_frozen)
+ snapshotConflictHorizon = presult.frz_conflict_horizon;
else
{
- TransactionId snapshotConflictHorizon;
+ /* Avoids false conflicts when hot_standby_feedback in use */
+ snapshotConflictHorizon = pagefrz.cutoffs->OldestXmin;
+ TransactionIdRetreat(snapshotConflictHorizon);
+ }
- vacrel->frozen_pages++;
+ /* Using same cutoff when setting VM is now unnecessary */
+ if (presult.all_visible_except_removable && presult.all_frozen)
+ presult.frz_conflict_horizon = InvalidTransactionId;
- /*
- * We can use frz_conflict_horizon as our cutoff for conflicts
- * when the whole page is eligible to become all-frozen in the VM
- * once we're done with it. Otherwise we generate a conservative
- * cutoff by stepping back from OldestXmin.
- */
- if (presult.all_visible_except_removable && presult.all_frozen)
- {
- /* Using same cutoff when setting VM is now unnecessary */
- snapshotConflictHorizon = presult.frz_conflict_horizon;
- presult.frz_conflict_horizon = InvalidTransactionId;
- }
- else
- {
- /* Avoids false conflicts when hot_standby_feedback in use */
- snapshotConflictHorizon = vacrel->cutoffs.OldestXmin;
- TransactionIdRetreat(snapshotConflictHorizon);
- }
+ /* Execute all freeze plans for page as a single atomic action */
+ heap_freeze_execute_prepared(vacrel->rel, buf,
+ snapshotConflictHorizon,
+ presult.frozen, presult.nfrozen);
+ }
+ else if (presult.all_frozen && presult.nfrozen == 0)
+ {
+ /* Page should be all visible except to-be-removed tuples */
+ Assert(presult.all_visible_except_removable);
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(vacrel->rel, buf,
- snapshotConflictHorizon,
- presult.frozen, presult.nfrozen);
- }
+ /*
+ * We have no freeze plans to execute, so there's no added cost from
+ * following the freeze path. That's why it was chosen. This is
+ * important in the case where the page only contains totally frozen
+ * tuples at this point (perhaps only following pruning). Such pages
+ * can be marked all-frozen in the VM by our caller, even though none
+ * of its tuples were newly frozen here (note that the "no freeze"
+ * path never sets pages all-frozen).
+ *
+ * We never increment the frozen_pages instrumentation counter here,
+ * since it only counts pages with newly frozen tuples (don't confuse
+ * that with pages newly set all-frozen in VM).
+ */
+ vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
+ vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
}
else
{
--
2.40.1
[text/x-diff] v4-0009-Execute-freezing-in-heap_page_prune.patch (26.7K, ../../20240320013602.6sypr4cx6sefpemg@liskov/10-v4-0009-Execute-freezing-in-heap_page_prune.patch)
download | inline diff:
From aec400aa0289ecb8cab6dfaeb8fee050db6487c3 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Fri, 8 Mar 2024 16:45:57 -0500
Subject: [PATCH v4 09/19] Execute freezing in heap_page_prune()
As a step toward combining the prune and freeze WAL records, execute
freezing in heap_page_prune(). The logic to determine whether or not to
execute freeze plans was moved from lazy_scan_prune() over to
heap_page_prune() with little modification.
---
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/access/heap/pruneheap.c | 155 ++++++++++++++++++-----
src/backend/access/heap/vacuumlazy.c | 134 +++++---------------
src/backend/storage/ipc/procarray.c | 6 +-
src/include/access/heapam.h | 39 +++---
src/tools/pgindent/typedefs.list | 2 +-
6 files changed, 182 insertions(+), 156 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 680a50bf8b1..5e522f5b0ba 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1046,7 +1046,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
* We ignore unused and redirect line pointers. DEAD line pointers
* should be counted as dead, because we need vacuum to run to get rid
* of them. Note that this rule agrees with the way that
- * heap_page_prune() counts things.
+ * heap_page_prune_and_freeze() counts things.
*/
if (!ItemIdIsNormal(itemid))
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index afc5ea5e0e7..20907ba5408 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -17,16 +17,19 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
+#include "access/multixact.h"
#include "access/transam.h"
#include "access/xlog.h"
+#include "commands/vacuum.h"
#include "access/xloginsert.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
-/* Working data for heap_page_prune and subroutines */
+/* Working data for heap_page_prune_and_freeze() and subroutines */
typedef struct
{
/* tuple visibility test, initialized for the relation */
@@ -51,6 +54,11 @@ typedef struct
* 1. Otherwise every access would need to subtract 1.
*/
bool marked[MaxHeapTuplesPerPage + 1];
+
+ /*
+ * One entry for every tuple that we may freeze.
+ */
+ HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
} PruneState;
/* Local functions */
@@ -59,14 +67,15 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult);
+ PruneState *prstate, PruneFreezeResult *presult);
+
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
@@ -146,15 +155,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
{
- PruneResult presult;
+ PruneFreezeResult presult;
/*
* For now, pass mark_unused_now as false regardless of whether or
* not the relation has indexes, since we cannot safely determine
* that during on-access pruning with the current implementation.
*/
- heap_page_prune(relation, buffer, vistest, false, NULL,
- &presult, NULL);
+ heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL,
+ &presult, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -188,7 +197,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
/*
- * Prune and repair fragmentation in the specified page.
+ * Prune and repair fragmentation and potentially freeze tuples on the
+ * specified page.
+ *
+ * If the page can be marked all-frozen in the visibility map, we may
+ * opportunistically freeze tuples on the page if either its tuples are old
+ * enough or freezing will be cheap enough.
*
* Caller must have pin and buffer cleanup lock on the page. Note that we
* don't update the FSM information for page on caller's behalf. Caller might
@@ -206,23 +220,24 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* mark_unused_now indicates whether or not dead items can be set LP_UNUSED
* during pruning.
*
- * pagefrz contains both input and output parameters used if the caller is
- * interested in potentially freezing tuples on the page.
+ * pagefrz is an input parameter containing visibility cutoff information and
+ * the current relfrozenxid and relminmxids used if the caller is interested in
+ * freezing tuples on the page.
*
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
- * heap_page_prune() is responsible for initializing it.
+ * heap_page_prune_and_freeze() is responsible for initializing it.
*
* off_loc is the offset location required by the caller to use in error
* callback.
*/
void
-heap_page_prune(Relation relation, Buffer buffer,
- GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc)
+heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc)
{
Page page = BufferGetPage(buffer);
BlockNumber blockno = BufferGetBlockNumber(buffer);
@@ -230,6 +245,8 @@ heap_page_prune(Relation relation, Buffer buffer,
maxoff;
PruneState prstate;
HeapTupleData tup;
+ bool do_freeze;
+ int64 fpi_before = pgWalUsage.wal_fpi;
/*
* First, initialize the new pd_prune_xid value to zero (indicating no
@@ -265,6 +282,10 @@ heap_page_prune(Relation relation, Buffer buffer,
/* for recovery conflicts */
presult->frz_conflict_horizon = InvalidTransactionId;
+ /* For advancing relfrozenxid and relminmxid */
+ presult->new_relfrozenxid = InvalidTransactionId;
+ presult->new_relminmxid = InvalidMultiXactId;
+
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(relation);
@@ -400,11 +421,11 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Tuple with storage -- consider need to freeze */
if ((heap_prepare_freeze_tuple(htup, pagefrz,
- &presult->frozen[presult->nfrozen],
+ &prstate.frozen[presult->nfrozen],
&totally_frozen)))
{
/* Save prepared freeze plan for later */
- presult->frozen[presult->nfrozen++].offset = offnum;
+ prstate.frozen[presult->nfrozen++].offset = offnum;
}
/*
@@ -557,6 +578,72 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ */
+ if (pagefrz)
+ do_freeze = pagefrz->freeze_required ||
+ (presult->all_visible_except_removable && presult->all_frozen &&
+ presult->nfrozen > 0 &&
+ fpi_before != pgWalUsage.wal_fpi);
+ else
+ do_freeze = false;
+
+ if (do_freeze)
+ {
+ /*
+ * We can use frz_conflict_horizon as our cutoff for conflicts when
+ * the whole page is eligible to become all-frozen in the VM once
+ * we're done with it. Otherwise we generate a conservative cutoff by
+ * stepping back from OldestXmin.
+ */
+ if (!(presult->all_visible_except_removable && presult->all_frozen))
+ {
+ /* Avoids false conflicts when hot_standby_feedback in use */
+ presult->frz_conflict_horizon = pagefrz->cutoffs->OldestXmin;
+ TransactionIdRetreat(presult->frz_conflict_horizon);
+ }
+
+ /* Execute all freeze plans for page as a single atomic action */
+ heap_freeze_execute_prepared(relation, buffer,
+ presult->frz_conflict_horizon,
+ prstate.frozen, presult->nfrozen);
+ }
+ else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
+ {
+ /*
+ * If we will neither freeze tuples on the page nor set the page all
+ * frozen in the visibility map, the page is not all frozen and there
+ * will be no newly frozen tuples.
+ */
+ presult->all_frozen = false;
+ presult->nfrozen = 0; /* avoid miscounts in instrumentation */
+ }
+
+ /* Caller won't update new_relfrozenxid and new_relminmxid */
+ if (!pagefrz)
+ return;
+
+ /*
+ * If we will freeze tuples on the page or, even if we don't freeze tuples
+ * on the page, if we will set the page all-frozen in the visibility map,
+ * we can advance relfrozenxid and relminmxid to the values in
+ * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid.
+ */
+ if (presult->all_frozen || presult->nfrozen > 0)
+ {
+ presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
+ }
+ else
+ {
+ presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
+ }
}
@@ -614,7 +701,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
*/
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult)
+ PruneState *prstate, PruneFreezeResult *presult)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -879,10 +966,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
{
/*
* We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state or LP_UNUSED if the caller indicated.
+ * item. This can happen if the loop in heap_page_prune_and_freeze()
+ * caused us to visit the dead successor of a redirect item before
+ * visiting the redirect item. We can clean up by setting the
+ * redirect item to DEAD state or LP_UNUSED if the caller indicated.
*/
heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
}
@@ -922,7 +1009,7 @@ heap_prune_record_redirect(PruneState *prstate,
/* Record line pointer to be marked dead */
static void
heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
@@ -945,7 +1032,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
*/
static void
heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
/*
* If the caller set mark_unused_now to true, we can remove dead tuples
@@ -972,9 +1059,9 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum)
/*
- * Perform the actual page changes needed by heap_page_prune.
- * It is expected that the caller has a full cleanup lock on the
- * buffer.
+ * Perform the actual page pruning modifications needed by
+ * heap_page_prune_and_freeze(). It is expected that the caller has a full
+ * cleanup lock on the buffer.
*/
void
heap_page_prune_execute(Buffer buffer,
@@ -1088,11 +1175,11 @@ heap_page_prune_execute(Buffer buffer,
#ifdef USE_ASSERT_CHECKING
/*
- * When heap_page_prune() was called, mark_unused_now may have been
- * passed as true, which allows would-be LP_DEAD items to be made
- * LP_UNUSED instead. This is only possible if the relation has no
- * indexes. If there are any dead items, then mark_unused_now was not
- * true and every item being marked LP_UNUSED must refer to a
+ * When heap_page_prune_and_freeze() was called, mark_unused_now may
+ * have been passed as true, which allows would-be LP_DEAD items to be
+ * made LP_UNUSED instead. This is only possible if the relation has
+ * no indexes. If there are any dead items, then mark_unused_now was
+ * not true and every item being marked LP_UNUSED must refer to a
* heap-only tuple.
*/
if (ndead > 0)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 74ebab25a95..c4553a4159c 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -430,12 +430,12 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
- * Next acquire vistest, a related cutoff that's used in heap_page_prune.
- * We expect vistest will always make heap_page_prune remove any deleted
- * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
- * confused about whether a tuple should be frozen or removed. (In the
- * future we might want to teach lazy_scan_prune to recompute vistest from
- * time to time, to increase the number of dead tuples it can prune away.)
+ * Next acquire vistest, a related cutoff that's used in
+ * heap_page_prune_and_freeze(). We expect vistest will always make
+ * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is <
+ * OldestXmin. (In the future we might want to teach lazy_scan_prune to
+ * recompute vistest from time to time, to increase the number of dead
+ * tuples it can prune away.)
*/
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
@@ -1378,21 +1378,21 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
*
* Caller must hold pin and buffer cleanup lock on the buffer.
*
- * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune()
- * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about
- * whether or not a tuple should be considered DEAD. This happened when an
- * inserting transaction concurrently aborted (after our heap_page_prune()
- * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot
- * of complexity just so we could deal with tuples that were DEAD to VACUUM,
- * but nevertheless were left with storage after pruning.
+ * Prior to PostgreSQL 14 there were very rare cases where
+ * heap_page_prune_and_freeze() was allowed to disagree with our
+ * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be
+ * considered DEAD. This happened when an inserting transaction concurrently
+ * aborted (after our heap_page_prune_and_freeze() call, before our
+ * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just
+ * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
+ * left with storage after pruning.
*
* As of Postgres 17, we circumvent this problem altogether by reusing the
- * result of heap_page_prune()'s visibility check. Without the second call to
- * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no
- * disagreement. We'll just handle such tuples as if they had become fully dead
- * right after this operation completes instead of in the middle of it. Note that
- * any tuple that becomes dead after the call to heap_page_prune() can't need to
- * be frozen, because it was visible to another session when vacuum started.
+ * result of heap_page_prune_and_freeze()'s visibility check. Without the
+ * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
+ * there can be no disagreement. We'll just handle such tuples as if they had
+ * become fully dead right after this operation completes instead of in the
+ * middle of it.
*
* vmbuffer is the buffer containing the VM block with visibility information
* for the heap block, blkno. all_visible_according_to_vm is the saved
@@ -1415,26 +1415,24 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- PruneResult presult;
+ PruneFreezeResult presult;
int lpdead_items,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool do_freeze;
- int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
/*
* maxoff might be reduced following line pointer array truncation in
- * heap_page_prune. That's safe for us to ignore, since the reclaimed
- * space will continue to look like LP_UNUSED items below.
+ * heap_page_prune_and_freeze(). That's safe for us to ignore, since the
+ * reclaimed space will continue to look like LP_UNUSED items below.
*/
maxoff = PageGetMaxOffsetNumber(page);
- /* Initialize (or reset) page-level state */
+ /* Initialize pagefrz */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid;
@@ -1446,7 +1444,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples = 0;
/*
- * Prune all HOT-update chains in this page.
+ * Prune all HOT-update chains and potentially freeze tuples on this page.
*
* We count the number of tuples removed from the page by the pruning step
* in presult.ndeleted. It should not be confused with lpdead_items;
@@ -1457,8 +1455,8 @@ lazy_scan_prune(LVRelState *vacrel,
* items LP_UNUSED, so mark_unused_now should be true if no indexes and
* false otherwise.
*/
- heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
- &pagefrz, &presult, &vacrel->offnum);
+ heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
+ &pagefrz, &presult, &vacrel->offnum);
/*
* Now scan the page to collect LP_DEAD items and check for tuples
@@ -1575,85 +1573,23 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = InvalidOffsetNumber;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- do_freeze = pagefrz.freeze_required ||
- (presult.all_visible_except_removable && presult.all_frozen &&
- presult.nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
+ Assert(MultiXactIdIsValid(presult.new_relminmxid));
+ vacrel->NewRelfrozenXid = presult.new_relfrozenxid;
+ Assert(TransactionIdIsValid(presult.new_relfrozenxid));
+ vacrel->NewRelminMxid = presult.new_relminmxid;
- if (do_freeze)
+ if (presult.nfrozen > 0)
{
- TransactionId snapshotConflictHorizon;
-
/*
- * We're freezing the page. Our final NewRelfrozenXid doesn't need to
- * be affected by the XIDs that are just about to be frozen anyway.
+ * We never increment the frozen_pages instrumentation counter when
+ * nfrozen == 0, since it only counts pages with newly frozen tuples
+ * (don't confuse that with pages newly set all-frozen in VM).
*/
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
-
vacrel->frozen_pages++;
- /*
- * We can use frz_conflict_horizon as our cutoff for conflicts when
- * the whole page is eligible to become all-frozen in the VM once
- * we're done with it. Otherwise we generate a conservative cutoff by
- * stepping back from OldestXmin.
- */
- if (presult.all_visible_except_removable && presult.all_frozen)
- snapshotConflictHorizon = presult.frz_conflict_horizon;
- else
- {
- /* Avoids false conflicts when hot_standby_feedback in use */
- snapshotConflictHorizon = pagefrz.cutoffs->OldestXmin;
- TransactionIdRetreat(snapshotConflictHorizon);
- }
-
/* Using same cutoff when setting VM is now unnecessary */
- if (presult.all_visible_except_removable && presult.all_frozen)
+ if (presult.all_frozen)
presult.frz_conflict_horizon = InvalidTransactionId;
-
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(vacrel->rel, buf,
- snapshotConflictHorizon,
- presult.frozen, presult.nfrozen);
- }
- else if (presult.all_frozen && presult.nfrozen == 0)
- {
- /* Page should be all visible except to-be-removed tuples */
- Assert(presult.all_visible_except_removable);
-
- /*
- * We have no freeze plans to execute, so there's no added cost from
- * following the freeze path. That's why it was chosen. This is
- * important in the case where the page only contains totally frozen
- * tuples at this point (perhaps only following pruning). Such pages
- * can be marked all-frozen in the VM by our caller, even though none
- * of its tuples were newly frozen here (note that the "no freeze"
- * path never sets pages all-frozen).
- *
- * We never increment the frozen_pages instrumentation counter here,
- * since it only counts pages with newly frozen tuples (don't confuse
- * that with pages newly set all-frozen in VM).
- */
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- }
- else
- {
- /*
- * Page requires "no freeze" processing. It might be set all-visible
- * in the visibility map, but it can never be set all-frozen.
- */
- vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid;
- presult.all_frozen = false;
- presult.nfrozen = 0; /* avoid miscounts in instrumentation */
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index b3cd248fb64..88a6d504dff 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid)
* Note: the approximate horizons (see definition of GlobalVisState) are
* updated by the computations done here. That's currently required for
* correctness and a small optimization. Without doing so it's possible that
- * heap vacuum's call to heap_page_prune() uses a more conservative horizon
- * than later when deciding which tuples can be removed - which the code
- * doesn't expect (breaking HOT).
+ * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative
+ * horizon than later when deciding which tuples can be removed - which the
+ * code doesn't expect (breaking HOT).
*/
static void
ComputeXidHorizons(ComputeXidHorizonsResult *h)
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 2339abfd28a..b2a4caeb33a 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -195,7 +195,7 @@ typedef struct HeapPageFreeze
/*
* Per-page state returned from pruning
*/
-typedef struct PruneResult
+typedef struct PruneFreezeResult
{
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
@@ -210,9 +210,10 @@ typedef struct PruneResult
/*
* Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
+ * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
+ * details. This is of type int8[], instead of HTSV_Result[], so we can
+ * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
+ * items.
*
* This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
* 1. Otherwise every access would need to subtract 1.
@@ -220,17 +221,18 @@ typedef struct PruneResult
int8 htsv[MaxHeapTuplesPerPage + 1];
- /*
- * One entry for every tuple that we may freeze.
- */
- HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
-} PruneResult;
+ /* New value of relfrozenxid found by heap_page_prune_and_freeze() */
+ TransactionId new_relfrozenxid;
+
+ /* New value of relminmxid found by heap_page_prune_and_freeze() */
+ MultiXactId new_relminmxid;
+} PruneFreezeResult;
/*
* Pruning calculates tuple visibility once and saves the results in an array
- * of int8. See PruneResult.htsv for details. This helper function is meant to
- * guard against examining visibility status array members which have not yet
- * been computed.
+ * of int8. See PruneFreezeResult.htsv for details. This helper function is
+ * meant to guard against examining visibility status array members which have
+ * not yet been computed.
*/
static inline HTSV_Result
htsv_get_valid_status(int status)
@@ -306,6 +308,7 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple,
Buffer *buffer, struct TM_FailureData *tmfd);
extern void heap_inplace_update(Relation relation, HeapTuple tuple);
+
extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
HeapPageFreeze *pagefrz,
HeapTupleFreeze *frz, bool *totally_frozen);
@@ -332,12 +335,12 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
/* in heap/pruneheap.c */
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
-extern void heap_page_prune(Relation relation, Buffer buffer,
- struct GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc);
+extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ struct GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 042d04c8de2..b2ddc1e2549 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2179,7 +2179,7 @@ ProjectionPath
PromptInterruptContext
ProtocolVersion
PrsStorage
-PruneResult
+PruneFreezeResult
PruneState
PruneStepResult
PsqlScanCallbacks
--
2.40.1
[text/x-diff] v4-0010-Make-opp-freeze-heuristic-compatible-with-prune-f.patch (4.3K, ../../20240320013602.6sypr4cx6sefpemg@liskov/11-v4-0010-Make-opp-freeze-heuristic-compatible-with-prune-f.patch)
download | inline diff:
From dd6bdad1253dc6c2e62cfe8144d62456ee7be2e8 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Sun, 7 Jan 2024 16:11:35 -0500
Subject: [PATCH v4 10/19] Make opp freeze heuristic compatible with
prune+freeze record
Once the prune and freeze records are combined, we will no longer be
able to use a test of whether or not pruning emitted an FPI to decide
whether or not to opportunistically freeze a freezable page.
While this heuristic should be improved, for now, approximate the
previous logic by keeping track of whether or not a hint bit FPI was
emitted during visibility checks (when checksums are on) and combine
that with checking XLogCheckBufferNeedsBackup(). If we just finished
deciding whether or not to prune and the current buffer seems to need an
FPI after modification, it is likely that pruning would have emitted an
FPI.
---
src/backend/access/heap/pruneheap.c | 57 +++++++++++++++++++++--------
1 file changed, 42 insertions(+), 15 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 20907ba5408..9edf6bf72d7 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -246,6 +246,10 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
PruneState prstate;
HeapTupleData tup;
bool do_freeze;
+ bool do_prune;
+ bool whole_page_freezable;
+ bool hint_bit_fpi;
+ bool prune_fpi = false;
int64 fpi_before = pgWalUsage.wal_fpi;
/*
@@ -439,6 +443,13 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
}
}
+ /*
+ * If checksums are enabled, heap_prune_satisfies_vacuum() may have caused
+ * an FPI to be emitted. Then reset fpi_before for no prune case.
+ */
+ hint_bit_fpi = fpi_before != pgWalUsage.wal_fpi;
+ fpi_before = pgWalUsage.wal_fpi;
+
/*
* For vacuum, if the whole page will become frozen, we consider
* opportunistically freezing tuples. Dead tuples which will be removed by
@@ -483,11 +494,41 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
if (off_loc)
*off_loc = InvalidOffsetNumber;
+ do_prune = prstate.nredirected > 0 ||
+ prstate.ndead > 0 ||
+ prstate.nunused > 0;
+
+ /*
+ * Only incur overhead of checking if we will do an FPI if we might use
+ * the information.
+ */
+ if (do_prune && pagefrz)
+ prune_fpi = XLogCheckBufferNeedsBackup(buffer);
+
+ /* Is the whole page freezable? And is there something to freeze */
+ whole_page_freezable = presult->all_visible_except_removable &&
+ presult->all_frozen;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ * XXX: Previously, we knew if pruning emitted an FPI by checking
+ * pgWalUsage.wal_fpi before and after pruning. Once the freeze and prune
+ * records are combined, this heuristic couldn't be used anymore. The
+ * opportunistic freeze heuristic must be improved; however, for now, try
+ * to approximate it.
+ */
+ do_freeze = pagefrz &&
+ (pagefrz->freeze_required ||
+ (whole_page_freezable && presult->nfrozen > 0 && (prune_fpi || hint_bit_fpi)));
+
/* Any error while applying the changes is critical */
START_CRIT_SECTION();
/* Have we found any prunable items? */
- if (prstate.nredirected > 0 || prstate.ndead > 0 || prstate.nunused > 0)
+ if (do_prune)
{
/*
* Apply the planned item changes, then repair page fragmentation, and
@@ -579,20 +620,6 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- if (pagefrz)
- do_freeze = pagefrz->freeze_required ||
- (presult->all_visible_except_removable && presult->all_frozen &&
- presult->nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
- else
- do_freeze = false;
-
if (do_freeze)
{
/*
--
2.40.1
[text/x-diff] v4-0011-Separate-tuple-pre-freeze-checks-and-invoke-earli.patch (6.6K, ../../20240320013602.6sypr4cx6sefpemg@liskov/12-v4-0011-Separate-tuple-pre-freeze-checks-and-invoke-earli.patch)
download | inline diff:
From 2f9a262a6b85e3bfc25f2c066634ccb9958529e3 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Sun, 7 Jan 2024 16:53:45 -0500
Subject: [PATCH v4 11/19] Separate tuple pre freeze checks and invoke earlier
When combining the prune and freeze records their critical sections will
have to be combined. heap_freeze_execute_prepared() does a set of pre
freeze validations before starting its critical section. Move these
validations into a helper function, heap_pre_freeze_checks(), and invoke
it in heap_page_prune() before the pruning critical section.
Also move up the calculation of the freeze snapshot conflict horizon.
---
src/backend/access/heap/heapam.c | 58 ++++++++++++++++-------------
src/backend/access/heap/pruneheap.c | 31 ++++++++-------
src/include/access/heapam.h | 3 ++
3 files changed, 54 insertions(+), 38 deletions(-)
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 7261c4988d7..16e3f2520a4 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -6659,35 +6659,19 @@ heap_execute_freeze_tuple(HeapTupleHeader tuple, HeapTupleFreeze *frz)
}
/*
- * heap_freeze_execute_prepared
- *
- * Executes freezing of one or more heap tuples on a page on behalf of caller.
- * Caller passes an array of tuple plans from heap_prepare_freeze_tuple.
- * Caller must set 'offset' in each plan for us. Note that we destructively
- * sort caller's tuples array in-place, so caller had better be done with it.
- *
- * WAL-logs the changes so that VACUUM can advance the rel's relfrozenxid
- * later on without any risk of unsafe pg_xact lookups, even following a hard
- * crash (or when querying from a standby). We represent freezing by setting
- * infomask bits in tuple headers, but this shouldn't be thought of as a hint.
- * See section on buffer access rules in src/backend/storage/buffer/README.
- */
+* Perform xmin/xmax XID status sanity checks before calling
+* heap_freeze_execute_prepared().
+*
+* heap_prepare_freeze_tuple doesn't perform these checks directly because
+* pg_xact lookups are relatively expensive. They shouldn't be repeated
+* by successive VACUUMs that each decide against freezing the same page.
+*/
void
-heap_freeze_execute_prepared(Relation rel, Buffer buffer,
- TransactionId snapshotConflictHorizon,
- HeapTupleFreeze *tuples, int ntuples)
+heap_pre_freeze_checks(Buffer buffer,
+ HeapTupleFreeze *tuples, int ntuples)
{
Page page = BufferGetPage(buffer);
- Assert(ntuples > 0);
-
- /*
- * Perform xmin/xmax XID status sanity checks before critical section.
- *
- * heap_prepare_freeze_tuple doesn't perform these checks directly because
- * pg_xact lookups are relatively expensive. They shouldn't be repeated
- * by successive VACUUMs that each decide against freezing the same page.
- */
for (int i = 0; i < ntuples; i++)
{
HeapTupleFreeze *frz = tuples + i;
@@ -6726,6 +6710,30 @@ heap_freeze_execute_prepared(Relation rel, Buffer buffer,
xmax)));
}
}
+}
+
+/*
+ * heap_freeze_execute_prepared
+ *
+ * Executes freezing of one or more heap tuples on a page on behalf of caller.
+ * Caller passes an array of tuple plans from heap_prepare_freeze_tuple.
+ * Caller must set 'offset' in each plan for us. Note that we destructively
+ * sort caller's tuples array in-place, so caller had better be done with it.
+ *
+ * WAL-logs the changes so that VACUUM can advance the rel's relfrozenxid
+ * later on without any risk of unsafe pg_xact lookups, even following a hard
+ * crash (or when querying from a standby). We represent freezing by setting
+ * infomask bits in tuple headers, but this shouldn't be thought of as a hint.
+ * See section on buffer access rules in src/backend/storage/buffer/README.
+ */
+void
+heap_freeze_execute_prepared(Relation rel, Buffer buffer,
+ TransactionId snapshotConflictHorizon,
+ HeapTupleFreeze *tuples, int ntuples)
+{
+ Page page = BufferGetPage(buffer);
+
+ Assert(ntuples > 0);
START_CRIT_SECTION();
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 9edf6bf72d7..87f99497865 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -524,6 +524,24 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
(pagefrz->freeze_required ||
(whole_page_freezable && presult->nfrozen > 0 && (prune_fpi || hint_bit_fpi)));
+ if (do_freeze)
+ {
+ heap_pre_freeze_checks(buffer, prstate.frozen, presult->nfrozen);
+
+ /*
+ * We can use frz_conflict_horizon as our cutoff for conflicts when
+ * the whole page is eligible to become all-frozen in the VM once
+ * we're done with it. Otherwise we generate a conservative cutoff by
+ * stepping back from OldestXmin.
+ */
+ if (!(presult->all_visible_except_removable && presult->all_frozen))
+ {
+ /* Avoids false conflicts when hot_standby_feedback in use */
+ presult->frz_conflict_horizon = pagefrz->cutoffs->OldestXmin;
+ TransactionIdRetreat(presult->frz_conflict_horizon);
+ }
+ }
+
/* Any error while applying the changes is critical */
START_CRIT_SECTION();
@@ -622,19 +640,6 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
if (do_freeze)
{
- /*
- * We can use frz_conflict_horizon as our cutoff for conflicts when
- * the whole page is eligible to become all-frozen in the VM once
- * we're done with it. Otherwise we generate a conservative cutoff by
- * stepping back from OldestXmin.
- */
- if (!(presult->all_visible_except_removable && presult->all_frozen))
- {
- /* Avoids false conflicts when hot_standby_feedback in use */
- presult->frz_conflict_horizon = pagefrz->cutoffs->OldestXmin;
- TransactionIdRetreat(presult->frz_conflict_horizon);
- }
-
/* Execute all freeze plans for page as a single atomic action */
heap_freeze_execute_prepared(relation, buffer,
presult->frz_conflict_horizon,
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index b2a4caeb33a..02e33f213e1 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -312,6 +312,9 @@ extern void heap_inplace_update(Relation relation, HeapTuple tuple);
extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
HeapPageFreeze *pagefrz,
HeapTupleFreeze *frz, bool *totally_frozen);
+
+extern void heap_pre_freeze_checks(Buffer buffer,
+ HeapTupleFreeze *tuples, int ntuples);
extern void heap_freeze_execute_prepared(Relation rel, Buffer buffer,
TransactionId snapshotConflictHorizon,
HeapTupleFreeze *tuples, int ntuples);
--
2.40.1
[text/x-diff] v4-0012-Remove-heap_freeze_execute_prepared.patch (11.2K, ../../20240320013602.6sypr4cx6sefpemg@liskov/13-v4-0012-Remove-heap_freeze_execute_prepared.patch)
download | inline diff:
From 2a2e5407f8e6f8200dbda1b95cd3ec8d379282dd Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Sun, 7 Jan 2024 17:03:17 -0500
Subject: [PATCH v4 12/19] Remove heap_freeze_execute_prepared()
In order to merge freeze and prune records, the execution of tuple
freezing and the WAL logging of the changes to the page must be
separated so that the WAL logging can be combined with prune WAL
logging. This commit makes a helper for the tuple freezing and then
inlines the contents of heap_freeze_execute_prepared() where it is
called in heap_page_prune().
---
src/backend/access/heap/heapam.c | 79 +++++------------------------
src/backend/access/heap/pruneheap.c | 51 +++++++++++++++++--
src/include/access/heapam.h | 31 ++++++-----
3 files changed, 77 insertions(+), 84 deletions(-)
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 16e3f2520a4..e47b56e7856 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -91,9 +91,6 @@ static void compute_new_xmax_infomask(TransactionId xmax, uint16 old_infomask,
static TM_Result heap_lock_updated_tuple(Relation rel, HeapTuple tuple,
ItemPointer ctid, TransactionId xid,
LockTupleMode mode);
-static int heap_log_freeze_plan(HeapTupleFreeze *tuples, int ntuples,
- xl_heap_freeze_plan *plans_out,
- OffsetNumber *offsets_out);
static void GetMultiXactIdHintBits(MultiXactId multi, uint16 *new_infomask,
uint16 *new_infomask2);
static TransactionId MultiXactIdGetUpdateXid(TransactionId xmax,
@@ -6343,9 +6340,9 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
* XIDs or MultiXactIds that will need to be processed by a future VACUUM.
*
* VACUUM caller must assemble HeapTupleFreeze freeze plan entries for every
- * tuple that we returned true for, and call heap_freeze_execute_prepared to
- * execute freezing. Caller must initialize pagefrz fields for page as a
- * whole before first call here for each heap page.
+ * tuple that we returned true for, and then execute freezing. Caller must
+ * initialize pagefrz fields for page as a whole before first call here for
+ * each heap page.
*
* VACUUM caller decides on whether or not to freeze the page as a whole.
* We'll often prepare freeze plans for a page that caller just discards.
@@ -6659,8 +6656,8 @@ heap_execute_freeze_tuple(HeapTupleHeader tuple, HeapTupleFreeze *frz)
}
/*
-* Perform xmin/xmax XID status sanity checks before calling
-* heap_freeze_execute_prepared().
+* Perform xmin/xmax XID status sanity checks before actually executing freeze
+* plans.
*
* heap_prepare_freeze_tuple doesn't perform these checks directly because
* pg_xact lookups are relatively expensive. They shouldn't be repeated
@@ -6713,30 +6710,17 @@ heap_pre_freeze_checks(Buffer buffer,
}
/*
- * heap_freeze_execute_prepared
- *
- * Executes freezing of one or more heap tuples on a page on behalf of caller.
- * Caller passes an array of tuple plans from heap_prepare_freeze_tuple.
- * Caller must set 'offset' in each plan for us. Note that we destructively
- * sort caller's tuples array in-place, so caller had better be done with it.
- *
- * WAL-logs the changes so that VACUUM can advance the rel's relfrozenxid
- * later on without any risk of unsafe pg_xact lookups, even following a hard
- * crash (or when querying from a standby). We represent freezing by setting
- * infomask bits in tuple headers, but this shouldn't be thought of as a hint.
- * See section on buffer access rules in src/backend/storage/buffer/README.
+ * Helper which executes freezing of one or more heap tuples on a page on
+ * behalf of caller. Caller passes an array of tuple plans from
+ * heap_prepare_freeze_tuple. Caller must set 'offset' in each plan for us.
+ * Must be called in a critical section that also marks the buffer dirty and,
+ * if needed, emits WAL.
*/
void
-heap_freeze_execute_prepared(Relation rel, Buffer buffer,
- TransactionId snapshotConflictHorizon,
- HeapTupleFreeze *tuples, int ntuples)
+heap_freeze_prepared_tuples(Buffer buffer, HeapTupleFreeze *tuples, int ntuples)
{
Page page = BufferGetPage(buffer);
- Assert(ntuples > 0);
-
- START_CRIT_SECTION();
-
for (int i = 0; i < ntuples; i++)
{
HeapTupleFreeze *frz = tuples + i;
@@ -6746,45 +6730,6 @@ heap_freeze_execute_prepared(Relation rel, Buffer buffer,
htup = (HeapTupleHeader) PageGetItem(page, itemid);
heap_execute_freeze_tuple(htup, frz);
}
-
- MarkBufferDirty(buffer);
-
- /* Now WAL-log freezing if necessary */
- if (RelationNeedsWAL(rel))
- {
- xl_heap_freeze_plan plans[MaxHeapTuplesPerPage];
- OffsetNumber offsets[MaxHeapTuplesPerPage];
- int nplans;
- xl_heap_freeze_page xlrec;
- XLogRecPtr recptr;
-
- /* Prepare deduplicated representation for use in WAL record */
- nplans = heap_log_freeze_plan(tuples, ntuples, plans, offsets);
-
- xlrec.snapshotConflictHorizon = snapshotConflictHorizon;
- xlrec.isCatalogRel = RelationIsAccessibleInLogicalDecoding(rel);
- xlrec.nplans = nplans;
-
- XLogBeginInsert();
- XLogRegisterData((char *) &xlrec, SizeOfHeapFreezePage);
-
- /*
- * The freeze plan array and offset array are not actually in the
- * buffer, but pretend that they are. When XLogInsert stores the
- * whole buffer, the arrays need not be stored too.
- */
- XLogRegisterBuffer(0, buffer, REGBUF_STANDARD);
- XLogRegisterBufData(0, (char *) plans,
- nplans * sizeof(xl_heap_freeze_plan));
- XLogRegisterBufData(0, (char *) offsets,
- ntuples * sizeof(OffsetNumber));
-
- recptr = XLogInsert(RM_HEAP2_ID, XLOG_HEAP2_FREEZE_PAGE);
-
- PageSetLSN(page, recptr);
- }
-
- END_CRIT_SECTION();
}
/*
@@ -6874,7 +6819,7 @@ heap_log_freeze_new_plan(xl_heap_freeze_plan *plan, HeapTupleFreeze *frz)
* (actually there is one array per freeze plan, but that's not of immediate
* concern to our caller).
*/
-static int
+int
heap_log_freeze_plan(HeapTupleFreeze *tuples, int ntuples,
xl_heap_freeze_plan *plans_out,
OffsetNumber *offsets_out)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 87f99497865..7bd479cfd4e 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -640,10 +640,53 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
if (do_freeze)
{
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(relation, buffer,
- presult->frz_conflict_horizon,
- prstate.frozen, presult->nfrozen);
+ START_CRIT_SECTION();
+
+ Assert(presult->nfrozen > 0);
+
+ heap_freeze_prepared_tuples(buffer, prstate.frozen, presult->nfrozen);
+
+ MarkBufferDirty(buffer);
+
+ /* Now WAL-log freezing if necessary */
+ if (RelationNeedsWAL(relation))
+ {
+ xl_heap_freeze_plan plans[MaxHeapTuplesPerPage];
+ OffsetNumber offsets[MaxHeapTuplesPerPage];
+ int nplans;
+ xl_heap_freeze_page xlrec;
+ XLogRecPtr recptr;
+
+ /*
+ * Prepare deduplicated representation for use in WAL record
+ * Destructively sorts tuples array in-place.
+ */
+ nplans = heap_log_freeze_plan(prstate.frozen, presult->nfrozen, plans, offsets);
+
+ xlrec.snapshotConflictHorizon = presult->frz_conflict_horizon;
+ xlrec.isCatalogRel = RelationIsAccessibleInLogicalDecoding(relation);
+ xlrec.nplans = nplans;
+
+ XLogBeginInsert();
+ XLogRegisterData((char *) &xlrec, SizeOfHeapFreezePage);
+
+ /*
+ * The freeze plan array and offset array are not actually in the
+ * buffer, but pretend that they are. When XLogInsert stores the
+ * whole buffer, the arrays need not be stored too.
+ */
+ XLogRegisterBuffer(0, buffer, REGBUF_STANDARD);
+ XLogRegisterBufData(0, (char *) plans,
+ nplans * sizeof(xl_heap_freeze_plan));
+ XLogRegisterBufData(0, (char *) offsets,
+ presult->nfrozen * sizeof(OffsetNumber));
+
+ recptr = XLogInsert(RM_HEAP2_ID, XLOG_HEAP2_FREEZE_PAGE);
+
+ PageSetLSN(page, recptr);
+ }
+
+ END_CRIT_SECTION();
}
else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
{
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 02e33f213e1..321a46185e1 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -14,6 +14,7 @@
#ifndef HEAPAM_H
#define HEAPAM_H
+#include "access/heapam_xlog.h"
#include "access/relation.h" /* for backward compatibility */
#include "access/relscan.h"
#include "access/sdir.h"
@@ -101,8 +102,8 @@ typedef enum
} HTSV_Result;
/*
- * heap_prepare_freeze_tuple may request that heap_freeze_execute_prepared
- * check any tuple's to-be-frozen xmin and/or xmax status using pg_xact
+ * heap_prepare_freeze_tuple may request that any tuple's to-be-frozen xmin
+ * and/or xmax status is checked using pg_xact during freezing execution.
*/
#define HEAP_FREEZE_CHECK_XMIN_COMMITTED 0x01
#define HEAP_FREEZE_CHECK_XMAX_ABORTED 0x02
@@ -154,14 +155,14 @@ typedef struct HeapPageFreeze
/*
* "Freeze" NewRelfrozenXid/NewRelminMxid trackers.
*
- * Trackers used when heap_freeze_execute_prepared freezes, or when there
- * are zero freeze plans for a page. It is always valid for vacuumlazy.c
- * to freeze any page, by definition. This even includes pages that have
- * no tuples with storage to consider in the first place. That way the
- * 'totally_frozen' results from heap_prepare_freeze_tuple can always be
- * used in the same way, even when no freeze plans need to be executed to
- * "freeze the page". Only the "freeze" path needs to consider the need
- * to set pages all-frozen in the visibility map under this scheme.
+ * Trackers used when tuples will be frozen, or when there are zero freeze
+ * plans for a page. It is always valid for vacuumlazy.c to freeze any
+ * page, by definition. This even includes pages that have no tuples with
+ * storage to consider in the first place. That way the 'totally_frozen'
+ * results from heap_prepare_freeze_tuple can always be used in the same
+ * way, even when no freeze plans need to be executed to "freeze the
+ * page". Only the "freeze" path needs to consider the need to set pages
+ * all-frozen in the visibility map under this scheme.
*
* When we freeze a page, we generally freeze all XIDs < OldestXmin, only
* leaving behind XIDs that are ineligible for freezing, if any. And so
@@ -315,12 +316,16 @@ extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
extern void heap_pre_freeze_checks(Buffer buffer,
HeapTupleFreeze *tuples, int ntuples);
-extern void heap_freeze_execute_prepared(Relation rel, Buffer buffer,
- TransactionId snapshotConflictHorizon,
- HeapTupleFreeze *tuples, int ntuples);
+
+extern void heap_freeze_prepared_tuples(Buffer buffer,
+ HeapTupleFreeze *tuples, int ntuples);
extern bool heap_freeze_tuple(HeapTupleHeader tuple,
TransactionId relfrozenxid, TransactionId relminmxid,
TransactionId FreezeLimit, TransactionId MultiXactCutoff);
+
+extern int heap_log_freeze_plan(HeapTupleFreeze *tuples, int ntuples,
+ xl_heap_freeze_plan *plans_out,
+ OffsetNumber *offsets_out);
extern bool heap_tuple_should_freeze(HeapTupleHeader tuple,
const struct VacuumCutoffs *cutoffs,
TransactionId *NoFreezePageRelfrozenXid,
--
2.40.1
[text/x-diff] v4-0013-Merge-prune-and-freeze-records.patch (31.4K, ../../20240320013602.6sypr4cx6sefpemg@liskov/14-v4-0013-Merge-prune-and-freeze-records.patch)
download | inline diff:
From b13f4d1d5fb4e8fcb3f97fe1f0043fdfaf319b4c Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Sun, 7 Jan 2024 17:55:31 -0500
Subject: [PATCH v4 13/19] Merge prune and freeze records
Eliminate xl_heap_freeze and XLOG_HEAP2_FREEZE record. When vacuum
freezes tuples, the information needed to replay those changes is
now recorded in the xl_heap_prune record.
When both pruning and freezing is done, this means a single, combined
WAL record is emitted for both operations. This will reduce the number
of WAL records emitted.
When there are only tuples to freeze present, we can avoid taking a full
cleanup lock when replaying the record.
The XLOG_HEAP2_PRUNE record is now bigger than it was previously and
bigger than the XLOG_HEAP2_FREEZE record. A future commit will
streamline the record.
---
src/backend/access/heap/heapam.c | 146 ++++------
src/backend/access/heap/pruneheap.c | 326 ++++++++++++-----------
src/backend/access/rmgrdesc/heapdesc.c | 95 ++++---
src/backend/replication/logical/decode.c | 1 -
src/include/access/heapam_xlog.h | 97 ++++---
5 files changed, 318 insertions(+), 347 deletions(-)
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index e47b56e7856..532868039d5 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -8706,8 +8706,6 @@ ExtractReplicaIdentity(Relation relation, HeapTuple tp, bool key_required,
/*
* Handles XLOG_HEAP2_PRUNE record type.
- *
- * Acquires a full cleanup lock.
*/
static void
heap_xlog_prune(XLogReaderState *record)
@@ -8718,12 +8716,22 @@ heap_xlog_prune(XLogReaderState *record)
RelFileLocator rlocator;
BlockNumber blkno;
XLogRedoAction action;
+ bool get_cleanup_lock;
XLogRecGetBlockTag(record, 0, &rlocator, NULL, &blkno);
/*
- * We're about to remove tuples. In Hot Standby mode, ensure that there's
- * no queries running for which the removed tuples are still visible.
+ * If there are dead, redirected, or unused items set unused by
+ * heap_page_prune_and_freeze(), heap_page_prune_execute() will call
+ * PageRepairFragementation() which expects a full cleanup lock.
+ */
+ get_cleanup_lock = xlrec->nredirected > 0 ||
+ xlrec->ndead > 0 || xlrec->nunused > 0;
+
+ /*
+ * We are either about to remove tuples or freeze them. In Hot Standby
+ * mode, ensure that there's no queries running for which any removed
+ * tuples are still visible or which consider the frozen xids as running.
*/
if (InHotStandby)
ResolveRecoveryConflictWithSnapshot(xlrec->snapshotConflictHorizon,
@@ -8731,38 +8739,69 @@ heap_xlog_prune(XLogReaderState *record)
rlocator);
/*
- * If we have a full-page image, restore it (using a cleanup lock) and
- * we're done.
+ * If we have a full-page image, restore it and we're done.
*/
- action = XLogReadBufferForRedoExtended(record, 0, RBM_NORMAL, true,
- &buffer);
+ action = XLogReadBufferForRedoExtended(record, 0, RBM_NORMAL,
+ get_cleanup_lock, &buffer);
+
if (action == BLK_NEEDS_REDO)
{
Page page = (Page) BufferGetPage(buffer);
- OffsetNumber *end;
OffsetNumber *redirected;
OffsetNumber *nowdead;
OffsetNumber *nowunused;
int nredirected;
int ndead;
int nunused;
+ int nplans;
Size datalen;
+ xl_heap_freeze_plan *plans;
+ OffsetNumber *frz_offsets;
+ int curoff = 0;
- redirected = (OffsetNumber *) XLogRecGetBlockData(record, 0, &datalen);
-
+ nplans = xlrec->nplans;
nredirected = xlrec->nredirected;
ndead = xlrec->ndead;
- end = (OffsetNumber *) ((char *) redirected + datalen);
+ nunused = xlrec->nunused;
+
+ plans = (xl_heap_freeze_plan *) XLogRecGetBlockData(record, 0, &datalen);
+ redirected = (OffsetNumber *) &plans[nplans];
nowdead = redirected + (nredirected * 2);
nowunused = nowdead + ndead;
- nunused = (end - nowunused);
- Assert(nunused >= 0);
+ frz_offsets = nowunused + nunused;
/* Update all line pointers per the record, and repair fragmentation */
- heap_page_prune_execute(buffer,
- redirected, nredirected,
- nowdead, ndead,
- nowunused, nunused);
+ if (nredirected > 0 || ndead > 0 || nunused > 0)
+ heap_page_prune_execute(buffer,
+ redirected, nredirected,
+ nowdead, ndead,
+ nowunused, nunused);
+
+ for (int p = 0; p < nplans; p++)
+ {
+ HeapTupleFreeze frz;
+
+ /*
+ * Convert freeze plan representation from WAL record into
+ * per-tuple format used by heap_execute_freeze_tuple
+ */
+ frz.xmax = plans[p].xmax;
+ frz.t_infomask2 = plans[p].t_infomask2;
+ frz.t_infomask = plans[p].t_infomask;
+ frz.frzflags = plans[p].frzflags;
+ frz.offset = InvalidOffsetNumber; /* unused, but be tidy */
+
+ for (int i = 0; i < plans[p].ntuples; i++)
+ {
+ OffsetNumber offset = frz_offsets[curoff++];
+ ItemId lp;
+ HeapTupleHeader tuple;
+
+ lp = PageGetItemId(page, offset);
+ tuple = (HeapTupleHeader) PageGetItem(page, lp);
+ heap_execute_freeze_tuple(tuple, &frz);
+ }
+ }
/*
* Note: we don't worry about updating the page's prunability hints.
@@ -9001,74 +9040,6 @@ heap_xlog_visible(XLogReaderState *record)
UnlockReleaseBuffer(vmbuffer);
}
-/*
- * Replay XLOG_HEAP2_FREEZE_PAGE records
- */
-static void
-heap_xlog_freeze_page(XLogReaderState *record)
-{
- XLogRecPtr lsn = record->EndRecPtr;
- xl_heap_freeze_page *xlrec = (xl_heap_freeze_page *) XLogRecGetData(record);
- Buffer buffer;
-
- /*
- * In Hot Standby mode, ensure that there's no queries running which still
- * consider the frozen xids as running.
- */
- if (InHotStandby)
- {
- RelFileLocator rlocator;
-
- XLogRecGetBlockTag(record, 0, &rlocator, NULL, NULL);
- ResolveRecoveryConflictWithSnapshot(xlrec->snapshotConflictHorizon,
- xlrec->isCatalogRel,
- rlocator);
- }
-
- if (XLogReadBufferForRedo(record, 0, &buffer) == BLK_NEEDS_REDO)
- {
- Page page = BufferGetPage(buffer);
- xl_heap_freeze_plan *plans;
- OffsetNumber *offsets;
- int curoff = 0;
-
- plans = (xl_heap_freeze_plan *) XLogRecGetBlockData(record, 0, NULL);
- offsets = (OffsetNumber *) ((char *) plans +
- (xlrec->nplans *
- sizeof(xl_heap_freeze_plan)));
- for (int p = 0; p < xlrec->nplans; p++)
- {
- HeapTupleFreeze frz;
-
- /*
- * Convert freeze plan representation from WAL record into
- * per-tuple format used by heap_execute_freeze_tuple
- */
- frz.xmax = plans[p].xmax;
- frz.t_infomask2 = plans[p].t_infomask2;
- frz.t_infomask = plans[p].t_infomask;
- frz.frzflags = plans[p].frzflags;
- frz.offset = InvalidOffsetNumber; /* unused, but be tidy */
-
- for (int i = 0; i < plans[p].ntuples; i++)
- {
- OffsetNumber offset = offsets[curoff++];
- ItemId lp;
- HeapTupleHeader tuple;
-
- lp = PageGetItemId(page, offset);
- tuple = (HeapTupleHeader) PageGetItem(page, lp);
- heap_execute_freeze_tuple(tuple, &frz);
- }
- }
-
- PageSetLSN(page, lsn);
- MarkBufferDirty(buffer);
- }
- if (BufferIsValid(buffer))
- UnlockReleaseBuffer(buffer);
-}
-
/*
* Given an "infobits" field from an XLog record, set the correct bits in the
* given infomask and infomask2 for the tuple touched by the record.
@@ -9975,9 +9946,6 @@ heap2_redo(XLogReaderState *record)
case XLOG_HEAP2_VACUUM:
heap_xlog_vacuum(record);
break;
- case XLOG_HEAP2_FREEZE_PAGE:
- heap_xlog_freeze_page(record);
- break;
case XLOG_HEAP2_VISIBLE:
heap_xlog_visible(record);
break;
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 7bd479cfd4e..19b50931b90 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -79,6 +79,9 @@ static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber o
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
+static void log_heap_prune_and_freeze(Relation relation, Buffer buffer,
+ PruneState *prstate, PruneFreezeResult *presult);
+
/*
* Optionally prune and repair fragmentation in the specified page.
@@ -247,9 +250,9 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
HeapTupleData tup;
bool do_freeze;
bool do_prune;
+ bool do_hint;
bool whole_page_freezable;
bool hint_bit_fpi;
- bool prune_fpi = false;
int64 fpi_before = pgWalUsage.wal_fpi;
/*
@@ -445,10 +448,9 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
/*
* If checksums are enabled, heap_prune_satisfies_vacuum() may have caused
- * an FPI to be emitted. Then reset fpi_before for no prune case.
+ * an FPI to be emitted.
*/
hint_bit_fpi = fpi_before != pgWalUsage.wal_fpi;
- fpi_before = pgWalUsage.wal_fpi;
/*
* For vacuum, if the whole page will become frozen, we consider
@@ -498,14 +500,18 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
prstate.ndead > 0 ||
prstate.nunused > 0;
+ /* Record number of newly-set-LP_DEAD items for caller */
+ presult->nnewlpdead = prstate.ndead;
+
/*
- * Only incur overhead of checking if we will do an FPI if we might use
- * the information.
+ * Even if we don't prune anything, if we found a new value for the
+ * pd_prune_xid field or the page was marked full, we will update the hint
+ * bit.
*/
- if (do_prune && pagefrz)
- prune_fpi = XLogCheckBufferNeedsBackup(buffer);
+ do_hint = ((PageHeader) page)->pd_prune_xid != prstate.new_prune_xid ||
+ PageIsFull(page);
- /* Is the whole page freezable? And is there something to freeze */
+ /* Is the whole page freezable? And is there something to freeze? */
whole_page_freezable = presult->all_visible_except_removable &&
presult->all_frozen;
@@ -520,43 +526,51 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
* opportunistic freeze heuristic must be improved; however, for now, try
* to approximate it.
*/
- do_freeze = pagefrz &&
- (pagefrz->freeze_required ||
- (whole_page_freezable && presult->nfrozen > 0 && (prune_fpi || hint_bit_fpi)));
+ do_freeze = false;
+ if (pagefrz)
+ {
+ if (pagefrz->freeze_required)
+ do_freeze = true;
+ else if (whole_page_freezable && presult->nfrozen > 0)
+ {
+ /*
+ * Freezing would make the page all-frozen. In this case, we will
+ * freeze if we have already emitted an FPI or will do so anyway.
+ * Be sure only to incur the overhead of checking if we will do an
+ * FPI if we may use that information.
+ */
+ if (hint_bit_fpi ||
+ ((do_prune || do_hint) && XLogCheckBufferNeedsBackup(buffer)))
+ {
+ do_freeze = true;
+ }
+ }
+ }
+ /*
+ * Validate the tuples we are considering freezing. We do this even if
+ * pruning and hint bit setting have not emitted an FPI so far because we
+ * still may emit an FPI while setting the page hint bit later. But we
+ * want to avoid doing the pre-freeze checks in a critical section.
+ */
if (do_freeze)
- {
heap_pre_freeze_checks(buffer, prstate.frozen, presult->nfrozen);
+ if (!do_freeze && (!pagefrz || !presult->all_frozen || presult->nfrozen > 0))
+ {
/*
- * We can use frz_conflict_horizon as our cutoff for conflicts when
- * the whole page is eligible to become all-frozen in the VM once
- * we're done with it. Otherwise we generate a conservative cutoff by
- * stepping back from OldestXmin.
+ * If we will neither freeze tuples on the page nor set the page all
+ * frozen in the visibility map, the page is not all-frozen and there
+ * will be no newly frozen tuples.
*/
- if (!(presult->all_visible_except_removable && presult->all_frozen))
- {
- /* Avoids false conflicts when hot_standby_feedback in use */
- presult->frz_conflict_horizon = pagefrz->cutoffs->OldestXmin;
- TransactionIdRetreat(presult->frz_conflict_horizon);
- }
+ presult->all_frozen = false;
+ presult->nfrozen = 0; /* avoid miscounts in instrumentation */
}
- /* Any error while applying the changes is critical */
START_CRIT_SECTION();
- /* Have we found any prunable items? */
- if (do_prune)
+ if (do_hint)
{
- /*
- * Apply the planned item changes, then repair page fragmentation, and
- * update the page's hint bit about whether it has free line pointers.
- */
- heap_page_prune_execute(buffer,
- prstate.redirected, prstate.nredirected,
- prstate.nowdead, prstate.ndead,
- prstate.nowunused, prstate.nunused);
-
/*
* Update the page's pd_prune_xid field to either zero, or the lowest
* XID of any soon-prunable tuple.
@@ -564,163 +578,159 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
((PageHeader) page)->pd_prune_xid = prstate.new_prune_xid;
/*
- * Also clear the "page is full" flag, since there's no point in
- * repeating the prune/defrag process until something else happens to
- * the page.
+ * Clear the "page is full" flag if it is set since there's no point
+ * in repeating the prune/defrag process until something else happens
+ * to the page.
*/
PageClearFull(page);
- MarkBufferDirty(buffer);
+ /*
+ * We only needed to update pd_prune_xid and clear the page-is-full
+ * hint bit, this is a non-WAL-logged hint. If we will also freeze or
+ * prune the page, we will mark the buffer dirty below.
+ */
+ if (!do_freeze && !do_prune)
+ MarkBufferDirtyHint(buffer, true);
+ }
+ if (do_prune || do_freeze)
+ {
/*
- * Emit a WAL XLOG_HEAP2_PRUNE record showing what we did
+ * Apply the planned item changes, then repair page fragmentation, and
+ * update the page's hint bit about whether it has free line pointers.
*/
- if (RelationNeedsWAL(relation))
+ if (do_prune)
{
- xl_heap_prune xlrec;
- XLogRecPtr recptr;
-
- xlrec.isCatalogRel = RelationIsAccessibleInLogicalDecoding(relation);
- xlrec.snapshotConflictHorizon = prstate.snapshotConflictHorizon;
- xlrec.nredirected = prstate.nredirected;
- xlrec.ndead = prstate.ndead;
-
- XLogBeginInsert();
- XLogRegisterData((char *) &xlrec, SizeOfHeapPrune);
-
- XLogRegisterBuffer(0, buffer, REGBUF_STANDARD);
+ heap_page_prune_execute(buffer,
+ prstate.redirected, prstate.nredirected,
+ prstate.nowdead, prstate.ndead,
+ prstate.nowunused, prstate.nunused);
+ }
+ if (do_freeze)
+ {
/*
- * The OffsetNumber arrays are not actually in the buffer, but we
- * pretend that they are. When XLogInsert stores the whole
- * buffer, the offset arrays need not be stored too.
+ * We can use frz_conflict_horizon as our cutoff for conflicts
+ * when the whole page is eligible to become all-frozen in the VM
+ * once we're done with it. Otherwise we generate a conservative
+ * cutoff by stepping back from OldestXmin. This avoids false
+ * conflicts when hot_standby_feedback is in use.
*/
- if (prstate.nredirected > 0)
- XLogRegisterBufData(0, (char *) prstate.redirected,
- prstate.nredirected *
- sizeof(OffsetNumber) * 2);
-
- if (prstate.ndead > 0)
- XLogRegisterBufData(0, (char *) prstate.nowdead,
- prstate.ndead * sizeof(OffsetNumber));
-
- if (prstate.nunused > 0)
- XLogRegisterBufData(0, (char *) prstate.nowunused,
- prstate.nunused * sizeof(OffsetNumber));
+ if (!(presult->all_visible_except_removable && presult->all_frozen))
+ {
+ presult->frz_conflict_horizon = pagefrz->cutoffs->OldestXmin;
+ TransactionIdRetreat(presult->frz_conflict_horizon);
+ }
+ heap_freeze_prepared_tuples(buffer, prstate.frozen, presult->nfrozen);
+ }
- recptr = XLogInsert(RM_HEAP2_ID, XLOG_HEAP2_PRUNE);
+ MarkBufferDirty(buffer);
- PageSetLSN(BufferGetPage(buffer), recptr);
- }
- }
- else
- {
/*
- * If we didn't prune anything, but have found a new value for the
- * pd_prune_xid field, update it and mark the buffer dirty. This is
- * treated as a non-WAL-logged hint.
- *
- * Also clear the "page is full" flag if it is set, since there's no
- * point in repeating the prune/defrag process until something else
- * happens to the page.
+ * Emit a WAL XLOG_HEAP2_PRUNE record showing what we did
*/
- if (((PageHeader) page)->pd_prune_xid != prstate.new_prune_xid ||
- PageIsFull(page))
- {
- ((PageHeader) page)->pd_prune_xid = prstate.new_prune_xid;
- PageClearFull(page);
- MarkBufferDirtyHint(buffer, true);
- }
+ if (RelationNeedsWAL(relation))
+ log_heap_prune_and_freeze(relation, buffer, &prstate, presult);
}
END_CRIT_SECTION();
- /* Record number of newly-set-LP_DEAD items for caller */
- presult->nnewlpdead = prstate.ndead;
-
- if (do_freeze)
+ /*
+ * If we froze tuples on the page, the caller can advance relfrozenxid and
+ * relminmxid to the values in pagefrz->FreezePageRelfrozenXid and
+ * pagefrz->FreezePageRelminMxid. Otherwise, it is only safe to advance to
+ * the values in pagefrz->NoFreezePage[RelfrozenXid|RelminMxid]
+ */
+ if (pagefrz)
{
- START_CRIT_SECTION();
+ if (presult->nfrozen > 0)
+ {
+ presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
+ }
+ else
+ {
+ presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
+ }
+ }
+}
- Assert(presult->nfrozen > 0);
- heap_freeze_prepared_tuples(buffer, prstate.frozen, presult->nfrozen);
+static void
+log_heap_prune_and_freeze(Relation relation, Buffer buffer,
+ PruneState *prstate, PruneFreezeResult *presult)
+{
+ xl_heap_prune xlrec;
+ XLogRecPtr recptr;
- MarkBufferDirty(buffer);
+ xl_heap_freeze_plan plans[MaxHeapTuplesPerPage];
+ OffsetNumber offsets[MaxHeapTuplesPerPage];
+ bool do_freeze = presult->nfrozen > 0;
- /* Now WAL-log freezing if necessary */
- if (RelationNeedsWAL(relation))
- {
- xl_heap_freeze_plan plans[MaxHeapTuplesPerPage];
- OffsetNumber offsets[MaxHeapTuplesPerPage];
- int nplans;
- xl_heap_freeze_page xlrec;
- XLogRecPtr recptr;
+ xlrec.isCatalogRel = RelationIsAccessibleInLogicalDecoding(relation);
+ xlrec.nredirected = prstate->nredirected;
+ xlrec.ndead = prstate->ndead;
+ xlrec.nunused = prstate->nunused;
+ xlrec.nplans = 0;
- /*
- * Prepare deduplicated representation for use in WAL record
- * Destructively sorts tuples array in-place.
- */
- nplans = heap_log_freeze_plan(prstate.frozen, presult->nfrozen, plans, offsets);
+ /*
+ * The snapshotConflictHorizon for the whole record should be the most
+ * conservative of all the horizons calculated for any of the possible
+ * modifications. If this record will prune tuples, any transactions on
+ * the standby older than the youngest xmax of the most recently removed
+ * tuple this record will prune will conflict. If this record will freeze
+ * tuples, any transactions on the standby with xids older than the
+ * youngest tuple this record will freeze will conflict.
+ */
+ if (do_freeze)
+ xlrec.snapshotConflictHorizon = Max(prstate->snapshotConflictHorizon,
+ presult->frz_conflict_horizon);
+ else
+ xlrec.snapshotConflictHorizon = prstate->snapshotConflictHorizon;
- xlrec.snapshotConflictHorizon = presult->frz_conflict_horizon;
- xlrec.isCatalogRel = RelationIsAccessibleInLogicalDecoding(relation);
- xlrec.nplans = nplans;
+ /*
+ * Prepare deduplicated representation for use in WAL record Destructively
+ * sorts tuples array in-place.
+ */
+ if (do_freeze)
+ xlrec.nplans = heap_log_freeze_plan(prstate->frozen,
+ presult->nfrozen, plans, offsets);
- XLogBeginInsert();
- XLogRegisterData((char *) &xlrec, SizeOfHeapFreezePage);
+ XLogBeginInsert();
+ XLogRegisterData((char *) &xlrec, SizeOfHeapPrune);
- /*
- * The freeze plan array and offset array are not actually in the
- * buffer, but pretend that they are. When XLogInsert stores the
- * whole buffer, the arrays need not be stored too.
- */
- XLogRegisterBuffer(0, buffer, REGBUF_STANDARD);
- XLogRegisterBufData(0, (char *) plans,
- nplans * sizeof(xl_heap_freeze_plan));
- XLogRegisterBufData(0, (char *) offsets,
- presult->nfrozen * sizeof(OffsetNumber));
+ XLogRegisterBuffer(0, buffer, REGBUF_STANDARD);
- recptr = XLogInsert(RM_HEAP2_ID, XLOG_HEAP2_FREEZE_PAGE);
+ /*
+ * The OffsetNumber arrays are not actually in the buffer, but we pretend
+ * that they are. When XLogInsert stores the whole buffer, the offset
+ * arrays need not be stored too.
+ */
+ if (xlrec.nplans > 0)
+ XLogRegisterBufData(0, (char *) plans,
+ xlrec.nplans * sizeof(xl_heap_freeze_plan));
- PageSetLSN(page, recptr);
- }
+ if (prstate->nredirected > 0)
+ XLogRegisterBufData(0, (char *) prstate->redirected,
+ prstate->nredirected *
+ sizeof(OffsetNumber) * 2);
- END_CRIT_SECTION();
- }
- else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
- {
- /*
- * If we will neither freeze tuples on the page nor set the page all
- * frozen in the visibility map, the page is not all frozen and there
- * will be no newly frozen tuples.
- */
- presult->all_frozen = false;
- presult->nfrozen = 0; /* avoid miscounts in instrumentation */
- }
+ if (prstate->ndead > 0)
+ XLogRegisterBufData(0, (char *) prstate->nowdead,
+ prstate->ndead * sizeof(OffsetNumber));
- /* Caller won't update new_relfrozenxid and new_relminmxid */
- if (!pagefrz)
- return;
+ if (prstate->nunused > 0)
+ XLogRegisterBufData(0, (char *) prstate->nowunused,
+ prstate->nunused * sizeof(OffsetNumber));
+ if (xlrec.nplans > 0)
+ XLogRegisterBufData(0, (char *) offsets,
+ presult->nfrozen * sizeof(OffsetNumber));
- /*
- * If we will freeze tuples on the page or, even if we don't freeze tuples
- * on the page, if we will set the page all-frozen in the visibility map,
- * we can advance relfrozenxid and relminmxid to the values in
- * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid.
- */
- if (presult->all_frozen || presult->nfrozen > 0)
- {
- presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
- presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
- }
- else
- {
- presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
- presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
- }
-}
+ recptr = XLogInsert(RM_HEAP2_ID, XLOG_HEAP2_PRUNE);
+ PageSetLSN(BufferGetPage(buffer), recptr);
+}
/*
* Perform visibility checks for heap pruning.
diff --git a/src/backend/access/rmgrdesc/heapdesc.c b/src/backend/access/rmgrdesc/heapdesc.c
index 36a3d83c8c2..9f0a0341d40 100644
--- a/src/backend/access/rmgrdesc/heapdesc.c
+++ b/src/backend/access/rmgrdesc/heapdesc.c
@@ -179,43 +179,67 @@ heap2_desc(StringInfo buf, XLogReaderState *record)
{
xl_heap_prune *xlrec = (xl_heap_prune *) rec;
- appendStringInfo(buf, "snapshotConflictHorizon: %u, nredirected: %u, ndead: %u, isCatalogRel: %c",
+ appendStringInfo(buf, "snapshotConflictHorizon: %u, isCatalogRel: %c",
xlrec->snapshotConflictHorizon,
- xlrec->nredirected,
- xlrec->ndead,
xlrec->isCatalogRel ? 'T' : 'F');
if (XLogRecHasBlockData(record, 0))
{
- OffsetNumber *end;
OffsetNumber *redirected;
OffsetNumber *nowdead;
OffsetNumber *nowunused;
int nredirected;
+ int ndead;
int nunused;
+ int nplans;
Size datalen;
+ xl_heap_freeze_plan *plans;
+ OffsetNumber *frz_offsets;
- redirected = (OffsetNumber *) XLogRecGetBlockData(record, 0,
- &datalen);
-
+ nplans = xlrec->nplans;
nredirected = xlrec->nredirected;
- end = (OffsetNumber *) ((char *) redirected + datalen);
- nowdead = redirected + (nredirected * 2);
- nowunused = nowdead + xlrec->ndead;
- nunused = (end - nowunused);
- Assert(nunused >= 0);
+ ndead = xlrec->ndead;
+ nunused = xlrec->nunused;
- appendStringInfo(buf, ", nunused: %d", nunused);
-
- appendStringInfoString(buf, ", redirected:");
- array_desc(buf, redirected, sizeof(OffsetNumber) * 2,
- nredirected, &redirect_elem_desc, NULL);
- appendStringInfoString(buf, ", dead:");
- array_desc(buf, nowdead, sizeof(OffsetNumber), xlrec->ndead,
- &offset_elem_desc, NULL);
- appendStringInfoString(buf, ", unused:");
- array_desc(buf, nowunused, sizeof(OffsetNumber), nunused,
- &offset_elem_desc, NULL);
+ plans = (xl_heap_freeze_plan *) XLogRecGetBlockData(record, 0, &datalen);
+ redirected = (OffsetNumber *) &plans[nplans];
+ nowdead = redirected + (nredirected * 2);
+ nowunused = nowdead + ndead;
+ frz_offsets = nowunused + nunused;
+
+ appendStringInfo(buf, ", nredirected: %u, ndead: %u, nunused: %u, nplans: %u,",
+ nredirected,
+ ndead,
+ nunused,
+ nplans);
+
+ if (nredirected > 0)
+ {
+ appendStringInfoString(buf, ", redirected:");
+ array_desc(buf, redirected, sizeof(OffsetNumber) * 2,
+ nredirected, &redirect_elem_desc, NULL);
+ }
+
+ if (ndead > 0)
+ {
+ appendStringInfoString(buf, ", dead:");
+ array_desc(buf, nowdead, sizeof(OffsetNumber), ndead,
+ &offset_elem_desc, NULL);
+ }
+
+ if (nunused > 0)
+ {
+ appendStringInfoString(buf, ", unused:");
+ array_desc(buf, nowunused, sizeof(OffsetNumber), nunused,
+ &offset_elem_desc, NULL);
+ }
+
+ if (nplans > 0)
+ {
+ appendStringInfoString(buf, ", plans:");
+ array_desc(buf, plans, sizeof(xl_heap_freeze_plan), nplans,
+ &plan_elem_desc, &frz_offsets);
+ }
}
}
else if (info == XLOG_HEAP2_VACUUM)
@@ -235,28 +259,6 @@ heap2_desc(StringInfo buf, XLogReaderState *record)
&offset_elem_desc, NULL);
}
}
- else if (info == XLOG_HEAP2_FREEZE_PAGE)
- {
- xl_heap_freeze_page *xlrec = (xl_heap_freeze_page *) rec;
-
- appendStringInfo(buf, "snapshotConflictHorizon: %u, nplans: %u, isCatalogRel: %c",
- xlrec->snapshotConflictHorizon, xlrec->nplans,
- xlrec->isCatalogRel ? 'T' : 'F');
-
- if (XLogRecHasBlockData(record, 0))
- {
- xl_heap_freeze_plan *plans;
- OffsetNumber *offsets;
-
- plans = (xl_heap_freeze_plan *) XLogRecGetBlockData(record, 0, NULL);
- offsets = (OffsetNumber *) ((char *) plans +
- (xlrec->nplans *
- sizeof(xl_heap_freeze_plan)));
- appendStringInfoString(buf, ", plans:");
- array_desc(buf, plans, sizeof(xl_heap_freeze_plan), xlrec->nplans,
- &plan_elem_desc, &offsets);
- }
- }
else if (info == XLOG_HEAP2_VISIBLE)
{
xl_heap_visible *xlrec = (xl_heap_visible *) rec;
@@ -361,9 +363,6 @@ heap2_identify(uint8 info)
case XLOG_HEAP2_VACUUM:
id = "VACUUM";
break;
- case XLOG_HEAP2_FREEZE_PAGE:
- id = "FREEZE_PAGE";
- break;
case XLOG_HEAP2_VISIBLE:
id = "VISIBLE";
break;
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index e5ab7b78b78..f77051572fd 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -445,7 +445,6 @@ heap2_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
* Everything else here is just low level physical stuff we're not
* interested in.
*/
- case XLOG_HEAP2_FREEZE_PAGE:
case XLOG_HEAP2_PRUNE:
case XLOG_HEAP2_VACUUM:
case XLOG_HEAP2_VISIBLE:
diff --git a/src/include/access/heapam_xlog.h b/src/include/access/heapam_xlog.h
index 6488dad5e64..fe4a8ff0620 100644
--- a/src/include/access/heapam_xlog.h
+++ b/src/include/access/heapam_xlog.h
@@ -53,11 +53,10 @@
#define XLOG_HEAP2_REWRITE 0x00
#define XLOG_HEAP2_PRUNE 0x10
#define XLOG_HEAP2_VACUUM 0x20
-#define XLOG_HEAP2_FREEZE_PAGE 0x30
-#define XLOG_HEAP2_VISIBLE 0x40
-#define XLOG_HEAP2_MULTI_INSERT 0x50
-#define XLOG_HEAP2_LOCK_UPDATED 0x60
-#define XLOG_HEAP2_NEW_CID 0x70
+#define XLOG_HEAP2_VISIBLE 0x30
+#define XLOG_HEAP2_MULTI_INSERT 0x40
+#define XLOG_HEAP2_LOCK_UPDATED 0x50
+#define XLOG_HEAP2_NEW_CID 0x60
/*
* xl_heap_insert/xl_heap_multi_insert flag values, 8 bits are available.
@@ -226,28 +225,65 @@ typedef struct xl_heap_update
#define SizeOfHeapUpdate (offsetof(xl_heap_update, new_offnum) + sizeof(OffsetNumber))
+/*
+ * This struct represents a 'freeze plan', which describes how to freeze a
+ * group of one or more heap tuples (appears in xl_heap_prune record)
+ */
+/* 0x01 was XLH_FREEZE_XMIN */
+#define XLH_FREEZE_XVAC 0x02
+#define XLH_INVALID_XVAC 0x04
+
+typedef struct xl_heap_freeze_plan
+{
+ TransactionId xmax;
+ uint16 t_infomask2;
+ uint16 t_infomask;
+ uint8 frzflags;
+
+ /* Length of individual page offset numbers array for this plan */
+ uint16 ntuples;
+} xl_heap_freeze_plan;
+
+/*
+ * As of Postgres 17, XLOG_HEAP2_PRUNE records replace
+ * XLOG_HEAP2_FREEZE_PAGE records.
+ */
+
/*
* This is what we need to know about page pruning (both during VACUUM and
* during opportunistic pruning)
*
* The array of OffsetNumbers following the fixed part of the record contains:
+ * * for each freeze plan: the freeze plan
* * for each redirected item: the item offset, then the offset redirected to
* * for each now-dead item: the item offset
* * for each now-unused item: the item offset
- * The total number of OffsetNumbers is therefore 2*nredirected+ndead+nunused.
- * Note that nunused is not explicitly stored, but may be found by reference
- * to the total record length.
+ * * for each tuple frozen by the freeze plans: the offset of the item corresponding to that tuple
+ * The total number of OffsetNumbers is therefore
+ * (2*nredirected) + ndead + nunused + (sum[plan.ntuples for plan in plans])
*
- * Acquires a full cleanup lock.
+ * Acquires a full cleanup lock if heap_page_prune_execute() must be called
*/
typedef struct xl_heap_prune
{
TransactionId snapshotConflictHorizon;
+ uint16 nplans;
uint16 nredirected;
uint16 ndead;
+ uint16 nunused;
bool isCatalogRel; /* to handle recovery conflict during logical
* decoding on standby */
- /* OFFSET NUMBERS are in the block reference 0 */
+ /*--------------------------------------------------------------------
+ * OFFSET NUMBERS and freeze plans are in the block reference 0 in the
+ * following order:
+ *
+ * * xl_heap_freeze_plan plans[nplans];
+ * * OffsetNumber redirected[2 * nredirected];
+ * * OffsetNumber nowdead[ndead];
+ * * OffsetNumber nowunused[nunused];
+ * * OffsetNumber frz_offsets[...];
+ *--------------------------------------------------------------------
+ */
} xl_heap_prune;
#define SizeOfHeapPrune (offsetof(xl_heap_prune, isCatalogRel) + sizeof(bool))
@@ -315,47 +351,6 @@ typedef struct xl_heap_inplace
#define SizeOfHeapInplace (offsetof(xl_heap_inplace, offnum) + sizeof(OffsetNumber))
-/*
- * This struct represents a 'freeze plan', which describes how to freeze a
- * group of one or more heap tuples (appears in xl_heap_freeze_page record)
- */
-/* 0x01 was XLH_FREEZE_XMIN */
-#define XLH_FREEZE_XVAC 0x02
-#define XLH_INVALID_XVAC 0x04
-
-typedef struct xl_heap_freeze_plan
-{
- TransactionId xmax;
- uint16 t_infomask2;
- uint16 t_infomask;
- uint8 frzflags;
-
- /* Length of individual page offset numbers array for this plan */
- uint16 ntuples;
-} xl_heap_freeze_plan;
-
-/*
- * This is what we need to know about a block being frozen during vacuum
- *
- * Backup block 0's data contains an array of xl_heap_freeze_plan structs
- * (with nplans elements), followed by one or more page offset number arrays.
- * Each such page offset number array corresponds to a single freeze plan
- * (REDO routine freezes corresponding heap tuples using freeze plan).
- */
-typedef struct xl_heap_freeze_page
-{
- TransactionId snapshotConflictHorizon;
- uint16 nplans;
- bool isCatalogRel; /* to handle recovery conflict during logical
- * decoding on standby */
-
- /*
- * In payload of blk 0 : FREEZE PLANS and OFFSET NUMBER ARRAY
- */
-} xl_heap_freeze_page;
-
-#define SizeOfHeapFreezePage (offsetof(xl_heap_freeze_page, isCatalogRel) + sizeof(bool))
-
/*
* This is what we need to know about setting a visibility map bit
*
--
2.40.1
[text/x-diff] v4-0014-Vacuum-second-pass-emits-XLOG_HEAP2_PRUNE-record.patch (14.9K, ../../20240320013602.6sypr4cx6sefpemg@liskov/15-v4-0014-Vacuum-second-pass-emits-XLOG_HEAP2_PRUNE-record.patch)
download | inline diff:
From 94f7f2fe6b6b7fdcbe4d60a157d87e607ef70dc0 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Tue, 19 Mar 2024 18:50:33 -0400
Subject: [PATCH v4 14/19] Vacuum second pass emits XLOG_HEAP2_PRUNE record
Remove the XLOG_HEAP2_VACUUM record and update vacuum's second pass to
emit a XLOG_HEAP2_PRUNE record. This temporarily wastes some space but a
future commit will streamline xl_heap_prune and ensure that no unused
members are included in the WAL record.
---
src/backend/access/heap/heapam.c | 94 ++++--------------------
src/backend/access/heap/pruneheap.c | 67 ++++++++++-------
src/backend/access/heap/vacuumlazy.c | 12 ++-
src/backend/access/rmgrdesc/heapdesc.c | 20 -----
src/backend/replication/logical/decode.c | 1 -
src/include/access/heapam.h | 2 +-
src/include/access/heapam_xlog.h | 33 +++++----
7 files changed, 85 insertions(+), 144 deletions(-)
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 532868039d5..16bab55ba02 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -8717,23 +8717,34 @@ heap_xlog_prune(XLogReaderState *record)
BlockNumber blkno;
XLogRedoAction action;
bool get_cleanup_lock;
+ bool lp_truncate_only;
XLogRecGetBlockTag(record, 0, &rlocator, NULL, &blkno);
+ lp_truncate_only = xlrec->flags & XLHP_LP_TRUNCATE_ONLY;
+
/*
* If there are dead, redirected, or unused items set unused by
* heap_page_prune_and_freeze(), heap_page_prune_execute() will call
* PageRepairFragementation() which expects a full cleanup lock.
*/
get_cleanup_lock = xlrec->nredirected > 0 ||
- xlrec->ndead > 0 || xlrec->nunused > 0;
+ xlrec->ndead > 0 ||
+ (xlrec->nunused > 0 && !lp_truncate_only);
+
+ if (lp_truncate_only)
+ {
+ Assert(xlrec->nredirected == 0);
+ Assert(xlrec->ndead == 0);
+ Assert(xlrec->nunused > 0);
+ }
/*
* We are either about to remove tuples or freeze them. In Hot Standby
* mode, ensure that there's no queries running for which any removed
* tuples are still visible or which consider the frozen xids as running.
*/
- if (InHotStandby)
+ if (xlrec->flags & XLHP_HAS_CONFLICT_HORIZON && InHotStandby)
ResolveRecoveryConflictWithSnapshot(xlrec->snapshotConflictHorizon,
xlrec->isCatalogRel,
rlocator);
@@ -8772,7 +8783,7 @@ heap_xlog_prune(XLogReaderState *record)
/* Update all line pointers per the record, and repair fragmentation */
if (nredirected > 0 || ndead > 0 || nunused > 0)
- heap_page_prune_execute(buffer,
+ heap_page_prune_execute(buffer, lp_truncate_only,
redirected, nredirected,
nowdead, ndead,
nowunused, nunused);
@@ -8819,7 +8830,7 @@ heap_xlog_prune(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
/*
- * After pruning records from a page, it's useful to update the FSM
+ * After modifying records on a page, it's useful to update the FSM
* about it, as it may cause the page become target for insertions
* later even if vacuum decides not to visit it (which is possible if
* gets marked all-visible.)
@@ -8831,78 +8842,6 @@ heap_xlog_prune(XLogReaderState *record)
}
}
-/*
- * Handles XLOG_HEAP2_VACUUM record type.
- *
- * Acquires an ordinary exclusive lock only.
- */
-static void
-heap_xlog_vacuum(XLogReaderState *record)
-{
- XLogRecPtr lsn = record->EndRecPtr;
- xl_heap_vacuum *xlrec = (xl_heap_vacuum *) XLogRecGetData(record);
- Buffer buffer;
- BlockNumber blkno;
- XLogRedoAction action;
-
- /*
- * If we have a full-page image, restore it (without using a cleanup lock)
- * and we're done.
- */
- action = XLogReadBufferForRedoExtended(record, 0, RBM_NORMAL, false,
- &buffer);
- if (action == BLK_NEEDS_REDO)
- {
- Page page = (Page) BufferGetPage(buffer);
- OffsetNumber *nowunused;
- Size datalen;
- OffsetNumber *offnum;
-
- nowunused = (OffsetNumber *) XLogRecGetBlockData(record, 0, &datalen);
-
- /* Shouldn't be a record unless there's something to do */
- Assert(xlrec->nunused > 0);
-
- /* Update all now-unused line pointers */
- offnum = nowunused;
- for (int i = 0; i < xlrec->nunused; i++)
- {
- OffsetNumber off = *offnum++;
- ItemId lp = PageGetItemId(page, off);
-
- Assert(ItemIdIsDead(lp) && !ItemIdHasStorage(lp));
- ItemIdSetUnused(lp);
- }
-
- /* Attempt to truncate line pointer array now */
- PageTruncateLinePointerArray(page);
-
- PageSetLSN(page, lsn);
- MarkBufferDirty(buffer);
- }
-
- if (BufferIsValid(buffer))
- {
- Size freespace = PageGetHeapFreeSpace(BufferGetPage(buffer));
- RelFileLocator rlocator;
-
- XLogRecGetBlockTag(record, 0, &rlocator, NULL, &blkno);
-
- UnlockReleaseBuffer(buffer);
-
- /*
- * After vacuuming LP_DEAD items from a page, it's useful to update
- * the FSM about it, as it may cause the page become target for
- * insertions later even if vacuum decides not to visit it (which is
- * possible if gets marked all-visible.)
- *
- * Do this regardless of a full-page image being applied, since the
- * FSM data is not in the page anyway.
- */
- XLogRecordPageWithFreeSpace(rlocator, blkno, freespace);
- }
-}
-
/*
* Replay XLOG_HEAP2_VISIBLE record.
*
@@ -9943,9 +9882,6 @@ heap2_redo(XLogReaderState *record)
case XLOG_HEAP2_PRUNE:
heap_xlog_prune(record);
break;
- case XLOG_HEAP2_VACUUM:
- heap_xlog_vacuum(record);
- break;
case XLOG_HEAP2_VISIBLE:
heap_xlog_visible(record);
break;
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 19b50931b90..135fe2dba3e 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -601,7 +601,7 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
*/
if (do_prune)
{
- heap_page_prune_execute(buffer,
+ heap_page_prune_execute(buffer, false,
prstate.redirected, prstate.nredirected,
prstate.nowdead, prstate.ndead,
prstate.nowunused, prstate.nunused);
@@ -668,12 +668,16 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
OffsetNumber offsets[MaxHeapTuplesPerPage];
bool do_freeze = presult->nfrozen > 0;
+ xlrec.flags = 0;
+
xlrec.isCatalogRel = RelationIsAccessibleInLogicalDecoding(relation);
xlrec.nredirected = prstate->nredirected;
xlrec.ndead = prstate->ndead;
xlrec.nunused = prstate->nunused;
xlrec.nplans = 0;
+ xlrec.flags |= XLHP_HAS_CONFLICT_HORIZON;
+
/*
* The snapshotConflictHorizon for the whole record should be the most
* conservative of all the horizons calculated for any of the possible
@@ -1149,7 +1153,7 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum)
* cleanup lock on the buffer.
*/
void
-heap_page_prune_execute(Buffer buffer,
+heap_page_prune_execute(Buffer buffer, bool lp_truncate_only,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
OffsetNumber *nowunused, int nunused)
@@ -1171,6 +1175,7 @@ heap_page_prune_execute(Buffer buffer,
ItemId tolp PG_USED_FOR_ASSERTS_ONLY;
#ifdef USE_ASSERT_CHECKING
+ Assert(!lp_truncate_only);
/*
* Any existing item that we set as an LP_REDIRECT (any 'from' item)
@@ -1226,6 +1231,7 @@ heap_page_prune_execute(Buffer buffer,
ItemId lp = PageGetItemId(page, off);
#ifdef USE_ASSERT_CHECKING
+ Assert(!lp_truncate_only);
/*
* An LP_DEAD line pointer must be left behind when the original item
@@ -1259,23 +1265,29 @@ heap_page_prune_execute(Buffer buffer,
#ifdef USE_ASSERT_CHECKING
- /*
- * When heap_page_prune_and_freeze() was called, mark_unused_now may
- * have been passed as true, which allows would-be LP_DEAD items to be
- * made LP_UNUSED instead. This is only possible if the relation has
- * no indexes. If there are any dead items, then mark_unused_now was
- * not true and every item being marked LP_UNUSED must refer to a
- * heap-only tuple.
- */
- if (ndead > 0)
+ if (lp_truncate_only)
{
- Assert(ItemIdHasStorage(lp) && ItemIdIsNormal(lp));
- htup = (HeapTupleHeader) PageGetItem(page, lp);
- Assert(HeapTupleHeaderIsHeapOnly(htup));
+ /* Setting LP_DEAD to LP_UNUSED in vacuum's second pass */
+ Assert(ItemIdIsDead(lp) && !ItemIdHasStorage(lp));
}
else
{
- Assert(ItemIdIsUsed(lp));
+ /*
+ * When heap_page_prune_and_freeze() was called, mark_unused_now
+ * may have been passed as true, which allows would-be LP_DEAD
+ * items to be made LP_UNUSED instead. This is only possible if
+ * the relation has no indexes. If there are any dead items, then
+ * mark_unused_now was not true and every item being marked
+ * LP_UNUSED must refer to a heap-only tuple.
+ */
+ if (ndead > 0)
+ {
+ Assert(ItemIdHasStorage(lp) && ItemIdIsNormal(lp));
+ htup = (HeapTupleHeader) PageGetItem(page, lp);
+ Assert(HeapTupleHeaderIsHeapOnly(htup));
+ }
+ else
+ Assert(ItemIdIsUsed(lp));
}
#endif
@@ -1283,17 +1295,22 @@ heap_page_prune_execute(Buffer buffer,
ItemIdSetUnused(lp);
}
- /*
- * Finally, repair any fragmentation, and update the page's hint bit about
- * whether it has free pointers.
- */
- PageRepairFragmentation(page);
+ if (lp_truncate_only)
+ PageTruncateLinePointerArray(page);
+ else
+ {
+ /*
+ * Finally, repair any fragmentation, and update the page's hint bit
+ * about whether it has free pointers.
+ */
+ PageRepairFragmentation(page);
- /*
- * Now that the page has been modified, assert that redirect items still
- * point to valid targets.
- */
- page_verify_redirects(page);
+ /*
+ * Now that the page has been modified, assert that redirect items
+ * still point to valid targets.
+ */
+ page_verify_redirects(page);
+ }
}
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index c4553a4159c..9dfb56475cf 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -2394,18 +2394,24 @@ lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
/* XLOG stuff */
if (RelationNeedsWAL(vacrel->rel))
{
- xl_heap_vacuum xlrec;
+ xl_heap_prune xlrec;
XLogRecPtr recptr;
+ xlrec.flags = XLHP_LP_TRUNCATE_ONLY;
+ xlrec.snapshotConflictHorizon = InvalidTransactionId;
+ xlrec.nplans = 0;
+ xlrec.nredirected = 0;
+ xlrec.ndead = 0;
xlrec.nunused = nunused;
+ xlrec.isCatalogRel = RelationIsAccessibleInLogicalDecoding(vacrel->rel);
XLogBeginInsert();
- XLogRegisterData((char *) &xlrec, SizeOfHeapVacuum);
+ XLogRegisterData((char *) &xlrec, SizeOfHeapPrune);
XLogRegisterBuffer(0, buffer, REGBUF_STANDARD);
XLogRegisterBufData(0, (char *) unused, nunused * sizeof(OffsetNumber));
- recptr = XLogInsert(RM_HEAP2_ID, XLOG_HEAP2_VACUUM);
+ recptr = XLogInsert(RM_HEAP2_ID, XLOG_HEAP2_PRUNE);
PageSetLSN(page, recptr);
}
diff --git a/src/backend/access/rmgrdesc/heapdesc.c b/src/backend/access/rmgrdesc/heapdesc.c
index 9f0a0341d40..ea03f902fc4 100644
--- a/src/backend/access/rmgrdesc/heapdesc.c
+++ b/src/backend/access/rmgrdesc/heapdesc.c
@@ -242,23 +242,6 @@ heap2_desc(StringInfo buf, XLogReaderState *record)
}
}
}
- else if (info == XLOG_HEAP2_VACUUM)
- {
- xl_heap_vacuum *xlrec = (xl_heap_vacuum *) rec;
-
- appendStringInfo(buf, "nunused: %u", xlrec->nunused);
-
- if (XLogRecHasBlockData(record, 0))
- {
- OffsetNumber *nowunused;
-
- nowunused = (OffsetNumber *) XLogRecGetBlockData(record, 0, NULL);
-
- appendStringInfoString(buf, ", unused:");
- array_desc(buf, nowunused, sizeof(OffsetNumber), xlrec->nunused,
- &offset_elem_desc, NULL);
- }
- }
else if (info == XLOG_HEAP2_VISIBLE)
{
xl_heap_visible *xlrec = (xl_heap_visible *) rec;
@@ -360,9 +343,6 @@ heap2_identify(uint8 info)
case XLOG_HEAP2_PRUNE:
id = "PRUNE";
break;
- case XLOG_HEAP2_VACUUM:
- id = "VACUUM";
- break;
case XLOG_HEAP2_VISIBLE:
id = "VISIBLE";
break;
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index f77051572fd..38d1bdd825e 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -446,7 +446,6 @@ heap2_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
* interested in.
*/
case XLOG_HEAP2_PRUNE:
- case XLOG_HEAP2_VACUUM:
case XLOG_HEAP2_VISIBLE:
case XLOG_HEAP2_LOCK_UPDATED:
break;
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 321a46185e1..d5cb8f99cac 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -349,7 +349,7 @@ extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer,
HeapPageFreeze *pagefrz,
PruneFreezeResult *presult,
OffsetNumber *off_loc);
-extern void heap_page_prune_execute(Buffer buffer,
+extern void heap_page_prune_execute(Buffer buffer, bool lp_truncate_only,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
OffsetNumber *nowunused, int nunused);
diff --git a/src/include/access/heapam_xlog.h b/src/include/access/heapam_xlog.h
index fe4a8ff0620..2393540cf68 100644
--- a/src/include/access/heapam_xlog.h
+++ b/src/include/access/heapam_xlog.h
@@ -52,11 +52,10 @@
*/
#define XLOG_HEAP2_REWRITE 0x00
#define XLOG_HEAP2_PRUNE 0x10
-#define XLOG_HEAP2_VACUUM 0x20
-#define XLOG_HEAP2_VISIBLE 0x30
-#define XLOG_HEAP2_MULTI_INSERT 0x40
-#define XLOG_HEAP2_LOCK_UPDATED 0x50
-#define XLOG_HEAP2_NEW_CID 0x60
+#define XLOG_HEAP2_VISIBLE 0x20
+#define XLOG_HEAP2_MULTI_INSERT 0x30
+#define XLOG_HEAP2_LOCK_UPDATED 0x40
+#define XLOG_HEAP2_NEW_CID 0x50
/*
* xl_heap_insert/xl_heap_multi_insert flag values, 8 bits are available.
@@ -266,6 +265,7 @@ typedef struct xl_heap_freeze_plan
*/
typedef struct xl_heap_prune
{
+ uint8 flags;
TransactionId snapshotConflictHorizon;
uint16 nplans;
uint16 nredirected;
@@ -288,19 +288,22 @@ typedef struct xl_heap_prune
#define SizeOfHeapPrune (offsetof(xl_heap_prune, isCatalogRel) + sizeof(bool))
+/* Flags for xl_heap_prune */
+
/*
- * The vacuum page record is similar to the prune record, but can only mark
- * already LP_DEAD items LP_UNUSED (during VACUUM's second heap pass)
- *
- * Acquires an ordinary exclusive lock only.
+ * During vacuum's second pass which sets LP_DEAD items LP_UNUSED, we will only
+ * truncate the line pointer array, not call PageRepairFragmentation. We need
+ * this flag to differentiate what kind of lock (exclusive or cleanup) to take
+ * on the buffer and whether to call PageTruncateLinePointerArray() or
+ * PageRepairFragementation().
*/
-typedef struct xl_heap_vacuum
-{
- uint16 nunused;
- /* OFFSET NUMBERS are in the block reference 0 */
-} xl_heap_vacuum;
+#define XLHP_LP_TRUNCATE_ONLY (1 << 1)
-#define SizeOfHeapVacuum (offsetof(xl_heap_vacuum, nunused) + sizeof(uint16))
+/*
+ * Vacuum's first pass and on-access pruning may need to include a snapshot
+ * conflict horizon.
+ */
+#define XLHP_HAS_CONFLICT_HORIZON (1 << 2)
/* flags for infobits_set */
#define XLHL_XMAX_IS_MULTI 0x01
--
2.40.1
[text/x-diff] v4-0015-Set-hastup-in-heap_page_prune.patch (7.7K, ../../20240320013602.6sypr4cx6sefpemg@liskov/16-v4-0015-Set-hastup-in-heap_page_prune.patch)
download | inline diff:
From 77a8d20e1c332444d1bde1a41682186713e51e5f Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Mon, 18 Mar 2024 20:12:18 -0400
Subject: [PATCH v4 15/19] Set hastup in heap_page_prune
lazy_scan_prune() loops through the line pointers and tuple visibility
information for each tuple on a page, setting hastup to true if there
are any LP_REDIRECT line pointers or tuples with storage which will not
be removed. We want to remove this extra loop from lazy_scan_prune(),
and we know about non-removable tuples during heap_page_prune() anyway.
Set hastup when recording LP_REDIRECT line pointers in
heap_prune_chain() and when LP_NORMAL line pointers refer to tuples
whose visibility status is not HEAPTUPLE_DEAD.
---
src/backend/access/heap/pruneheap.c | 64 ++++++++++++++++++----------
src/backend/access/heap/vacuumlazy.c | 24 +----------
src/include/access/heapam.h | 2 +
3 files changed, 45 insertions(+), 45 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 135fe2dba3e..d183912a402 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -71,7 +71,8 @@ static int heap_prune_chain(Buffer buffer,
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
- OffsetNumber offnum, OffsetNumber rdoffnum);
+ OffsetNumber offnum, OffsetNumber rdoffnum,
+ PruneFreezeResult *presult);
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
PruneFreezeResult *presult);
static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
@@ -277,6 +278,8 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
presult->nnewlpdead = 0;
presult->nfrozen = 0;
+ presult->hastup = false;
+
/*
* Caller will update the VM after pruning, collecting LP_DEAD items, and
* freezing tuples. Keep track of whether or not the page is all_visible
@@ -419,30 +422,42 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
break;
}
- /*
- * Consider freezing any normal tuples which will not be removed
- */
- if (presult->htsv[offnum] != HEAPTUPLE_DEAD && pagefrz)
+ if (presult->htsv[offnum] != HEAPTUPLE_DEAD)
{
- bool totally_frozen;
+ /*
+ * Deliberately don't set hastup for LP_DEAD items. We make the
+ * soft assumption that any LP_DEAD items encountered here will
+ * become LP_UNUSED later on, before count_nondeletable_pages is
+ * reached. If we don't make this assumption then rel truncation
+ * will only happen every other VACUUM, at most. Besides, VACUUM
+ * must treat hastup/nonempty_pages as provisional no matter how
+ * LP_DEAD items are handled (handled here, or handled later on).
+ */
+ presult->hastup = true;
- /* Tuple with storage -- consider need to freeze */
- if ((heap_prepare_freeze_tuple(htup, pagefrz,
- &prstate.frozen[presult->nfrozen],
- &totally_frozen)))
+ /* Consider freezing any normal tuples which will not be removed */
+ if (pagefrz)
{
- /* Save prepared freeze plan for later */
- prstate.frozen[presult->nfrozen++].offset = offnum;
- }
+ bool totally_frozen;
- /*
- * If any tuple isn't either totally frozen already or eligible to
- * become totally frozen (according to its freeze plan), then the
- * page definitely cannot be set all-frozen in the visibility map
- * later on
- */
- if (!totally_frozen)
- presult->all_frozen = false;
+ /* Tuple with storage -- consider need to freeze */
+ if ((heap_prepare_freeze_tuple(htup, pagefrz,
+ &prstate.frozen[presult->nfrozen],
+ &totally_frozen)))
+ {
+ /* Save prepared freeze plan for later */
+ prstate.frozen[presult->nfrozen++].offset = offnum;
+ }
+
+ /*
+ * If any tuple isn't either totally frozen already or
+ * eligible to become totally frozen (according to its freeze
+ * plan), then the page definitely cannot be set all-frozen in
+ * the visibility map later on
+ */
+ if (!totally_frozen)
+ presult->all_frozen = false;
+ }
}
}
@@ -1049,7 +1064,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
if (i >= nchain)
heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
else
- heap_prune_record_redirect(prstate, rootoffnum, chainitems[i]);
+ heap_prune_record_redirect(prstate, rootoffnum, chainitems[i], presult);
}
else if (nchain < 2 && ItemIdIsRedirected(rootlp))
{
@@ -1083,7 +1098,8 @@ heap_prune_record_prunable(PruneState *prstate, TransactionId xid)
/* Record line pointer to be redirected */
static void
heap_prune_record_redirect(PruneState *prstate,
- OffsetNumber offnum, OffsetNumber rdoffnum)
+ OffsetNumber offnum, OffsetNumber rdoffnum,
+ PruneFreezeResult *presult)
{
Assert(prstate->nredirected < MaxHeapTuplesPerPage);
prstate->redirected[prstate->nredirected * 2] = offnum;
@@ -1093,6 +1109,8 @@ heap_prune_record_redirect(PruneState *prstate,
prstate->marked[offnum] = true;
Assert(!prstate->marked[rdoffnum]);
prstate->marked[rdoffnum] = true;
+
+ presult->hastup = true;
}
/* Record line pointer to be marked dead */
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 9dfb56475cf..370721a619a 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -1420,7 +1420,6 @@ lazy_scan_prune(LVRelState *vacrel,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
- bool hastup = false;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
@@ -1477,28 +1476,12 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = offnum;
itemid = PageGetItemId(page, offnum);
- if (!ItemIdIsUsed(itemid))
- continue;
-
/* Redirect items mustn't be touched */
- if (ItemIdIsRedirected(itemid))
- {
- /* page makes rel truncation unsafe */
- hastup = true;
+ if (ItemIdIsRedirected(itemid) || !ItemIdIsUsed(itemid))
continue;
- }
if (ItemIdIsDead(itemid))
{
- /*
- * Deliberately don't set hastup for LP_DEAD items. We make the
- * soft assumption that any LP_DEAD items encountered here will
- * become LP_UNUSED later on, before count_nondeletable_pages is
- * reached. If we don't make this assumption then rel truncation
- * will only happen every other VACUUM, at most. Besides, VACUUM
- * must treat hastup/nonempty_pages as provisional no matter how
- * LP_DEAD items are handled (handled here, or handled later on).
- */
deadoffsets[lpdead_items++] = offnum;
continue;
}
@@ -1566,9 +1549,6 @@ lazy_scan_prune(LVRelState *vacrel,
elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result");
break;
}
-
- hastup = true; /* page makes rel truncation unsafe */
-
}
vacrel->offnum = InvalidOffsetNumber;
@@ -1650,7 +1630,7 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->recently_dead_tuples += recently_dead_tuples;
/* Can't truncate this page */
- if (hastup)
+ if (presult.hastup)
vacrel->nonempty_pages = blkno + 1;
/* Did we find LP_DEAD items? */
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index d5cb8f99cac..25dbae8139e 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -202,6 +202,8 @@ typedef struct PruneFreezeResult
int nnewlpdead; /* Number of newly LP_DEAD items */
bool all_visible; /* Whether or not the page is all visible */
bool all_visible_except_removable;
+ bool hastup; /* Does page make rel truncation unsafe */
+
/* Whether or not the page can be set all frozen in the VM */
bool all_frozen;
--
2.40.1
[text/x-diff] v4-0016-Count-tuples-for-vacuum-logging-in-heap_page_prun.patch (15.6K, ../../20240320013602.6sypr4cx6sefpemg@liskov/17-v4-0016-Count-tuples-for-vacuum-logging-in-heap_page_prun.patch)
download | inline diff:
From f9356564f651fa9f91e99adbbac50914601e3551 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Mon, 22 Jan 2024 17:25:56 -0500
Subject: [PATCH v4 16/19] Count tuples for vacuum logging in heap_page_prune
lazy_scan_prune() loops through all of the tuple visibility information
that was recorded in heap_page_prune() and then counts live and recently
dead tuples. That information is available in heap_page_prune(), so just
record it there. Add live and recently dead tuple counters to the
PruneResult. Doing this counting in heap_page_prune() eliminates the
need for saving the tuple visibility status information in the
PruneResult. Instead, save it in the PruneState where it can be
referenced by heap_prune_chain().
---
src/backend/access/heap/pruneheap.c | 99 ++++++++++++++++++++++++----
src/backend/access/heap/vacuumlazy.c | 93 +-------------------------
src/include/access/heapam.h | 29 +-------
3 files changed, 93 insertions(+), 128 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index d183912a402..f59b03222b0 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -55,6 +55,18 @@ typedef struct
*/
bool marked[MaxHeapTuplesPerPage + 1];
+ /*
+ * Tuple visibility is only computed once for each tuple, for correctness
+ * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
+ * details. This is of type int8[], instead of HTSV_Result[], so we can
+ * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
+ * items.
+ *
+ * This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
+ * 1. Otherwise every access would need to subtract 1.
+ */
+ int8 htsv[MaxHeapTuplesPerPage + 1];
+
/*
* One entry for every tuple that we may freeze.
*/
@@ -69,6 +81,7 @@ static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
PruneState *prstate, PruneFreezeResult *presult);
+static inline HTSV_Result htsv_get_valid_status(int status);
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum,
@@ -269,7 +282,7 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
prstate.nredirected = prstate.ndead = prstate.nunused = 0;
/*
- * presult->htsv is not initialized here because all ntuple spots in the
+ * prstate.htsv is not initialized here because all ntuple spots in the
* array will be set either to a valid HTSV_Result value or -1.
*/
memset(prstate.marked, 0, sizeof(prstate.marked));
@@ -280,6 +293,9 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
presult->hastup = false;
+ presult->live_tuples = 0;
+ presult->recently_dead_tuples = 0;
+
/*
* Caller will update the VM after pruning, collecting LP_DEAD items, and
* freezing tuples. Keep track of whether or not the page is all_visible
@@ -329,7 +345,7 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
/* Nothing to do if slot doesn't contain a tuple */
if (!ItemIdIsNormal(itemid))
{
- presult->htsv[offnum] = -1;
+ prstate.htsv[offnum] = -1;
continue;
}
@@ -345,9 +361,29 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
if (off_loc)
*off_loc = offnum;
- presult->htsv[offnum] = heap_prune_satisfies_vacuum(&prstate, &tup,
- buffer);
- switch (presult->htsv[offnum])
+ prstate.htsv[offnum] = heap_prune_satisfies_vacuum(&prstate, &tup,
+ buffer);
+
+ /*
+ * The criteria for counting a tuple as live in this block need to
+ * match what analyze.c's acquire_sample_rows() does, otherwise VACUUM
+ * and ANALYZE may produce wildly different reltuples values, e.g.
+ * when there are many recently-dead tuples.
+ *
+ * The logic here is a bit simpler than acquire_sample_rows(), as
+ * VACUUM can't run inside a transaction block, which makes some cases
+ * impossible (e.g. in-progress insert from the same transaction).
+ *
+ * We treat LP_DEAD items (which are the closest thing to DEAD tuples
+ * that might be seen here) differently, too: we assume that they'll
+ * become LP_UNUSED before VACUUM finishes. This difference is only
+ * superficial. VACUUM effectively agrees with ANALYZE about DEAD
+ * items, in the end. VACUUM won't remember LP_DEAD items, but only
+ * because they're not supposed to be left behind when it is done.
+ * (Cases where we bypass index vacuuming will violate this optimistic
+ * assumption, but the overall impact of that should be negligible.)
+ */
+ switch (prstate.htsv[offnum])
{
case HEAPTUPLE_DEAD:
@@ -367,6 +403,12 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
break;
case HEAPTUPLE_LIVE:
+ /*
+ * Count it as live. Not only is this natural, but it's also
+ * what acquire_sample_rows() does.
+ */
+ presult->live_tuples++;
+
/*
* Is the tuple definitely visible to all transactions?
*
@@ -408,13 +450,34 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
}
break;
case HEAPTUPLE_RECENTLY_DEAD:
+
+ /*
+ * If tuple is recently dead then we must not remove it from
+ * the relation. (We only remove items that are LP_DEAD from
+ * pruning.)
+ */
+ presult->recently_dead_tuples++;
presult->all_visible = false;
break;
case HEAPTUPLE_INSERT_IN_PROGRESS:
+
+ /*
+ * We do not count these rows as live, because we expect the
+ * inserting transaction to update the counters at commit, and
+ * we assume that will happen only after we report our
+ * results. This assumption is a bit shaky, but it is what
+ * acquire_sample_rows() does, so be consistent.
+ */
presult->all_visible = false;
break;
case HEAPTUPLE_DELETE_IN_PROGRESS:
- /* This is an expected case during concurrent vacuum */
+
+ /*
+ * This an expected case during concurrent vacuum. Count such
+ * rows as live. As above, we assume the deleting transaction
+ * will commit and update the counters after we report.
+ */
+ presult->live_tuples++;
presult->all_visible = false;
break;
default:
@@ -422,7 +485,7 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
break;
}
- if (presult->htsv[offnum] != HEAPTUPLE_DEAD)
+ if (prstate.htsv[offnum] != HEAPTUPLE_DEAD)
{
/*
* Deliberately don't set hastup for LP_DEAD items. We make the
@@ -772,10 +835,24 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
}
+/*
+ * Pruning calculates tuple visibility once and saves the results in an array
+ * of int8. See PruneState.htsv for details. This helper function is meant to
+ * guard against examining visibility status array members which have not yet
+ * been computed.
+ */
+static inline HTSV_Result
+htsv_get_valid_status(int status)
+{
+ Assert(status >= HEAPTUPLE_DEAD &&
+ status <= HEAPTUPLE_DELETE_IN_PROGRESS);
+ return (HTSV_Result) status;
+}
+
/*
* Prune specified line pointer or a HOT chain originating at line pointer.
*
- * Tuple visibility information is provided in presult->htsv.
+ * Tuple visibility information is provided in prstate->htsv.
*
* If the item is an index-referenced tuple (i.e. not a heap-only tuple),
* the HOT chain is pruned by removing all DEAD tuples at the start of the HOT
@@ -826,7 +903,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
*/
if (ItemIdIsNormal(rootlp))
{
- Assert(presult->htsv[rootoffnum] != -1);
+ Assert(prstate->htsv[rootoffnum] != -1);
htup = (HeapTupleHeader) PageGetItem(dp, rootlp);
if (HeapTupleHeaderIsHeapOnly(htup))
@@ -849,7 +926,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
* either here or while following a chain below. Whichever path
* gets there first will mark the tuple unused.
*/
- if (presult->htsv[rootoffnum] == HEAPTUPLE_DEAD &&
+ if (prstate->htsv[rootoffnum] == HEAPTUPLE_DEAD &&
!HeapTupleHeaderIsHotUpdated(htup))
{
heap_prune_record_unused(prstate, rootoffnum);
@@ -950,7 +1027,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
*/
tupdead = recent_dead = false;
- switch (htsv_get_valid_status(presult->htsv[offnum]))
+ switch (htsv_get_valid_status(prstate->htsv[offnum]))
{
case HEAPTUPLE_DEAD:
tupdead = true;
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 370721a619a..a3c971cd26d 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -1378,22 +1378,6 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
*
* Caller must hold pin and buffer cleanup lock on the buffer.
*
- * Prior to PostgreSQL 14 there were very rare cases where
- * heap_page_prune_and_freeze() was allowed to disagree with our
- * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be
- * considered DEAD. This happened when an inserting transaction concurrently
- * aborted (after our heap_page_prune_and_freeze() call, before our
- * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just
- * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
- * left with storage after pruning.
- *
- * As of Postgres 17, we circumvent this problem altogether by reusing the
- * result of heap_page_prune_and_freeze()'s visibility check. Without the
- * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
- * there can be no disagreement. We'll just handle such tuples as if they had
- * become fully dead right after this operation completes instead of in the
- * middle of it.
- *
* vmbuffer is the buffer containing the VM block with visibility information
* for the heap block, blkno. all_visible_according_to_vm is the saved
* visibility status of the heap block looked up earlier by the caller. We
@@ -1415,10 +1399,8 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
+ int lpdead_items = 0;
PruneFreezeResult presult;
- int lpdead_items,
- live_tuples,
- recently_dead_tuples;
HeapPageFreeze pagefrz;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
@@ -1438,9 +1420,6 @@ lazy_scan_prune(LVRelState *vacrel,
pagefrz.NoFreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.NoFreezePageRelminMxid = vacrel->NewRelminMxid;
pagefrz.cutoffs = &vacrel->cutoffs;
- lpdead_items = 0;
- live_tuples = 0;
- recently_dead_tuples = 0;
/*
* Prune all HOT-update chains and potentially freeze tuples on this page.
@@ -1476,9 +1455,6 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = offnum;
itemid = PageGetItemId(page, offnum);
- /* Redirect items mustn't be touched */
- if (ItemIdIsRedirected(itemid) || !ItemIdIsUsed(itemid))
- continue;
if (ItemIdIsDead(itemid))
{
@@ -1486,69 +1462,6 @@ lazy_scan_prune(LVRelState *vacrel,
continue;
}
- Assert(ItemIdIsNormal(itemid));
-
- /*
- * The criteria for counting a tuple as live in this block need to
- * match what analyze.c's acquire_sample_rows() does, otherwise VACUUM
- * and ANALYZE may produce wildly different reltuples values, e.g.
- * when there are many recently-dead tuples.
- *
- * The logic here is a bit simpler than acquire_sample_rows(), as
- * VACUUM can't run inside a transaction block, which makes some cases
- * impossible (e.g. in-progress insert from the same transaction).
- *
- * We treat LP_DEAD items (which are the closest thing to DEAD tuples
- * that might be seen here) differently, too: we assume that they'll
- * become LP_UNUSED before VACUUM finishes. This difference is only
- * superficial. VACUUM effectively agrees with ANALYZE about DEAD
- * items, in the end. VACUUM won't remember LP_DEAD items, but only
- * because they're not supposed to be left behind when it is done.
- * (Cases where we bypass index vacuuming will violate this optimistic
- * assumption, but the overall impact of that should be negligible.)
- */
- switch (htsv_get_valid_status(presult.htsv[offnum]))
- {
- case HEAPTUPLE_LIVE:
-
- /*
- * Count it as live. Not only is this natural, but it's also
- * what acquire_sample_rows() does.
- */
- live_tuples++;
- break;
- case HEAPTUPLE_RECENTLY_DEAD:
-
- /*
- * If tuple is recently dead then we must not remove it from
- * the relation. (We only remove items that are LP_DEAD from
- * pruning.)
- */
- recently_dead_tuples++;
- break;
- case HEAPTUPLE_INSERT_IN_PROGRESS:
-
- /*
- * We do not count these rows as live, because we expect the
- * inserting transaction to update the counters at commit, and
- * we assume that will happen only after we report our
- * results. This assumption is a bit shaky, but it is what
- * acquire_sample_rows() does, so be consistent.
- */
- break;
- case HEAPTUPLE_DELETE_IN_PROGRESS:
-
- /*
- * This an expected case during concurrent vacuum. Count such
- * rows as live. As above, we assume the deleting transaction
- * will commit and update the counters after we report.
- */
- live_tuples++;
- break;
- default:
- elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result");
- break;
- }
}
vacrel->offnum = InvalidOffsetNumber;
@@ -1626,8 +1539,8 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->tuples_deleted += presult.ndeleted;
vacrel->tuples_frozen += presult.nfrozen;
vacrel->lpdead_items += lpdead_items;
- vacrel->live_tuples += live_tuples;
- vacrel->recently_dead_tuples += recently_dead_tuples;
+ vacrel->live_tuples += presult.live_tuples;
+ vacrel->recently_dead_tuples += presult.recently_dead_tuples;
/* Can't truncate this page */
if (presult.hastup)
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 25dbae8139e..23ba23b5b01 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -198,6 +198,8 @@ typedef struct HeapPageFreeze
*/
typedef struct PruneFreezeResult
{
+ int live_tuples;
+ int recently_dead_tuples;
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
bool all_visible; /* Whether or not the page is all visible */
@@ -211,19 +213,6 @@ typedef struct PruneFreezeResult
int nfrozen;
TransactionId frz_conflict_horizon; /* Newest xmin on the page */
- /*
- * Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
- * details. This is of type int8[], instead of HTSV_Result[], so we can
- * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
- * items.
- *
- * This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
- * 1. Otherwise every access would need to subtract 1.
- */
- int8 htsv[MaxHeapTuplesPerPage + 1];
-
-
/* New value of relfrozenxid found by heap_page_prune_and_freeze() */
TransactionId new_relfrozenxid;
@@ -231,20 +220,6 @@ typedef struct PruneFreezeResult
MultiXactId new_relminmxid;
} PruneFreezeResult;
-/*
- * Pruning calculates tuple visibility once and saves the results in an array
- * of int8. See PruneFreezeResult.htsv for details. This helper function is
- * meant to guard against examining visibility status array members which have
- * not yet been computed.
- */
-static inline HTSV_Result
-htsv_get_valid_status(int status)
-{
- Assert(status >= HEAPTUPLE_DEAD &&
- status <= HEAPTUPLE_DELETE_IN_PROGRESS);
- return (HTSV_Result) status;
-}
-
/* ----------------
* function prototypes for heap access method
*
--
2.40.1
[text/x-diff] v4-0017-Save-dead-tuple-offsets-during-heap_page_prune.patch (7.0K, ../../20240320013602.6sypr4cx6sefpemg@liskov/18-v4-0017-Save-dead-tuple-offsets-during-heap_page_prune.patch)
download | inline diff:
From a322901bb7fa86a0a7bc64b339d52c1446488915 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Mon, 22 Jan 2024 16:55:28 -0500
Subject: [PATCH v4 17/19] Save dead tuple offsets during heap_page_prune
After heap_page_prune() returned, lazy_scan_prune() looped through all
of the offsets of LP_DEAD items which it later added to
LVRelState->dead_items. Instead take care of this when marking a line
pointer or when an existing non-removable LP_DEAD item is encountered in
heap_prune_chain().
---
src/backend/access/heap/pruneheap.c | 7 ++++
src/backend/access/heap/vacuumlazy.c | 61 ++++++----------------------
src/include/access/heapam.h | 2 +
3 files changed, 22 insertions(+), 48 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index f59b03222b0..5decb1127d0 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -295,6 +295,7 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
presult->live_tuples = 0;
presult->recently_dead_tuples = 0;
+ presult->lpdead_items = 0;
/*
* Caller will update the VM after pruning, collecting LP_DEAD items, and
@@ -1002,7 +1003,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
if (unlikely(prstate->mark_unused_now))
heap_prune_record_unused(prstate, offnum);
else
+ {
presult->all_visible = false;
+ presult->deadoffsets[presult->lpdead_items++] = offnum;
+ }
break;
}
@@ -1206,6 +1210,9 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
* all_visible.
*/
presult->all_visible = false;
+
+ /* Record the dead offset for vacuum */
+ presult->deadoffsets[presult->lpdead_items++] = offnum;
}
/*
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index a3c971cd26d..295846b854f 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -1396,23 +1396,11 @@ lazy_scan_prune(LVRelState *vacrel,
bool *has_lpdead_items)
{
Relation rel = vacrel->rel;
- OffsetNumber offnum,
- maxoff;
- ItemId itemid;
- int lpdead_items = 0;
PruneFreezeResult presult;
HeapPageFreeze pagefrz;
- OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
- /*
- * maxoff might be reduced following line pointer array truncation in
- * heap_page_prune_and_freeze(). That's safe for us to ignore, since the
- * reclaimed space will continue to look like LP_UNUSED items below.
- */
- maxoff = PageGetMaxOffsetNumber(page);
-
/* Initialize pagefrz */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
@@ -1425,9 +1413,9 @@ lazy_scan_prune(LVRelState *vacrel,
* Prune all HOT-update chains and potentially freeze tuples on this page.
*
* We count the number of tuples removed from the page by the pruning step
- * in presult.ndeleted. It should not be confused with lpdead_items;
- * lpdead_items's final value can be thought of as the number of tuples
- * that were deleted from indexes.
+ * in presult.ndeleted. It should not be confused with
+ * presult.lpdead_items; presult.lpdead_items's final value can be thought
+ * of as the number of tuples that were deleted from indexes.
*
* If the relation has no indexes, we can immediately mark would-be dead
* items LP_UNUSED, so mark_unused_now should be true if no indexes and
@@ -1437,33 +1425,10 @@ lazy_scan_prune(LVRelState *vacrel,
&pagefrz, &presult, &vacrel->offnum);
/*
- * Now scan the page to collect LP_DEAD items and check for tuples
- * requiring freezing among remaining tuples with storage. We will update
- * the VM after collecting LP_DEAD items and freezing tuples. Pruning will
- * have determined whether or not the page is all_visible and able to
- * become all_frozen.
- *
+ * We will update the VM after collecting LP_DEAD items and freezing
+ * tuples. Pruning will have determined whether or not the page is
+ * all_visible.
*/
- for (offnum = FirstOffsetNumber;
- offnum <= maxoff;
- offnum = OffsetNumberNext(offnum))
- {
- /*
- * Set the offset number so that we can display it along with any
- * error that occurred while processing this tuple.
- */
- vacrel->offnum = offnum;
- itemid = PageGetItemId(page, offnum);
-
-
- if (ItemIdIsDead(itemid))
- {
- deadoffsets[lpdead_items++] = offnum;
- continue;
- }
-
- }
-
vacrel->offnum = InvalidOffsetNumber;
Assert(MultiXactIdIsValid(presult.new_relminmxid));
@@ -1499,7 +1464,7 @@ lazy_scan_prune(LVRelState *vacrel,
TransactionId debug_cutoff;
bool debug_all_frozen;
- Assert(lpdead_items == 0);
+ Assert(presult.lpdead_items == 0);
if (!heap_page_is_all_visible(vacrel, buf,
&debug_cutoff, &debug_all_frozen))
@@ -1515,7 +1480,7 @@ lazy_scan_prune(LVRelState *vacrel,
/*
* Now save details of the LP_DEAD items from the page in vacrel
*/
- if (lpdead_items > 0)
+ if (presult.lpdead_items > 0)
{
VacDeadItems *dead_items = vacrel->dead_items;
ItemPointerData tmp;
@@ -1524,9 +1489,9 @@ lazy_scan_prune(LVRelState *vacrel,
ItemPointerSetBlockNumber(&tmp, blkno);
- for (int i = 0; i < lpdead_items; i++)
+ for (int i = 0; i < presult.lpdead_items; i++)
{
- ItemPointerSetOffsetNumber(&tmp, deadoffsets[i]);
+ ItemPointerSetOffsetNumber(&tmp, presult.deadoffsets[i]);
dead_items->items[dead_items->num_items++] = tmp;
}
@@ -1538,7 +1503,7 @@ lazy_scan_prune(LVRelState *vacrel,
/* Finally, add page-local counts to whole-VACUUM counts */
vacrel->tuples_deleted += presult.ndeleted;
vacrel->tuples_frozen += presult.nfrozen;
- vacrel->lpdead_items += lpdead_items;
+ vacrel->lpdead_items += presult.lpdead_items;
vacrel->live_tuples += presult.live_tuples;
vacrel->recently_dead_tuples += presult.recently_dead_tuples;
@@ -1547,7 +1512,7 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->nonempty_pages = blkno + 1;
/* Did we find LP_DEAD items? */
- *has_lpdead_items = (lpdead_items > 0);
+ *has_lpdead_items = (presult.lpdead_items > 0);
Assert(!presult.all_visible || !(*has_lpdead_items));
@@ -1615,7 +1580,7 @@ lazy_scan_prune(LVRelState *vacrel,
* There should never be LP_DEAD items on a page with PD_ALL_VISIBLE set,
* however.
*/
- else if (lpdead_items > 0 && PageIsAllVisible(page))
+ else if (presult.lpdead_items > 0 && PageIsAllVisible(page))
{
elog(WARNING, "page containing LP_DEAD items is marked as all-visible in relation \"%s\" page %u",
vacrel->relname, blkno);
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 23ba23b5b01..bdde17eb230 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -218,6 +218,8 @@ typedef struct PruneFreezeResult
/* New value of relminmxid found by heap_page_prune_and_freeze() */
MultiXactId new_relminmxid;
+ int lpdead_items; /* includes existing LP_DEAD items */
+ OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
} PruneFreezeResult;
/* ----------------
--
2.40.1
[text/x-diff] v4-0018-Initialize-xl_heap_prune-deserialization-variable.patch (2.2K, ../../20240320013602.6sypr4cx6sefpemg@liskov/19-v4-0018-Initialize-xl_heap_prune-deserialization-variable.patch)
download | inline diff:
From 89c1686279f59ac3e8536fcd6618291e3a22c199 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Tue, 19 Mar 2024 16:05:42 -0400
Subject: [PATCH v4 18/19] Initialize xl_heap_prune deserialization variables
Future commits will depend on these being initialized
---
src/backend/access/heap/heapam.c | 18 +++++++++---------
src/backend/access/rmgrdesc/heapdesc.c | 16 ++++++++--------
2 files changed, 17 insertions(+), 17 deletions(-)
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 16bab55ba02..2ef7decdd05 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -8758,16 +8758,16 @@ heap_xlog_prune(XLogReaderState *record)
if (action == BLK_NEEDS_REDO)
{
Page page = (Page) BufferGetPage(buffer);
- OffsetNumber *redirected;
- OffsetNumber *nowdead;
- OffsetNumber *nowunused;
- int nredirected;
- int ndead;
- int nunused;
- int nplans;
+ OffsetNumber *redirected = NULL;
+ OffsetNumber *nowdead = NULL;
+ OffsetNumber *nowunused = NULL;
+ int nredirected = 0;
+ int ndead = 0;
+ int nunused = 0;
+ int nplans = 0;
Size datalen;
- xl_heap_freeze_plan *plans;
- OffsetNumber *frz_offsets;
+ xl_heap_freeze_plan *plans = NULL;
+ OffsetNumber *frz_offsets = NULL;
int curoff = 0;
nplans = xlrec->nplans;
diff --git a/src/backend/access/rmgrdesc/heapdesc.c b/src/backend/access/rmgrdesc/heapdesc.c
index ea03f902fc4..1fe5c78031f 100644
--- a/src/backend/access/rmgrdesc/heapdesc.c
+++ b/src/backend/access/rmgrdesc/heapdesc.c
@@ -185,15 +185,15 @@ heap2_desc(StringInfo buf, XLogReaderState *record)
if (XLogRecHasBlockData(record, 0))
{
- OffsetNumber *redirected;
- OffsetNumber *nowdead;
- OffsetNumber *nowunused;
- int nredirected;
- int ndead;
- int nunused;
- int nplans;
Size datalen;
- xl_heap_freeze_plan *plans;
+ OffsetNumber *redirected = NULL;
+ OffsetNumber *nowdead = NULL;
+ OffsetNumber *nowunused = NULL;
+ int nredirected = 0;
+ int nunused = 0;
+ int ndead = 0;
+ int nplans = 0;
+ xl_heap_freeze_plan *plans = NULL;
OffsetNumber *frz_offsets;
nplans = xlrec->nplans;
--
2.40.1
[text/x-diff] v4-0019-Streamline-XLOG_HEAP2_PRUNE-record-and-use-for-fr.patch (20.4K, ../../20240320013602.6sypr4cx6sefpemg@liskov/20-v4-0019-Streamline-XLOG_HEAP2_PRUNE-record-and-use-for-fr.patch)
download | inline diff:
From 74411532903ec17cc9c29e17786fdd35ba1b0eac Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Tue, 19 Mar 2024 20:48:28 -0400
Subject: [PATCH v4 19/19] Streamline XLOG_HEAP2_PRUNE record and use for
freeze and vacuum
xl_heap_prune struct for the XLOG_HEAP2_PRUNE record type had members
for counting the number of freeze plans and number of redirected, dead,
and newly unused line pointers. However, only some of those are used in
many XLOG_HEAP2_PRUNE records.
Put all of those members in the XLOG buffer data and use flags to
indicate which are present. This makes it feasible to use the
XLOG_HEAP2_PRUNE record smaller than it was and smaller than the
previously used XLOG_HEAP2_VACUUM and XLOG_HEAP2_FREEZE records.
The snapshot conflict horizon is not used for vacuum but is for the
other record types. It must go in the main data (not per buffer data) so
that it can be used even if the record contains an FPI.
The new prune record is composed of sub-records for each type of
modification freezing tuples and setting line pointers unused, redirect,
or dead.
---
src/backend/access/heap/heapam.c | 33 +++----
src/backend/access/heap/pruneheap.c | 98 ++++++++++++++-----
src/backend/access/heap/vacuumlazy.c | 24 +++--
src/backend/access/rmgrdesc/heapdesc.c | 27 ++---
src/backend/access/rmgrdesc/xactdesc.c | 68 +++++++++++++
src/include/access/heapam_xlog.h | 130 +++++++++++++++----------
src/tools/pgindent/typedefs.list | 3 +
7 files changed, 273 insertions(+), 110 deletions(-)
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 2ef7decdd05..adc259fdca7 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -8728,15 +8728,15 @@ heap_xlog_prune(XLogReaderState *record)
* heap_page_prune_and_freeze(), heap_page_prune_execute() will call
* PageRepairFragementation() which expects a full cleanup lock.
*/
- get_cleanup_lock = xlrec->nredirected > 0 ||
- xlrec->ndead > 0 ||
- (xlrec->nunused > 0 && !lp_truncate_only);
+ get_cleanup_lock = xlrec->flags & XLHP_HAS_REDIRECTIONS ||
+ xlrec->flags & XLHP_HAS_DEAD_ITEMS ||
+ (xlrec->flags & XLHP_HAS_NOW_UNUSED_ITEMS && !lp_truncate_only);
if (lp_truncate_only)
{
- Assert(xlrec->nredirected == 0);
- Assert(xlrec->ndead == 0);
- Assert(xlrec->nunused > 0);
+ Assert(!(xlrec->flags & XLHP_HAS_REDIRECTIONS));
+ Assert(!(xlrec->flags & XLHP_HAS_DEAD_ITEMS));
+ Assert(xlrec->flags & XLHP_HAS_NOW_UNUSED_ITEMS);
}
/*
@@ -8745,9 +8745,13 @@ heap_xlog_prune(XLogReaderState *record)
* tuples are still visible or which consider the frozen xids as running.
*/
if (xlrec->flags & XLHP_HAS_CONFLICT_HORIZON && InHotStandby)
- ResolveRecoveryConflictWithSnapshot(xlrec->snapshotConflictHorizon,
- xlrec->isCatalogRel,
+ {
+ xlhp_conflict_horizon *horizon = (xlhp_conflict_horizon *) (xlrec + SizeOfHeapPrune);
+
+ ResolveRecoveryConflictWithSnapshot(horizon->xid,
+ xlrec->flags & XLHP_IS_CATALOG_REL,
rlocator);
+ }
/*
* If we have a full-page image, restore it and we're done.
@@ -8770,16 +8774,11 @@ heap_xlog_prune(XLogReaderState *record)
OffsetNumber *frz_offsets = NULL;
int curoff = 0;
- nplans = xlrec->nplans;
- nredirected = xlrec->nredirected;
- ndead = xlrec->ndead;
- nunused = xlrec->nunused;
+ char *cursor = XLogRecGetBlockData(record, 0, &datalen);
- plans = (xl_heap_freeze_plan *) XLogRecGetBlockData(record, 0, &datalen);
- redirected = (OffsetNumber *) &plans[nplans];
- nowdead = redirected + (nredirected * 2);
- nowunused = nowdead + ndead;
- frz_offsets = nowunused + nunused;
+ heap_xlog_deserialize_prune_and_freeze(cursor, xlrec->flags,
+ &nredirected, &redirected, &ndead, &nowdead,
+ &nunused, &nowunused, &nplans, &plans, &frz_offsets);
/* Update all line pointers per the record, and repair fragmentation */
if (nredirected > 0 || ndead > 0 || nunused > 0)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 5decb1127d0..26147f63c4c 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -741,19 +741,22 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
PruneState *prstate, PruneFreezeResult *presult)
{
xl_heap_prune xlrec;
+ xlhp_conflict_horizon horizon;
XLogRecPtr recptr;
+ xlhp_freeze freeze;
+ xlhp_prune_items redirect,
+ dead,
+ unused;
+ int nplans = 0;
xl_heap_freeze_plan plans[MaxHeapTuplesPerPage];
- OffsetNumber offsets[MaxHeapTuplesPerPage];
- bool do_freeze = presult->nfrozen > 0;
+ OffsetNumber frz_offsets[MaxHeapTuplesPerPage];
+ bool do_freeze = (presult->nfrozen > 0);
xlrec.flags = 0;
- xlrec.isCatalogRel = RelationIsAccessibleInLogicalDecoding(relation);
- xlrec.nredirected = prstate->nredirected;
- xlrec.ndead = prstate->ndead;
- xlrec.nunused = prstate->nunused;
- xlrec.nplans = 0;
+ if (RelationIsAccessibleInLogicalDecoding(relation))
+ xlrec.flags |= XLHP_IS_CATALOG_REL;
xlrec.flags |= XLHP_HAS_CONFLICT_HORIZON;
@@ -767,22 +770,25 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
* youngest tuple this record will freeze will conflict.
*/
if (do_freeze)
- xlrec.snapshotConflictHorizon = Max(prstate->snapshotConflictHorizon,
- presult->frz_conflict_horizon);
+ horizon.xid = Max(prstate->snapshotConflictHorizon,
+ presult->frz_conflict_horizon);
else
- xlrec.snapshotConflictHorizon = prstate->snapshotConflictHorizon;
+ horizon.xid = prstate->snapshotConflictHorizon;
/*
* Prepare deduplicated representation for use in WAL record Destructively
* sorts tuples array in-place.
*/
if (do_freeze)
- xlrec.nplans = heap_log_freeze_plan(prstate->frozen,
- presult->nfrozen, plans, offsets);
-
+ nplans = heap_log_freeze_plan(prstate->frozen,
+ presult->nfrozen, plans,
+ frz_offsets);
XLogBeginInsert();
XLogRegisterData((char *) &xlrec, SizeOfHeapPrune);
+ xlrec.flags |= XLHP_HAS_CONFLICT_HORIZON;
+ XLogRegisterData((char *) &horizon, SizeOfSnapshotConflictHorizon);
+
XLogRegisterBuffer(0, buffer, REGBUF_STANDARD);
/*
@@ -790,25 +796,73 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
* that they are. When XLogInsert stores the whole buffer, the offset
* arrays need not be stored too.
*/
- if (xlrec.nplans > 0)
+ if (nplans > 0)
+ {
+ xlrec.flags |= XLHP_HAS_FREEZE_PLANS;
+
+ freeze = (xlhp_freeze)
+ {
+ .nplans = nplans
+ };
+
+ XLogRegisterBufData(0, (char *) &freeze, offsetof(xlhp_freeze, plans));
+
XLogRegisterBufData(0, (char *) plans,
- xlrec.nplans * sizeof(xl_heap_freeze_plan));
+ sizeof(xl_heap_freeze_plan) * freeze.nplans);
+ }
+
if (prstate->nredirected > 0)
+ {
+ xlrec.flags |= XLHP_HAS_REDIRECTIONS;
+
+ redirect = (xlhp_prune_items)
+ {
+ .ntargets = prstate->nredirected
+ };
+
+ XLogRegisterBufData(0, (char *) &redirect,
+ offsetof(xlhp_prune_items, data));
+
XLogRegisterBufData(0, (char *) prstate->redirected,
- prstate->nredirected *
- sizeof(OffsetNumber) * 2);
+ sizeof(OffsetNumber[2]) * prstate->nredirected);
+ }
if (prstate->ndead > 0)
+ {
+ xlrec.flags |= XLHP_HAS_DEAD_ITEMS;
+
+ dead = (xlhp_prune_items)
+ {
+ .ntargets = prstate->ndead
+ };
+
+ XLogRegisterBufData(0, (char *) &dead,
+ offsetof(xlhp_prune_items, data));
+
XLogRegisterBufData(0, (char *) prstate->nowdead,
- prstate->ndead * sizeof(OffsetNumber));
+ sizeof(OffsetNumber) * dead.ntargets);
+ }
if (prstate->nunused > 0)
+ {
+ xlrec.flags |= XLHP_HAS_NOW_UNUSED_ITEMS;
+
+ unused = (xlhp_prune_items)
+ {
+ .ntargets = prstate->nunused
+ };
+
+ XLogRegisterBufData(0, (char *) &unused,
+ offsetof(xlhp_prune_items, data));
+
XLogRegisterBufData(0, (char *) prstate->nowunused,
- prstate->nunused * sizeof(OffsetNumber));
- if (xlrec.nplans > 0)
- XLogRegisterBufData(0, (char *) offsets,
- presult->nfrozen * sizeof(OffsetNumber));
+ sizeof(OffsetNumber) * unused.ntargets);
+ }
+
+ if (nplans > 0)
+ XLogRegisterBufData(0, (char *) frz_offsets,
+ sizeof(OffsetNumber) * presult->nfrozen);
recptr = XLogInsert(RM_HEAP2_ID, XLOG_HEAP2_PRUNE);
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 295846b854f..1e79cbbb107 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -2253,21 +2253,27 @@ lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
if (RelationNeedsWAL(vacrel->rel))
{
xl_heap_prune xlrec;
+ xlhp_prune_items unused_rec;
XLogRecPtr recptr;
- xlrec.flags = XLHP_LP_TRUNCATE_ONLY;
- xlrec.snapshotConflictHorizon = InvalidTransactionId;
- xlrec.nplans = 0;
- xlrec.nredirected = 0;
- xlrec.ndead = 0;
- xlrec.nunused = nunused;
- xlrec.isCatalogRel = RelationIsAccessibleInLogicalDecoding(vacrel->rel);
+ xlrec.flags = XLHP_HAS_NOW_UNUSED_ITEMS;
+
+ xlrec.flags |= XLHP_LP_TRUNCATE_ONLY;
+
+ unused_rec = (xlhp_prune_items)
+ {
+ .ntargets = nunused
+ };
XLogBeginInsert();
XLogRegisterData((char *) &xlrec, SizeOfHeapPrune);
-
XLogRegisterBuffer(0, buffer, REGBUF_STANDARD);
- XLogRegisterBufData(0, (char *) unused, nunused * sizeof(OffsetNumber));
+
+ XLogRegisterBufData(0, (char *) &unused_rec,
+ offsetof(xlhp_prune_items, data));
+
+ XLogRegisterBufData(0, (char *) unused,
+ sizeof(OffsetNumber) * unused_rec.ntargets);
recptr = XLogInsert(RM_HEAP2_ID, XLOG_HEAP2_PRUNE);
diff --git a/src/backend/access/rmgrdesc/heapdesc.c b/src/backend/access/rmgrdesc/heapdesc.c
index 1fe5c78031f..f542bcb94b6 100644
--- a/src/backend/access/rmgrdesc/heapdesc.c
+++ b/src/backend/access/rmgrdesc/heapdesc.c
@@ -179,9 +179,16 @@ heap2_desc(StringInfo buf, XLogReaderState *record)
{
xl_heap_prune *xlrec = (xl_heap_prune *) rec;
- appendStringInfo(buf, "snapshotConflictHorizon: %u, isCatalogRel: %c",
- xlrec->snapshotConflictHorizon,
- xlrec->isCatalogRel ? 'T' : 'F');
+ if (xlrec->flags & XLHP_HAS_CONFLICT_HORIZON)
+ {
+ xlhp_conflict_horizon *horizon = (xlhp_conflict_horizon *) (xlrec + SizeOfHeapPrune);
+
+ appendStringInfo(buf, "snapshotConflictHorizon: %u",
+ horizon->xid);
+ }
+
+ appendStringInfo(buf, ", isCatalogRel: %c",
+ xlrec->flags & XLHP_IS_CATALOG_REL ? 'T' : 'F');
if (XLogRecHasBlockData(record, 0))
{
@@ -196,16 +203,12 @@ heap2_desc(StringInfo buf, XLogReaderState *record)
xl_heap_freeze_plan *plans = NULL;
OffsetNumber *frz_offsets;
- nplans = xlrec->nplans;
- nredirected = xlrec->nredirected;
- ndead = xlrec->ndead;
- nunused = xlrec->nunused;
+ char *cursor = XLogRecGetBlockData(record, 0, &datalen);
+
+ heap_xlog_deserialize_prune_and_freeze(cursor, xlrec->flags,
+ &nredirected, &redirected, &ndead, &nowdead,
+ &nunused, &nowunused, &nplans, &plans, &frz_offsets);
- plans = (xl_heap_freeze_plan *) XLogRecGetBlockData(record, 0, &datalen);
- redirected = (OffsetNumber *) &plans[nplans];
- nowdead = redirected + (nredirected * 2);
- nowunused = nowdead + ndead;
- frz_offsets = nowunused + nunused;
appendStringInfo(buf, ", nredirected: %u, ndead: %u, nunused: %u, nplans: %u,",
nredirected,
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index 41b842d80ec..e120805e5e0 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -14,6 +14,7 @@
*/
#include "postgres.h"
+#include "access/heapam_xlog.h"
#include "access/transam.h"
#include "access/xact.h"
#include "replication/origin.h"
@@ -21,6 +22,73 @@
#include "storage/standbydefs.h"
#include "utils/timestamp.h"
+/*
+ * Given a MAXALIGNed buffer returned by XLogRecGetBlockData() and pointed to
+ * by cursor and any xl_heap_prune flags, deserialize the arrays of
+ * OffsetNumbers contained in an xl_heap_prune record. This is in this file so
+ * it can be shared between heap2_redo and heap2_desc code, the latter of which
+ * is used in frontend code.
+ */
+void
+heap_xlog_deserialize_prune_and_freeze(char *cursor, uint8 flags,
+ int *nredirected, OffsetNumber **redirected,
+ int *ndead, OffsetNumber **nowdead,
+ int *nunused, OffsetNumber **nowunused,
+ int *nplans, xl_heap_freeze_plan **plans,
+ OffsetNumber **frz_offsets)
+{
+ if (flags & XLHP_HAS_FREEZE_PLANS)
+ {
+ xlhp_freeze *freeze = (xlhp_freeze *) cursor;
+
+ *nplans = freeze->nplans;
+ Assert(*nplans > 0);
+ *plans = freeze->plans;
+
+ cursor += offsetof(xlhp_freeze, plans);
+ cursor += sizeof(xl_heap_freeze_plan) * freeze->nplans;
+ }
+
+ if (flags & XLHP_HAS_REDIRECTIONS)
+ {
+ xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
+
+ *nredirected = subrecord->ntargets;
+ Assert(nredirected > 0);
+ *redirected = &subrecord->data[0];
+
+ cursor += offsetof(xlhp_prune_items, data);
+ cursor += sizeof(OffsetNumber[2]) * *nredirected;
+ }
+
+ if (flags & XLHP_HAS_DEAD_ITEMS)
+ {
+ xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
+
+ *ndead = subrecord->ntargets;
+ Assert(ndead > 0);
+ *nowdead = subrecord->data;
+
+ cursor += offsetof(xlhp_prune_items, data);
+ cursor += sizeof(OffsetNumber) * *ndead;
+ }
+
+ if (flags & XLHP_HAS_NOW_UNUSED_ITEMS)
+ {
+ xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
+
+ *nunused = subrecord->ntargets;
+ Assert(nunused > 0);
+ *nowunused = subrecord->data;
+
+ cursor += offsetof(xlhp_prune_items, data);
+ cursor += sizeof(OffsetNumber) * *nunused;
+ }
+
+ if (nplans > 0)
+ *frz_offsets = (OffsetNumber *) cursor;
+}
+
/*
* Parse the WAL format of an xact commit and abort records into an easier to
* understand format.
diff --git a/src/include/access/heapam_xlog.h b/src/include/access/heapam_xlog.h
index 2393540cf68..dfeb703d136 100644
--- a/src/include/access/heapam_xlog.h
+++ b/src/include/access/heapam_xlog.h
@@ -224,9 +224,64 @@ typedef struct xl_heap_update
#define SizeOfHeapUpdate (offsetof(xl_heap_update, new_offnum) + sizeof(OffsetNumber))
+/*
+ * This is what we need to know about page pruning and freezing, both during
+ * VACUUM and during opportunistic pruning.
+ *
+ * If XLPH_HAS_REDIRECTIONS, XLHP_HAS_DEAD_ITEMS, or XLHP_HAS_NOW_UNUSED is set,
+ * acquires a full cleanup lock. Otherwise an ordinary exclusive lock is
+ * enough. This can happen if freezing was the only modification to the page.
+ *
+ * The data for block reference 0 contains "sub-records" depending on which
+ * of the XLHP_HAS_* flags are set. See xlhp_* struct definitions below.
+ * The layout is in the same order as the XLHP_* flags.
+ *
+ * OFFSET NUMBERS are in the block reference 0
+ *
+ * If only unused item offsets are included because the record is constructed
+ * during vacuum's second pass (marking LP_DEAD items LP_UNUSED) then only an
+ * ordinary exclusive lock is required to replay.
+ */
+typedef struct xl_heap_prune
+{
+ uint8 flags;
+} xl_heap_prune;
+
+/* to handle recovery conflict during logical decoding on standby */
+#define XLHP_IS_CATALOG_REL (1 << 1)
+
+/*
+ * During vacuum's second pass which sets LP_DEAD items LP_UNUSED, we will only
+ * truncate the line pointer array, not call PageRepairFragmentation. We need
+ * this flag to differentiate what kind of lock (exclusive or cleanup) to take
+ * on the buffer and whether to call PageTruncateLinePointerArray() or
+ * PageRepairFragementation().
+ */
+#define XLHP_LP_TRUNCATE_ONLY (1 << 2)
+
+/*
+ * Vacuum's first pass and on-access pruning may need to include a snapshot
+ * conflict horizon.
+ */
+#define XLHP_HAS_CONFLICT_HORIZON (1 << 3)
+#define XLHP_HAS_FREEZE_PLANS (1 << 4)
+#define XLHP_HAS_REDIRECTIONS (1 << 5)
+#define XLHP_HAS_DEAD_ITEMS (1 << 6)
+#define XLHP_HAS_NOW_UNUSED_ITEMS (1 << 7)
+
+#define SizeOfHeapPrune (offsetof(xl_heap_prune, flags) + sizeof(uint8))
+
+typedef struct xlhp_conflict_horizon
+{
+ TransactionId xid;
+} xlhp_conflict_horizon;
+
+#define SizeOfSnapshotConflictHorizon (offsetof(xlhp_conflict_horizon, xid) + sizeof(uint32))
+
/*
* This struct represents a 'freeze plan', which describes how to freeze a
- * group of one or more heap tuples (appears in xl_heap_prune record)
+ * group of one or more heap tuples (appears in xl_heap_prune's xlhp_freeze
+ * record)
*/
/* 0x01 was XLH_FREEZE_XMIN */
#define XLH_FREEZE_XVAC 0x02
@@ -246,64 +301,32 @@ typedef struct xl_heap_freeze_plan
/*
* As of Postgres 17, XLOG_HEAP2_PRUNE records replace
* XLOG_HEAP2_FREEZE_PAGE records.
- */
-
-/*
- * This is what we need to know about page pruning (both during VACUUM and
- * during opportunistic pruning)
*
- * The array of OffsetNumbers following the fixed part of the record contains:
- * * for each freeze plan: the freeze plan
- * * for each redirected item: the item offset, then the offset redirected to
- * * for each now-dead item: the item offset
- * * for each now-unused item: the item offset
- * * for each tuple frozen by the freeze plans: the offset of the item corresponding to that tuple
- * The total number of OffsetNumbers is therefore
- * (2*nredirected) + ndead + nunused + (sum[plan.ntuples for plan in plans])
+ * This is what we need to know about a block being frozen during vacuum
*
- * Acquires a full cleanup lock if heap_page_prune_execute() must be called
+ * Backup block 0's data contains an array of xl_heap_freeze_plan structs
+ * (with nplans elements), followed by one or more page offset number arrays.
+ * Each such page offset number array corresponds to a single freeze plan
+ * (REDO routine freezes corresponding heap tuples using freeze plan).
*/
-typedef struct xl_heap_prune
+typedef struct xlhp_freeze
{
- uint8 flags;
- TransactionId snapshotConflictHorizon;
uint16 nplans;
- uint16 nredirected;
- uint16 ndead;
- uint16 nunused;
- bool isCatalogRel; /* to handle recovery conflict during logical
- * decoding on standby */
- /*--------------------------------------------------------------------
- * OFFSET NUMBERS and freeze plans are in the block reference 0 in the
- * following order:
- *
- * * xl_heap_freeze_plan plans[nplans];
- * * OffsetNumber redirected[2 * nredirected];
- * * OffsetNumber nowdead[ndead];
- * * OffsetNumber nowunused[nunused];
- * * OffsetNumber frz_offsets[...];
- *--------------------------------------------------------------------
- */
-} xl_heap_prune;
-
-#define SizeOfHeapPrune (offsetof(xl_heap_prune, isCatalogRel) + sizeof(bool))
-
-/* Flags for xl_heap_prune */
+ xl_heap_freeze_plan plans[FLEXIBLE_ARRAY_MEMBER];
+} xlhp_freeze;
/*
- * During vacuum's second pass which sets LP_DEAD items LP_UNUSED, we will only
- * truncate the line pointer array, not call PageRepairFragmentation. We need
- * this flag to differentiate what kind of lock (exclusive or cleanup) to take
- * on the buffer and whether to call PageTruncateLinePointerArray() or
- * PageRepairFragementation().
+ * Sub-record type contained in block reference 0 of a prune record if
+ * XLHP_HAS_REDIRECTIONS/XLHP_HAS_DEAD_ITEMS/XLHP_HAS_NOW_UNUSED_ITEMS is set.
+ * Note that in the XLHP_HAS_REDIRECTIONS variant, there are actually 2 *
+ * length number of OffsetNumbers in the data.
*/
-#define XLHP_LP_TRUNCATE_ONLY (1 << 1)
+typedef struct xlhp_prune_items
+{
+ uint16 ntargets;
+ OffsetNumber data[FLEXIBLE_ARRAY_MEMBER];
+} xlhp_prune_items;
-/*
- * Vacuum's first pass and on-access pruning may need to include a snapshot
- * conflict horizon.
- */
-#define XLHP_HAS_CONFLICT_HORIZON (1 << 2)
/* flags for infobits_set */
#define XLHL_XMAX_IS_MULTI 0x01
@@ -416,4 +439,11 @@ extern XLogRecPtr log_heap_visible(Relation rel, Buffer heap_buffer,
TransactionId snapshotConflictHorizon,
uint8 vmflags);
+extern void heap_xlog_deserialize_prune_and_freeze(char *cursor, uint8 flags,
+ int *nredirected, OffsetNumber **redirected,
+ int *ndead, OffsetNumber **nowdead,
+ int *nunused, OffsetNumber **nowunused,
+ int *nplans, xl_heap_freeze_plan **plans,
+ OffsetNumber **frz_offsets);
+
#endif /* HEAPAM_XLOG_H */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index b2ddc1e2549..40fb694c836 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -4007,6 +4007,9 @@ xl_xact_stats_items
xl_xact_subxacts
xl_xact_twophase
xl_xact_xinfo
+xlhp_conflict_horizon
+xlhp_freeze
+xlhp_prune_items
xmlBuffer
xmlBufferPtr
xmlChar
--
2.40.1
^ permalink raw reply [nested|flat] 55+ messages in thread
end of thread, other threads:[~2024-03-20 01:36 UTC | newest]
Thread overview: 55+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2017-05-24 03:25 Shortened URLs for commit messages Bruce Momjian <[email protected]>
2017-06-15 17:05 ` Bruce Momjian <[email protected]>
2017-06-16 13:51 ` Bruce Momjian <[email protected]>
2017-06-21 15:11 ` Alvaro Herrera <[email protected]>
2017-06-21 16:27 ` Bruce Momjian <[email protected]>
2018-11-01 03:58 move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-01 04:02 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-01 04:03 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-05 07:21 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 06:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-07 07:37 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-07 15:04 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
2018-11-08 02:48 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-08 03:59 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-08 06:11 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-12 13:55 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-13 01:34 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-13 02:34 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-13 03:40 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-13 04:04 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-13 12:58 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
2018-11-13 13:35 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-13 13:59 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
2018-11-14 01:04 ` Re: move PartitionBoundInfo creation code Michael Paquier <[email protected]>
2018-11-14 01:09 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-08 04:31 ` Re: move PartitionBoundInfo creation code Amit Langote <[email protected]>
2018-11-08 13:30 ` Re: move PartitionBoundInfo creation code Alvaro Herrera <[email protected]>
2019-02-01 10:47 [PATCH v33 4/5] Subscripting documentation erthalion <[email protected]>
2019-02-01 10:47 [PATCH v33 4/5] Subscripting documentation erthalion <[email protected]>
2019-02-01 10:47 [PATCH v33 4/5] Subscripting documentation erthalion <[email protected]>
2019-02-01 10:47 [PATCH v33 4/5] Subscripting documentation erthalion <[email protected]>
2019-02-01 10:47 [PATCH v33 4/5] Subscripting documentation erthalion <[email protected]>
2019-02-01 10:47 [PATCH v33 4/5] Subscripting documentation erthalion <[email protected]>
2019-02-01 10:47 [PATCH v33 4/5] Subscripting documentation erthalion <[email protected]>
2019-02-01 10:47 [PATCH v33 4/5] Subscripting documentation erthalion <[email protected]>
2019-02-01 10:47 [PATCH v31 4/6] Subscripting documentation erthalion <[email protected]>
2019-02-01 10:47 [PATCH v33 4/5] Subscripting documentation erthalion <[email protected]>
2019-02-01 10:47 [PATCH v34 4/5] Subscripting documentation erthalion <[email protected]>
2019-02-01 10:47 [PATCH v33 4/5] Subscripting documentation erthalion <[email protected]>
2019-02-01 10:47 [PATCH v33 4/5] Subscripting documentation erthalion <[email protected]>
2019-02-01 10:47 [PATCH v33 4/5] Subscripting documentation erthalion <[email protected]>
2019-02-01 10:47 [PATCH v29 4/6] Subscripting documentation erthalion <[email protected]>
2019-02-01 10:47 [PATCH v33 4/5] Subscripting documentation erthalion <[email protected]>
2019-02-01 10:47 [PATCH v33 4/5] Subscripting documentation erthalion <[email protected]>
2019-02-01 10:47 [PATCH v35 4/5] Subscripting documentation erthalion <[email protected]>
2019-02-01 10:47 [PATCH v30 4/6] Subscripting documentation erthalion <[email protected]>
2019-02-01 10:47 [PATCH v33 4/5] Subscripting documentation erthalion <[email protected]>
2019-02-01 10:47 [PATCH v33 4/5] Subscripting documentation erthalion <[email protected]>
2019-02-01 10:47 [PATCH v32 4/6] Subscripting documentation erthalion <[email protected]>
2019-02-01 10:47 [PATCH v33 4/5] Subscripting documentation erthalion <[email protected]>
2019-02-01 10:47 [PATCH v33 4/5] Subscripting documentation erthalion <[email protected]>
2019-02-01 10:47 [PATCH v33 4/5] Subscripting documentation erthalion <[email protected]>
2019-02-01 10:47 [PATCH v32 4/6] Subscripting documentation erthalion <[email protected]>
2019-02-01 10:47 [PATCH v33 4/5] Subscripting documentation erthalion <[email protected]>
2024-03-20 01:36 Re: Combine Prune and Freeze records emitted by vacuum Melanie Plageman <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox