Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1WiWyT-00035U-BL for pgsql-hackers@arkaria.postgresql.org; Thu, 08 May 2014 22:39:33 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.80) (envelope-from ) id 1WiWyS-0004gQ-G9 for pgsql-hackers@arkaria.postgresql.org; Thu, 08 May 2014 22:39:32 +0000 Received: from makus.postgresql.org ([2001:4800:7903:4::125]) by malur.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1WiWyQ-0004gH-8Z for pgsql-hackers@postgresql.org; Thu, 08 May 2014 22:39:30 +0000 Received: from sss.pgh.pa.us ([66.207.139.130]) by makus.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1WiWyL-00043c-7T for pgsql-hackers@postgresql.org; Thu, 08 May 2014 22:39:29 +0000 Received: from sss1.sss.pgh.pa.us (localhost [127.0.0.1]) by sss.pgh.pa.us (8.14.4/8.14.4) with ESMTP id s48MdB40001889; Thu, 8 May 2014 18:39:11 -0400 From: Tom Lane To: Bruce Momjian cc: "David E. Wheeler" , Greg Stark , Robert Haas , Heikki Linnakangas , Andrew Dunstan , Peter Geoghegan , "pgsql-hackers@postgresql.org" Subject: Re: default opclass for jsonb (was Re: Call for GIST/GIN/SP-GIST opclass documentation) In-reply-to: <5819.1399558614@sss.pgh.pa.us> References: <30137.1397057056@sss.pgh.pa.us> <20140422223230.GL10046@momjian.us> <16527.1398214220@sss.pgh.pa.us> <20140506201048.GI30817@momjian.us> <16769.1399407530@sss.pgh.pa.us> <20140506212020.GK30817@momjian.us> <57E8AA44-F816-45F2-BB61-5A854FFB0A97@justatheory.com> <28554.1399414853@sss.pgh.pa.us> <20140508134701.GO30817@momjian.us> <5819.1399558614@sss.pgh.pa.us> Comments: In-reply-to Tom Lane message dated "Thu, 08 May 2014 10:16:54 -0400" MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----- =_aaaaaaaaaa0" Content-ID: <1835.1399588661.0@sss.pgh.pa.us> Date: Thu, 08 May 2014 18:39:11 -0400 Message-ID: <1888.1399588751@sss.pgh.pa.us> X-Pg-Spam-Score: -2.6 (--) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-hackers Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org ------- =_aaaaaaaaaa0 Content-Type: text/plain; charset="us-ascii" Content-ID: <1835.1399588661.1@sss.pgh.pa.us> I wrote: > I think the idea of hashing only keys/values that are "too long" is a > reasonable compromise. I've not finished coding it (because I keep > getting distracted by other problems in the code :-() but it does not > look to be very difficult. I'm envisioning the cutoff as being something > like 128 bytes; in practice that would mean that few if any keys get > hashed, I think. Attached is a draft patch for this. In addition to the hash logic per se, I made these changes: * Replaced the K/V prefix bytes with a code that distinguishes the types of JSON values. While this is not of any huge significance for the current index search operators, it's basically free to store the info, so I think we should do it for possible future use. * Fixed the problem with "exists" returning rows it shouldn't. I concluded that the best fix is just to force recheck for exists, which allows considerable simplification in the consistent functions. * Tried to improve the comments in jsonb_gin.c. Barring objections I'll commit this tomorrow, and also try to improve the user-facing documentation about the jsonb opclasses. regards, tom lane ------- =_aaaaaaaaaa0 Content-Type: text/x-diff; name="jsonb-gin-fixes-1.patch"; charset="us-ascii" Content-ID: <1835.1399588661.2@sss.pgh.pa.us> Content-Description: jsonb-gin-fixes-1.patch Content-Transfer-Encoding: quoted-printable diff --git a/src/backend/utils/adt/jsonb_gin.c b/src/backend/utils/adt/json= b_gin.c index 592036a..2c4ade2 100644 *** a/src/backend/utils/adt/jsonb_gin.c --- b/src/backend/utils/adt/jsonb_gin.c *************** *** 14,19 **** --- 14,20 ---- #include "postgres.h" =20=20 #include "access/gin.h" + #include "access/hash.h" #include "access/skey.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" *************** typedef struct PathHashStack *** 26,39 **** struct PathHashStack *parent; } PathHashStack; =20=20 ! static text *make_text_key(const char *str, int len, char flag); ! static text *make_scalar_key(const JsonbValue *scalarVal, char flag); =20=20 /* * * jsonb_ops GIN opclass support functions * */ Datum gin_compare_jsonb(PG_FUNCTION_ARGS) { --- 27,41 ---- struct PathHashStack *parent; } PathHashStack; =20=20 ! static Datum make_text_key(char flag, const char *str, int len); ! static Datum make_scalar_key(const JsonbValue *scalarVal, bool is_key); =20=20 /* * * jsonb_ops GIN opclass support functions * */ +=20 Datum gin_compare_jsonb(PG_FUNCTION_ARGS) { *************** gin_extract_jsonb(PG_FUNCTION_ARGS) *** 65,144 **** { Jsonb *jb =3D (Jsonb *) PG_GETARG_JSONB(0); int32 *nentries =3D (int32 *) PG_GETARG_POINTER(1); - Datum *entries =3D NULL; int total =3D 2 * JB_ROOT_COUNT(jb); - int i =3D 0, - r; JsonbIterator *it; JsonbValue v; =20=20 if (total =3D=3D 0) { *nentries =3D 0; PG_RETURN_POINTER(NULL); } =20=20 entries =3D (Datum *) palloc(sizeof(Datum) * total); =20=20 it =3D JsonbIteratorInit(&jb->root); =20=20 while ((r =3D JsonbIteratorNext(&it, &v, false)) !=3D WJB_DONE) { if (i >=3D total) { total *=3D 2; entries =3D (Datum *) repalloc(entries, sizeof(Datum) * total); } =20=20 - /* - * Serialize keys and elements equivalently, but only when elements - * are Jsonb strings. Otherwise, serialize elements as values. Array - * elements are indexed as keys, for the benefit of - * JsonbExistsStrategyNumber. Our definition of existence does not - * allow for checking the existence of a non-jbvString element (just - * like the definition of the underlying operator), because the - * operator takes a text rhs argument (which is taken as a proxy for - * an equivalent Jsonb string). - * - * The way existence is represented does not preclude an alternative - * existence operator, that takes as its rhs value an arbitrarily - * internally-typed Jsonb. The only reason that isn't the case here - * is that the existence operator is only really intended to determine - * if an object has a certain key (object pair keys are of course - * invariably strings), which is extended to jsonb arrays. You could - * think of the default Jsonb definition of existence as being - * equivalent to a definition where all types of scalar array elements - * are keys that we can check the existence of, while just forbidding - * non-string notation. This inflexibility prevents the user from - * having to qualify that the rhs string is a raw scalar string (that - * is, naturally no internal string quoting in required for the text - * argument), and allows us to not set the reset flag for - * JsonbExistsStrategyNumber, since we know that keys are strings for - * both objects and arrays, and don't have to further account for type - * mismatch. Not having to set the reset flag makes it less than - * tempting to tighten up the definition of existence to preclude - * array elements entirely, which would arguably be a simpler - * alternative. In any case the infrastructure used to implement the - * existence operator could trivially support this hypothetical, - * slightly distinct definition of existence. - */ switch (r) { case WJB_KEY: ! /* Serialize key separately, for existence strategies */ ! entries[i++] =3D PointerGetDatum(make_scalar_key(&v, JKEYELEM)); break; case WJB_ELEM: ! if (v.type =3D=3D jbvString) ! entries[i++] =3D PointerGetDatum(make_scalar_key(&v, JKEYELEM)); ! else ! entries[i++] =3D PointerGetDatum(make_scalar_key(&v, JVAL)); break; case WJB_VALUE: ! entries[i++] =3D PointerGetDatum(make_scalar_key(&v, JVAL)); break; default: ! continue; } } =20=20 --- 67,115 ---- { Jsonb *jb =3D (Jsonb *) PG_GETARG_JSONB(0); int32 *nentries =3D (int32 *) PG_GETARG_POINTER(1); int total =3D 2 * JB_ROOT_COUNT(jb); JsonbIterator *it; JsonbValue v; + int i =3D 0, + r; + Datum *entries; =20=20 + /* If the root level is empty, we certainly have no keys */ if (total =3D=3D 0) { *nentries =3D 0; PG_RETURN_POINTER(NULL); } =20=20 + /* Otherwise, use 2 * root count as initial estimate of result size */ entries =3D (Datum *) palloc(sizeof(Datum) * total); =20=20 it =3D JsonbIteratorInit(&jb->root); =20=20 while ((r =3D JsonbIteratorNext(&it, &v, false)) !=3D WJB_DONE) { + /* Since we recurse into the object, we might need more space */ if (i >=3D total) { total *=3D 2; entries =3D (Datum *) repalloc(entries, sizeof(Datum) * total); } =20=20 switch (r) { case WJB_KEY: ! entries[i++] =3D make_scalar_key(&v, true); break; case WJB_ELEM: ! /* Pretend string array elements are keys, see jsonb.h */ ! entries[i++] =3D make_scalar_key(&v, (v.type =3D=3D jbvString)); break; case WJB_VALUE: ! entries[i++] =3D make_scalar_key(&v, false); break; default: ! /* we can ignore structural items */ ! break; } } =20=20 *************** gin_extract_jsonb_query(PG_FUNCTION_ARGS *** 168,192 **** } else if (strategy =3D=3D JsonbExistsStrategyNumber) { text *query =3D PG_GETARG_TEXT_PP(0); - text *item; =20=20 *nentries =3D 1; entries =3D (Datum *) palloc(sizeof(Datum)); ! item =3D make_text_key(VARDATA_ANY(query), VARSIZE_ANY_EXHDR(query), ! JKEYELEM); ! entries[0] =3D PointerGetDatum(item); } else if (strategy =3D=3D JsonbExistsAnyStrategyNumber || strategy =3D=3D JsonbExistsAllStrategyNumber) { ArrayType *query =3D PG_GETARG_ARRAYTYPE_P(0); Datum *key_datums; bool *key_nulls; int key_count; int i, j; - text *item; =20=20 deconstruct_array(query, TEXTOID, -1, false, 'i', --- 139,163 ---- } else if (strategy =3D=3D JsonbExistsStrategyNumber) { + /* Query is a text string, which we treat as a key */ text *query =3D PG_GETARG_TEXT_PP(0); =20=20 *nentries =3D 1; entries =3D (Datum *) palloc(sizeof(Datum)); ! entries[0] =3D make_text_key(JGINFLAG_KEY, ! VARDATA_ANY(query), ! VARSIZE_ANY_EXHDR(query)); } else if (strategy =3D=3D JsonbExistsAnyStrategyNumber || strategy =3D=3D JsonbExistsAllStrategyNumber) { + /* Query is a text array; each element is treated as a key */ ArrayType *query =3D PG_GETARG_ARRAYTYPE_P(0); Datum *key_datums; bool *key_nulls; int key_count; int i, j; =20=20 deconstruct_array(query, TEXTOID, -1, false, 'i', *************** gin_extract_jsonb_query(PG_FUNCTION_ARGS *** 194,208 **** =20=20 entries =3D (Datum *) palloc(sizeof(Datum) * key_count); =20=20 ! for (i =3D 0, j =3D 0; i < key_count; ++i) { /* Nulls in the array are ignored */ if (key_nulls[i]) continue; ! item =3D make_text_key(VARDATA(key_datums[i]), ! VARSIZE(key_datums[i]) - VARHDRSZ, ! JKEYELEM); ! entries[j++] =3D PointerGetDatum(item); } =20=20 *nentries =3D j; --- 165,178 ---- =20=20 entries =3D (Datum *) palloc(sizeof(Datum) * key_count); =20=20 ! for (i =3D 0, j =3D 0; i < key_count; i++) { /* Nulls in the array are ignored */ if (key_nulls[i]) continue; ! entries[j++] =3D make_text_key(JGINFLAG_KEY, ! VARDATA_ANY(key_datums[i]), ! VARSIZE_ANY_EXHDR(key_datums[i])); } =20=20 *nentries =3D j; *************** gin_consistent_jsonb(PG_FUNCTION_ARGS) *** 236,248 **** if (strategy =3D=3D JsonbContainsStrategyNumber) { /* ! * Index doesn't have information about correspondence of Jsonb keys ! * and values (as distinct from GIN keys, which a key/value pair is ! * stored as), so invariably we recheck. Besides, there are some ! * special rules around the containment of raw scalar arrays and ! * regular arrays that are not represented here. However, if all of ! * the keys are not present, that's sufficient reason to return false ! * and finish immediately. */ *recheck =3D true; for (i =3D 0; i < nkeys; i++) --- 206,217 ---- if (strategy =3D=3D JsonbContainsStrategyNumber) { /* ! * We must always recheck, since we can't tell from the index whether ! * the positions of the matched items match the structure of the query ! * object. (Even if we could, we'd also have to worry about hashed ! * keys and the index's failure to distinguish keys from string array ! * elements.) However, the tuple certainly doesn't match unless it ! * contains all the query keys. */ *recheck =3D true; for (i =3D 0; i < nkeys; i++) *************** gin_consistent_jsonb(PG_FUNCTION_ARGS) *** 256,275 **** } else if (strategy =3D=3D JsonbExistsStrategyNumber) { ! /* Existence of key guaranteed in default search mode */ ! *recheck =3D false; res =3D true; } else if (strategy =3D=3D JsonbExistsAnyStrategyNumber) { ! /* Existence of key guaranteed in default search mode */ ! *recheck =3D false; res =3D true; } else if (strategy =3D=3D JsonbExistsAllStrategyNumber) { ! /* Testing for the presence of all keys gives an exact result */ ! *recheck =3D false; for (i =3D 0; i < nkeys; i++) { if (!check[i]) --- 225,251 ---- } else if (strategy =3D=3D JsonbExistsStrategyNumber) { ! /* ! * Although the key is certainly present in the index, we must recheck ! * because (1) the key might be hashed, and (2) the index match might ! * be for a key that's not at top level of the JSON object. For (1), ! * we could look at the query key to see if it's hashed and not ! * recheck if not, but the index lacks enough info to tell about (2). ! */ ! *recheck =3D true; res =3D true; } else if (strategy =3D=3D JsonbExistsAnyStrategyNumber) { ! /* As for plain exists, we must recheck */ ! *recheck =3D true; res =3D true; } else if (strategy =3D=3D JsonbExistsAllStrategyNumber) { ! /* As for plain exists, we must recheck */ ! *recheck =3D true; ! /* ... but unless all the keys are present, we can say "false" */ for (i =3D 0; i < nkeys; i++) { if (!check[i]) *************** gin_triconsistent_jsonb(PG_FUNCTION_ARGS *** 295,313 **** int32 nkeys =3D PG_GETARG_INT32(3); =20=20 /* Pointer *extra_data =3D (Pointer *) PG_GETARG_POINTER(4); */ ! GinTernaryValue res =3D GIN_TRUE; !=20 int32 i; =20=20 ! if (strategy =3D=3D JsonbContainsStrategyNumber) { ! bool has_maybe =3D false; !=20 ! /* ! * All extracted keys must be present. Combination of GIN_MAYBE and ! * GIN_TRUE gives GIN_MAYBE result because then all keys may be ! * present. ! */ for (i =3D 0; i < nkeys; i++) { if (check[i] =3D=3D GIN_FALSE) --- 271,288 ---- int32 nkeys =3D PG_GETARG_INT32(3); =20=20 /* Pointer *extra_data =3D (Pointer *) PG_GETARG_POINTER(4); */ ! GinTernaryValue res =3D GIN_MAYBE; int32 i; =20=20 ! /* ! * Note that we never return GIN_TRUE, only GIN_MAYBE or GIN_FALSE; this ! * corresponds to always forcing recheck in the regular consistent ! * function, for the reasons listed there. ! */ ! if (strategy =3D=3D JsonbContainsStrategyNumber || ! strategy =3D=3D JsonbExistsAllStrategyNumber) { ! /* All extracted keys must be present */ for (i =3D 0; i < nkeys; i++) { if (check[i] =3D=3D GIN_FALSE) *************** gin_triconsistent_jsonb(PG_FUNCTION_ARGS *** 315,369 **** res =3D GIN_FALSE; break; } - if (check[i] =3D=3D GIN_MAYBE) - { - res =3D GIN_MAYBE; - has_maybe =3D true; - } } -=20 - /* - * Index doesn't have information about correspondence of Jsonb keys - * and values (as distinct from GIN keys, which a key/value pair is - * stored as), so invariably we recheck. This is also reflected in - * how GIN_MAYBE is given in response to there being no GIN_MAYBE - * input. - */ - if (!has_maybe && res =3D=3D GIN_TRUE) - res =3D GIN_MAYBE; } else if (strategy =3D=3D JsonbExistsStrategyNumber || strategy =3D=3D JsonbExistsAnyStrategyNumber) { ! /* Existence of key guaranteed in default search mode */ res =3D GIN_FALSE; for (i =3D 0; i < nkeys; i++) { ! if (check[i] =3D=3D GIN_TRUE) ! { ! res =3D GIN_TRUE; ! break; ! } ! if (check[i] =3D=3D GIN_MAYBE) { res =3D GIN_MAYBE; - } - } - } - else if (strategy =3D=3D JsonbExistsAllStrategyNumber) - { - /* Testing for the presence of all keys gives an exact result */ - for (i =3D 0; i < nkeys; i++) - { - if (check[i] =3D=3D GIN_FALSE) - { - res =3D GIN_FALSE; break; } - if (check[i] =3D=3D GIN_MAYBE) - { - res =3D GIN_MAYBE; - } } } else --- 290,310 ---- res =3D GIN_FALSE; break; } } } else if (strategy =3D=3D JsonbExistsStrategyNumber || strategy =3D=3D JsonbExistsAnyStrategyNumber) { ! /* At least one extracted key must be present */ res =3D GIN_FALSE; for (i =3D 0; i < nkeys; i++) { ! if (check[i] =3D=3D GIN_TRUE || ! check[i] =3D=3D GIN_MAYBE) { res =3D GIN_MAYBE; break; } } } else *************** gin_triconsistent_jsonb(PG_FUNCTION_ARGS *** 376,382 **** --- 317,330 ---- * * jsonb_hash_ops GIN opclass support functions * + * In a jsonb_hash_ops index, the keys are uint32 hashes, one per value; = but + * the key(s) leading to each value are also included in its hash computa= tion. + * This means we can only support containment queries, but the index can + * distinguish, for example, {"foo": 42} from {"bar": 42} since different + * hashes will be generated. + * */ +=20 Datum gin_consistent_jsonb_hash(PG_FUNCTION_ARGS) { *************** gin_consistent_jsonb_hash(PG_FUNCTION_AR *** 395,407 **** elog(ERROR, "unrecognized strategy number: %d", strategy); =20=20 /* ! * jsonb_hash_ops index doesn't have information about correspondence of ! * Jsonb keys and values (as distinct from GIN keys, which a key/value ! * pair is stored as), so invariably we recheck. Besides, there are some * special rules around the containment of raw scalar arrays and regular ! * arrays that are not represented here. However, if all of the keys are ! * not present, that's sufficient reason to return false and finish ! * immediately. */ *recheck =3D true; for (i =3D 0; i < nkeys; i++) --- 343,355 ---- elog(ERROR, "unrecognized strategy number: %d", strategy); =20=20 /* ! * jsonb_hash_ops is necessarily lossy, not only because of hash ! * collisions but also because it doesn't preserve complete information ! * about the structure of the JSON object. Besides, there are some * special rules around the containment of raw scalar arrays and regular ! * arrays that are not handled here. So we must always recheck a match. ! * However, if not all of the keys are present, the tuple certainly ! * doesn't match. */ *recheck =3D true; for (i =3D 0; i < nkeys; i++) *************** gin_triconsistent_jsonb_hash(PG_FUNCTION *** 426,442 **** int32 nkeys =3D PG_GETARG_INT32(3); =20=20 /* Pointer *extra_data =3D (Pointer *) PG_GETARG_POINTER(4); */ ! GinTernaryValue res =3D GIN_TRUE; int32 i; - bool has_maybe =3D false; =20=20 if (strategy !=3D JsonbContainsStrategyNumber) elog(ERROR, "unrecognized strategy number: %d", strategy); =20=20 /* ! * All extracted keys must be present. A combination of GIN_MAYBE and ! * GIN_TRUE induces a GIN_MAYBE result, because then all keys may be ! * present. */ for (i =3D 0; i < nkeys; i++) { --- 374,389 ---- int32 nkeys =3D PG_GETARG_INT32(3); =20=20 /* Pointer *extra_data =3D (Pointer *) PG_GETARG_POINTER(4); */ ! GinTernaryValue res =3D GIN_MAYBE; int32 i; =20=20 if (strategy !=3D JsonbContainsStrategyNumber) elog(ERROR, "unrecognized strategy number: %d", strategy); =20=20 /* ! * Note that we never return GIN_TRUE, only GIN_MAYBE or GIN_FALSE; this ! * corresponds to always forcing recheck in the regular consistent ! * function, for the reasons listed there. */ for (i =3D 0; i < nkeys; i++) { *************** gin_triconsistent_jsonb_hash(PG_FUNCTION *** 445,467 **** res =3D GIN_FALSE; break; } - if (check[i] =3D=3D GIN_MAYBE) - { - res =3D GIN_MAYBE; - has_maybe =3D true; - } } =20=20 - /* - * jsonb_hash_ops index doesn't have information about correspondence of - * Jsonb keys and values (as distinct from GIN keys, which for this - * opclass are a hash of a pair, or a hash of just an element), so - * invariably we recheck. This is also reflected in how GIN_MAYBE is - * given in response to there being no GIN_MAYBE input. - */ - if (!has_maybe && res =3D=3D GIN_TRUE) - res =3D GIN_MAYBE; -=20 PG_RETURN_GIN_TERNARY_VALUE(res); } =20=20 --- 392,399 ---- *************** gin_extract_jsonb_hash(PG_FUNCTION_ARGS) *** 477,502 **** PathHashStack *stack; int i =3D 0, r; ! Datum *entries =3D NULL; =20=20 if (total =3D=3D 0) { *nentries =3D 0; PG_RETURN_POINTER(NULL); } =20=20 entries =3D (Datum *) palloc(sizeof(Datum) * total); =20=20 ! it =3D JsonbIteratorInit(&jb->root); !=20 tail.parent =3D NULL; tail.hash =3D 0; stack =3D &tail; =20=20 while ((r =3D JsonbIteratorNext(&it, &v, false)) !=3D WJB_DONE) { ! PathHashStack *tmp; =20=20 if (i >=3D total) { total *=3D 2; --- 409,438 ---- PathHashStack *stack; int i =3D 0, r; ! Datum *entries; =20=20 + /* If the root level is empty, we certainly have no keys */ if (total =3D=3D 0) { *nentries =3D 0; PG_RETURN_POINTER(NULL); } =20=20 + /* Otherwise, use 2 * root count as initial estimate of result size */ entries =3D (Datum *) palloc(sizeof(Datum) * total); =20=20 ! /* We keep a stack of hashes corresponding to parent key levels */ tail.parent =3D NULL; tail.hash =3D 0; stack =3D &tail; =20=20 + it =3D JsonbIteratorInit(&jb->root); +=20 while ((r =3D JsonbIteratorNext(&it, &v, false)) !=3D WJB_DONE) { ! PathHashStack *parent; =20=20 + /* Since we recurse into the object, we might need more space */ if (i >=3D total) { total *=3D 2; *************** gin_extract_jsonb_hash(PG_FUNCTION_ARGS) *** 507,521 **** { case WJB_BEGIN_ARRAY: case WJB_BEGIN_OBJECT: ! tmp =3D stack; stack =3D (PathHashStack *) palloc(sizeof(PathHashStack)); =20=20 ! /* ! * Nesting an array within another array will not alter ! * innermost scalar element hash values, but that seems ! * inconsequential ! */ ! if (tmp->parent) { /* * We pass forward hashes from previous container nesting --- 443,453 ---- { case WJB_BEGIN_ARRAY: case WJB_BEGIN_OBJECT: ! /* Push a stack level for this object */ ! parent =3D stack; stack =3D (PathHashStack *) palloc(sizeof(PathHashStack)); =20=20 ! if (parent->parent) { /* * We pass forward hashes from previous container nesting *************** gin_extract_jsonb_hash(PG_FUNCTION_ARGS) *** 524,561 **** * outermost key. It's also somewhat useful to have * nested objects innermost values have hashes that are a * function of not just their own key, but outer keys too. */ ! stack->hash =3D tmp->hash; } else { /* ! * At least nested level, initialize with stable container ! * type proxy value */ stack->hash =3D (r =3D=3D WJB_BEGIN_ARRAY) ? JB_FARRAY : JB_FOBJECT; } ! stack->parent =3D tmp; break; case WJB_KEY: ! /* Initialize hash from parent */ stack->hash =3D stack->parent->hash; JsonbHashScalarValue(&v, &stack->hash); break; case WJB_ELEM: ! /* Elements have parent hash mixed in separately */ stack->hash =3D stack->parent->hash; case WJB_VALUE: ! /* Element/value case */ JsonbHashScalarValue(&v, &stack->hash); entries[i++] =3D UInt32GetDatum(stack->hash); break; case WJB_END_ARRAY: case WJB_END_OBJECT: /* Pop the stack */ ! tmp =3D stack->parent; pfree(stack); ! stack =3D tmp; break; default: elog(ERROR, "invalid JsonbIteratorNext rc: %d", r); --- 456,504 ---- * outermost key. It's also somewhat useful to have * nested objects innermost values have hashes that are a * function of not just their own key, but outer keys too. + * + * Nesting an array within another array will not alter + * innermost scalar element hash values, but that seems + * inconsequential. */ ! stack->hash =3D parent->hash; } else { /* ! * At the outermost level, initialize hash with container ! * type proxy value. Note that this makes JB_FARRAY and ! * JB_FOBJECT part of the on-disk representation, but they ! * are that in the base jsonb object storage already. */ stack->hash =3D (r =3D=3D WJB_BEGIN_ARRAY) ? JB_FARRAY : JB_FOBJECT; } ! stack->parent =3D parent; break; case WJB_KEY: ! /* initialize hash from parent */ stack->hash =3D stack->parent->hash; + /* and mix in this key */ JsonbHashScalarValue(&v, &stack->hash); + /* hash is now ready to incorporate the value */ break; case WJB_ELEM: ! /* array elements use parent hash mixed with element's hash */ stack->hash =3D stack->parent->hash; + /* FALL THRU */ case WJB_VALUE: ! /* mix the element or value's hash into the prepared hash */ JsonbHashScalarValue(&v, &stack->hash); + /* and emit an index entry */ entries[i++] =3D UInt32GetDatum(stack->hash); + /* Note: we assume we'll see KEY before another VALUE */ break; case WJB_END_ARRAY: case WJB_END_OBJECT: /* Pop the stack */ ! parent =3D stack->parent; pfree(stack); ! stack =3D parent; break; default: elog(ERROR, "invalid JsonbIteratorNext rc: %d", r); *************** gin_extract_jsonb_query_hash(PG_FUNCTION *** 592,605 **** } =20=20 /* ! * Build a text value from a cstring and flag suitable for storage as a k= ey ! * value */ ! static text * ! make_text_key(const char *str, int len, char flag) { text *item; =20=20 item =3D (text *) palloc(VARHDRSZ + len + 1); SET_VARSIZE(item, VARHDRSZ + len + 1); =20=20 --- 535,563 ---- } =20=20 /* ! * Construct a GIN key from a flag byte and a textual representation ! * (which need not be null-terminated). This function is responsible ! * for hashing overlength text representations; it will add the ! * JGINFLAG_HASHED bit to the flag value if it does that. */ ! static Datum ! make_text_key(char flag, const char *str, int len) { text *item; + char hashbuf[10]; +=20 + if (len > JGIN_MAXLENGTH) + { + uint32 hashval; =20=20 + hashval =3D DatumGetUInt32(hash_any((const unsigned char *) str, len)); + snprintf(hashbuf, sizeof(hashbuf), "%08x", hashval); + str =3D hashbuf; + len =3D 8; + flag |=3D JGINFLAG_HASHED; + } +=20 + /* Now build the text Datum */ item =3D (text *) palloc(VARHDRSZ + len + 1); SET_VARSIZE(item, VARHDRSZ + len + 1); =20=20 *************** make_text_key(const char *str, int len,=20 *** 607,637 **** =20=20 memcpy(VARDATA(item) + 1, str, len); =20=20 ! return item; } =20=20 /* ! * Create a textual representation of a jsonbValue for GIN storage. */ ! static text * ! make_scalar_key(const JsonbValue *scalarVal, char flag) { ! text *item; char *cstr; =20=20 switch (scalarVal->type) { case jbvNull: ! item =3D make_text_key("n", 1, flag); break; case jbvBool: ! item =3D make_text_key(scalarVal->val.boolean ? "t" : "f", 1, flag); break; case jbvNumeric: =20=20 /* ! * A normalized textual representation, free of trailing zeroes is ! * is required. * * It isn't ideal that numerics are stored in a relatively bulky * textual format. However, it's a notationally convenient way of --- 565,603 ---- =20=20 memcpy(VARDATA(item) + 1, str, len); =20=20 ! return PointerGetDatum(item); } =20=20 /* ! * Create a textual representation of a JsonbValue that will serve as a G= IN ! * key in a jsonb_ops index. is_key is true if the JsonbValue is a key, ! * or if it is a string array element (since we pretend those are keys, ! * see jsonb.h). */ ! static Datum ! make_scalar_key(const JsonbValue *scalarVal, bool is_key) { ! Datum item; char *cstr; =20=20 switch (scalarVal->type) { case jbvNull: ! Assert(!is_key); ! item =3D make_text_key(JGINFLAG_NULL, "", 0); break; case jbvBool: ! Assert(!is_key); ! item =3D make_text_key(JGINFLAG_BOOL, ! scalarVal->val.boolean ? "t" : "f", 1); break; case jbvNumeric: + Assert(!is_key); =20=20 /* ! * A normalized textual representation, free of trailing zeroes, ! * is required so that numerically equal values will produce equal ! * strings. * * It isn't ideal that numerics are stored in a relatively bulky * textual format. However, it's a notationally convenient way of *************** make_scalar_key(const JsonbValue *scalar *** 639,653 **** * strings takes precedence. */ cstr =3D numeric_normalize(scalarVal->val.numeric); ! item =3D make_text_key(cstr, strlen(cstr), flag); pfree(cstr); break; case jbvString: ! item =3D make_text_key(scalarVal->val.string.val, scalarVal->val.strin= g.len, ! flag); break; default: ! elog(ERROR, "invalid jsonb scalar type"); } =20=20 return item; --- 605,622 ---- * strings takes precedence. */ cstr =3D numeric_normalize(scalarVal->val.numeric); ! item =3D make_text_key(JGINFLAG_NUM, cstr, strlen(cstr)); pfree(cstr); break; case jbvString: ! item =3D make_text_key(is_key ? JGINFLAG_KEY : JGINFLAG_STR, ! scalarVal->val.string.val, ! scalarVal->val.string.len); break; default: ! elog(ERROR, "unrecognized jsonb scalar type: %d", scalarVal->type); ! item =3D 0; /* keep compiler quiet */ ! break; } =20=20 return item; diff --git a/src/include/utils/jsonb.h b/src/include/utils/jsonb.h index fc746c8..1a6409a 100644 *** a/src/include/utils/jsonb.h --- b/src/include/utils/jsonb.h *************** typedef enum *** 29,53 **** WJB_END_OBJECT } JsonbIteratorToken; =20=20 ! /* ! * When using a GIN index for jsonb, we choose to index both keys and val= ues. ! * The storage format is text, with K, or V prepended to the string to in= dicate ! * key/element or value/element. ! * ! * Jsonb Keys and string array elements are treated equivalently when ! * serialized to text index storage. One day we may wish to create an op= class ! * that only indexes values, but for now keys and values are stored in GIN ! * indexes in a way that doesn't really consider their relationship to ea= ch ! * other. ! */ ! #define JKEYELEM 'K' ! #define JVAL 'V' !=20 #define JsonbContainsStrategyNumber 7 #define JsonbExistsStrategyNumber 9 #define JsonbExistsAnyStrategyNumber 10 #define JsonbExistsAllStrategyNumber 11 =20=20 /* Convenience macros */ #define DatumGetJsonb(d) ((Jsonb *) PG_DETOAST_DATUM(d)) #define JsonbGetDatum(p) PointerGetDatum(p) --- 29,69 ---- WJB_END_OBJECT } JsonbIteratorToken; =20=20 ! /* Strategy numbers for GIN index opclasses */ #define JsonbContainsStrategyNumber 7 #define JsonbExistsStrategyNumber 9 #define JsonbExistsAnyStrategyNumber 10 #define JsonbExistsAllStrategyNumber 11 =20=20 + /* + * In the standard jsonb_ops GIN opclass for jsonb, we choose to index bo= th + * keys and values. The storage format is text. The first byte of the t= ext + * string distinguishes whether this is a key (always a string), null val= ue, + * boolean value, numeric value, or string value. However, array elements + * that are strings are marked as though they were keys; this imprecision + * supports the definition of the "exists" operator, which treats array + * elements like keys. The remainder of the text string is empty for a n= ull + * value, "t" or "f" for a boolean value, a normalized print representati= on of + * a numeric value, or the text of a string value. However, if the lengt= h of + * this text representation would exceed JGIN_MAXLENGTH bytes, we instead= hash + * the text representation and store an 8-hex-digit representation of the + * uint32 hash value, marking the prefix byte with an additional bit to + * distinguish that this has happened. Hashing long strings saves space = and + * ensures that we won't overrun the maximum entry length for a GIN index. + * (But JGIN_MAXLENGTH is quite a bit shorter than GIN's limit. It's cho= sen + * to ensure that the on-disk text datum will have a short varlena header= .) + * Note that when any hashed item appears in a query, we must recheck ind= ex + * matches against the heap tuple; currently, this costs nothing because = we + * must always recheck for other reasons. + */ + #define JGINFLAG_KEY 0x01 /* key (or string array element) */ + #define JGINFLAG_NULL 0x02 /* null value */ + #define JGINFLAG_BOOL 0x03 /* boolean value */ + #define JGINFLAG_NUM 0x04 /* numeric value */ + #define JGINFLAG_STR 0x05 /* string value (if not an array element) */ + #define JGINFLAG_HASHED 0x10 /* OR'd into flag if value was hashed */ + #define JGIN_MAXLENGTH 125 /* max length of text part before hashing */ +=20 /* Convenience macros */ #define DatumGetJsonb(d) ((Jsonb *) PG_DETOAST_DATUM(d)) #define JsonbGetDatum(p) PointerGetDatum(p) ------- =_aaaaaaaaaa0 Content-Type: text/plain Content-Disposition: inline Content-Transfer-Encoding: 8bit MIME-Version: 1.0 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers ------- =_aaaaaaaaaa0--