Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kz9Ok-00030P-5y for pgsql-hackers@arkaria.postgresql.org; Tue, 12 Jan 2021 02:27:07 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1kz9Oj-0002hR-02 for pgsql-hackers@arkaria.postgresql.org; Tue, 12 Jan 2021 02:27:05 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kz9Oh-0002hJ-Eg for pgsql-hackers@lists.postgresql.org; Tue, 12 Jan 2021 02:27:04 +0000 Received: from wout5-smtp.messagingengine.com ([64.147.123.21]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kz9Od-0006lU-4V for pgsql-hackers@postgresql.org; Tue, 12 Jan 2021 02:27:03 +0000 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id B7BE5205A; Mon, 11 Jan 2021 21:26:55 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Mon, 11 Jan 2021 21:26:55 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=paquier.xyz; h= date:from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm3; bh=z3Lye39+v+SmfKtmHx7DeztOd75 DzzYGd4e+m1s+CZ8=; b=IBWwpPivYymh8IrQAFItjurG8Vm89/7EyMB38apTFbT AaBuCRLWhyvnlNoZ8blX4x/2oMai3XmcUHJ1/n7vBQLWhNNAyMoAE0UL0WZbLdbg EBk2kvDgOMpxGPDf3yfi+dJzI6hYlAxL8SIcLkTEipJ/lI9XV1vBHEl5RLSMdPvD PG/brJj7YVHkTHJ2yCWlSaJ3mt+Q8UX3PSEMRlJkdmzKyP8q7dnXp8522gDAzASj nYA128YthWUCrXLkNMcdU7b8PG1Bebg54d5SKrWkwMyuPDG/sXY4o38kdvZB6HY2 TyriJe6Ah3SU3/V6w/E7dARETtIpVxkIcXm4xQevoHg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=z3Lye3 9+v+SmfKtmHx7DeztOd75DzzYGd4e+m1s+CZ8=; b=EmG47LzCp8fn91JLIMoj1D 5CrXP/GrCzBQMTObfzidcpTRfyeGw7fgnc+LnGnlz6ORx68NK5MD1CotWWn0I3H2 cwcnGos3MLthMhwALs/w2fszuwIvSHOJNU2Eg67pYsSusF9GMHAOtEme4U8jCvWE wAQZEF2zuouh5n8wCtfTaKwG9dRLo2low9+qVhubF2luCZn5ogJCOlRBBfBalWHb Yi4MYW/fBUKont5EOcOux7/1HcRVRaH+9+lUfGPqB97Ydl2kGVQsyD8vo1T8fVkb j1aIQ6bVWCWOBd8NdFDoSg0e5esB75bH9g/0bQQnzFoxXQt6rDoudYbWinljyh7w == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedujedrvdehvddggeeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne gfrhhlucfvnfffucdljedtmdenucfjughrpeffhffvuffkfhggtggujgesghdtreertddt vdenucfhrhhomhepofhitghhrggvlhcurfgrqhhuihgvrhcuoehmihgthhgrvghlsehprg hquhhivghrrdighiiiqeenucggtffrrghtthgvrhhnpedvgeduuefhtdeuleettdevjeeh heeiveeuieegleetgeeljeelieeuieehgeevhfenucfkphepudduuddruddtvddrudektd drudekheenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhm pehmihgthhgrvghlsehprghquhhivghrrdighiii X-ME-Proxy: Received: from paquier.xyz (ee0822lan1.rev.em-net.ne.jp [111.102.180.185]) by mail.messagingengine.com (Postfix) with ESMTPA id E4A5D1080059; Mon, 11 Jan 2021 21:26:53 -0500 (EST) Date: Tue, 12 Jan 2021 11:26:51 +0900 From: Michael Paquier To: Bruce Momjian Cc: PostgreSQL-development Subject: Re: Moving other hex functions to /common Message-ID: References: <20210101200613.GB7432@momjian.us> <20210105034739.GG7432@momjian.us> <20210105172109.GH7432@momjian.us> <20210106135823.GD32172@momjian.us> <20210111162730.GC4320@momjian.us> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="8CFd2oXuwoXoOSrw" Content-Disposition: inline In-Reply-To: <20210111162730.GC4320@momjian.us> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --8CFd2oXuwoXoOSrw Content-Type: multipart/mixed; boundary="EbfbO4bElJzk1fEW" Content-Disposition: inline --EbfbO4bElJzk1fEW Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Mon, Jan 11, 2021 at 11:27:30AM -0500, Bruce Momjian wrote: > Sure, I realize the elog/pg_log is odd, but the alternatives seem worse. I guess that it depends on the use cases. If there is no need to worry about shared libraries, elog/pg_log would do just fine. > You can take ownership of my hex patch and just add to it. I know you > already did the hex length part, and have other ideas of what you want. OK, thanks. I have been looking at it, and tweaked the patch as per the attached. That's basically what you did on the following points: - Use size_t for the arguments, uint64 as return result. - Leave ECPG out of the equation. - Use pg_log()/elog() to report failures in src/common/, rather than error codes. - Renamed the arguments of encode.c to src, srclen, dst, dstlen. The two only things that were not present are the set of checks for overflows, and the adjustments for varlena.c. The first point makes the code of encode.c safer, as previously the code would issue a FATAL *after* writing out-of-bound data. Now it issues an ERROR before any overwrite is done, and I have added some assertions as an extra safety net. For the second, I think that it makes the allocation pattern easier to follow, similarly to checksum manifests. Thoughts? -- Michael --EbfbO4bElJzk1fEW Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v5-0001-Refactor-hex-code.patch" Content-Transfer-Encoding: quoted-printable =46rom 3e5c9fa42ff981933bc6eeede92335ae89e94e21 Mon Sep 17 00:00:00 2001 =46rom: Michael Paquier Date: Tue, 12 Jan 2021 11:21:42 +0900 Subject: [PATCH v5] Refactor hex code --- src/include/common/hex.h | 25 +++ src/include/common/hex_decode.h | 16 -- src/include/utils/builtins.h | 3 - src/backend/replication/backup_manifest.c | 28 ++-- src/backend/utils/adt/encode.c | 96 ++++++----- src/backend/utils/adt/varlena.c | 16 +- src/common/Makefile | 2 +- src/common/hex.c | 196 ++++++++++++++++++++++ src/common/hex_decode.c | 106 ------------ src/tools/msvc/Mkvcbuild.pm | 2 +- 10 files changed, 308 insertions(+), 182 deletions(-) create mode 100644 src/include/common/hex.h delete mode 100644 src/include/common/hex_decode.h create mode 100644 src/common/hex.c delete mode 100644 src/common/hex_decode.c diff --git a/src/include/common/hex.h b/src/include/common/hex.h new file mode 100644 index 0000000000..3c3c956bb6 --- /dev/null +++ b/src/include/common/hex.h @@ -0,0 +1,25 @@ +/*------------------------------------------------------------------------ + * + * hex.h + * Encoding and decoding routines for hex strings. + * + * Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * IDENTIFICATION + * src/include/common/hex.h + * + *------------------------------------------------------------------------ + */ + +#ifndef COMMON_HEX_H +#define COMMON_HEX_H + +extern uint64 pg_hex_decode(const char *src, size_t srclen, + char *dst, size_t dstlen); +extern uint64 pg_hex_encode(const char *src, size_t srclen, + char *dst, size_t dstlen); +extern uint64 pg_hex_enc_len(size_t srclen); +extern uint64 pg_hex_dec_len(size_t srclen); + +#endif /* COMMON_HEX_H */ diff --git a/src/include/common/hex_decode.h b/src/include/common/hex_decod= e.h deleted file mode 100644 index 29ab248458..0000000000 --- a/src/include/common/hex_decode.h +++ /dev/null @@ -1,16 +0,0 @@ -/* - * hex_decode.h - * hex decoding - * - * Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group - * Portions Copyright (c) 1994, Regents of the University of California - * - * src/include/common/hex_decode.h - */ -#ifndef COMMON_HEX_DECODE_H -#define COMMON_HEX_DECODE_H - -extern uint64 hex_decode(const char *src, size_t len, char *dst); - - -#endif /* COMMON_HEX_DECODE_H */ diff --git a/src/include/utils/builtins.h b/src/include/utils/builtins.h index 6f739a8822..27d2f2ffb3 100644 --- a/src/include/utils/builtins.h +++ b/src/include/utils/builtins.h @@ -31,9 +31,6 @@ extern void domain_check(Datum value, bool isnull, Oid do= mainType, extern int errdatatype(Oid datatypeOid); extern int errdomainconstraint(Oid datatypeOid, const char *conname); =20 -/* encode.c */ -extern uint64 hex_encode(const char *src, size_t len, char *dst); - /* int.c */ extern int2vector *buildint2vector(const int16 *int2s, int n); =20 diff --git a/src/backend/replication/backup_manifest.c b/src/backend/replic= ation/backup_manifest.c index 8af94610b3..0df5186828 100644 --- a/src/backend/replication/backup_manifest.c +++ b/src/backend/replication/backup_manifest.c @@ -13,11 +13,11 @@ #include "postgres.h" =20 #include "access/timeline.h" +#include "common/hex.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" #include "replication/backup_manifest.h" -#include "utils/builtins.h" #include "utils/json.h" =20 static void AppendStringToManifest(backup_manifest_info *manifest, char *s= ); @@ -150,10 +150,12 @@ AddFileToBackupManifest(backup_manifest_info *manifes= t, const char *spcoid, } else { + uint64 dstlen =3D pg_hex_enc_len(pathlen); + appendStringInfoString(&buf, "{ \"Encoded-Path\": \""); - enlargeStringInfo(&buf, 2 * pathlen); - buf.len +=3D hex_encode(pathname, pathlen, - &buf.data[buf.len]); + enlargeStringInfo(&buf, dstlen); + buf.len +=3D pg_hex_encode(pathname, pathlen, + &buf.data[buf.len], dstlen); appendStringInfoString(&buf, "\", "); } =20 @@ -176,6 +178,7 @@ AddFileToBackupManifest(backup_manifest_info *manifest,= const char *spcoid, { uint8 checksumbuf[PG_CHECKSUM_MAX_LENGTH]; int checksumlen; + uint64 dstlen; =20 checksumlen =3D pg_checksum_final(checksum_ctx, checksumbuf); if (checksumlen < 0) @@ -185,9 +188,10 @@ AddFileToBackupManifest(backup_manifest_info *manifest= , const char *spcoid, appendStringInfo(&buf, ", \"Checksum-Algorithm\": \"%s\", \"Checksum\": \"", pg_checksum_type_name(checksum_ctx->type)); - enlargeStringInfo(&buf, 2 * checksumlen); - buf.len +=3D hex_encode((char *) checksumbuf, checksumlen, - &buf.data[buf.len]); + dstlen =3D pg_hex_enc_len(checksumlen); + enlargeStringInfo(&buf, dstlen); + buf.len +=3D pg_hex_encode((char *) checksumbuf, checksumlen, + &buf.data[buf.len], dstlen); appendStringInfoChar(&buf, '"'); } =20 @@ -307,8 +311,9 @@ SendBackupManifest(backup_manifest_info *manifest) { StringInfoData protobuf; uint8 checksumbuf[PG_SHA256_DIGEST_LENGTH]; - char checksumstringbuf[PG_SHA256_DIGEST_STRING_LENGTH]; + char *checksumstringbuf; size_t manifest_bytes_done =3D 0; + uint64 dstlen; =20 if (!IsManifestEnabled(manifest)) return; @@ -328,8 +333,11 @@ SendBackupManifest(backup_manifest_info *manifest) if (pg_cryptohash_final(manifest->manifest_ctx, checksumbuf) < 0) elog(ERROR, "failed to finalize checksum of backup manifest"); AppendStringToManifest(manifest, "\"Manifest-Checksum\": \""); - hex_encode((char *) checksumbuf, sizeof checksumbuf, checksumstringbuf); - checksumstringbuf[PG_SHA256_DIGEST_STRING_LENGTH - 1] =3D '\0'; + dstlen =3D pg_hex_enc_len(PG_SHA256_DIGEST_LENGTH); + checksumstringbuf =3D palloc0(dstlen + 1); /* includes \0 */ + pg_hex_encode((char *) checksumbuf, sizeof checksumbuf, + checksumstringbuf, dstlen); + checksumstringbuf[dstlen] =3D '\0'; AppendStringToManifest(manifest, checksumstringbuf); AppendStringToManifest(manifest, "\"}\n"); =20 diff --git a/src/backend/utils/adt/encode.c b/src/backend/utils/adt/encode.c index 4759be25e9..6e26f2af8a 100644 --- a/src/backend/utils/adt/encode.c +++ b/src/backend/utils/adt/encode.c @@ -15,7 +15,7 @@ =20 #include =20 -#include "common/hex_decode.h" +#include "common/hex.h" #include "mb/pg_wchar.h" #include "utils/builtins.h" #include "utils/memutils.h" @@ -32,10 +32,12 @@ */ struct pg_encoding { - uint64 (*encode_len) (const char *data, size_t dlen); - uint64 (*decode_len) (const char *data, size_t dlen); - uint64 (*encode) (const char *data, size_t dlen, char *res); - uint64 (*decode) (const char *data, size_t dlen, char *res); + uint64 (*encode_len) (const char *src, size_t srclen); + uint64 (*decode_len) (const char *src, size_t srclen); + uint64 (*encode) (const char *src, size_t srclen, + char *dst, size_t dstlen); + uint64 (*decode) (const char *src, size_t srclen, + char *dst, size_t dstlen); }; =20 static const struct pg_encoding *pg_find_encoding(const char *name); @@ -81,11 +83,7 @@ binary_encode(PG_FUNCTION_ARGS) =20 result =3D palloc(VARHDRSZ + resultlen); =20 - res =3D enc->encode(dataptr, datalen, VARDATA(result)); - - /* Make this FATAL 'cause we've trodden on memory ... */ - if (res > resultlen) - elog(FATAL, "overflow - encode estimate too small"); + res =3D enc->encode(dataptr, datalen, VARDATA(result), resultlen); =20 SET_VARSIZE(result, VARHDRSZ + res); =20 @@ -129,11 +127,7 @@ binary_decode(PG_FUNCTION_ARGS) =20 result =3D palloc(VARHDRSZ + resultlen); =20 - res =3D enc->decode(dataptr, datalen, VARDATA(result)); - - /* Make this FATAL 'cause we've trodden on memory ... */ - if (res > resultlen) - elog(FATAL, "overflow - decode estimate too small"); + res =3D enc->decode(dataptr, datalen, VARDATA(result), resultlen); =20 SET_VARSIZE(result, VARHDRSZ + res); =20 @@ -145,32 +139,20 @@ binary_decode(PG_FUNCTION_ARGS) * HEX */ =20 -static const char hextbl[] =3D "0123456789abcdef"; - -uint64 -hex_encode(const char *src, size_t len, char *dst) -{ - const char *end =3D src + len; - - while (src < end) - { - *dst++ =3D hextbl[(*src >> 4) & 0xF]; - *dst++ =3D hextbl[*src & 0xF]; - src++; - } - return (uint64) len * 2; -} - +/* + * Those two wrappers are still needed to match with the layer of + * src/common/. + */ static uint64 hex_enc_len(const char *src, size_t srclen) { - return (uint64) srclen << 1; + return pg_hex_enc_len(srclen); } =20 static uint64 hex_dec_len(const char *src, size_t srclen) { - return (uint64) srclen >> 1; + return pg_hex_dec_len(srclen); } =20 /* @@ -192,12 +174,12 @@ static const int8 b64lookup[128] =3D { }; =20 static uint64 -pg_base64_encode(const char *src, size_t len, char *dst) +pg_base64_encode(const char *src, size_t srclen, char *dst, size_t dstlen) { char *p, *lend =3D dst + 76; const char *s, - *end =3D src + len; + *end =3D src + srclen; int pos =3D 2; uint32 buf =3D 0; =20 @@ -213,6 +195,8 @@ pg_base64_encode(const char *src, size_t len, char *dst) /* write it out */ if (pos < 0) { + if ((p - dst + 4) > dstlen) + elog(ERROR, "overflow - encode estimate too small"); *p++ =3D _base64[(buf >> 18) & 0x3f]; *p++ =3D _base64[(buf >> 12) & 0x3f]; *p++ =3D _base64[(buf >> 6) & 0x3f]; @@ -223,25 +207,30 @@ pg_base64_encode(const char *src, size_t len, char *d= st) } if (p >=3D lend) { + if ((p - dst + 1) > dstlen) + elog(ERROR, "overflow - encode estimate too small"); *p++ =3D '\n'; lend =3D p + 76; } } if (pos !=3D 2) { + if ((p - dst + 4) > dstlen) + elog(ERROR, "overflow - encode estimate too small"); *p++ =3D _base64[(buf >> 18) & 0x3f]; *p++ =3D _base64[(buf >> 12) & 0x3f]; *p++ =3D (pos =3D=3D 0) ? _base64[(buf >> 6) & 0x3f] : '=3D'; *p++ =3D '=3D'; } =20 + Assert((p - dst) <=3D dstlen); return p - dst; } =20 static uint64 -pg_base64_decode(const char *src, size_t len, char *dst) +pg_base64_decode(const char *src, size_t srclen, char *dst, size_t dstlen) { - const char *srcend =3D src + len, + const char *srcend =3D src + srclen, *s =3D src; char *p =3D dst; char c; @@ -289,11 +278,21 @@ pg_base64_decode(const char *src, size_t len, char *d= st) pos++; if (pos =3D=3D 4) { + if ((p - dst + 1) > dstlen) + elog(ERROR, "overflow - decode estimate too small"); *p++ =3D (buf >> 16) & 255; if (end =3D=3D 0 || end > 1) + { + if ((p - dst + 1) > dstlen) + elog(ERROR, "overflow - decode estimate too small"); *p++ =3D (buf >> 8) & 255; + } if (end =3D=3D 0 || end > 2) + { + if ((p - dst + 1) > dstlen) + elog(ERROR, "overflow - decode estimate too small"); *p++ =3D buf & 255; + } buf =3D 0; pos =3D 0; } @@ -305,6 +304,7 @@ pg_base64_decode(const char *src, size_t len, char *dst) errmsg("invalid base64 end sequence"), errhint("Input data is missing padding, is truncated, or is otherwise= corrupted."))); =20 + Assert((p - dst) <=3D dstlen); return p - dst; } =20 @@ -340,7 +340,7 @@ pg_base64_dec_len(const char *src, size_t srclen) #define DIG(VAL) ((VAL) + '0') =20 static uint64 -esc_encode(const char *src, size_t srclen, char *dst) +esc_encode(const char *src, size_t srclen, char *dst, size_t dstlen) { const char *end =3D src + srclen; char *rp =3D dst; @@ -352,6 +352,8 @@ esc_encode(const char *src, size_t srclen, char *dst) =20 if (c =3D=3D '\0' || IS_HIGHBIT_SET(c)) { + if ((rp - dst + 4) > dstlen) + elog(ERROR, "overflow - encode estimate too small"); rp[0] =3D '\\'; rp[1] =3D DIG(c >> 6); rp[2] =3D DIG((c >> 3) & 7); @@ -361,6 +363,8 @@ esc_encode(const char *src, size_t srclen, char *dst) } else if (c =3D=3D '\\') { + if ((rp - dst + 2) > dstlen) + elog(ERROR, "overflow - encode estimate too small"); rp[0] =3D '\\'; rp[1] =3D '\\'; rp +=3D 2; @@ -368,6 +372,8 @@ esc_encode(const char *src, size_t srclen, char *dst) } else { + if ((rp - dst + 1) > dstlen) + elog(ERROR, "overflow - encode estimate too small"); *rp++ =3D c; len++; } @@ -375,11 +381,12 @@ esc_encode(const char *src, size_t srclen, char *dst) src++; } =20 + Assert((rp - dst) <=3D dstlen); return len; } =20 static uint64 -esc_decode(const char *src, size_t srclen, char *dst) +esc_decode(const char *src, size_t srclen, char *dst, size_t dstlen) { const char *end =3D src + srclen; char *rp =3D dst; @@ -388,7 +395,11 @@ esc_decode(const char *src, size_t srclen, char *dst) while (src < end) { if (src[0] !=3D '\\') + { + if ((rp - dst + 1) > dstlen) + elog(ERROR, "overflow - decode estimate too small"); *rp++ =3D *src++; + } else if (src + 3 < end && (src[1] >=3D '0' && src[1] <=3D '3') && (src[2] >=3D '0' && src[2] <=3D '7') && @@ -400,12 +411,16 @@ esc_decode(const char *src, size_t srclen, char *dst) val <<=3D 3; val +=3D VAL(src[2]); val <<=3D 3; + if ((rp - dst + 1) > dstlen) + elog(ERROR, "overflow - decode estimate too small"); *rp++ =3D val + VAL(src[3]); src +=3D 4; } else if (src + 1 < end && (src[1] =3D=3D '\\')) { + if ((rp - dst + 1) > dstlen) + elog(ERROR, "overflow - decode estimate too small"); *rp++ =3D '\\'; src +=3D 2; } @@ -423,6 +438,7 @@ esc_decode(const char *src, size_t srclen, char *dst) len++; } =20 + Assert((rp - dst) <=3D dstlen); return len; } =20 @@ -504,7 +520,7 @@ static const struct { "hex", { - hex_enc_len, hex_dec_len, hex_encode, hex_decode + hex_enc_len, hex_dec_len, pg_hex_encode, pg_hex_decode } }, { diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlen= a.c index 4838bfb4ff..479ed9ae54 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -21,8 +21,8 @@ #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" +#include "common/hex.h" #include "common/int.h" -#include "common/hex_decode.h" #include "common/unicode_norm.h" #include "lib/hyperloglog.h" #include "libpq/pqformat.h" @@ -304,10 +304,12 @@ byteain(PG_FUNCTION_ARGS) if (inputText[0] =3D=3D '\\' && inputText[1] =3D=3D 'x') { size_t len =3D strlen(inputText); + uint64 dstlen =3D pg_hex_dec_len(len - 2); =20 - bc =3D (len - 2) / 2 + VARHDRSZ; /* maximum possible length */ + bc =3D dstlen + VARHDRSZ; /* maximum possible length */ result =3D palloc(bc); - bc =3D hex_decode(inputText + 2, len - 2, VARDATA(result)); + + bc =3D pg_hex_decode(inputText + 2, len - 2, VARDATA(result), dstlen); SET_VARSIZE(result, bc + VARHDRSZ); /* actual length */ =20 PG_RETURN_BYTEA_P(result); @@ -396,11 +398,15 @@ byteaout(PG_FUNCTION_ARGS) =20 if (bytea_output =3D=3D BYTEA_OUTPUT_HEX) { + uint64 dstlen =3D pg_hex_enc_len(VARSIZE_ANY_EXHDR(vlena)); + /* Print hex format */ - rp =3D result =3D palloc(VARSIZE_ANY_EXHDR(vlena) * 2 + 2 + 1); + rp =3D result =3D palloc(dstlen + 2 + 1); *rp++ =3D '\\'; *rp++ =3D 'x'; - rp +=3D hex_encode(VARDATA_ANY(vlena), VARSIZE_ANY_EXHDR(vlena), rp); + + rp +=3D pg_hex_encode(VARDATA_ANY(vlena), VARSIZE_ANY_EXHDR(vlena), rp, + dstlen); } else if (bytea_output =3D=3D BYTEA_OUTPUT_ESCAPE) { diff --git a/src/common/Makefile b/src/common/Makefile index f624977939..93eb27a2aa 100644 --- a/src/common/Makefile +++ b/src/common/Makefile @@ -58,7 +58,7 @@ OBJS_COMMON =3D \ file_perm.o \ file_utils.o \ hashfn.o \ - hex_decode.o \ + hex.o \ ip.o \ jsonapi.o \ keywords.o \ diff --git a/src/common/hex.c b/src/common/hex.c new file mode 100644 index 0000000000..0123c69697 --- /dev/null +++ b/src/common/hex.c @@ -0,0 +1,196 @@ +/*------------------------------------------------------------------------- + * + * hex.c + * Encoding and decoding routines for hex. + * + * Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * IDENTIFICATION + * src/common/hex.c + * + *------------------------------------------------------------------------- + */ + + +#ifndef FRONTEND +#include "postgres.h" +#else +#include "postgres_fe.h" +#endif + +#include "common/hex.h" +#ifdef FRONTEND +#include "common/logging.h" +#endif +#include "mb/pg_wchar.h" + + +static const int8 hexlookup[128] =3D { + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, -1, -1, -1, -1, -1, -1, + -1, 10, 11, 12, 13, 14, 15, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, 10, 11, 12, 13, 14, 15, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, +}; + +static const char hextbl[] =3D "0123456789abcdef"; + +static inline char +get_hex(const char *cp) +{ + unsigned char c =3D (unsigned char) *cp; + int res =3D -1; + + if (c < 127) + res =3D hexlookup[c]; + + if (res < 0) + { +#ifdef FRONTEND + pg_log_fatal("invalid hexadecimal digit"); + exit(EXIT_FAILURE); +#else + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("invalid hexadecimal digit: \"%.*s\"", + pg_mblen(cp), cp))); +#endif + } + + return (char) res; +} + +/* + * pg_hex_encode + * + * Encode into hex the given string. Returns the length of the encoded + * string. + */ +uint64 +pg_hex_encode(const char *src, size_t srclen, char *dst, size_t dstlen) +{ + const char *end =3D src + srclen; + char *p; + + p =3D dst; + + while (src < end) + { + /* + * Leave if there is an overflow in the area allocated for the + * encoded string. + */ + if ((p - dst + 2) > dstlen) + { +#ifdef FRONTEND + pg_log_fatal("overflow of destination buffer during hex encoding"); + exit(EXIT_FAILURE); +#else + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("overflow of destination buffer during hex encoding"))); +#endif + } + + *p++ =3D hextbl[(*src >> 4) & 0xF]; + *p++ =3D hextbl[*src & 0xF]; + src++; + } + + Assert((p - dst) <=3D dstlen); + return p - dst; +} + +/* + * pg_hex_decode + * + * Decode the given hex string. Returns the length of the decoded string. + */ +uint64 +pg_hex_decode(const char *src, size_t srclen, char *dst, size_t dstlen) +{ + const char *s, + *srcend; + char v1, + v2, + *p; + + srcend =3D src + srclen; + s =3D src; + p =3D dst; + while (s < srcend) + { + if (*s =3D=3D ' ' || *s =3D=3D '\n' || *s =3D=3D '\t' || *s =3D=3D '\r') + { + s++; + continue; + } + v1 =3D get_hex(s) << 4; + s++; + + if (s >=3D srcend) + { +#ifdef FRONTEND + pg_log_fatal("invalid hexadecimal data: odd number of digits"); + exit(EXIT_FAILURE); +#else + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("invalid hexadecimal data: odd number of digits"))); +#endif + } + + v2 =3D get_hex(s); + s++; + + /* overflow check */ + if ((p - dst + 1) > dstlen) + { +#ifdef FRONTEND + pg_log_fatal("overflow of destination buffer during hex decoding"); + exit(EXIT_FAILURE); +#else + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("overflow of destination buffer during hex decoding"))); +#endif + } + + *p++ =3D v1 | v2; + } + + Assert((p - dst) <=3D dstlen); + return p - dst; +} + +/* + * pg_hex_enc_len + * + * Returns to caller the length of the string if it were encoded with + * hex based on the length provided by caller. This is useful to estimate + * how large a buffer allocation needs to be done before doing the actual + * encoding. + */ +uint64 +pg_hex_enc_len(size_t srclen) +{ + return (srclen << 1); +} + +/* + * pg_hex_dec_len + * + * Returns to caller the length of the string if it were to be decoded + * with hex, based on the length given by caller. This is useful to + * estimate how large a buffer allocation needs to be done before doing + * the actual decoding. + */ +uint64 +pg_hex_dec_len(size_t srclen) +{ + return (srclen >> 1); +} diff --git a/src/common/hex_decode.c b/src/common/hex_decode.c deleted file mode 100644 index dbc0e1c30a..0000000000 --- a/src/common/hex_decode.c +++ /dev/null @@ -1,106 +0,0 @@ -/*------------------------------------------------------------------------- - * - * hex_decode.c - * hex decoding - * - * - * Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group - * Portions Copyright (c) 1994, Regents of the University of California - * - * - * IDENTIFICATION - * src/common/hex_decode.c - * - *------------------------------------------------------------------------- - */ - - -#ifndef FRONTEND -#include "postgres.h" -#else -#include "postgres_fe.h" -#endif - -#ifdef FRONTEND -#include "common/logging.h" -#else -#include "mb/pg_wchar.h" -#endif -#include "common/hex_decode.h" - - -static const int8 hexlookup[128] =3D { - -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, - -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, - -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, - 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, -1, -1, -1, -1, -1, -1, - -1, 10, 11, 12, 13, 14, 15, -1, -1, -1, -1, -1, -1, -1, -1, -1, - -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, - -1, 10, 11, 12, 13, 14, 15, -1, -1, -1, -1, -1, -1, -1, -1, -1, - -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -}; - -static inline char -get_hex(const char *cp) -{ - unsigned char c =3D (unsigned char) *cp; - int res =3D -1; - - if (c < 127) - res =3D hexlookup[c]; - - if (res < 0) - { -#ifdef FRONTEND - pg_log_fatal("invalid hexadecimal digit"); - exit(EXIT_FAILURE); -#else - ereport(ERROR, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("invalid hexadecimal digit: \"%.*s\"", - pg_mblen(cp), cp))); -#endif - } - - return (char) res; -} - -uint64 -hex_decode(const char *src, size_t len, char *dst) -{ - const char *s, - *srcend; - char v1, - v2, - *p; - - srcend =3D src + len; - s =3D src; - p =3D dst; - while (s < srcend) - { - if (*s =3D=3D ' ' || *s =3D=3D '\n' || *s =3D=3D '\t' || *s =3D=3D '\r') - { - s++; - continue; - } - v1 =3D get_hex(s) << 4; - s++; - if (s >=3D srcend) - { -#ifdef FRONTEND - pg_log_fatal("invalid hexadecimal data: odd number of digits"); - exit(EXIT_FAILURE); -#else - ereport(ERROR, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("invalid hexadecimal data: odd number of digits"))); -#endif - } - v2 =3D get_hex(s); - s++; - *p++ =3D v1 | v2; - } - - return p - dst; -} diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm index 7f014a12c9..60b216cce0 100644 --- a/src/tools/msvc/Mkvcbuild.pm +++ b/src/tools/msvc/Mkvcbuild.pm @@ -121,7 +121,7 @@ sub mkvcbuild our @pgcommonallfiles =3D qw( archive.c base64.c checksum_helper.c config_info.c controldata_utils.c d2s.c encnames.c exec.c - f2s.c file_perm.c file_utils.c hashfn.c hex_decode.c ip.c jsonapi.c + f2s.c file_perm.c file_utils.c hashfn.c hex.c ip.c jsonapi.c keywords.c kwlookup.c link-canary.c md5_common.c pg_get_line.c pg_lzcompress.c pgfnames.c psprintf.c relpath.c rmtree.c saslprep.c scram-common.c string.c stringinfo.c unicode_norm.c username= =2Ec --=20 2.30.0 --EbfbO4bElJzk1fEW-- --8CFd2oXuwoXoOSrw Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEG72nH6vTowiyblFKnvQgOdbyQH0FAl/9COsACgkQnvQgOdby QH1N8A/7Bld8yHT9g1QahaN2ChYOjxm8j/sE6zVGkvOZCOv62+6T2YuKjDZOUTE7 8RjeJ1/ZLpSscQgT3JdaTg4F2ug9t6Ou3a3hOwsQlodjvJ7AyYJIIP8flMpjsUMw SzcERRate4CT3Lh+Tw22fsTcSzxhZ7Wqg4r0DLPNehpoTfr0rHQSlDZx3AgVFjAh J8NjgvF0wirVknG/lNRtLV55hpUAudVyZWI6HngfR2jeH6+JY87zZqQyB9jk9ev5 zjYpWLM3fzaJkZiV0Gff1ARUeBXq1W5ctVsWuOqV8MgdLjCOvr2//7eKIOqyL7d7 pHBqxH81m234s+z7yVgjBD6R+cGeFVGFxtElLcUykaD9FzfQeb1QIforKIkJYUGv hBoZqrn3+pOeXMzdU4+ONSDWsje5ymsC2sKMgwTuFVGSkM6gVUZyJzNQ1d3N4obX 5dxzgW8jQPQyZ+QlHghCOgFIwqaNT+pAHfTFel1LDWfQfA4Jyf22m1EkQBiqoRQ3 r3lra+UOeJ3SHfSVdwVfs2Wd/r616gt/KWfFWbBjaBYjMDtdkzh9NXdq2n/ZDkH6 52ja7aaDLFvi+qtj+RLWBSp2JJnBtr1qBpdO56xoWhU+NZMBy0TlQYw7hhIfBRYU YDIokpN7yULTUwXZy7U0lNsq06tUjyfmDo9u2NlQo+eYFytQWdU= =ntx1 -----END PGP SIGNATURE----- --8CFd2oXuwoXoOSrw--