Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xDg2V-00000000e7F-2fjo for pgsql-general@arkaria.postgresql.org; Mon, 05 Oct 2026 10:35:23 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.98.2) (envelope-from ) id 1xDg2U-00000009UnH-2jVR for pgsql-general@arkaria.postgresql.org; Mon, 05 Oct 2026 10:35:22 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xDg2U-00000009Un9-1FPL for pgsql-general@lists.postgresql.org; Mon, 05 Oct 2026 10:35:22 +0000 Received: from mail-ej2-x0f.google.com ([2a00:1450:4864:34::f]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.98.2) (envelope-from ) id 1xDg2S-00000000VgQ-0AnM for pgsql-general@lists.postgresql.org; Mon, 05 Oct 2026 10:35:22 +0000 Received: by mail-ej2-x0f.google.com with SMTP id a640c23a62f3a-c2e2050feb1so292021366b.1 for ; Mon, 05 Oct 2026 03:35:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cybertec.at; s=google; t=1791196519; x=1791801319; darn=lists.postgresql.org; h=mime-version:user-agent:content-transfer-encoding:content-type :references:in-reply-to:date:to:from:subject:message-id:from:to:cc :subject:date:message-id:reply-to:content-type; bh=azySEzjK0jbtVuJSw77Rv1KUbuJIfo2ioOgbDp/Uj7k=; b=bSxINMtEp0CxAr3iA4Otd1GZrb+OiSk8gia75lh3pTfZlLGfnX9/+TpsemT3aSHbyd Mo7WOaS0cySWSDe0DDRrc2lSM5Lrt7ve1HZVE4k+ZX+3H8birqyxupCpeQ14WHzNi6zG /g6lJ9neCXuWExmDXW2wKxcW66x73ZyLVRrYF/tfc7RyS0NigbM5IP6zlQcNV4JHS5Tv 0xFida+T4OSJeu3Tr9F8HsQywzn07+t1Me1STAJ8XhMa5H6Mac8xQYfRkdoP5KpSHRqN rdVBdix8qUs39zkLe2RIh4/cMHcC5FNa9CgFplxFWCFLEA5iBMiDMMy2Fr8VIC+dhPSB d1yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1791196519; x=1791801319; h=mime-version:user-agent:content-transfer-encoding:content-type :references:in-reply-to:date:to:from:subject:message-id:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=azySEzjK0jbtVuJSw77Rv1KUbuJIfo2ioOgbDp/Uj7k=; b=Yr1VwJdXYd6UOY2vQiOKsKcW/W/YTzph0zWlIUe9xYA6KZojfSER1UPDTnGQqa/IbY HQaYM/YE95MYzoEUWHHkJ7zQc4qWWuJiFCk2D90WNRsg4NZitfXQFklqtqloq2kP/2At OA1uBKwiHxknD3QEVdMImOgMJYN8InTRcunc6puv7rFjXrz+XYF+LKjQI0VXg7mTvKT2 Cw7I59YtvOHQsheYnCcggtDD8fPJn9FVAlDeBCX25EqOOEFlRsUOl57oOYvbjcfXsxaD ZZGk2FwCTNi6fD1ucfoYvnNUwhHB6T0Z/nydIKfwLg10U2fLU8Pu0GnokeXfBYwsY/k3 ezRw== X-Forwarded-Encrypted: i=1; AKwUvBywULKUAXMOops7jAvqfCOl7WbNNatOljuoj1dMT1FnbCwRABKr8U6HuzUjVhsNtLzDM+FKHN/0WNPwK1kw@lists.postgresql.org X-Gm-Message-State: AFq9FYLjjnvyHSxt6bVBN65EGjcCjgDkHwt1jPZaLVKUe0itxdfQPsbJ k9MR73eTHBYwfWE6HCGeWsWh2fUOUXfA445H8otFxf5qTp20aZZlT+9PG8VSdeYPPzw= X-Gm-Gg: AYBFou0JoGPQVFXL2df/f5Rvr+cClVsHCocn5+1EemJoQsYWw+hj5jOL68HI6EdTefM uEkJKydPU1x9a+Ydcd9dGRUCbvo11uUbNRy4OTQK0yyhN+rFFMM3WKVn8nxBbjO1Qgq6EVKxGav tOjUAJEgW4nsll25M5VlReSXGdqKJUPpDwDmIsWjIU2qOC8rn6aPnwgdZI13Qk5iAwGDt8c/LXY gRpot7d55M1WyKoxwEbVPUDUGExJXnfNIAXqcbistvAH6Tjjjiye79WQXauysvWH/qcSZm4MDoR S1xLpHJelFng1vP5bKq9QP2Bo7GlNxBzuCRwDP4/mKPy32BY1vC0v4TPjoXh1h2XTFP1JbK9wZd D3kyeuGbPWs2gwNLMU6C35r7IpZPbHeJTMEPHsZxihGrFvCE7oVYMwHbDTfPh3dua+C8NPJ0lYx sFFzfqWqs34UgozudR8RE0dnD4NkO+N7PqemJGYm8YAOA/ttdTD4pv2SI1bNo4kpicvo9+Z5VLs rHtra1FfaHhI440eDkxF9zFD40J2wiwnlbDOg== X-Received: by 2002:a17:907:3d52:b0:c2a:46a2:5b57 with SMTP id a640c23a62f3a-c2e4ad568c9mr809585266b.10.1791196518744; Mon, 05 Oct 2026 03:35:18 -0700 (PDT) Received: from laurenz.albe-K4N0CV00F97414D ([2001:871:70:8b5b:c776:83dc:e5e8:cfc]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-c3156333b0fsm57116866b.62.2026.10.05.03.35.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Oct 2026 03:35:17 -0700 (PDT) Message-ID: <088bd47f443ee4b8f657e3dbfadf611a560aa95d.camel@cybertec.at> Subject: Re: Parsing of hex encoding strings From: Laurenz Albe To: Thomas de Zeeuw , pgsql-general@lists.postgresql.org Date: Mon, 05 Oct 2026 12:35:17 +0200 In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.60.2 (3.60.2-2.fc44) MIME-Version: 1.0 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On Mon, 2026-10-05 at 11:50 +0200, Thomas de Zeeuw wrote: > I=E2=80=99m working on a system where I need to search for hex encoded st= rings, for > which I=E2=80=99m using full text search with the =E2=80=9Csimple=E2=80= =9D directory. This is working > fine for the most part, but I=E2=80=99ve run into an issue which may or m= ay not be > considered a bug. >=20 > When creating the ts vector and queries I want to achieve that the entire= hex > encoded string is seen as a single word. For a string such as =E2=80=9Cfe= 82fbb28=E2=80=9D > this works using the =E2=80=9Csimple=E2=80=9D directory. However a string= such as =E2=80=9C7e82fbb28=E2=80=9D > is parsed as two words: =E2=80=9C7e82=E2=80=9D as scientific notation and= =E2=80=9Cfbb28" as numword. > I=E2=80=99ve put the output of ts_debug at the bottom of this email for c= onvince. >=20 > I would have expected that the entire string would be considered as one t= oken > considering it doesn=E2=80=99t have any whitespace. And since that token = isn=E2=80=99t a valid > scientific number as whole (though it can be, just not the examples I=E2= =80=99ve > shared above) I would expected it be considered a single word (numword). > Reading the URL example from chapter 12.5 > (https://www.postgresql.org/docs/current/textsearch-parsers.html) maybe i= t > could have to two tokens, one for the scientific number (=E2=80=9C7e82=E2= =80=9D) and another > numword token for the entire string (7e82fbb28=E2=80=9D), this would also= work for my > use case. >=20 > Finally, my question: is this considered behaviour a bug or working as in= tended? That is working as intended. The problem is that you are trying to use ful= l-text search for something that isn't a text. Storing strings in hex encoding wa= s a design error. Not only does it keep you from using full-text search, but y= ou are also wasting storage space (unless TOAST compression takes care of that= ). You could use substring search with pg_trgm, but it is not the same as full-text search. Yours, Laurenz Albe