pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedBUG #19748: tsvectorrecv accepts duplicate lexemes and position 0 that tsvectorin rejects
3+ messages / 2 participants
[nested] [flat]
Commits from this thread:
2026-10-07 10cc5aa96a Tighten data validity checks in tsvectorrecv().
* BUG #19748: tsvectorrecv accepts duplicate lexemes and position 0 that tsvectorin rejects
@ 2026-10-04 05:19 PG Bug reporting form <noreply@postgresql.org>
2026-10-05 18:48 ` Re: BUG #19748: tsvectorrecv accepts duplicate lexemes and position 0 that tsvectorin rejects Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 3+ messages in thread
From: PG Bug reporting form @ 2026-10-04 05:19 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org; +Cc: kehan5800@gmail.com
The following bug has been logged on the website:
Bug reference: 19748
Logged by: Ke
Email address: kehan5800@gmail.com
PostgreSQL version: 18.6
Operating system: Ubuntu 22.04.2 x86_64
Description:
tsvectorrecv() (src/backend/utils/adt/tsvector.c:446) has this comment
(tsvector.c:552-556):
* Enforce that datalen is still within MAXSTRPOS, ie the last lexeme
* didn't go past that. We could allow that, since no "pos" field
* overflowed, but tsvectorrecv shouldn't accept values that other
* tsvector-constructing routines wouldn't.
Two checks are still missing relative to tsvectorin():
1. Duplicate lexemes are not merged. tsvectorin() runs uniqueentry()
(tsvector.c:265). tsvectorrecv() detects "compareentry(...) <= 0"
(tsvector.c:514-517), but only sorts (tsvector.c:563); equal entries
stay. The result is a tsvector with repeated lexemes, which no other
constructor produces, and which silently changes on a text round trip.
2. The first position is never checked. tsvectorin() rejects position 0
("wrong position info in tsvector", tsvector_parser.c:334); the receive
loop only checks that positions ascend for j > 0 (tsvector.c:540-545),
so a leading position 0 is accepted.
Self-contained reproduction (bash + psql; the files are COPY BINARY files
with one tsvector column):
psql -c "CREATE TABLE tv (v tsvector)"
# one row: tsvector with 2 entries, 'a' (no positions), 'a' (no
positions)
printf
'PGCOPY\n\377\r\n\0\0\0\0\0\0\0\0\0\0\001\0\0\0\014\0\0\0\002a\0\0\0a\0\0\0\377\377'
> dup.bin
# one row: tsvector with 1 entry, 'a' with one position, 0
printf
'PGCOPY\n\377\r\n\0\0\0\0\0\0\0\0\0\0\001\0\0\0\012\0\0\0\001a\0\0\001\0\0\377\377'
> pos0.bin
psql -c "\copy tv from 'dup.bin' with (format binary)" -- COPY 1
psql -c "\copy tv from 'pos0.bin' with (format binary)" -- COPY 1
SELECT v, length(v), pg_input_is_valid(v::text, 'tsvector') FROM tv;
v | length | pg_input_is_valid
---------+--------+-------------------
'a':0 | 1 | f
'a' 'a' | 2 | t
SELECT v, length(v), v::text::tsvector AS reparsed,
length(v::text::tsvector), v = v::text::tsvector AS same
FROM tv WHERE length(v) = 2;
v | length | reparsed | length | same
---------+--------+----------+--------+------
'a' 'a' | 2 | 'a' | 1 | f
SELECT v::text::tsvector FROM tv WHERE length(v) = 1;
ERROR: wrong position info in tsvector: "'a':0"
Consequences through pg_dump (plain format, psql restore), from the
attached transcript:
duplicates: 'a' 'a' -> 'a', 'a':1 'a':2 -> 'a':1,2, 'a' 'a' 'b' -> 'a'
'b';
restore reports no error and length(v) changes
position 0: ERROR: wrong position info in tsvector: "'a':0";
0 of 2 rows of that table restored
Expected: tsvectorrecv() rejects (or normalises, as tsvectorin() does for
duplicates) values that tsvectorin() would not produce, as its own comment
says it should.
Actual: both are accepted; the duplicate case later changes silently on a
text dump/restore, and the position-0 case makes the dump unrestorable.
Suggested fix:
- after the qsort in tsvectorrecv(), merge adjacent equal entries the way
uniqueentry() does (merging and de-duplicating their position lists),
or reject them with ERRCODE_INVALID_BINARY_REPRESENTATION;
- in the position loop, also reject WEP_GETPOS(wepptr[0]) == 0.
The existing elog(ERROR, ...) calls in tsvectorrecv() could at the same
time become ereport(ERROR, errcode(ERRCODE_INVALID_BINARY_REPRESENTATION),
...), since they are reachable from client input (COPY ... FORMAT binary,
binary-format parameters).
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: BUG #19748: tsvectorrecv accepts duplicate lexemes and position 0 that tsvectorin rejects
2026-10-04 05:19 BUG #19748: tsvectorrecv accepts duplicate lexemes and position 0 that tsvectorin rejects PG Bug reporting form <noreply@postgresql.org>
@ 2026-10-05 18:48 ` Tom Lane <tgl@sss.pgh.pa.us>
2026-10-06 21:27 ` Re: BUG #19748: tsvectorrecv accepts duplicate lexemes and position 0 that tsvectorin rejects Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 3+ messages in thread
From: Tom Lane @ 2026-10-05 18:48 UTC (permalink / raw)
To: kehan5800@gmail.com; +Cc: pgsql-bugs@lists.postgresql.org
PG Bug reporting form <noreply@postgresql.org> writes:
> tsvectorrecv() (src/backend/utils/adt/tsvector.c:446) has this comment
> (tsvector.c:552-556):
> * Enforce that datalen is still within MAXSTRPOS, ie the last lexeme
> * didn't go past that. We could allow that, since no "pos" field
> * overflowed, but tsvectorrecv shouldn't accept values that other
> * tsvector-constructing routines wouldn't.
> Two checks are still missing relative to tsvectorin():
> 1. Duplicate lexemes are not merged. tsvectorin() runs uniqueentry()
> (tsvector.c:265). tsvectorrecv() detects "compareentry(...) <= 0"
> (tsvector.c:514-517), but only sorts (tsvector.c:563); equal entries
> stay. The result is a tsvector with repeated lexemes, which no other
> constructor produces, and which silently changes on a text round trip.
Hmph. Making that happen would be a lot of mess, since that'd require
also merging and de-duping positions for equal lexemes, and the de-dup
logic used by tsvectorin() can't be re-used because it doesn't work on
the same data representation. But it seems to me that tsvectorrecv()
is being quite schizophrenic here: it will let you be sloppy about
lexeme order, but not about duplicated or out-of-order positions.
I don't see a good reason for it to be forgiving about either thing.
So my proposal is to just throw an error if the lexeme sequence isn't
strictly increasing, as it already does for positions. That is
probably not something to back-patch, but this issue is surely not
important enough to worry about back-patching.
> 2. The first position is never checked. tsvectorin() rejects position 0
> ("wrong position info in tsvector", tsvector_parser.c:334); the receive
> loop only checks that positions ascend for j > 0 (tsvector.c:540-545),
> so a leading position 0 is accepted.
Agreed, that's an oversight.
regards, tom lane
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: BUG #19748: tsvectorrecv accepts duplicate lexemes and position 0 that tsvectorin rejects
2026-10-04 05:19 BUG #19748: tsvectorrecv accepts duplicate lexemes and position 0 that tsvectorin rejects PG Bug reporting form <noreply@postgresql.org>
2026-10-05 18:48 ` Re: BUG #19748: tsvectorrecv accepts duplicate lexemes and position 0 that tsvectorin rejects Tom Lane <tgl@sss.pgh.pa.us>
@ 2026-10-06 21:27 ` Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 3+ messages in thread
From: Tom Lane @ 2026-10-06 21:27 UTC (permalink / raw)
To: kehan5800@gmail.com; +Cc: pgsql-bugs@lists.postgresql.org
I wrote:
> PG Bug reporting form <noreply@postgresql.org> writes:
>> Two checks are still missing relative to tsvectorin():
>> 1. Duplicate lexemes are not merged.
> So my proposal is to just throw an error if the lexeme sequence isn't
> strictly increasing, as it already does for positions. That is
> probably not something to back-patch, but this issue is surely not
> important enough to worry about back-patching.
>> 2. ... a leading position 0 is accepted.
> Agreed, that's an oversight.
Here's what that would look like.
regards, tom lane
Attachments:
v1-0001-Tighten-data-validity-checks-in-tsvectorrecv.patch (text/x-diff, 3.5K) download
From 3c32c2cefc572489300d6fab5690e76a0e76cfd4 Mon Sep 17 00:00:00 2001
From: Tom Lane <tgl@sss.pgh.pa.us>
Date: Tue, 6 Oct 2026 17:23:45 -0400
Subject: [PATCH v1] Tighten data validity checks in tsvectorrecv().
Insist that the input lexemes be already sorted and non-duplicate.
There's no good reason for them not to be, since the required ordering
is simple and platform-independent. The previous coding was willing
to re-sort them, but that left a gap: it didn't de-duplicate them.
De-duplicating would potentially require merging position lists,
which would be painful; we can't easily re-use tsvectorin()'s logic
for that, since it uses a different temporary representation.
Let's just dodge the problem by insisting the data be fully correct
as-presented, as we were already doing for the position lists.
Also, there was an oversight in the position-list checking: it did
not verify that the first position isn't zero.
While these problems allow tsvectorrecv() to construct tsvectors
that would not be generated internally, the vectors aren't so broken
that we can't work with them. Therefore, the best course of action
seems to be to apply these tighter rules in master but not back-patch.
Back-patching might break application workflows that aren't causing
any serious problems. Any such users will need to clean their data
before loading it into PG >= v20, but they'd face that issue anyway,
and a major update is a better time for it than a minor update.
Bug: #19748
Reported-by: Ke <kehan5800@gmail.com>
Author: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/19748-6075f0508fa1ffc0@postgresql.org
---
src/backend/utils/adt/tsvector.c | 17 +++++++++--------
1 file changed, 9 insertions(+), 8 deletions(-)
diff --git a/src/backend/utils/adt/tsvector.c b/src/backend/utils/adt/tsvector.c
index 21d29a7e4f6..2fe782374e7 100644
--- a/src/backend/utils/adt/tsvector.c
+++ b/src/backend/utils/adt/tsvector.c
@@ -454,7 +454,6 @@ tsvectorrecv(PG_FUNCTION_ARGS)
* WordEntries */
Size hdrlen;
Size len; /* allocated size of vec */
- bool needSort = false;
nentries = pq_getmsgint(buf, sizeof(int32));
@@ -512,16 +511,18 @@ tsvectorrecv(PG_FUNCTION_ARGS)
datalen += lex_len;
+ /* Verify correct sorting of lexemes, with no duplicates */
if (i > 0 && compareentry(&vec->entries[i],
&vec->entries[i - 1],
STRPTR(vec)) <= 0)
- needSort = true;
+ elog(ERROR, "tsvector lexemes are misordered");
/* Receive positions */
if (npos > 0)
{
uint16 j;
WordEntryPos *wepptr;
+ WordEntryPos lastpos = 0;
/*
* Pad to 2-byte alignment if necessary. Though we used palloc0
@@ -539,9 +540,13 @@ tsvectorrecv(PG_FUNCTION_ARGS)
wepptr = POSDATAPTR(vec, &vec->entries[i]);
for (j = 0; j < npos; j++)
{
- wepptr[j] = (WordEntryPos) pq_getmsgint(buf, sizeof(WordEntryPos));
- if (j > 0 && WEP_GETPOS(wepptr[j]) <= WEP_GETPOS(wepptr[j - 1]))
+ WordEntryPos thispos;
+
+ thispos = (WordEntryPos) pq_getmsgint(buf, sizeof(WordEntryPos));
+ /* Verify positions are sorted, nonduplicate, and not zero */
+ if (WEP_GETPOS(thispos) <= WEP_GETPOS(lastpos))
elog(ERROR, "position information is misordered");
+ wepptr[j] = lastpos = thispos;
}
datalen += sizeof(uint16) + npos * sizeof(WordEntryPos);
@@ -559,9 +564,5 @@ tsvectorrecv(PG_FUNCTION_ARGS)
SET_VARSIZE(vec, hdrlen + datalen);
- if (needSort)
- qsort_arg(ARRPTR(vec), vec->size, sizeof(WordEntry),
- compareentry, STRPTR(vec));
-
PG_RETURN_TSVECTOR(vec);
}
--
2.52.0
^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2026-10-06 21:27 UTC | newest]
Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-10-04 05:19 BUG #19748: tsvectorrecv accepts duplicate lexemes and position 0 that tsvectorin rejects PG Bug reporting form <noreply@postgresql.org>
2026-10-05 18:48 ` Tom Lane <tgl@sss.pgh.pa.us>
2026-10-06 21:27 ` Tom Lane <tgl@sss.pgh.pa.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox