From: John Naylor <john.naylor@postgresql.org>
To: pgsql-committers@lists.postgresql.org
Subject: pgsql: Add fast path for validating UTF-8 text
Date: Mon, 20 Dec 2021 14:10:07 +0000
Message-ID: <E1mzJMd-0007LH-5j@gemulon.postgresql.org> (raw)
Add fast path for validating UTF-8 text
Our previous validator used a traditional algorithm that performed
comparison and branching one byte at a time. It's useful in that
we always know exactly how many bytes we have validated, but that
precision comes at a cost. Input validation can show up prominently
in profiles of COPY FROM, and future improvements to COPY FROM such
as parallelism or faster line parsing will put more pressure on input
validation. Hence, add fast paths for both ASCII and multibyte UTF-8:
Use bitwise operations to check 16 bytes at a time for ASCII. If
that fails, use a "shift-based" DFA on those bytes to handle the
general case, including multibyte. These paths are relatively free
of branches and thus robust against all kinds of byte patterns. With
these algorithms, UTF-8 validation is several times faster, depending
on platform and the input byte distribution.
The previous coding in pg_utf8_verifystr() is retained for short
strings and for when the fast path returns an error.
Review, performance testing, and additional hacking by: Heikki
Linakangas, Vladimir Sitnikov, Amit Khandekar, Thomas Munro, and
Greg Stark
Discussion:
https://www.postgresql.org/message-id/CAFBsxsEV_SzH%2BOLyCiyon%3DiwggSyMh_eF6A3LU2tiWf3Cy2ZQg%40mail...
Branch
------
master
Details
-------
https://git.postgresql.org/pg/commitdiff/911588a3f816d875261d8f7d89e2517978831cd5
Modified Files
--------------
src/common/wchar.c | 215 +++++++++++++++++++++++++++++++
src/include/mb/pg_wchar.h | 53 ++++++++
src/test/regress/expected/conversion.out | 169 ++++++++++++++++++++++++
src/test/regress/sql/conversion.sql | 133 +++++++++++++++++++
4 files changed, 570 insertions(+)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-committers@postgresql.org
Cc: john.naylor@postgresql.org
Subject: Re: pgsql: Add fast path for validating UTF-8 text
In-Reply-To: <E1mzJMd-0007LH-5j@gemulon.postgresql.org>
References: <E1mzJMd-0007LH-5j@gemulon.postgresql.org>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox