Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1vycgT-000Gj5-0a for pgsql-hackers@arkaria.postgresql.org; Fri, 06 Mar 2026 21:26:09 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.96) (envelope-from ) id 1vycgQ-0072mj-1J for pgsql-hackers@arkaria.postgresql.org; Fri, 06 Mar 2026 21:26:06 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1vycgQ-0072mb-09 for pgsql-hackers@lists.postgresql.org; Fri, 06 Mar 2026 21:26:06 +0000 Received: from mail-dy1-x132c.google.com ([2607:f8b0:4864:20::132c]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.98.2) (envelope-from ) id 1vycgL-00000001FKX-1Z7Z for pgsql-hackers@postgresql.org; Fri, 06 Mar 2026 21:26:05 +0000 Received: by mail-dy1-x132c.google.com with SMTP id 5a478bee46e88-2b4520f6b32so11096266eec.0 for ; Fri, 06 Mar 2026 13:26:00 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1772832359; cv=none; d=google.com; s=arc-20240605; b=YRg8YrzWljPGnS7p6Z18YeFc3TQ7MJEzhHJq6PfJKIpwCtGqcBtwGsEcRshDJ/tDQh nZHPnPMQqizapvftSBR2MoleBa6g2yEBjxMio2BM4VCI2fJ4abOJySydBB5puzAlRScr Y9K3CA65U4rC/ZX4lSDyylGfu2ElmBDN+cFGXeCPuzFUWvLXzg1UkelSMzd2m82B66o1 DCZ5/Cahk6y1CDlP3LAec2ztp7wpo1tRXOvaVaMgOJ2EmhC1lDewgQ8gzXybhOO/SWha F8nyPcdZeyUFa7fs+/KzavE26QK1R7OG0B+ricE+CX0QjY2TvhfpYN5qvOf1R58CVyPY azHw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:dkim-signature; bh=CzYPtiXpC8weli6aGhKamcdAzu5Qj30kChqWE2zLqi8=; fh=gpLPB8DtmG9ZQS2GpOzXITVdMOxTo7fLmx7jDZOIE6g=; b=KqOjZbTh/8B0nfvhmuczE/gClan4XFK5ShEBxl5zy75GHeBbVOp0jmEfi9BuZGgxjK aJxRnxP0h03z4Hj/sBijEE4ymkv/Sr4vCH7FCelh7c1lzTgTG+5eW+lA5MwsX/qrzQ/J 0yMxGuaeOzqB9XykB4zkaTCSwEGD+3QtaDJ9qXpmngtlaXwpSIohFjQxCU0ABCnC9x7g t6ESvM4ymtmhVdZDM1azXUj9/IWSOmBhWoRL4bgTtr64oR72MRLRw/A3YqgQA7QjOnih jTmKTHElsqrPKvYuaFzKvsOzkb8nOhV1aAiiZNYdCqK9x1cS+iXMNyZimrEd9gxDyH6s Mmvw==; darn=postgresql.org ARC-Authentication-Results: i=1; mx.google.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=enterprisedb.com; s=google; t=1772832359; x=1773437159; darn=postgresql.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=CzYPtiXpC8weli6aGhKamcdAzu5Qj30kChqWE2zLqi8=; b=bZ2rdnWXF8pMm5gt0b9LLWI7ezJFgPfn4gbiVf5DD22MNpE7YmoOEEUniAnujsmzVR bQioddOatCrYVlmeB9uVWQrKA+4Le4TydJhBZs5ozZSzUMKWJag3qTnUS02N9ynO4k9H qsknhJuUHUhXeZJf34US+xO5u4PBz0G4j1Dfpj80eEatTq59PTPWc3kZoZAoo1SYUDze WZ+V6MOkGL4ZtDJSeeG+NVRZtGk79M4rzpwKcXo2hSvarBjEW5+wEdslOPAoj5Re60cz N6eJO7UnQSxGDm4r09mahggA6khesL7dNjIBsr1Cu1O4E5yBzQYdA73U+JLT9NZadb79 qU8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772832359; x=1773437159; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=CzYPtiXpC8weli6aGhKamcdAzu5Qj30kChqWE2zLqi8=; b=SkI8JRxmVwn6WsDk2ehYJKZx/V4IYskPmmQNDUDHnInMsZoBXPOujbgL0xe8+gCX8q nKTSROL7UWiT7BgLYLbqABK09MujcspI7Ciwoum0mXHg4hQezSLlmrrfTDnm+iA4Al1A Ve1gnh+siq2o+twvNhGQ7CCDVZ80MRqoFmFJ7Qz1VOXfdlQ8RGIwjBHtc/jLERGS9CLO mjkoEd4prcDdvl7fxFKRur5suEGYPF6PnQOL2ZN4CBu+RM7p1JF3+vpq5J6JN4xSouyB eiKZewFqyeGJzUHumgN/NGzOo6keaREgsT2tNke8TmPrZsqTuWx3qvFY4T0K3GonUSBw ewrg== X-Forwarded-Encrypted: i=1; AJvYcCURbp0yldI3EMvhgTCp+zdW3uw5MpdvtY+Yj5NaEaiu1ppQ99ND+BTUEvBqhk99q2dK0YHTUEGqRvNaepKb@postgresql.org X-Gm-Message-State: AOJu0YxPhs8JSBFoHXcznsDfQxxoarMnqXsWGZcA3HfOgkMeBxN7HMYw p6N1Q7zdWnjwkx0McYd9k94B0XMKRBjr/Fq7DbWPIdxTE5xZMTE2liVSfEXdbj/iwqGJ+CXt6fX QLoD6V1eHJd5Ah/9N+X5Xq9Vaoq/oOuOf7I09E3E4 X-Gm-Gg: ATEYQzwirX773ORihdXh+6X/F+m4WOcUoSfa6fBSFhn1j2yKbMg3GLkk5WGTnjzIxqY x9HZ26gf5XfPCJC/2w3QuzbRIg7rfO/3mx+TufR4HunGWkxFk8f8WGSvKV2scyBwYAxXFDVfRV/ OoXUz80BShp2RvLJoiFAh2Lmwl32vsFyI6LCJqzqMT8xDXOdjP+4CcCxmzgs/VWVA/7WidaX0i5 opkF9LrMu4f9VzF5/+LDi77ssusqILPMkjsiWxnXozz4Q5yafEQ/p6nK2e5GSvuAJhQ4Rtl7mcK 4jfD6aM9 X-Received: by 2002:a05:7300:1489:b0:2b8:6ae5:79c3 with SMTP id 5a478bee46e88-2be4e090105mr1381488eec.38.1772832358686; Fri, 06 Mar 2026 13:25:58 -0800 (PST) MIME-Version: 1.0 References: <91acb778-42c4-44ef-8888-f18ad9b12a5b@dunslane.net> In-Reply-To: From: Manni Wood Date: Fri, 6 Mar 2026 15:25:46 -0600 X-Gm-Features: AaiRm53p-utYWedgmJso5aUvPyPn1gKyJwbBxG_mRdDTWzsjrmpqUFSEgj7_mAY Message-ID: Subject: Re: Speed up COPY FROM text/CSV parsing using SIMD To: Nazir Bilal Yavuz Cc: Andrew Dunstan , Nathan Bossart , KAZAR Ayoub , Neil Conway , Shinya Kato , PostgreSQL-development Content-Type: multipart/alternative; boundary="000000000000fc712b064c61b24e" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --000000000000fc712b064c61b24e Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Fri, Mar 6, 2026 at 12:55=E2=80=AFPM Nazir Bilal Yavuz wrote: > Hi Manni! > > On Fri, 6 Mar 2026 at 21:13, Manni Wood > wrote: > > > > When I ran `meson setup build --buildtype=3Ddebugoptimized` on both my = x86 > machine and my arm machine, the response on both was: > > > > "External libraries > > " lz4 : NO" > > > > However, I did not remove commit 34dfca2934 from any of my Postgres > builds; I left that commit in place. > > > > Let me know if that helps! > > That definitely helps, thanks! If you have a chance, could you also > run the benchmark with LZ4 enabled? I expect you may see significantly > better performance, similar to what I observed. > > -- > Regards, > Nazir Bilal Yavuz > Microsoft > Hi, Nazir. Well, golly! Look at these numbers. Old master with no lz4, your v11 patch with no lz4, and then your v11 patch with lz4 compiled in. (Aside: I assume everything is good for my lz4 build after installing the lz4 dev library and seeing this with my meson config: " External libraries " lz4 : YES 1.9.4" and this from the db itself: $ ./bin/psql -U mwood -d postgres -c 'show default_toast_compression' default_toast_compression --------------------------- lz4 ) x86 NARROW old master TXT : 25909.060500 ms CSV : 28137.591250 ms TXT with 1/3 escapes: 27794.177000 ms CSV with 1/3 quotes: 34541.704750 ms x86 NARROW v11 TXT : 27212.945750 ms -5.032545% regression CSV : 26985.971250 ms 4.092817% improvement TXT with 1/3 escapes: 27216.510000 ms 2.078374% improvement CSV with 1/3 quotes: 32817.267500 ms 4.992334% improvement x86 NARROW v11 lz4 TXT : 26471.776500 ms -2.171889% regression CSV : 25607.026250 ms 8.993538% improvement TXT with 1/3 escapes: 28628.729750 ms -3.002617% regression CSV with 1/3 quotes: 34729.006750 ms -0.542249% regression x86 WIDE old master TXT : 28778.426500 ms CSV : 35671.908000 ms TXT with 1/3 escapes: 32441.549750 ms CSV with 1/3 quotes: 47024.416000 ms x86 WIDE v11 TXT : 22571.305750 ms 21.568659% improvement CSV : 22711.524750 ms 36.332184% improvement TXT with 1/3 escapes: 29236.453000 ms 9.879604% improvement CSV with 1/3 quotes: 40022.110750 ms 14.890786% improvement x86 WIDE v11 lz4 TXT : 8032.912750 ms 72.087033% improvement CSV : 8047.098000 ms 77.441358% improvement TXT with 1/3 escapes: 15428.139500 ms 52.443272% improvement CSV with 1/3 quotes: 27517.084500 ms 41.483410% improvement arm NARROW old master TXT : 10997.568250 ms CSV : 10797.549000 ms TXT with 1/3 escapes: 10299.047000 ms CSV with 1/3 quotes: 12559.385750 ms arm NARROW v11 TXT : 10340.966250 ms 5.970429% improvement CSV : 10224.399500 ms 5.308144% improvement TXT with 1/3 escapes: 10438.216750 ms -1.351288% regression CSV with 1/3 quotes: 11865.934000 ms 5.521383% improvement arm NARROW v11 lz4 TXT : 9783.737000 ms 11.037270% improvement CSV : 10122.890750 ms 6.248254% improvement TXT with 1/3 escapes: 10298.780250 ms 0.002590% improvement CSV with 1/3 quotes: 11738.992250 ms 6.532115% improvement arm WIDE old master TXT : 11825.771250 ms CSV : 13907.074000 ms TXT with 1/3 escapes: 13430.691250 ms CSV with 1/3 quotes: 17557.954500 ms arm WIDE v11 TXT : 9001.442250 ms 23.882831% improvement CSV : 8940.928750 ms 35.709490% improvement TXT with 1/3 escapes: 12049.668500 ms 10.282589% improvement CSV with 1/3 quotes: 15277.843250 ms 12.986201% improvement arm WIDE v11 lz4 TXT : 3186.825500 ms 73.051859% improvement CSV : 3142.526500 ms 77.403396% improvement TXT with 1/3 escapes: 6180.176000 ms 53.984677% improvement CSV with 1/3 quotes: 9460.505500 ms 46.118407% improvement Cheers, -Manni --=20 -- Manni Wood EDB: https://www.enterprisedb.com --000000000000fc712b064c61b24e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Fri, Mar 6, = 2026 at 12:55=E2=80=AFPM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
Hi Manni!

On Fri, 6 Mar 2026 at 21:13, Manni Wood <manni.wood@enterprisedb.com> wrote= :
>
> When I ran `meson setup build --buildtype=3Ddebugoptimized` on both my= x86 machine and my arm machine, the response on both was:
>
> "External libraries
> "=C2=A0 lz4=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 : NO"
>
> However, I did not remove commit 34dfca2934 from any of my Postgres bu= ilds; I left that commit in place.
>
> Let me know if that helps!

That definitely helps, thanks! If you have a chance, could you also
run the benchmark with LZ4 enabled? I expect you may see significantly
better performance, similar to what I observed.

--
Regards,
Nazir Bilal Yavuz
Microsoft

Hi, Nazir.
<= br>
Well, golly! Look at these numbers. Old master with no lz4, y= our v11 patch with no lz4, and then your v11 patch with lz4 compiled in.

(Aside: I assume everything is good for my lz4 build= after installing the lz4 dev library and seeing this with my meson config:=

"=C2=A0 External libraries
"=C2=A0= =C2=A0 lz4 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0: YES 1.9.4"

and this from the db itse= lf:

$ ./bin/psql -U mwood -d postgres -c 'show= default_toast_compression'
=C2=A0default_toast_compression
----= -----------------------
=C2=A0lz4
)

x= 86 NARROW old master
TXT : =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 25909.060500 ms
CSV : =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 28137.591250 ms
TXT with 1/3 escapes: 27794.177000 ms<= br>CSV with 1/3 quotes: =C2=A034541.704750 ms

x86 NARROW v11
TXT = : =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 27212.945750 ms = =C2=A0-5.032545% regression
CSV : =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 26985.971250 ms =C2=A04.092817% improvement
TXT with 1= /3 escapes: 27216.510000 ms =C2=A02.078374% improvement
CSV with 1/3 quo= tes: =C2=A032817.267500 ms =C2=A04.992334% improvement

x86 NARROW v1= 1 lz4
TXT : =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 2647= 1.776500 ms =C2=A0-2.171889% regression
CSV : =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 25607.026250 ms =C2=A08.993538% improvement=
TXT with 1/3 escapes: 28628.729750 ms =C2=A0-3.002617% regression
CS= V with 1/3 quotes: =C2=A034729.006750 ms =C2=A0-0.542249% regression

x86 WIDE old master
TXT : =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 28778.426500 ms
CSV : =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 35671.908000 ms
TXT with 1/3 escapes: 32441.549750 = ms
CSV with 1/3 quotes: =C2=A047024.416000 ms

x86 WIDE v11
TXT= : =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 22571.305750 ms = =C2=A021.568659% improvement
CSV : =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 22711.524750 ms =C2=A036.332184% improvement
TXT wi= th 1/3 escapes: 29236.453000 ms =C2=A09.879604% improvement
CSV with 1/3= quotes: =C2=A040022.110750 ms =C2=A014.890786% improvement

x86 WIDE= v11 lz4
TXT : =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 8= 032.912750 ms =C2=A072.087033% improvement
CSV : =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 8047.098000 ms =C2=A077.441358% improvem= ent
TXT with 1/3 escapes: 15428.139500 ms =C2=A052.443272% improvementCSV with 1/3 quotes: =C2=A027517.084500 ms =C2=A041.483410% improvement


arm NARROW old master
TXT : =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 10997.568250 ms
CSV : =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 10797.549000 ms
TXT with 1/3 escapes:= 10299.047000 ms
CSV with 1/3 quotes: =C2=A012559.385750 ms

arm N= ARROW v11
TXT : =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = 10340.966250 ms =C2=A05.970429% improvement
CSV : =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 10224.399500 ms =C2=A05.308144% improvem= ent
TXT with 1/3 escapes: 10438.216750 ms =C2=A0-1.351288% regressionCSV with 1/3 quotes: =C2=A011865.934000 ms =C2=A05.521383% improvement
=
arm NARROW v11 lz4
TXT : =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 9783.737000 ms =C2=A011.037270% improvement
CSV : =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 10122.890750 ms =C2=A06.24= 8254% improvement
TXT with 1/3 escapes: 10298.780250 ms =C2=A00.002590% = improvement
CSV with 1/3 quotes: =C2=A011738.992250 ms =C2=A06.532115% i= mprovement


arm WIDE old master
TXT : =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 11825.771250 ms
CSV : =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 13907.074000 ms
TXT with 1/3 esca= pes: 13430.691250 ms
CSV with 1/3 quotes: =C2=A017557.954500 ms

a= rm WIDE v11
TXT : =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 9001.442250 ms =C2=A023.882831% improvement
CSV : =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 8940.928750 ms =C2=A035.709490% impr= ovement
TXT with 1/3 escapes: 12049.668500 ms =C2=A010.282589% improveme= nt
CSV with 1/3 quotes: =C2=A015277.843250 ms =C2=A012.986201% improveme= nt

arm WIDE v11 lz4
TXT : =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 3186.825500 ms =C2=A073.051859% improvement
CSV : =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 3142.526500 ms =C2=A07= 7.403396% improvement
TXT with 1/3 escapes: 6180.176000 ms =C2=A053.9846= 77% improvement
CSV with 1/3 quotes: =C2=A09460.505500 ms =C2=A046.11840= 7% improvement

Cheers,

-M= anni

--
-- Manni Wood EDB: https://www.enterprisedb.com
--000000000000fc712b064c61b24e--