agora inbox for pgsql-hackers@postgresql.org
help / color / mirror / Atom feedFrom: Greg Burd <greg@burd.me>
To: Nazir Bilal Yavuz <byavuz81@gmail.com>
To: Nathan Bossart <nathandbossart@gmail.com>
Cc: Manni Wood <manni.wood@enterprisedb.com>
Cc: KAZAR Ayoub <ma_kazar@esi.dz>
Cc: Neil Conway <neil.conway@gmail.com>
Cc: Andrew Dunstan <andrew@dunslane.net>
Cc: Shinya Kato <shinya11.kato@gmail.com>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: Re: Speed up COPY FROM text/CSV parsing using SIMD
Date: Tue, 01 Sep 2026 00:41:20 -0400
Message-ID: <78e736c8-7f4e-46c2-9fcf-a6839e393d64@app.fastmail.com> (raw)
In-Reply-To: <CAN55FZ1utgaKb1DbW7tr+2=s4Hm1yiCEeNDY9m=qP_M_drafCg@mail.gmail.com>
References: <abHTvkeIK37hj9oS@nathan>
<CAN55FZ1sn-2tVX_n9C5UNBCfDPjSDOCT4zkMeDsB7MaZ9SUBTw@mail.gmail.com>
<abL5ylYjmopRRmvy@nathan>
<CAN55FZ0oYuVHZ=TdJofj0WA3mG77XumVsGJgG8gnav9kexr+wg@mail.gmail.com>
<CAN55FZ0ocS6cBHEWqHv2s-dK91U6OdVLBqj7VexTehtBtioDbA@mail.gmail.com>
<abQZmZ0KZesCxfR0@nathan>
<43de48dc-701b-4735-881b-50bca6870f39@app.fastmail.com>
<CAN55FZ1b1Jstrr-K7xcXY9z1ObDRxi48dAhwdHpTZn1C=S-jHA@mail.gmail.com>
<b26a1cf8-335d-4054-90ad-d94457c45947@app.fastmail.com>
<abRKUiInG6Gu728Q@nathan>
<apWOr1kgVSqweBnU@nathan>
<CAN55FZ1utgaKb1DbW7tr+2=s4Hm1yiCEeNDY9m=qP_M_drafCg@mail.gmail.com>
Hey Nathan, Nazir,
I've also reproduced the error and confirmed the patch addresses it
cleanly.
For anyone following along, the mechanism: commit e0a3a3fd53 added
CopyReadLineTextSIMDHelper, which refills the input buffer once fewer
than sizeof(Vector8) bytes remain:
if (copy_buf_len - input_buf_ptr < sizeof(Vector8)) /* reads ahead */
whereas scalar CopyReadLineText only refills when actually empty:
if (input_buf_ptr >= copy_buf_len || need_data)
That difference is the issue. CopyConvertBuf() deliberately tolerates an
invalid byte sequence after the \. end-of-copy marker (see the comment
near its convertedlen == 0 block), deferring the error. But
CopyLoadInputBuf() raises it the moment input_reached_error is set. The
SIMD read-ahead reaches that raise before the scalar loop has consumed
\., so a file ending in \.\n<valid bytes><invalid byte> errors out where
the scalar path would have stopped cleanly at \.
+1 on adding the reproducer as a test and calling it good.
best,
-greg
view thread (178+ messages) latest in thread
Message-ID: <78e736c8-7f4e-46c2-9fcf-a6839e393d64@app.fastmail.com>
Permalink: ../78e736c8-7f4e-46c2-9fcf-a6839e393d64@app.fastmail.com/
Also on: postgresql.org/message-id/78e736c8-7f4e-46c2-9fcf-a6839e393d64@app.fastmail.com
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: greg@burd.me, byavuz81@gmail.com, nathandbossart@gmail.com, manni.wood@enterprisedb.com, ma_kazar@esi.dz, neil.conway@gmail.com, andrew@dunslane.net, shinya11.kato@gmail.com
Subject: Re: Speed up COPY FROM text/CSV parsing using SIMD
In-Reply-To: <78e736c8-7f4e-46c2-9fcf-a6839e393d64@app.fastmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox