Feedback-ID: i675e48f3:Fastmail
MIME-Version: 1.0
Date: Tue, 01 Sep 2026 00:41:20 -0400
From: "Greg Burd" <greg@burd.me>
To: "Nazir Bilal Yavuz" <byavuz81@gmail.com>,
 "Nathan Bossart" <nathandbossart@gmail.com>
Cc: "Manni Wood" <manni.wood@enterprisedb.com>,
 "KAZAR Ayoub" <ma_kazar@esi.dz>, "Neil Conway" <neil.conway@gmail.com>,
 "Andrew Dunstan" <andrew@dunslane.net>,
 "Shinya Kato" <shinya11.kato@gmail.com>,
 pgsql-hackers <pgsql-hackers@postgresql.org>
Message-Id: <78e736c8-7f4e-46c2-9fcf-a6839e393d64@app.fastmail.com>
In-Reply-To: 
 <CAN55FZ1utgaKb1DbW7tr+2=s4Hm1yiCEeNDY9m=qP_M_drafCg@mail.gmail.com>
References: <abHTvkeIK37hj9oS@nathan>
 <CAN55FZ1sn-2tVX_n9C5UNBCfDPjSDOCT4zkMeDsB7MaZ9SUBTw@mail.gmail.com>
 <abL5ylYjmopRRmvy@nathan>
 <CAN55FZ0oYuVHZ=TdJofj0WA3mG77XumVsGJgG8gnav9kexr+wg@mail.gmail.com>
 <CAN55FZ0ocS6cBHEWqHv2s-dK91U6OdVLBqj7VexTehtBtioDbA@mail.gmail.com>
 <abQZmZ0KZesCxfR0@nathan>
 <43de48dc-701b-4735-881b-50bca6870f39@app.fastmail.com>
 <CAN55FZ1b1Jstrr-K7xcXY9z1ObDRxi48dAhwdHpTZn1C=S-jHA@mail.gmail.com>
 <b26a1cf8-335d-4054-90ad-d94457c45947@app.fastmail.com>
 <abRKUiInG6Gu728Q@nathan> <apWOr1kgVSqweBnU@nathan>
 <CAN55FZ1utgaKb1DbW7tr+2=s4Hm1yiCEeNDY9m=qP_M_drafCg@mail.gmail.com>
Subject: Re: Speed up COPY FROM text/CSV parsing using SIMD
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Archived-At: 
 <https://www.postgresql.org/message-id/78e736c8-7f4e-46c2-9fcf-a6839e393d64%40app.fastmail.com>
Precedence: bulk

Hey Nathan, Nazir,

I've also reproduced the error and confirmed the patch addresses it
cleanly.

For anyone following along, the mechanism: commit e0a3a3fd53 added
CopyReadLineTextSIMDHelper, which refills the input buffer once fewer
than sizeof(Vector8) bytes remain:

  if (copy_buf_len - input_buf_ptr < sizeof(Vector8))   /* reads ahead */

whereas scalar CopyReadLineText only refills when actually empty:

  if (input_buf_ptr >= copy_buf_len || need_data)

That difference is the issue. CopyConvertBuf() deliberately tolerates an
invalid byte sequence after the \. end-of-copy marker (see the comment
near its convertedlen == 0 block), deferring the error.  But
CopyLoadInputBuf() raises it the moment input_reached_error is set. The
SIMD read-ahead reaches that raise before the scalar loop has consumed
\., so a file ending in \.\n<valid bytes><invalid byte> errors out where
the scalar path would have stopped cleanly at \.

+1 on adding the reproducer as a test and calling it good.

best,

-greg