pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedpsql include text file with bom
2+ messages / 2 participants
[nested] [flat]
* psql include text file with bom
@ 2023-11-26 05:46 Rick Parrish <ai5jt@unitrunker.net>
0 siblings, 1 reply; 2+ messages in thread
From: Rick Parrish @ 2023-11-26 05:46 UTC (permalink / raw)
To: bugs <pgsql-bugs@lists.postgresql.org>
*Summary:*
psql "include" or "\i" command chokes on UTF8 text files prefixed with BOM.
*Steps to reproduce:*
1. create a UTF8 file with three byte BOM 'EF BB BF'.
2. include the file from psql via the "include" or \i command.
Example output for file named "test.sql" below:
redacted-# \i test.sql
psql:test.sql:1: ERROR: syntax error at or near ""
LINE 1: 
*Background*
https://en.wikipedia.org/wiki/Byte_order_mark
Some text editors save text to a file prefixed by a BOM or byte marker.
This includes Visual Studio, VSCode and others.
I think it would be reasonable for the include command to skip over any
BOM found in the first two or three bytes of a file.
^ permalink raw reply [nested|flat] 2+ messages in thread
* Re: psql include text file with bom
@ 2023-11-27 14:49 Tom Lane <tgl@sss.pgh.pa.us>
parent: Rick Parrish <ai5jt@unitrunker.net>
0 siblings, 0 replies; 2+ messages in thread
From: Tom Lane @ 2023-11-27 14:49 UTC (permalink / raw)
To: Rick Parrish <ai5jt@unitrunker.net>; +Cc: bugs <pgsql-bugs@lists.postgresql.org>
Rick Parrish <ai5jt@unitrunker.net> writes:
> I think it would be reasonable for the include command to skip over any
> BOM found in the first two or three bytes of a file.
This has been proposed before, and rejected before. psql has no
inherent knowledge of what encoding an input file is in, and therefore
no justification to assume that a bit-pattern it sees there is a BOM.
In non-UTF8 encodings it could very easily be valid data.
(For that matter, it's also valid data in UTF8: it's the same bit
pattern as U+FEFF ZERO WIDTH NO-BREAK SPACE. Programs that emit
one into UTF8 streams, and expect it not to be taken as data,
are frankly broken.)
regards, tom lane
^ permalink raw reply [nested|flat] 2+ messages in thread
end of thread, other threads:[~2023-11-27 14:49 UTC | newest]
Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2023-11-26 05:46 psql include text file with bom Rick Parrish <ai5jt@unitrunker.net>
2023-11-27 14:49 ` Tom Lane <tgl@sss.pgh.pa.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox