From: Ron <ronljohnsonjr@gmail.com>
To: pgsql-admin@lists.postgresql.org
Subject: Re: COPY from STDIN vs file with large CSVs
Date: Wed, 8 Jan 2020 11:08:56 -0600
Message-ID: <c2e587ae-d58d-8d21-0a21-78c323867d08@gmail.com> (raw)
In-Reply-To: <CAOC+FBVjk4W=nhn3FZUiTgroFFhaQ6bvh-NiU207PhD9nO3Qww@mail.gmail.com>
References: <CAOC+FBVjk4W=nhn3FZUiTgroFFhaQ6bvh-NiU207PhD9nO3Qww@mail.gmail.com>
On 1/8/20 10:54 AM, Wells Oliver wrote:
> I have a CSV that's ~30GB. Some 400m rows. Would there be a meaningful
> performance difference to run COPY from STDIN using: cat f.csv | psql
> "COPY .. FROM STDIN WITH CSV" versus just doing "COPY ... FROM 'f.csv'
> WITH CSV"?
>
> Thanks. It took about four hours to copy one and I felt that was a little
> much.
catting the file starts another process, and opens a pipe. That can't be
faster than "COPY ... FROM ... WITH CSV".
pg_bulkload (which might be in your repository) is probably what you really
want.
--
Angular momentum makes the world go 'round.
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-admin@postgresql.org
Cc: ronljohnsonjr@gmail.com, pgsql-admin@lists.postgresql.org
Subject: Re: COPY from STDIN vs file with large CSVs
In-Reply-To: <c2e587ae-d58d-8d21-0a21-78c323867d08@gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox