agora inbox for pgsql-sql@postgresql.org
help / color / mirror / Atom feedFrom: Rob Sargent <robjsargent@gmail.com>
To: Shaozhong SHI <shishaozhong@gmail.com>
Cc: Adrian Klaver <adrian.klaver@aklaver.com>
Cc: pgsql-general <pgsql-general@lists.postgresql.org>
Subject: Re: Testing of a fast method to bulk insert a Pandas DataFrame into Postgres
Date: Mon, 4 Oct 2021 10:25:12 -0600
Message-ID: <3D84E92F-28DF-460C-90C7-160F41BE8B11@gmail.com> (raw)
In-Reply-To: <CA+i5JwZkQJ2su4n14tg0Cyer5=ek17KnWhOYdiq-rnw9MRrt2Q@mail.gmail.com>
References: <CA+i5JwYnUBCKSnWn96OvtF66oGYHNGefMd5V_Bgceu=4PtcDNw@mail.gmail.com>
<CA+i5JwY1jDx3jyBCkuJQhOyPELAoYVFtOmCqgxotTXmGc3wO9g@mail.gmail.com>
<a6105435-6bfd-8776-fe38-13ffab752f23@aklaver.com>
<CA+i5JwZkQJ2su4n14tg0Cyer5=ek17KnWhOYdiq-rnw9MRrt2Q@mail.gmail.com>
> On Oct 4, 2021, at 10:20 AM, Shaozhong SHI <shishaozhong@gmail.com> wrote:
>
> Hello, Adrian Klaver,
>
> Pandas version is 0.23.0.
>
> I used the following code:
>
> def psql_insert_copy(table, conn, keys, data_iter):
> # gets a DBAPI connection that can provide a cursor
> dbapi_conn = conn.connection
> with dbapi_conn.cursor() as cur:
> s_buf = StringIO()
> writer = csv.writer(s_buf)
> writer.writerows(data_iter)
> s_buf.seek(0)
>
> columns = ', '.join('"{}"'.format(k) for k in keys)
> if table.schema:
> table_name = '{}.{}'.format(table.schema, table.name <http://table.name/;)
> else:
> table_name = table.name <http://table.name/;
>
> sql = 'COPY {} ({}) FROM STDIN WITH CSV'.format(
> table_name, columns)
> cur.copy_expert(sql=sql, file=s_buf)
>
> engine = create_engine('postgresql+psycopg2://:5432/postgres')
> try:
> df.to_sql('test1', engine, schema='public', if_exists='append', index=False, method=psql_insert_copy)
you need to quote ‘psql_insert_copy'
view thread (12+ messages) latest in thread
Message-ID: <3D84E92F-28DF-460C-90C7-160F41BE8B11@gmail.com>
Permalink: ../3D84E92F-28DF-460C-90C7-160F41BE8B11@gmail.com/
Also on: postgresql.org/message-id/3D84E92F-28DF-460C-90C7-160F41BE8B11@gmail.com
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-sql@postgresql.org
Cc: robjsargent@gmail.com, shishaozhong@gmail.com, adrian.klaver@aklaver.com, pgsql-general@lists.postgresql.org
Subject: Re: Testing of a fast method to bulk insert a Pandas DataFrame into Postgres
In-Reply-To: <3D84E92F-28DF-460C-90C7-160F41BE8B11@gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox