agora inbox for pgsql-sql@postgresql.org
help / color / mirror / Atom feedFrom: Adrian Klaver <adrian.klaver@aklaver.com>
To: Shaozhong SHI <shishaozhong@gmail.com>
Cc: pgsql-general <pgsql-general@lists.postgresql.org>
Subject: Re: Testing of a fast method to bulk insert a Pandas DataFrame into Postgres
Date: Mon, 4 Oct 2021 09:50:52 -0700
Message-ID: <1b86e754-0317-84df-848f-9218e8df73e9@aklaver.com> (raw)
In-Reply-To: <CA+i5JwZkQJ2su4n14tg0Cyer5=ek17KnWhOYdiq-rnw9MRrt2Q@mail.gmail.com>
References: <CA+i5JwYnUBCKSnWn96OvtF66oGYHNGefMd5V_Bgceu=4PtcDNw@mail.gmail.com>
<CA+i5JwY1jDx3jyBCkuJQhOyPELAoYVFtOmCqgxotTXmGc3wO9g@mail.gmail.com>
<a6105435-6bfd-8776-fe38-13ffab752f23@aklaver.com>
<CA+i5JwZkQJ2su4n14tg0Cyer5=ek17KnWhOYdiq-rnw9MRrt2Q@mail.gmail.com>
On 10/4/21 9:20 AM, Shaozhong SHI wrote:
> Hello, Adrian Klaver,
>
> Pandas version is 0.23.0.
The reason the below does not work is method did not show up until
pandas 0.24.0.
>
> I used the following code:
>
> def psql_insert_copy(table, conn, keys, data_iter):
> # gets a DBAPI connection that can provide a cursor
> dbapi_conn = conn.connection
> with dbapi_conn.cursor() as cur:
> s_buf = StringIO()
> writer = csv.writer(s_buf)
> writer.writerows(data_iter)
> s_buf.seek(0)
>
> columns = ', '.join('"{}"'.format(k) for k in keys)
> if table.schema:
> table_name = '{}.{}'.format(table.schema, table.name
> <http://table.name;)
> else:
> table_name = table.name <http://table.name;
>
> sql = 'COPY {} ({}) FROM STDIN WITH CSV'.format(
> table_name, columns)
> cur.copy_expert(sql=sql, file=s_buf)
> engine = create_engine('postgresql+psycopg2://:5432/postgres')
> try:
> df.to_sql('test1', engine, schema='public', if_exists='append',
> index=False, method=psql_insert_copy)
>
> I could not find obvious reasons.
>
> Regards,
>
> David
>
--
Adrian Klaver
adrian.klaver@aklaver.com
view thread (12+ messages) latest in thread
Message-ID: <1b86e754-0317-84df-848f-9218e8df73e9@aklaver.com>
Permalink: ../1b86e754-0317-84df-848f-9218e8df73e9@aklaver.com/
Also on: postgresql.org/message-id/1b86e754-0317-84df-848f-9218e8df73e9@aklaver.com
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-sql@postgresql.org
Cc: adrian.klaver@aklaver.com, shishaozhong@gmail.com, pgsql-general@lists.postgresql.org
Subject: Re: Testing of a fast method to bulk insert a Pandas DataFrame into Postgres
In-Reply-To: <1b86e754-0317-84df-848f-9218e8df73e9@aklaver.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox