Subject: Re: Testing of a fast method to bulk insert a Pandas DataFrame into
 Postgres
To: Shaozhong SHI <shishaozhong@gmail.com>
Cc: pgsql-general <pgsql-general@lists.postgresql.org>
References: 
 <CA+i5JwYnUBCKSnWn96OvtF66oGYHNGefMd5V_Bgceu=4PtcDNw@mail.gmail.com>
 <CA+i5JwY1jDx3jyBCkuJQhOyPELAoYVFtOmCqgxotTXmGc3wO9g@mail.gmail.com>
 <a6105435-6bfd-8776-fe38-13ffab752f23@aklaver.com>
 <CA+i5JwZkQJ2su4n14tg0Cyer5=ek17KnWhOYdiq-rnw9MRrt2Q@mail.gmail.com>
From: Adrian Klaver <adrian.klaver@aklaver.com>
Message-ID: <1b86e754-0317-84df-848f-9218e8df73e9@aklaver.com>
Date: Mon, 4 Oct 2021 09:50:52 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.7.0
MIME-Version: 1.0
In-Reply-To: 
 <CA+i5JwZkQJ2su4n14tg0Cyer5=ek17KnWhOYdiq-rnw9MRrt2Q@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
Archived-At: 
 <https://www.postgresql.org/message-id/1b86e754-0317-84df-848f-9218e8df73e9%40aklaver.com>
Precedence: bulk

On 10/4/21 9:20 AM, Shaozhong SHI wrote:
> Hello, Adrian Klaver,
> 
> Pandas version is 0.23.0.

The reason the below does not work is method did not show up until 
pandas 0.24.0.

> 
> I used the following code:
> 
> def psql_insert_copy(table, conn, keys, data_iter):
>      # gets a DBAPI connection that can provide a cursor
>      dbapi_conn = conn.connection
>      with dbapi_conn.cursor() as cur:
>          s_buf = StringIO()
>          writer = csv.writer(s_buf)
>          writer.writerows(data_iter)
>          s_buf.seek(0)
> 
>          columns = ', '.join('"{}"'.format(k) for k in keys)
>          if table.schema:
>              table_name = '{}.{}'.format(table.schema, table.name 
> <http://table.name>)
>          else:
>              table_name = table.name <http://table.name>
> 
>          sql = 'COPY {} ({}) FROM STDIN WITH CSV'.format(
>              table_name, columns)
>          cur.copy_expert(sql=sql, file=s_buf)
> engine = create_engine('postgresql+psycopg2://:5432/postgres')
> try:
>      df.to_sql('test1', engine, schema='public', if_exists='append', 
> index=False, method=psql_insert_copy)
> 
> I could not find obvious reasons.


> 
> Regards,
> 
> David
> 


-- 
Adrian Klaver
adrian.klaver@aklaver.com