Subject: Data transfer from PG to SQL Server

From: Paul van Rixel <[email-masked]>
Date: Wed, 9 Jun 2021 14:48:58 +0200

Hi all,

I'm looking for the best way to transfer data on a weekly basis to a SQL
Server Instance which will be the staging area for BI reporting. A SQL
Server Linked Server configuration is because of security issues not
allowed so now I need some help!

What I did previously was installing the  Postgres ODBC Driver 12_02
version, configuration is ok because a test is successful.  So next step
was to configure a linked server and in that way I can see the tables. To
get the data from straigtforwarded tables without JSON or XML columns I
used openquery from SQL Server. That works for 80% of the tables. I also
see in the Postgresql logfiles that I'm connected and read that table.
Large tables (> 3GB) however did not transfer!
Also tried with Export/Import wizard from SQL Server Managememnt Studio but
also from SSIS. Both failed.

So my simple question is how to transfer Postgresql data/tables to SQL
Server without Linked Server or other paid tools like pgOledb? I could not
find answer so that's why I'm here!

Any help would be appreciated!

Best regards,

Paul van Rixel
DBA

---

From: David Parenteau <[email-masked]>
Date: Wed, 9 Jun 2021 13:53:55 +0000

Maybe you could use a Scheduled Powershell script to get your data using PostGres ODBC and save them in SQL Server using the Bulk Copy object? Otherwise, getting data to json/csv files and then BULK INSERT these file sinto SQL Server might also be wuick based on my experience.

Ex:
$bc = new-object ("System.Data.SqlClient.SqlBulkCopy") $cn

NOTE: On my side, to get my ODBC long calls to work I had to set the tcp timeout managed by pgbouncer to 120 instead of default.

HTH.

David

________________________________
From: Paul van Rixel <[email-masked]>
Sent: Wednesday, June 9, 2021 8:48 AM
To: [email-masked] <[email-masked]>
Subject: Data transfer from PG to SQL Server

Hi all,

I'm looking for the best way to transfer data on a weekly basis to a SQL Server Instance which will be the staging area for BI reporting. A SQL Server Linked Server configuration is because of security issues not allowed so now I need some help!

What I did previously was installing the  Postgres ODBC Driver 12_02 version, configuration is ok because a test is successful.  So next step was to configure a linked server and in that way I can see the tables. To get the data from straigtforwarded tables without JSON or XML columns I used openquery from SQL Server. That works for 80% of the tables. I also see in the Postgresql logfiles that I'm connected and read that table.
Large tables (> 3GB) however did not transfer!
Also tried with Export/Import wizard from SQL Server Managememnt Studio but also from SSIS. Both failed.

So my simple question is how to transfer Postgresql data/tables to SQL Server without Linked Server or other paid tools like pgOledb? I could not find answer so that's why I'm here!

Best regards,

Paul van Rixel
DBA