postgresql-interfaces/psqlodbc GitHub issues and pull requests (mirror)  

help / color / mirror / Atom feed
From: ezekieloyeniran1 (@ezekieloyeniran1) <[email protected]>
To: postgresql-interfaces/psqlodbc <[email protected]>
Subject: [postgresql-interfaces/psqlodbc] issue #96: Memory allocation error from PSQLODBC35W.DLL
Date: Mon, 10 Mar 2025 19:44:11 +0000
Message-ID: <[email protected]> (raw)

I am currently experiencing an out-of-memory (OOM) error while using Copy Activity in a Pipeline to transfer 100M+ records from an On-Prem PostgreSQL database to a Microsoft Fabric Lakehouse.
 
The pipeline works for smaller subsets (~1M records) with parallelism set to 10 but consistently fails when attempting to load over 1 million records.
 
The PostgreSQL source is connected via ODBC (psqlodbc35w.dll), and an On Prem Data Gateway (version 3000.246.5) with 4 logical processors.  
 
I would love an insight on knowing the limitation on the volume of data PSQLODBC35W.DLL driver can handle. 

Do you think the constraints is from the On-Prem Data Gateway to transfer large data via Microsoft On-Prem Data Gateway? 

Are there best practices for optimizing PostgreSQL reads to prevent failures?

Appreciate any insights!


Here is the error I got:
"```
_{
	"errorCode": 2200,
	"message": "Failure happened on 'Source' side. ErrorCode=UserErrorOdbcOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ERROR [HY001] Out of memory while reading tuples.;\nmemory allocation error???,Source=Microsoft.DataTransfer.Runtime.GenericOdbcConnectors,''Type=System.Data.Odbc.OdbcException,Message=ERROR [HY001] Out of memory while reading tuples.;\nmemory allocation error???,Source=PSQLODBC35W.DLL,'",
	"details": []
}_"
```

view thread (3+ messages)  latest in thread

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: github://postgresql-interfaces/psqlodbc
  Cc: [email protected], [email protected]
  Subject: Re: [postgresql-interfaces/psqlodbc] issue #96: Memory allocation error from PSQLODBC35W.DLL
  In-Reply-To: <<[email protected]>>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox