Message-ID: From: "ezekieloyeniran1 (@ezekieloyeniran1)" To: "postgresql-interfaces/psqlodbc" Date: Mon, 10 Mar 2025 19:44:11 +0000 Subject: [postgresql-interfaces/psqlodbc] issue #96: Memory allocation error from PSQLODBC35W.DLL List-Id: X-GitHub-Author-Id: 188894415 X-GitHub-Author-Login: ezekieloyeniran1 X-GitHub-Issue: 96 X-GitHub-Repo: postgresql-interfaces/psqlodbc X-GitHub-State: open X-GitHub-Type: issue X-GitHub-Url: https://github.com/postgresql-interfaces/psqlodbc/issues/96 Content-Type: text/plain; charset=utf-8 I am currently experiencing an out-of-memory (OOM) error while using Copy Activity in a Pipeline to transfer 100M+ records from an On-Prem PostgreSQL database to a Microsoft Fabric Lakehouse. The pipeline works for smaller subsets (~1M records) with parallelism set to 10 but consistently fails when attempting to load over 1 million records. The PostgreSQL source is connected via ODBC (psqlodbc35w.dll), and an On Prem Data Gateway (version 3000.246.5) with 4 logical processors. I would love an insight on knowing the limitation on the volume of data PSQLODBC35W.DLL driver can handle. Do you think the constraints is from the On-Prem Data Gateway to transfer large data via Microsoft On-Prem Data Gateway? Are there best practices for optimizing PostgreSQL reads to prevent failures? Appreciate any insights! Here is the error I got: "``` _{ "errorCode": 2200, "message": "Failure happened on 'Source' side. ErrorCode=UserErrorOdbcOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ERROR [HY001] Out of memory while reading tuples.;\nmemory allocation error???,Source=Microsoft.DataTransfer.Runtime.GenericOdbcConnectors,''Type=System.Data.Odbc.OdbcException,Message=ERROR [HY001] Out of memory while reading tuples.;\nmemory allocation error???,Source=PSQLODBC35W.DLL,'", "details": [] }_" ```