postgresql-interfaces/psqlodbc GitHub issues and pull requests (mirror)
help / color / mirror / Atom feedFrom: markmaker (@markmaker) <[email protected]>
To: postgresql-interfaces/psqlodbc <[email protected]>
Subject: Re: [postgresql-interfaces/psqlodbc] issue #96: Memory allocation error from PSQLODBC35W.DLL
Date: Wed, 21 May 2025 08:03:31 +0000
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
References: <[email protected]>
I had the same problem, and this is how I had to solve it:
1. Use the original query `where` expression etc. but only fetch the primary keys of the result set records. Assuming the primary key is some rather small ID data type, it will allow for very large result sets, even in constrained RAM.
2. Then select the actual records in batches of a comfortable size, i.e. `select ... where yourPrimaryKey in (?, ?, ?, ?, ...)`
This seems to work well, PostgreSQL does the batch fetching fast (as it happens, the overall performance is still faster than the original DB we used before). We even just prepare one batch fetching statement, and fill any surplus bound params with `null` for the last remainder fetch, and it doesn't seem to bother.
During the query processing, if you are updating or even deleting records further down the data set, be aware this will now be reflected in the batch-fetched records, unlike with the original full-fetch query. We actually wanted that, it solved a secondary problem for us.
_Mark
view thread (3+ messages)
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: github://postgresql-interfaces/psqlodbc
Cc: [email protected], [email protected]
Subject: Re: [postgresql-interfaces/psqlodbc] issue #96: Memory allocation error from PSQLODBC35W.DLL
In-Reply-To: <<[email protected]>>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox