Subject: How to build statically on Windows

From: Dan Davis <[email-masked]>
Date: Mon, 4 Oct 2021 19:01:04 -0400

Can anyone give me a solution to build psycopg2 statically on Windows?

I have succeeded in building it, but when I run dumpbin /dependents on the
generated file (the PYD file), it still depends on libpq.dll even when I
pass --static-libpq.

*Environment*

- OS: Windows 10
- Psycopg version: psycopg2-2.8.5
- Python version: 3.9
- PostgreSQL version: 14.0 (from ZIP)
- pip version: 21.2.4
- Visual C++ version: 2019

*Procedure*

- Make sure pg_config and psql are in the path
- Download as follows - pip download psycopg2==2.8.5 --no-binary :all:
- Expand the tarball
- Build in that directory as follows (following appveyor.py
<https://github.com/psycopg/psycopg2/blob/master/scripts/build/appveyor.py#L291>
)

      python setup.py build_ext -l "libpgcommon libpgport"

- Try to verify it is indeed static

      dumpbin /dependents
build\lib.win-amd64-3.9\psycopg2\_psycopg.cp39-win_amd64.pyd

*Background*
The purpose here is 2 fold:

* Backfill support for Python 3.9 and psycopg2 to versions that may have
come out before 3.9 was available.
* Make sure our psycopg2 is built against a more recent version of the
PostgreSQL client libraries than 9.x

---

From: Daniele Varrazzo <[email-masked]>
Date: Tue, 5 Oct 2021 01:12:47 +0200

Hi Dan,

On Tue, 5 Oct 2021 at 01:01, Dan Davis <[email-masked]> wrote:
> I have succeeded in building it, but when I run dumpbin /dependents on the generated file (the PYD file), it still depends on libpq.dll even when I pass --static-libpq.

I haven't personally used --static-libpq in a long time, and neither
have I used windows for a while. As far as I know that part of the set
up might have bitrotten.

If anyone can help Dan it would be appreciated.

Dan, there is a ticket/MR in the tracker of which I've never been able
to make completely sense: https://github.com/psycopg/psycopg2/pull/758
Would you like to check if that's the issue that doesn't allow
building the lib statically and if so can you propose a MR or just
acknowledge that the proposed one works as expected?

Thank you everyone

-- Daniele

---

From: Jason Erickson <[email-masked]>
Date: Tue, 5 Oct 2021 12:10:51 -0600

Hi Dan,

Yeah, unfortunately --static-libpq doesn't do what it should anymore.  We
should remove that option.

When building pyscopg2 from source, the binary Windows Postgres packages
does not include the statically linked library (In fairness, I haven't
checked the latest packages, but that is how it has been in the past).  The
library included with the binary packages was the DLL import library and it
was named libpq.lib.  This is an issue, because originally when you built
from source on windows, the file libpq.lib was the static link library,
whereas the DLL import library was named libpqdll.lib.  So we have a name
inconsistency and no way to link statically with the Postgres binary
distribution.

This got even more convoluted a few major versions back with the source of
Postgres, as the Windows perl build scripts quit creating the build file
for the static link library, only building the DLL import library AND
naming it libpq.lib.  With our Appveyor build script, I cheated and
modified the perl script to build the library instead of the DLL at this
line:
      file_replace('Mkvcbuild.pm', "'libpq', 'dll'", "'libpq', 'lib'")

With that said, I am not happy with that solution and always intended to
revisit the setup script portion for windows, but always had more questions
than answers, some of them:
* If static libraries are not part of the Postgres binary distribution (or
even the build from source option by default), do we concern ourselves with
them? Personally I prefer static libraries because I think it has less
support issues, but???
* If people are building psycopg from source, what libraries do we assume
they have installed?  pg_config probably would not be working, so
include/library paths would have to be passed. What about other
dependencies (ie openssl, still use the has_ssl flag?)?  There are slightly
different link libraries between some Postgres, so versions matter, too.
* Do we try to differentiate between the DLL import library and the static
library since the name can be the same between them (size?)?
* Or do we say heck with it and just link against a file named libpq.lib,
letting the builder point to the libpq they want to use?

Maybe an option is to have setup.py on windows call the appveyor script
(with some modifications) and build everything from scratch?  It takes
about 40 minutes to build everything, tho, and does require some storage
space for the source and build files, so not the best solution.

-jason

On Mon, Oct 4, 2021 at 5:13 PM Daniele Varrazzo <[email-masked]>
wrote:

> > I have succeeded in building it, but when I run dumpbin /dependents on
> the generated file (the PYD file), it still depends on libpq.dll even when
> I pass --static-libpq.
>

---

From: Dan Davis <[email-masked]>
Date: Tue, 5 Oct 2021 16:40:48 -0400

Jason,

I tried to call the appveyor script, but it was looking for a number of
Environment variables I did not have.  I got through PY_VER=39 and then
thought better of it.

Do you think it would be easier to do it this way:
* fork psycopg2
* make my own appveyor account that builds the versions I need...

Simulate the appveyor environment on my own workstation

Wow, what a lot I've forgotten since before manylinux1 - I used to do this
for lxml, PyCrypto, and a lot of other packages :).   Still, they were all
more or less one-offs - I never got as automated as modern DevOps allows
these days.

On Tue, Oct 5, 2021 at 2:11 PM Jason Erickson <[email-masked]>
wrote:

> On Mon, Oct 4, 2021 at 5:13 PM Daniele Varrazzo <
> [email-masked]> wrote:
>
>> Hi Dan,
>>

---

From: Dan Davis <[email-masked]>
Date: Tue, 5 Oct 2021 17:04:05 -0400

I am going to tell my team this is much, much harder than cx_Oracle, and we
may have no choice but to coordinate updates to more recent copies. I would
never ask you guys to go back and backfill, e.g. build 2.8.5 for Python 3.9
just for us.

I decided to at least try the "easy button" and setup a fork of psycopg2
with appveyor.  I changed the files in my fork for the .appveyor
sub-directory to only build for Python 3.9 and Python 3.6 64 bit.  However,
it seems that appveyor has too many settings for me to easily duplicate the
environment - not enough is controlled simply from the files in .appveyor
to make this easy button super easy.  If it is possible for someone to
document the settings, that probably would be of benefit for you guys as
well.

I know most people creating pull requests can simply rely on github.com and
appveyor to test that everything works, however.

On Tue, Oct 5, 2021 at 4:40 PM Dan Davis <[email-masked]> wrote:

>> This got even more convoluted a few major versions back with the source
>> of Postgres, as the Windows perl build scripts quit creating the build file
>> * If static libraries are not part of the Postgres binary distribution
>> (or even the build from source option by default), do we concern ourselves
>> with them? Personally I prefer static libraries because I think it has less
>> support issues, but???
>> * Do we try to differentiate between the DLL import library and the
>> static library since the name can be the same between them (size?)?

---

From: Dan Davis <[email-masked]>
Date: Tue, 5 Oct 2021 17:21:15 -0400

Daniele (and Jason),

One more comment - digging into this a bit is reassuring.  I had worried
that by using psycopg2 windows builds and psycopg2-binary builds, we were
using builds against ancient Postgresql 9.x client libraries.  By digging
into the builds process, I can see that as long as my users upgrade to
psycopg2>=2.9.1 and I build that on Linux with the version of PostgreSQL
libraries we install, then the Windows users will have a library built with
PostgreSQL 13.3.    Even those application developers who have not
heeded me and are running psycopg2-binary will have libraries built with
PostgreSQL 13.3 (as long as they upgrade).

So, I think my digging in has helped me anyway, and you guys can consider
me helped :)

On Tue, Oct 5, 2021 at 5:04 PM Dan Davis <[email-masked]> wrote:

>
> I am going to tell my team this is much, much harder than cx_Oracle, and
> we may have no choice but to coordinate updates to more recent copies. I
> would never ask you guys to go back and backfill, e.g. build 2.8.5 for
> Python 3.9 just for us.
>
> I know most people creating pull requests can simply rely on github.com
> and appveyor to test that everything works, however.
>
>
>
>
>> Wow, what a lot I've forgotten since before manylinux1 - I used to do
>> this for lxml, PyCrypto, and a lot of other packages :).   Still, they were
>> all more or less one-offs - I never got as automated as modern DevOps
>> allows these days.
>>
>>
>>
>>> Yeah, unfortunately --static-libpq doesn't do what it should anymore.
>>> We should remove that option.
>>>
>>> * If people are building psycopg from source, what libraries do we
>>> assume they have installed?  pg_config probably would not be working, so
>>> * Or do we say heck with it and just link against a file named
>>> libpq.lib, letting the builder point to the libpq they want to use?
>>>
>>>> > I have succeeded in building it, but when I run dumpbin /dependents
>>>> on the generated file (the PYD file), it still depends on libpq.dll even
>>>> when I pass --static-libpq.
>>>>

---

From: Daniele Varrazzo <[email-masked]>
Date: Tue, 5 Oct 2021 23:46:38 +0200

Hey Dan,

> I would never ask you guys to go back and backfill, e.g. build 2.8.5 for Python 3.9 just for us.

If your "backfill" needs are not extremely onerous we would have no
problem in helping you out. If it's only a matter of adding the python
version to an already working pipeline and release the resulting
artifact we could be up for itl.

-- Daniele

---

From: Jason Erickson <[email-masked]>
Date: Tue, 5 Oct 2021 15:51:08 -0600

Hi Dan,

Yeah, the appveyor script expects the appveyor environment, path locations,
etc, so some work would be needed to get it to function.  Forking it and
using appveyor to build would be the simplest.

One thing we haven't done yet with the builds is try building against the
PostgreSQL 14.x source, so if you build against that, keep it in mind.

-jason

---

From: Daniele Varrazzo <[email-masked]>
Date: Wed, 6 Oct 2021 00:03:18 +0200

On Tue, 5 Oct 2021 at 23:51, Jason Erickson <[email-masked]> wrote:

> One thing we haven't done yet with the builds is try building against the PostgreSQL 14.x source, so if you build against that, keep it in mind.

There are Postgres 14 and Python 3.10 out there in the same week, for
extra fun :D

I have already prepared a branch to build with Python 3.10
(https://github.com/psycopg/psycopg2/tree/py310), but it was failing
because the images weren't there yet.

-- Daniele