Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mXrsV-000120-Lp for psycopg@arkaria.postgresql.org; Tue, 05 Oct 2021 21:21:36 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1mXrsU-0004BJ-Fi for psycopg@arkaria.postgresql.org; Tue, 05 Oct 2021 21:21:34 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mXrsT-00049n-Uo for psycopg@lists.postgresql.org; Tue, 05 Oct 2021 21:21:34 +0000 Received: from mail-ua1-x936.google.com ([2607:f8b0:4864:20::936]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1mXrsM-0006Sk-W4 for psycopg@lists.postgresql.org; Tue, 05 Oct 2021 21:21:32 +0000 Received: by mail-ua1-x936.google.com with SMTP id f13so279684uan.6 for ; Tue, 05 Oct 2021 14:21:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=YqCifR3n2ZLOh7Yg9oajZ/RhB25dz0+RnL2KZcRplBc=; b=aPcAbm6Hv52SjSbHMbMCnur93kp6Sf3Zya14vHK2aBKft3qdzUay7wyRvzlM3Hgw2l bmRog8lQvYP0HnI+6rnzOLNVOb/x9AiKnrnj/lia28CNb1X9otJvQDOXaNlGmcoMPRgI cqIeHi4VT+fohCe603IYrQRTiJQuaEqkNsyxzstssZEB+U6dQEU2o9crXd6apoizua6H lPSKmZOvzsRb3DYxWXzjWwSZLoo/O/2cXA5VYbSFaGQoorYpunYJwPqP9hsLslctrEsK Rh9dDrcNZZur54ZXvIqr3PIWe17naE1m9jungNoibsqR9MqmZGx863zM/gl9zodU1MpD odzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=YqCifR3n2ZLOh7Yg9oajZ/RhB25dz0+RnL2KZcRplBc=; b=KZh0zXvVZE7KEP/zoG+U7s7gVvafw1umuxg9OmmVnYcgcMcC1ysmpohO5rp8LfouTn HHiIZMCJ0Dak75e9xKSs97MZ4RVGstyvUoiDhTiasTOxTYff2dl/ROngshXQz3hqoOmQ WP3ZMEyj6GjVutefEbnuUdPUrVuGzKyrDeeGpC8unc7uC3LLtGRbqBjbKnaPjH+tgxd8 6Fyy0xnAuSs0o3J8/hC+eTdEyBapNLz4qr/7FPt4ZUnE3WKN5eU6rkOaC9SwFAWGl1V+ fel91AS/JlcxVS1xQlorQxARleKiuxxwB1l2LJUzi0f3udGP/80TCwmEOFeDSj+LjOOw QAcQ== X-Gm-Message-State: AOAM531SZ5GlXtnP88ojio9HBJPcVsd0Ujnptxxo413E4/wfWTifojv6 gYWBu6P5ME8H78HzW3knUFlBUhBmKlxAa0Mod/c= X-Google-Smtp-Source: ABdhPJzKtrj8Tn4OD5NWaUAcryF+U2iIEM+4A3gZ9DV14xs2zKZa3igQ9EJSJt/d7OMyormdYuA9ZUI0aXQTP9j62qw= X-Received: by 2002:ab0:7587:: with SMTP id q7mr15317333uap.92.1633468885998; Tue, 05 Oct 2021 14:21:25 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Dan Davis Date: Tue, 5 Oct 2021 17:21:15 -0400 Message-ID: Subject: Re: How to build statically on Windows To: Jason Erickson Cc: Daniele Varrazzo , psycopg@lists.postgresql.org Content-Type: multipart/alternative; boundary="000000000000b39f0c05cda19c92" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --000000000000b39f0c05cda19c92 Content-Type: text/plain; charset="UTF-8" Daniele (and Jason), One more comment - digging into this a bit is reassuring. I had worried that by using psycopg2 windows builds and psycopg2-binary builds, we were using builds against ancient Postgresql 9.x client libraries. By digging into the builds process, I can see that as long as my users upgrade to psycopg2>=2.9.1 and I build that on Linux with the version of PostgreSQL libraries we install, then the Windows users will have a library built with PostgreSQL 13.3. Even those application developers who have not heeded me and are running psycopg2-binary will have libraries built with PostgreSQL 13.3 (as long as they upgrade). So, I think my digging in has helped me anyway, and you guys can consider me helped :) On Tue, Oct 5, 2021 at 5:04 PM Dan Davis wrote: > > I am going to tell my team this is much, much harder than cx_Oracle, and > we may have no choice but to coordinate updates to more recent copies. I > would never ask you guys to go back and backfill, e.g. build 2.8.5 for > Python 3.9 just for us. > > I decided to at least try the "easy button" and setup a fork of psycopg2 > with appveyor. I changed the files in my fork for the .appveyor > sub-directory to only build for Python 3.9 and Python 3.6 64 bit. However, > it seems that appveyor has too many settings for me to easily duplicate the > environment - not enough is controlled simply from the files in .appveyor > to make this easy button super easy. If it is possible for someone to > document the settings, that probably would be of benefit for you guys as > well. > > I know most people creating pull requests can simply rely on github.com > and appveyor to test that everything works, however. > > > > > On Tue, Oct 5, 2021 at 4:40 PM Dan Davis wrote: > >> Jason, >> >> I tried to call the appveyor script, but it was looking for a number of >> Environment variables I did not have. I got through PY_VER=39 and then >> thought better of it. >> >> Do you think it would be easier to do it this way: >> * fork psycopg2 >> * make my own appveyor account that builds the versions I need... >> >> Simulate the appveyor environment on my own workstation >> >> Wow, what a lot I've forgotten since before manylinux1 - I used to do >> this for lxml, PyCrypto, and a lot of other packages :). Still, they were >> all more or less one-offs - I never got as automated as modern DevOps >> allows these days. >> >> >> >> On Tue, Oct 5, 2021 at 2:11 PM Jason Erickson >> wrote: >> >>> Hi Dan, >>> >>> Yeah, unfortunately --static-libpq doesn't do what it should anymore. >>> We should remove that option. >>> >>> When building pyscopg2 from source, the binary Windows Postgres packages >>> does not include the statically linked library (In fairness, I haven't >>> checked the latest packages, but that is how it has been in the past). The >>> library included with the binary packages was the DLL import library and it >>> was named libpq.lib. This is an issue, because originally when you built >>> from source on windows, the file libpq.lib was the static link library, >>> whereas the DLL import library was named libpqdll.lib. So we have a name >>> inconsistency and no way to link statically with the Postgres binary >>> distribution. >>> >>> This got even more convoluted a few major versions back with the source >>> of Postgres, as the Windows perl build scripts quit creating the build file >>> for the static link library, only building the DLL import library AND >>> naming it libpq.lib. With our Appveyor build script, I cheated and >>> modified the perl script to build the library instead of the DLL at this >>> line: >>> file_replace('Mkvcbuild.pm', "'libpq', 'dll'", "'libpq', 'lib'") >>> >>> With that said, I am not happy with that solution and always intended to >>> revisit the setup script portion for windows, but always had more questions >>> than answers, some of them: >>> * If static libraries are not part of the Postgres binary distribution >>> (or even the build from source option by default), do we concern ourselves >>> with them? Personally I prefer static libraries because I think it has less >>> support issues, but??? >>> * If people are building psycopg from source, what libraries do we >>> assume they have installed? pg_config probably would not be working, so >>> include/library paths would have to be passed. What about other >>> dependencies (ie openssl, still use the has_ssl flag?)? There are slightly >>> different link libraries between some Postgres, so versions matter, too. >>> * Do we try to differentiate between the DLL import library and the >>> static library since the name can be the same between them (size?)? >>> * Or do we say heck with it and just link against a file named >>> libpq.lib, letting the builder point to the libpq they want to use? >>> >>> Maybe an option is to have setup.py on windows call the appveyor script >>> (with some modifications) and build everything from scratch? It takes >>> about 40 minutes to build everything, tho, and does require some storage >>> space for the source and build files, so not the best solution. >>> >>> -jason >>> >>> >>> On Mon, Oct 4, 2021 at 5:13 PM Daniele Varrazzo < >>> daniele.varrazzo@gmail.com> wrote: >>> >>>> Hi Dan, >>>> >>>> On Tue, 5 Oct 2021 at 01:01, Dan Davis wrote: >>>> > >>>> > Can anyone give me a solution to build psycopg2 statically on Windows? >>>> > >>>> > I have succeeded in building it, but when I run dumpbin /dependents >>>> on the generated file (the PYD file), it still depends on libpq.dll even >>>> when I pass --static-libpq. >>>> >>>> I haven't personally used --static-libpq in a long time, and neither >>>> have I used windows for a while. As far as I know that part of the set >>>> up might have bitrotten. >>>> >>>> If anyone can help Dan it would be appreciated. >>>> >>>> Dan, there is a ticket/MR in the tracker of which I've never been able >>>> to make completely sense: https://github.com/psycopg/psycopg2/pull/758 >>>> Would you like to check if that's the issue that doesn't allow >>>> building the lib statically and if so can you propose a MR or just >>>> acknowledge that the proposed one works as expected? >>>> >>>> Thank you everyone >>>> >>>> -- Daniele >>>> >>>> >>>> --000000000000b39f0c05cda19c92 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Daniele (and Jason),

One more comment -= digging into this a bit is reassuring.=C2=A0 I had worried that by using p= sycopg2 windows builds and psycopg2-binary builds, we were using builds aga= inst ancient Postgresql 9.x client libraries.=C2=A0 By digging into the bui= lds process, I can see that as long as my users upgrade to psycopg2>=3D2= .9.1 and I build that on Linux with the version of PostgreSQL libraries we = install, then the Windows users will have a library built with PostgreSQL 1= 3.3.=C2=A0 =C2=A0 Even those application developers who have not heeded=C2= =A0me and are running=C2=A0psycopg2-binary will have libraries built with P= ostgreSQL 13.3 (as long as they upgrade).

So, I th= ink my digging in has helped me anyway, and you guys can consider me helped= :)

On Tue, Oct 5, 2021 at 5:04 PM Dan Davis <dansmood@gmail.com> wrote:

I am going to tell my team this is much, much harder than cx_Oracle, and = we may have no choice but to coordinate=C2=A0updates to more recent copies.= I would never ask you guys to go back and backfill, e.g. build 2.8.5 for P= ython 3.9 just for us.

I decided to at least try = the "easy button" and setup a fork of psycopg2 with appveyor.=C2= =A0 I changed the files in my fork for the .appveyor sub-directory to only = build for Python 3.9 and Python 3.6 64 bit.=C2=A0 However, it seems that ap= pveyor has too many settings for me to easily duplicate the environment - n= ot enough is controlled simply from the files in .appveyor to make this eas= y button super easy.=C2=A0 If it is possible for someone to document the se= ttings, that probably would be of benefit for you guys as well.

I know most people creating pull requests can simply rely on github.com and appveyor to te= st that everything works, however.




On Tue, Oct 5, 2021 at 4:40 PM Dan Davis <dansmood@gmail.com>= wrote:
Jason,

I tried to call the appveyor script, but= it was looking for a number of Environment variables I did not have.=C2=A0= I got through PY_VER=3D39 and then thought better of it.=C2=A0
<= br>
Do you think it would be easier to do it this way:
= * fork psycopg2
* make my own appveyor account that builds the=C2= =A0versions I need...

Simulate the appveyor enviro= nment on my own workstation

Wow, what a lot I'= ve forgotten since before manylinux1 - I used to do this for lxml, PyCrypto= , and a lot of other packages :).=C2=A0 =C2=A0Still, they were all more or = less one-offs - I never got as automated as modern DevOps allows these days= .



<= div dir=3D"ltr" class=3D"gmail_attr">On Tue, Oct 5, 2021 at 2:11 PM Jason E= rickson <j= erickso@stickpeople.com> wrote:
Hi Dan,

Yeah, unfortunately --static-libpq doesn't do what it should anymore= .=C2=A0 We should remove that option.

When buildin= g pyscopg2 from source, the binary Windows Postgres packages does not inclu= de the statically linked library (In fairness, I haven't checked the la= test=C2=A0packages, but that is how it has been in the past).=C2=A0 The lib= rary included with the binary packages was the DLL import library and it wa= s named libpq.lib.=C2=A0 This is an issue, because originally when you buil= t from source on windows, the file libpq.lib was the static link library, w= hereas the DLL import library was named libpqdll.lib.=C2=A0 So we have a na= me inconsistency and=C2=A0no way to link statically with the Postgres binar= y distribution.

This got even more c= onvoluted a few major versions back with the source of Postgres, as the Win= dows perl build scripts quit creating the build file for the static link li= brary, only building the DLL import library AND naming it libpq.lib.=C2=A0 = With our Appveyor build script, I cheated and modified the perl script to b= uild the library instead of the DLL at this line:
=C2=A0 =C2=A0 = =C2=A0 file_replace('Mkvcbuild.pm', "'libpq', 'dll'", "'libpq&= #39;, 'lib'")

With that said, = I am not happy with that solution and always intended to revisit the setup = script portion for windows, but always had more questions than answers, som= e of them:
* If static libraries are not part of the = Postgres=C2=A0binary distribution (or even the build from source option by = default), do we concern ourselves with them? Personally I prefer static lib= raries because I think it has less support issues, but???
* I= f people are building psycopg from source, what libraries do we assume they= have installed?=C2=A0 pg_config probably would not be working, so include/= library paths would have to be passed. What about other dependencies (ie op= enssl, still use the has_ssl flag?)?=C2=A0 There are slightly different lin= k libraries between some Postgres, so versions matter, too.
*= Do we try to differentiate between the DLL import library and the static l= ibrary since the name can be the same between them (size?)?
* Or = do we say heck with it and just link against a file named libpq.lib, lettin= g the builder point to the libpq they want to use?

Maybe an option is to have setup.py on windows call the appveyor script (w= ith some modifications) and build everything from scratch?=C2=A0 It takes a= bout 40 minutes to build everything, tho, and does require some storage spa= ce for the source and build files, so not the best solution.

=
-jason


=
On Mon, Oct 4, 2021 at 5:13 PM Daniel= e Varrazzo <daniele.varrazzo@gmail.com> wrote:
Hi Dan,

On Tue, 5 Oct 2021 at 01:01, Dan Davis <dansmood@gmail.com> wrote:
>
> Can anyone give me a solution to build psycopg2 statically on Windows?=
>
> I have succeeded in building it, but when I run dumpbin /dependents on= the generated file (the PYD file), it still depends on libpq.dll even when= I pass --static-libpq.

I haven't personally used --static-libpq in a long time, and neither have I used windows for a while. As far as I know that part of the set
up might have bitrotten.

If anyone can help Dan it would be appreciated.

Dan, there is a ticket/MR in the tracker of which I've never been able<= br> to make completely sense: https://github.com/psycopg/psy= copg2/pull/758
Would you like to check if that's the issue that doesn't allow
building the lib statically and if so can you propose a MR or just
acknowledge that the proposed one works as expected?

Thank you everyone

-- Daniele


--000000000000b39f0c05cda19c92--