Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mXrbv-0000WH-8X for psycopg@arkaria.postgresql.org; Tue, 05 Oct 2021 21:04:27 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1mXrbt-0007aR-1v for psycopg@arkaria.postgresql.org; Tue, 05 Oct 2021 21:04:25 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mXrbs-0007aG-HM for psycopg@lists.postgresql.org; Tue, 05 Oct 2021 21:04:24 +0000 Received: from mail-vs1-xe2c.google.com ([2607:f8b0:4864:20::e2c]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1mXrbl-0006Lr-Ht for psycopg@lists.postgresql.org; Tue, 05 Oct 2021 21:04:23 +0000 Received: by mail-vs1-xe2c.google.com with SMTP id w13so710323vsa.2 for ; Tue, 05 Oct 2021 14:04:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=i7M0+z5rkacl3gYkmHy7+oxMBYrZ/Bev1jnmUy0O7O8=; b=Euf2Ol/OwKguMdYCUl8UULtStP0vEY7gX27MpkM9jviNdaSYkSAmF8be6v2sKlREXq jUaOFLpAg/dgrMBvuzLpGd21vEgV6t2SHCteWpu9FmzD8UFYELu3PeLV/v96WgjR05hA fm2AbUbOROdE8Pe19qYm2PYVOBOjaoF1FPuO5ZXvZyZ/kwZ/6eZen6L7tMw2mTga7zkF FsobScRTSvSXZj1E3CrmRK7WYQhLtvOSZFstrGd6URiSzDu2G6PXjjgwHlNkOyWHPvwF UQWJYL/UtGL95Bx+Z/f5fq1QQNhdszl+BgNZohDf+0qKGOzT+fvXhw5iaU8j0QgfFftk 5K5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=i7M0+z5rkacl3gYkmHy7+oxMBYrZ/Bev1jnmUy0O7O8=; b=7w2GJZWx0SLwORzWzr3YhGrhYJVNZogqH6+6MBlrtRqQLDWcigUe84bnkxligQE9nf BNI2MIrymwjCOxQgyZX8ndr2LtwavVajiSiaRd6l+oBppN2QJ2S4izJ/Wa2+saFuWNGy mt7B8zen58fdYsyDjEwsNQ6YzNVnwvxRPmwdSpth4NkXAO/uigkRyOyhELhvXYOsHL0d wmJ7koYktnFg8aYA+sDlEyyPvWkYBdEQFjk4z7UItTlwzAHqY+Icuiyx55PxOgO/DDY5 7Y5NXiHRyvyAOTXc3WQehZ94vi+nZQ3jCxLkivX4NeygvCfqfQCTfp3hlkewkjBNqY+6 X3mg== X-Gm-Message-State: AOAM530zCpuFbVzXADEtCBnfHClrE8CI9J6GEOGwZQM0kA80yx6oKmpr 0vvSUPOeIBDiSSOhSI5ikXOBNuOSh9lIN0NCzf0= X-Google-Smtp-Source: ABdhPJy9x5CXE0PkuSApLCICiBvMK59bby8EgQRtQBJNqwSYBabb3KWc/I5tv42qMG9fnpTI/7Xrkz7YXqoOnqPbhLg= X-Received: by 2002:a67:ae0a:: with SMTP id x10mr21379197vse.21.1633467856056; Tue, 05 Oct 2021 14:04:16 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Dan Davis Date: Tue, 5 Oct 2021 17:04:05 -0400 Message-ID: Subject: Re: How to build statically on Windows To: Jason Erickson Cc: Daniele Varrazzo , psycopg@lists.postgresql.org Content-Type: multipart/alternative; boundary="0000000000004ff51605cda15f19" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --0000000000004ff51605cda15f19 Content-Type: text/plain; charset="UTF-8" I am going to tell my team this is much, much harder than cx_Oracle, and we may have no choice but to coordinate updates to more recent copies. I would never ask you guys to go back and backfill, e.g. build 2.8.5 for Python 3.9 just for us. I decided to at least try the "easy button" and setup a fork of psycopg2 with appveyor. I changed the files in my fork for the .appveyor sub-directory to only build for Python 3.9 and Python 3.6 64 bit. However, it seems that appveyor has too many settings for me to easily duplicate the environment - not enough is controlled simply from the files in .appveyor to make this easy button super easy. If it is possible for someone to document the settings, that probably would be of benefit for you guys as well. I know most people creating pull requests can simply rely on github.com and appveyor to test that everything works, however. On Tue, Oct 5, 2021 at 4:40 PM Dan Davis wrote: > Jason, > > I tried to call the appveyor script, but it was looking for a number of > Environment variables I did not have. I got through PY_VER=39 and then > thought better of it. > > Do you think it would be easier to do it this way: > * fork psycopg2 > * make my own appveyor account that builds the versions I need... > > Simulate the appveyor environment on my own workstation > > Wow, what a lot I've forgotten since before manylinux1 - I used to do this > for lxml, PyCrypto, and a lot of other packages :). Still, they were all > more or less one-offs - I never got as automated as modern DevOps allows > these days. > > > > On Tue, Oct 5, 2021 at 2:11 PM Jason Erickson > wrote: > >> Hi Dan, >> >> Yeah, unfortunately --static-libpq doesn't do what it should anymore. We >> should remove that option. >> >> When building pyscopg2 from source, the binary Windows Postgres packages >> does not include the statically linked library (In fairness, I haven't >> checked the latest packages, but that is how it has been in the past). The >> library included with the binary packages was the DLL import library and it >> was named libpq.lib. This is an issue, because originally when you built >> from source on windows, the file libpq.lib was the static link library, >> whereas the DLL import library was named libpqdll.lib. So we have a name >> inconsistency and no way to link statically with the Postgres binary >> distribution. >> >> This got even more convoluted a few major versions back with the source >> of Postgres, as the Windows perl build scripts quit creating the build file >> for the static link library, only building the DLL import library AND >> naming it libpq.lib. With our Appveyor build script, I cheated and >> modified the perl script to build the library instead of the DLL at this >> line: >> file_replace('Mkvcbuild.pm', "'libpq', 'dll'", "'libpq', 'lib'") >> >> With that said, I am not happy with that solution and always intended to >> revisit the setup script portion for windows, but always had more questions >> than answers, some of them: >> * If static libraries are not part of the Postgres binary distribution >> (or even the build from source option by default), do we concern ourselves >> with them? Personally I prefer static libraries because I think it has less >> support issues, but??? >> * If people are building psycopg from source, what libraries do we assume >> they have installed? pg_config probably would not be working, so >> include/library paths would have to be passed. What about other >> dependencies (ie openssl, still use the has_ssl flag?)? There are slightly >> different link libraries between some Postgres, so versions matter, too. >> * Do we try to differentiate between the DLL import library and the >> static library since the name can be the same between them (size?)? >> * Or do we say heck with it and just link against a file named libpq.lib, >> letting the builder point to the libpq they want to use? >> >> Maybe an option is to have setup.py on windows call the appveyor script >> (with some modifications) and build everything from scratch? It takes >> about 40 minutes to build everything, tho, and does require some storage >> space for the source and build files, so not the best solution. >> >> -jason >> >> >> On Mon, Oct 4, 2021 at 5:13 PM Daniele Varrazzo < >> daniele.varrazzo@gmail.com> wrote: >> >>> Hi Dan, >>> >>> On Tue, 5 Oct 2021 at 01:01, Dan Davis wrote: >>> > >>> > Can anyone give me a solution to build psycopg2 statically on Windows? >>> > >>> > I have succeeded in building it, but when I run dumpbin /dependents on >>> the generated file (the PYD file), it still depends on libpq.dll even when >>> I pass --static-libpq. >>> >>> I haven't personally used --static-libpq in a long time, and neither >>> have I used windows for a while. As far as I know that part of the set >>> up might have bitrotten. >>> >>> If anyone can help Dan it would be appreciated. >>> >>> Dan, there is a ticket/MR in the tracker of which I've never been able >>> to make completely sense: https://github.com/psycopg/psycopg2/pull/758 >>> Would you like to check if that's the issue that doesn't allow >>> building the lib statically and if so can you propose a MR or just >>> acknowledge that the proposed one works as expected? >>> >>> Thank you everyone >>> >>> -- Daniele >>> >>> >>> --0000000000004ff51605cda15f19 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

I am going to tell my team this i= s much, much harder than cx_Oracle, and we may have no choice but to coordi= nate=C2=A0updates to more recent copies. I would never ask you guys to go b= ack and backfill, e.g. build 2.8.5 for Python 3.9 just for us.
<= div>
I decided to at least try the "easy button" and set= up a fork of psycopg2 with appveyor.=C2=A0 I changed the files in my fork f= or the .appveyor sub-directory to only build for Python 3.9 and Python 3.6 = 64 bit.=C2=A0 However, it seems that appveyor has too many settings for me = to easily duplicate the environment - not enough is controlled simply from = the files in .appveyor to make this easy button super easy.=C2=A0 If it is = possible for someone to document the settings, that probably would be of be= nefit for you guys as well.

I know most people creating = pull requests can simply rely on github.com and appveyor to test that everything works, however.

On Tue, Oct 5, 2021 at 4:40 PM Da= n Davis <dansmood@gmail.com>= ; wrote:
Jason,

I tried to call the appveyor script, bu= t it was looking for a number of Environment variables I did not have.=C2= =A0 I got through PY_VER=3D39 and then thought better of it.=C2=A0

Do you think it would be easier to do it this way:
* fork psycopg2
* make my own appveyor account that builds the= =C2=A0versions I need...

Simulate the appveyor env= ironment on my own workstation

Wow, what a lot I&#= 39;ve forgotten since before manylinux1 - I used to do this for lxml, PyCry= pto, and a lot of other packages :).=C2=A0 =C2=A0Still, they were all more = or less one-offs - I never got as automated as modern DevOps allows these d= ays.



On Tue, Oct 5, 2021 at 2:11 PM Jaso= n Erickson <jerickso@stickpeople.com> wrote:
Hi Dan,

Yeah, unfortunately --static-libpq doesn't do what it should anym= ore.=C2=A0 We should remove that option.

When buil= ding pyscopg2 from source, the binary Windows Postgres packages does not in= clude the statically linked library (In fairness, I haven't checked the= latest=C2=A0packages, but that is how it has been in the past).=C2=A0 The = library included with the binary packages was the DLL import library and it= was named libpq.lib.=C2=A0 This is an issue, because originally when you b= uilt from source on windows, the file libpq.lib was the static link library= , whereas the DLL import library was named libpqdll.lib.=C2=A0 So we have a= name inconsistency and=C2=A0no way to link statically with the Postgres bi= nary distribution.

This got even mor= e convoluted a few major versions back with the source of Postgres, as the = Windows perl build scripts quit creating the build file for the static link= library, only building the DLL import library AND naming it libpq.lib.=C2= =A0 With our Appveyor build script, I cheated and modified the perl script = to build the library instead of the DLL at this line:
=C2=A0 =C2= =A0 =C2=A0 file_replace('Mkvcbuild.pm', <= span>"'libpq', 'dll'", "'li= bpq', 'lib'")

With that sa= id, I am not happy with that solution and always intended to revisit the se= tup script portion for windows, but always had more questions than answers,= some of them:
* If static libraries are not part of = the Postgres=C2=A0binary distribution (or even the build from source option= by default), do we concern ourselves with them? Personally I prefer static= libraries because I think it has less support issues, but???
* If people are building psycopg from source, what libraries do we assume = they have installed?=C2=A0 pg_config probably would not be working, so incl= ude/library paths would have to be passed. What about other dependencies (i= e openssl, still use the has_ssl flag?)?=C2=A0 There are slightly different= link libraries between some Postgres, so versions matter, too.
* Do we try to differentiate between the DLL import library and the stat= ic library since the name can be the same between them (size?)?
*= Or do we say heck with it and just link against a file named libpq.lib, le= tting the builder point to the libpq they want to use?

=
Maybe an option is to have setup.py on windows call the appveyor scrip= t (with some modifications) and build everything from scratch?=C2=A0 It tak= es about 40 minutes to build everything, tho, and does require some storage= space for the source and build files, so not the best solution.
=
-jason


On Mon, Oct 4, 2021 at 5:13 PM Da= niele Varrazzo <daniele.varrazzo@gmail.com> wrote:
Hi Dan,

On Tue, 5 Oct 2021 at 01:01, Dan Davis <dansmood@gmail.com> wrote:
>
> Can anyone give me a solution to build psycopg2 statically on Windows?=
>
> I have succeeded in building it, but when I run dumpbin /dependents on= the generated file (the PYD file), it still depends on libpq.dll even when= I pass --static-libpq.

I haven't personally used --static-libpq in a long time, and neither have I used windows for a while. As far as I know that part of the set
up might have bitrotten.

If anyone can help Dan it would be appreciated.

Dan, there is a ticket/MR in the tracker of which I've never been able<= br> to make completely sense: https://github.com/psycopg/psy= copg2/pull/758
Would you like to check if that's the issue that doesn't allow
building the lib statically and if so can you propose a MR or just
acknowledge that the proposed one works as expected?

Thank you everyone

-- Daniele


--0000000000004ff51605cda15f19--