Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1q5W6P-0004BX-3G for psycopg@arkaria.postgresql.org; Sat, 03 Jun 2023 18:35:49 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1q5W6N-00009K-Vu for psycopg@arkaria.postgresql.org; Sat, 03 Jun 2023 18:35:47 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1q5W6N-000091-Kt for psycopg@lists.postgresql.org; Sat, 03 Jun 2023 18:35:47 +0000 Received: from mail-yb1-xb2f.google.com ([2607:f8b0:4864:20::b2f]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1q5W6G-0000vr-UH for psycopg@postgresql.org; Sat, 03 Jun 2023 18:35:46 +0000 Received: by mail-yb1-xb2f.google.com with SMTP id 3f1490d57ef6-bad97da58adso3966357276.1 for ; Sat, 03 Jun 2023 11:35:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1685817340; x=1688409340; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=WiWwK2NenpZL1A5pAM7KUDfdMhJhGmN0nAp1yZxIGd0=; b=T0W9P1MX8+g3II91itwZCDUVe4UTZpfXNGu1il6TLlCfwGKfd/qBaOwPUcwtmp+pOz WTB+0UCSmA8SohbVGjAw0DxZ14DEU/GL3QXcYCETnhnOOQuhBedRkBaXXN6yy4CGXGWN lWFJtGSOdVG/Z/MF3ZwfoOvF24sdXfVX/f7DIS2fS9bF2DW1Oelv2Urzr071hZmtT3ug AQ56HeFwQrVq3z+aFa4bodDyicx/IkMCVyflHVwjsCBG0Z0h9gtaoBYHphNom7+WC9l8 mm+odEYbwiUSFF5utp+bJxwUxknBdjKSzVvitWrbRoll1GnAsZOvtoWJXgPEP/yLN53Y RscA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685817340; x=1688409340; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=WiWwK2NenpZL1A5pAM7KUDfdMhJhGmN0nAp1yZxIGd0=; b=KxecFU6zvxUWHaZn3QIpWPmbGMR+v00wIbPkT+E2Qar0IWqlFV1Dg+033g9CdT42eH JUpyBVZrypJnaVUeaUY4dU0XbdaRcgnNmssSVxmXIvhTCv4IjtWaNEWbzsADKvkTaI2G ddr25QmLUYQFtwVhV/vXZ2ac9y+uqVTsXkOaoY3T2Rztn0HyqTZx4cwnd7x4xmPhuJtY peKkkLy5ZO78tmQM/4a0tkcLNrACb/s3QWPMS/VPs94DPCS63N+mZRbx6mftnEl21468 hs8iNfeiRLw6xVggiFO3y0BwPBaTbhIB6F5AjEH89M+QpPoX86xc+0D+eKooRb7pcwz2 d9kg== X-Gm-Message-State: AC+VfDzXm+p0eW134Twd8+vezMwuIYLJcx73Ji3F6H1+udA8XVz+JZNT SSMtaQxNKPC/kmCnRH4wSCBwv/16LEo0DlKWKbHQA8eI X-Google-Smtp-Source: ACHHUZ4k7SWxMp9Q2tvtA7N4BL4sA1TaD3DFV5iL+flFVfr4pSEfRh+TVp0aReGI6ENVsSMD4APH2xJhfO3vt0fEWGI= X-Received: by 2002:a05:6870:e290:b0:195:cd3d:1134 with SMTP id v16-20020a056870e29000b00195cd3d1134mr3403091oad.17.1685816922688; Sat, 03 Jun 2023 11:28:42 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Daniele Varrazzo Date: Sat, 3 Jun 2023 20:28:29 +0200 Message-ID: Subject: Re: Seeking Insights on Roll-out Process for Multiple PyPI Packages To: Jan-Hendrik Lorenz Cc: Psycopg Content-Type: multipart/alternative; boundary="000000000000d552f505fd3dd6c3" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --000000000000d552f505fd3dd6c3 Content-Type: text/plain; charset="UTF-8" Hello, On Fri, 2 Jun 2023, 16:25 Jan-Hendrik Lorenz, < jan-hendrik.lorenz@tngtech.com> wrote: > > 1) How did the Psycopg team handle deploying and distributing two > PyPI packages from a single source repository? > With one line of sed: https://github.com/psycopg/psycopg2/blob/8b17e218bea6dbaf43f27d6aa97bafb33e2fb66f/scripts/build/wheel_linux_before_all.sh#L48 The package name is replaced in the setup.py, and that's enough to create a different distribution package from the same Python module. 2) Were there any challenges encountered during the roll-out > process, and if so, how were they addressed? > The biggest problem is faced with dependency tracking, because pypi metadata don't actually consider the possibility of different distribution package providing the same python package. Both psycopg2 and psycopg2-binary can be installed in the same environment, but they will trample on each other's files, with undefined results. Libraries shouldn't depend on -binary, but of course many either don't read the indication or they consider themselves superior and do it otherwise, causing problems when two libraries both depending on psycopg2 are installed alongside. In Psycopg 3 we are doing something similar, but we solved large part of the problems. There is a pure python package defining the interface, which can be used as dependency target, and optional speedup. The psycopg_c speedup requires to be built on the client, so it's similar to psycopg2; the psycopg_binary speedup is precompiled and distributed as wheel together with the library dependencies, so it's similar to psycopg2-binary. The two distribution packages install two distinct python packages, so they can be installed alongside. See https://www.psycopg.org/psycopg3/docs/basic/install.html In this case too, both packages are created from the same source code; modifications are a bit more than sed'ing a single file but also not difficult: see the script https://github.com/psycopg/psycopg/blob/master/tools/build/copy_to_binary.py 3) Are there any recommended tools, scripts, or configuration > adjustments for effectively managing the two PyPI packages? > Feel free to use the scripts and techniques above for your own use. I strongly advise against distributing the same python package in different distribution packages and suggest you to follow the approach of psycopg 3. Best -- Daniele --000000000000d552f505fd3dd6c3 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello,

On Fri, 2 Jun 2023, 16:25 Jan-Hendrik Lorenz, &l= t;jan-hendrik.lorenz@tngt= ech.com> wrote:

=C2=A0 =C2=A0 =C2=A01) How did the Psycopg team handle deploying and distri= buting two
PyPI packages from a single source repository?
=


The package name is replaced in the setup.py, and that's en= ough to create a different distribution package from the same Python module= .


=C2=A0 =C2=A0 =C2=A02) Were there any challenges encountered during the rol= l-out
process, and if so, how were they addressed?

The biggest problem is faced wi= th dependency tracking, because pypi metadata don't actually consider t= he possibility of different distribution package providing the same python = package. Both psycopg2 and psycopg2-binary can be installed in the same env= ironment, but they will trample on each other's files, with undefined r= esults. Libraries shouldn't depend on -binary, but of course many eithe= r don't read the indication or they consider themselves superior and do= it otherwise, causing problems when two libraries both depending on psycop= g2 are installed alongside.

In Psycopg 3 we are doing something similar, but we solved large part o= f the problems. There is a pure python package defining the interface, whic= h can be used as dependency target, and optional speedup. The psycopg_c spe= edup requires to be built on the client, so it's similar to psycopg2; t= he psycopg_binary speedup is precompiled and distributed as wheel together = with the library dependencies, so it's similar to psycopg2-binary. The = two distribution packages install two distinct python packages, so they can= be installed alongside. See=C2=A0https://www.psycopg.org/psycopg3/docs/basic/ins= tall.html

In this ca= se too, both packages are created from the same source code; modifications = are a bit more than sed'ing a single file but also not difficult: see t= he script=C2=A0https://github.com/psycopg/psycopg/blob/master= /tools/build/copy_to_binary.py

=C2=A0 =C2=A0 =C2=A03) Are there any recommended tools, scripts, or configu= ration
adjustments for effectively managing the two PyPI packages?

Feel free to use= the scripts and techniques above for your own use. I strongly advise again= st distributing the same python package in different distribution packages = and suggest you to follow the approach of psycopg 3.

Best

-- Daniele=C2=A0
--000000000000d552f505fd3dd6c3--