Subject: Seeking Insights on Roll-out Process for Multiple PyPI Packages

From: Jan-Hendrik Lorenz <[email-masked]>
Date: Fri, 2 Jun 2023 11:10:05 +0200

Dear Psycopg mailing list members,

I'm reaching out to inquire about the roll-out process for managing two 
PyPI packages from a single source repository,
similar to the setup employed by the Psycopg Python package (psycopg2 
and psycopg2-binary).

As I work on a project with a comparable requirement, I would greatly 
appreciate any insights the Psycopg team can offer in this regard.
Specifically, I'm interested in the following:

     1) How did the Psycopg team handle deploying and distributing two 
PyPI packages from a single source repository?
     2) Were there any challenges encountered during the roll-out 
process, and if so, how were they addressed?
     3) Are there any recommended tools, scripts, or configuration 
adjustments for effectively managing the two PyPI packages?

Thank you for your time and consideration. If you need more information 
or have any questions, I'm happy to answer them.

Best regards,

Jan-Hendrik Lorenz
-- 
Dr. Jan-Hendrik Lorenz * Software Consultant * 
[email-masked]
TNG Technology Consulting GmbH, Beta-Str. 13a, 85774 Unterföhring
Geschäftsführer: Henrik Klagges, Dr. Robert Dahlke, Thomas Endres
Sitz: Unterföhring * Amtsgericht München * HRB 135082

---

From: Daniele Varrazzo <[email-masked]>
Date: Sat, 3 Jun 2023 20:28:29 +0200

Hello,

> PyPI packages from a single source repository?

With one line of sed:
https://github.com/psycopg/psycopg2/blob/8b17e218bea6dbaf43f27d6aa97bafb33e2fb66f/scripts/build/wheel_linux_before_all.sh#L48

The package name is replaced in the setup.py, and that's enough to create a
different distribution package from the same Python module.

> process, and if so, how were they addressed?

The biggest problem is faced with dependency tracking, because pypi
metadata don't actually consider the possibility of different distribution
package providing the same python package. Both psycopg2 and
psycopg2-binary can be installed in the same environment, but they will
trample on each other's files, with undefined results. Libraries shouldn't
depend on -binary, but of course many either don't read the indication or
they consider themselves superior and do it otherwise, causing problems
when two libraries both depending on psycopg2 are installed alongside.

In Psycopg 3 we are doing something similar, but we solved large part of
the problems. There is a pure python package defining the interface, which
can be used as dependency target, and optional speedup. The psycopg_c
speedup requires to be built on the client, so it's similar to psycopg2;
the psycopg_binary speedup is precompiled and distributed as wheel together
with the library dependencies, so it's similar to psycopg2-binary. The two
distribution packages install two distinct python packages, so they can be
installed alongside. See
https://www.psycopg.org/psycopg3/docs/basic/install.html

In this case too, both packages are created from the same source code;
modifications are a bit more than sed'ing a single file but also not
difficult: see the script
https://github.com/psycopg/psycopg/blob/master/tools/build/copy_to_binary.py

> adjustments for effectively managing the two PyPI packages?

Feel free to use the scripts and techniques above for your own use. I
strongly advise against distributing the same python package in different
distribution packages and suggest you to follow the approach of psycopg 3.

Best

-- Daniele