agora inbox for pgsql-pkg-debian@postgresql.org
help / color / mirror / Atom feedMigration of pgBackRest internal support tools to Rust.
2+ messages / 1 participants
[nested] [flat]
* Migration of pgBackRest internal support tools to Rust.
@ 2026-07-15 02:44 David Steele <david@pgbackrest.org>
0 siblings, 1 reply; 2+ messages in thread
From: David Steele @ 2026-07-15 02:44 UTC (permalink / raw)
To: pgsql-pkg-debian@lists.postgresql.org; pgsql-pkg-yum@lists.postgresql.org
Greetings Packagers,
I'm considering migrating the pgBackRest support tools to Rust and I'd
appreciate any feedback on how this would affect packaging.
To be clear, the pgBackRest binary itself would stay in C, so this only
affects the internal build and support tooling.
We have three internal tools that support pgBackRest:
- Code generation (/src/build), which is written entirely in C and is
the simplest of the three. It reads various YAML files and does code
generation for configuration, help, errors, and support for various
PostgreSQL versions. Most of the generated code is checked in with a
couple of exceptions:
- src/command/help/help.auto.c.inc - this is compressed binary data
for the command line help. It is not checked in because even small
changes to the help tend to rewrite the entire file.
- src/postgres/interface.auto.c.inc - this defines the interface to
each supported version of PostgreSQL using data structures pulled from
PostgreSQL and macros that define functions. It is low churn but is not
that useful for review purposes so it is not checked in, but it could be.
All packages currently run code generation since it is baked into the
meson build. So if the generator moves to Rust, every package would need
Rust at build time unless the generated files are shipped with the release.
- Second is doc generation (/doc). This is about 1/3 C and the rest is
still in Perl. The code builds and executes the user guide and other
source XML and outputs HTML, markdown, and man pages. Most of the
complexity is in the Perl code but none of it is too complicated. The
Debian package uses the doc generator.
- Last is the test harness (/test), which is also about 1/3 C and the
rest in Perl. In addition to running tests this code manages code
generation, linting, container builds, code formatting, etc. The unit
and integration tests have to stay in C but the harness can be in any
language. I'm not aware of any packages that run our tests, though.
Rather than complete the migration from Perl to C, I would prefer to
just migrate all of it to Rust. Rust is a powerful language with rich
libraries that would make my life much easier and in the long run save
time. I prototyped migrating the test harness C code to Rust and it
looks pretty good.
The easiest thing would be for the packagers to provide Rust in the
build environment so code generation can run and packages can build
documentation or run tests as they please, but my understanding is that
Rust is problematic for some packages, and even where it is supported
the selection of crates would be limited with potential version issues
for older distros.
My proposal is to provide the files that are currently generated at
build time (html documentation, man pages, help.auto.c.inc,
interface.auto.c.inc) in each release either by checking them into the
repo with each release or by providing a dist tarball that adds the
generated files. The former option would require no changes for packages
while the latter would require pointing at a new tarball. Either way, no
new dependencies are required and in fact a few could be dropped, e.g.
Perl and libyaml.
Would either approach cause problems for your packaging, and do you have
a preference? Please let me know if you see any other issues.
Thanks,
-David
^ permalink raw reply [nested|flat] 2+ messages in thread
* Re: Migration of pgBackRest internal support tools to Rust.
@ 2026-07-17 09:47 David Steele <david@pgbackrest.org>
parent: David Steele <david@pgbackrest.org>
0 siblings, 0 replies; 2+ messages in thread
From: David Steele @ 2026-07-17 09:47 UTC (permalink / raw)
To: pgsql-pkg-debian@lists.postgresql.org; pgsql-pkg-yum@lists.postgresql.org
On 7/15/26 09:44, David Steele wrote:
>
> My proposal is to provide the files that are currently generated at
> build time (html documentation, man pages, help.auto.c.inc,
> interface.auto.c.inc) in each release
<...>
> by providing a dist tarball that contains the
> generated files.
This is the approach I decided to go with. In retrospect it was a
mistake on my part to expose so many details of our toolchain to
packaging. Live and learn!
Starting with the 2.59.0 release on July 20 there will be a distribution
tarball as a release asset that contains pre-built documentation in the
form of HTML and a man page, help.auto.c.inc and interface.auto.c.inc
pre-generated in the source tree, and a smoke test that exercises basic
functionality and can be run with meson test. The test needs only the
same core Python that is provided for the meson build environment.
The packagers will get a consistent deliverable that we test for
regressions and the pgBackRest developers can use whatever tooling is
deemed best for the project, as long as it does not leak into the dist
tarball.
You can see an example of what the release will look like going forward
(with dist tarball asset) here:
https://github.com/dwsteele/pgbackrest/releases/tag/release%2F2.59.0dev
This is exactly what the 2.59.0 release will look like except for the
dev tag. For 2.59.0 packages can still be built as before (from a source
tarball) and if there are issues with the dist tarball we will address
them in patch releases before 2.60.0.
I have attached the README.md that will be included in the dist tarball.
It has more details on what is provided.
Regards,
-David
Attachments:
[text/markdown] README.md (3.2K, ../../bbe4e843-2e04-4e3c-a040-09c5afea84bc@pgbackrest.org/2-README.md)
download | inline:
# pgBackRest Distribution
This is a pgBackRest build distribution. Unlike a checkout of the [git repository](https://github.com/pgbackrest/pgbackrest), it ships the generated source and the rendered documentation pre-built, so pgBackRest can be built and installed without the code generation or the documentation tooling.
See <https://pgbackrest.org> for complete documentation.
## Contents
- `src` -- pgBackRest source, including the pre-generated `*.auto.c.inc` files
- `doc/man` -- command reference
- `doc/html` -- HTML documentation
- `test` -- smoke test to verify the build
- `meson.build`, `meson_options.txt` -- build configuration
- `LICENSE`
## Build
pgBackRest builds with meson and ninja. The required libraries are libpq, OpenSSL (>= 1.1.1), libxml2, lz4, bz2, and zlib; zstd, libssh2 (for SFTP), and libsystemd are optional. For example, on Debian/Ubuntu:
```
apt-get install meson gcc pkg-config libpq-dev libssl-dev libxml2-dev \
liblz4-dev libzstd-dev libbz2-dev libz-dev libssh2-1-dev libsystemd-dev
```
Then configure, build, and install:
```
meson setup build .
ninja -C build
meson install -C build
```
The `pgbackrest` binary is built at `build/src/pgbackrest`.
## Test
A smoke test is included to verify that the build works. It finds each PostgreSQL installation on the system and runs the newly-built `pgbackrest` through a backup and restore cycle against every supported version found: create a cluster, configure archiving, create the stanza, check, back up, restore, and verify that the restored data matches. Versions that pgBackRest does not support are reported and skipped. The test requires nothing beyond the build environment and PostgreSQL -- no additional Python packages.
Run it with meson after building:
```
meson test -C build --suite smoke
```
Add `-v` to stream the output of each step as it runs:
```
meson test -C build --suite smoke -v
```
PostgreSQL is searched for in the standard packaging paths:
- `/usr/lib/postgresql/<version>/bin` (Debian/Ubuntu)
- `/usr/pgsql-<version>/bin` (RHEL)
- `/usr/libexec/postgresql<version>` (Alpine)
- `/usr/bin`
Each version is tested in a temporary directory with its own repository and a private unix socket, so a running PostgreSQL cluster is not disturbed. The test fails when there is no PostgreSQL to test against, since a test that silently does nothing does not verify the build. Since PostgreSQL will not run as root, running the test as root drops to the `postgres` user and fails if that user does not exist.
The test can also be run directly, which allows the defaults to be overridden:
```
python3 test/smoke.py --pgbackrest build/src/pgbackrest
```
Useful options are `--pg-bin-path` to search an additional path for PostgreSQL, `--version` to test only a specific version, and `--user` to select the user to drop to when running as root. Use `--help` for the complete list.
## Documentation
The rendered documentation is in `doc`: the command reference in `doc/man` and the HTML documentation in `doc/html`.
Two user guides are provided, Debian/Ubuntu and RHEL/Rocky/Alma. If only one user guide is required, replace user-guide-index.html with the desired user guide. For example, to keep only the Debian/Ubuntu user guide:
```
mv user-guide.html user-guide-index.html
rm user-guide-rhel.html
```
^ permalink raw reply [nested|flat] 2+ messages in thread
end of thread, other threads:[~2026-07-17 09:47 UTC | newest]
Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-07-15 02:44 Migration of pgBackRest internal support tools to Rust. David Steele <david@pgbackrest.org>
2026-07-17 09:47 ` David Steele <david@pgbackrest.org>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox