pg.ddx.io  pgbackrest/pgbackrest GitHub issues and pull requests  
help / color / mirror / Atom feed
[pgbackrest/pgbackrest] PR #2917: Add an info helper to load, cache, and save info files.
13+ messages / 3 participants
[nested] [flat]

* [pgbackrest/pgbackrest] PR #2917: Add an info helper to load, cache, and save info files.
@ 2026-11-10 17:47  "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
  0 siblings, 0 replies; 13+ messages in thread

From: dwsteele (@dwsteele) @ 2026-11-10 17:47 UTC (permalink / raw)
  To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>

Commands loaded and saved archive.info and backup.info themselves, passing the repository storage and cipher spec each time. This made it easy to get wrong, e.g. annotate loaded backup.info from the repository where the backup set was found but saved it to the default repository (fixed in b49e758f). The helper would have prevented this. Commands also had to get the repository storage before loading an info file so a remote repository could set the cipher options.

Add an info helper that loads archive.info and backup.info for a repository on first use and caches them until the configuration is loaded again. The save functions are moved to internal headers so info files are only saved through the helper, which saves to the repository the info was loaded from. The helper gets the repository storage before the cipher spec, so commands that load or save info files through the helper no longer need to. A command that runs after another in the same process, e.g. expire after backup, uses the cached info rather than loading it again.

The helper calls cfgRepoFormatSet() when backup.info is loaded, so the repository format is set for every command that loads backup.info. Previously only expire called cfgRepoFormatSet(). An option that requires a higher repository format is now reported when backup starts rather than by expire after the backup completes.

Expire and restore got the repository storage outside of the per-repository error handling, so an error getting the storage of one repository stopped the command. Now the error is handled the same as any other error for the repository: expire continues with the next repository and restore warns and selects a backup from another repository.

Replace infoBackupLoadFileReconstruct() with infoBackupReconstruct(), which reconstructs backup info that is already loaded. Add cfgInitTotal() to report how many times the configuration has been initialized, which the helper uses to detect that the configuration was loaded again, e.g. when check switches stanzas.

In unit tests a harness shim clears the cached info at the start of each test statement so each statement loads the info files from the repository. Within a statement the shim checks that cached info matches the info file each time it is accessed.

^ permalink  raw  [nested|flat] 13+ messages in thread

* Re: [pgbackrest/pgbackrest] PR #2917: Add an info helper to load, cache, and save info files.
@ 2026-11-10 17:47  "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
  11 siblings, 0 replies; 13+ messages in thread

From: dwsteele (@dwsteele) @ 2026-11-10 17:47 UTC (permalink / raw)
  To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>

@dwsteele assigned @dwsteele.

^ permalink  raw  [nested|flat] 13+ messages in thread

* Re: [pgbackrest/pgbackrest] PR #2917: Add an info helper to load, cache, and save info files.
@ 2026-11-10 17:48  "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
  11 siblings, 0 replies; 13+ messages in thread

From: dwsteele (@dwsteele) @ 2026-11-10 17:48 UTC (permalink / raw)
  To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>

@dwsteele applied label "module [core]".

^ permalink  raw  [nested|flat] 13+ messages in thread

* Re: [pgbackrest/pgbackrest] PR #2917: Add an info helper to load, cache, and save info files.
@ 2026-11-10 17:48  "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
  11 siblings, 0 replies; 13+ messages in thread

From: dwsteele (@dwsteele) @ 2026-11-10 17:48 UTC (permalink / raw)
  To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>

@dwsteele applied label "enhancement".

^ permalink  raw  [nested|flat] 13+ messages in thread

* Re: [pgbackrest/pgbackrest] PR #2917: Add an info helper to load, cache, and save info files.
@ 2026-11-10 17:48  "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
  11 siblings, 0 replies; 13+ messages in thread

From: dwsteele (@dwsteele) @ 2026-11-10 17:48 UTC (permalink / raw)
  To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>

@dwsteele set milestone to "2.61.0".

^ permalink  raw  [nested|flat] 13+ messages in thread

* Re: [pgbackrest/pgbackrest] PR #2917 (approved): Add an info helper to load, cache, and save info files.
@ 2026-11-11 14:14  "hunleyd (@hunleyd)" <noreply+hunleyd@github.com>
  11 siblings, 0 replies; 13+ messages in thread

From: hunleyd (@hunleyd) @ 2026-11-11 14:14 UTC (permalink / raw)
  To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>



^ permalink  raw  [nested|flat] 13+ messages in thread

* Re: [pgbackrest/pgbackrest] PR #2917: Add an info helper to load, cache, and save info files.
@ 2026-11-13 09:46  "pgstef (@pgstef)" <noreply+pgstef@github.com>
  11 siblings, 0 replies; 13+ messages in thread

From: pgstef (@pgstef) @ 2026-11-13 09:46 UTC (permalink / raw)
  To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>

The helper approach looks good to me! However, if I'm reading the code correctly, there might be a small issue with the initialization order in `infoArchiveRepoIdxSave()` and `infoBackupRepoIdxSave()`.

Both currently pass their info accessor (`infoHelperArchive(repoIdx)` or `infoHelperBackup(repoIdx)`), `storageRepoIdxWrite(repoIdx)`, and `cfgCipherSpecMainIdx(repoIdx)` as arguments to the same save call. Since C doesn't guarantee argument evaluation order, the cipher spec can be resolved before the remote repository has provided its cipher options. This looks like the same ordering issue addressed by `078489fa3`.

On a PostgreSQL host where encryption is configured only on the remote repository host, this can cache a `none` cipher spec. Initializing storage afterwards updates the cipher options but leaves that cached spec unchanged. A `Set` followed by `Save` as the first repository access can therefore write the info file without encryption.

The current callers appear to initialize repository storage first, so this looks like a latent issue rather than a regression in an existing command. Still, since the helper is intended to handle this initialization itself, could we make the ordering explicit in both save functions, as the load paths already do?

In `infoArchiveRepoIdxSave()`, replace the save call with:

```c
InfoArchive *const infoArchive = infoHelperArchive(repoIdx);

// Initialize repo storage before resolving the cipher spec, even when the info is already cached.
const Storage *const storage = storageRepoIdxWrite(repoIdx);

infoArchiveSaveFile(infoArchive, storage, INFO_ARCHIVE_PATH_FILE_STR, cfgCipherSpecMainIdx(repoIdx));
```

And similarly in `infoBackupRepoIdxSave()`:

```c
InfoBackup *const infoBackup = infoHelperBackup(repoIdx);

// Initialize repo storage before resolving the cipher spec, even when the info is already cached.
const Storage *const storage = storageRepoIdxWrite(repoIdx);

infoBackupSaveFile(infoBackup, storage, INFO_BACKUP_PATH_FILE_STR, cfgCipherSpecMainIdx(repoIdx));
```

Does that make sense, or am I missing something about the initialization order?

^ permalink  raw  [nested|flat] 13+ messages in thread

* Re: [pgbackrest/pgbackrest] PR #2917: Add an info helper to load, cache, and save info files.
@ 2026-11-13 10:51  "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
  11 siblings, 0 replies; 13+ messages in thread

From: dwsteele (@dwsteele) @ 2026-11-13 10:51 UTC (permalink / raw)
  To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>

> there might be a small issue with the initialization order

And indeed there is. Fixed in 9c999eee2. This is a bit tricky but is the reason the calls were separated originally so I should have seen it.  Thank you!

^ permalink  raw  [nested|flat] 13+ messages in thread

* Re: [pgbackrest/pgbackrest] PR #2917 (approved): Add an info helper to load, cache, and save info files.
@ 2026-11-13 11:07  "pgstef (@pgstef)" <noreply+pgstef@github.com>
  11 siblings, 0 replies; 13+ messages in thread

From: pgstef (@pgstef) @ 2026-11-13 11:07 UTC (permalink / raw)
  To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>

> Fixed in [9c999ee](https://github.com/pgbackrest/pgbackrest/commit/9c999eee26aa6a41ffece0044102be21f66ccba7).

Thanks, the PR looks good to me now!

One small nitpick: I don't think the initialization-order fix in the save helpers has a dedicated regression test yet. Would it be worth adding one along the lines of the diff below?

```c
inline patch 1: test/define.yaml, test/src/module/info/helperTest.c (+89/-1 lines) download inline-1.patchdiff --git a/test/define.yaml b/test/define.yaml index f8d45cb80..379462586 100644 --- a/test/define.yaml +++ b/test/define.yaml @@ -726,7 +726,7 @@ unit: # -------------------------------------------------------------------------------------------------------------------------------- - name: info/helper - total: 1 + total: 2 harness: name: infoHelper shim: diff --git a/test/src/module/info/helperTest.c b/test/src/module/info/helperTest.c index ad2bd7d06..0b7aa08b5 100644 --- a/test/src/module/info/helperTest.c +++ b/test/src/module/info/helperTest.c @@ -1,13 +1,24 @@ /*********************************************************************************************************************************** Test Info Helper ***********************************************************************************************************************************/ +#include "common/crypto/cipherBlock.h" +#include "common/crypto/hash.h" #include "common/format/format.h" +#include "common/io/filter/sink.h" +#include "common/io/filter/size.h" +#include "config/protocol.h" +#include "info/helper.intern.h" +#include "protocol/helper.h" +#include "storage/remote/protocol.h" #include "storage/helper.h" +#include "storage/posix/storage.h" #include "harness/config.h" #include "harness/info.h" #include "harness/infoHelper.h" #include "harness/postgres.h" +#include "harness/protocol.h" +#include "harness/storage.h" /*********************************************************************************************************************************** Test Run @@ -163,5 +174,82 @@ testRun(void) TEST_RESULT_STR_Z(infoArchiveId(infoArchiveRepoIdx(1)), "14-1", "archive info loaded from file"); } + // ***************************************************************************************************************************** + if (testBegin("infoArchiveRepoIdxSave() as first repo access encrypts with cipher configured only on the repo host")) + { + const Storage *const storageTest = storagePosixNewP(TEST_PATH_STR, .write = true); + + static const ProtocolServerHandler testRemoteHandlerList[] = + { + PROTOCOL_SERVER_HANDLER_OPTION_LIST + PROTOCOL_SERVER_HANDLER_STORAGE_REMOTE_LIST + }; + + hrnProtocolRemoteShimInstall(LSTDEF(testRemoteHandlerList)); + + static const StorageRemoteFilterHandler storageRemoteFilterHandlerList[] = + { + {.type = CIPHER_BLOCK_FILTER_TYPE, .handlerParam = cipherBlockNewPack}, + {.type = CRYPTO_HASH_FILTER_TYPE, .handlerParam = cryptoHashNewPack}, + {.type = SINK_FILTER_TYPE, .handlerNoParam = ioSinkNew}, + {.type = SIZE_FILTER_TYPE, .handlerNoParam = ioSizeNew}, + }; + + storageRemoteFilterHandlerSet(LSTDEF(storageRemoteFilterHandlerList)); + + hrnInfoHelperStatementClearSet(false); + + HRN_STORAGE_PUT_Z( + storageTest, "repo-host.conf", "[global]\nrepo1-cipher-type=aes-256-cbc\nrepo1-cipher-pass=remotepass\n"); + + StringList *argList = strLstNew(); + hrnCfgArgRawZ(argList, cfgOptStanza, "db"); + hrnCfgArgRawZ(argList, cfgOptProtocolTimeout, "20"); + hrnCfgArgKeyRawZ(argList, cfgOptPgPath, 1, TEST_PATH "/pg"); + hrnCfgArgKeyRawZ(argList, cfgOptRepoHost, 1, "localhost"); + hrnCfgArgKeyRawZ(argList, cfgOptRepoHostUser, 1, TEST_USER); + hrnCfgArgKeyRawZ(argList, cfgOptRepoHostConfig, 1, TEST_PATH "/repo-host.conf"); + hrnCfgArgKeyRawZ(argList, cfgOptRepoPath, 1, TEST_PATH "/repo-enc"); + HRN_CFG_LOAD(cfgCmdStanzaCreate, argList); + + TEST_RESULT_UINT(cfgOptionIdxStrId(cfgOptRepoCipherType, 0), cipherTypeNone, "cipher type not configured locally"); + + TEST_RESULT_VOID( + infoArchiveRepoIdxSet(0, infoArchiveNew(PG_VERSION_14, 6569239123849665679, REPOSITORY_FORMAT_5, cipherSpecMapNew())), + "cache archive info"); + TEST_RESULT_VOID(infoArchiveRepoIdxSave(0), "save archive info as the first repo access"); + + TEST_RESULT_UINT(cfgOptionIdxStrId(cfgOptRepoCipherType, 0), cipherTypeAes256Cbc, "cipher type pulled from repo host"); + TEST_RESULT_UINT( + cipherSpecType(cfgCipherSpecMainIdx(0)), cipherTypeAes256Cbc, "cipher spec resolved after remote set cipher options"); + + const String *const content = strNewBuf( + storageGetP(storageNewReadP(storageTest, STRDEF("repo-enc/archive/db/archive.info")))); + + TEST_RESULT_BOOL(strBeginsWithZ(content, "[backrest]"), false, "archive.info written encrypted"); + + // ------------------------------------------------------------------------------------------------------------------------- + TEST_TITLE("control: storage obtained first gives an encrypted archive.info"); + + protocolFree(); + HRN_STORAGE_PATH_REMOVE(storageTest, "repo-enc", .recurse = true); + HRN_CFG_LOAD(cfgCmdStanzaCreate, argList); + + TEST_RESULT_VOID(storageRepoIdxWrite(0), "get repo storage first"); + TEST_RESULT_VOID( + infoArchiveRepoIdxSet(0, infoArchiveNew(PG_VERSION_14, 6569239123849665679, REPOSITORY_FORMAT_5, cipherSpecMapNew())), + "cache archive info"); + TEST_RESULT_VOID(infoArchiveRepoIdxSave(0), "save archive info"); + TEST_RESULT_UINT(cipherSpecType(cfgCipherSpecMainIdx(0)), cipherTypeAes256Cbc, "main cipher spec is aes"); + + TEST_RESULT_BOOL( + strBeginsWithZ( + strNewBuf(storageGetP(storageNewReadP(storageTest, STRDEF("repo-enc/archive/db/archive.info")))), "[backrest]"), + false, "archive.info encrypted"); + + protocolFree(); + hrnProtocolRemoteShimUninstall(); + } + FUNCTION_HARNESS_RETURN_VOID(); }
``` ^ permalink raw [nested|flat] 13+ messages in thread

* Re: [pgbackrest/pgbackrest] PR #2917: Add an info helper to load, cache, and save info files.
@ 2026-11-13 12:15  "pgstef (@pgstef)" <noreply+pgstef@github.com>
  11 siblings, 0 replies; 13+ messages in thread

From: pgstef (@pgstef) @ 2026-11-13 12:15 UTC (permalink / raw)
  To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>

21ee6ce looks great!

If I understand correctly, the order the compiler evaluates arguments in no longer matters: the save call can stay as one statement because the cipher spec now gets the storage itself.

The simplified regression test looks good too! 👍

^ permalink  raw  [nested|flat] 13+ messages in thread

* Re: [pgbackrest/pgbackrest] PR #2917: Add an info helper to load, cache, and save info files.
@ 2026-11-13 13:28  "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
  11 siblings, 0 replies; 13+ messages in thread

From: dwsteele (@dwsteele) @ 2026-11-13 13:28 UTC (permalink / raw)
  To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>

> If I understand correctly, the order the compiler evaluates arguments in no longer matters: the save call can stay as one statement because the cipher spec now gets the storage itself.

Correct. And I simplified the test a bit. It is good to verify that the helper correctly pulls remote credentials but the test as a limited ability to test the ordering issue. After all, the prior code already worked on multiple compilers and architectures with no issue. Compilers *can* reorder the calls but I think they strive not to in order to prevent this kind of bug.

^ permalink  raw  [nested|flat] 13+ messages in thread

* Re: [pgbackrest/pgbackrest] PR #2917: Add an info helper to load, cache, and save info files.
@ 2026-11-13 14:20  "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
  11 siblings, 0 replies; 13+ messages in thread

From: dwsteele (@dwsteele) @ 2026-11-13 14:20 UTC (permalink / raw)
  To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>

@dwsteele merged this (commit 66351527593f).

^ permalink  raw  [nested|flat] 13+ messages in thread

* Re: [pgbackrest/pgbackrest] PR #2917: Add an info helper to load, cache, and save info files.
@ 2026-11-13 14:20  "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
  11 siblings, 0 replies; 13+ messages in thread

From: dwsteele (@dwsteele) @ 2026-11-13 14:20 UTC (permalink / raw)
  To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>

@dwsteele closed this.

^ permalink  raw  [nested|flat] 13+ messages in thread


end of thread, other threads:[~2026-11-13 14:20 UTC | newest]

Thread overview: 13+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-11-10 17:47 [pgbackrest/pgbackrest] PR #2917: Add an info helper to load, cache, and save info files. "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
2026-11-10 17:47 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
2026-11-10 17:48 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
2026-11-10 17:48 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
2026-11-10 17:48 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
2026-11-11 14:14 ` Re: [pgbackrest/pgbackrest] PR #2917 (approved): Add an info helper to load, cache, and save info files. "hunleyd (@hunleyd)" <noreply+hunleyd@github.com>
2026-11-13 09:46 ` "pgstef (@pgstef)" <noreply+pgstef@github.com>
2026-11-13 10:51 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
2026-11-13 11:07 ` Re: [pgbackrest/pgbackrest] PR #2917 (approved): Add an info helper to load, cache, and save info files. "pgstef (@pgstef)" <noreply+pgstef@github.com>
2026-11-13 12:15 ` "pgstef (@pgstef)" <noreply+pgstef@github.com>
2026-11-13 13:28 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
2026-11-13 14:20 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
2026-11-13 14:20 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox