pg.ddx.io  pgbackrest/pgbackrest GitHub issues and pull requests  
help / color / mirror / Atom feed
From: Jyrgal (@Jyrgal) <noreply+Jyrgal@github.com>
To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>
Subject: [pgbackrest/pgbackrest] PR #2918: Add repo-storage-tmp-retention option.
Date: Thu, 12 Nov 2026 05:01:52 +0000
Message-ID: <gh-pgbackrest-pgbackrest-2918@github.com> (raw)

Original issue with context: #2913.

When `resume` is enabled the manifest is saved to `backup.manifest.copy` periodically during a backup. On S3 with Object Lock or Azure with version-level immutability and a default retention, each save creates a new version that is retained for the full default period. A single backup can leave many locked versions of a file that is only useful until the backup completes.

This adds an option to set an explicit, shorter retention on these temporary files that overrides the default.

**Changes**

- Add a generic `tmp` flag to `StorageNewWriteParam` / `StorageInterfaceNewWriteParam` for files that are not intended to last longer than the current operation. It is passed through `storageWriteNew()` and the remote protocol to the driver, which decides what to do with it. Posix, CIFS, SFTP, and GCS ignore it.
- Backup sets `.tmp = !final` in `backupManifestSaveCopy()`, so only checkpoint copies are tmp. The final `backup.manifest.copy` and `backup.manifest` keep the default retention.
- Add `repo-storage-tmp-retention` (valid for `s3` and `azure`). The drivers get it from their constructors and apply it only to tmp files:
  - **S3**: `x-amz-object-lock-mode` / `x-amz-object-lock-retain-until-date` on PutObject and CreateMultipartUpload.
  - **Azure**: `x-ms-immutability-policy-mode` / `x-ms-immutability-policy-until-date` on Put Blob and Put Block List.
- Add `repo-s3-tmp-retention-mode` (`governance`/`compliance`) and `repo-azure-tmp-retention-mode` (`unlocked`/`locked`). These are separate options because the config builder does not support a per-type `allow-list`.

**Not supported: GCS**

An object retention cannot shorten a bucket retention policy (the object is retained until both are satisfied). Without a bucket policy, it would add retention that prevents the next checkpoint save from replacing the object. This is noted in the option docs.

inline patch 1: build/config.yaml, doc/xml/reference.xml, doc/xml/release/2020s/2027/2.61.0.xml and 30 more (+731/-36 lines) download inline-1.patchdiff --git a/build/config.yaml b/build/config.yaml index d893815786..1e681e086d 100644 --- a/build/config.yaml +++ b/build/config.yaml @@ -2280,6 +2280,19 @@ option: - gcs - s3 + repo-storage-tmp-retention: + section: global + group: repo + type: time + required: false + allow-range: [1m, 36500d] + command: repo-type + depend: + option: repo-type + list: + - azure + - s3 + # Repository Azure storage options #--------------------------------------------------------------------------------------------------------------------------------- repo-azure-account: @@ -2337,6 +2350,19 @@ option: command: repo-type depend: repo-azure-account + repo-azure-tmp-retention-mode: + section: global + group: repo + type: string-id + sequence: true + default: unlocked + # Keep synced with StorageAzureImmutabilityPolicyMode enum + allow-list: + - unlocked + - locked + command: repo-type + depend: repo-azure-account + # Repository GCS storage options #--------------------------------------------------------------------------------------------------------------------------------- repo-gcs-bucket: @@ -2447,6 +2473,16 @@ option: inherit: repo-s3-bucket required: false + repo-s3-tmp-retention-mode: + inherit: repo-s3-bucket + type: string-id + sequence: true + default: governance + # Keep synced with StorageS3ObjectLockMode enum + allow-list: + - governance + - compliance + repo-s3-sse-customer-key: inherit: repo-s3-bucket required: false diff --git a/doc/xml/reference.xml b/doc/xml/reference.xml index c06240f205..8d60f5e718 100644 --- a/doc/xml/reference.xml +++ b/doc/xml/reference.xml @@ -613,6 +613,21 @@ <example>sas</example> </config-key> + <config-key id="repo-azure-tmp-retention-mode" name="Azure Repository Temporary File Retention Mode"> + <summary>Azure temporary file retention mode.</summary> + + <text> + <p>Immutability policy mode applied to temporary files when <br-option>repo-storage-tmp-retention</br-option> is set.</p> + + <allow-list caption="retention modes"> + <allow-item id="unlocked">The retention can be shortened, extended, or removed.</allow-item> + <allow-item id="locked">The retention can be extended but cannot be shortened or removed until it expires.</allow-item> + </allow-list> + </text> + + <example>locked</example> + </config-key> + <config-key id="repo-azure-uri-style" name="Azure Repository URI Style"> <summary>Azure URI Style.</summary> @@ -1089,6 +1104,21 @@ <example>bceb4f13-6939-4be3-910d-df54dee817b7</example> </config-key> + <config-key id="repo-s3-tmp-retention-mode" name="S3 Repository Temporary File Retention Mode"> + <summary>S3 temporary file retention mode.</summary> + + <text> + <p>Object Lock retention mode applied to temporary files when <br-option>repo-storage-tmp-retention</br-option> is set.</p> + + <allow-list caption="retention modes"> + <allow-item id="governance">Users with the required permissions can shorten or remove the retention.</allow-item> + <allow-item id="compliance">The retention cannot be shortened or removed by any user until it expires.</allow-item> + </allow-list> + </text> + + <example>compliance</example> + </config-key> + <config-key id="repo-s3-bucket" name="S3 Repository Bucket"> <summary>S3 repository bucket.</summary> @@ -1380,6 +1410,24 @@ <example>key1=value1</example> </config-key> + <config-key id="repo-storage-tmp-retention" name="Repository Storage Temporary File Retention"> + <summary>Repository temporary file retention.</summary> + + <text> + <p>Retention period for temporary files written to the repository. Temporary files are not intended to last longer than the current operation, e.g. the checkpoint copies of <file>backup.manifest.copy</file> that are saved periodically during a backup when <br-option>resume</br-option> is enabled. When the storage is configured to retain every version of a file for a default period, each save creates a new version that is retained for that period, which may be far longer than these files are useful.</p> + + <p>When this option is set, <backrest/> sets an explicit retention on temporary files that overrides the default retention of the storage. Other files, including the final <file>backup.manifest.copy</file> and <file>backup.manifest</file>, are not affected and continue to use the default retention. The retention must be between one minute and 36500 days.</p> + + <p>For <proper>S3</proper> an Object Lock retention is set with the mode from <br-option>repo-s3-tmp-retention-mode</br-option>. Object Lock must be enabled on the bucket and the credentials must have the <id>s3:PutObjectRetention</id> permission or writes will fail.</p> + + <p>For <proper>Azure</proper> an immutability policy is set with the mode from <br-option>repo-azure-tmp-retention-mode</br-option>. Version-level immutability must be enabled on the container or account or writes will fail.</p> + + <p>This option is not supported for <proper>GCS</proper> since an object retention cannot shorten the retention policy of a bucket.</p> + </text> + + <example>1d</example> + </config-key> + <config-key id="repo-storage-upload-chunk-size" name="Repository Storage Upload Chunk Size"> <summary>Repository storage upload chunk size.</summary> diff --git a/doc/xml/release/2020s/2027/2.61.0.xml b/doc/xml/release/2020s/2027/2.61.0.xml index 883c5e19cd..c43795f354 100644 --- a/doc/xml/release/2020s/2027/2.61.0.xml +++ b/doc/xml/release/2020s/2027/2.61.0.xml @@ -106,6 +106,19 @@ <p>Set <br-option>tls-min-version=1.3</br-option> to allow only TLS 1.3, e.g. for compliance. The default is <id>1.2</id>. A higher minimum set in the <proper>OpenSSL</proper> config is not lowered. The option is ignored for repository storage connections when <br-option>repo-storage-verify-tls=n</br-option>.</p> </release-item> + + <release-item> + <github-issue id="2913"/> + <github-pull-request id="2918"/> + + <release-item-contributor-list> + <release-item-contributor id="james.pan"/> + </release-item-contributor-list> + + <p>Add <br-option>repo-storage-tmp-retention</br-option> option.</p> + + <p>When <br-option>resume</br-option> is enabled the manifest is saved to <file>backup.manifest.copy</file> periodically during a backup. On <proper>S3</proper> with Object Lock or <proper>Azure</proper> with version-level immutability and a default retention, each save creates a new version that is retained for the full default period. This option sets an explicit, typically shorter, retention on temporary files such as these checkpoint copies that overrides the default. The mode is set with <br-option>repo-s3-tmp-retention-mode</br-option> or <br-option>repo-azure-tmp-retention-mode</br-option>. The final <file>backup.manifest.copy</file> and <file>backup.manifest</file> are not affected.</p> + </release-item> </release-feature-list> <release-improvement-list> diff --git a/doc/xml/release/contributor.xml b/doc/xml/release/contributor.xml index 072dd292d8..749a0fd854 100644 --- a/doc/xml/release/contributor.xml +++ b/doc/xml/release/contributor.xml @@ -591,6 +591,11 @@ <contributor-id type="github">jcoleman</contributor-id> </contributor> +<contributor id="james.pan"> + <contributor-name-display>James Pan</contributor-name-display> + <contributor-id type="github">Jyrgal</contributor-id> +</contributor> + <contributor id="james.sewell"> <contributor-name-display>James Sewell</contributor-name-display> <contributor-id type="github">jamessewell</contributor-id> diff --git a/src/command/backup/complete.c.inc b/src/command/backup/complete.c.inc index 6140d9721f..b3218fd624 100644 --- a/src/command/backup/complete.c.inc +++ b/src/command/backup/complete.c.inc @@ -17,12 +17,13 @@ backupManifestSaveCopy(Manifest *const manifest, const CipherSpec *const cipherS { MEM_CONTEXT_TEMP_BEGIN() { - // Open file for write + // Open file for write. Checkpoint copies are tmp since they will be overwritten by the next save. IoWrite *const write = storageWriteIo( storageNewWriteP( storageRepoWrite(), strNewFmt( - STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE INFO_COPY_EXT, strZ(manifestData(manifest)->backupLabel)))); + STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE INFO_COPY_EXT, strZ(manifestData(manifest)->backupLabel)), + .tmp = !final)); // Add encryption filter if required cipherBlockFilterGroupAdd(ioWriteFilterGroup(write), cipherModeEncrypt, cipherSpecManifest); diff --git a/src/config/config.auto.h b/src/config/config.auto.h index 9ad6e5b780..399de4c204 100644 --- a/src/config/config.auto.h +++ b/src/config/config.auto.h @@ -147,7 +147,7 @@ Option constants #define CFGOPT_VERBOSE "verbose" #define CFGOPT_VERSION "version" -#define CFG_OPTION_TOTAL 203 +#define CFG_OPTION_TOTAL 206 /*********************************************************************************************************************************** Option value constants @@ -294,6 +294,13 @@ Option value constants #define CFGOPTVAL_REPO_AZURE_KEY_TYPE_AUTO_STRID STRID5S("auto", 2, 0x7d2a16) #define CFGOPTVAL_REPO_AZURE_KEY_TYPE_AUTO_Z "auto" +#define CFGOPTVAL_REPO_AZURE_TMP_RETENTION_MODE_UNLOCKED 0 +#define CFGOPTVAL_REPO_AZURE_TMP_RETENTION_MODE_UNLOCKED_STRID STRID5S("unlocked", 0, 0x215637b1d52) +#define CFGOPTVAL_REPO_AZURE_TMP_RETENTION_MODE_UNLOCKED_Z "unlocked" +#define CFGOPTVAL_REPO_AZURE_TMP_RETENTION_MODE_LOCKED 1 +#define CFGOPTVAL_REPO_AZURE_TMP_RETENTION_MODE_LOCKED_STRID STRID5S("locked", 1, 0x8558dec4) +#define CFGOPTVAL_REPO_AZURE_TMP_RETENTION_MODE_LOCKED_Z "locked" + #define CFGOPTVAL_REPO_AZURE_URI_STYLE_HOST 0 #define CFGOPTVAL_REPO_AZURE_URI_STYLE_HOST_STRID STRID5S("host", 0, 0xa4de82) #define CFGOPTVAL_REPO_AZURE_URI_STYLE_HOST_Z "host" @@ -354,6 +361,13 @@ Option value constants #define CFGOPTVAL_REPO_S3_KEY_TYPE_PROCESS_STRID STRID5S("process", 4, 0x4e651be50a) #define CFGOPTVAL_REPO_S3_KEY_TYPE_PROCESS_Z "process" +#define CFGOPTVAL_REPO_S3_TMP_RETENTION_MODE_GOVERNANCE 0 +#define CFGOPTVAL_REPO_S3_TMP_RETENTION_MODE_GOVERNANCE_STRID STRID5S("governance", 0, 0xa3705d22d9e72) +#define CFGOPTVAL_REPO_S3_TMP_RETENTION_MODE_GOVERNANCE_Z "governance" +#define CFGOPTVAL_REPO_S3_TMP_RETENTION_MODE_COMPLIANCE 1 +#define CFGOPTVAL_REPO_S3_TMP_RETENTION_MODE_COMPLIANCE_STRID STRID5S("compliance", 1, 0xa37052c835e34) +#define CFGOPTVAL_REPO_S3_TMP_RETENTION_MODE_COMPLIANCE_Z "compliance" + #define CFGOPTVAL_REPO_S3_URI_STYLE_HOST 0 #define CFGOPTVAL_REPO_S3_URI_STYLE_HOST_STRID STRID5S("host", 0, 0xa4de82) #define CFGOPTVAL_REPO_S3_URI_STYLE_HOST_Z "host" @@ -580,6 +594,7 @@ typedef enum cfgOptRepoAzureEndpoint, cfgOptRepoAzureKey, cfgOptRepoAzureKeyType, + cfgOptRepoAzureTmpRetentionMode, cfgOptRepoAzureUriStyle, cfgOptRepoBlock, cfgOptRepoBlockAgeMap, @@ -633,6 +648,7 @@ typedef enum cfgOptRepoS3Service, cfgOptRepoS3SseCustomerKey, cfgOptRepoS3StsHost, + cfgOptRepoS3TmpRetentionMode, cfgOptRepoS3Token, cfgOptRepoS3UriStyle, cfgOptRepoSftpHost, @@ -652,6 +668,7 @@ typedef enum cfgOptRepoStoragePrefetch, cfgOptRepoStorageReadOver, cfgOptRepoStorageTag, + cfgOptRepoStorageTmpRetention, cfgOptRepoStorageUploadChunkSize, cfgOptRepoStorageVerifyTls, cfgOptRepoSymlink, diff --git a/src/config/parse.auto.c.inc b/src/config/parse.auto.c.inc index 7579f27e9a..f704b0fd06 100644 --- a/src/config/parse.auto.c.inc +++ b/src/config/parse.auto.c.inc @@ -59,6 +59,7 @@ static const StringPubConst parseRuleValueStr[] = PARSE_RULE_STRPUB("32KiB"), // val/str PARSE_RULE_STRPUB("360"), // val/str PARSE_RULE_STRPUB("3600"), // val/str + PARSE_RULE_STRPUB("36500d"), // val/str PARSE_RULE_STRPUB("443"), // val/str PARSE_RULE_STRPUB("4MiB"), // val/str PARSE_RULE_STRPUB("4PiB"), // val/str @@ -87,6 +88,7 @@ static const StringPubConst parseRuleValueStr[] = PARSE_RULE_STRPUB("blob.core.windows.net"), // val/str PARSE_RULE_STRPUB("bz2"), // val/str PARSE_RULE_STRPUB("cifs"), // val/str + PARSE_RULE_STRPUB("compliance"), // val/str PARSE_RULE_STRPUB("count"), // val/str PARSE_RULE_STRPUB("debug"), // val/str PARSE_RULE_STRPUB("default"), // val/str @@ -97,6 +99,7 @@ static const StringPubConst parseRuleValueStr[] = PARSE_RULE_STRPUB("fingerprint"), // val/str PARSE_RULE_STRPUB("full"), // val/str PARSE_RULE_STRPUB("gcs"), // val/str + PARSE_RULE_STRPUB("governance"), // val/str PARSE_RULE_STRPUB("gz"), // val/str PARSE_RULE_STRPUB("host"), // val/str PARSE_RULE_STRPUB("immediate"), // val/str @@ -105,6 +108,7 @@ static const StringPubConst parseRuleValueStr[] = PARSE_RULE_STRPUB("json"), // val/str PARSE_RULE_STRPUB("latest"), // val/str PARSE_RULE_STRPUB("localhost"), // val/str + PARSE_RULE_STRPUB("locked"), // val/str PARSE_RULE_STRPUB("lsn"), // val/str PARSE_RULE_STRPUB("lz4"), // val/str PARSE_RULE_STRPUB("md5"), // val/str @@ -144,6 +148,7 @@ static const StringPubConst parseRuleValueStr[] = PARSE_RULE_STRPUB("tls"), // val/str PARSE_RULE_STRPUB("token"), // val/str PARSE_RULE_STRPUB("trace"), // val/str + PARSE_RULE_STRPUB("unlocked"), // val/str PARSE_RULE_STRPUB("warn"), // val/str PARSE_RULE_STRPUB("web-id"), // val/str PARSE_RULE_STRPUB("xid"), // val/str @@ -203,6 +208,7 @@ typedef enum parseRuleValStrQT_32KiB_QT, // val/str/enum parseRuleValStrQT_360_QT, // val/str/enum parseRuleValStrQT_3600_QT, // val/str/enum + parseRuleValStrQT_36500d_QT, // val/str/enum parseRuleValStrQT_443_QT, // val/str/enum parseRuleValStrQT_4MiB_QT, // val/str/enum parseRuleValStrQT_4PiB_QT, // val/str/enum @@ -231,6 +237,7 @@ typedef enum parseRuleValStrQT_blob_DT_core_DT_windows_DT_net_QT, // val/str/enum parseRuleValStrQT_bz2_QT, // val/str/enum parseRuleValStrQT_cifs_QT, // val/str/enum + parseRuleValStrQT_compliance_QT, // val/str/enum parseRuleValStrQT_count_QT, // val/str/enum parseRuleValStrQT_debug_QT, // val/str/enum parseRuleValStrQT_default_QT, // val/str/enum @@ -241,6 +248,7 @@ typedef enum parseRuleValStrQT_fingerprint_QT, // val/str/enum parseRuleValStrQT_full_QT, // val/str/enum parseRuleValStrQT_gcs_QT, // val/str/enum + parseRuleValStrQT_governance_QT, // val/str/enum parseRuleValStrQT_gz_QT, // val/str/enum parseRuleValStrQT_host_QT, // val/str/enum parseRuleValStrQT_immediate_QT, // val/str/enum @@ -249,6 +257,7 @@ typedef enum parseRuleValStrQT_json_QT, // val/str/enum parseRuleValStrQT_latest_QT, // val/str/enum parseRuleValStrQT_localhost_QT, // val/str/enum + parseRuleValStrQT_locked_QT, // val/str/enum parseRuleValStrQT_lsn_QT, // val/str/enum parseRuleValStrQT_lz4_QT, // val/str/enum parseRuleValStrQT_md5_QT, // val/str/enum @@ -288,6 +297,7 @@ typedef enum parseRuleValStrQT_tls_QT, // val/str/enum parseRuleValStrQT_token_QT, // val/str/enum parseRuleValStrQT_trace_QT, // val/str/enum + parseRuleValStrQT_unlocked_QT, // val/str/enum parseRuleValStrQT_warn_QT, // val/str/enum parseRuleValStrQT_web_DS_id_QT, // val/str/enum parseRuleValStrQT_xid_QT, // val/str/enum @@ -495,6 +505,7 @@ static const int64_t parseRuleValueTime[] = 604800000, // val/time 1296000000, // val/time 7776000000, // val/time + 3153600000000, // val/time 863999913600000, // val/time }; @@ -512,6 +523,7 @@ static const uint8_t parseRuleValueTimeStrMap[] = parseRuleValStrQT_7d_QT, // val/time/strmap parseRuleValStrQT_15d_QT, // val/time/strmap parseRuleValStrQT_90d_QT, // val/time/strmap + parseRuleValStrQT_36500d_QT, // val/time/strmap parseRuleValStrQT_9999999d_QT, // val/time/strmap }; @@ -529,6 +541,7 @@ typedef enum parseRuleValTime7d, // val/time/enum parseRuleValTime15d, // val/time/enum parseRuleValTime90d, // val/time/enum + parseRuleValTime36500d, // val/time/enum parseRuleValTime9999999d, // val/time/enum } ParseRuleValueTime; @@ -5532,6 +5545,95 @@ static const ParseRuleOption parseRuleOption[CFG_OPTION_TOTAL] = ), // opt/repo-azure-key-type ), // opt/repo-azure-key-type // ----------------------------------------------------------------------------------------------------------------------------- + PARSE_RULE_OPTION // opt/repo-azure-tmp-retention-mode + ( // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_NAME("repo-azure-tmp-retention-mode"), // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_TYPE(StringId), // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_SEQUENCE(true), // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_RESET(true), // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_SECTION(Global), // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-azure-tmp-retention-mode + // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-azure-tmp-retention-mode + ( // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-tmp-retention-mode + ), // opt/repo-azure-tmp-retention-mode + // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-azure-tmp-retention-mode + ( // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-tmp-retention-mode + ), // opt/repo-azure-tmp-retention-mode + // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-azure-tmp-retention-mode + ( // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-tmp-retention-mode + ), // opt/repo-azure-tmp-retention-mode + // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-azure-tmp-retention-mode + ( // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-tmp-retention-mode + ), // opt/repo-azure-tmp-retention-mode + // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTIONAL // opt/repo-azure-tmp-retention-mode + ( // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTIONAL_GROUP // opt/repo-azure-tmp-retention-mode + ( // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTIONAL_DEPEND // opt/repo-azure-tmp-retention-mode + ( // opt/repo-azure-tmp-retention-mode + PARSE_RULE_VAL_OPT(RepoType), // opt/repo-azure-tmp-retention-mode + PARSE_RULE_VAL_STRID(azure), // opt/repo-azure-tmp-retention-mode + ), // opt/repo-azure-tmp-retention-mode + // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/repo-azure-tmp-retention-mode + ( // opt/repo-azure-tmp-retention-mode + PARSE_RULE_VAL_STRID(unlocked), // opt/repo-azure-tmp-retention-mode + PARSE_RULE_VAL_STRID(locked), // opt/repo-azure-tmp-retention-mode + ), // opt/repo-azure-tmp-retention-mode + // opt/repo-azure-tmp-retention-mode + PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-azure-tmp-retention-mode + ( // opt/repo-azure-tmp-retention-mode + PARSE_RULE_VAL_STRID(unlocked), // opt/repo-azure-tmp-retention-mode + ), // opt/repo-azure-tmp-retention-mode + ), // opt/repo-azure-tmp-retention-mode + ), // opt/repo-azure-tmp-retention-mode + ), // opt/repo-azure-tmp-retention-mode + // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-azure-uri-style ( // opt/repo-azure-uri-style PARSE_RULE_OPTION_NAME("repo-azure-uri-style"), // opt/repo-azure-uri-style @@ -8674,6 +8776,95 @@ static const ParseRuleOption parseRuleOption[CFG_OPTION_TOTAL] = ), // opt/repo-s3-sts-host ), // opt/repo-s3-sts-host // ----------------------------------------------------------------------------------------------------------------------------- + PARSE_RULE_OPTION // opt/repo-s3-tmp-retention-mode + ( // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_NAME("repo-s3-tmp-retention-mode"), // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_TYPE(StringId), // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_SEQUENCE(true), // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-tmp-retention-mode + // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-tmp-retention-mode + ( // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-tmp-retention-mode + ), // opt/repo-s3-tmp-retention-mode + // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-tmp-retention-mode + ( // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-tmp-retention-mode + ), // opt/repo-s3-tmp-retention-mode + // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-tmp-retention-mode + ( // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-tmp-retention-mode + ), // opt/repo-s3-tmp-retention-mode + // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-tmp-retention-mode + ( // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-tmp-retention-mode + ), // opt/repo-s3-tmp-retention-mode + // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTIONAL // opt/repo-s3-tmp-retention-mode + ( // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-tmp-retention-mode + ( // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-tmp-retention-mode + ( // opt/repo-s3-tmp-retention-mode + PARSE_RULE_VAL_OPT(RepoType), // opt/repo-s3-tmp-retention-mode + PARSE_RULE_VAL_STRID(s3), // opt/repo-s3-tmp-retention-mode + ), // opt/repo-s3-tmp-retention-mode + // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/repo-s3-tmp-retention-mode + ( // opt/repo-s3-tmp-retention-mode + PARSE_RULE_VAL_STRID(governance), // opt/repo-s3-tmp-retention-mode + PARSE_RULE_VAL_STRID(compliance), // opt/repo-s3-tmp-retention-mode + ), // opt/repo-s3-tmp-retention-mode + // opt/repo-s3-tmp-retention-mode + PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-s3-tmp-retention-mode + ( // opt/repo-s3-tmp-retention-mode + PARSE_RULE_VAL_STRID(governance), // opt/repo-s3-tmp-retention-mode + ), // opt/repo-s3-tmp-retention-mode + ), // opt/repo-s3-tmp-retention-mode + ), // opt/repo-s3-tmp-retention-mode + ), // opt/repo-s3-tmp-retention-mode + // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-token ( // opt/repo-s3-token PARSE_RULE_OPTION_NAME("repo-s3-token"), // opt/repo-s3-token @@ -10231,6 +10422,90 @@ static const ParseRuleOption parseRuleOption[CFG_OPTION_TOTAL] = ), // opt/repo-storage-tag ), // opt/repo-storage-tag // ----------------------------------------------------------------------------------------------------------------------------- + PARSE_RULE_OPTION // opt/repo-storage-tmp-retention + ( // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_NAME("repo-storage-tmp-retention"), // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_TYPE(Time), // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_RESET(true), // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_SECTION(Global), // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-storage-tmp-retention + // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-storage-tmp-retention + ( // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-tmp-retention + ), // opt/repo-storage-tmp-retention + // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-storage-tmp-retention + ( // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-tmp-retention + ), // opt/repo-storage-tmp-retention + // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-storage-tmp-retention + ( // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-tmp-retention + ), // opt/repo-storage-tmp-retention + // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-storage-tmp-retention + ( // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-tmp-retention + PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-tmp-retention + ), // opt/repo-storage-tmp-retention + // opt/repo-storage-tmp-retention + PARSE_RULE_OPTIONAL // opt/repo-storage-tmp-retention + ( // opt/repo-storage-tmp-retention + PARSE_RULE_OPTIONAL_GROUP // opt/repo-storage-tmp-retention + ( // opt/repo-storage-tmp-retention + PARSE_RULE_OPTIONAL_DEPEND // opt/repo-storage-tmp-retention + ( // opt/repo-storage-tmp-retention + PARSE_RULE_VAL_OPT(RepoType), // opt/repo-storage-tmp-retention + PARSE_RULE_VAL_STRID(azure), // opt/repo-storage-tmp-retention + PARSE_RULE_VAL_STRID(s3), // opt/repo-storage-tmp-retention + ), // opt/repo-storage-tmp-retention + // opt/repo-storage-tmp-retention + PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo-storage-tmp-retention + ( // opt/repo-storage-tmp-retention + PARSE_RULE_VAL_TIME(1m), // opt/repo-storage-tmp-retention + PARSE_RULE_VAL_TIME(36500d), // opt/repo-storage-tmp-retention + ), // opt/repo-storage-tmp-retention + ), // opt/repo-storage-tmp-retention + ), // opt/repo-storage-tmp-retention + ), // opt/repo-storage-tmp-retention + // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-storage-upload-chunk-size ( // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_NAME("repo-storage-upload-chunk-size"), // opt/repo-storage-upload-chunk-size @@ -12356,6 +12631,7 @@ static const uint8_t optionResolveOrder[] = cfgOptRepoAzureContainer, // opt-resolve-order cfgOptRepoAzureEndpoint, // opt-resolve-order cfgOptRepoAzureKeyType, // opt-resolve-order + cfgOptRepoAzureTmpRetentionMode, // opt-resolve-order cfgOptRepoAzureUriStyle, // opt-resolve-order cfgOptRepoBlock, // opt-resolve-order cfgOptRepoBlockAgeMap, // opt-resolve-order @@ -12383,6 +12659,7 @@ static const uint8_t optionResolveOrder[] = cfgOptRepoS3Service, // opt-resolve-order cfgOptRepoS3SseCustomerKey, // opt-resolve-order cfgOptRepoS3StsHost, // opt-resolve-order + cfgOptRepoS3TmpRetentionMode, // opt-resolve-order cfgOptRepoS3Token, // opt-resolve-order cfgOptRepoS3UriStyle, // opt-resolve-order cfgOptRepoSftpHost, // opt-resolve-order @@ -12401,6 +12678,7 @@ static const uint8_t optionResolveOrder[] = cfgOptRepoStoragePrefetch, // opt-resolve-order cfgOptRepoStorageReadOver, // opt-resolve-order cfgOptRepoStorageTag, // opt-resolve-order + cfgOptRepoStorageTmpRetention, // opt-resolve-order cfgOptRepoStorageUploadChunkSize, // opt-resolve-order cfgOptRepoStorageVerifyTls, // opt-resolve-order cfgOptRepoSymlink, // opt-resolve-order diff --git a/src/storage/azure/helper.c b/src/storage/azure/helper.c index c1376e1d8f..c63ae19d46 100644 --- a/src/storage/azure/helper.c +++ b/src/storage/azure/helper.c @@ -76,6 +76,9 @@ storageAzureHelper(const unsigned int repoIdx, const bool write, StoragePathExpr if (cfgOptionIdxSource(cfgOptRepoStoragePort, repoIdx) != cfgSourceDefault) port = cfgOptionIdxUInt(cfgOptRepoStoragePort, repoIdx); + // Immutability policy retention for tmp files + const bool tmpRetention = cfgOptionIdxTest(cfgOptRepoStorageTmpRetention, repoIdx); + MEM_CONTEXT_PRIOR_BEGIN() { result = storageAzureNew( @@ -84,7 +87,10 @@ storageAzureHelper(const unsigned int repoIdx, const bool write, StoragePathExpr (size_t)cfgOptionIdxUInt64(cfgOptRepoStorageUploadChunkSize, repoIdx), cfgOptionIdxKvNull(cfgOptRepoStorageTag, repoIdx), endpoint, uriStyle, port, ioTimeoutMs(), protocolType, cfgOptionIdxBool(cfgOptRepoStorageVerifyTls, repoIdx), cfgOptionIdxStrNull(cfgOptRepoStorageCaFile, repoIdx), - cfgOptionIdxStrNull(cfgOptRepoStorageCaPath, repoIdx), cfgOptionIdxUInt(cfgOptRepoStoragePrefetch, repoIdx), + cfgOptionIdxStrNull(cfgOptRepoStorageCaPath, repoIdx), + tmpRetention ? cfgOptionIdxUInt64(cfgOptRepoStorageTmpRetention, repoIdx) : 0, + (StorageAzureImmutabilityPolicyMode)cfgOptionIdxSeq(cfgOptRepoAzureTmpRetentionMode, repoIdx), + cfgOptionIdxUInt(cfgOptRepoStoragePrefetch, repoIdx), cfgOptionIdxUInt64(cfgOptRepoStorageReadOver, repoIdx)); } MEM_CONTEXT_PRIOR_END(); diff --git a/src/storage/azure/storage.c b/src/storage/azure/storage.c index dd1ebfd7d7..b3e61c259a 100644 --- a/src/storage/azure/storage.c +++ b/src/storage/azure/storage.c @@ -30,6 +30,10 @@ Defaults Azure http headers ***********************************************************************************************************************************/ STRING_STATIC(AZURE_HEADER_TAGS, "x-ms-tags"); +STRING_STATIC(AZURE_HEADER_IMMUTABILITY_POLICY_MODE_STR, "x-ms-immutability-policy-mode"); +STRING_STATIC(AZURE_HEADER_IMMUTABILITY_POLICY_MODE_UNLOCKED_STR, "unlocked"); +STRING_STATIC(AZURE_HEADER_IMMUTABILITY_POLICY_MODE_LOCKED_STR, "locked"); +STRING_STATIC(AZURE_HEADER_IMMUTABILITY_POLICY_UNTIL_DATE_STR, "x-ms-immutability-policy-until-date"); STRING_STATIC(AZURE_HEADER_VERSION_STR, "x-ms-version"); STRING_STATIC(AZURE_HEADER_VERSION_VALUE_STR, "2024-08-04"); @@ -105,6 +109,8 @@ struct StorageAzure size_t blockSize; // Block size for multi-block upload unsigned int deleteMax; // Maximum objects that can be deleted in one request const String *tag; // Tags to be applied to objects + TimeMSec tmpRetention; // Immutability policy retention for tmp files (0 = default) + StorageAzureImmutabilityPolicyMode tmpRetentionMode; // Immutability policy mode for tmp files const String *pathPrefix; // Account/container prefix uint64_t fileId; // Id to used to make file block identifiers unique @@ -303,6 +309,7 @@ storageAzureRequestAsync(StorageAzure *const this, const String *const verb, Sto FUNCTION_LOG_PARAM(BUFFER, param.content); FUNCTION_LOG_PARAM(LIST, param.contentList); FUNCTION_LOG_PARAM(BOOL, param.tag); + FUNCTION_LOG_PARAM(BOOL, param.immutabilityPolicy); FUNCTION_LOG_END(); ASSERT(this != NULL); @@ -364,6 +371,19 @@ storageAzureRequestAsync(StorageAzure *const this, const String *const verb, Sto if (param.tag && this->tag != NULL) httpHeaderPut(requestHeader, AZURE_HEADER_TAGS, this->tag); + // Set immutability policy for tmp files when configured. A policy set on upload overrides the default policy configured + // on the container or account. + if (param.immutabilityPolicy && this->tmpRetention != 0) + { + httpHeaderPut( + requestHeader, AZURE_HEADER_IMMUTABILITY_POLICY_MODE_STR, + this->tmpRetentionMode == storageAzureImmutabilityPolicyModeLocked ? + AZURE_HEADER_IMMUTABILITY_POLICY_MODE_LOCKED_STR : AZURE_HEADER_IMMUTABILITY_POLICY_MODE_UNLOCKED_STR); + httpHeaderPut( + requestHeader, AZURE_HEADER_IMMUTABILITY_POLICY_UNTIL_DATE_STR, + httpDateFromTime(time(NULL) + (time_t)(this->tmpRetention / MSEC_PER_SEC))); + } + // Encode path const String *const path = httpUriEncode(param.path, true); @@ -433,11 +453,12 @@ storageAzureRequest(StorageAzure *const this, const String *const verb, const St FUNCTION_LOG_PARAM(BOOL, param.allowMissing); FUNCTION_LOG_PARAM(BOOL, param.contentIo); FUNCTION_LOG_PARAM(BOOL, param.tag); + FUNCTION_LOG_PARAM(BOOL, param.immutabilityPolicy); FUNCTION_LOG_END(); HttpRequest *const request = storageAzureRequestAsyncP( this, verb, .path = param.path, .header = param.header, .query = param.query, .content = param.content, - .contentList = param.contentList, .tag = param.tag); + .contentList = param.contentList, .tag = param.tag, .immutabilityPolicy = param.immutabilityPolicy); HttpResponse *const result = storageAzureResponseP(request, .allowMissing = param.allowMissing, .contentIo = param.contentIo); httpRequestFree(request); @@ -766,7 +787,7 @@ storageAzureNewWrite(THIS_VOID, const String *const file, const StorageInterface FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_AZURE, this); FUNCTION_LOG_PARAM(STRING, file); - (void)param; // No parameters are used + FUNCTION_LOG_PARAM(BOOL, param.tmp); FUNCTION_LOG_END(); ASSERT(this != NULL); @@ -777,7 +798,7 @@ storageAzureNewWrite(THIS_VOID, const String *const file, const StorageInterface ASSERT(param.group == NULL); ASSERT(param.timeModified == 0); - FUNCTION_LOG_RETURN(STORAGE_WRITE_AZURE, storageWriteAzureNew(this, file, this->fileId++, this->blockSize)); + FUNCTION_LOG_RETURN(STORAGE_WRITE_AZURE, storageWriteAzureNew(this, file, this->fileId++, this->blockSize, param.tmp)); } /**********************************************************************************************************************************/ @@ -1003,7 +1024,8 @@ storageAzureNew( const String *const container, const String *const account, const StorageAzureKeyType keyType, const String *const key, const size_t blockSize, const KeyValue *const tag, const String *const endpoint, const StorageAzureUriStyle uriStyle, const unsigned int port, const TimeMSec timeout, const HttpProtocolType protocolType, const bool verifyPeer, - const String *const caFile, const String *const caPath, const unsigned int prefetch, const uint64_t readOver) + const String *const caFile, const String *const caPath, const TimeMSec tmpRetention, + const StorageAzureImmutabilityPolicyMode tmpRetentionMode, const unsigned int prefetch, const uint64_t readOver) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, path); @@ -1024,6 +1046,8 @@ storageAzureNew( FUNCTION_LOG_PARAM(BOOL, verifyPeer); FUNCTION_LOG_PARAM(STRING, caFile); FUNCTION_LOG_PARAM(STRING, caPath); + FUNCTION_LOG_PARAM(TIME_MSEC, tmpRetention); + FUNCTION_LOG_PARAM(ENUM, tmpRetentionMode); FUNCTION_LOG_PARAM(UINT, prefetch); FUNCTION_LOG_PARAM(UINT64, readOver); FUNCTION_LOG_END(); @@ -1049,6 +1073,8 @@ storageAzureNew( strNewFmt("/%s", strZ(container)) : strNewFmt("/%s/%s", strZ(account), strZ(container)), .deleteMax = STORAGE_AZURE_DELETE_MAX, .keyType = keyType, + .tmpRetention = tmpRetention, + .tmpRetentionMode = tmpRetentionMode, }; // Set prefetch and read over diff --git a/src/storage/azure/storage.h b/src/storage/azure/storage.h index 2993af1548..3c04cc7833 100644 --- a/src/storage/azure/storage.h +++ b/src/storage/azure/storage.h @@ -31,6 +31,15 @@ typedef enum storageAzureUriStylePath, } StorageAzureUriStyle; +/*********************************************************************************************************************************** +Immutability policy mode +***********************************************************************************************************************************/ +typedef enum +{ + storageAzureImmutabilityPolicyModeUnlocked, // Retention can be shortened or removed + storageAzureImmutabilityPolicyModeLocked, // Retention cannot be shortened or removed +} StorageAzureImmutabilityPolicyMode; + /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ @@ -38,7 +47,7 @@ FN_EXTERN Storage *storageAzureNew( const String *path, bool write, time_t targetTime, StoragePathExpressionCallback pathExpressionFunction, const String *container, const String *account, StorageAzureKeyType keyType, const String *key, size_t blockSize, const KeyValue *tag, const String *endpoint, StorageAzureUriStyle uriStyle, unsigned int port, TimeMSec timeout, - HttpProtocolType protocolType, bool verifyPeer, const String *caFile, const String *caPath, unsigned int prefetch, - uint64_t readOver); + HttpProtocolType protocolType, bool verifyPeer, const String *caFile, const String *caPath, TimeMSec tmpRetention, + StorageAzureImmutabilityPolicyMode tmpRetentionMode, unsigned int prefetch, uint64_t readOver); #endif diff --git a/src/storage/azure/storage.intern.h b/src/storage/azure/storage.intern.h index 4fc9e579c9..6d10924632 100644 --- a/src/storage/azure/storage.intern.h +++ b/src/storage/azure/storage.intern.h @@ -44,6 +44,7 @@ typedef struct StorageAzureRequestAsyncParam const Buffer *content; // Request content const List *contentList; // Request content part list bool tag; // Add tags when available? + bool immutabilityPolicy; // Add tmp immutability policy when configured? } StorageAzureRequestAsyncParam; #define storageAzureRequestAsyncP(this, verb, ...) \ @@ -75,6 +76,7 @@ typedef struct StorageAzureRequestParam bool allowMissing; // Allow missing files (caller can check response code) bool contentIo; // Is IoRead interface required to read content? bool tag; // Add tags when available? + bool immutabilityPolicy; // Add tmp immutability policy when configured? } StorageAzureRequestParam; #define storageAzureRequestP(this, verb, ...) \ diff --git a/src/storage/azure/write.c b/src/storage/azure/write.c index ac2f0466af..cf304a1c64 100644 --- a/src/storage/azure/write.c +++ b/src/storage/azure/write.c @@ -50,6 +50,7 @@ struct StorageWriteAzure HttpRequest *request; // Async block upload request uint64_t fileId; // Id to used to make file block identifiers unique size_t blockSize; // Size of blocks for multi-block upload + bool tmp; // File will not last longer than the current operation Buffer *blockBuffer; // Block buffer (stores data until blockSize is reached) StringList *blockIdList; // List of uploaded block ids }; @@ -220,7 +221,7 @@ storageWriteAzureClose(THIS_VOID) storageAzureRequestP( this->storage, HTTP_VERB_PUT_STR, .path = this->name, .query = httpQueryAdd(httpQueryNewP(), AZURE_QUERY_COMP_STR, AZURE_QUERY_VALUE_BLOCK_LIST_STR), - .content = xmlDocumentBuf(blockXml), .tag = true); + .content = xmlDocumentBuf(blockXml), .tag = true, .immutabilityPolicy = this->tmp); } // Else upload all the data in a single block else @@ -228,7 +229,7 @@ storageWriteAzureClose(THIS_VOID) storageAzureRequestP( this->storage, HTTP_VERB_PUT_STR, .path = this->name, httpHeaderAdd(httpHeaderNew(NULL), AZURE_HEADER_BLOB_TYPE_STR, AZURE_HEADER_VALUE_BLOCK_BLOB_STR), - .content = this->blockBuffer, .tag = true); + .content = this->blockBuffer, .tag = true, .immutabilityPolicy = this->tmp); } bufFree(this->blockBuffer); @@ -248,13 +249,15 @@ static const StorageWriteInterface storageWriteAzureInterface = }; FN_EXTERN StorageWriteAzure * -storageWriteAzureNew(StorageAzure *const storage, const String *const name, const uint64_t fileId, const size_t blockSize) +storageWriteAzureNew( + StorageAzure *const storage, const String *const name, const uint64_t fileId, const size_t blockSize, const bool tmp) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_AZURE, storage); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(UINT64, fileId); FUNCTION_LOG_PARAM(UINT64, blockSize); + FUNCTION_LOG_PARAM(BOOL, tmp); FUNCTION_LOG_END(); ASSERT(storage != NULL); @@ -269,6 +272,7 @@ storageWriteAzureNew(StorageAzure *const storage, const String *const name, cons .name = strDup(name), .fileId = fileId, .blockSize = blockSize, + .tmp = tmp, .blockBuffer = bufNew(0), }; } diff --git a/src/storage/azure/write.h b/src/storage/azure/write.h index 0f2fa94195..5bdfaa5389 100644 --- a/src/storage/azure/write.h +++ b/src/storage/azure/write.h @@ -15,7 +15,8 @@ typedef struct StorageWriteAzure StorageWriteAzure; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ -FN_EXTERN StorageWriteAzure *storageWriteAzureNew(StorageAzure *storage, const String *name, uint64_t fileId, size_t blockSize); +FN_EXTERN StorageWriteAzure *storageWriteAzureNew( + StorageAzure *storage, const String *name, uint64_t fileId, size_t blockSize, bool tmp); /*********************************************************************************************************************************** Macros for function logging diff --git a/src/storage/remote/protocol.c b/src/storage/remote/protocol.c index 5230dbd119..8a123ea2b2 100644 --- a/src/storage/remote/protocol.c +++ b/src/storage/remote/protocol.c @@ -572,11 +572,12 @@ storageRemoteWriteOpenProtocol(PackRead *const param) const bool syncFile = pckReadBoolP(param); const bool syncPath = pckReadBoolP(param); const bool atomic = pckReadBoolP(param); + const bool tmp = pckReadBoolP(param); const Pack *const filter = pckReadPackP(param); StorageWrite *const fileWrite = storageWriteNew( storageRemoteProtocolLocal.storage, file, modeFile, modePath, user, group, timeModified, createPath, syncFile, - syncPath, atomic, true, false); + syncPath, atomic, true, false, tmp); // Set filter group based on passed filters storageRemoteFilterGroup(ioWriteFilterGroup(storageWriteIo(fileWrite)), filter); diff --git a/src/storage/remote/storage.c b/src/storage/remote/storage.c index 230c7375cb..cf4b82c053 100644 --- a/src/storage/remote/storage.c +++ b/src/storage/remote/storage.c @@ -319,6 +319,7 @@ storageRemoteNewWrite(THIS_VOID, const String *const file, const StorageInterfac FUNCTION_LOG_PARAM(BOOL, param.syncFile); FUNCTION_LOG_PARAM(BOOL, param.syncPath); FUNCTION_LOG_PARAM(BOOL, param.atomic); + FUNCTION_LOG_PARAM(BOOL, param.tmp); FUNCTION_LOG_PARAM(BOOL, param.compressible); FUNCTION_LOG_END(); @@ -330,7 +331,7 @@ storageRemoteNewWrite(THIS_VOID, const String *const file, const StorageInterfac STORAGE_WRITE_REMOTE, storageWriteRemoteNew( this, this->client, file, param.modeFile, param.modePath, param.user, param.group, param.timeModified, param.createPath, - param.syncFile, param.syncPath, param.atomic, this->compressLevel > 0 ? param.compressible : false, + param.syncFile, param.syncPath, param.atomic, param.tmp, this->compressLevel > 0 ? param.compressible : false, this->compressLevel)); } diff --git a/src/storage/remote/write.c b/src/storage/remote/write.c index 991bd407ea..7329eff020 100644 --- a/src/storage/remote/write.c +++ b/src/storage/remote/write.c @@ -34,6 +34,7 @@ struct StorageWriteRemote bool syncFile; // Sync file bool syncPath; // Sync path bool atomic; // Atomic write + bool tmp; // File will not last longer than the current operation bool compressible; // Is this file compressible? unsigned int compressLevel; // Level to use for compression @@ -95,6 +96,7 @@ storageWriteRemoteOpen(THIS_VOID) pckWriteBoolP(param, this->syncFile); pckWriteBoolP(param, this->syncPath); pckWriteBoolP(param, this->atomic); + pckWriteBoolP(param, this->tmp); pckWritePackP(param, ioFilterGroupParamAll(this->filterGroup)); protocolClientSessionOpenP(this->session, .param = param); @@ -196,7 +198,8 @@ FN_EXTERN StorageWriteRemote * storageWriteRemoteNew( StorageRemote *const storage, ProtocolClient *const client, const String *const name, const mode_t modeFile, const mode_t modePath, const String *const user, const String *const group, const time_t timeModified, const bool createPath, - const bool syncFile, const bool syncPath, const bool atomic, const bool compressible, const unsigned int compressLevel) + const bool syncFile, const bool syncPath, const bool atomic, const bool tmp, const bool compressible, + const unsigned int compressLevel) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_REMOTE, storage); @@ -210,6 +213,7 @@ storageWriteRemoteNew( FUNCTION_LOG_PARAM(BOOL, syncFile); FUNCTION_LOG_PARAM(BOOL, syncPath); FUNCTION_LOG_PARAM(BOOL, atomic); + FUNCTION_LOG_PARAM(BOOL, tmp); FUNCTION_LOG_PARAM(BOOL, compressible); FUNCTION_LOG_PARAM(UINT, compressLevel); FUNCTION_LOG_END(); @@ -238,6 +242,7 @@ storageWriteRemoteNew( .syncFile = syncFile, .syncPath = syncPath, .atomic = atomic, + .tmp = tmp, .compressible = compressible, .compressLevel = compressLevel, }; diff --git a/src/storage/remote/write.h b/src/storage/remote/write.h index ebc556685b..a6f374e15b 100644 --- a/src/storage/remote/write.h +++ b/src/storage/remote/write.h @@ -18,8 +18,8 @@ Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageWriteRemote *storageWriteRemoteNew( StorageRemote *storage, ProtocolClient *client, const String *name, mode_t modeFile, mode_t modePath, const String *user, - const String *group, time_t timeModified, bool createPath, bool syncFile, bool syncPath, bool atomic, bool compressible, - unsigned int compressLevel); + const String *group, time_t timeModified, bool createPath, bool syncFile, bool syncPath, bool atomic, bool tmp, + bool compressible, unsigned int compressLevel); /*********************************************************************************************************************************** Macros for function logging diff --git a/src/storage/s3/helper.c b/src/storage/s3/helper.c index 54eedbe51d..a897217e15 100644 --- a/src/storage/s3/helper.c +++ b/src/storage/s3/helper.c @@ -109,6 +109,9 @@ storageS3Helper(const unsigned int repoIdx, const bool write, StoragePathExpress const StringList *const credCmd = keyType == storageS3KeyTypeProcess ? strLstNewVarLst(cfgOptionIdxLst(cfgOptRepoS3ProcessCmd, repoIdx)) : NULL; + // Object lock retention for tmp files + const bool tmpRetention = cfgOptionIdxTest(cfgOptRepoStorageTmpRetention, repoIdx); + MEM_CONTEXT_PRIOR_BEGIN() { result = storageS3New( @@ -123,6 +126,8 @@ storageS3Helper(const unsigned int repoIdx, const bool write, StoragePathExpress cfgOptionIdxKvNull(cfgOptRepoStorageTag, repoIdx), host, port, ioTimeoutMs(), protocolType, cfgOptionIdxBool(cfgOptRepoStorageVerifyTls, repoIdx), cfgOptionIdxStrNull(cfgOptRepoStorageCaFile, repoIdx), cfgOptionIdxStrNull(cfgOptRepoStorageCaPath, repoIdx), cfgOptionIdxBool(cfgOptRepoS3RequesterPays, repoIdx), + tmpRetention ? cfgOptionIdxUInt64(cfgOptRepoStorageTmpRetention, repoIdx) : 0, + (StorageS3ObjectLockMode)cfgOptionIdxSeq(cfgOptRepoS3TmpRetentionMode, repoIdx), cfgOptionIdxUInt(cfgOptRepoStoragePrefetch, repoIdx), cfgOptionIdxUInt64(cfgOptRepoStorageReadOver, repoIdx)); } MEM_CONTEXT_PRIOR_END(); diff --git a/src/storage/s3/storage.c b/src/storage/s3/storage.c index 039492b794..c12595a758 100644 --- a/src/storage/s3/storage.c +++ b/src/storage/s3/storage.c @@ -42,6 +42,10 @@ STRING_STATIC(S3_HEADER_SSECUSTKEY_KEY_MD5_STR, "x-amz-serve STRING_STATIC(S3_HEADER_TAGGING, "x-amz-tagging"); STRING_STATIC(S3_HEADER_REQUEST_PAYER, "x-amz-request-payer"); STRING_STATIC(S3_HEADER_REQUEST_PAYER_STR, "requester"); +STRING_STATIC(S3_HEADER_OBJECT_LOCK_MODE_STR, "x-amz-object-lock-mode"); +STRING_STATIC(S3_HEADER_OBJECT_LOCK_MODE_GOVERNANCE_STR, "GOVERNANCE"); +STRING_STATIC(S3_HEADER_OBJECT_LOCK_MODE_COMPLIANCE_STR, "COMPLIANCE"); +STRING_STATIC(S3_HEADER_OBJECT_LOCK_RETAIN_UNTIL_DATE_STR, "x-amz-object-lock-retain-until-date"); /*********************************************************************************************************************************** S3 query tokens @@ -118,6 +122,8 @@ struct StorageS3 StorageS3UriStyle uriStyle; // Path or host style URIs const String *bucketEndpoint; // Set to {bucket}.{endpoint} bool requesterPays; // Requester pays? + TimeMSec tmpRetention; // Object lock retention for tmp files (0 = bucket default) + StorageS3ObjectLockMode tmpRetentionMode; // Object lock mode for tmp files // For retrieving temporary security credentials HttpClient *credHttpClient; // HTTP client to service credential requests @@ -563,6 +569,7 @@ storageS3RequestAsync(StorageS3 *const this, const String *const verb, const Str FUNCTION_LOG_PARAM(BOOL, param.sseKms); FUNCTION_LOG_PARAM(BOOL, param.sseC); FUNCTION_LOG_PARAM(BOOL, param.tag); + FUNCTION_LOG_PARAM(BOOL, param.objectLock); FUNCTION_LOG_END(); ASSERT(this != NULL); @@ -612,6 +619,19 @@ storageS3RequestAsync(StorageS3 *const this, const String *const verb, const Str if (param.tag && this->tag != NULL) httpHeaderPut(requestHeader, S3_HEADER_TAGGING, this->tag); + // Set object lock retention for tmp files when configured. Explicit retention on an object overrides the default retention + // configured on the bucket. + if (param.objectLock && this->tmpRetention != 0) + { + httpHeaderPut( + requestHeader, S3_HEADER_OBJECT_LOCK_MODE_STR, + this->tmpRetentionMode == storageS3ObjectLockModeCompliance ? + S3_HEADER_OBJECT_LOCK_MODE_COMPLIANCE_STR : S3_HEADER_OBJECT_LOCK_MODE_GOVERNANCE_STR); + httpHeaderPut( + requestHeader, S3_HEADER_OBJECT_LOCK_RETAIN_UNTIL_DATE_STR, + strNewTimeP("%Y-%m-%dT%H:%M:%SZ", time(NULL) + (time_t)(this->tmpRetention / MSEC_PER_SEC), .utc = true)); + } + // When using path-style URIs the bucket name needs to be prepended if (this->uriStyle == storageS3UriStylePath) path = strNewFmt("/%s%s", strZ(this->bucket), strZ(path)); @@ -733,11 +753,12 @@ storageS3Request(StorageS3 *const this, const String *const verb, const String * FUNCTION_LOG_PARAM(BOOL, param.sseKms); FUNCTION_LOG_PARAM(BOOL, param.sseC); FUNCTION_LOG_PARAM(BOOL, param.tag); + FUNCTION_LOG_PARAM(BOOL, param.objectLock); FUNCTION_LOG_END(); HttpRequest *const request = storageS3RequestAsyncP( this, verb, path, .header = param.header, .query = param.query, .content = param.content, .sseKms = param.sseKms, - .sseC = param.sseC, .tag = param.tag); + .sseC = param.sseC, .tag = param.tag, .objectLock = param.objectLock); HttpResponse *const result = storageS3ResponseP( request, .allowMissing = param.allowMissing, .contentIo = param.contentIo); @@ -1112,7 +1133,7 @@ storageS3NewWrite(THIS_VOID, const String *const file, const StorageInterfaceNew FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_S3, this); FUNCTION_LOG_PARAM(STRING, file); - (void)param; // No parameters are used + FUNCTION_LOG_PARAM(BOOL, param.tmp); FUNCTION_LOG_END(); ASSERT(this != NULL); @@ -1123,7 +1144,7 @@ storageS3NewWrite(THIS_VOID, const String *const file, const StorageInterfaceNew ASSERT(param.group == NULL); ASSERT(param.timeModified == 0); - FUNCTION_LOG_RETURN(STORAGE_WRITE_S3, storageWriteS3New(this, file, this->partSize)); + FUNCTION_LOG_RETURN(STORAGE_WRITE_S3, storageWriteS3New(this, file, this->partSize, param.tmp)); } /**********************************************************************************************************************************/ @@ -1331,7 +1352,8 @@ storageS3New( const StringList *const credCmd, const String *const stsHost, const size_t partSize, const KeyValue *const tag, const String *host, const unsigned int port, const TimeMSec timeout, const HttpProtocolType protocolType, const bool verifyPeer, const String *const caFile, const String *const caPath, const bool requesterPays, - const unsigned int prefetch, const uint64_t readOver) + const TimeMSec tmpRetention, const StorageS3ObjectLockMode tmpRetentionMode, const unsigned int prefetch, + const uint64_t readOver) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, path); @@ -1364,6 +1386,8 @@ storageS3New( FUNCTION_LOG_PARAM(STRING, caFile); FUNCTION_LOG_PARAM(STRING, caPath); FUNCTION_LOG_PARAM(BOOL, requesterPays); + FUNCTION_LOG_PARAM(TIME_MSEC, tmpRetention); + FUNCTION_LOG_PARAM(ENUM, tmpRetentionMode); FUNCTION_LOG_PARAM(UINT, prefetch); FUNCTION_LOG_PARAM(UINT64, readOver); FUNCTION_LOG_END(); @@ -1386,6 +1410,8 @@ storageS3New( .keyType = keyType, .kmsKeyId = strDup(kmsKeyId), .requesterPays = requesterPays, + .tmpRetention = tmpRetention, + .tmpRetentionMode = tmpRetentionMode, .sseCustomerKey = strDup(sseCustomerKey), .partSize = partSize, .deleteMax = STORAGE_S3_DELETE_MAX, diff --git a/src/storage/s3/storage.h b/src/storage/s3/storage.h index 6b251b0fb5..d7d8a6450b 100644 --- a/src/storage/s3/storage.h +++ b/src/storage/s3/storage.h @@ -33,6 +33,15 @@ typedef enum storageS3UriStylePath, } StorageS3UriStyle; +/*********************************************************************************************************************************** +Object lock mode +***********************************************************************************************************************************/ +typedef enum +{ + storageS3ObjectLockModeGovernance, // Retention can be removed by users with special permissions + storageS3ObjectLockModeCompliance, // Retention cannot be removed by any user +} StorageS3ObjectLockMode; + /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ @@ -43,6 +52,7 @@ FN_EXTERN Storage *storageS3New( const String *sseCustomerKey, const String *credRole, const String *tokenFile, const String *credUrl, const StringList *credCmd, const String *stsHost, size_t partSize, const KeyValue *tag, const String *host, unsigned int port, TimeMSec timeout, HttpProtocolType protocolType, bool verifyPeer, const String *caFile, - const String *caPath, bool requesterPays, unsigned int prefetch, uint64_t readOver); + const String *caPath, bool requesterPays, TimeMSec tmpRetention, StorageS3ObjectLockMode tmpRetentionMode, + unsigned int prefetch, uint64_t readOver); #endif diff --git a/src/storage/s3/storage.intern.h b/src/storage/s3/storage.intern.h index 58c2b44e83..35e63d882b 100644 --- a/src/storage/s3/storage.intern.h +++ b/src/storage/s3/storage.intern.h @@ -25,6 +25,7 @@ typedef struct StorageS3RequestAsyncParam bool sseKms; // Enable server-side encryption? bool sseC; // Enable server-side encryption with customer-provided keys? bool tag; // Add tags when available? + bool objectLock; // Add tmp object lock retention when configured? } StorageS3RequestAsyncParam; #define storageS3RequestAsyncP(this, verb, path, ...) \ @@ -58,6 +59,7 @@ typedef struct StorageS3RequestParam bool sseKms; // Enable server-side encryption? bool sseC; // Enable server-side encryption with customer-provided keys? bool tag; // Add tags when available? + bool objectLock; // Add tmp object lock retention when configured? } StorageS3RequestParam; #define storageS3RequestP(this, verb, path, ...) \ diff --git a/src/storage/s3/write.c b/src/storage/s3/write.c index ec180dd54b..04357055d9 100644 --- a/src/storage/s3/write.c +++ b/src/storage/s3/write.c @@ -43,6 +43,7 @@ struct StorageWriteS3 HttpRequest *request; // Async request size_t partSize; + bool tmp; // File will not last longer than the current operation Buffer *partBuffer; const String *uploadId; StringList *uploadPartList; @@ -102,7 +103,7 @@ storageWriteS3PartAsync(StorageWriteS3 *const this) storageS3RequestP( this->storage, HTTP_VERB_POST_STR, this->name, .query = httpQueryAdd(httpQueryNewP(), S3_QUERY_UPLOADS_STR, EMPTY_STR), .sseKms = true, - .sseC = true, .tag = true)))); + .sseC = true, .tag = true, .objectLock = this->tmp)))); // Store the upload id MEM_CONTEXT_OBJ_BEGIN(this) @@ -235,7 +236,7 @@ storageWriteS3Close(THIS_VOID) { storageS3RequestP( this->storage, HTTP_VERB_PUT_STR, this->name, .content = this->partBuffer, .sseKms = true, - .sseC = true, .tag = true); + .sseC = true, .tag = true, .objectLock = this->tmp); } bufFree(this->partBuffer); @@ -255,12 +256,13 @@ static const StorageWriteInterface storageWriteS3Interface = }; FN_EXTERN StorageWriteS3 * -storageWriteS3New(StorageS3 *const storage, const String *const name, const size_t partSize) +storageWriteS3New(StorageS3 *const storage, const String *const name, const size_t partSize, const bool tmp) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_S3, storage); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(SIZE, partSize); + FUNCTION_LOG_PARAM(BOOL, tmp); FUNCTION_LOG_END(); ASSERT(storage != NULL); @@ -274,6 +276,7 @@ storageWriteS3New(StorageS3 *const storage, const String *const name, const size .storage = storage, .name = strDup(name), .partSize = partSize, + .tmp = tmp, .partBuffer = bufNew(0), }; } diff --git a/src/storage/s3/write.h b/src/storage/s3/write.h index 13b10405d8..cc5894776a 100644 --- a/src/storage/s3/write.h +++ b/src/storage/s3/write.h @@ -15,7 +15,7 @@ typedef struct StorageWriteS3 StorageWriteS3; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ -FN_EXTERN StorageWriteS3 *storageWriteS3New(StorageS3 *storage, const String *name, size_t partSize); +FN_EXTERN StorageWriteS3 *storageWriteS3New(StorageS3 *storage, const String *name, size_t partSize, bool tmp); /*********************************************************************************************************************************** Macros for function logging diff --git a/src/storage/storage.c b/src/storage/storage.c index d65ff11daf..ff09a26091 100644 --- a/src/storage/storage.c +++ b/src/storage/storage.c @@ -569,6 +569,7 @@ storageNewWrite(const Storage *const this, const String *const fileExp, const St FUNCTION_LOG_PARAM(BOOL, param.noAtomic); FUNCTION_LOG_PARAM(BOOL, param.noTruncate); FUNCTION_LOG_PARAM(BOOL, param.compressible); + FUNCTION_LOG_PARAM(BOOL, param.tmp); FUNCTION_LOG_END(); ASSERT(this != NULL); @@ -585,7 +586,7 @@ storageNewWrite(const Storage *const this, const String *const fileExp, const St this, storagePathP(this, fileExp), param.modeFile != 0 ? param.modeFile : this->modeFile, param.modePath != 0 ? param.modePath : this->modePath, param.user, param.group, param.timeModified, !param.noCreatePath, !param.noSyncFile, !param.noSyncPath, !param.noAtomic, !param.noTruncate, - param.compressible), + param.compressible, param.tmp), memContextPrior()); } MEM_CONTEXT_TEMP_END(); diff --git a/src/storage/storage.h b/src/storage/storage.h index ddc436ca9d..b9f4ac115a 100644 --- a/src/storage/storage.h +++ b/src/storage/storage.h @@ -193,6 +193,11 @@ typedef struct StorageNewWriteParam bool noTruncate; bool compressible; + + // The file is not intended to last longer than the current operation, e.g. a checkpoint that will be overwritten. Storage may + // treat the file differently, e.g. by applying a shorter retention. + bool tmp; + mode_t modeFile; mode_t modePath; time_t timeModified; diff --git a/src/storage/storage.intern.h b/src/storage/storage.intern.h index ede898b438..5277e07cfc 100644 --- a/src/storage/storage.intern.h +++ b/src/storage/storage.intern.h @@ -171,6 +171,10 @@ typedef struct StorageInterfaceNewWriteParam // Is the file compressible? This is used when the file must be moved across a network and temporary compression is helpful. bool compressible; + + // The file is not intended to last longer than the current operation, e.g. a checkpoint that will be overwritten. The driver + // decides what to do with this, e.g. apply a shorter retention, or ignore it. + bool tmp; } StorageInterfaceNewWriteParam; typedef void *StorageInterfaceNewWrite(void *thisVoid, const String *file, StorageInterfaceNewWriteParam param); diff --git a/src/storage/write.c b/src/storage/write.c index 540050f568..7cdecccad6 100644 --- a/src/storage/write.c +++ b/src/storage/write.c @@ -126,7 +126,7 @@ FN_EXTERN StorageWrite * storageWriteNew( const Storage *const storage, const String *const name, const mode_t modeFile, const mode_t modePath, const String *const user, const String *const group, const time_t timeModified, const bool createPath, const bool syncFile, const bool syncPath, - const bool atomic, const bool truncate, const bool compressible) + const bool atomic, const bool truncate, const bool compressible, const bool tmp) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE, storage); @@ -142,6 +142,7 @@ storageWriteNew( FUNCTION_LOG_PARAM(BOOL, atomic); FUNCTION_LOG_PARAM(BOOL, truncate); FUNCTION_LOG_PARAM(BOOL, compressible); + FUNCTION_LOG_PARAM(BOOL, tmp); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); @@ -156,7 +157,7 @@ storageWriteNew( .driver = storageInterfaceNewWriteP( storageDriver(storage), name, .modeFile = modeFile, .modePath = modePath, .user = user, .group = group, .timeModified = timeModified, .createPath = createPath, .syncFile = syncFile, .syncPath = syncPath, - .atomic = atomic, .truncate = truncate, .compressible = compressible), + .atomic = atomic, .truncate = truncate, .compressible = compressible, .tmp = tmp), .pub = { .type = storageType(storage), diff --git a/src/storage/write.intern.h b/src/storage/write.intern.h index 10e59c1f65..4d4fc227d5 100644 --- a/src/storage/write.intern.h +++ b/src/storage/write.intern.h @@ -31,7 +31,8 @@ Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageWrite *storageWriteNew( const Storage *storage, const String *name, mode_t modeFile, mode_t modePath, const String *user, const String *group, - time_t timeModified, bool createPath, bool syncFile, bool syncPath, bool atomic, bool truncate, bool compressible); + time_t timeModified, bool createPath, bool syncFile, bool syncPath, bool atomic, bool truncate, bool compressible, + bool tmp); /*********************************************************************************************************************************** Getters diff --git a/test/src/harness/host.c b/test/src/harness/host.c index 6492c946d8..a1740077f9 100644 --- a/test/src/harness/host.c +++ b/test/src/harness/host.c @@ -742,7 +742,8 @@ hrnHostConfig(HrnHost *const this) this->pub.repo1Storage = storageAzureNew( hrnHostRepo1Path(this), true, 0, NULL, STRDEF(HRN_HOST_AZURE_CONTAINER), STRDEF(HRN_HOST_AZURE_ACCOUNT), storageAzureKeyTypeShared, STRDEF(HRN_HOST_AZURE_KEY), 4 * 1024 * 1024, NULL, hrnHostIp(azure), - storageAzureUriStylePath, 443, ioTimeoutMs(), httpProtocolTypeHttps, false, NULL, NULL, 2, 8192); + storageAzureUriStylePath, 443, ioTimeoutMs(), httpProtocolTypeHttps, false, NULL, NULL, 0, + storageAzureImmutabilityPolicyModeUnlocked, 2, 8192); } MEM_CONTEXT_OBJ_END(); @@ -790,7 +791,7 @@ hrnHostConfig(HrnHost *const this) STR(HRN_HOST_S3_REGION), STRDEF("s3"), storageS3KeyTypeShared, storageS3UriStyleHost, STRDEF(HRN_HOST_S3_ACCESS_KEY), STRDEF(HRN_HOST_S3_ACCESS_SECRET_KEY), NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 5 * 1024 * 1024, NULL, hrnHostIp(s3), 443, ioTimeoutMs(), httpProtocolTypeHttps, - false, NULL, NULL, NULL, 2, 8192); + false, NULL, NULL, NULL, 0, storageS3ObjectLockModeGovernance, 2, 8192); } MEM_CONTEXT_OBJ_END(); diff --git a/test/src/module/command/helpTest.c b/test/src/module/command/helpTest.c index f49913a320..09d862d263 100644 --- a/test/src/module/command/helpTest.c +++ b/test/src/module/command/helpTest.c @@ -303,6 +303,7 @@ testRun(void) " --repo-azure-endpoint azure repository endpoint\n" " --repo-azure-key azure repository key\n" " --repo-azure-key-type azure repository key type\n" + " --repo-azure-tmp-retention-mode azure temporary file retention mode\n" " --repo-azure-uri-style azure URI Style\n" " --repo-cipher-pass repository cipher passphrase\n" " [current=<redacted>]\n" @@ -349,6 +350,7 @@ testRun(void) " --repo-s3-service S3 signing service\n" " --repo-s3-sse-customer-key S3 repository SSE customer key\n" " --repo-s3-sts-host S3 repository STS endpoint\n" + " --repo-s3-tmp-retention-mode S3 temporary file retention mode\n" " --repo-s3-token S3 repository security token\n" " --repo-s3-uri-style S3 URI Style\n" " --repo-sftp-host SFTP repository host\n" @@ -368,6 +370,7 @@ testRun(void) " --repo-storage-prefetch repository storage prefetch\n" " --repo-storage-read-over repository storage read over\n" " --repo-storage-tag repository storage tag(s)\n" + " --repo-storage-tmp-retention repository temporary file retention\n" " --repo-storage-upload-chunk-size repository storage upload chunk size\n" " --repo-storage-verify-tls repository storage TLS verify\n" " --repo-target-time target time for repository\n" diff --git a/test/src/module/storage/azureTest.c b/test/src/module/storage/azureTest.c index 43311eeac5..6a3e8825f7 100644 --- a/test/src/module/storage/azureTest.c +++ b/test/src/module/storage/azureTest.c @@ -37,6 +37,7 @@ typedef struct TestRequestParam const char *blobType; const char *range; const char *tag; + const char *immutabilityPolicyMode; bool multiPart; } TestRequestParam; @@ -112,6 +113,13 @@ testRequest(IoWrite *write, const char *verb, const char *path, TestRequestParam if (param.blobType != NULL) strCatFmt(request, "x-ms-blob-type:%s\r\n", param.blobType); + // Add immutability policy mode and until date + if (param.immutabilityPolicyMode != NULL) + { + strCatFmt(request, "x-ms-immutability-policy-mode:%s\r\n", param.immutabilityPolicyMode); + strCatZ(request, "x-ms-immutability-policy-until-date:???, ?? ??? ???? ??:??:?? GMT\r\n"); + } + // Add tags if (param.tag != NULL) strCatFmt(request, "x-ms-tags:%s\r\n", param.tag); @@ -421,7 +429,7 @@ testRun(void) storageAzureNew( STRDEF("/repo"), false, 0, NULL, TEST_CONTAINER_STR, TEST_ACCOUNT_STR, storageAzureKeyTypeShared, TEST_KEY_SHARED_STR, 16, NULL, STRDEF("blob.core.windows.net"), storageAzureUriStyleHost, 443, 1000, - httpProtocolTypeHttps, true, NULL, NULL, 1, 0)), + httpProtocolTypeHttps, true, NULL, NULL, 0, storageAzureImmutabilityPolicyModeUnlocked, 1, 0)), "new azure storage - shared key"); // ------------------------------------------------------------------------------------------------------------------------- @@ -464,7 +472,7 @@ testRun(void) storageAzureNew( STRDEF("/repo"), false, 0, NULL, TEST_CONTAINER_STR, TEST_ACCOUNT_STR, storageAzureKeyTypeSas, TEST_KEY_SAS_STR, 16, NULL, STRDEF("blob.core.usgovcloudapi.net"), storageAzureUriStyleHost, 443, 1000, httpProtocolTypeHttps, - true, NULL, NULL, 1, 0)), + true, NULL, NULL, 0, storageAzureImmutabilityPolicyModeUnlocked, 1, 0)), "new azure storage - sas key"); query = httpQueryAdd(httpQueryNewP(), STRDEF("a"), STRDEF("b")); @@ -520,6 +528,8 @@ testRun(void) hrnCfgEnvRawZ(cfgOptRepoAzureKey, TEST_KEY_SHARED); hrnCfgArgRawZ(argList, cfgOptRepoStorageTag, "Key1=Value1"); hrnCfgArgRawZ(argList, cfgOptRepoStorageTag, " Key 2= Value 2"); + hrnCfgArgRawZ(argList, cfgOptRepoStorageTmpRetention, "1d"); + hrnCfgArgRawZ(argList, cfgOptRepoAzureTmpRetentionMode, "locked"); HRN_CFG_LOAD(cfgCmdArchivePush, argList); Storage *storage = NULL; @@ -527,6 +537,12 @@ testRun(void) driver = (StorageAzure *)storageDriver(storage); TEST_RESULT_STR(driver->host, hrnServerHost(), "check host"); + TEST_RESULT_UINT(driver->tmpRetention, 86400000, "check tmp retention"); + TEST_RESULT_UINT(driver->tmpRetentionMode, storageAzureImmutabilityPolicyModeLocked, "check tmp retention mode"); + + // Tests expect the unlocked mode unless they set it + driver->tmpRetentionMode = storageAzureImmutabilityPolicyModeUnlocked; + TEST_RESULT_STR_Z(driver->pathPrefix, "/" TEST_ACCOUNT "/" TEST_CONTAINER, "check path prefix"); TEST_RESULT_BOOL(driver->fileId == 0, false, "check file id"); @@ -753,6 +769,63 @@ testRun(void) TEST_ASSIGN(write, storageNewWriteP(storage, STRDEF("file.txt")), "new write"); TEST_RESULT_VOID(storagePutP(write, BUFSTRDEF("12345678901234567890123456789012")), "write"); + // ----------------------------------------------------------------------------------------------------------------- + TEST_TITLE("write tmp file in one part with unlocked immutability policy"); + + testRequestP( + service, HTTP_VERB_PUT, "/file.txt", .blobType = "BlockBlob", .content = "ABCD", + .tag = "%20Key%202=%20Value%202&Key1=Value1", .immutabilityPolicyMode = "unlocked"); + testResponseP(service); + + TEST_ASSIGN(write, storageNewWriteP(storage, STRDEF("file.txt"), .tmp = true), "new write"); + TEST_RESULT_VOID(storagePutP(write, BUFSTRDEF("ABCD")), "write"); + + // ----------------------------------------------------------------------------------------------------------------- + TEST_TITLE("write tmp file without tmp retention"); + + driver->tmpRetention = 0; + + testRequestP( + service, HTTP_VERB_PUT, "/file.txt", .blobType = "BlockBlob", .content = "ABCD", + .tag = "%20Key%202=%20Value%202&Key1=Value1"); + testResponseP(service); + + TEST_ASSIGN(write, storageNewWriteP(storage, STRDEF("file.txt"), .tmp = true), "new write"); + TEST_RESULT_VOID(storagePutP(write, BUFSTRDEF("ABCD")), "write"); + + driver->tmpRetention = 86400000; + + // ----------------------------------------------------------------------------------------------------------------- + TEST_TITLE("write tmp file in chunks with locked immutability policy only on block list"); + + driver->tmpRetentionMode = storageAzureImmutabilityPolicyModeLocked; + + testRequestP( + service, HTTP_VERB_PUT, "/file.txt?blockid=0AAAAAAACCCCCCCCx0000000&comp=block", .content = "1234567890123456"); + testResponseP(service); + + testRequestP( + service, HTTP_VERB_PUT, "/file.txt?blockid=0AAAAAAACCCCCCCCx0000001&comp=block", .content = "7890123456789012"); + testResponseP(service); + + testRequestP( + service, HTTP_VERB_PUT, "/file.txt?comp=blocklist", + .content = + "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" + "<BlockList>" + "<Uncommitted>0AAAAAAACCCCCCCCx0000000</Uncommitted>" + "<Uncommitted>0AAAAAAACCCCCCCCx0000001</Uncommitted>" + "</BlockList>\n", + .tag = "%20Key%202=%20Value%202&Key1=Value1", .immutabilityPolicyMode = "locked"); + testResponseP(service); + + driver->fileId = 0x0AAAAAAACCCCCCCC; + + TEST_ASSIGN(write, storageNewWriteP(storage, STRDEF("file.txt"), .tmp = true), "new write"); + TEST_RESULT_VOID(storagePutP(write, BUFSTRDEF("12345678901234567890123456789012")), "write"); + + driver->tmpRetentionMode = storageAzureImmutabilityPolicyModeUnlocked; + // Stop writing tags driver->tag = NULL; diff --git a/test/src/module/storage/remoteTest.c b/test/src/module/storage/remoteTest.c index 81316c45c3..900a0c155f 100644 --- a/test/src/module/storage/remoteTest.c +++ b/test/src/module/storage/remoteTest.c @@ -597,6 +597,14 @@ testRun(void) TEST_RESULT_VOID(storagePutP(write, contentBuf), "write file"); TEST_RESULT_BOOL(bufEq(storageGetP(storageNewReadP(storageRepo, STRDEF("test.txt"))), contentBuf), true, "check file"); + // ------------------------------------------------------------------------------------------------------------------------- + TEST_TITLE("write tmp file"); + + TEST_ASSIGN(write, storageNewWriteP(storageRepoWrite, STRDEF("test.txt"), .tmp = true), "new write file"); + TEST_RESULT_BOOL(((StorageWriteRemote *)write->driver)->tmp, true, "check tmp"); + TEST_RESULT_VOID(storagePutP(write, contentBuf), "write file"); + TEST_RESULT_BOOL(bufEq(storageGetP(storageNewReadP(storageRepo, STRDEF("test.txt"))), contentBuf), true, "check file"); + // Enable protocol compression in the storage object ((StorageRemote *)storageDriver(storageRepoWrite))->compressLevel = 3; diff --git a/test/src/module/storage/s3Test.c b/test/src/module/storage/s3Test.c index 9cb8de6d0a..49ec4c4a2f 100644 --- a/test/src/module/storage/s3Test.c +++ b/test/src/module/storage/s3Test.c @@ -82,6 +82,7 @@ typedef struct TestRequestParam const char *ttl; const char *token; const char *tag; + const char *objectLockMode; bool requesterPays; } TestRequestParam; @@ -127,6 +128,9 @@ testRequest(IoWrite *write, Storage *s3, const char *verb, const char *path, Tes strCatZ(request, "x-amz-content-sha256;x-amz-date"); + if (param.objectLockMode != NULL) + strCatZ(request, ";x-amz-object-lock-mode;x-amz-object-lock-retain-until-date"); + if (param.requesterPays) strCatZ(request, ";x-amz-request-payer"); @@ -190,6 +194,13 @@ testRequest(IoWrite *write, Storage *s3, const char *verb, const char *path, Tes param.content == NULL ? HASH_TYPE_SHA256_ZERO : strZ(strNewEncode(encodingHex, cryptoHashOne(hashTypeSha256, BUFSTRZ(param.content))))); + // Add object lock mode and retain until date + if (param.objectLockMode != NULL) + { + strCatFmt(request, "x-amz-object-lock-mode:%s\r\n", param.objectLockMode); + strCatZ(request, "x-amz-object-lock-retain-until-date:\?\?\?\?-\?\?-\?\?T\?\?:\?\?:\?\?Z\r\n"); + } + // Add security token if (securityToken != NULL) strCatFmt(request, "x-amz-security-token:%s\r\n", securityToken); @@ -571,6 +582,7 @@ testRun(void) TEST_RESULT_STR(s3->path, path, "check path"); TEST_RESULT_BOOL(storageFeature(s3, storageFeaturePath), false, "check path feature"); TEST_RESULT_UINT(driver->partSize, 5 * 1024 * 1024, "check part size"); + TEST_RESULT_UINT(driver->tmpRetention, 0, "check tmp retention"); // ----------------------------------------------------------------------------------------------------------------- TEST_TITLE("check part sizes"); @@ -781,6 +793,7 @@ testRun(void) hrnCfgEnvRawZ(cfgOptRepoS3SseCustomerKey, "rA1P"); hrnCfgArgRawZ(argList, cfgOptRepoStorageTag, "Key1=Value1"); hrnCfgArgRawZ(argList, cfgOptRepoStorageTag, " Key 2= Value 2"); + hrnCfgArgRawZ(argList, cfgOptRepoStorageTmpRetention, "1d"); HRN_CFG_LOAD(cfgCmdArchivePush, argList); hrnCfgEnvRemoveRaw(cfgOptRepoS3SseCustomerKey); @@ -789,6 +802,8 @@ testRun(void) TEST_RESULT_STR(s3->path, path, "check path"); TEST_RESULT_STR(driver->credRole, credRole, "check role"); + TEST_RESULT_UINT(driver->tmpRetention, 86400000, "check tmp retention"); + TEST_RESULT_UINT(driver->tmpRetentionMode, storageS3ObjectLockModeGovernance, "check tmp retention mode"); TEST_RESULT_BOOL(storageFeature(s3, storageFeaturePath), false, "check path feature"); // Set partSize to a small value for testing @@ -1044,6 +1059,90 @@ testRun(void) TEST_ASSIGN(write, storageNewWriteP(s3, STRDEF("file.txt")), "new write"); TEST_RESULT_VOID(storagePutP(write, NULL), "write"); + // ----------------------------------------------------------------------------------------------------------------- + TEST_TITLE("write tmp file in one part with governance retention"); + + testRequestP( + service, s3, HTTP_VERB_PUT, "/file.txt", .content = "ABCD", .kms = "kmskey1", .sseC = "rA1P", + .tag = "%20Key%202=%20Value%202&Key1=Value1", .objectLockMode = "GOVERNANCE"); + testResponseP(service); + + TEST_ASSIGN(write, storageNewWriteP(s3, STRDEF("file.txt"), .tmp = true), "new write"); + TEST_RESULT_VOID(storagePutP(write, BUFSTRDEF("ABCD")), "write"); + + // ----------------------------------------------------------------------------------------------------------------- + TEST_TITLE("write tmp file without tmp retention"); + + driver->tmpRetention = 0; + + testRequestP( + service, s3, HTTP_VERB_PUT, "/file.txt", .content = "ABCD", .kms = "kmskey1", .sseC = "rA1P", + .tag = "%20Key%202=%20Value%202&Key1=Value1"); + testResponseP(service); + + TEST_ASSIGN(write, storageNewWriteP(s3, STRDEF("file.txt"), .tmp = true), "new write"); + TEST_RESULT_VOID(storagePutP(write, BUFSTRDEF("ABCD")), "write"); + + driver->tmpRetention = 86400000; + + // ----------------------------------------------------------------------------------------------------------------- + TEST_TITLE("write tmp file in one part with compliance retention"); + + driver->tmpRetentionMode = storageS3ObjectLockModeCompliance; + + testRequestP( + service, s3, HTTP_VERB_PUT, "/file.txt", .content = "ABCD", .kms = "kmskey1", .sseC = "rA1P", + .tag = "%20Key%202=%20Value%202&Key1=Value1", .objectLockMode = "COMPLIANCE"); + testResponseP(service); + + TEST_ASSIGN(write, storageNewWriteP(s3, STRDEF("file.txt"), .tmp = true), "new write"); + TEST_RESULT_VOID(storagePutP(write, BUFSTRDEF("ABCD")), "write"); + + // ----------------------------------------------------------------------------------------------------------------- + TEST_TITLE("write tmp file in chunks with retention only on multipart upload initiation"); + + testRequestP( + service, s3, HTTP_VERB_POST, "/file.txt?uploads=", .kms = "kmskey1", .sseC = "rA1P", + .tag = "%20Key%202=%20Value%202&Key1=Value1", .objectLockMode = "COMPLIANCE"); + testResponseP( + service, + .content = + "<?xml version=\"1.0\" encoding=\"UTF-8\"?>" + "<InitiateMultipartUploadResult xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\">" + "<Bucket>bucket</Bucket>" + "<Key>file.txt</Key>" + "<UploadId>WxRt</UploadId>" + "</InitiateMultipartUploadResult>"); + + testRequestP( + service, s3, HTTP_VERB_PUT, "/file.txt?partNumber=1&uploadId=WxRt", .content = "1234567890123456", + .sseC = "rA1P"); + testResponseP(service, .header = "etag:WxRt1"); + + testRequestP( + service, s3, HTTP_VERB_PUT, "/file.txt?partNumber=2&uploadId=WxRt", .content = "7890123456789012", + .sseC = "rA1P"); + testResponseP(service, .header = "eTag:WxRt2"); + + testRequestP( + service, s3, HTTP_VERB_POST, "/file.txt?uploadId=WxRt", + .content = + "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" + "<CompleteMultipartUpload>" + "<Part><PartNumber>1</PartNumber><ETag>WxRt1</ETag></Part>" + "<Part><PartNumber>2</PartNumber><ETag>WxRt2</ETag></Part>" + "</CompleteMultipartUpload>\n"); + testResponseP( + service, + .content = + "<?xml version=\"1.0\" encoding=\"UTF-8\"?>" + "<CompleteMultipartUploadResult><ETag>XXX</ETag></CompleteMultipartUploadResult>"); + + TEST_ASSIGN(write, storageNewWriteP(s3, STRDEF("file.txt"), .tmp = true), "new write"); + TEST_RESULT_VOID(storagePutP(write, BUFSTRDEF("12345678901234567890123456789012")), "write"); + + driver->tmpRetentionMode = storageS3ObjectLockModeGovernance; + // ----------------------------------------------------------------------------------------------------------------- TEST_TITLE("write file in chunks with nothing left over on close");

Message-ID: <gh-pgbackrest-pgbackrest-2918@github.com>
Permalink:  ../gh-pgbackrest-pgbackrest-2918@github.com/

 ·  · 


This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox