From: Tomas Vondra <tomas.vondra@enterprisedb.com>
To: Michael Paquier <michael@paquier.xyz>
Cc: Daniel Gustafsson <daniel@yesql.se>
Cc: Christoph Berg <myon@debian.org>
Cc: Tomas Vondra <tomas.vondra@postgresql.org>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: lz4 --rm on Ubuntu 18.04 (Add LZ4 compression to pg_dump)
Date: Tue, 14 Mar 2023 00:16:16 +0100
Message-ID: <74f02417-d18a-71dd-a406-8af41cbeb6c4@enterprisedb.com> (raw)
In-Reply-To: <0e8e53fa-5087-2151-dcd6-a1d3ad3ed493@enterprisedb.com>
References: <E1pVI5z-000MHR-Ch@gemulon.postgresql.org>
<ZAjL96N9ZW84U59p@msg.df7cb.de>
<E6DD8AC9-3EE4-4E7B-A2BC-53FDF04D1ED0@yesql.se>
<819f6883-f7cf-74fd-e918-4c06a9d76d0b@enterprisedb.com>
<ZAkol+4UyWuBR7PS@paquier.xyz>
<0e8e53fa-5087-2151-dcd6-a1d3ad3ed493@enterprisedb.com>
On 3/9/23 19:00, Tomas Vondra wrote:
>
>
> On 3/9/23 01:30, Michael Paquier wrote:
>> On Thu, Mar 09, 2023 at 12:39:08AM +0100, Tomas Vondra wrote:
>>> IMO we should fix that. We have a bunch of buildfarm members running on
>>> Ubuntu 18.04 (or older) - it's true none of them seems to be running TAP
>>> tests. But considering how trivial the fix is ...
>>>
>>> Barring objections, I'll push a fix early next week.
>>
>> +1, better to change that, thanks. Actually, would --rm be OK even on
>> Windows? As far as I can see, the CI detects a LZ4 command for the
>> VS2019 environment but not the liblz4 libraries that would be needed
>> to trigger the set of tests.
>
> Thanks for noticing that. I'll investigate next week.
>
So, here's a fix that should (I think) replace the 'lz4 --rm' with a
simple 'rm'. I have two doubts about this, though:
1) I haven't found a simple way to inject additional command into the
test. The pg_dump runs have a predefined list of "steps" to run:
-- compress_cmd
-- glob_patterns
-- command_like
-- restore_cmd
and I don't think there's a good place to inject the 'rm' so I ended up
adding a 'cleanup_cmd' right after 'compress_cmd'. But it seems a bit
strange / hacky. Maybe there's a better way?
2) I wonder if Windows will know what 'rm' means. I haven't found any
TAP test doing 'rm' and don't see 'rm' in any $ENV either.
That being said, I have no idea how to make this work on our Windows CI.
As mentioned, the environment is missing the lz4 library - there's a
setup_additional_packages_script: |
REM choco install -y --no-progress ...
in the .yml file, but AFAICS the chocolatey does not have lz4 :-/
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:lz4-rm-tap-test.patch (text/x-patch, 1.4K) downloaddiff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.plindex 9c354213ce..b3dcf6ff6d 100644--- a/src/bin/pg_dump/t/002_pg_dump.pl+++ b/src/bin/pg_dump/t/002_pg_dump.pl@@ -178,11 +178,18 @@ my %pgdump_runs = (
compress_cmd => {
program => $ENV{'LZ4'},
args => [
- '-z', '-f', '--rm',+ '-z', '-f',
"$tempdir/compression_lz4_dir/blobs.toc",
"$tempdir/compression_lz4_dir/blobs.toc.lz4",
],
},
+ # remove the source (uncompressed) .toc file+ cleanup_cmd => {+ program => 'rm',+ args => [+ "$tempdir/compression_lz4_dir/blobs.toc",+ ],+ },
# Verify that data files were compressed
glob_patterns => [
"$tempdir/compression_lz4_dir/toc.dat",
@@ -4274,6 +4281,20 @@ foreach my $run (sort keys %pgdump_runs)
command_ok(\@full_compress_cmd, "$run: compression commands");
}
+ if ($pgdump_runs{$run}->{cleanup_cmd})+ {+ my ($cleanup_cmd) = $pgdump_runs{$run}->{cleanup_cmd};+ my $cleanup_program = $cleanup_cmd->{program};++ # Skip the rest of the test if the compression program is+ # not defined.+ next if (!defined($cleanup_cmd) || $cleanup_cmd eq '');++ my @full_cleanup_cmd =+ ($cleanup_cmd->{program}, @{ $cleanup_cmd->{args} });+ command_ok(\@full_cleanup_cmd, "$run: cleanup commands");+ }+
if ($pgdump_runs{$run}->{glob_patterns})
{
my $glob_patterns = $pgdump_runs{$run}->{glob_patterns};
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-committers@postgresql.org
Cc: tomas.vondra@enterprisedb.com, michael@paquier.xyz, daniel@yesql.se, myon@debian.org, tomas.vondra@postgresql.org, pgsql-hackers@lists.postgresql.org
Subject: Re: lz4 --rm on Ubuntu 18.04 (Add LZ4 compression to pg_dump)
In-Reply-To: <74f02417-d18a-71dd-a406-8af41cbeb6c4@enterprisedb.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox