pg.ddx.io  pgsql-committers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Tomas Vondra <tomas.vondra@enterprisedb.com>
To: Michael Paquier <michael@paquier.xyz>
Cc: Daniel Gustafsson <daniel@yesql.se>
Cc: Christoph Berg <myon@debian.org>
Cc: Tomas Vondra <tomas.vondra@postgresql.org>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: lz4 --rm on Ubuntu 18.04 (Add LZ4 compression to pg_dump)
Date: Tue, 14 Mar 2023 00:16:16 +0100
Message-ID: <74f02417-d18a-71dd-a406-8af41cbeb6c4@enterprisedb.com> (raw)
In-Reply-To: <0e8e53fa-5087-2151-dcd6-a1d3ad3ed493@enterprisedb.com>
References: <E1pVI5z-000MHR-Ch@gemulon.postgresql.org>
	<ZAjL96N9ZW84U59p@msg.df7cb.de>
	<E6DD8AC9-3EE4-4E7B-A2BC-53FDF04D1ED0@yesql.se>
	<819f6883-f7cf-74fd-e918-4c06a9d76d0b@enterprisedb.com>
	<ZAkol+4UyWuBR7PS@paquier.xyz>
	<0e8e53fa-5087-2151-dcd6-a1d3ad3ed493@enterprisedb.com>



On 3/9/23 19:00, Tomas Vondra wrote:
> 
> 
> On 3/9/23 01:30, Michael Paquier wrote:
>> On Thu, Mar 09, 2023 at 12:39:08AM +0100, Tomas Vondra wrote:
>>> IMO we should fix that. We have a bunch of buildfarm members running on
>>> Ubuntu 18.04 (or older) - it's true none of them seems to be running TAP
>>> tests. But considering how trivial the fix is ...
>>>
>>> Barring objections, I'll push a fix early next week.
>>
>> +1, better to change that, thanks.  Actually, would --rm be OK even on
>> Windows?  As far as I can see, the CI detects a LZ4 command for the
>> VS2019 environment but not the liblz4 libraries that would be needed
>> to trigger the set of tests.
> 
> Thanks for noticing that. I'll investigate next week.
> 

So, here's a fix that should (I think) replace the 'lz4 --rm' with a
simple 'rm'. I have two doubts about this, though:


1) I haven't found a simple way to inject additional command into the
test. The pg_dump runs have a predefined list of "steps" to run:

  -- compress_cmd
  -- glob_patterns
  -- command_like
  -- restore_cmd

and I don't think there's a good place to inject the 'rm' so I ended up
adding a 'cleanup_cmd' right after 'compress_cmd'. But it seems a bit
strange / hacky. Maybe there's a better way?


2) I wonder if Windows will know what 'rm' means. I haven't found any
TAP test doing 'rm' and don't see 'rm' in any $ENV either.


That being said, I have no idea how to make this work on our Windows CI.
As mentioned, the environment is missing the lz4 library - there's a

  setup_additional_packages_script: |
    REM choco install -y --no-progress ...

in the .yml file, but AFAICS the chocolatey does not have lz4 :-/


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Attachments:

lz4-rm-tap-test.patch (text/x-patch, 1.4K) download diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 9c354213ce..b3dcf6ff6d 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -178,11 +178,18 @@ my %pgdump_runs = ( compress_cmd => { program => $ENV{'LZ4'}, args => [ - '-z', '-f', '--rm', + '-z', '-f', "$tempdir/compression_lz4_dir/blobs.toc", "$tempdir/compression_lz4_dir/blobs.toc.lz4", ], }, + # remove the source (uncompressed) .toc file + cleanup_cmd => { + program => 'rm', + args => [ + "$tempdir/compression_lz4_dir/blobs.toc", + ], + }, # Verify that data files were compressed glob_patterns => [ "$tempdir/compression_lz4_dir/toc.dat", @@ -4274,6 +4281,20 @@ foreach my $run (sort keys %pgdump_runs) command_ok(\@full_compress_cmd, "$run: compression commands"); } + if ($pgdump_runs{$run}->{cleanup_cmd}) + { + my ($cleanup_cmd) = $pgdump_runs{$run}->{cleanup_cmd}; + my $cleanup_program = $cleanup_cmd->{program}; + + # Skip the rest of the test if the compression program is + # not defined. + next if (!defined($cleanup_cmd) || $cleanup_cmd eq ''); + + my @full_cleanup_cmd = + ($cleanup_cmd->{program}, @{ $cleanup_cmd->{args} }); + command_ok(\@full_cleanup_cmd, "$run: cleanup commands"); + } + if ($pgdump_runs{$run}->{glob_patterns}) { my $glob_patterns = $pgdump_runs{$run}->{glob_patterns};

view thread (10+ messages)  latest in thread

Message-ID: <74f02417-d18a-71dd-a406-8af41cbeb6c4@enterprisedb.com>
Permalink:  ../74f02417-d18a-71dd-a406-8af41cbeb6c4@enterprisedb.com/
Also on:    postgresql.org/message-id/74f02417-d18a-71dd-a406-8af41cbeb6c4@enterprisedb.com

 ·  · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-committers@postgresql.org
  Cc: tomas.vondra@enterprisedb.com, michael@paquier.xyz, daniel@yesql.se, myon@debian.org, tomas.vondra@postgresql.org, pgsql-hackers@lists.postgresql.org
  Subject: Re: lz4 --rm on Ubuntu 18.04 (Add LZ4 compression to pg_dump)
  In-Reply-To: <74f02417-d18a-71dd-a406-8af41cbeb6c4@enterprisedb.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox