Date: Mon, 14 Mar 2022 11:35:47 -0500
From: Justin Pryzby <pryzby@telsasoft.com>
To: Dipesh Pandit <dipesh.pandit@gmail.com>
Cc: Robert Haas <robertmhaas@gmail.com>, Andres Freund <andres@anarazel.de>,
	Jeevan Ladhe <jeevanladhe.os@gmail.com>,
	Alvaro Herrera <alvherre@alvh.no-ip.org>,
	"Shinoda, Noriyoshi (PN Japan FSIP)" <noriyoshi.shinoda@hpe.com>,
	Abhijit Menon-Sen <ams@toroid.org>,
	Dmitry Dolgov <9erthalion6@gmail.com>,
	Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>,
	Mark Dilger <mark.dilger@enterprisedb.com>,
	pgsql-hackers@postgresql.org,
	tushar <tushar.ahuja@enterprisedb.com>
Subject: Re: refactoring basebackup.c (zstd workers)
Message-ID: <20220314163547.GT28503@telsasoft.com>
References: 
 <CANm22CgVMa85O1akgs+DOPE8NSrT1zbz5_vYfS83_r+6nCivLQ@mail.gmail.com>
 <CA+Tgmoap+hTD2-QNPJLH4tffeFE8MX5+xkbFKMU3FKBy=ZSNKA@mail.gmail.com>
 <CANm22Ciseat_kJwY_vixu6Wen48E7EVV5E+tWzZ9qN7SYJYOfQ@mail.gmail.com>
 <CA+TgmoZZmZctarpJyRiLNcePeTUS0z-Y75tAPUnFYQvRcj1Xzw@mail.gmail.com>
 <CANm22CiVwmix7JGOGhJYsbNt9M8AH-svG2HQfPP=MdZ2kMHfzw@mail.gmail.com>
 <20220311010223.GI28503@telsasoft.com>
 <CA+Tgmob6Rnjz-Qv32h3yJn8nnUkLhrtQDAS4y5AtsgtorAFHRA@mail.gmail.com>
 <20220312015253.zokhkwdn5xbbd4vf@alap3.anarazel.de>
 <CA+TgmoZ10QPYaUxe8+RfJT5ptyDe-ML=oE_FyyFa+sNS2GBgZw@mail.gmail.com>
 <CAN1g5_F0KBgitEVLO8T0ynaBcQBfRMCjH9wKguDJZpX1hF3gFA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: 
 <CAN1g5_F0KBgitEVLO8T0ynaBcQBfRMCjH9wKguDJZpX1hF3gFA@mail.gmail.com>
User-Agent: Mutt/1.9.4 (2018-02-28)
Archived-At: 
 <https://www.postgresql.org/message-id/20220314163547.GT28503%40telsasoft.com>
Precedence: bulk

On Mon, Mar 14, 2022 at 09:41:35PM +0530, Dipesh Pandit wrote:
> I tried to implement support for parallel ZSTD compression. The
> library provides an option (ZSTD_c_nbWorkers) to specify the
> number of compression workers. The number of parallel
> workers can be set as part of compression parameter and if this
> option is specified then the library performs parallel compression
> based on the specified number of workers.
> 
> User can specify the number of parallel worker as part of
> --compress option by appending an integer value after at sign (@).
> (-Z, --compress=[{client|server}-]{gzip|lz4|zstd}[:LEVEL][@WORKERS])

I suggest to use a syntax that's more general than that, maybe something like

:[level=]N,parallel=N,flag,flag,...

For example, someone may want to use zstd "long" mode or (when it's released)
rsyncable mode, or specify fine-grained compression parameters (strategy,
windowLog, hashLog, etc).

I hope the same syntax will be shared with wal_compression and pg_dump.
And libpq, if that patch progresses.

BTW, I think this may be better left for PG16.

-- 
Justin