Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1sEVOk-001pEs-5I for pgsql-admin@arkaria.postgresql.org; Tue, 04 Jun 2024 14:44:27 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1sEVOk-00HNK2-2B for pgsql-admin@arkaria.postgresql.org; Tue, 04 Jun 2024 14:44:26 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1sEVOj-00HNJu-H6 for pgsql-admin@lists.postgresql.org; Tue, 04 Jun 2024 14:44:25 +0000 Received: from sonic310-24.consmr.mail.ne1.yahoo.com ([66.163.186.205]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1sEVOg-002JSi-5s for pgsql-admin@lists.postgresql.org; Tue, 04 Jun 2024 14:44:24 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1717512259; bh=cPfrpDx4ErgfbEkpqEEeV+5RMld6/2m9vMPYyI6csG0=; h=Date:From:To:In-Reply-To:References:Subject:From:Subject:Reply-To; b=T+AOpov+Py7SuR22Z4tKydPVjlw6Am/UrzzVyz68Bvj315LlNS/7qDZGsNQzcNiydoCW9MXf+QFSjQe0X8ee+0ypk66DR3r6YFmjAVLBr6C6G7GDjI1sUV5/dh5oVXzTO+7h4/gI5GPJmEUtrIofGn5hJVHDicV5BOtl8KJ6CZE+x8wDJ/uKZbUTBPwnwKlHeQkFDy45kOmf1/1WFtxafQvGYeDplscoE/fC1WQG4PgWpZoJ0xGbL1VVL443jPLF47IjTnVih4cwipSkMtkLZukvwovmQkc/Md04X7JAPTnXOAOaEdxU4yJqPO5vJi9In6LkKwTrDi4gfA4ajHmQtg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1717512259; bh=zt3MiJ0DFgrpCy9U01WPmekwUKg0lyipJcjIoqgMVnx=; h=X-Sonic-MF:Date:From:To:Subject:From:Subject; b=Ne4NVitgHyd/d8K/DGOmla2GQGIF/5BTqfmf8evHwQKmwPJ4aXMsN4C5KJewD9t6V/ep3ELC2Af93i3A0+j6RBWIz0fXZT5mty1zY/EOp1LzmrboMUWphl7VFgwuoDEbeTXuxrhxRZcC9Ws6DEF3hVcC9Z+y/4OacqguBaOYu6YZD4bZDXLBIWADKaZROcVUD66BYsFmui11PnocFVIk4S1wfdF8DJKxV8Lsu0X1z6aOo7pg5qCitVE8OhsdHtwme5VRChucxHu/YO7mWlibgJ1NivBbIQqar1u135at4xQSGioCsJTj6h9ZqFfQqbQ/DWPsnGJnlz/eA/dBBo7qZA== X-YMail-OSG: K.tA6dMVM1nPJnRJIUvNIoYSZeiX7_l0epCPjAWqd14lOnjnwBD6KIpr7rgPvhs a4PA_Sj8hYBctle8HDddrM6dBOlLR_PV.wMsUYaW.SATalw.2i1G320TuSE7ln0kUS.Bb7i0Kjtq 5NsXsP3e6qKI6F7L4A1R8__qlMAgyc6cKdKGsQqjNtbBqnfs1XmFRrPkxX3sl9InCNHkzl_n6AM7 E5TL1OYU4HqhloBn22qdU8XGU48sMnxqiFPKmYKU68jY.OEysVAiv8RV0xRF891HXc4vfkeOX4Qt BSsam6CuKX2oD2RLQQY3bxyb.yHENqiuVDLl2V9o7O4W2FlcjCqbrh16zxr9zr713LRAE.PRU2Dm X_0knwtG4yuX2H6nLdyH28F10Ds5NMvxUgoiiXqp11gKcN45aOM9rzzLWVZO.gpq_hzQiYgTucEt v4fqvbu3HmlDQ48O6MbQbAr0DW.y0fPfKxHfx.3oXqphdW_rd0.Csse.4XfC8XIJzsVbqN7XtFf_ CTMn0zOdrdjbVqWz8zGLnAvWOEhe0R5ELfQ13bXLcJkWKMz.HhHJua4bixuVdrZWI3Ic0utLym3A 5RjLlgXYna19_X_rbbbGaJ9qi7YjjWn.Rab0pB.CzbLVRyDS4s8EaW82fEEngV0t_VqyW7nP1bW_ G7kruhre4rkH1PqQJZa7gPcJKbxJZ2wNSvxJyfPbEbKk4ClVgmvbdhqydZz6NBGo3GU.qTTRVy7T t3UwBvESKc8dupWMI0meyzSuAlo1JChkc7NmHtqOEw0saHUYiFhLGRvdbGUQHcAuntgHIJYZkvVX mcnbmyvVghWwRtTIUmW2123Axca_6jJWNoT73asCrBzi.tPuK3Y_9SaRXKD2UIITFuVDIvjXyLH7 XoUKP8aXKVlG7YGpy6zFAu0xmxwcPvSsw5QLNu8MDKBTGn_a9pWcVZAAnbvWydl6p36igNqHxO45 F2WINmNNDd5luklt5GK01PIcf48qeaNonPDeagJr6ZCPXhvaAJuE9T8TqvdRWpLxLiX7AUgha.AD UKVdFeLBof8ItXXY_bm1HQiXYRNfJmd9LA.LWYk3Fa2MGpKbe5RV8JYWmoSM1EhsMSxTGnMmZ2F2 iK2yGLdHxFIKXNWGJb1KrM8zefxOaSuF3.nrcZyuj.w96MEe8rG3r_oYZDFNpX1Fbgeb9cjjDsma DG0JoD7ElE6rtBGwy6x9k1oER.OCuCItyWOuZu6aSB3tCtRRo8lQ51wOWDcLz4rJUVz.NVf3H0mr VRh7DwaU9NC4.aEYHrcnQbDz7Q8vs7g87Hgyz_mGkEwuz88r.r6XXq_qzcO8MRraA2IBcINFCApE mZszUnwGtoLIqHJoN9Kj5CHliWJFQMIQbhiuxfq04rODlH7_aub_YYba9IlN.7bMQjVhD0Ex0wSm Lbf6Z8CbT.XnFdhVXL9bDbLQPeSuH7yfuQXSqo0kXYpQG5VaIxQXhSQYqea199rdcGO2HvWh9yEf 0UF7tz9CiRTnAvQ8RdwsGhxS26ASAuVIbHNceMqB10AVtKy1bAnhompSdOTnVlO1U6ALvSOBFw1h sU8sYK0WZQeBk29kg0hrhfI20ae0GvdDkdNUA4zxMrgkxQSQWoZUGButw4rUcE9PFDh.VZawJA8P T_PMtQ4NllwdMzKFf2MF4tyGZx6qgLKSxxnOd9kgCmT_9Ex7vG1C9kUy0ONiy2eK1Bvxo5bK0Lfx qQbnZJbF8fsxY2XM3OFOzUZklic7BlDKjtfe3fG3oldu_AaVwWEtkht5bGBFSYV4RcMvP1F6gKHJ sDxccu.lZEA7V.mzdSyuxEtHYa39S0FiCPdgdS_n0g5BKlrzNbISx6pLZ.CTEoue9uU1AOb_OSuh mYHOvKbbO1q7A7zbQe5D1tLLxDQ2yefvq3fxOVjJCZRCxXfG7M7yp.bkl8rO9SJ1QWT.2it0iJt1 .Rq805tNDPZu_i5_Zpbmn9kNooNgHWrkNAcE_7G5QMBku5NErKk1TiZsXX2fULZp.6Xe1o_Bs_hw ActqJiwN1.Pcn80N_LCMar1vwncmVC_SJJj4vet17k1B7O06pfmxIsIbo4fQj7SmTp4OjJ0277JR .T.Vdtdm9vMat708vVYZAeouBLm0DADXcmTC6Viom64NGe5bk2T80f2O7tUENAeknoTtCkS_UTvx MmBmuvEGCwHShxZ1eXJ.OCVgF8U1TjP6d5WvEFSMSJwNoVO7.gre5yx4epJcacWqW0McojAJS_Wy V6nqorg9LPPCEEF6nEJG_rNoqqZXqS7_r2OJOoJULGNjiw_zX86pXHUGXIkg- X-Sonic-MF: X-Sonic-ID: 4b27c53b-82ad-414e-9bb2-079619fc5075 Received: from sonic.gate.mail.ne1.yahoo.com by sonic310.consmr.mail.ne1.yahoo.com with HTTP; Tue, 4 Jun 2024 14:44:19 +0000 Date: Tue, 4 Jun 2024 14:44:18 +0000 (UTC) From: Naveed Iftikhar To: "pgsql-admin@lists.postgresql.org" , Scott Ribe Message-ID: <579383683.1720596.1717512258251@mail.yahoo.com> In-Reply-To: <96F639B3-D9FC-49D2-9FAC-5799010C314E@elevated-dev.com> References: <1353402555.1511878.1717467795288.ref@mail.yahoo.com> <1353402555.1511878.1717467795288@mail.yahoo.com> <96F639B3-D9FC-49D2-9FAC-5799010C314E@elevated-dev.com> Subject: Re: pgBackRest configuration with DDBoost MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_1720595_1508247542.1717512258250" X-Mailer: WebService/1.1.22407 YMailNorrin Content-Length: 3998 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk ------=_Part_1720595_1508247542.1717512258250 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Thanks Scott for your input, Our database is keep growing and application o= wner want to have incremental backups daily. Can you share how you have con= figured the lz4 compression with pgbackrest.=C2=A0 Any special storage requ= irements? Best Regards=C2=A0-Naveed-=20 On Monday, June 3, 2024 at 10:22:28 PM CDT, Scott Ribe wrote: =20 =20 > On Jun 3, 2024, at 9:06=E2=80=AFPM, Ron Johnson wrote: >=20 > Takes 2 hr 45 min for us to do a weekly full backup of the 6.1TB instance= using integrated gzip compression (it's an old RHEL6 system, so nothing be= tter) and 24 threads.=C2=A0 Size 3.3T after compression, which is surprisin= g based on how much already-compressed bytea data there is. FYI, Naveed, there's a tradeoff available in current versions of pgbackrest= . It can be configured to use lz4 compression. For us, that means close to = 50% larger backups than gzip, but many times faster. 3.2TB in 30 minutes, u= sing 8 processes, 725GB backup size--no already compressed bytea, but a goo= d bit of incompressible UUID columns. Good enough that I stopped doing daily incrementals, and just take full bac= kups daily. =20 ------=_Part_1720595_1508247542.1717512258250 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thanks Scott for your input,= Our database is keep growing and application owner want to have incrementa= l backups daily. Can you share how you have configured the lz4 compression = with pgbackrest.  Any special storage requirements?

Best Regards
 -Naveed-


=20
=20
On Monday, June 3, 2024 at 10:22:28 PM CDT, Scott R= ibe <scott_ribe@elevated-dev.com> wrote:


=20 =20
> On Jun 3, 2024, at 9:06=E2=80=AF= PM, Ron Johnson <ronljohnsonjr@gmail.com> wrote= :

>
> Takes 2 hr 45 min for us to do a= weekly full backup of the 6.1TB instance using integrated gzip compression= (it's an old RHEL6 system, so nothing better) and 24 threads.  Size 3= .3T after compression, which is surprising based on how much already-compre= ssed bytea data there is.


FYI, N= aveed, there's a tradeoff available in current versions of pgbackrest. It c= an be configured to use lz4 compression. For us, that means close to 50% la= rger backups than gzip, but many times faster. 3.2TB in 30 minutes, using 8= processes, 725GB backup size--no already compressed bytea, but a good bit = of incompressible UUID columns.

Good e= nough that I stopped doing daily incrementals, and just take full backups d= aily.
<= br clear=3D"none">
------=_Part_1720595_1508247542.1717512258250--