Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mA8NX-00088E-9B for pgsql-admin@arkaria.postgresql.org; Sun, 01 Aug 2021 10:07:31 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1mA8NW-0006Qp-2F for pgsql-admin@arkaria.postgresql.org; Sun, 01 Aug 2021 10:07:30 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mA8NV-0006Qh-PV for pgsql-admin@lists.postgresql.org; Sun, 01 Aug 2021 10:07:29 +0000 Received: from mail.am-soft.de ([83.218.36.120]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mA8NT-0003dJ-5u for pgsql-admin@lists.postgresql.org; Sun, 01 Aug 2021 10:07:29 +0000 Received: from mail.am-soft.de (localhost [127.0.0.1]) by mail.am-soft.de (Postfix) with ESMTP id 94B8D3E71E87; Sun, 1 Aug 2021 12:07:25 +0200 (CEST) Received: from Bit02EX.bitstoregruppe.local (exchange01.bitstore.group [116.202.141.225]) by mail.am-soft.de (Postfix) with ESMTPA id 874613E71E85 for ; Sun, 1 Aug 2021 12:07:25 +0200 (CEST) Received: from tschoening-nb.fritz.box (77.78.163.99) by Bit02EX.bitstoregruppe.local (192.168.254.21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2106.2; Sun, 1 Aug 2021 12:07:02 +0200 Date: Sun, 1 Aug 2021 12:07:09 +0200 From: =?utf-8?B?VGhvcnN0ZW4gU2Now7ZuaW5n?= Organization: AM-SoFT IT-Service - Bitstore Hameln GmbH Message-ID: <1429691210.20210801120709@am-soft.de> To: Subject: Any experiences with de-duplicating backup of dumps? MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [77.78.163.99] X-ClientProxiedBy: Bit02EX.bitstoregruppe.local (192.168.254.21) To Bit02EX.bitstoregruppe.local (192.168.254.21) X-C2ProcessedOrg: e1a4acb5-ea26-4a8f-9559-e23e579650e8 X-AV-Checked: ClamAV using ClamSMTP List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk Hi everyone, I'm in the process of changing backups from simply storing encrypted dumps (encrypted by the host) on some external NAS to using the software Borg, a de-duplicating archiver. > Deduplication is performed across multiple backups, both on data and > metadata, using Chunks created by the chunker using the Buzhash > algorithm. https://borgbackup.readthedocs.io/en/0.30.0/internals.html I've created dumps with highest compression available to make the backups as small as possible in the past, to use the available storage on the NAS as best as possible. Now I'm wondering if I should keep compression before de-duplication (and again compression) of Borg or if things might behave better with less or none compression at all. I could e.g. use the plain text format, though, the custom one seems more flexible and supports no compression as well. What makes me additionally wonder is if Postgres processes things in constant order, which de-duplication should benefit from most likely. Especially if e.g. rows are mostly added instead of updated, which is some of my use cases. Do you have any test results and experiences already about that topic? Which formats do you use with de-duplication? Or wasn't it worth it at all? Thanks! Mit freundlichen Gr=C3=BC=C3=9Fen Thorsten Sch=C3=B6ning --=20 AM-SoFT IT-Service - Bitstore Hameln GmbH Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f=C3=BCr IT u= nd TK E-Mail: Thorsten.Schoening@AM-SoFT.de Web: http://www.AM-SoFT.de/ Tel: 05151- 9468- 0 Tel: 05151- 9468-55 Fax: 05151- 9468-88 Mobil: 0178-8 9468-04 AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Ham= eln AG Hannover HRB 221853 - Gesch=C3=A4ftsf=C3=BChrer: Janine Galonska