Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pu5B7-0003aK-0J for pgsql-hackers@arkaria.postgresql.org; Wed, 03 May 2023 05:37:25 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1pu5B5-0001YZ-T4 for pgsql-hackers@arkaria.postgresql.org; Wed, 03 May 2023 05:37:23 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pu5B5-0001YQ-Jf for pgsql-hackers@lists.postgresql.org; Wed, 03 May 2023 05:37:23 +0000 Received: from mail-ed1-x536.google.com ([2a00:1450:4864:20::536]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1pu5B3-000FUC-9X for pgsql-hackers@postgresql.org; Wed, 03 May 2023 05:37:22 +0000 Received: by mail-ed1-x536.google.com with SMTP id 4fb4d7f45d1cf-5083bd8e226so7266437a12.3 for ; Tue, 02 May 2023 22:37:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1683092240; x=1685684240; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=WEpUH/OJo6HI2bTInaH+Qmzs2ZqIM5thXphHbkD2AEs=; b=pbh7TC8+h7Hwm4UkRBcmrS3NfsdlHPkxakmSvU51gbqptJNLURqERkLHA/QgLws8PI jP3ZY40t+uS6mB8HBns6/dyMN7APXjxIyOAP8cAHF+HO2eYGfuEAsqN+aAmp8Ud3lTjb Ni6qRzvhVpHpl89yzi/R6+eywGVvL69jhGyE7/cu4x6AAjxt4+h1H+mOcXHJb5XznhUd FSmt0Q080ITQo4awtUrk/mIiW3gXIqfJEmhEfqP/S9R7jNIPeqFgpgNLoxyyIbkwBdbX t83aA7+7uU+c+fixTLdebybtq0lJ79zPOTpSQSGbsmnL3Kjb3qEduKQda8LZe7RVXA8h YU7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683092240; x=1685684240; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WEpUH/OJo6HI2bTInaH+Qmzs2ZqIM5thXphHbkD2AEs=; b=G+C+ztfaC0DS/C6knWrYID7a9SFEnOIpXheXBw1ghagfIhUNeZ+Z7SOsKmKU8fk7YC 0lQ9cK/SrDaQrB/2CgiRaZw87Eb6iQUXV/3+M8JlikCEVuRWo6MayT7G0KajQSrwMtSC qP8YfhA8Jd3RASF3gjUNSJlytXfrj4nH4c8ZNJNH+ahLfto1HyZPUnH7fTSjDHpTtJMU 2ITUPhkhaRDQhGC4fi/lGGbnnBBp/JzvuFM5Ye8xw1Z9u+gCzYN/+J5eqSmJDkuyRyXC jlejJo41OVh24Ts3wJIbN0DsQvPN4lDSxYS0XEtcySrdxLgxiby1+C9rUgid5pp3nIC3 K5Dg== X-Gm-Message-State: AC+VfDzjL66V4OfFrwIrR1Qp1iIZldvyHr1c1RNfDD09jV2gy7nVZUOS QKJXl6uk3u98/zgW7v2w8BkwBYig7fY2MsyPha3DY8ibn+Q= X-Google-Smtp-Source: ACHHUZ4UfTqj/xr1GVBXO828dqva5tRbSOdVjKtCVHsnkewzNWHFjT7tpeBSJEqk6G9DQrkAD1oV/3JTUPwzIjZdyuw= X-Received: by 2002:a17:907:c21:b0:933:3814:e0f4 with SMTP id ga33-20020a1709070c2100b009333814e0f4mr2811709ejc.16.1683092239558; Tue, 02 May 2023 22:37:19 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Thomas Munro Date: Wed, 3 May 2023 17:36:42 +1200 Message-ID: Subject: Re: Large files for relations To: Pavel Stehule Cc: pgsql-hackers Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On Wed, May 3, 2023 at 5:21=E2=80=AFPM Thomas Munro wrote: > rsync --link-dest I wonder if rsync will grow a mode that can use copy_file_range() to share blocks with a reference file (=3D previous backup). Something like --copy-range-dest. That'd work for large-file relations (assuming a file system that has block sharing, like XFS and ZFS). You wouldn't get the "mtime is enough, I don't even need to read the bytes" optimisation, which I assume makes all database hackers feel a bit queasy anyway, but you'd get the space savings via the usual rolling checksum or a cheaper version that only looks for strong checksum matches at the same offset, or whatever other tricks rsync might have up its sleeve.