public inbox for [email protected]  

help / color / mirror / Atom feed

From: Thomas Munro <[email protected]>
To: Pavel Stehule <[email protected]>
Cc: pgsql-hackers <[email protected]>
Subject: Re: Large files for relations
Date: Wed, 3 May 2023 17:21:14 +1200
Message-ID: <CA+hUKGJxtfM3hLrmjP910_nm3DFCFZaUzbHG5Rhw6FpdLwpc6A@mail.gmail.com> (raw)
In-Reply-To: <CAFj8pRCMQXtOUhxnGLYLL+T-5HwfjWLKYVN_hxi2VcG_PDD_GA@mail.gmail.com>
References: <CA+hUKG+BGXwMbrvzXAjL8VMGf25y_ga_XnO741g10y0=m6dDiA@mail.gmail.com>
	<CAFj8pRCMQXtOUhxnGLYLL+T-5HwfjWLKYVN_hxi2VcG_PDD_GA@mail.gmail.com>

On Tue, May 2, 2023 at 3:28 PM Pavel Stehule <[email protected]> wrote:
> I like this patch - it can save some system sources - I am not sure how much, because bigger tables usually use partitioning usually.

Yeah, if you only use partitions of < 1GB it won't make a difference.
Larger partitions are not uncommon, though.

> Important note - this feature breaks sharing files on the backup side - so before disabling 1GB sized files, this issue should be solved.

Hmm, right, so there is a backup granularity continuum with "whole
database cluster" at one end, "only files whose size, mtime [or
optionally also checksum] changed since last backup" in the middle,
and "only blocks that changed since LSN of last backup" at the other
end.  Getting closer to the right end of that continuum can make
backups require less reading, less network transfer, less writing
and/or less storage space depending on details.  But this proposal
moves the middle thing further to the left by changing the granularity
from 1GB to whole relation, which can be gargantuan with this patch.
Ultimately we need to be all the way at the right on that continuum,
and there are clearly several people working on that goal.

I'm not involved in any of those projects, but it's fun to think about
an alien technology that produces complete standalone backups like
rsync --link-dest (as opposed to "full" backups followed by a chain of
"incremental" backups that depend on it so you need to retain them
carefully) while still sharing disk blocks with older backups, and
doing so with block granularity.  TL;DW something something WAL
something something copy_file_range().

view thread (4+ messages)  latest in thread

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected]
  Subject: Re: Large files for relations
  In-Reply-To: <CA+hUKGJxtfM3hLrmjP910_nm3DFCFZaUzbHG5Rhw6FpdLwpc6A@mail.gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox