MIME-Version: 1.0
References: <013D63E2-5D75-492E-85FF-1D5CC0148C82@gmail.com>
 <CAJ7c6TMYvYBQpgv8k+4QnajqPcroOzrj7d3snBBTKX4tHQfVYw@mail.gmail.com>
 <499686.1759250489@sss.pgh.pa.us>
 <68dcd1f2.df0a0220.3300c0.f7af@mx.google.com>
 <CAKZiRmwHKY=KTjBEL3S2cVQpo1OjHyky4BdgJm4Hkv1-ig9PfQ@mail.gmail.com>
 <CAJ7c6TMCyhX4Z7FrS=ORCcKS1F_iZzfUuQM3shx64WKynoxjsA@mail.gmail.com>
In-Reply-To: 
 <CAJ7c6TMCyhX4Z7FrS=ORCcKS1F_iZzfUuQM3shx64WKynoxjsA@mail.gmail.com>
From: Jakub Wartak <jakub.wartak@enterprisedb.com>
Date: Wed, 1 Oct 2025 14:05:53 +0200
Message-ID: 
 <CAKZiRmy0CK3m0-raCdTDELg0JjY7qAqzEN9P5n4N4wGw6ys4tw@mail.gmail.com>
Subject: Re: The ability of postgres to determine loss of files of the main
 fork
To: Aleksander Alekseev <aleksander@tigerdata.com>
Cc: pgsql-hackers@lists.postgresql.org, Michael Banck <mbanck@gmx.net>,
	Tom Lane <tgl@sss.pgh.pa.us>, Frits Hoogland <frits.hoogland@gmail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: 
 <https://www.postgresql.org/message-id/CAKZiRmy0CK3m0-raCdTDELg0JjY7qAqzEN9P5n4N4wGw6ys4tw%40mail.gmail.com>
Precedence: bulk

On Wed, Oct 1, 2025 at 1:46=E2=80=AFPM Aleksander Alekseev
<aleksander@tigerdata.com> wrote:
>
> Hi Jakub,
>
> > IMHO all files should be opened at least on startup to check integrity,
>
> That might be a lot of files to open.

I was afraid of that, but let's say modern high-end is 200TB big DB,
that's like 200*1024 1GB files, but I'm getting such time(1) timings
for 204k files on ext4:

$ time ./createfiles                      # real    0m2.157s, it's
open(O_CREAT)+close()
$ time ls -l many_files_dir/ > /dev/null # real    0m0.734s
$ time ./openfiles                          # real    0m0.297s , for
already existing ones (hot)
$ time ./openfiles                          # real    0m1.456s , for
already existing ones (cold, echo 3 > drop_caches sysctl)

Not bad in my book as a one time activity. It could pose a problem
potentially with some high latency open() calls, maybe NFS or
something remote I guess.

> Even if you can open a file it doesn't mean it's not empty

Correct, I haven't investigated that rabbithole...

> or is not corrupted.

I think checksums guard users well in this case as they would get
notified that stuff is wonky (much better than wrong result/silent
data loss)

-J.