Subject: Re: post-freeze damage control

From: Tom Kincaid <[email-masked]>
Date: Wed, 10 Apr 2024 20:23:19 -0400

>
> > Yeah, that's an excellent practive, but is why I'm less worried for
> > this feature.  The docs at [1] caution about "not to remove earlier
> > backups if they might be needed when restoring later incremental
> > backups".  Like Alvaro said, should we insist a bit more about the WAL
> > retention part in this section of the docs, down to the last full
> > backup?
>
> I think that would make sense in general. But if we are doing it because
> we lack confidence in the incremental backup feature maybe that's a sign
> that the feature should be released as experimental (or not released at
> all).
>
>
The extensive Beta process we have can be used to build confidence we need
in a feature that has extensive review and currently has no known issues or
outstanding objections.

> Regards,
> -David

-- 
Thomas John Kincaid

---

From: David Steele <[email-masked]>
Date: Thu, 11 Apr 2024 11:52:00 +1000

On 4/11/24 10:23, Tom Kincaid wrote:
> 
> The extensive Beta process we have can be used to build confidence we 
> need in a feature that has extensive review and currently has no known 
> issues or outstanding objections.

I did have objections, here [1] and here [2]. I think the complexity, 
space requirements, and likely performance issues involved in restores 
are going to be a real problem for users. Some of these can be addressed 
in future releases, but I can't escape the feeling that what we are 
releasing here is half-baked.

Also, there are outstanding issues here [3] and now here [4].

Regards,
-David

[1] 
https://www.postgresql.org/message-id/590e3017-da1f-4af6-9bf0-1679511ca7e5%40pgmasters.net
[2] 
https://www.postgresql.org/message-id/11b38a96-6ded-4668-b772-40f992132797%40pgmasters.net
[3] 
https://www.postgresql.org/message-id/flat/05fb32c9-18d8-4f72-9af3-f41576c33119%40pgmasters.net#bb04b896f0f0147c10cee944a1391c1e
[4] 
https://www.postgresql.org/message-id/flat/9badd24d-5bd9-4c35-ba85-4c38a2feb73e%40pgmasters.net

---

From: Tomas Vondra <[email-masked]>
Date: Thu, 11 Apr 2024 12:26:38 +0200

> releasing here is half-baked.

I haven't been part of those discussions, and that part of the thread is
a couple months old already, so I'll share my view here instead.

I do not think it's half-baked. I certainly agree there are limitations,
and there's all kinds of bells and whistles we could add, but I think
the fundamental infrastructure is corrent and a meaningful step forward.
Would I wish it to handle .tar for example? Sure I would. But I think
it's something we can add in the future - if we require all of this to
happen in a single release, it'll never happen.

FWIW that discussion also mentions stuff that I think the feature should
not do. In particular, I don't think the ambition was (or should be) to
make pg_basebackup into a stand-alone tool. I always saw pg_basebackup
more as an interface to "backup steps" correctly rather than a complete
backup solution that'd manage backup registry, retention, etc.

> Also, there are outstanding issues here [3] and now here [4].

I agree with some of this, I'll respond in the threads.

regards
Tomas

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

---

From: David Steele <[email-masked]>
Date: Fri, 12 Apr 2024 07:48:12 +1000

> happen in a single release, it'll never happen.

Fair enough, but the current release is extremely limited and it would 
be best if that was well understood by users.

> backup solution that'd manage backup registry, retention, etc.

Right -- this is exactly my issue. pg_basebackup was never easy to use 
as a backup solution and this feature makes it significantly more 
complicated. Complicated enough that it would be extremely difficult for 
most users to utilize in a meaningful way.

But they'll try because it is a new pg_basebackup feature and they'll 
assume it is there to be used. Maybe it would be a good idea to make it 
clear in the documentation that significant tooling will be required to 
make it work.

Regards,
-David

---

From: Robert Haas <[email-masked]>
Date: Thu, 11 Apr 2024 22:15:27 -0400

> clear in the documentation that significant tooling will be required to

I don't agree with that idea. LOTS of what we ship takes a significant
amount of effort to make it work. You may well need a connection
pooler. You may well need a failover manager which may or may not be
separate from your connection pooler. You need a backup tool. You need
a replication management tool which may or may not be separate from
your backup tool and may or may not be separate from your failover
tool. You probably need various out-of-core connections for the
programming languages you need. You may need a management tool, and
you probably need a monitoring tool. Some of the tools you might
choose to do all that stuff themselves have a whole bunch of complex
dependencies. It's a mess.

Now, if someone were to say that we ought to talk about these issues
in our documentation and maybe give people some ideas about how to get
started, I would likely be in favor of that, modulo the small
political problem that various people would want their solution to be
the canonical one to which everyone gets referred. But I think it's
wrong to pretend like this feature is somehow special, that it's
somehow more raw or unfinished than tons of other things. I actually
think it's significantly *better* than a lot of other things. If we
add a disclaimer to the documentation saying "hey, this new
incremental backup feature is half-finished garbage!", and meanwhile
the documentation still says "hey, you can use cp as your
archive_command," then we have completely lost our minds.

I also think that you're being more negative about this than the facts
justify. As I said to several colleagues today, I *fully* acknowledge
that you have a lot more practical experience in this area than I do,
and a bunch of good ideas. I was really pleased to see you talking
about how it would be good if these tools worked on tar files - and I
completely agree, and I hope that will happen, and I hope to help in
making that happen. I think there are a bunch of other problems too,
only some of which I can guess at. However, I think saying that this
feature is not realistically intended to be used by end-users or that
they will not be able to do so is over the top, and is actually kind
of insulting. There has been more enthusiasm for this feature on this
mailing list and elsewhere than I've gotten for anything I've
developed in years. And I don't think that's because all of the people
who have expressed enthusiasm are silly geese who don't understand how
terrible it is.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

---

From: David Steele <[email-masked]>
Date: Fri, 12 Apr 2024 16:42:13 +1000

> happen in a single release, it'll never happen.

I'm not sure that I really buy this argument, anyway. It is not uncommon 
for significant features to spend years in development before they are 
committed. This feature went from first introduction to commit in just 
over six months. Obviously Robert had been working on it for a while, 
but for a feature this large six months is a sprint.

Regards,
-David

---

From: Tomas Vondra <[email-masked]>
Date: Fri, 12 Apr 2024 14:12:34 +0200

> most users to utilize in a meaningful way.

Perhaps, I agree we could/should try to do better job to do backups, no
argument there. But I still don't quite see why introducing such
infrastructure to "manage backups" should be up to the patch adding
incremental backups. I see it as something to build on top of
pg_basebackup/pg_combinebackup, not into those tools.

> clear in the documentation that significant tooling will be required to

Sure, I'm not against making it clearer pg_combinebackup is not a
complete backup solution, and documenting the existing restrictions.

regards

---

From: David Steele <[email-masked]>
Date: Sat, 13 Apr 2024 09:03:28 +1000

> pg_basebackup/pg_combinebackup, not into those tools.

I'm not saying that managing backups needs to be part of pg_basebackup, 
but I am saying without that it is not a complete backup solution. 
Therefore introducing advanced features that the user then has to figure 
out how to manage puts a large burden on them. Implementing 
pg_combinebackup inefficiently out of the gate just makes their life harder.

> complete backup solution, and documenting the existing restrictions.

Let's do that then. I think it would make sense to add caveats to the 
pg_combinebackup docs including space requirements, being explicit about 
the format required (e.g. plain), and also possible mitigation with COW 
filesystems.

Regards,
-David

---

From: Tomas Vondra <[email-masked]>
Date: Sat, 13 Apr 2024 12:18:08 +0200

> pg_combinebackup inefficiently out of the gate just makes their life

I agree with this in general, but I fail to see how it'd be the fault of
this patch. It merely extends what pg_basebackup did before, so if it's
not a complete solution now, it wasn't a complete solution before.

Sure, I 100% agree it'd be great to have a more efficient
pg_combinebackup with fewer restrictions. But if we make these
limitations clear, I think it's better to have this than having nothing.

> the format required (e.g. plain), and also possible mitigation with COW

OK. I'll add this as an open item, for me and Robert.

regards

---

From: Stefan Fercot <[email-masked]>
Date: Tue, 16 Apr 2024 08:47:14 +0000

Hi,

> > > > most users to utilize in a meaningful way.

pg_basebackup has its own use-cases IMHO. It is very handy to simply take a copy of the running cluster, thanks to its ability to carry on the needed WAL segs without the user even needing to think about archive_command/pg_receivewal. And for this kind of tasks, the new incremental thing will not really be interesting and won't make things more complicated.

I totally agree that we don't have any complete backup solution in core though. And adding more tools to the picture (pg_basebackup, pg_combinebackup, pg_receivewal, pg_verifybackup,...) will increase the need of on-top orchestration. But that's not new. And for people already having such orchestration, having the incremental feature will help.

> not a complete solution now, it wasn't a complete solution before.

+1. We can see it as a step to having a better backup solution in core for the future, but we definitely shouldn't rule out the fact that lots of people already developed such orchestration (home-made or relying to pgbarman, pgbackrest, wal-g,...). IMHO, if we're trying to extend the in core features, we should also aim at giving more lights and features for those tools (like adding more fields to the backup functions,...).

> OK. I'll add this as an open item, for me and Robert.

Thanks for this! It's probably not up to core docs to state all the steps that would be needed to develop a complete backup solution but documenting the links between the tools and mostly all the caveats (the "don't use INCREMENTAL.* filenames",...) will help users not be caught off guard. And as I mentioned in [1], IMO the earlier we can catch a potential issue (wrong filename, missing file,...), the better for the user.

Thank you all for working on this.
Kind Regards,
--
Stefan FERCOT
Data Egret (https://dataegret.com)

[1] https://www.postgresql.org/message-id/vJnnuiaye5rNnCPN8h3xN1Y3lyUDESIgEQnR-Urat9_ld_fozShSJbEk8JDM_3K6BVt5HXT-CatWpSfEZkYVeymlrxKO2_kfKmVZNWyCuJc%3D%40protonmail.com