Subject: Re: CLUSTER vs. VACUUM FULL

From: Tom Lane <[email-masked]>
Date: Sun, 21 Apr 2024 18:45:14 -0400

Ron Johnson <[email-masked]> writes:
> Why is VACUUM FULL recommended for compressing a table, when CLUSTER does
> the same thing (similarly doubling disk space), and apparently runs just as
> fast?

CLUSTER makes the additional effort to sort the data per the ordering
of the specified index.  I'm surprised that's not noticeable in your
test case.

			regards, tom lane

---

From: Ron Johnson <[email-masked]>
Date: Sun, 21 Apr 2024 20:06:52 -0400

> of the specified index.  I'm surprised that's not noticeable in your

It's in a freshly restored database.  However, regular deletions of old
records, and normal vacuums would have led me to expect that the "fresh"
public.log would have been in relatively random order.

---

From: Ron Johnson <[email-masked]>
Date: Sun, 21 Apr 2024 20:15:48 -0400

> of the specified index.  I'm surprised that's not noticeable in your

Clustering on a completely different index  was also 44 seconds.

---

From: David Rowley <[email-masked]>
Date: Mon, 22 Apr 2024 13:34:58 +1200

> Clustering on a completely different index  was also 44 seconds.

Both VACUUM FULL and CLUSTER go through a very similar code path. Both
use cluster_rel().  VACUUM FULL just won't make use of an existing
index to provide presorted input or perform a sort, whereas CLUSTER
will attempt to choose the cheapest out of these two to get sorted
results.

If the timing for each is similar, it just means that using an index
scan or sorting isn't very expensive compared to the other work that's
being done.  Both CLUSTER and VACUUM FULL require reading every heap
page and writing out new pages into a new heap and maintaining  all
indexes on the new heap. That's quite an effort.

To satisfy your curiosity, you could always run some EXPLAIN ANALYZE
SELECT queries to measure how much time was spent sorting the entire
table. You'd have to set work_mem to the value of
maintenance_work_mem.

David

---

From: Ron Johnson <[email-masked]>
Date: Sun, 21 Apr 2024 22:50:46 -0400

> indexes on the new heap. That's quite an effort.

My original CLUSTER command didn't have to change the order of the data
very much, thus, the sort didn't have to do much work.

CLUSTER on a different index was indeed much slower than VACUUM FULL.

---

From: Marcos Pegoraro <[email-masked]>
Date: Mon, 22 Apr 2024 08:42:36 -0300

> will attempt to choose the cheapest out of these two to get sorted

But wouldn't it be good that VACUUM FULL uses that index defined by
Cluster, if it exists ? Maybe an additional option for VACUUM FULL ?
Because if I periodically reorganize all tables I have to run CLUSTER once,
which will run on clustered tables, and VACUUM FULL on every table that is
not clustered, because if I run VACUUM FULL for entire database it'll just
ignore cluster index defined for every table. So, INDISCLUSTERED is used
when running CLUSTER but is ignored when running VACUUM FULL.

regards
Marcos

---

From: Tom Lane <[email-masked]>
Date: Mon, 22 Apr 2024 10:25:53 -0400

Marcos Pegoraro <[email-masked]> writes:
> Cluster, if it exists ?

No ... what would be the difference then?

			regards, tom lane

---

From: Ron Johnson <[email-masked]>
Date: Mon, 22 Apr 2024 11:37:33 -0400

> No ... what would be the difference then?

What the VACUUM docs "should" do, it seems, is suggest CLUSTER on the PK,
if the PK is a sequence (whether that be an actual sequence, or a timestamp
or something else that grows monotonically).

That's because the data is already roughly in PK order.

---

From: Adrian Klaver <[email-masked]>
Date: Mon, 22 Apr 2024 08:51:38 -0700

On 4/22/24 08:37, Ron Johnson wrote:
> On Mon, Apr 22, 2024 at 10:25 AM Tom Lane <[email-masked] 
> <mailto:[email-masked]>> wrote:
> 
>     Marcos Pegoraro <[email-masked] <mailto:[email-masked]>> writes:
> What the VACUUM docs "should" do, it seems, is suggest CLUSTER on the 
> PK, if the PK is a sequence (whether that be an actual sequence, or a 
> timestamp or something else that grows monotonically).

Why?

That would, per David Rowley's comments, impose a sort cost on top of 
the cost of hitting every heap page and rewriting it. You end up with 
sorted table granted, until such time as you start making changes to it. 
If you are to the point of running VACUUM FULL that indicates to me the 
table has seen a heavy load of changes that you want to clean out. Given 
the temporary nature of the effects of a  CLUSTER under a change load I 
don't see why it would be the way to go to clean up a changing table.

-- 
Adrian Klaver
[email-masked]

---

From: "David G. Johnston" <[email-masked]>
Date: Mon, 22 Apr 2024 09:29:21 -0700

> That's because the data is already roughly in PK order.

If things are bad enough to require a vacuum full that doesn't seem like a
good assumption.  Any insert-only table or one with a reduced fill-factor
maybe.

David J

---

From: Marcos Pegoraro <[email-masked]>
Date: Mon, 22 Apr 2024 14:50:24 -0300

Em seg., 22 de abr. de 2024 às 11:25, Tom Lane <[email-masked]> escreveu:

> No ... what would be the difference then

Well, I think if a cluster index was defined sometime in a table, it should
be respected for next commands, including VACUUM FULL.
If I want to come back to PK or any other index I would use CLUSTER ...
USING PK_INDEX.

regards
Marcos

---

From: Ron Johnson <[email-masked]>
Date: Mon, 22 Apr 2024 14:45:16 -0400

On Mon, Apr 22, 2024 at 12:29 PM David G. Johnston <
[email-masked]> wrote:

> good assumption.

Sure it does.

For example, I just deleted the oldest half of the records in 30 tables.
Tables who's CREATED_ON timestamp value strongly correlates to the
synthetic PK sequence values.

Thus, the remaining records were still mostly in PK order.  CLUSTERs on the
PK values would have taken just about as much time as the VACUUM FULL
statements which I *did* run.

---

From: Adrian Klaver <[email-masked]>
Date: Mon, 22 Apr 2024 12:14:38 -0700

On 4/22/24 11:45 AM, Ron Johnson wrote:
> On Mon, Apr 22, 2024 at 12:29 PM David G. Johnston 
> <[email-masked] <mailto:[email-masked]>> wrote:
> 
> 
> 
>     On Mon, Apr 22, 2024, 08:37 Ron Johnson <[email-masked]
>     <mailto:[email-masked]>> wrote:
> 
>         What the VACUUM docs "should" do, it seems, is suggest CLUSTER
>         on the PK, if the PK is a sequence (whether that be an actual
>         sequence, or a timestamp or something else that grows
>         monotonically).
> 
>     If things are bad enough to require a vacuum full that doesn't seem
>     like a good assumption.
> 
> 
> Sure it does.
> 
> For example, I just deleted the oldest half of the records in 30 
> tables.  Tables who's CREATED_ON timestamp value strongly correlates to 
> the synthetic PK sequence values.
> 
> Thus, the remaining records were still mostly in PK order.  CLUSTERs on 
> the PK values would have taken just about as much time as the VACUUM 
> FULL statements which I /did/ run.

1) If they are already in enough of a PK order that the CLUSTER time vs 
VACUUM FULL time would not be material as there is not much or any 
sorting to do then what does the CLUSTER gain you? Unless this table 
then became read only whatever small gain arose from the CLUSTER would 
fade away as UPDATEs and DELETEs where done.

2) What evidence is there that the records where still in PK order just 
because you deleted based on CREATED_ON? I understand the correlation 
between CREATED_ON and the PK just not sure why that would necessarily 
translate to an on disk order by PK?

---

From: Ron Johnson <[email-masked]>
Date: Mon, 22 Apr 2024 15:51:54 -0400

On Mon, Apr 22, 2024 at 3:14 PM Adrian Klaver <[email-masked]>
wrote:

> sorting to do then what does the CLUSTER gain you?

Not much.  Now they're just "slightly more ordered" instead of "slightly
less ordered" for little if any extra effort.

> translate to an on disk order by PK?

1. Records are appended to tables in INSERT order, and INSERT order is
highly correlated to synthetic PK, by the nature of sequences.
2. My original email showed that CLUSTER took just as long as VACUUM FULL.
That means not many records had to be sorted, because... the on-disk order
was strongly correlated to PK and CREATED_ON.

Will that happen *every time* in *every circumstance* in *every database*?
No, and I never said it would.  But it does in *my *database in *this *
application.

---

From: Adrian Klaver <[email-masked]>
Date: Mon, 22 Apr 2024 13:21:33 -0700

On 4/22/24 12:51, Ron Johnson wrote:
> On Mon, Apr 22, 2024 at 3:14 PM Adrian Klaver <[email-masked] 
> <mailto:[email-masked]>> wrote:
> 
> 
> 

> highly correlated to synthetic PK, by the nature of sequences.

Not something I would count on, see:

https://www.postgresql.org/docs/current/sql-createsequence.html

Notes

for how that may not always be the case.

Also any UPDATE or DELETE is going to change that. There is no guarantee 
of order for the data in the table. If there where you would not need to 
run CLUSTER.

> 2. My original email showed that CLUSTER took just as long as VACUUM 
> FULL.  That means not many records had to be sorted, because... the 
> on-disk order was strongly correlated to PK and CREATED_ON. >
> Will that happen *every time* in *every circumstance* in *every 
> database*?  No, and I never said it would.  But it does in *my *database 
> in *this *application.
> 

Which gets us back to your comment upstream:

"What the VACUUM docs "should" do, it seems, is suggest CLUSTER on the 
PK, if the PK is a sequence (whether that be an actual sequence, or a 
timestamp or something else that grows monotonically)."

This is a case specific to you and this particular circumstance, not a 
general rule for VACUUM FULL. If for no other reason then it might make 
more sense for the application that the CLUSTER be done on some other 
index then the PK.

---

From: Ron Johnson <[email-masked]>
Date: Mon, 22 Apr 2024 16:59:44 -0400

On Mon, Apr 22, 2024 at 4:21 PM Adrian Klaver <[email-masked]>
wrote:
[snip]

> more sense for the application that the CLUSTER be done on some other

On Stack Exchange, I've got a question on how to determine when to run
CLUSTER.  It ties in strongly with this thread..

---

From: Adrian Klaver <[email-masked]>
Date: Mon, 22 Apr 2024 14:03:35 -0700

On 4/22/24 13:59, Ron Johnson wrote:
> On Mon, Apr 22, 2024 at 4:21 PM Adrian Klaver <[email-masked] 

And the link is?

---

From: Ron Johnson <[email-masked]>
Date: Mon, 22 Apr 2024 17:35:57 -0400

> > CLUSTER.  It ties in strongly with this thread..

Sorry.  Got distracted by the answer.

https://dba.stackexchange.com/questions/338870/when-to-rerun-cluster

---

From: Adrian Klaver <[email-masked]>
Date: Mon, 22 Apr 2024 14:56:44 -0700

>      > CLUSTER.  It ties in strongly with this thread..

It should have been the initial question of this thread and it explains 
what you are really after.

> <https://dba.stackexchange.com/questions/338870/when-to-rerun-cluster>

---

From: Ron Johnson <[email-masked]>
Date: Mon, 22 Apr 2024 18:06:40 -0400

> what you are really after.

It was already a long email.