public inbox for [email protected]
help / color / mirror / Atom feedFrom: Chao Li <[email protected]>
To: Andres Freund <[email protected]>
Cc: [email protected]
Cc: Nathan Bossart <[email protected]>
Cc: Tom Lane <[email protected]>
Cc: Alexander Lakhin <[email protected]>
Cc: Sami Imseih <[email protected]>
Cc: Bharath Rupireddy <[email protected]>
Cc: Treat <[email protected]>
Cc: [email protected]
Cc: [email protected]
Subject: Re: pgstat vs aset
Date: Thu, 9 Apr 2026 09:51:26 +0800
Message-ID: <[email protected]> (raw)
In-Reply-To: <rtskcyc2o6rt7yebeej4byb62fnv7mcbuiodyuduqmi4ek2mnn@y5pqlz5fbsbd>
References: <rtskcyc2o6rt7yebeej4byb62fnv7mcbuiodyuduqmi4ek2mnn@y5pqlz5fbsbd>
> On Apr 9, 2026, at 07:04, Andres Freund <[email protected]> wrote:
>
> Hi,
>
> In [1] I looked at pgstat memory usage after sort of a complaint by Nathan.
> The conversion of "PgStat Shared Ref" to slab seems like an improvement we
> obviously should make [2].
>
> In the email I also showed how much CacheMemoryContext using a lot of memory
> after doing a database wide VACUUM (with ~100k tables), not very surprising.
>
>
> I didn't immediately look at the next few entries:
> ┌────────────────────────┬─────────────┬───────────────┬────────────┬─────────────┬────────────┐
> │ name │ total_bytes │ total_nblocks │ free_bytes │ free_chunks │ used_bytes │
> ├────────────────────────┼─────────────┼───────────────┼────────────┼─────────────┼────────────┤
> │ CacheMemoryContext │ 168820784 │ 31 │ 2363064 │ 33 │ 166457720 │
> │ PgStat Pending │ 27983872 │ 3419 │ 27846912 │ 112699 │ 136960 │
> │ smgr relation table │ 16777216 │ 12 │ 2797592 │ 50 │ 13979624 │
> │ MdSmgr │ 8388608 │ 11 │ 3475256 │ 1 │ 4913352 │
> │ PgStat Shared Ref │ 5767344 │ 88 │ 19712 │ 352 │ 5747632 │
> │ PgStat Shared Ref Hash │ 4195376 │ 2 │ 712 │ 0 │ 4194664 │
> │ Relcache by OID │ 4194304 │ 10 │ 57448 │ 24 │ 4136856 │
> │ TopMemoryContext │ 173264 │ 7 │ 32968 │ 79 │ 140296 │
> └────────────────────────┴─────────────┴───────────────┴────────────┴─────────────┴────────────┘
>
> When I did I was quite surprised to see "PgStat Pending" using so much, as it
> should all be freed after the stats have been submitted, a few seconds later
> at most.
>
> After a moment of worry about having introduced a leak, and adding the
> necessary columns to see more information, it's clear that all the memory was
> actually freed. It's just that we don't ever free aset blocks, even if all the
> constituent memory has been freed. So the overall context size doesn't shrink.
>
> That seems decidedly not great.
>
> I wonder how bad it would be to teach aset to recognize this situation.
>
> But I think for this use case we actually have a more fitting memory context
> type for this workload, i.e. GenerationContext. With that the size after the
> same vacuum is
>
> │ PgStat Pending │ 134144 │ 3 │ 133848 │ 0 │ 296 │
>
> Seems like a fairly obvious change.
>
>
> My local experimental changes attached.
>
>
> Greetings,
>
> Andres Freund
>
> [1] https://postgr.es/m/fcvnawwq32mamvf66q5i3sk73xudxz5corqlgqljtocepspjps%40ypvl6yjzy5xk
> [2] It's a big enough saving that I'm kinda wondering about whether we should
> try to sneak it into 19.
> <pgstat-memory.diff>
I tested this patch by creating a large number of tables and then running VACUUM.
Before the patch:
```
evantest=# select name, total_bytes, free_bytes, used_bytes, total_nblocks, free_chunks
from pg_backend_memory_contexts
where name in ('PgStat Pending', 'PgStat Shared Ref', 'PgStat Shared Ref Hash')
order by name;
name | total_bytes | free_bytes | used_bytes | total_nblocks | free_chunks
------------------------+-------------+------------+------------+---------------+-------------
PgStat Pending | 16384 | 8368 | 8016 | 5 | 8
PgStat Shared Ref | 8192 | 3672 | 4520 | 4 | 2
PgStat Shared Ref Hash | 9280 | 704 | 8576 | 2 | 0
(3 rows)
evantest=# vacuum;
VACUUM
evantest=#
evantest=# select name, total_bytes, free_bytes, used_bytes, total_nblocks, free_chunks
from pg_backend_memory_contexts
where name in ('PgStat Pending', 'PgStat Shared Ref', 'PgStat Shared Ref Hash’)
order by name;
name | total_bytes | free_bytes | used_bytes | total_nblocks | free_chunks
------------------------+-------------+------------+------------+---------------+-------------
PgStat Pending | 65536 | 64896 | 640 | 11 | 244
PgStat Shared Ref | 8142848 | 79328 | 8063520 | 997 | 1987
PgStat Shared Ref Hash | 4195392 | 704 | 4194688 | 2 | 0
(3 rows)
```
After the patch:
```
evantest=# select name, total_bytes, free_bytes, used_bytes, total_nblocks, free_chunks
from pg_backend_memory_contexts
where name in ('PgStat Pending', 'PgStat Shared Ref', 'PgStat Shared Ref Hash’)
order by name;
name | total_bytes | free_bytes | used_bytes | total_nblocks | free_chunks
------------------------+-------------+------------+------------+---------------+-------------
PgStat Pending | 8192 | 3360 | 4832 | 4 | 0
PgStat Shared Ref | 16802 | 13536 | 3266 | 1 | 188
PgStat Shared Ref Hash | 9280 | 704 | 8576 | 2 | 0
(3 rows)
evantest=#
evantest=# vacuum;
VACUUM
evantest=#
evantest=#
evantest=#
evantest=# select name, total_bytes, free_bytes, used_bytes, total_nblocks, free_chunks
from pg_backend_memory_contexts
where name in ('PgStat Pending', 'PgStat Shared Ref', 'PgStat Shared Ref Hash’)
order by name;
name | total_bytes | free_bytes | used_bytes | total_nblocks | free_chunks
------------------------+-------------+------------+------------+---------------+-------------
PgStat Pending | 37888 | 37592 | 296 | 3 | 0
PgStat Shared Ref | 7274914 | 3672 | 7271242 | 444 | 51
PgStat Shared Ref Hash | 4195392 | 704 | 4194688 | 2 | 0
(3 rows)
```
For PgStat Pending, total_bytes went from about 64kB down to about 36kB.
For PgStat Shared Ref, total_bytes went from about 8MB down to about 7MB.
As a comparison point, PgStat Shared Ref Hash showed no change before and after the patch.
So this does show improvement to me.
Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/
view thread (4+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: pgstat vs aset
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox