Subject: Crash: invalid DSA memory alloc request

From: Andreas 'ads' Scherbaum <[email-masked]>
Date: Thu, 12 Dec 2024 22:28:18 +0100

Hello,

I'm running a couple of large tests, and in this particular test I have 
a few million tables more.

At some point it fails, and I gathered the following trace:

2024-12-12 22:22:55.307 CET [1496210] ERROR: invalid DSA memory alloc 
request size 1073741824
2024-12-12 22:22:55.307 CET [1496210] BACKTRACE:
         postgres: ads tabletest [local] CREATE TABLE(+0x15e570) 
[0x6309c379c570]
         postgres: ads tabletest [local] CREATE 
TABLE(dshash_find_or_insert+0x1a4) [0x6309c39882d4]
         postgres: ads tabletest [local] CREATE 
TABLE(pgstat_get_entry_ref+0x440) [0x6309c3b0a530]
         postgres: ads tabletest [local] CREATE 
TABLE(pgstat_prep_pending_entry+0x3a) [0x6309c3b0676a]
         postgres: ads tabletest [local] CREATE 
TABLE(pgstat_assoc_relation+0x32) [0x6309c3b086c2]
         postgres: ads tabletest [local] CREATE 
TABLE(StartReadBuffer+0x3c0) [0x6309c3ab9870]
         postgres: ads tabletest [local] CREATE 
TABLE(ReadBufferExtended+0xa1) [0x6309c3abb271]
         postgres: ads tabletest [local] CREATE TABLE(+0x2c6caa) 
[0x6309c3904caa]
         postgres: ads tabletest [local] CREATE 
TABLE(AlterSequence+0xc0) [0x6309c3905860]
         postgres: ads tabletest [local] CREATE TABLE(+0x4b6336) 
[0x6309c3af4336]
         postgres: ads tabletest [local] CREATE 
TABLE(standard_ProcessUtility+0x259) [0x6309c3af33f9]
         postgres: ads tabletest [local] CREATE TABLE(+0x4b6e64) 
[0x6309c3af4e64]
         postgres: ads tabletest [local] CREATE 
TABLE(standard_ProcessUtility+0x259) [0x6309c3af33f9]
         postgres: ads tabletest [local] CREATE TABLE(+0x4b3d2f) 
[0x6309c3af1d2f]
         postgres: ads tabletest [local] CREATE TABLE(+0x4b3e4b) 
[0x6309c3af1e4b]
         postgres: ads tabletest [local] CREATE TABLE(PortalRun+0x16f) 
[0x6309c3af226f]
         postgres: ads tabletest [local] CREATE TABLE(+0x4b06cc) 
[0x6309c3aee6cc]
         postgres: ads tabletest [local] CREATE 
TABLE(PostgresMain+0xf67) [0x6309c3aefa87]
         postgres: ads tabletest [local] CREATE TABLE(+0x4accc5) 
[0x6309c3aeacc5]
         postgres: ads tabletest [local] CREATE 
TABLE(postmaster_child_launch+0x8f) [0x6309c3a5b95f]
         postgres: ads tabletest [local] CREATE TABLE(+0x421479) 
[0x6309c3a5f479]
         postgres: ads tabletest [local] CREATE 
TABLE(PostmasterMain+0xd71) [0x6309c3a61251]
         postgres: ads tabletest [local] CREATE TABLE(main+0x207) 
[0x6309c379efc7]
         /lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca) [0x710c33a2a1ca]
         /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b) 
[0x710c33a2a28b]
         postgres: ads tabletest [local] CREATE TABLE(_start+0x25) 
[0x6309c379f595]
2024-12-12 22:22:55.307 CET [1496210] STATEMENT:  CREATE TABLE IF NOT 
EXISTS test_16718629 (id SERIAL PRIMARY KEY, d VARCHAR(200), e 
VARCHAR(200), f VARCHAR(200), i INTEGER, j INTEGER);

PostgreSQL Version is 17.2, compiled with debug symbols.

tabletest=# select version();
version
--------------------------------------------------------------------------------------------------
  PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 
13.2.0-23ubuntu4) 13.2.0, 64-bit
(1 row)

I'm not able to reproduce this for every DDL statement, but grouping 
together about 50 of them it fails at some point.

Regards,

-- 
				Andreas 'ads' Scherbaum
German PostgreSQL User Group
European PostgreSQL User Group - Board of Directors
Volunteer Regional Contact, Germany - PostgreSQL Project

---

From: Matthias van de Meent <[email-masked]>
Date: Thu, 12 Dec 2024 22:49:35 +0100

> TABLE(pgstat_get_entry_ref+0x440) [0x6309c3b0a530]

It looks like the dshash table used in the pgstats system uses
resize(), which only specifies DSA_ALLOC_ZERO, not DSA_ALLOC_HUGE,
causing issues when the table grows larger than 1 GB.

I expect that error to disappear when you replace the
dsa_allocate0(...) call in dshash.c's resize function with
dsa_allocate_extended(..., DSA_ALLOC_HUGE | DSA_ALLOC_ZERO) as
attached, but haven't tested it due to a lack of database with
millions of relations.

Kind regards,

Matthias van de Meent

---

From: "Andreas 'ads' Scherbaum" <[email-masked]>
Date: Mon, 16 Dec 2024 08:00:00 +0100

Hi Matthias,

> attached, but haven't tested it due to a lack of database with

Can confirm that the crash no longer happens when applying your patch.

Was able to both continue the old and crashed test, as well as run a new
test:

tabletest=# select count(*) from information_schema.tables;
  count
----------
 20000211
(1 row)

Thanks,

---

From: Nathan Bossart <[email-masked]>
Date: Mon, 16 Dec 2024 16:18:26 -0600

> Can confirm that the crash no longer happens when applying your patch.

The patch looks reasonable to me.  I'll commit it soon unless someone
objects.  I was surprised to learn that the DSA_ALLOC_HUGE flag is only
intended to catch faulty allocation requests [0].

> tabletest=# select count(*) from information_schema.tables;

That's a lot of tables...

[0] https://postgr.es/m/28062.1487456862%40sss.pgh.pa.us

-- 
nathan

---

From: "Andreas 'ads' Scherbaum" <[email-masked]>
Date: Tue, 17 Dec 2024 08:00:00 +0100

Hello,

> intended to catch faulty allocation requests [0].

Is there a way to test it, except by creating so many tables?
There might be more such problems.
I did run a few basic queries in the database, but that's far from a full
test.

> That's a lot of tables...

Started as a discussion, got me curious and it's only about a magnitude or
so off
from what I've seen in production.

Not unrealistic to find out when and where it breaks.

Thanks,

---

From: Michael Paquier <[email-masked]>
Date: Tue, 17 Dec 2024 16:50:45 +0900

> intended to catch faulty allocation requests [0].

No objections.

Most likely this issue gets by a large degree easier to reach now that
we can plug into the backend custom pgstats kinds.  If pgstats or an
equivalent implementation uses pgstats, I don't think that we'll be
able to live without lifting this limit (500k query entries are
common, at 2kB each it would be enough to blow things), so using
DSA_ALLOC_HUGE sounds good to me.  I don't see a huge point in
backpatching, FWIW.

> [0] https://postgr.es/m/28062.1487456862%40sss.pgh.pa.us

And most likely don't do that.  If you want to play more in this area,
there is also the join-1M-tables-in-a-single-query game.
--
Michael