Subject: shmat() fails with Not enough space on wrasse

From: Alexander Lakhin <[email-masked]>
Date: Thu, 3 Oct 2024 07:00:00 +0300

Hello Noah,

Could you please look at sporadic test failures occurring on wrasse?

Yesterday wrasse failed pg_rewindCheck ([3]) and it's the third failure of
the same kind since July. The failure log regress_log_001_basic contains:
# Running: pg_rewind --debug --source-server port=17304 host=/home/nm/farm/tmp/sbGTlHPVwD dbname='postgres' 
user=rewind_user 
--target-pgdata=/export/home/nm/farm/studio64v12_6/REL_14_STABLE/pgsql.build/src/bin/pg_rewind/tmp_check/t_001_basic_primary_remote_data/pgdata 
--no-sync --write-recovery-conf
pg_rewind: executing ".../inst/bin/postgres" for target server to complete crash recovery
2024-10-03 01:32:24.149 CEST [29411:1] FATAL:  shmat(id=1392509011, addr=0, flags=0x4000) failed: Not enough space
pg_rewind: error: postgres single-user mode in target cluster failed
pg_rewind: fatal: Command was: ".../inst/bin/postgres" --single -F -D 
"...src/bin/pg_rewind/tmp_check/t_001_basic_primary_remote_data/pgdata" template1 < "/dev/null"
not ok 11 - pg_rewind remote

I don't know SPARC Solaris, but I've found a recommendation to increase
zone.max-locked-memory [4] in a similar situation.

As wrasse tests REL_16_STABLE and I hope it will continue doing that for
another 4 years, maybe you would find it sensible to fix this issue if it's
not more complicated...

[1] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=wrasse&dt=2024-07-22%2023%3A48%3A35
[2] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=wrasse&dt=2024-09-15%2000%3A40%3A56
[3] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=wrasse&dt=2024-10-02%2023%3A12%3A02
[4] https://www.ibm.com/support/pages/db2-activate-database-fails-sql1084c-solaris

Best regards,
Alexander

---

From: Noah Misch <[email-masked]>
Date: Sat, 5 Oct 2024 10:08:00 -0700

On Thu, Oct 03, 2024 at 07:00:00AM +0300, Alexander Lakhin wrote:
> # Running: pg_rewind --debug --source-server port=17304
> host=/home/nm/farm/tmp/sbGTlHPVwD dbname='postgres' user=rewind_user --target-pgdata=/export/home/nm/farm/studio64v12_6/REL_14_STABLE/pgsql.build/src/bin/pg_rewind/tmp_check/t_001_basic_primary_remote_data/pgdata

I'm seeing no limit on that one and generous limits on other memory-related
parameters.  (I checked it from inside a cron job, just in case that
environment had been different.)

prctl $$ | ggrep -A4 -E '(memory|shm|^process:)'

process: 21611: sh -c { date; prctl $$ | ggrep -A4 -E '(memory|shm|^process:)'; } >/ex
NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT
process.max-deferred-posts
        basic              32       -   deny                             21611
        privileged        100       -   deny                                 -
--
project.max-adi-metadata-memory
        usage               0B   
        system          16.0EB    max   deny                                 -
project.max-locked-memory
        usage           1.02MB   
        system          16.0EB    max   deny                                 -
project.max-port-ids
        privileged      8.19K       -   deny                                 -
--
project.max-shm-memory
        usage           1.02MB   
        privileged      7.85GB      -   deny                                 -
        system          16.0EB    max   deny                                 -
project.max-shm-ids
        privileged        128       -   deny                                 -
        system          16.8M     max   deny                                 -
project.max-msg-ids
        privileged        128       -   deny                                 -
--
project.max-crypto-memory
        usage               0B   
        privileged      7.85GB      -   deny                                 -
        system          16.0EB    max   deny                                 -
project.max-tasks
--
zone.max-adi-metadata-memory
        usage               0B   
        system          16.0EB    max   deny                                 -
zone.max-locked-memory
        usage           1.02MB   
        system          16.0EB    max   deny                                 -
zone.max-mrp-ids
        usage               0    
--
zone.max-shm-memory
        usage           1.02MB   
        system          16.0EB    max   deny                                 -
zone.max-shm-ids
        usage               3    
        system          16.8M     max   deny                                 -
zone.max-sem-ids
        usage               8