pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedfails to start - pg_stat_tmp ?
5+ messages / 4 participants
[nested] [flat]
* fails to start - pg_stat_tmp ?
@ 2023-11-10 10:33 lejeczek <peljasz@yahoo.co.uk>
0 siblings, 1 reply; 5+ messages in thread
From: lejeczek @ 2023-11-10 10:33 UTC (permalink / raw)
To: pgsql-admin@lists.postgresql.org
Hi guys.
I have simple cluster managed by a popular toolset, I
believe, and I get a standby/slave node which after it was
shutdown, rebooted, is unable to start, not by the that
cluster-manager nor manually.
I get:
...
2023-11-10 10:18:45.491 UTC [54858] LOG: starting
PostgreSQL 14.9 (Ubuntu 14.9-0ubuntu0.22.04.1) on
x86_64-pc-linux-gnu, compiled by gcc (Ubuntu
11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit
2023-11-10 10:18:45.491 UTC [54858] LOG: listening on IPv4
address "0.0.0.0", port 5433
2023-11-10 10:18:45.491 UTC [54858] LOG: listening on IPv6
address "::", port 5433
2023-11-10 10:18:45.495 UTC [54858] LOG: listening on Unix
socket "/var/run/postgresql/.s.PGSQL.5433"
2023-11-10 10:18:45.505 UTC [54860] LOG: database system
was interrupted while in recovery at log time 2023-11-10
09:53:18 UTC
2023-11-10 10:18:45.505 UTC [54860] HINT: If this has
occurred more than once some data might be corrupted and you
might need to choose an earlier recovery target.
2023-11-10 10:18:45.540 UTC [54862]
drwszelakiadmmdisc@drwszelaki_discourse FATAL: the database
system is starting up
2023-11-10 10:18:45.545 UTC [54863]
drwszelakiadmmdisc@drwszelaki_discourse FATAL: the database
system is starting up
2023-11-10 10:18:45.795 UTC [54860] LOG: entering standby mode
2023-11-10 10:18:45.799 UTC [54860] FATAL: could not open
directory "/var/run/postgresql/14-paf.pg_stat_tmp": No such
file or directory
2023-11-10 10:18:45.800 UTC [54858] LOG: startup process
(PID 54860) exited with exit code 1
2023-11-10 10:18:45.800 UTC [54858] LOG: aborting startup
due to startup process failure
2023-11-10 10:18:45.802 UTC [54858] LOG: database system is
shut down
Indeed that dir, that path does not exist on the node - from
what I can see, among the nodes comprising the cluster this
path exists only on the _master_ node.
What would that specific errors be suggesting or meaning -
do I have something mis-configured, perhaps the toolset
itself does not deal with "mere" reboot well, or at all.
Toolset is HA/pcs(pacemaker) cluster tools.
many thanks, L.
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: fails to start - pg_stat_tmp ?
@ 2023-11-10 12:20 Laurenz Albe <laurenz.albe@cybertec.at>
parent: lejeczek <peljasz@yahoo.co.uk>
0 siblings, 2 replies; 5+ messages in thread
From: Laurenz Albe @ 2023-11-10 12:20 UTC (permalink / raw)
To: lejeczek <peljasz@yahoo.co.uk>; pgsql-admin@lists.postgresql.org
On Fri, 2023-11-10 at 11:33 +0100, lejeczek wrote:
> I have simple cluster managed by a popular toolset, I believe, and I get a standby/slave node which
> after it was shutdown, rebooted, is unable to start, not by the that cluster-manager nor manually.
Thatks for telling us all the detail...
> FATAL: could not open directory "/var/run/postgresql/14-paf.pg_stat_tmp": No such file or directory
> LOG: startup process (PID 54860) exited with exit code 1
>
> Indeed that dir, that path does not exist on the node - from what I can see, among the nodes
> comprising the cluster this path exists only on the _master_ node.
Looks like somebody randomly deleted directories on that machine.
Create it again, and make sure it belongs to the postgres user and
nobody else has write access.
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: fails to start - pg_stat_tmp ?
@ 2023-11-10 12:45 Scott Ribe <scott_ribe@elevated-dev.com>
parent: Laurenz Albe <laurenz.albe@cybertec.at>
1 sibling, 1 reply; 5+ messages in thread
From: Scott Ribe @ 2023-11-10 12:45 UTC (permalink / raw)
To: lejeczek <peljasz@yahoo.co.uk>; +Cc: Pgsql-admin <pgsql-admin@lists.postgresql.org>
/var/run is intended for volatile data--depending on files there being permanent is dangerous
There could easily be some "cleanup" utility removing whatever its idea of "old" files is
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: fails to start - pg_stat_tmp ?
@ 2023-11-10 14:01 Ron <ronljohnsonjr@gmail.com>
parent: Scott Ribe <scott_ribe@elevated-dev.com>
0 siblings, 0 replies; 5+ messages in thread
From: Ron @ 2023-11-10 14:01 UTC (permalink / raw)
To: pgsql-admin@lists.postgresql.org
On 11/10/23 06:45, Scott Ribe wrote:
> /var/run is intended for volatile data--depending on files there being permanent is dangerous
>
> There could easily be some "cleanup" utility removing whatever its idea of "old" files is
And wonder what other critical files were "cleaned up"...
--
Born in Arizona, moved to Babylonia.
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: fails to start - pg_stat_tmp ?
@ 2023-11-10 17:33 lejeczek <peljasz@yahoo.co.uk>
parent: Laurenz Albe <laurenz.albe@cybertec.at>
1 sibling, 0 replies; 5+ messages in thread
From: lejeczek @ 2023-11-10 17:33 UTC (permalink / raw)
To: pgsql-admin@lists.postgresql.org
On 10/11/2023 13:20, Laurenz Albe wrote:
> On Fri, 2023-11-10 at 11:33 +0100, lejeczek wrote:
>> I have simple cluster managed by a popular toolset, I believe, and I get a standby/slave node which
>> after it was shutdown, rebooted, is unable to start, not by the that cluster-manager nor manually.
> Thatks for telling us all the detail...
>
>> FATAL: could not open directory "/var/run/postgresql/14-paf.pg_stat_tmp": No such file or directory
>> LOG: startup process (PID 54860) exited with exit code 1
>>
>> Indeed that dir, that path does not exist on the node - from what I can see, among the nodes
>> comprising the cluster this path exists only on the _master_ node.
> Looks like somebody randomly deleted directories on that machine.
>
> Create it again, and make sure it belongs to the postgres user and
> nobody else has write access.
>
> Yours,
> Laurenz Albe
>
>
Indeed something removes at/after reboot the path, last bit
of with stats - I started fiddling with _systemd_ tmpfiles
which I thought was the culprit but it seems systemd neither
creates nor cleans it.
I also have bucardo - cleaned it up, did not help nor after
I completely removed bucardo
pg_ctlcluster start - does work, with this wrapper path gets
created and server starts
OS journal boot logs do not mention this path neither.
It's a true sticky-wicket for me right now - to add, other
two virtually identical nodes seem free from this issue.
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2023-11-10 17:33 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2023-11-10 10:33 fails to start - pg_stat_tmp ? lejeczek <peljasz@yahoo.co.uk>
2023-11-10 12:20 ` Laurenz Albe <laurenz.albe@cybertec.at>
2023-11-10 12:45 ` Scott Ribe <scott_ribe@elevated-dev.com>
2023-11-10 14:01 ` Ron <ronljohnsonjr@gmail.com>
2023-11-10 17:33 ` lejeczek <peljasz@yahoo.co.uk>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox