pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedFATAL: the database system is starting up
11+ messages / 6 participants
[nested] [flat]
* FATAL: the database system is starting up
@ 2010-09-22 16:37 Joe Carr <joe.carr@gmail.com>
2010-09-23 12:48 ` Re: FATAL: the database system is starting up Fujii Masao <masao.fujii@gmail.com>
2010-09-23 17:01 ` Re: FATAL: the database system is starting up Kevin Grittner <Kevin.Grittner@wicourts.gov>
0 siblings, 2 replies; 11+ messages in thread
From: Joe Carr @ 2010-09-22 16:37 UTC (permalink / raw)
To: pgsql-admin
Hello all,
OS = Windows Server 2003
PostGres = 9.0
I'm trying to bring up a second machine as a hot standby. When I attempted
to start the service on the second machine I receive the following :
2010-09-22 09:33:56 PDT LOG: database system was interrupted; last known up
at 2010-09-22 09:07:51 PDT
2010-09-22 09:33:56 PDT LOG: creating missing WAL directory
"pg_xlog/archive_status"
2010-09-22 09:33:56 PDT LOG: entering standby mode
2010-09-22 09:33:57 PDT FATAL: the database system is starting up
2010-09-22 09:33:57 PDT LOG: restored log file "00000001000000000000002E"
from archive
2010-09-22 09:33:57 PDT LOG: redo starts at 0/2E000020
2010-09-22 09:33:57 PDT LOG: consistent recovery state reached at
0/2F000000
2010-09-22 09:33:57 PDT LOG: streaming replication successfully connected
to primary
2010-09-22 09:33:58 PDT FATAL: the database system is starting up
2010-09-22 09:33:59 PDT FATAL: the database system is starting up
and the "FATAL: the database system is starting up" message will continue.
Do you know any steps to debug this message? I have a feeling it is a file
permissions error, but I cannot determine the cause.
Thanks beforehand for any assistance.
-Joe
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: FATAL: the database system is starting up
2010-09-22 16:37 FATAL: the database system is starting up Joe Carr <joe.carr@gmail.com>
@ 2010-09-23 12:48 ` Fujii Masao <masao.fujii@gmail.com>
2010-09-28 18:19 ` Re: FATAL: the database system is starting up Joe Carr <joe.carr@gmail.com>
1 sibling, 1 reply; 11+ messages in thread
From: Fujii Masao @ 2010-09-23 12:48 UTC (permalink / raw)
To: Joe Carr <joe.carr@gmail.com>; +Cc: pgsql-admin
On Thu, Sep 23, 2010 at 1:37 AM, Joe Carr <joe.carr@gmail.com> wrote:
> Hello all,
> OS = Windows Server 2003
> PostGres = 9.0
>
> I'm trying to bring up a second machine as a hot standby. When I attempted
> to start the service on the second machine I receive the following :
>
> 2010-09-22 09:33:56 PDT LOG: database system was interrupted; last known up
> at 2010-09-22 09:07:51 PDT
> 2010-09-22 09:33:56 PDT LOG: creating missing WAL directory
> "pg_xlog/archive_status"
> 2010-09-22 09:33:56 PDT LOG: entering standby mode
> 2010-09-22 09:33:57 PDT FATAL: the database system is starting up
> 2010-09-22 09:33:57 PDT LOG: restored log file "00000001000000000000002E"
> from archive
> 2010-09-22 09:33:57 PDT LOG: redo starts at 0/2E000020
> 2010-09-22 09:33:57 PDT LOG: consistent recovery state reached at
> 0/2F000000
> 2010-09-22 09:33:57 PDT LOG: streaming replication successfully connected
> to primary
> 2010-09-22 09:33:58 PDT FATAL: the database system is starting up
> 2010-09-22 09:33:59 PDT FATAL: the database system is starting up
>
> and the "FATAL: the database system is starting up" message will continue.
>
> Do you know any steps to debug this message? I have a feeling it is a file
> permissions error, but I cannot determine the cause.
Did you set hot_standby to true in postgresql.conf of the standby server?
Regards,
--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: FATAL: the database system is starting up
2010-09-22 16:37 FATAL: the database system is starting up Joe Carr <joe.carr@gmail.com>
2010-09-23 12:48 ` Re: FATAL: the database system is starting up Fujii Masao <masao.fujii@gmail.com>
@ 2010-09-28 18:19 ` Joe Carr <joe.carr@gmail.com>
0 siblings, 0 replies; 11+ messages in thread
From: Joe Carr @ 2010-09-28 18:19 UTC (permalink / raw)
To: Fujii Masao <masao.fujii@gmail.com>; +Cc: pgsql-admin
Hello Fujii,
Sorry for the delay in my response, and thank you very much for the help.
I did not have the server configured to be a hot_standby in the
postgresql.conf, and once I did replication is working perfectly for me.
-Joe
On Thu, Sep 23, 2010 at 8:48 AM, Fujii Masao <masao.fujii@gmail.com> wrote:
> On Thu, Sep 23, 2010 at 1:37 AM, Joe Carr <joe.carr@gmail.com> wrote:
> > Hello all,
> > OS = Windows Server 2003
> > PostGres = 9.0
> >
> > I'm trying to bring up a second machine as a hot standby. When I
> attempted
> > to start the service on the second machine I receive the following :
> >
> > 2010-09-22 09:33:56 PDT LOG: database system was interrupted; last known
> up
> > at 2010-09-22 09:07:51 PDT
> > 2010-09-22 09:33:56 PDT LOG: creating missing WAL directory
> > "pg_xlog/archive_status"
> > 2010-09-22 09:33:56 PDT LOG: entering standby mode
> > 2010-09-22 09:33:57 PDT FATAL: the database system is starting up
> > 2010-09-22 09:33:57 PDT LOG: restored log file
> "00000001000000000000002E"
> > from archive
> > 2010-09-22 09:33:57 PDT LOG: redo starts at 0/2E000020
> > 2010-09-22 09:33:57 PDT LOG: consistent recovery state reached at
> > 0/2F000000
> > 2010-09-22 09:33:57 PDT LOG: streaming replication successfully
> connected
> > to primary
> > 2010-09-22 09:33:58 PDT FATAL: the database system is starting up
> > 2010-09-22 09:33:59 PDT FATAL: the database system is starting up
> >
> > and the "FATAL: the database system is starting up" message will
> continue.
> >
> > Do you know any steps to debug this message? I have a feeling it is a
> file
> > permissions error, but I cannot determine the cause.
>
> Did you set hot_standby to true in postgresql.conf of the standby server?
>
> Regards,
>
> --
> Fujii Masao
> NIPPON TELEGRAPH AND TELEPHONE CORPORATION
> NTT Open Source Software Center
>
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: FATAL: the database system is starting up
2010-09-22 16:37 FATAL: the database system is starting up Joe Carr <joe.carr@gmail.com>
@ 2010-09-23 17:01 ` Kevin Grittner <Kevin.Grittner@wicourts.gov>
1 sibling, 0 replies; 11+ messages in thread
From: Kevin Grittner @ 2010-09-23 17:01 UTC (permalink / raw)
To: Joe Carr <joe.carr@gmail.com>; pgsql-admin
Joe Carr <joe.carr@gmail.com> wrote:
> and the "FATAL: the database system is starting up" message will
> continue.
I believe that you get that when there is an attempt to connect to a
database which is in recovery mode. I think the pg_ctl program
makes attempts for about one minute before it gives up. If you have
some other monitoring software which is attempting connection, or
some software for managing failover, perhaps that's the cause?
-Kevin
^ permalink raw reply [nested|flat] 11+ messages in thread
* "FATAL: the database system is starting up" and hot_standby = on
@ 2017-12-07 05:49 Ilnur Khanov <khanov.i@gmail.com>
2017-12-07 08:11 ` Re: "FATAL: the database system is starting up" and hot_standby = on David Guyot <david.guyot@europecamions-interactive.com>
0 siblings, 1 reply; 11+ messages in thread
From: Ilnur Khanov @ 2017-12-07 05:49 UTC (permalink / raw)
To: pgsql-admin@lists.postgresql.org
Hi,
There is a standalone server with Postgresql 9.6.3 on Windows 2016 Server.
I had been faced with "FATAL: the database system is starting up" after
disk failure. This mailing list helped me to solve issue by setting
"hot_standby = on" in postgresql.conf.
But I can't fully understand why it had worked.
Comments in conf says, that the setting has to be ignored in my case:
===
# - Standby Servers -
# These settings are ignored on a master server.
hot_standby = on # "on" allows queries during recovery
# (change requires restart)
===
But it is standalone installation, not slave, there is no replication.
Though there is no recovery.conf and any sign of pending recovery nor in
log, nor in postgres.exe output with debug level 5.
Why setting "hot_standby = on" had helped me with "FATAL: the database
system is starting up" error on standalone server? Any ideas?
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: "FATAL: the database system is starting up" and hot_standby = on
2017-12-07 05:49 "FATAL: the database system is starting up" and hot_standby = on Ilnur Khanov <khanov.i@gmail.com>
@ 2017-12-07 08:11 ` David Guyot <david.guyot@europecamions-interactive.com>
2017-12-07 08:35 ` Re: "FATAL: the database system is starting up" and hot_standby = on Ilnur Khanov <khanov.i@gmail.com>
0 siblings, 1 reply; 11+ messages in thread
From: David Guyot @ 2017-12-07 08:11 UTC (permalink / raw)
To: pgsql-admin@lists.postgresql.org
Hello.
I can't say why this tip worked, but I can say you that you shouldn't
bother about the "FATAL: the database system is starting up" message:
PgSQL routinely displays this message when starting up. "FATAL" is
called "gravity", that is to say it describes the importance of the
following message. "FATAL" is the highest importance and, indeed,
knowing if your database is up, or not, is of the highest importance,
but it is not a fatal error.
Regards.
Le jeudi 07 décembre 2017 à 10:49 +0500, Ilnur Khanov a écrit :
> Hi,
>
> There is a standalone server with Postgresql 9.6.3 on Windows 2016
> Server.
>
> I had been faced with "FATAL: the database system is starting up"
> after disk failure. This mailing list helped me to solve issue by
> setting "hot_standby = on" in postgresql.conf.
> But I can't fully understand why it had worked.
>
> Comments in conf says, that the setting has to be ignored in my case:
> ===
> # - Standby Servers -
> # These settings are ignored on a master server.
> hot_standby = on # "on" allows queries during
> recovery
> # (change requires restart)
> ===
>
> But it is standalone installation, not slave, there is no
> replication. Though there is no recovery.conf and any sign of pending
> recovery nor in log, nor in postgres.exe output with debug level 5.
>
> Why setting "hot_standby = on" had helped me with "FATAL: the
> database system is starting up" error on standalone server? Any
> ideas?
>
--
David Guyot
Administrateur système / Sysadmin
Europe Camions Interactive / Stockway
Moulin Collot F-88500 Ambacourt
Tél : +33 (0)3 29 30 47 85
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: "FATAL: the database system is starting up" and hot_standby = on
2017-12-07 05:49 "FATAL: the database system is starting up" and hot_standby = on Ilnur Khanov <khanov.i@gmail.com>
2017-12-07 08:11 ` Re: "FATAL: the database system is starting up" and hot_standby = on David Guyot <david.guyot@europecamions-interactive.com>
@ 2017-12-07 08:35 ` Ilnur Khanov <khanov.i@gmail.com>
2017-12-07 17:37 ` Re: "FATAL: the database system is starting up" and hot_standby = on Laurenz Albe <laurenz.albe@cybertec.at>
0 siblings, 1 reply; 11+ messages in thread
From: Ilnur Khanov @ 2017-12-07 08:35 UTC (permalink / raw)
To: pgsql-admin@lists.postgresql.org
For me, when I saw "FATAL: the database system is starting up", nothing
else hasn't been working: pgAdmin, psql, business apps.
Logs has been showing cycles: client connect - worker starts -
"the-FATAL-error" - worker shutting down with exit code (1). No any
informative errors.
I think, database engine is in some sort of recovery mode after crush, but
I was waiting long enough - no CPU or HDD activity. If it's some sort of
warning and waiting for me to check consistency and manually reset this
mode flag, but then how to do this?!
More important: now problem is not solved, "hot_standby = on" is just
workaround letting query databases. But PostgreSQL is still in some strange
mode and I don't understand how serious is it and how resolve it correctly.
By "PostgreSQL is still in some strange mode" I mean that setting "hot_standby
= off" returns "the-FATAL-error" and nothing works then.
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: "FATAL: the database system is starting up" and hot_standby = on
2017-12-07 05:49 "FATAL: the database system is starting up" and hot_standby = on Ilnur Khanov <khanov.i@gmail.com>
2017-12-07 08:11 ` Re: "FATAL: the database system is starting up" and hot_standby = on David Guyot <david.guyot@europecamions-interactive.com>
2017-12-07 08:35 ` Re: "FATAL: the database system is starting up" and hot_standby = on Ilnur Khanov <khanov.i@gmail.com>
@ 2017-12-07 17:37 ` Laurenz Albe <laurenz.albe@cybertec.at>
2017-12-08 04:31 ` Re: "FATAL: the database system is starting up" and hot_standby = on Ilnur Khanov <khanov.i@gmail.com>
0 siblings, 1 reply; 11+ messages in thread
From: Laurenz Albe @ 2017-12-07 17:37 UTC (permalink / raw)
To: Ilnur Khanov <khanov.i@gmail.com>; pgsql-admin@lists.postgresql.org
Ilnur Khanov wrote:
> For me, when I saw "FATAL: the database system is starting up", nothing else hasn't been working: pgAdmin, psql, business apps.
"FATAL" just means that a client connection is terminated.
In this case, because the database is not up.
> Logs has been showing cycles: client connect - worker starts - "the-FATAL-error" -
> worker shutting down with exit code (1). No any informative errors.
>
> I think, database engine is in some sort of recovery mode after crush, but I was
> waiting long enough - no CPU or HDD activity. If it's some sort of warning and waiting for me to check consistency and manually reset this mode flag, but then how to do this?!
>
> More important: now problem is not solved, "hot_standby = on" is just workaround letting query databases.
> But PostgreSQL is still in some strange mode and I don't understand how serious is it and how resolve it correctly.
You cannot modify the database, right?
And "SELECT pg_is_in_recovery();" returns TRUE, right?
Could you show us what is written to the log after a new startup?
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: "FATAL: the database system is starting up" and hot_standby = on
2017-12-07 05:49 "FATAL: the database system is starting up" and hot_standby = on Ilnur Khanov <khanov.i@gmail.com>
2017-12-07 08:11 ` Re: "FATAL: the database system is starting up" and hot_standby = on David Guyot <david.guyot@europecamions-interactive.com>
2017-12-07 08:35 ` Re: "FATAL: the database system is starting up" and hot_standby = on Ilnur Khanov <khanov.i@gmail.com>
2017-12-07 17:37 ` Re: "FATAL: the database system is starting up" and hot_standby = on Laurenz Albe <laurenz.albe@cybertec.at>
@ 2017-12-08 04:31 ` Ilnur Khanov <khanov.i@gmail.com>
2017-12-08 05:49 ` Re: "FATAL: the database system is starting up" and hot_standby = on Laurenz Albe <laurenz.albe@cybertec.at>
0 siblings, 1 reply; 11+ messages in thread
From: Ilnur Khanov @ 2017-12-08 04:31 UTC (permalink / raw)
To: pgsql-admin@lists.postgresql.org
Hi,
Thank you all for suggestions.
I've just disabled hot_standby and restarted server. Everything is OK now.
I assume there was continuous recovery mode.
You cannot modify the database, right?
>
No, I can. As long it was in this recovery mode with hot_standby=on I was
able query and modify database, with hot_standby=off I couldn't even
connect.
Here is another my wonder: relying on manual, it had to be read-only mode.
And "SELECT pg_is_in_recovery();" returns TRUE, right?
I don't know :( Can't check it anymore. May be.
That is the question: If I can't connect with "FATAL: the database system
is starting up" how I can check pg_is_in_recovery()?
What else I could check to ensure that postgres recovering databases and I
just have to wait?
I was strongly confused by NO CPU and HDD activity during my wait.
So my conclusion is:
If postgres says "FATAL: the database system is starting up" after
failure without any other errors in logs and postgres output, than you can:
- wait for some long period of time
or
- you can turn on hot_standby and use your database with care while
it's recovering and turn hot_standby off when it finishes.
Hoping this thread helps someone sometime.
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: "FATAL: the database system is starting up" and hot_standby = on
2017-12-07 05:49 "FATAL: the database system is starting up" and hot_standby = on Ilnur Khanov <khanov.i@gmail.com>
2017-12-07 08:11 ` Re: "FATAL: the database system is starting up" and hot_standby = on David Guyot <david.guyot@europecamions-interactive.com>
2017-12-07 08:35 ` Re: "FATAL: the database system is starting up" and hot_standby = on Ilnur Khanov <khanov.i@gmail.com>
2017-12-07 17:37 ` Re: "FATAL: the database system is starting up" and hot_standby = on Laurenz Albe <laurenz.albe@cybertec.at>
2017-12-08 04:31 ` Re: "FATAL: the database system is starting up" and hot_standby = on Ilnur Khanov <khanov.i@gmail.com>
@ 2017-12-08 05:49 ` Laurenz Albe <laurenz.albe@cybertec.at>
2017-12-08 06:25 ` Re: "FATAL: the database system is starting up" and hot_standby = on Ilnur Khanov <khanov.i@gmail.com>
0 siblings, 1 reply; 11+ messages in thread
From: Laurenz Albe @ 2017-12-08 05:49 UTC (permalink / raw)
To: Ilnur Khanov <khanov.i@gmail.com>; pgsql-admin@lists.postgresql.org
Ilnur Khanov wrote:
> I've just disabled hot_standby and restarted server. Everything is OK now.
>
> I assume there was continuous recovery mode.
>
> > You cannot modify the database, right?
>
> No, I can. As long it was in this recovery mode with hot_standby=on I was able query and modify database, with hot_standby=off I couldn't even connect.
> Here is another my wonder: relying on manual, it had to be read-only mode.
Now that would be really strange if you could modify the database while in recovery mode.
Unfortunately you cannot test that now...
> That is the question: If I can't connect with "FATAL: the database system is starting up" how I can check pg_is_in_recovery()?
If you get the fatal message you *know* the system is recovering.
That query is to ascertain you are still in recovery if you set hot_standby=on.
> So my conclusion is:
> If postgres says "FATAL: the database system is starting up" after failure without any other errors in logs and postgres output, than you can:
> - wait for some long period of time
> or
> - you can turn on hot_standby and use your database with care while it's recovering and turn hot_standby off when it finishes.
>
> Hoping this thread helps someone sometime.
In my opinion the main question is why crash recovery took so long.
The server log should be able to tell you more about it.
Is your checkpoint_timeout very high?
Was there a backup_label file in the data directory (now backup_label.old)?
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: "FATAL: the database system is starting up" and hot_standby = on
2017-12-07 05:49 "FATAL: the database system is starting up" and hot_standby = on Ilnur Khanov <khanov.i@gmail.com>
2017-12-07 08:11 ` Re: "FATAL: the database system is starting up" and hot_standby = on David Guyot <david.guyot@europecamions-interactive.com>
2017-12-07 08:35 ` Re: "FATAL: the database system is starting up" and hot_standby = on Ilnur Khanov <khanov.i@gmail.com>
2017-12-07 17:37 ` Re: "FATAL: the database system is starting up" and hot_standby = on Laurenz Albe <laurenz.albe@cybertec.at>
2017-12-08 04:31 ` Re: "FATAL: the database system is starting up" and hot_standby = on Ilnur Khanov <khanov.i@gmail.com>
2017-12-08 05:49 ` Re: "FATAL: the database system is starting up" and hot_standby = on Laurenz Albe <laurenz.albe@cybertec.at>
@ 2017-12-08 06:25 ` Ilnur Khanov <khanov.i@gmail.com>
0 siblings, 0 replies; 11+ messages in thread
From: Ilnur Khanov @ 2017-12-08 06:25 UTC (permalink / raw)
To: pgsql-admin@lists.postgresql.org
>
> Now that would be really strange if you could modify the database while in
> recovery mode.
> Unfortunately you cannot test that now...
I tried that right after enabling hot_standby, so databases was writable.
In my opinion the main question is why crash recovery took so long.
> The server log should be able to tell you more about it.
server logs with debug level 5 didn't tell anything usefull.
there is no something in logs about recovery mode start, nor about recovery
mode end.
so it's hard to say how long crash recovery took.
No more 24h, no less 2h. Databases are very small - about 60Gb. Resources
are high available (SSD, Xeon E7), but not used.
Is your checkpoint_timeout very high?
>
It's default (5 min).
> Was there a backup_label file in the data directory (now backup_label.old)?
No, there is no.
^ permalink raw reply [nested|flat] 11+ messages in thread
end of thread, other threads:[~2017-12-08 06:25 UTC | newest]
Thread overview: 11+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2010-09-22 16:37 FATAL: the database system is starting up Joe Carr <joe.carr@gmail.com>
2010-09-23 12:48 ` Fujii Masao <masao.fujii@gmail.com>
2010-09-28 18:19 ` Joe Carr <joe.carr@gmail.com>
2010-09-23 17:01 ` Kevin Grittner <Kevin.Grittner@wicourts.gov>
2017-12-07 05:49 "FATAL: the database system is starting up" and hot_standby = on Ilnur Khanov <khanov.i@gmail.com>
2017-12-07 08:11 ` Re: "FATAL: the database system is starting up" and hot_standby = on David Guyot <david.guyot@europecamions-interactive.com>
2017-12-07 08:35 ` Re: "FATAL: the database system is starting up" and hot_standby = on Ilnur Khanov <khanov.i@gmail.com>
2017-12-07 17:37 ` Re: "FATAL: the database system is starting up" and hot_standby = on Laurenz Albe <laurenz.albe@cybertec.at>
2017-12-08 04:31 ` Re: "FATAL: the database system is starting up" and hot_standby = on Ilnur Khanov <khanov.i@gmail.com>
2017-12-08 05:49 ` Re: "FATAL: the database system is starting up" and hot_standby = on Laurenz Albe <laurenz.albe@cybertec.at>
2017-12-08 06:25 ` Re: "FATAL: the database system is starting up" and hot_standby = on Ilnur Khanov <khanov.i@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox