pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedpg_ctl: server does not shut down
4+ messages / 2 participants
[nested] [flat]
* pg_ctl: server does not shut down
@ 2010-10-25 07:36 Gnanakumar <gnanam@zoniac.com>
0 siblings, 1 reply; 4+ messages in thread
From: Gnanakumar @ 2010-10-25 07:36 UTC (permalink / raw)
To: pgsql-admin
Hi,
My production server is running PostgreSQL v8.2.3 on CentOS release 5.2
(Final).
I've setup PITR in my production server. For some reason, after setting up
PITR, we're not able to manage and maintain it.
PITR SETUP DETAILS
We've 2 drives. Primary drive (pgsql/data/ directory resides) is 400 GB and
secondary drive (WAL archive) is 300 GB. All WAL archives are written to
secondary drive.
Base backup taken on: Aug03, 2009
WAL archive drive become full (100% use) on: Jul01, 2010
Because this WAL archive drive has become full, all WAL archive segments to
be archived are accumulated into pg_xlog/ directory itself. Eventually,
almost 4 months (as of today from Jul01, 2010) of WAL archives are residing
in pg_xlog/ directory.
ISSUE
We then decided to stop PITR in our database. Steps which I followed are:
1. service postgresql stop
2. Detached (unmounted) 300 GB secondary drive (WAL archive)
3. Disabled WAL archiving by commenting (#) 2 postgresql.conf parameters:
"archive_command" and "archive_timeout".
4. service postgresql start
Now, the database has started without any issues. But when I tried to stop
database using 'service postgresql stop' within few minutes after it's been
started, it was not shutting down and responded with the following message
after few minutes:
"Stopping PostgreSQL: pg_ctl: server does not shut down"
I even checked serverlog for whether any errors/issues are reported, but I
don't find any. Also, when I tried to login to my database at this time, it
says:
psql mydb mydbuser
psql: FATAL: the database system is shutting down
but not actually shutting down ever. I'm clueless why it is happening so
and not shutting down ever.
My questions are:
1. What could be the root cause of PostgreSQL server not shutting
down ever?
2. Am I following a different/incorrect way of disabling PITR?
3. Considering this situation, now how do I safely disable PITR in
my database?
Experts advice/idea/suggestion on this are appreciated.
Regards,
Gnanam
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: pg_ctl: server does not shut down
@ 2010-10-25 15:22 Greg Smith <greg@2ndquadrant.com>
parent: Gnanakumar <gnanam@zoniac.com>
0 siblings, 1 reply; 4+ messages in thread
From: Greg Smith @ 2010-10-25 15:22 UTC (permalink / raw)
To: gnanam@zoniac.com; +Cc: pgsql-admin
Gnanakumar wrote:
> 1. What could be the root cause of PostgreSQL server not shutting
> down ever?
> 2. Am I following a different/incorrect way of disabling PITR?
>
You can check if PITR is disabled by executing:
SHOW archive_command;
At a psql prompt. If that's empty, then you turned it off, and it isn't
involved in the shutdown issue you have anymore.
Normally when the server doesn't want to shut down, that's because of a
client that won't exit normally, not something in the main server. Take
a look at what processes are still running against the database and see
if there are clients attached after the fast shutdown attempt. If so,
those are your problem, not something to PITR.
If you know every client has been stopped or is doing nothing useful,
you can do an immediate shutdown of the server and kick everyone off.
You'll get messages about unclean shutdown, but no data should be lost.
The "service" scripts don't do that though. You'll need to do something
like this instead, presuming you start as root:
su postgres
pg_ctl -D /var/lib/pgsql/data -m immedate stop
--
Greg Smith, 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD
PostgreSQL Training, Services and Support www.2ndQuadrant.us
Author, "PostgreSQL 9.0 High Performance" Pre-ordering at:
https://www.packtpub.com/postgresql-9-0-high-performance/book
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: pg_ctl: server does not shut down
@ 2010-10-26 05:32 Gnanakumar <gnanam@zoniac.com>
parent: Greg Smith <greg@2ndquadrant.com>
0 siblings, 1 reply; 4+ messages in thread
From: Gnanakumar @ 2010-10-26 05:32 UTC (permalink / raw)
To: 'Greg Smith' <greg@2ndquadrant.com>; +Cc: pgsql-admin
> You can check if PITR is disabled by executing:
> SHOW archive_command;
> At a psql prompt. If that's empty, then you turned it off, and it isn't
involved in the shutdown issue you have anymore.
Yes, I could see empty value here.
> Take a look at what processes are still running against the database and
see if there are clients attached after the fast shutdown attempt.
> If so, those are your problem, not something to PITR.
Yes, I can see the following processes still running. I used the command 'ps
-ef |grep postgres' to list.
postgres 3376 1 0 07:20 ? 00:00:00
/mnt/database/pgsql/bin/postgres -D /usr/local/pgsql/data
postgres 3381 3376 0 07:20 ? 00:00:00 postgres: logger process
postgres 3383 3376 0 07:20 ? 00:00:00 postgres: writer process
But why logger and writer process are still running/showing up here for a
very long time after issuing 'service postgresql stop'. What could be the
problem? Also, am sure here that no other clients (database connections)
are connected at this time.
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: pg_ctl: server does not shut down
@ 2010-10-28 07:57 Gnanakumar <gnanam@zoniac.com>
parent: Gnanakumar <gnanam@zoniac.com>
0 siblings, 0 replies; 4+ messages in thread
From: Gnanakumar @ 2010-10-28 07:57 UTC (permalink / raw)
To: 'Greg Smith' <greg@2ndquadrant.com>; +Cc: pgsql-admin
>> You can check if PITR is disabled by executing:
>> SHOW archive_command;
>> At a psql prompt. If that's empty, then you turned it off, and it isn't
>> involved in the shutdown issue you have anymore.
> Yes, I could see empty value here.
>> Take a look at what processes are still running against the database and
>> see if there are clients attached after the fast shutdown attempt.
>> If so, those are your problem, not something to PITR.
> Yes, I can see the following processes still running. I used the command
'ps
> -ef |grep postgres' to list.
> postgres 3376 1 0 07:20 ? 00:00:00
> /mnt/database/pgsql/bin/postgres -D /usr/local/pgsql/data
> postgres 3381 3376 0 07:20 ? 00:00:00 postgres: logger process
> postgres 3383 3376 0 07:20 ? 00:00:00 postgres: writer process
> But why logger and writer process are still running/showing up here for a
> very long time after issuing 'service postgresql stop'. What could be the
> problem? Also, am sure here that no other clients (database connections)
> are connected at this time.
I can CONSISTENTLY reproduce this problem (PITR walarchive drive is not
attached/mounted) and I still don't know the reason behind this issue:
1) If I disable PITR and start postgresql service, consistently server does
not shut down on 'service postgresql stop' command. I have tried and
checked this upto 4 times.
NOTE: After disabling PITR, on checking "SHOW archive_command" from psql,
it is showing empty value. But on 'ps -ef |grep postgres', it is always
showing up
logger and writer process running.
2) But if I enable back PITR and start postgresql service, consistently
server shuts down properly on 'service postgresql stop' command. Again I
have tried this cycle also upto 4 times.
When I mean enable/disable PITR, it is about enabling/disabling 2
configuration parameters: "archive_command" and "archive_timeout".
Since this is our Production server, experts assistance to solve this issue
are highly appreciated.
^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2010-10-28 07:57 UTC | newest]
Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2010-10-25 07:36 pg_ctl: server does not shut down Gnanakumar <gnanam@zoniac.com>
2010-10-25 15:22 ` Greg Smith <greg@2ndquadrant.com>
2010-10-26 05:32 ` Gnanakumar <gnanam@zoniac.com>
2010-10-28 07:57 ` Gnanakumar <gnanam@zoniac.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox