pg.ddx.io  pgsql-admin@postgresql.org mailing list archive  
help / color / mirror / Atom feed
Re: Problem with pgstat timeouts
4+ messages / 2 participants
[nested] [flat]

* Re: Problem with pgstat timeouts
@ 2012-01-04 17:54 Benjamin Krajmalnik <kraj@servoyant.com>
  2012-01-04 18:43 ` Re: Problem with pgstat timeouts Kevin Grittner <Kevin.Grittner@wicourts.gov>
  0 siblings, 1 reply; 4+ messages in thread

From: Benjamin Krajmalnik @ 2012-01-04 17:54 UTC (permalink / raw)
  To: Benjamin Krajmalnik <kraj@servoyant.com>; pgsql-admin

I have proceeded to do some more checking, and I see in iostat that the pg_xlog drive has a significantly higher busy state than before.  Whereas it was barely busy when we first spun up the server (total %busy since we started the server is about 6%) it is now in its 80's almost steady state.  We have a set of partitioned tables which are continuously updated, and based on the size of them they no longer fit in the shared memory which was allocated.  Pg_xlog is in a SAS RAID 1.  The server is set up with streaming replication to an identical server.  The one thing which I just checked is the RAID mode on the server.
db1# megacli64 -LDInfo -lAll -aAll
                                     

Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name                :
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0
Size                : 2.452 TB
State               : Optimal
Strip Size          : 256 KB
Number Of Drives per span:2
Span Depth          : 6
Default Cache Policy: WriteBack, ReadAhead, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteThrough, ReadAhead, Direct, No Write Cache if Bad BBU
Access Policy       : Read/Write
Disk Cache Policy   : Disk's Default
Encryption Type     : None


Virtual Drive: 1 (Target Id: 1)
Name                :
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0
Size                : 418.656 GB
State               : Optimal
Strip Size          : 256 KB
Number Of Drives    : 2
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAhead, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteThrough, ReadAhead, Direct, No Write Cache if Bad BBU
Access Policy       : Read/Write
Disk Cache Policy   : Disk's Default
Encryption Type     : None

Is it possible that the WriteThrough is what is causing the high io (and maybe the pgstat wait timeouts as well)?
If this is the case, would it be safe to change the cache to Write back?

Additionally, and somewhat unrelated, is there anything special which I need to do when restarting the primary server vis-à-vis the streaming replication server?  In other words, if I were to restart the main server, will the streaming replication server reconnect and pick up once the primary comes online?




^ permalink  raw  reply  [nested|flat] 4+ messages in thread

* Re: Problem with pgstat timeouts
  2012-01-04 17:54 Re: Problem with pgstat timeouts Benjamin Krajmalnik <kraj@servoyant.com>
@ 2012-01-04 18:43 ` Kevin Grittner <Kevin.Grittner@wicourts.gov>
  2012-01-04 18:48   ` Re: Problem with pgstat timeouts Benjamin Krajmalnik <kraj@servoyant.com>
  0 siblings, 1 reply; 4+ messages in thread

From: Kevin Grittner @ 2012-01-04 18:43 UTC (permalink / raw)
  To: pgsql-admin; Benjamin Krajmalnik <kraj@servoyant.com>

"Benjamin Krajmalnik" <kraj@servoyant.com> wrote:
 
> Default Cache Policy: WriteBack, ReadAhead, Direct, No Write Cache
>   if Bad BBU
> Current Cache Policy: WriteThrough, ReadAhead, Direct, No Write
>   Cache if Bad BBU
 
> Is it possible that the WriteThrough is what is causing the high
> io (and maybe the pgstat wait timeouts as well)?
 
Yes.
 
> If this is the case, would it be safe to change the cache to Write
> back?
 
Maybe.  How did it get into this (non-default) state?  Are batteries
installed and healthy?
 
> Additionally, and somewhat unrelated, is there anything special
> which I need to do when restarting the primary server vis-à-vis
> the streaming replication server?  In other words, if I were to
> restart the main server, will the streaming replication server
> reconnect and pick up once the primary comes online?
 
I think that should be pretty automatic as long as you haven't
promoted the standby to be the new master.
 
-Kevin



^ permalink  raw  reply  [nested|flat] 4+ messages in thread

* Re: Problem with pgstat timeouts
  2012-01-04 17:54 Re: Problem with pgstat timeouts Benjamin Krajmalnik <kraj@servoyant.com>
  2012-01-04 18:43 ` Re: Problem with pgstat timeouts Kevin Grittner <Kevin.Grittner@wicourts.gov>
@ 2012-01-04 18:48   ` Benjamin Krajmalnik <kraj@servoyant.com>
  2012-01-04 22:48     ` Re: Problem with pgstat timeouts Benjamin Krajmalnik <kraj@servoyant.com>
  0 siblings, 1 reply; 4+ messages in thread

From: Benjamin Krajmalnik @ 2012-01-04 18:48 UTC (permalink / raw)
  To: Kevin Grittner <Kevin.Grittner@wicourts.gov>; pgsql-admin

Batteries are installed and healthy.  I wonder if it did it during one of the charge/discharge cycles and just did not revert (interestingly, I have one more server that did just that).  I am trying to get the info from LSI as to what may have caused it, if it is safe to revert the cache mode while it is running, and, of course, the exact command so I don't foobar things up.


> -----Original Message-----
> From: Kevin Grittner [mailto:Kevin.Grittner@wicourts.gov]
> Sent: Wednesday, January 04, 2012 11:44 AM
> To: pgsql-admin; Benjamin Krajmalnik
> Subject: Re: [ADMIN] Problem with pgstat timeouts
> 
> "Benjamin Krajmalnik" <kraj@servoyant.com> wrote:
> 
> > Default Cache Policy: WriteBack, ReadAhead, Direct, No Write Cache
> >   if Bad BBU
> > Current Cache Policy: WriteThrough, ReadAhead, Direct, No Write
> >   Cache if Bad BBU
> 
> > Is it possible that the WriteThrough is what is causing the high
> > io (and maybe the pgstat wait timeouts as well)?
> 
> Yes.
> 
> > If this is the case, would it be safe to change the cache to Write
> > back?
> 
> Maybe.  How did it get into this (non-default) state?  Are batteries
> installed and healthy?
> 
> > Additionally, and somewhat unrelated, is there anything special
> > which I need to do when restarting the primary server vis-à-vis
> > the streaming replication server?  In other words, if I were to
> > restart the main server, will the streaming replication server
> > reconnect and pick up once the primary comes online?
> 
> I think that should be pretty automatic as long as you haven't
> promoted the standby to be the new master.
> 
> -Kevin



^ permalink  raw  reply  [nested|flat] 4+ messages in thread

* Re: Problem with pgstat timeouts
  2012-01-04 17:54 Re: Problem with pgstat timeouts Benjamin Krajmalnik <kraj@servoyant.com>
  2012-01-04 18:43 ` Re: Problem with pgstat timeouts Kevin Grittner <Kevin.Grittner@wicourts.gov>
  2012-01-04 18:48   ` Re: Problem with pgstat timeouts Benjamin Krajmalnik <kraj@servoyant.com>
@ 2012-01-04 22:48     ` Benjamin Krajmalnik <kraj@servoyant.com>
  0 siblings, 0 replies; 4+ messages in thread

From: Benjamin Krajmalnik @ 2012-01-04 22:48 UTC (permalink / raw)
  To: Benjamin Krajmalnik <kraj@servoyant.com>; Kevin Grittner <Kevin.Grittner@wicourts.gov>; pgsql-admin

Just a followup - thanks to Tom, Kevin, and the rest of the user group, as usual, for their great support.
It appears that I have hit a nice firmware bug, so I am going to have to upgrade the controller firmware.  Apparently, it has some issues communicating with the BBU so although the BBU is showing as good, trying to bring it to writeback says it is "below threshold".  Since I am going to be taking down 4 servers (our entire infrastructure) to perform the firmware upgrades, I will end up replacing the BBU's for good measure.

Once again, thanks to everyone for their assistance, and I guess this pgstat timeout can be attributed to the controller :)



> -----Original Message-----
> From: pgsql-admin-owner@postgresql.org [mailto:pgsql-admin-
> owner@postgresql.org] On Behalf Of Benjamin Krajmalnik
> Sent: Wednesday, January 04, 2012 11:48 AM
> To: Kevin Grittner; pgsql-admin
> Subject: Re: [ADMIN] Problem with pgstat timeouts
> 
> Batteries are installed and healthy.  I wonder if it did it during one
> of the charge/discharge cycles and just did not revert (interestingly,
> I have one more server that did just that).  I am trying to get the
> info from LSI as to what may have caused it, if it is safe to revert
> the cache mode while it is running, and, of course, the exact command
> so I don't foobar things up.
> 
> 
> > -----Original Message-----
> > From: Kevin Grittner [mailto:Kevin.Grittner@wicourts.gov]
> > Sent: Wednesday, January 04, 2012 11:44 AM
> > To: pgsql-admin; Benjamin Krajmalnik
> > Subject: Re: [ADMIN] Problem with pgstat timeouts
> >
> > "Benjamin Krajmalnik" <kraj@servoyant.com> wrote:
> >
> > > Default Cache Policy: WriteBack, ReadAhead, Direct, No Write Cache
> > >   if Bad BBU
> > > Current Cache Policy: WriteThrough, ReadAhead, Direct, No Write
> > >   Cache if Bad BBU
> >
> > > Is it possible that the WriteThrough is what is causing the high
> > > io (and maybe the pgstat wait timeouts as well)?
> >
> > Yes.
> >
> > > If this is the case, would it be safe to change the cache to Write
> > > back?
> >
> > Maybe.  How did it get into this (non-default) state?  Are batteries
> > installed and healthy?
> >
> > > Additionally, and somewhat unrelated, is there anything special
> > > which I need to do when restarting the primary server vis-à-vis
> > > the streaming replication server?  In other words, if I were to
> > > restart the main server, will the streaming replication server
> > > reconnect and pick up once the primary comes online?
> >
> > I think that should be pretty automatic as long as you haven't
> > promoted the standby to be the new master.
> >
> > -Kevin
> 
> --
> Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-admin



^ permalink  raw  reply  [nested|flat] 4+ messages in thread


end of thread, other threads:[~2012-01-04 22:48 UTC | newest]

Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2012-01-04 17:54 Re: Problem with pgstat timeouts Benjamin Krajmalnik <kraj@servoyant.com>
2012-01-04 18:43 ` Kevin Grittner <Kevin.Grittner@wicourts.gov>
2012-01-04 18:48   ` Benjamin Krajmalnik <kraj@servoyant.com>
2012-01-04 22:48     ` Benjamin Krajmalnik <kraj@servoyant.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox