Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1WRA2q-0000NP-8E for pgsql-admin@arkaria.postgresql.org; Sat, 22 Mar 2014 00:44:16 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.80) (envelope-from ) id 1WRA2p-0000g4-Nj for pgsql-admin@arkaria.postgresql.org; Sat, 22 Mar 2014 00:44:15 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1WRA2o-0000er-7S for pgsql-admin@postgresql.org; Sat, 22 Mar 2014 00:44:14 +0000 Received: from cerberus.pinpointresearch.com ([66.7.238.130] helo=polaris.pinpointresearch.com) by magus.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1WRA2k-000075-OU for pgsql-admin@postgresql.org; Sat, 22 Mar 2014 00:44:13 +0000 Received: from [192.168.1.179] (betelgeuse.pinpointresearch.com [192.168.1.179]) by polaris.pinpointresearch.com (Postfix) with ESMTP id D8F10E00EC82; Fri, 21 Mar 2014 17:44:08 -0700 (PDT) Message-ID: <532CDCD8.30200@pinpointresearch.com> Date: Fri, 21 Mar 2014 17:44:08 -0700 From: Steve Crawford User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.3.0 MIME-Version: 1.0 To: "Chatha, Karan (CMG-Atlanta)" , "pgsql-admin@postgresql.org" Subject: Re: Replication Lag References: <378D940FF9AE0145BA853EE92442D64334660540@CMGATLPMS2003.cmg.int> <532C8F44.4030301@pinpointresearch.com> <378D940FF9AE0145BA853EE92442D64334660CF3@CMGATLPMS2003.cmg.int> In-Reply-To: <378D940FF9AE0145BA853EE92442D64334660CF3@CMGATLPMS2003.cmg.int> Content-Type: multipart/alternative; boundary="------------090307000208010808090907" X-Pg-Spam-Score: -1.9 (-) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-admin Precedence: bulk Sender: pgsql-admin-owner@postgresql.org This is a multi-part message in MIME format. --------------090307000208010808090907 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit On 03/21/2014 12:43 PM, Chatha, Karan (CMG-Atlanta) wrote: > > 1)It was working until March 8 > And then what changed? *Anything* that might have happened. Config change, unclean reboot, out of disk, firewall updates, network changes, anything at all... > > 2)We upgraded Postgres from 9.03 to 9.015 on Feb 19 > There are a few items that require special handling between 9.03 and 9.0.15. Did you read all the release notes and make sure that the extra steps were completed or didn't apply to you? (I'm not sure that any directly impact replication but haven't been running anything earlier than 9.1 for quite a while.) > > 3)Streaming Replication > > 4)Right now we have master on 9.015 and 7 slaves on 9.015 and one > slave on 9.0.16 > > 5)We have full logging enable to syslog > What do the logs tell you? Have you thoroughly examined them both for current messages and anything unusual around the time that the issue appeared? > > 6)What we see is that there are no loads or io but archives get stuck > on one archive. We have > > 7)max_standby_archive_delay = 60000 # max delay before > canceling queries > > max_standby_streaming_delay = 60000 > > It is almost like these values are not being honored. > Cheers, Steve --------------090307000208010808090907 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit
On 03/21/2014 12:43 PM, Chatha, Karan (CMG-Atlanta) wrote:

1)      It was working until March 8

And then what changed? *Anything* that might have happened. Config change, unclean reboot, out of disk, firewall updates, network changes, anything at all...

2)      We upgraded Postgres from 9.03 to 9.015 on Feb 19

There are a few items that require special handling between 9.03 and 9.0.15. Did you read all the release notes and make sure that the extra steps were completed or didn't apply to you? (I'm not sure that any directly impact replication but haven't been running anything earlier than 9.1 for quite a while.)

3)      Streaming Replication

4)      Right now we have master on 9.015 and 7 slaves on 9.015 and one slave on 9.0.16

5)      We have full logging enable to syslog

What do the logs tell you? Have you thoroughly examined them both for current messages and anything unusual around the time that the issue appeared?

6)      What we see is that there are no loads or io but archives get stuck on one archive.  We have

7)      max_standby_archive_delay = 60000               # max delay before canceling queries

max_standby_streaming_delay = 60000

 

It is almost like these values are not being honored.

Cheers,
Steve
--------------090307000208010808090907--