Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1XbZ1M-00058A-K4 for pgsql-admin@arkaria.postgresql.org; Tue, 07 Oct 2014 17:58:00 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.80) (envelope-from ) id 1XbZ1L-0001FH-PP for pgsql-admin@arkaria.postgresql.org; Tue, 07 Oct 2014 17:57:59 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256) (Exim 4.80) (envelope-from ) id 1XbZ1H-00015l-BR for pgsql-admin@postgresql.org; Tue, 07 Oct 2014 17:57:55 +0000 Received: from resqmta-ch2-05v.sys.comcast.net ([2001:558:fe21:29:69:252:207:37]) by makus.postgresql.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.80) (envelope-from ) id 1XbZ1D-0007qn-N2 for pgsql-admin@postgresql.org; Tue, 07 Oct 2014 17:57:52 +0000 Received: from resomta-ch2-15v.sys.comcast.net ([69.252.207.111]) by resqmta-ch2-05v.sys.comcast.net with comcast id 0HxY1p0042Qkjl901HxpF9; Tue, 07 Oct 2014 17:57:49 +0000 Received: from jerry.enova.com ([38.98.177.132]) by resomta-ch2-15v.sys.comcast.net with comcast id 0Hxf1p0022rmZlA01HxhoX; Tue, 07 Oct 2014 17:57:47 +0000 Received: by jerry.enova.com (Postfix, from userid 1001) id F19CB7401F4; Tue, 7 Oct 2014 12:56:14 -0500 (CDT) From: Jerry Sievers To: John Scalia Cc: "pgsql-admin\@postgresql.org" Subject: Re: Error messages in a hot standby server's logfiles References: <5434239E.50508@gmail.com> Date: Tue, 07 Oct 2014 12:56:14 -0500 In-Reply-To: <5434239E.50508@gmail.com> (John Scalia's message of "Tue, 07 Oct 2014 13:32:14 -0400") Message-ID: <86d2a3zrup.fsf@jerry.enova.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.4 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net; s=q20140121; t=1412704669; bh=3ehG5tnhmJs3bC11m5YBPl113sEiNuHt2Z16pqn8Szo=; h=Received:Received:Received:From:To:Subject:Date:Message-ID: MIME-Version:Content-Type; b=hldeFJHnvYsSQ7Qo8d93QyDlscxXWc8E2UYHo/jE9/m1SY8lh/XU8msngn968+IFY l/dH6p/CNM5iGSy7lQCYafZImo9q1s8lKLZu+B8qtsbOYK3Uie6KVcZmrIo44r/sPq 9mI6uDGuPz6NLA/sHJM/aAdEZCiTY+bAO/DDc/mRmPTmUjTLdYbqpZ14u64Uw/dnVT I/m5IfWnrH4b+O3lkEwpDFmW3QYYmXHjOFQoeoG6CnQ55wgf6QU+LzNTtu3FLwKvP+ XS3v+t7G1EYD1nElE3vvUxgVeIwQaR/HFPQVIp+patjBb9jiPLaYcKKJmrOcqbaUzT ceczhOSu0rYUg== X-Pg-Spam-Score: -0.8 (/) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-admin Precedence: bulk Sender: pgsql-admin-owner@postgresql.org John Scalia writes: > Hi all, > > My setup is: postgresql V9.3.3 running on a CentOS 6.5 (kernel > 2.6.32-358.18.1.el6.x86_64) and I have 3 servers, one primary and two > hot standbys. In our failover and loss of communications testing, I > have seen a couple of issues that I'm hard to explain. For instance, > we took one hot standby out of service by shutting down postgresql on > it. Now, we're hot standby with log shipping as an insurance policy, > so the WAL segments continued to be copied onto that out of service > standby for a few minutes. On restart, I see: > > cp: cannot stat '/mnt/wallogs/archive/0000000C.history': No such file or directory > cp: cannot stat '/mnt/wallogs/archive/0000000B0000001900000077': No such file or directory > > Later in the logfile, I see another failure for 000000B.history. These .history files may or may not exist depending on whether timeline branching has been done. The only way Pg knows how to check for them is invoking archive_command which is your case is cp and thus the message. HTH > > In looking at the /mnt/wallogs/archive directory, those files aren't > there, but as the primary never had an issue and continued to copy WAL > segments to this directory, why was the standby looking for them? What > triggered this? Also, in that directory, I often see files generated > by the pg_basebackup command used to build the standby, files like > "0000000B0000001900000000.00000028.backup" or generally files ending > with .backup in their names. These never get removed automatically by > the standby server. We have to manually remove them. So, I'm guessing > they weren't necessary, so why did the primary copy them here using > its archive_command? Why aren't they removed by some mechanism on the > standby? > > -- > Jay -- Jerry Sievers Postgres DBA/Development Consulting e: postgres.consulting@comcast.net p: 312.241.7800 -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin