Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cE0E9-0007GO-KO for pgsql-admin@arkaria.postgresql.org; Mon, 05 Dec 2016 20:51:09 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.84_2) (envelope-from ) id 1cE0E9-0005AJ-6t for pgsql-admin@arkaria.postgresql.org; Mon, 05 Dec 2016 20:51:09 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1cE0E8-00055I-22 for pgsql-admin@postgresql.org; Mon, 05 Dec 2016 20:51:08 +0000 Received: from mail-yw0-x233.google.com ([2607:f8b0:4002:c05::233]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.84_2) (envelope-from ) id 1cE0E3-0006SL-Ue for pgsql-admin@postgresql.org; Mon, 05 Dec 2016 20:51:06 +0000 Received: by mail-yw0-x233.google.com with SMTP id a10so260317358ywa.3 for ; Mon, 05 Dec 2016 12:51:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=H60F55En9XyfxYOLC9JqVl1+pRQKcE8WJVjDMw+8sdI=; b=kzrUh+d2t+MEDw/J4FmVnz3On24ZljCb7kGnPAcPirSr1+rv78oG7OoO/FP/+xGGyq VTknTNBSLTz4wv9Pan9hEBq4uwo7Q9hIzlVPxNmcepG+7L9XvEW+aIbnZkmmMQCiadjx ux+P+TM9b7i6HNV2AE+Dd8zfb+zvUq+Dhi3ZfH4wbfE49Su3b9uabkbTgX/EeQ/FMQBU JatOV620oj2pku71CseN0My3NxxtXSn5UHeGejdqhcE/0jvWkF21MmYaxOZVES8it04t OXr0jMHbM2mG0Wvy349i7f93gdd90y7S2rLaAkx/Z66bns+LpHtu1RHHyiYQ8gd4EWXQ g1bA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=H60F55En9XyfxYOLC9JqVl1+pRQKcE8WJVjDMw+8sdI=; b=Ixa0/aR9h3XFNe/ZhSUeGm3Eaow9EXwpKrAz2f4ZsfwoTf1Io0kQ0W7oe/g0/4Jyvi 0slascAwQUk/3NHu/Q3px4MScK76GQKO5wFe0xWGuYvZr/GIBwGB9+JHqG/Yj69+kYPj DoaMC333A315SUoxTqx4ZMo5K+OelkJ4BQezLxObkxPvWAbjmDQSMK/VfTrVbpuoYucB 0RviWVySdaayVkunJKGdCaePdcPchZjGbW41lxdC0K41ennHtx/rrR0IjFQuxUAFt5O1 HB05dKBX1Nm2/m17aOLv9HVNiZIFNjv8AxDONkdzb70+1P659BTQ2vYhLFIIqJk5UJ9r 9cAg== X-Gm-Message-State: AKaTC00aCd8UwwbTCpUQWKs3J9B8SxDo3ZR1QdsayQog1qF+NkKgJDK6VJj97WyU70Bjmw== X-Received: by 10.129.73.65 with SMTP id w62mr58532906ywa.80.1480971063383; Mon, 05 Dec 2016 12:51:03 -0800 (PST) Received: from [192.168.1.130] (71-12-86-250.dhcp.spbg.sc.charter.com. [71.12.86.250]) by smtp.gmail.com with ESMTPSA id v66sm7352544ywc.27.2016.12.05.12.51.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 05 Dec 2016 12:51:02 -0800 (PST) Content-Type: multipart/alternative; boundary=Apple-Mail-122DEE4F-60A9-472B-967E-E792B51FC2C3 Mime-Version: 1.0 (1.0) Subject: Re: Req. for some help with resolving the error "pg_basebackup: could not get WAL end position from server: FATAL: requested WAL segment" From: John Scalia X-Mailer: iPad Mail (14B100) In-Reply-To: Date: Mon, 5 Dec 2016 15:51:02 -0500 Cc: "Tharmarajah, Sam" , "pgsql-admin@postgresql.org" Content-Transfer-Encoding: 7bit Message-Id: <38BC73D1-59DB-4E17-A252-C8DE9EE147CE@gmail.com> References: <873181a7a9164ffe8b74c6e425a1a724@Mail02.usask.ca> <11CEA700-3307-4425-8ADF-6E819447889E@gmail.com> <0bdba7c255474ff6973bbf5ba1ef9661@Mail02.usask.ca> To: Keith X-Pg-Spam-Score: -2.7 (--) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-admin Precedence: bulk Sender: pgsql-admin-owner@postgresql.org --Apple-Mail-122DEE4F-60A9-472B-967E-E792B51FC2C3 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable The only way to increase the segment size is to adjust the setting in the so= urce code and recompile the server. There are no runtime adjustments availab= le to do this. Sent from my iPad > On Dec 5, 2016, at 3:41 PM, Keith wrote: >=20 >=20 >=20 >> On Mon, Dec 5, 2016 at 3:27 PM, Tharmarajah, Sam wrote: >> Hi John, >>=20 >> =20 >>=20 >> Thank you for the feed back. Presently it is taking around 1hr to finish t= he backups. We have the wal_keep_segments set to 50. Is there way to adjust t= he segment size to be more than 16MB? At times I notice in the pg_xlog, ther= e are 10-15 segments on an invocation point. >>=20 >> =20 >>=20 >> =20 >>=20 >> Thanx, >>=20 >> Sam >>=20 >> =20 >>=20 >> From: John Scalia [mailto:jayknowsunix@gmail.com]=20 >> Sent: Monday, December 05, 2016 1:52 PM >> To: Tharmarajah, Sam >> Cc: pgsql-admin@postgresql.org >> Subject: Re: [ADMIN] Req. for some help with resolving the error "pg_base= backup: could not get WAL end position from server: FATAL: requested WAL seg= ment" >>=20 >> =20 >>=20 >> More than likely your keep_wal_segments setting is too low. How long is t= he pg_basebackup taking? Remember that segments are generated about every 5 m= inutes, so if the backup takes an hour, you need at least 12 segments stored= . At 2 hours, you need 24 etc., I'd set the value to about 12.2 segments/hou= r of backup. >>=20 >> -- >>=20 >> Jay >>=20 >>=20 >>=20 >> Sent from my iPad >>=20 >>=20 >> On Dec 5, 2016, at 1:57 PM, Tharmarajah, Sam wrote: >>=20 >> Hi >>=20 >> =20 >>=20 >> I am a newbie to postgres. We are on postgres 9.1 running on RedHat 6. W= hen we do the postgres backups we are noticing the following errors >>=20 >> =20 >>=20 >> pg_basebackup: could not get WAL end position from server: FATAL: reques= ted WAL segment 000000010000020B00000040 has already been removed >>=20 >> =20 >>=20 >> But when I look into pg_xlog, I notice >>=20 >> =20 >>=20 >> 000000010000020B00000040.00000118.backup >>=20 >> =20 >>=20 >> =20 >>=20 >> May I know how to resolve this error? Are the backups done with these err= ors, recoverable? >>=20 >> =20 >>=20 >> =20 >>=20 >> Thanks in advance for your support. >>=20 >> =20 >>=20 >> Thanx, >>=20 >> Sam >>=20 >> =20 >>=20 >> =20 >>=20 >> =20 >>=20 >> =20 >>=20 >=20 > If you're using pgbasebackup just for creating consistent backups, use the= --xlog-method=3Dstrean (-Xs) to ensure you get all the WAL files necessary f= or a consistent backup. This avoids having to set wal_keep_segments high eno= ugh to account for the time length of your entire backup. Note you will need= an extra WAL stream for this setting, so ensure max_wal_senders is set high= enough on the master. >=20 > Now if you're going to be using this backup to create a slave, you must ge= t that slave up and running before the master recycles all the WAL files nec= essary for replication to be able to catch up from when the backup stopped. O= therwise you will see that error you shared in your original post. That's wh= en you may need to adjust wal_keep_segments. Either that or you should set u= p an archive_command on the master to keep a secondary copy of all WAL files= .=20 >=20 > Keith --Apple-Mail-122DEE4F-60A9-472B-967E-E792B51FC2C3 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
The only way to increase the segment s= ize is to adjust the setting in the source code and recompile the server. Th= ere are no runtime adjustments available to do this.

Sent from my iPa= d

On Dec 5, 2016, at 3:41 PM, Keith <keith@keithf4.com> wrote:



On Mon, Dec 5, 2016 at 3:27 PM, Tharmarajah, Sam <sambavan.tharmarajah@usask.ca> wrote:

Hi John,<= u>

 =

Thank you for th= e feed back. Presently it is taking around 1hr to finish the backups. We hav= e the wal_keep_segments set to 50. Is there way to adjust the segment size t= o be more than 16MB? At times I notice in the pg_xlog, there are 10-15 segments on an invocation point.=

 =

 =

Thanx,=

Sam

 =

From: John Scalia [mailto:jayknowsunix@gmail.com]
Sent: Monday, December 05, 2016 1:52 PM
To: Tharmarajah, Sam <sambavan.tharmarajah@usask.ca>
Cc: p= gsql-admin@postgresql.org
Subject: Re: [ADMIN] Req. for some help with resolving the error "pg_= basebackup: could not get WAL end position from server: FATAL: requested WAL= segment"

 

More than likely your keep_wal_segments setting is to= o low. How long is the pg_basebackup taking? Remember that segments are gene= rated about every 5 minutes, so if the backup takes an hour, you need at lea= st 12 segments stored. At 2 hours, you need 24 etc., I'd set the value to about 12.2 segments/hour of backup.<= span style=3D"font-size:12pt">

--

Jay



Sent from my iPad


On Dec 5, 2016, at 1:57 PM, Tharmarajah, Sam <sambavan.tharmarajah@usask.ca> wrote:

Hi

 

I am a newbie to postgres. We are on postgres 9.1 run= ning on RedHat 6.  When we do the postgres backups we are noticing the f= ollowing errors

 

pg_basebackup: could not get WAL end position from se= rver: FATAL:  requested WAL segment  000000010000020B00000040 has a= lready been removed

 

But when I look into pg_xlog, I notice =

 

000000010000020B00000040.00000118.backup<= u>

 

 

May I know how to resolve this error? Are the backups= done with these errors, recoverable?

 

 

Thanks in advance for your support.

=

 

Thanx,

Sam

 

 

 

 


If you're using pgba= sebackup just for creating consistent backups, use the --xlog-method=3Dstrea= n (-Xs) to ensure you get all the WAL files necessary for a consistent backu= p. This avoids having to set wal_keep_segments high enough to account for th= e time length of your entire backup. Note you will need an extra WAL stream f= or this setting, so ensure max_wal_senders is set high enough on the master.=

Now if you're going to be using thi= s backup to create a slave, you must get that slave up and running before th= e master recycles all the WAL files necessary for replication to be able to c= atch up from when the backup stopped. Otherwise you will see that error you s= hared in your original post. That's when you may need to adjust wal_keep_seg= ments. Either that or you should set up an archive_command on the master to k= eep a secondary copy of all WAL files.

Keith
= --Apple-Mail-122DEE4F-60A9-472B-967E-E792B51FC2C3--