Subject: Bug in pg_upgrade standby rsync doc

From: Don Seiler <[email-masked]>
Date: Thu, 7 Dec 2017 11:28:12 -0600

In step 10 of the pg_upgrade doc at
https://www.postgresql.org/docs/9.6/static/pgupgrade.html, it uses this
example for rsyncing the main $PGDATA dir to the standby:

rsync --archive --delete --hard-links --size-only --no-inc-recursive
/opt/PostgreSQL/9.5/data \
      /opt/PostgreSQL/9.6/data standby.example.com:/opt/PostgreSQL

However when I ran this (substituting /var/lib/pgsql for /opt/PostgreSQL),
I found that it put a directory under /var/lib/pgsql/data, and my 9.6/data
dir was still empty. Furthermore, what was in /var/lib/pgsql/data appeared
to be the old 9.2 contents.

I suspect this is because the "data" directories are two levels below the
parent directory specified at the end of the command. When I used the
similar command for separate tablespaces, it worked as expected.

When I run the command with the "data" dir, so that I'm just specifying 9.2
and 9.6, it works as desired. eg:

rsync --archive --delete --hard-links --size-only --no-inc-recursive
/opt/PostgreSQL/9.5 \
      /opt/PostgreSQL/9.6 standby.example.com:/opt/PostgreSQL

FWIW I was upgrading from 9.2.22 to 9.6.6 on CentOS 6.

-- 
Don Seiler
www.seiler.us

---

From: Bruce Momjian <[email-masked]>
Date: Thu, 25 Jan 2018 12:46:52 -0500

On Thu, Dec  7, 2017 at 11:28:12AM -0600, Don Seiler wrote:
> In step 10 of the pg_upgrade doc at https://www.postgresql.org/docs/9.6/static/
> pgupgrade.html, it uses this example for rsyncing the main $PGDATA dir to the
> standby:
> 
> 
> rsync --archive --delete --hard-links --size-only --no-inc-recursive /opt/PostgreSQL/9.5/data \
> However when I ran this (substituting /var/lib/pgsql for /opt/PostgreSQL), I
> found that it put a directory under /var/lib/pgsql/data, and my 9.6/data dir
> was still empty. Furthermore, what was in /var/lib/pgsql/data appeared to be
> the old 9.2 contents.
> 
> parent directory specified at the end of the command. When I used the similar
> command for separate tablespaces, it worked as expected.
> 
> When I run the command with the "data" dir, so that I'm just specifying 9.2 and
> 9.6, it works as desired. eg:
> 
> 
> rsync --archive --delete --hard-links --size-only --no-inc-recursive /opt/PostgreSQL/9.5 \

Sorry I am just getting to this.  I was able to reproduce your results
with rsync version 3.1.1.  With /opt/PostgreSQL containing:

	/opt/PostgreSQL
	/opt/PostgreSQL/9.5
	/opt/PostgreSQL/9.5/data
	/opt/PostgreSQL/9.5/data/x
	/opt/PostgreSQL/9.6
	/opt/PostgreSQL/9.6/data
	/opt/PostgreSQL/9.6/data/y

This script:

	TMP="/tmp"
	
	rm -rf $TMP/PostgreSQL
	mkdir -p $TMP/PostgreSQL/9.5/data
	mkdir -p $TMP/PostgreSQL/9.6/data
	
	rsync --archive --delete --hard-links --size-only \
		--no-inc-recursive /opt/PostgreSQL/9.5 \
		/opt/PostgreSQL/9.6 $TMP/PostgreSQL
	
	find $TMP/PostgreSQL -print

Yields:

	/tmp/PostgreSQL
	/tmp/PostgreSQL/9.5
	/tmp/PostgreSQL/9.5/data
-->	/tmp/PostgreSQL/9.5/data/x
	/tmp/PostgreSQL/9.6
	/tmp/PostgreSQL/9.6/data
-->	/tmp/PostgreSQL/9.6/data/y

which is correct, but if I change rsync to match our docs:

	rsync --archive --delete --hard-links --size-only \
		--no-inc-recursive /opt/PostgreSQL/9.5/data \
		/opt/PostgreSQL/9.6/data $TMP/PostgreSQL

I get:

	/rtmp/PostgreSQL
	/rtmp/PostgreSQL/9.5
	/rtmp/PostgreSQL/9.5/data
	/rtmp/PostgreSQL/9.6
	/rtmp/PostgreSQL/9.6/data
-->	/rtmp/PostgreSQL/data
-->	/rtmp/PostgreSQL/data/x
-->	/rtmp/PostgreSQL/data/y

which is incorrect and the behavior you reported.

The incorrect example was added a few months ago:

	commit 9521ce4a7a1125385fb4de9689f345db594c516a
	Author: Bruce Momjian <[email-masked]>
	Date:   Wed Sep 13 09:11:28 2017 -0400
	
	    docs:  improve pg_upgrade standby instructions
	
	    This makes it clear that pg_upgrade standby upgrade instructions should
	    only be used in link mode, adds examples, and explains how rsync works
	    with links.
	
	    Reported-by: Andreas Joseph Krogh
	
	    Discussion: https://postgr.es/m/VisenaEmail.6c.c0e592c5af4ef0a2.15e785dcb61@tc7-visena
	
	    Backpatch-through: 9.5

but the generic syntax mentioning the data directory has been there for
a while.  I am wondering if people had to test this to get it working
and didn't report that saying datadir was inaccurate.

I propse the attached patch to fix the generic syntax and the example. 
I will backpatch it through 9.5.  As you stated, the tablespace example
is fine.

-- 
  Bruce Momjian  <[email-masked]>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +
diff --git a/doc/src/sgml/ref/pgupgrade.sgml b/doc/src/sgml/ref/pgupgrade.sgml
new file mode 100644
index 055eac3..baf67db
*** a/doc/src/sgml/ref/pgupgrade.sgml
--- b/doc/src/sgml/ref/pgupgrade.sgml
*************** pg_upgrade.exe
*** 494,503 ****
         server:
  
  <programlisting>
! rsync --archive --delete --hard-links --size-only --no-inc-recursive old_pgdata new_pgdata remote_dir
  </programlisting>
  
!        where <option>old_pgdata</option> and <option>new_pgdata</option> are relative
         to the current directory on the primary, and <option>remote_dir</option>
         is <emphasis>above</emphasis> the old and new cluster directories
         on the standby.  The directory structure under the specified
--- 494,503 ----
         server:
  
  <programlisting>
! rsync --archive --delete --hard-links --size-only --no-inc-recursive old_cluster new_cluster remote_dir
  </programlisting>
  
!        where <option>old_cluster</option> and <option>new_cluster</option> are relative
         to the current directory on the primary, and <option>remote_dir</option>
         is <emphasis>above</emphasis> the old and new cluster directories
         on the standby.  The directory structure under the specified
*************** rsync --archive --delete --hard-links --
*** 506,513 ****
         remote directory, e.g.
  
  <programlisting>
! rsync --archive --delete --hard-links --size-only --no-inc-recursive /opt/PostgreSQL/9.5/data \
!       /opt/PostgreSQL/9.6/data standby.example.com:/opt/PostgreSQL
  </programlisting>
  
         You can verify what the command will do using
--- 506,513 ----
         remote directory, e.g.
  
  <programlisting>
! rsync --archive --delete --hard-links --size-only --no-inc-recursive /opt/PostgreSQL/9.5 \
!       /opt/PostgreSQL/9.6 standby.example.com:/opt/PostgreSQL
  </programlisting>
  
         You can verify what the command will do using

---

From: Bruce Momjian <[email-masked]>
Date: Wed, 31 Jan 2018 16:44:01 -0500

> I will backpatch it through 9.5.  As you stated, the tablespace example

Done.  Thanks for the report.