Message-ID: <4E70ED3C.6020604@pinpointresearch.com>
Date: Wed, 14 Sep 2011 11:06:52 -0700
From: Steve Crawford <scrawford@pinpointresearch.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
	rv:1.9.2.21) Gecko/20110831 Lightning/1.0b2 Thunderbird/3.1.13
MIME-Version: 1.0
To: "pgsql-admin@postgresql.org Admin" <pgsql-admin@postgresql.org>
CC: devrim@gunduz.org, Tom Lane <tgl@sss.pgh.pa.us>
Subject: Re: pg_upgrade difficulties
References: <4E6FE821.5050403@pinpointresearch.com>
	<16242.1315960267@sss.pgh.pa.us>
	<4E70CCB7.5080506@pinpointresearch.com>
In-Reply-To: <4E70CCB7.5080506@pinpointresearch.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 09/14/2011 08:48 AM, Steve Crawford wrote:
> On 09/13/2011 05:31 PM, Tom Lane wrote:
>> Steve Crawford<scrawford@pinpointresearch.com>  writes:
>>> waiting for server to start....../usr/pgsql-9.1/bin/pg_ctl: symbol
>>> lookup error: /usr/pgsql-9.1/bin/pg_ctl: undefined symbol: PQping
>>> There were problems executing "/usr/pgsql-9.1/bin/pg_ctl" -w -l
>>> "upgrade.log" -D "/var/lib/pgsql/9.1/data" -o "-p 5432 -b" start>>
>>> "upgrade.log" 2>&1
>> This looks like your pg_ctl executable is mistakenly getting linked to a
>> pre-9.1 version of libpq.so.  Don't know enough about the file layout in
>> the current PGDG RPMs to speculate as to why.
>>
>>             regards, tom lane
>>
> Devrim, any thoughts?
>
> Cheers,
> Steve
>

OK, it's a packaging/library issue that seems likely to bite anyone 
trying to do a parallel install of 9.0 and 9.1 from PGDG CentOS repo. 
Don't know if if affects any other packages. Basically, everything seems 
to load the old libraries, not the new ones.

Here are the installed packages:
postgresql90.i386                   9.0.4-2PGDG.rhel5     installed
postgresql90-contrib.i386           9.0.4-2PGDG.rhel5     installed
postgresql90-devel.i386             9.0.4-2PGDG.rhel5     installed
postgresql90-libs.i386              9.0.4-2PGDG.rhel5     installed
postgresql90-server.i386            9.0.4-2PGDG.rhel5     installed
postgresql91.i386                   9.1.0-2PGDG.rhel5     installed
postgresql91-contrib.i386           9.1.0-2PGDG.rhel5     installed
postgresql91-devel.i386             9.1.0-2PGDG.rhel5     installed
postgresql91-libs.i386              9.1.0-2PGDG.rhel5     installed
postgresql91-server.i386            9.1.0-2PGDG.rhel5     installed

When pg_upgrade and psql do not work correctly:

root:~ ldconfig -p | grep pq
     libpqwalreceiver.so (libc6) => /usr/pgsql-9.0/lib/libpqwalreceiver.so
     libpqwalreceiver.so (libc6) => /usr/pgsql-9.1/lib/libpqwalreceiver.so
     libpq.so.5 (libc6) => /usr/pgsql-9.0/lib/libpq.so.5
     libpq.so.5 (libc6) => /usr/pgsql-9.1/lib/libpq.so.5
     libpq.so (libc6) => /usr/pgsql-9.0/lib/libpq.so
     libpq.so (libc6) => /usr/pgsql-9.1/lib/libpq.so

Here's how the libraries get listed for ldconfig:

root:ld.so.conf.d cd /etc/ld.so.conf.d
root:ld.so.conf.d ls -l post*
-rwx------ 1 root root 20 Sep  8 02:06 postgresql-9.0-libs.conf
-rwx------ 1 root root 20 Sep 12 10:57 postgresql-9.1-libs.conf

But if I rename postgresql-9.0-libs.conf so it sorts later and rerun 
ldconfig I see:

root:ld.so.conf.d mv postgresql-9.0-libs.conf postgresql-9.old-libs.conf

root:ld.so.conf.d ls -l post*
-rwx------ 1 root root 20 Sep 12 10:57 postgresql-9.1-libs.conf
-rwx------ 1 root root 20 Sep  8 02:06 postgresql-9.old-libs.conf

root:ld.so.conf.d ldconfig

root:ld.so.conf.d ldconfig -p | grep pq
     libpqwalreceiver.so (libc6) => /usr/pgsql-9.1/lib/libpqwalreceiver.so
     libpqwalreceiver.so (libc6) => /usr/pgsql-9.0/lib/libpqwalreceiver.so
     libpq.so.5 (libc6) => /usr/pgsql-9.1/lib/libpq.so.5
     libpq.so.5 (libc6) => /usr/pgsql-9.0/lib/libpq.so.5
     libpq.so (libc6) => /usr/pgsql-9.1/lib/libpq.so
     libpq.so (libc6) => /usr/pgsql-9.0/lib/libpq.so

Now pg_upgrade works fine, I can start the new cluster and psql works as 
expected. I can rename the 9.0 conf file back to its original name and 
use yum to remove 9.0 and everything seems to work. I suspect I could 
run 9.0 and 9.1 in parallel after fixing the library issue but did not try.

Cheers,
Steve