pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedHS/SR on AIX
9+ messages / 4 participants
[nested] [flat]
* HS/SR on AIX
@ 2010-08-24 21:39 Steve Singer <ssinger@ca.afilias.info>
2010-08-24 23:56 ` Re: HS/SR on AIX Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 9+ messages in thread
From: Steve Singer @ 2010-08-24 21:39 UTC (permalink / raw)
To: PostgreSQL-development Hackers <pgsql-hackers@postgresql.org>
I think I've been able to reproduce the issue floating around with
streaming replication on AIX.
LOG: could not bind IPv6 socket: The socket name is already in use.
HINT: Is another postmaster already running on port 5433? If not, wait
a few seconds and retry.
LOG: database system was shut down in recovery at 2010-08-24 21:08:37 UTC
LOG: entering standby mode
cp: cannot stat `/opt/rg/data_tb3/steve/wals/000000010000000000000001':
A file or directory in the path name does not exist.
LOG: redo starts at 0/1000020
LOG: record with zero length at 0/1012280
cp: cannot stat `/opt/rg/data_tb3/steve/wals/000000010000000000000001':
A file or directory in the path name does not exist.
FATAL: could not load library
"/opt/dbs/pgsql9-beta2/lib/libpqwalreceiver.so": 0509-022 Cannot
load module /opt/dbs/pgsql9-beta2/lib/libpqwalreceiver.so.
0509-150 Dependent module libpq.a(libpq.so.5) could
not be loaded.
0509-022 Cannot load module libpq.a(libpq.so.5).
0509-026 System error: A file or directory in the path
name does not exist.
0509-022 Cannot load module
/opt/dbs/pgsql9-beta2/lib/libpqwalreceiver.so.
0509-150 Dependent module
/opt/dbs/pgsql9-beta2/lib/libpqwalreceiver.so could not be loaded.
This worked fine with beta2 but now seems to be an issue on beta4.
If I do
export LIBPATH=/opt/dbs/pgsql9-beta2/lib/
before starting the standby postmaster then it seems to work.
I haven't yet tried running truss to try to look at where it is looking
for libpq from. liblibpqwalreceiver is being linked with this
gcc -maix64 -O2 -Wall -Wmissing-prototypes -Wpointer-arith
-Wdeclaration-after-statement -Wendif-labels -fno-strict-aliasing
-fwrapv -g -o libpqwalreceiver.so liblibpqwalreceiver.a
-Wl,-bE:liblibpqwalreceiver.exp -L../../../../src/port
-Wl,-bmaxdata:0x80000000,-bbigtoc -L/opt/freeware/lib
-Wl,-blibpath:/opt/dbs/pgsql9-beta2/lib:/opt/freeware/lib:/usr/lib:/lib
-Wl,-bnoentry -Wl,-H512 -Wl,-bM:SRE -L../../../../src/interfaces/libpq
-lpq -Wl,-bI:../../../../src/backend/postgres.imp
I'll try to look into this a bit more tomorrow or thursday.
--
Steve Singer
Afilias Canada
Data Services Developer
416-673-1142
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: HS/SR on AIX
2010-08-24 21:39 HS/SR on AIX Steve Singer <ssinger@ca.afilias.info>
@ 2010-08-24 23:56 ` Tom Lane <tgl@sss.pgh.pa.us>
2010-08-25 15:05 ` Re: HS/SR on AIX Steve Singer <ssinger@ca.afilias.info>
0 siblings, 1 reply; 9+ messages in thread
From: Tom Lane @ 2010-08-24 23:56 UTC (permalink / raw)
To: Steve Singer <ssinger@ca.afilias.info>; +Cc: PostgreSQL-development Hackers <pgsql-hackers@postgresql.org>
Steve Singer <ssinger@ca.afilias.info> writes:
> I think I've been able to reproduce the issue floating around with
> streaming replication on AIX.
Excellent, because we weren't getting much from the original reporter.
> This worked fine with beta2 but now seems to be an issue on beta4.
> If I do
> export LIBPATH=/opt/dbs/pgsql9-beta2/lib/
> before starting the standby postmaster then it seems to work.
Fascinating. That seems to prove that it's an rpath problem. My
first guess is that the relevant change between beta2 and beta4 is
my LDFLAGS changes. See
http://archives.postgresql.org/pgsql-committers/2010-07/msg00060.php
and following commits.
regards, tom lane
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: HS/SR on AIX
2010-08-24 21:39 HS/SR on AIX Steve Singer <ssinger@ca.afilias.info>
2010-08-24 23:56 ` Re: HS/SR on AIX Tom Lane <tgl@sss.pgh.pa.us>
@ 2010-08-25 15:05 ` Steve Singer <ssinger@ca.afilias.info>
2010-08-25 15:45 ` Re: HS/SR on AIX Steve Singer <ssinger@ca.afilias.info>
0 siblings, 1 reply; 9+ messages in thread
From: Steve Singer @ 2010-08-25 15:05 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: PostgreSQL-development Hackers <pgsql-hackers@postgresql.org>
Tom Lane wrote:
> Steve Singer <ssinger@ca.afilias.info> writes:
>> I think I've been able to reproduce the issue floating around with
>> streaming replication on AIX.
>
> Excellent, because we weren't getting much from the original reporter.
I'm withdrawing my comment, today on a clean install of the binaries I
am not able to reproduce any of this.
Today with beta4 I can stream replication to the standby and bring the
standby up to read-write without issues.
Yesterday when I had put beta4 on top of beta2 without explicitly
deleting all of the beta2 binaries/libraries first. I'm thinking maybe
some portions of beta2 where still laying around.
It is also possible that an old version of a shared library was still
sitting in memory and was being picked up by the newer postgresql (man
slibclean on AIX)
I will do another clean build from the beta4 source tar to confirm that
I'm not still having the issue but I'm thinking the original reporter
might have done something similar and had some old artifacts laying around.
>
>> This worked fine with beta2 but now seems to be an issue on beta4.
>
>> If I do
>> export LIBPATH=/opt/dbs/pgsql9-beta2/lib/
>> before starting the standby postmaster then it seems to work.
>
> Fascinating. That seems to prove that it's an rpath problem. My
> first guess is that the relevant change between beta2 and beta4 is
> my LDFLAGS changes. See
> http://archives.postgresql.org/pgsql-committers/2010-07/msg00060.php
> and following commits.
>
> regards, tom lane
>
--
Steve Singer
Afilias Canada
Data Services Developer
416-673-1142
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: HS/SR on AIX
2010-08-24 21:39 HS/SR on AIX Steve Singer <ssinger@ca.afilias.info>
2010-08-24 23:56 ` Re: HS/SR on AIX Tom Lane <tgl@sss.pgh.pa.us>
2010-08-25 15:05 ` Re: HS/SR on AIX Steve Singer <ssinger@ca.afilias.info>
@ 2010-08-25 15:45 ` Steve Singer <ssinger@ca.afilias.info>
2010-08-26 05:44 ` Re: HS/SR on AIX Fujii Masao <masao.fujii@gmail.com>
0 siblings, 1 reply; 9+ messages in thread
From: Steve Singer @ 2010-08-25 15:45 UTC (permalink / raw)
To: Steve Singer <ssinger@ca.afilias.info>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; PostgreSQL-development Hackers <pgsql-hackers@postgresql.org>
Steve Singer wrote:
> Tom Lane wrote:
>> Steve Singer <ssinger@ca.afilias.info> writes:
>>> I think I've been able to reproduce the issue floating around with
>>> streaming replication on AIX.
>>
>
> I will do another clean build from the beta4 source tar to confirm that
> I'm not still having the issue but I'm thinking the original reporter
> might have done something similar and had some old artifacts laying around.
>
A clean build from the beta4 source tarball where I'm careful to install
into a clean (ie no old beta2 artifacts laying around waiting to be
overwritten) isn't reproducing the issue.
I'm happy to try other things if people suggest them (or if the original
reporter is still getting this after making sure he cleans up old files
first) but I'm thinking that was the issue.
--
Steve Singer
Afilias Canada
Data Services Developer
416-673-1142
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: HS/SR on AIX
2010-08-24 21:39 HS/SR on AIX Steve Singer <ssinger@ca.afilias.info>
2010-08-24 23:56 ` Re: HS/SR on AIX Tom Lane <tgl@sss.pgh.pa.us>
2010-08-25 15:05 ` Re: HS/SR on AIX Steve Singer <ssinger@ca.afilias.info>
2010-08-25 15:45 ` Re: HS/SR on AIX Steve Singer <ssinger@ca.afilias.info>
@ 2010-08-26 05:44 ` Fujii Masao <masao.fujii@gmail.com>
2010-08-27 19:31 ` Re: [HACKERS] HS/SR on AIX Alanoly Andrews <alanolya@invera.com>
0 siblings, 1 reply; 9+ messages in thread
From: Fujii Masao @ 2010-08-26 05:44 UTC (permalink / raw)
To: Steve Singer <ssinger@ca.afilias.info>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; PostgreSQL-development Hackers <pgsql-hackers@postgresql.org>; alanolya@invera.com
On Thu, Aug 26, 2010 at 12:45 AM, Steve Singer <ssinger@ca.afilias.info> wrote:
> A clean build from the beta4 source tarball where I'm careful to install
> into a clean (ie no old beta2 artifacts laying around waiting to be
> overwritten) isn't reproducing the issue.
>
> I'm happy to try other things if people suggest them (or if the original
> reporter is still getting this after making sure he cleans up old files
> first) but I'm thinking that was the issue.
Thanks for the report!
Alanoly, could you do a clean install and try the test again?
Regards,
--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: [HACKERS] HS/SR on AIX
2010-08-24 21:39 HS/SR on AIX Steve Singer <ssinger@ca.afilias.info>
2010-08-24 23:56 ` Re: HS/SR on AIX Tom Lane <tgl@sss.pgh.pa.us>
2010-08-25 15:05 ` Re: HS/SR on AIX Steve Singer <ssinger@ca.afilias.info>
2010-08-25 15:45 ` Re: HS/SR on AIX Steve Singer <ssinger@ca.afilias.info>
2010-08-26 05:44 ` Re: HS/SR on AIX Fujii Masao <masao.fujii@gmail.com>
@ 2010-08-27 19:31 ` Alanoly Andrews <alanolya@invera.com>
2010-08-27 19:45 ` Re: [HACKERS] HS/SR on AIX Steve Singer <ssinger@ca.afilias.info>
0 siblings, 1 reply; 9+ messages in thread
From: Alanoly Andrews @ 2010-08-27 19:31 UTC (permalink / raw)
To: Fujii Masao <masao.fujii@gmail.com>; Steve Singer <ssinger@ca.afilias.info>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; PostgreSQL-development Hackers <pgsql-hackers@postgresql.org>; pgsql-admin
Fujii,
All my tests so far were done on "clean" installs. Every version I tested on, beta2 through beta4, was compiled and installed in different and unique directories.
Regards.
Alanoly.
-----Original Message-----
From: Fujii Masao [mailto:masao.fujii@gmail.com]
Sent: Thursday, August 26, 2010 1:44 AM
To: Steve Singer
Cc: Tom Lane; PostgreSQL-development Hackers; Alanoly Andrews
Subject: Re: [HACKERS] HS/SR on AIX
On Thu, Aug 26, 2010 at 12:45 AM, Steve Singer <ssinger@ca.afilias.info> wrote:
> A clean build from the beta4 source tarball where I'm careful to install
> into a clean (ie no old beta2 artifacts laying around waiting to be
> overwritten) isn't reproducing the issue.
>
> I'm happy to try other things if people suggest them (or if the original
> reporter is still getting this after making sure he cleans up old files
> first) but I'm thinking that was the issue.
Thanks for the report!
Alanoly, could you do a clean install and try the test again?
Regards,
--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
****************************************************
This e-mail may be privileged and/or confidential, and the sender does not waive any related rights and obligations. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. If you received this e-mail in error, please advise me (by return e-mail or otherwise) immediately.
Ce courriel est confidentiel et protégé. L'expéditeur ne renonce pas aux droits et obligations qui s'y rapportent. Toute diffusion, utilisation ou copie de ce message ou des renseignements qu'il contient par une personne autre que le (les) destinataire(s) désigné(s) est interdite. Si vous recevez ce courriel par erreur, veuillez m'en aviser immédiatement, par retour de courriel ou par un autre moyen.
****************************************************
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: [HACKERS] HS/SR on AIX
2010-08-24 21:39 HS/SR on AIX Steve Singer <ssinger@ca.afilias.info>
2010-08-24 23:56 ` Re: HS/SR on AIX Tom Lane <tgl@sss.pgh.pa.us>
2010-08-25 15:05 ` Re: HS/SR on AIX Steve Singer <ssinger@ca.afilias.info>
2010-08-25 15:45 ` Re: HS/SR on AIX Steve Singer <ssinger@ca.afilias.info>
2010-08-26 05:44 ` Re: HS/SR on AIX Fujii Masao <masao.fujii@gmail.com>
2010-08-27 19:31 ` Re: [HACKERS] HS/SR on AIX Alanoly Andrews <alanolya@invera.com>
@ 2010-08-27 19:45 ` Steve Singer <ssinger@ca.afilias.info>
2010-08-27 19:57 ` Re: [HACKERS] HS/SR on AIX Alanoly Andrews <alanolya@invera.com>
0 siblings, 1 reply; 9+ messages in thread
From: Steve Singer @ 2010-08-27 19:45 UTC (permalink / raw)
To: Alanoly Andrews <alanolya@invera.com>; +Cc: Fujii Masao <masao.fujii@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; PostgreSQL-development Hackers <pgsql-hackers@postgresql.org>; pgsql-admin
Alanoly Andrews wrote:
> Fujii,
>
> All my tests so far were done on "clean" installs. Every version I tested on, beta2 through beta4, was compiled and installed in different and unique directories.
>
> Regards.
Alanoly,
If you do an
export LIBPATH=/apps/pg_9.0_b4/lib
before starting postgres on the replica does it make a difference?
How about with a LIBPATH=/apps/pg_9.0_b4/lib/postgresql
(I'm not exactly sure where libpq.a is on your install)
>
> Alanoly.
--
Steve Singer
Afilias Canada
Data Services Developer
416-673-1142
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: [HACKERS] HS/SR on AIX
2010-08-24 21:39 HS/SR on AIX Steve Singer <ssinger@ca.afilias.info>
2010-08-24 23:56 ` Re: HS/SR on AIX Tom Lane <tgl@sss.pgh.pa.us>
2010-08-25 15:05 ` Re: HS/SR on AIX Steve Singer <ssinger@ca.afilias.info>
2010-08-25 15:45 ` Re: HS/SR on AIX Steve Singer <ssinger@ca.afilias.info>
2010-08-26 05:44 ` Re: HS/SR on AIX Fujii Masao <masao.fujii@gmail.com>
2010-08-27 19:31 ` Re: [HACKERS] HS/SR on AIX Alanoly Andrews <alanolya@invera.com>
2010-08-27 19:45 ` Re: [HACKERS] HS/SR on AIX Steve Singer <ssinger@ca.afilias.info>
@ 2010-08-27 19:57 ` Alanoly Andrews <alanolya@invera.com>
2010-08-27 20:21 ` Re: [HACKERS] HS/SR on AIX Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 9+ messages in thread
From: Alanoly Andrews @ 2010-08-27 19:57 UTC (permalink / raw)
To: Steve Singer <ssinger@ca.afilias.info>; +Cc: Fujii Masao <masao.fujii@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; PostgreSQL-development Hackers <pgsql-hackers@postgresql.org>; pgsql-admin
Steve,
I have tried all the LIBPATH's that you suggested. Besides, I don't think the problem is that postgres cannot find the "libpqwalreceiver" library. It does find it, but crashes on loading it. See below a repeat of the copy and paste from my first post, showing the sequence just before the crash:
(dbx) where
_alloc_initial_pthread(??) at 0x90000000049567c
__pth_init(??) at 0x900000000493ba4
uload(??, ??, ??, ??, ??, ??, ??, ??) at 0x9fffffff0001954
load_64.load(??, ??, ??) at 0x90000000004686c
loadAndInit() at 0x90000000047bd7c
dlopen(??, ??) at 0x90000000011cc4c
internal_load_library(libname = "/apps/pg_9.0_b4/lib/postgresql/libpqwalreceiver.so"), line 234 in "dfmgr.c"
load_file(filename = "libpqwalreceiver", restricted = '\0'), line 156 in "dfmgr.c"
WalReceiverMain(), line 248 in "walreceiver.c"
AuxiliaryProcessMain(argc = 2, argv = 0x0fffffffffffa8b8), line 428 in "bootstrap.c"
StartChildProcess(type = WalReceiverProcess), line 4405 in "postmaster.c"
sigusr1_handler(postgres_signal_arg = 30), line 4227 in "postmaster.c"
__fd_select(??, ??, ??, ??, ??) at 0x90000000011805c
postmaster.select(__fds = 5, __readlist = 0x0fffffffffffd0a8, __writelist = (nil), __exceptlist = (nil), __timeout = 0x0ffffffffffff0c0), line 229 in "time.h"
unnamed block in ServerLoop(), line 1391 in "postmaster.c"
unnamed block in ServerLoop(), line 1391 in "postmaster.c"
ServerLoop(), line 1391 in "postmaster.c"
PostmasterMain(argc = 1, argv = 0x00000001102aa4b0), line 1092 in "postmaster.c"
main(argc = 1, argv = 0x00000001102aa4b0), line 188 in "main.c"
Alanoly.
-----Original Message-----
From: Steve Singer [mailto:ssinger@ca.afilias.info]
Sent: Friday, August 27, 2010 3:45 PM
To: Alanoly Andrews
Cc: 'Fujii Masao'; Tom Lane; PostgreSQL-development Hackers; pgsql-admin@postgresql.org
Subject: Re: [HACKERS] HS/SR on AIX
Alanoly Andrews wrote:
> Fujii,
>
> All my tests so far were done on "clean" installs. Every version I tested on, beta2 through beta4, was compiled and installed in different and unique directories.
>
> Regards.
Alanoly,
If you do an
export LIBPATH=/apps/pg_9.0_b4/lib
before starting postgres on the replica does it make a difference?
How about with a LIBPATH=/apps/pg_9.0_b4/lib/postgresql
(I'm not exactly sure where libpq.a is on your install)
>
> Alanoly.
--
Steve Singer
Afilias Canada
Data Services Developer
416-673-1142
****************************************************
This e-mail may be privileged and/or confidential, and the sender does not waive any related rights and obligations. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. If you received this e-mail in error, please advise me (by return e-mail or otherwise) immediately.
Ce courriel est confidentiel et protégé. L'expéditeur ne renonce pas aux droits et obligations qui s'y rapportent. Toute diffusion, utilisation ou copie de ce message ou des renseignements qu'il contient par une personne autre que le (les) destinataire(s) désigné(s) est interdite. Si vous recevez ce courriel par erreur, veuillez m'en aviser immédiatement, par retour de courriel ou par un autre moyen.
****************************************************
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: [HACKERS] HS/SR on AIX
2010-08-24 21:39 HS/SR on AIX Steve Singer <ssinger@ca.afilias.info>
2010-08-24 23:56 ` Re: HS/SR on AIX Tom Lane <tgl@sss.pgh.pa.us>
2010-08-25 15:05 ` Re: HS/SR on AIX Steve Singer <ssinger@ca.afilias.info>
2010-08-25 15:45 ` Re: HS/SR on AIX Steve Singer <ssinger@ca.afilias.info>
2010-08-26 05:44 ` Re: HS/SR on AIX Fujii Masao <masao.fujii@gmail.com>
2010-08-27 19:31 ` Re: [HACKERS] HS/SR on AIX Alanoly Andrews <alanolya@invera.com>
2010-08-27 19:45 ` Re: [HACKERS] HS/SR on AIX Steve Singer <ssinger@ca.afilias.info>
2010-08-27 19:57 ` Re: [HACKERS] HS/SR on AIX Alanoly Andrews <alanolya@invera.com>
@ 2010-08-27 20:21 ` Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 9+ messages in thread
From: Tom Lane @ 2010-08-27 20:21 UTC (permalink / raw)
To: Alanoly Andrews <alanolya@invera.com>; +Cc: 'Steve Singer' <ssinger@ca.afilias.info>; 'Fujii Masao' <masao.fujii@gmail.com>; PostgreSQL-development Hackers <pgsql-hackers@postgresql.org>; pgsql-admin
Alanoly Andrews <alanolya@invera.com> writes:
> I have tried all the LIBPATH's that you suggested. Besides, I don't
> think the problem is that postgres cannot find the "libpqwalreceiver"
> library. It does find it, but crashes on loading it.
I thought the point of Steve's remarks was that it might be loading
the wrong version, either of libpqwalreceiver.so itself or libpq.so.
regards, tom lane
^ permalink raw reply [nested|flat] 9+ messages in thread
end of thread, other threads:[~2010-08-27 20:21 UTC | newest]
Thread overview: 9+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2010-08-24 21:39 HS/SR on AIX Steve Singer <ssinger@ca.afilias.info>
2010-08-24 23:56 ` Tom Lane <tgl@sss.pgh.pa.us>
2010-08-25 15:05 ` Steve Singer <ssinger@ca.afilias.info>
2010-08-25 15:45 ` Steve Singer <ssinger@ca.afilias.info>
2010-08-26 05:44 ` Fujii Masao <masao.fujii@gmail.com>
2010-08-27 19:31 ` Alanoly Andrews <alanolya@invera.com>
2010-08-27 19:45 ` Steve Singer <ssinger@ca.afilias.info>
2010-08-27 19:57 ` Alanoly Andrews <alanolya@invera.com>
2010-08-27 20:21 ` Tom Lane <tgl@sss.pgh.pa.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox