pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedpg_upgrade FAIL: can't find tablespaces
5+ messages / 2 participants
[nested] [flat]
* pg_upgrade FAIL: can't find tablespaces
@ 2014-08-18 18:59 Craig James <cjames@emolecules.com>
2014-08-18 19:15 ` Re: pg_upgrade FAIL: can't find tablespaces Craig James <cjames@emolecules.com>
0 siblings, 1 reply; 5+ messages in thread
From: Craig James @ 2014-08-18 18:59 UTC (permalink / raw)
To: pgsql-admin
I did pg_upgrade from 8.4.17 to 9.3.5. The upgrade claimed it was
successful. However, when I start Postgres 9.3.5, I get an error message
for every one of the roughly 250 databases:
LOG: could not open tablespace directory
"pg_tblspc/16828/PG_9.3_201306121": No such file or directory
LOG: could not open tablespace directory
"pg_tblspc/16523/PG_9.3_201306121": No such file or directory
LOG: could not open tablespace directory
"pg_tblspc/16768/PG_9.3_201306121": No such file or directory
LOG: could not open tablespace directory
"pg_tblspc/16715/PG_9.3_201306121": No such file or directory
... and so forth
My Postgres looks like this
/data/postgres/main - the PGDATA directory
/data/postgres/tablespaces - where most of the data live
Urgent question: Having started 9.3 (briefly), is it possible to revert to
8.4? Or do I have to fully revert my 750GB database from a backup?
Question 2: Is there a way to get 9.3 to work? Something I did wrong with
table spaces, or a step I missed?
I pretty much ran pg_upgrade per the instructions. See complete command and
output below.
Thanks,
Craig
/usr/local/pgsql-9.3.5/bin/pg_upgrade \
--link \
--user=postgres \
--old-bindir=/usr/local/pgsql-8.4.17/bin \
--new-bindir=/usr/local/pgsql-9.3.5/bin \
--old-datadir=/data/postgres/main \
--new-datadir=/data/postgres-9.3/main \
--jobs=4
Performing Consistency Checks
-----------------------------
Checking cluster versions ok
Checking database user is a superuser ok
Checking for prepared transactions ok
Checking for reg* system OID user data types ok
Checking for contrib/isn with bigint-passing mismatch ok
Checking for large objects warning
Your installation contains large objects. The new database has an
additional large object permission table. After upgrading, you will be
given a command to populate the pg_largeobject permission table with
default permissions.
Creating dump of global objects ok
Creating dump of database schemas
ok
Checking for presence of required libraries ok
Checking database user is a superuser ok
Checking for prepared transactions ok
If pg_upgrade fails after this point, you must re-initdb the
new cluster before continuing.
Performing Upgrade
------------------
Analyzing all rows in the new cluster ok
Freezing all rows on the new cluster ok
Deleting files from new pg_clog ok
Copying old pg_clog to new server ok
Setting next transaction ID for new cluster ok
Deleting files from new pg_multixact/offsets ok
Setting oldest multixact ID on new cluster ok
Resetting WAL archives ok
Setting frozenxid and minmxid counters in new cluster ok
Restoring global objects in the new cluster ok
Adding support functions to new cluster ok
Restoring database schemas in the new cluster
ok
Setting minmxid counter in new cluster ok
Removing support functions from new cluster ok
Adding ".old" suffix to old global/pg_control ok
If you want to start the old cluster, you will need to remove
the ".old" suffix from /data/postgres/main/global/pg_control.old.
Because "link" mode was used, the old cluster cannot be safely
started once the new cluster has been started.
Linking user relation files
ok
Setting next OID for new cluster ok
Sync data directory to disk ok
Creating script to analyze new cluster ok
Creating script to delete old cluster ok
Checking for large objects warning
Your installation contains large objects. The new database has an
additional large object permission table, so default permissions must be
defined for all large objects. The file
pg_largeobject.sql
when executed by psql by the database superuser will set the default
permissions.
Upgrade Complete
----------------
Optimizer statistics are not transferred by pg_upgrade so,
once you start the new server, consider running:
analyze_new_cluster.sh
Running this script will delete the old cluster's data files:
delete_old_cluster.sh
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: pg_upgrade FAIL: can't find tablespaces
2014-08-18 18:59 pg_upgrade FAIL: can't find tablespaces Craig James <cjames@emolecules.com>
@ 2014-08-18 19:15 ` Craig James <cjames@emolecules.com>
2014-08-21 02:23 ` Re: pg_upgrade FAIL: can't find tablespaces Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 5+ messages in thread
From: Craig James @ 2014-08-18 19:15 UTC (permalink / raw)
To: pgsql-admin
On Mon, Aug 18, 2014 at 11:59 AM, Craig James <cjames@emolecules.com> wrote:
> I did pg_upgrade from 8.4.17 to 9.3.5. The upgrade claimed it was
> successful. However, when I start Postgres 9.3.5, I get an error message
> for every one of the roughly 250 databases:
>
> LOG: could not open tablespace directory
> "pg_tblspc/16828/PG_9.3_201306121": No such file or directory
> LOG: could not open tablespace directory
> "pg_tblspc/16523/PG_9.3_201306121": No such file or directory
> LOG: could not open tablespace directory
> "pg_tblspc/16768/PG_9.3_201306121": No such file or directory
> LOG: could not open tablespace directory
> "pg_tblspc/16715/PG_9.3_201306121": No such file or directory
> ... and so forth
>
> My Postgres looks like this
>
> /data/postgres/main - the PGDATA directory
> /data/postgres/tablespaces - where most of the data live
>
OK, I figured this out: pg_upgrade didn't follow symbolic links. Each of
the missing files pointed to /data/postgres/tablespaces, but the actual
directory was in /data/postgres-8.4/tablespaces. When I replaced
/data/postgres/tablespaces with a symbolic link to
/data/postgres-8.4/tablespaces, everything worked.
Once everything is in order and I'm convinced 9.3.5 is working, I'll shut
down, remove the symlink, and move the tablespaces directory to the new
location.
Craig
> Urgent question: Having started 9.3 (briefly), is it possible to revert to
> 8.4? Or do I have to fully revert my 750GB database from a backup?
>
> Question 2: Is there a way to get 9.3 to work? Something I did wrong with
> table spaces, or a step I missed?
>
> I pretty much ran pg_upgrade per the instructions. See complete command
> and output below.
>
> Thanks,
> Craig
>
>
> /usr/local/pgsql-9.3.5/bin/pg_upgrade \
> --link \
> --user=postgres \
> --old-bindir=/usr/local/pgsql-8.4.17/bin \
> --new-bindir=/usr/local/pgsql-9.3.5/bin \
> --old-datadir=/data/postgres/main \
> --new-datadir=/data/postgres-9.3/main \
> --jobs=4
>
> Performing Consistency Checks
> -----------------------------
> Checking cluster versions ok
> Checking database user is a superuser ok
> Checking for prepared transactions ok
> Checking for reg* system OID user data types ok
> Checking for contrib/isn with bigint-passing mismatch ok
> Checking for large objects warning
>
> Your installation contains large objects. The new database has an
> additional large object permission table. After upgrading, you will be
> given a command to populate the pg_largeobject permission table with
> default permissions.
>
> Creating dump of global objects ok
> Creating dump of database schemas
> ok
> Checking for presence of required libraries ok
> Checking database user is a superuser ok
> Checking for prepared transactions ok
>
> If pg_upgrade fails after this point, you must re-initdb the
> new cluster before continuing.
>
> Performing Upgrade
> ------------------
> Analyzing all rows in the new cluster ok
> Freezing all rows on the new cluster ok
> Deleting files from new pg_clog ok
> Copying old pg_clog to new server ok
> Setting next transaction ID for new cluster ok
> Deleting files from new pg_multixact/offsets ok
> Setting oldest multixact ID on new cluster ok
> Resetting WAL archives ok
> Setting frozenxid and minmxid counters in new cluster ok
> Restoring global objects in the new cluster ok
> Adding support functions to new cluster ok
> Restoring database schemas in the new cluster
> ok
> Setting minmxid counter in new cluster ok
> Removing support functions from new cluster ok
> Adding ".old" suffix to old global/pg_control ok
>
> If you want to start the old cluster, you will need to remove
> the ".old" suffix from /data/postgres/main/global/pg_control.old.
> Because "link" mode was used, the old cluster cannot be safely
> started once the new cluster has been started.
>
> Linking user relation files
> ok
> Setting next OID for new cluster ok
> Sync data directory to disk ok
> Creating script to analyze new cluster ok
> Creating script to delete old cluster ok
> Checking for large objects warning
>
> Your installation contains large objects. The new database has an
> additional large object permission table, so default permissions must be
> defined for all large objects. The file
> pg_largeobject.sql
> when executed by psql by the database superuser will set the default
> permissions.
>
>
> Upgrade Complete
> ----------------
> Optimizer statistics are not transferred by pg_upgrade so,
> once you start the new server, consider running:
> analyze_new_cluster.sh
>
> Running this script will delete the old cluster's data files:
> delete_old_cluster.sh
>
>
--
---------------------------------
Craig A. James
Chief Technology Officer
eMolecules, Inc.
---------------------------------
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: pg_upgrade FAIL: can't find tablespaces
2014-08-18 18:59 pg_upgrade FAIL: can't find tablespaces Craig James <cjames@emolecules.com>
2014-08-18 19:15 ` Re: pg_upgrade FAIL: can't find tablespaces Craig James <cjames@emolecules.com>
@ 2014-08-21 02:23 ` Bruce Momjian <bruce@momjian.us>
2014-08-21 13:54 ` Re: pg_upgrade FAIL: can't find tablespaces Craig James <cjames@emolecules.com>
0 siblings, 1 reply; 5+ messages in thread
From: Bruce Momjian @ 2014-08-21 02:23 UTC (permalink / raw)
To: Craig James <cjames@emolecules.com>; +Cc: pgsql-admin
On Mon, Aug 18, 2014 at 12:15:57PM -0700, Craig James wrote:
>
>
> On Mon, Aug 18, 2014 at 11:59 AM, Craig James <cjames@emolecules.com> wrote:
>
> I did pg_upgrade from 8.4.17 to 9.3.5. The upgrade claimed it was
> successful. However, when I start Postgres 9.3.5, I get an error message
> for every one of the roughly 250 databases:
>
> LOG: could not open tablespace directory "pg_tblspc/16828/
> PG_9.3_201306121": No such file or directory
> LOG: could not open tablespace directory "pg_tblspc/16523/
> PG_9.3_201306121": No such file or directory
> LOG: could not open tablespace directory "pg_tblspc/16768/
> PG_9.3_201306121": No such file or directory
> LOG: could not open tablespace directory "pg_tblspc/16715/
> PG_9.3_201306121": No such file or directory
> ... and so forth
>
> My Postgres looks like this
>
> /data/postgres/main - the PGDATA directory
> /data/postgres/tablespaces - where most of the data live
>
>
> OK, I figured this out: pg_upgrade didn't follow symbolic links. Each of the
> missing files pointed to /data/postgres/tablespaces, but the actual directory
> was in /data/postgres-8.4/tablespaces. When I replaced /data/postgres/
> tablespaces with a symbolic link to /data/postgres-8.4/tablespaces, everything
> worked.
>
> Once everything is in order and I'm convinced 9.3.5 is working, I'll shut down,
> remove the symlink, and move the tablespaces directory to the new location.
Uh, I am not totally clear what happened here. I am unclear how
pg_upgrade would have moved the files yet the new server would not
start. Did you do anything between the completion of pg_upgrade and
trying to start the new server?
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ Everyone has their own god. +
--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: pg_upgrade FAIL: can't find tablespaces
2014-08-18 18:59 pg_upgrade FAIL: can't find tablespaces Craig James <cjames@emolecules.com>
2014-08-18 19:15 ` Re: pg_upgrade FAIL: can't find tablespaces Craig James <cjames@emolecules.com>
2014-08-21 02:23 ` Re: pg_upgrade FAIL: can't find tablespaces Bruce Momjian <bruce@momjian.us>
@ 2014-08-21 13:54 ` Craig James <cjames@emolecules.com>
2014-08-21 14:34 ` Re: pg_upgrade FAIL: can't find tablespaces Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 5+ messages in thread
From: Craig James @ 2014-08-21 13:54 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: pgsql-admin
On Wed, Aug 20, 2014 at 7:23 PM, Bruce Momjian <bruce@momjian.us> wrote:
> On Mon, Aug 18, 2014 at 12:15:57PM -0700, Craig James wrote:
> >
> >
> > On Mon, Aug 18, 2014 at 11:59 AM, Craig James <cjames@emolecules.com>
> wrote:
> >
> > I did pg_upgrade from 8.4.17 to 9.3.5. The upgrade claimed it was
> > successful. However, when I start Postgres 9.3.5, I get an error
> message
> > for every one of the roughly 250 databases:
> >
> > LOG: could not open tablespace directory "pg_tblspc/16828/
> > PG_9.3_201306121": No such file or directory
> > LOG: could not open tablespace directory "pg_tblspc/16523/
> > PG_9.3_201306121": No such file or directory
> > LOG: could not open tablespace directory "pg_tblspc/16768/
> > PG_9.3_201306121": No such file or directory
> > LOG: could not open tablespace directory "pg_tblspc/16715/
> > PG_9.3_201306121": No such file or directory
> > ... and so forth
> >
> > My Postgres looks like this
> >
> > /data/postgres/main - the PGDATA directory
> > /data/postgres/tablespaces - where most of the data live
> >
> >
> > OK, I figured this out: pg_upgrade didn't follow symbolic links. Each of
> the
> > missing files pointed to /data/postgres/tablespaces, but the actual
> directory
> > was in /data/postgres-8.4/tablespaces. When I replaced /data/postgres/
> > tablespaces with a symbolic link to /data/postgres-8.4/tablespaces,
> everything
> > worked.
> >
> > Once everything is in order and I'm convinced 9.3.5 is working, I'll
> shut down,
> > remove the symlink, and move the tablespaces directory to the new
> location.
>
> Uh, I am not totally clear what happened here. I am unclear how
> pg_upgrade would have moved the files yet the new server would not
> start. Did you do anything between the completion of pg_upgrade and
> trying to start the new server?
>
It's complicated. My situation is probably not relevant to anyone else.
Here is the history, which explains the how this occurred.
For consistency with our older systems, we built the database with PGDATA =
/postgres, even though it was actually in /data/postgres. /postgres was a
symlink.
Similarly, our tablespaces were specified as /postgres/tablespaces/xxx, but
in fact they were in /data/postgres/tablespaces/xxx (where xxx is the
tablespace name).
Prior running pg_upgrade, I renamed /data/postgres to /data/postgres-8.4
and updated the symlink /postgres. I verified that 8.4 still ran with no
problems.
pg_upgrade ran with no complaints. The new directory was /data/postgres-9.3.
Since I wasn't going to run 8.4 any more, I updated the symlink /postgres
to point to /data/postgres-9.3. Unfortunately, pg_upgrade hadn't created
/data/postgres-9.3/tablespaces, since it thought they were in
/postgres/tablespaces:
/data/postgres-9.3/main/pg_tblspc/nnnnn -> /postgres/tablespaces/xxx/
where "nnnnn" is the ID of the tablespace and "xxx" was the original
tablespace name. When I tried to start 9.3, I got the errors shown in my
original email.
To fix it, I simply moved /data/postgres-8.4/tablespaces to
/data/postgres-9.3/tablespaces.
One scary side effect: if I had run the cleanup script that pg_upgrade
produced, it would have deleted my entire database since
/data/postgres-8.4/tablespaces was NOT hard linked into the new
/data/postgres-9.3 directory.
Craig
> --
> Bruce Momjian <bruce@momjian.us> http://momjian.us
> EnterpriseDB http://enterprisedb.com
>
> + Everyone has their own god. +
>
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: pg_upgrade FAIL: can't find tablespaces
2014-08-18 18:59 pg_upgrade FAIL: can't find tablespaces Craig James <cjames@emolecules.com>
2014-08-18 19:15 ` Re: pg_upgrade FAIL: can't find tablespaces Craig James <cjames@emolecules.com>
2014-08-21 02:23 ` Re: pg_upgrade FAIL: can't find tablespaces Bruce Momjian <bruce@momjian.us>
2014-08-21 13:54 ` Re: pg_upgrade FAIL: can't find tablespaces Craig James <cjames@emolecules.com>
@ 2014-08-21 14:34 ` Bruce Momjian <bruce@momjian.us>
0 siblings, 0 replies; 5+ messages in thread
From: Bruce Momjian @ 2014-08-21 14:34 UTC (permalink / raw)
To: Craig James <cjames@emolecules.com>; +Cc: pgsql-admin
On Thu, Aug 21, 2014 at 06:54:06AM -0700, Craig James wrote:
> pg_upgrade ran with no complaints. The new directory was /data/postgres-9.3.
>
> Since I wasn't going to run 8.4 any more, I updated the symlink /postgres to
> point to /data/postgres-9.3. Unfortunately, pg_upgrade hadn't created /data/
> postgres-9.3/tablespaces, since it thought they were in /postgres/tablespaces:
>
> /data/postgres-9.3/main/pg_tblspc/nnnnn -> /postgres/tablespaces/xxx/
>
> where "nnnnn" is the ID of the tablespace and "xxx" was the original tablespace
> name. When I tried to start 9.3, I got the errors shown in my original email.
>
> To fix it, I simply moved /data/postgres-8.4/tablespaces to /data/postgres-9.3/
> tablespaces.
Thanks for the explanation. I wanted to make sure I wasn't missing
something.
> One scary side effect: if I had run the cleanup script that pg_upgrade
> produced, it would have deleted my entire database since /data/postgres-8.4/
> tablespaces was NOT hard linked into the new /data/postgres-9.3 directory.
Yes, very scary. We have a test in the code to see if you have placed
the tablespace inside the old cluster, but it can't check symlinks. We
also mention this in the docs:
Once you are satisfied with the upgrade, you can delete the old
cluster's data directories by running the script mentioned when
<command>pg_upgrade</command> completes. (Automatic deletion is not <--
possible if you have user-defined tablespaces inside the old data <--
directory.) You can also delete the old installation directories <--
(e.g. <filename>bin</>, <filename>share</>).
Not sure what more we can do.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ Everyone has their own god. +
--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2014-08-21 14:34 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2014-08-18 18:59 pg_upgrade FAIL: can't find tablespaces Craig James <cjames@emolecules.com>
2014-08-18 19:15 ` Craig James <cjames@emolecules.com>
2014-08-21 02:23 ` Bruce Momjian <bruce@momjian.us>
2014-08-21 13:54 ` Craig James <cjames@emolecules.com>
2014-08-21 14:34 ` Bruce Momjian <bruce@momjian.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox