From: Ron <ronljohnsonjr@gmail.com>
To: pgsql-admin@lists.postgresql.org
Subject: Re: Upgrade from PG12 to PG
Date: Thu, 20 Jul 2023 12:22:39 -0500
Message-ID: <78789644-f55a-51eb-8507-58415064414d@gmail.com> (raw)
In-Reply-To: <1cb1652e-286a-077e-950d-516970ecf499@gmail.com>
References: <3901e0da-819b-5b6d-bb57-c29f6d384725@gmail.com>
<CAG95seWz3dJxKQPg4_3EAEBCXUuJTDpZBUKAp0+bqP2amQaZ9g@mail.gmail.com>
<dc88d14d-6d0f-2d67-ecfc-c7495bf1c22b@gmail.com>
<B6D3FD80-5794-47B8-9074-E10C4945951B@elevated-dev.com>
<1cb1652e-286a-077e-950d-516970ecf499@gmail.com>
Jeff,
It does a "--schema-only" dump. The /gianter/ the schema, the longer that
"pg_dump --schema-only" takes.
Note also that there's a known issue with pg_upgrade and /millions/ of Large
Objects (not bytea or text, but lo_* columns).
On 7/20/23 12:05, Jef Mortelle wrote:
> running /usr/lib/postgresql15/bin/pg_upgrade -v -p 5431 -P 5432 -k
>
> gives you in the output :
>
> ...
> "/usr/lib/postgresql15/bin/pg_dump" --host /pg/PG15 --port 5431 --username
> postgres --schema-only --quote-all-identifiers --binary-upgrade
> --format=custom --verbose
> --file="/pg/PG15/system/pg_upgrade_output.d/20230720T184333.610/dump/pg_upgrade_dump_16385.custom"
> 'dbname=dbname' >>
> "/pg/PG15/system/pg_upgrade_output.d/20230720T184333.610/log/pg_upgrade_dump_16385.log"
> 2>&1
> ...
>
>
> so, yes pg_ugrade start a pg_dump session,
>
>
> even a pg_restore at the end, runs about 7 hours
>
> "/usr/lib/postgresql15/bin/pg_restore" --host /pg/PG15 --port 5432
> --username postgres --create --exit-on-error --verbose --dbname template1
> "/pg/PG15/system/pg_upgrade_output.d/20230720T184333.610/dump/pg_upgrade_dump_16385.custom"
> >>
> "/pg/PG15/system/pg_upgrade_output.d/20230720T184333.610/log/pg_upgrade_dump_16385.log"
> 2>&1
>
> with a lot of lines in the logfiles like: pg_restore: executing BLOB
> 11108809
>
>
> Server is a VM server, my VM has 64GB SuseSLES attached to a SAN with SSD
> disk (Hp3Par)
>
> 2)vacuumlo
>
> lovacum -n => I have vacuumlo, running this:
>
> vacuumlo dbname -v
>
> postgres@dbfluxd02:/pg/data> vacuumlo dflux -v
> Connected to database "dflux"
> Checking message_oid in fluxnaf.flux_message_content
> Checking payload_oid in fluxnaf.bridge_payload_content
> Successfully removed 0 large objects from database "dflux".
>
> and again same issues
>
>
>
> On 20/07/2023 16:51, Scott Ribe wrote:
>>> On Jul 20, 2023, at 7:46 AM, Jef Mortelle <jefmortelle@gmail.com> wrote:
>>>
>>> So: not possible to have very little downtime if you have a database
>>> with al lot rows containing text as datatype, as pg_upgrade needs 12hr
>>> for 24 milj rows in pg_largeobject.
>> We need to get terminology straight, as at the moment your posts are very
>> confusing. In PostgreSQL large objects and text are not the same. Text is
>> basically varchar without a specified length limit. Large object is a
>> blob (but not what SQL calls a BLOB)--it is kind of like a file stored
>> outside the normal table mechanism, and provides facilities for partial
>> reads, etc: https://www.postgresql.org/docs/15/largeobjects.html. There
>> are a number of ways to wind up with references to large objects all
>> deleted, but the orphaned large objects still in the database.
>>
>> First thing you should do: run lovacuum -n to find out if you have
>> orphaned large objects. If so, start cleaning those up, then see how long
>> pg_upgrade takes.
>>
>> Second, what's your hardware? I really don't see dump & restore of a 1TB
>> database taking 6 hours.
>>
>>> Alsready tried to use --link and --jobs, but you cannot ommit the
>>> "select lo_unlink ...." for every rows containing datatype text in
>>> your database that the pg_* program creates in the export/dump file.
>> Terminology again, or are you conflating two different issues? pg_upgrade
>> --link does not create a dump file.
>
>
--
Born in Arizona, moved to Babylonia.
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-admin@postgresql.org
Cc: ronljohnsonjr@gmail.com, pgsql-admin@lists.postgresql.org
Subject: Re: Upgrade from PG12 to PG
In-Reply-To: <78789644-f55a-51eb-8507-58415064414d@gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox