pg.ddx.io  pgsql-admin@postgresql.org mailing list archive  
help / color / mirror / Atom feed
psql enhancement idea
6+ messages / 2 participants
[nested] [flat]

* psql enhancement idea
@ 2004-10-21 19:05 Uwe C. Schroeder <uwe@oss4u.com>
  2004-10-21 19:40 ` Re: psql enhancement idea Tom Lane <tgl@sss.pgh.pa.us>
  0 siblings, 1 reply; 6+ messages in thread

From: Uwe C. Schroeder @ 2004-10-21 19:05 UTC (permalink / raw)
  To: pgsql-admin

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Hi everyone,

I'm just doing a load of database transfers using pg_dump. Because I changed 
the schema of quite some tables I have to dump full column inserts.
The normal -D dump doesn't use a transaction, and I can see that this 
certainly shouldn't be the default. Nonetheless I think it might be a nice 
feature if there is a optional switch that creates a BEGIN .... COMMIT block 
around the inserts of each table's data. Full column inserts are at least 10 
times faster if used in a transaction. Ok I understand that the transaction 
bites big chunks out of the ressources - which wouldn't matter if you do that 
on a development machine with plenty of everything and the tables don't have 
billions of rows.

Thought's anyone ?

	UC

- --
Open Source Solutions 4U, LLC	2570 Fleetwood Drive
Phone:  +1 650 872 2425		San Bruno, CA 94066
Cell:   +1 650 302 2405		United States
Fax:    +1 650 872 2417
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQFBeAhljqGXBvRToM4RAhHBAJ9pnOxF2owze5k1+qBsjg2BB39BuwCePdpJ
clYgE7K5TD7HYhhHy3JudHQ=
=Xpsr
-----END PGP SIGNATURE-----




^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: psql enhancement idea
  2004-10-21 19:05 psql enhancement idea Uwe C. Schroeder <uwe@oss4u.com>
@ 2004-10-21 19:40 ` Tom Lane <tgl@sss.pgh.pa.us>
  2004-10-21 22:00   ` Re: psql enhancement idea Uwe C. Schroeder <uwe@oss4u.com>
  0 siblings, 1 reply; 6+ messages in thread

From: Tom Lane @ 2004-10-21 19:40 UTC (permalink / raw)
  To: Uwe C. Schroeder <uwe@oss4u.com>; +Cc: pgsql-admin

"Uwe C. Schroeder" <uwe@oss4u.com> writes:
> I'm just doing a load of database transfers using pg_dump. Because I changed 
> the schema of quite some tables I have to dump full column inserts.

I don't think that conclusion follows from that premise.  In recent
pg_dump versions (any that use a column list with COPY, which I think is
7.3 or later) there is no fundamental disadvantage to using COPY; it
should be semantically equivalent to INSERT-with-column-list commands.

The only reason I can see for using the INSERT option anymore is if you
think that some of the row insertions might fail, and you don't want
that to stop the load altogether.  If that's what you want, then you'd
certainly not want to wrap the load in BEGIN/END, so there doesn't seem
to be much point in adding an option to do it.

			regards, tom lane



^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: psql enhancement idea
  2004-10-21 19:05 psql enhancement idea Uwe C. Schroeder <uwe@oss4u.com>
  2004-10-21 19:40 ` Re: psql enhancement idea Tom Lane <tgl@sss.pgh.pa.us>
@ 2004-10-21 22:00   ` Uwe C. Schroeder <uwe@oss4u.com>
  2004-10-21 22:03     ` Re: psql enhancement idea Tom Lane <tgl@sss.pgh.pa.us>
  0 siblings, 1 reply; 6+ messages in thread

From: Uwe C. Schroeder @ 2004-10-21 22:00 UTC (permalink / raw)
  To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-admin

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thursday 21 October 2004 12:40 pm, Tom Lane wrote:
> "Uwe C. Schroeder" <uwe@oss4u.com> writes:
> > I'm just doing a load of database transfers using pg_dump. Because I
> > changed the schema of quite some tables I have to dump full column
> > inserts.
>
> I don't think that conclusion follows from that premise.  In recent
> pg_dump versions (any that use a column list with COPY, which I think is
> 7.3 or later) there is no fundamental disadvantage to using COPY; it
> should be semantically equivalent to INSERT-with-column-list commands.
>
> The only reason I can see for using the INSERT option anymore is if you
> think that some of the row insertions might fail, and you don't want
> that to stop the load altogether.  If that's what you want, then you'd
> certainly not want to wrap the load in BEGIN/END, so there doesn't seem
> to be much point in adding an option to do it.
>
> 			regards, tom lane

The reason is that I made changes to the schema, i.e. changed the ordinal 
position of columns and added some columns. Since the positioning of the 
columns isn't the same anymore a copy will fail. Column Inserts will leave 
the new columns untouched (set them to NULL or whatever the default) but can 
handle the change in order of the columns (ok, the order thing is purely 
cosmetic...). 
I see your point that you don't want to fail the whole import when using full 
inserts - which would happen in a transaction block. My point was basically 
that I assume all inserts will succeed, however I can't use copy because of 
the changes but doing it in a transaction will increase the insert speed by 
at least 10 (chewing up memory certainly).

As a remark: normally I make pure data backups as copy and as full column 
inserts. In case of a recovery I then can easily switch to single inserts if 
the copy fails - eats a bit more diskspace but increases security (besides 
the slony replication I'm running on top of that...)

Uwe

- --
Open Source Solutions 4U, LLC	2570 Fleetwood Drive
Phone:  +1 650 872 2425		San Bruno, CA 94066
Cell:   +1 650 302 2405		United States
Fax:    +1 650 872 2417
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQFBeDF0jqGXBvRToM4RAmrkAKCoRXnlmnRpp0DI5/cFeCyx97nm1QCfWJ/A
8zSHT4NXzG8LG7GFqpPGSbE=
=c99W
-----END PGP SIGNATURE-----




^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: psql enhancement idea
  2004-10-21 19:05 psql enhancement idea Uwe C. Schroeder <uwe@oss4u.com>
  2004-10-21 19:40 ` Re: psql enhancement idea Tom Lane <tgl@sss.pgh.pa.us>
  2004-10-21 22:00   ` Re: psql enhancement idea Uwe C. Schroeder <uwe@oss4u.com>
@ 2004-10-21 22:03     ` Tom Lane <tgl@sss.pgh.pa.us>
  2004-10-21 22:18       ` Re: psql enhancement idea Uwe C. Schroeder <uwe@oss4u.com>
  0 siblings, 1 reply; 6+ messages in thread

From: Tom Lane @ 2004-10-21 22:03 UTC (permalink / raw)
  To: Uwe C. Schroeder <uwe@oss4u.com>; +Cc: pgsql-admin

"Uwe C. Schroeder" <uwe@oss4u.com> writes:
> On Thursday 21 October 2004 12:40 pm, Tom Lane wrote:
>> I don't think that conclusion follows from that premise.  In recent
>> pg_dump versions (any that use a column list with COPY, which I think is
>> 7.3 or later) there is no fundamental disadvantage to using COPY; it
>> should be semantically equivalent to INSERT-with-column-list commands.

> The reason is that I made changes to the schema, i.e. changed the ordinal 
> position of columns and added some columns. Since the positioning of the 
> columns isn't the same anymore a copy will fail.

No, it won't, not if you use a column list.

			regards, tom lane



^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: psql enhancement idea
  2004-10-21 19:05 psql enhancement idea Uwe C. Schroeder <uwe@oss4u.com>
  2004-10-21 19:40 ` Re: psql enhancement idea Tom Lane <tgl@sss.pgh.pa.us>
  2004-10-21 22:00   ` Re: psql enhancement idea Uwe C. Schroeder <uwe@oss4u.com>
  2004-10-21 22:03     ` Re: psql enhancement idea Tom Lane <tgl@sss.pgh.pa.us>
@ 2004-10-21 22:18       ` Uwe C. Schroeder <uwe@oss4u.com>
  2004-10-21 22:50         ` Re: psql enhancement idea Tom Lane <tgl@sss.pgh.pa.us>
  0 siblings, 1 reply; 6+ messages in thread

From: Uwe C. Schroeder @ 2004-10-21 22:18 UTC (permalink / raw)
  To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-admin

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thursday 21 October 2004 03:03 pm, Tom Lane wrote:
> "Uwe C. Schroeder" <uwe@oss4u.com> writes:
> > On Thursday 21 October 2004 12:40 pm, Tom Lane wrote:
> >> I don't think that conclusion follows from that premise.  In recent
> >> pg_dump versions (any that use a column list with COPY, which I think is
> >> 7.3 or later) there is no fundamental disadvantage to using COPY; it
> >> should be semantically equivalent to INSERT-with-column-list commands.
> >
> > The reason is that I made changes to the schema, i.e. changed the ordinal
> > position of columns and added some columns. Since the positioning of the
> > columns isn't the same anymore a copy will fail.
>
> No, it won't, not if you use a column list.

Ahhh - I see what you mean, thanks for that hint.

A little remark then: the pg_dump manpage (7.4.3) states

       --attribute-inserts
              Dump  data as INSERT commands with explicit column names (INSERT
              INTO table (column, ...) VALUES ...). This will make restoration
              very  slow,  but  it is necessary if you desire to rearrange the
              column ordering.

Maybe that should be fixed then. I was used to do the full insert statements 
(ok, I've been using postgres long before it even had SQL) - maybe a old 
habit then...

	Uwe

- --
Open Source Solutions 4U, LLC	2570 Fleetwood Drive
Phone:  +1 650 872 2425		San Bruno, CA 94066
Cell:   +1 650 302 2405		United States
Fax:    +1 650 872 2417
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQFBeDW4jqGXBvRToM4RAu4+AJ9gchqGNrP/IUUYy8ySMWLyP6I0JgCdG1SH
h0RLcQppb2yLaTkrG/W5cvc=
=5tEF
-----END PGP SIGNATURE-----




^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: psql enhancement idea
  2004-10-21 19:05 psql enhancement idea Uwe C. Schroeder <uwe@oss4u.com>
  2004-10-21 19:40 ` Re: psql enhancement idea Tom Lane <tgl@sss.pgh.pa.us>
  2004-10-21 22:00   ` Re: psql enhancement idea Uwe C. Schroeder <uwe@oss4u.com>
  2004-10-21 22:03     ` Re: psql enhancement idea Tom Lane <tgl@sss.pgh.pa.us>
  2004-10-21 22:18       ` Re: psql enhancement idea Uwe C. Schroeder <uwe@oss4u.com>
@ 2004-10-21 22:50         ` Tom Lane <tgl@sss.pgh.pa.us>
  0 siblings, 0 replies; 6+ messages in thread

From: Tom Lane @ 2004-10-21 22:50 UTC (permalink / raw)
  To: Uwe C. Schroeder <uwe@oss4u.com>; +Cc: pgsql-admin

"Uwe C. Schroeder" <uwe@oss4u.com> writes:
> A little remark then: the pg_dump manpage (7.4.3) states

>        --attribute-inserts
>               Dump  data as INSERT commands with explicit column names (INSERT
>               INTO table (column, ...) VALUES ...). This will make restoration
>               very  slow,  but  it is necessary if you desire to rearrange the
>               column ordering.

> Maybe that should be fixed then.

Mmm, you are right, that comment is a tad out of date.  Patch committed
--- thanks for pointing it out!

			regards, tom lane



^ permalink  raw  reply  [nested|flat] 6+ messages in thread


end of thread, other threads:[~2004-10-21 22:50 UTC | newest]

Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2004-10-21 19:05 psql enhancement idea Uwe C. Schroeder <uwe@oss4u.com>
2004-10-21 19:40 ` Tom Lane <tgl@sss.pgh.pa.us>
2004-10-21 22:00   ` Uwe C. Schroeder <uwe@oss4u.com>
2004-10-21 22:03     ` Tom Lane <tgl@sss.pgh.pa.us>
2004-10-21 22:18       ` Uwe C. Schroeder <uwe@oss4u.com>
2004-10-21 22:50         ` Tom Lane <tgl@sss.pgh.pa.us>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox