pg.ddx.io pgsql-sql@postgresql.org mailing list archive
help / color / mirror / Atom feedSpeeds using a transaction vrs not
6+ messages / 2 participants
[nested] [flat]
* Speeds using a transaction vrs not
@ 2005-02-22 22:19 Joel Fradkin <jfradkin@wazagua.com>
2005-02-23 08:02 ` Re: Speeds using a transaction vrs not Richard Huxton <dev@archonet.com>
0 siblings, 1 reply; 6+ messages in thread
From: Joel Fradkin @ 2005-02-22 22:19 UTC (permalink / raw)
To: pgsql-sql
I wrote a .net program to move my data to postgres (works great on
SQ_ASCII).
In fiddling around I tried it using the odbc driver and a transaction
originally, but converted it to using the .net connectivity but no
transaction.
What I found was it moved my database (4 gig in MSSQL) in 2 hours using the
.net, but 12 hours with the odbc and transaction.
Have any of you played around to see if using a transaction should be that
much slower or is it the odbc versus .net?
I paid a consultant to look at what would work best and his speed tests
indicated the odbc and the .net were pretty close, so I am assuming it is
because I am using a transaction on my odbc test.
I can run again without it or with it on .net driver, but thought I would
ask.
We only use transaction on important multiple table updates in our system
now, so should not be a huge thing, but was curious.
Joel Fradkin
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Speeds using a transaction vrs not
2005-02-22 22:19 Speeds using a transaction vrs not Joel Fradkin <jfradkin@wazagua.com>
@ 2005-02-23 08:02 ` Richard Huxton <dev@archonet.com>
2005-02-24 13:51 ` Re: Speeds using a transaction vrs not Joel Fradkin <jfradkin@wazagua.com>
0 siblings, 1 reply; 6+ messages in thread
From: Richard Huxton @ 2005-02-23 08:02 UTC (permalink / raw)
To: Joel Fradkin <jfradkin@wazagua.com>; +Cc: pgsql-sql
Joel Fradkin wrote:
> I wrote a .net program to move my data to postgres (works great on
> SQ_ASCII).
>
> In fiddling around I tried it using the odbc driver and a transaction
> originally, but converted it to using the .net connectivity but no
> transaction.
>
> What I found was it moved my database (4 gig in MSSQL) in 2 hours using the
> .net, but 12 hours with the odbc and transaction.
You *are* using transactions, you don't have a choice. Did you do the
transfer of all 4GB in ONE transaction with the ODBC? Please describe
the process in more detail.
--
Richard Huxton
Archonet Ltd
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Speeds using a transaction vrs not
2005-02-22 22:19 Speeds using a transaction vrs not Joel Fradkin <jfradkin@wazagua.com>
2005-02-23 08:02 ` Re: Speeds using a transaction vrs not Richard Huxton <dev@archonet.com>
@ 2005-02-24 13:51 ` Joel Fradkin <jfradkin@wazagua.com>
2005-02-24 14:21 ` Re: Speeds using a transaction vrs not Richard Huxton <dev@archonet.com>
0 siblings, 1 reply; 6+ messages in thread
From: Joel Fradkin @ 2005-02-24 13:51 UTC (permalink / raw)
To: 'Richard Huxton' <dev@archonet.com>; +Cc: pgsql-sql
No I did not do it in on transaction (although in .net I never started or
commited a transaction.
ODBC :
myTrans = myConnection.BeginTransaction(IsolationLevel.ReadCommitted)
' Assign transaction object for a pending local transaction
myCommand.Transaction = myTrans
'example of insert
'myCommand.CommandText = "Insert into Region (RegionID, RegionDescription)
VALUES (101, 'Description')"
myCommand.CommandText = insertsqltext
myCommand.ExecuteNonQuery()
myTrans.Commit()
.net driver:
Dim cmd As New NpgsqlCommand(insertsqltext, cnn)
cmd.ExecuteNonQuery()
cmd.Dispose()
Joel Fradkin
Wazagua, Inc.
2520 Trailmate Dr
Sarasota, Florida 34243
Tel. 941-753-7111 ext 305
jfradkin@wazagua.com
www.wazagua.com
Powered by Wazagua
Providing you with the latest Web-based technology & advanced tools.
C 2004. WAZAGUA, Inc. All rights reserved. WAZAGUA, Inc
This email message is for the use of the intended recipient(s) and may
contain confidential and privileged information. Any unauthorized review,
use, disclosure or distribution is prohibited. If you are not the intended
recipient, please contact the sender by reply email and delete and destroy
all copies of the original message, including attachments.
-----Original Message-----
From: Richard Huxton [mailto:dev@archonet.com]
Sent: Wednesday, February 23, 2005 3:03 AM
To: Joel Fradkin
Cc: pgsql-sql@postgresql.org
Subject: Re: [SQL] Speeds using a transaction vrs not
Joel Fradkin wrote:
> I wrote a .net program to move my data to postgres (works great on
> SQ_ASCII).
>
> In fiddling around I tried it using the odbc driver and a transaction
> originally, but converted it to using the .net connectivity but no
> transaction.
>
> What I found was it moved my database (4 gig in MSSQL) in 2 hours using
the
> .net, but 12 hours with the odbc and transaction.
You *are* using transactions, you don't have a choice. Did you do the
transfer of all 4GB in ONE transaction with the ODBC? Please describe
the process in more detail.
--
Richard Huxton
Archonet Ltd
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Speeds using a transaction vrs not
2005-02-22 22:19 Speeds using a transaction vrs not Joel Fradkin <jfradkin@wazagua.com>
2005-02-23 08:02 ` Re: Speeds using a transaction vrs not Richard Huxton <dev@archonet.com>
2005-02-24 13:51 ` Re: Speeds using a transaction vrs not Joel Fradkin <jfradkin@wazagua.com>
@ 2005-02-24 14:21 ` Richard Huxton <dev@archonet.com>
2005-02-24 15:47 ` Re: Speeds using a transaction vrs not Joel Fradkin <jfradkin@wazagua.com>
0 siblings, 1 reply; 6+ messages in thread
From: Richard Huxton @ 2005-02-24 14:21 UTC (permalink / raw)
To: Joel Fradkin <jfradkin@wazagua.com>; +Cc: pgsql-sql
Joel Fradkin wrote:
> No I did not do it in on transaction (although in .net I never started or
> commited a transaction.
All inserts/updates/etc take place within a transaction with PostgreSQL.
Some client libraries autocommit for you - you'll need to read the
documentation.
> ODBC :
> myCommand.CommandText = insertsqltext
> myCommand.ExecuteNonQuery()
> myTrans.Commit()
>
> .net driver:
> Dim cmd As New NpgsqlCommand(insertsqltext, cnn)
> cmd.ExecuteNonQuery()
> cmd.Dispose()
Both look to me like they are producing one transaction per insert (the
slowest possible way to bulk-copy data). That's assuming each block of
commands is within one loop.
Precisely what is happening will be easiest to see by turning statement
logging on in your postgresql.conf and comparing two runs. The delay
might be in overheads of setting up the transaction etc. with the ODBC
driver.
In any case, if bulk-copying data you'll find a huge improvement
grouping rows together in batches of 100 - 10,000.
--
Richard Huxton
Archonet Ltd
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Speeds using a transaction vrs not
2005-02-22 22:19 Speeds using a transaction vrs not Joel Fradkin <jfradkin@wazagua.com>
2005-02-23 08:02 ` Re: Speeds using a transaction vrs not Richard Huxton <dev@archonet.com>
2005-02-24 13:51 ` Re: Speeds using a transaction vrs not Joel Fradkin <jfradkin@wazagua.com>
2005-02-24 14:21 ` Re: Speeds using a transaction vrs not Richard Huxton <dev@archonet.com>
@ 2005-02-24 15:47 ` Joel Fradkin <jfradkin@wazagua.com>
2005-02-24 16:15 ` Re: Speeds using a transaction vrs not Richard Huxton <dev@archonet.com>
0 siblings, 1 reply; 6+ messages in thread
From: Joel Fradkin @ 2005-02-24 15:47 UTC (permalink / raw)
To: 'Richard Huxton' <dev@archonet.com>; +Cc: pgsql-sql
Thanks.
I guess I could add that logic, but this is a one time process going to run
the night we go live on postgres.
With .net it took like 2 hours to do the whole shebang, so I am happy.
But I was curious as we will be using odbc for our asp (aprx 90% of our
app).
I could test wrapping the .net in a transaction the way I do in odbc, but
you indicate it is doing that behind the scenes.
So far progress is going really great on our conversion.
I cant wait to see us up on the new platform.
I have not researched the details on doing autovacuum and backup.
I have tested manually backing up and restoring.
We are only like 10% done converting the code (I had hoped it going faster,
but I have not been hands on helping, so maybe it go faster when I get time
to start helping).
I did convert a fairly large app and it seemed to run faster on the postgres
box then the MSSQL box, so my first project after actually getting the data
and views has gone very smooth.
Everyone has been so helpful on the lists and that is the only reason I am
where I am so far so thanks again to everyone who has bothered to answer my
noob questions.
Joel Fradkin
Wazagua, Inc.
2520 Trailmate Dr
Sarasota, Florida 34243
Tel. 941-753-7111 ext 305
jfradkin@wazagua.com
www.wazagua.com
Powered by Wazagua
Providing you with the latest Web-based technology & advanced tools.
C 2004. WAZAGUA, Inc. All rights reserved. WAZAGUA, Inc
This email message is for the use of the intended recipient(s) and may
contain confidential and privileged information. Any unauthorized review,
use, disclosure or distribution is prohibited. If you are not the intended
recipient, please contact the sender by reply email and delete and destroy
all copies of the original message, including attachments.
-----Original Message-----
From: Richard Huxton [mailto:dev@archonet.com]
Sent: Thursday, February 24, 2005 9:21 AM
To: Joel Fradkin
Cc: pgsql-sql@postgresql.org
Subject: Re: [SQL] Speeds using a transaction vrs not
Joel Fradkin wrote:
> No I did not do it in on transaction (although in .net I never started or
> commited a transaction.
All inserts/updates/etc take place within a transaction with PostgreSQL.
Some client libraries autocommit for you - you'll need to read the
documentation.
> ODBC :
> myCommand.CommandText = insertsqltext
> myCommand.ExecuteNonQuery()
> myTrans.Commit()
>
> .net driver:
> Dim cmd As New NpgsqlCommand(insertsqltext, cnn)
> cmd.ExecuteNonQuery()
> cmd.Dispose()
Both look to me like they are producing one transaction per insert (the
slowest possible way to bulk-copy data). That's assuming each block of
commands is within one loop.
Precisely what is happening will be easiest to see by turning statement
logging on in your postgresql.conf and comparing two runs. The delay
might be in overheads of setting up the transaction etc. with the ODBC
driver.
In any case, if bulk-copying data you'll find a huge improvement
grouping rows together in batches of 100 - 10,000.
--
Richard Huxton
Archonet Ltd
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Speeds using a transaction vrs not
2005-02-22 22:19 Speeds using a transaction vrs not Joel Fradkin <jfradkin@wazagua.com>
2005-02-23 08:02 ` Re: Speeds using a transaction vrs not Richard Huxton <dev@archonet.com>
2005-02-24 13:51 ` Re: Speeds using a transaction vrs not Joel Fradkin <jfradkin@wazagua.com>
2005-02-24 14:21 ` Re: Speeds using a transaction vrs not Richard Huxton <dev@archonet.com>
2005-02-24 15:47 ` Re: Speeds using a transaction vrs not Joel Fradkin <jfradkin@wazagua.com>
@ 2005-02-24 16:15 ` Richard Huxton <dev@archonet.com>
0 siblings, 0 replies; 6+ messages in thread
From: Richard Huxton @ 2005-02-24 16:15 UTC (permalink / raw)
To: Joel Fradkin <jfradkin@wazagua.com>; +Cc: pgsql-sql
Joel Fradkin wrote:
> Thanks.
>
> I guess I could add that logic, but this is a one time process going to run
> the night we go live on postgres.
> With .net it took like 2 hours to do the whole shebang, so I am happy.
Fair enough. I tend to produce a text-file and use COPY for bulk
transfers, but that's just me.
> I did convert a fairly large app and it seemed to run faster on the postgres
> box then the MSSQL box, so my first project after actually getting the data
> and views has gone very smooth.
Don't forget to set aside some time for tweaking configuration settings
so you can tune the system.
--
Richard Huxton
Archonet Ltd
^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2005-02-24 16:15 UTC | newest]
Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2005-02-22 22:19 Speeds using a transaction vrs not Joel Fradkin <jfradkin@wazagua.com>
2005-02-23 08:02 ` Richard Huxton <dev@archonet.com>
2005-02-24 13:51 ` Joel Fradkin <jfradkin@wazagua.com>
2005-02-24 14:21 ` Richard Huxton <dev@archonet.com>
2005-02-24 15:47 ` Joel Fradkin <jfradkin@wazagua.com>
2005-02-24 16:15 ` Richard Huxton <dev@archonet.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox