pg.ddx.io  pgsql-sql@postgresql.org mailing list archive  
help / color / mirror / Atom feed
Re: persistent connections, AOLserver (Was: maybe Offtopic : PostgreSQL & PHP ?)
4+ messages / 3 participants
[nested] [flat]

* Re: persistent connections, AOLserver (Was: maybe Offtopic : PostgreSQL & PHP ?)
@ 2001-11-30 10:23 Steve Brett <steve.brett@e-mis.com>
  2001-12-03 19:07 ` Re: persistent connections, AOLserver (Was: maybe Offtopic : Stephan Szabo <sszabo@megazone23.bigpanda.com>
  0 siblings, 1 reply; 4+ messages in thread

From: Steve Brett @ 2001-11-30 10:23 UTC (permalink / raw)
  To: pgsql-sql

is there a limit/guide to the number of inserts you should use in a
transaction block ?

i have an app at the minute written indelphi that moves data from a cache
database and a sql server databse and produces one definitive customer
management database for the company i work for.

the transfer is quite slow but i do use single inserts (not through a lack
of knowledge for what a database is for though :-) and maybe wrapping in a
transaction block is the answer.

i'm moving (parsing and matching) pretty small recorsets (10,000 in one and
approx 60,000 in the other).

Steve


"Roberto Mello" <rmello@cc.usu.edu> wrote in message
news:20011119083547.A22031@cc.usu.edu...
> On Wed, Apr 18, 2001 at 08:28:14PM +0200, Mathijs Brands wrote:
> >
> > I've seen both MySQL and PostgreSQL give up (MySQL just hung, pgsql
> > cored) when I tried simple selects from a couple of hundred concurrent
> > connections; no transactions or other fancy stuff there. I think I was
> > using MySQL 3.22.?? and pgsql 6.5.3, so more modern versions may well be
> > able to cope with these numbers of connections. It's been more than a
> > year since I last tried it.
>
> PG 7.0 was released almost 2 years ago, so you had a very old version when
> you tried.
>
> > I don't know what the current state of affairs is, but it is my
> > understanding that, while pgsql performs admirably on tasks which mostly
> > read data, pgsql isn't really able to cope (performance wise) with an
> > application that has a very high insert to select ratio, such as OLTP.
>
> This is simply not true. It may have been true in the past, but not since
> PG 7.0 came out.
>
> A mistake that I see MySQL users do frequently is that when they have a
> bulk of inserts to do, they don't use the COPY command to bulk load the
> data and/or they don't wrap the inserts into one (or several) transaction
> blocks, leaving each INSERT in its own transaction, which obviously will
> hurt performance.
>
> They do that because they usually have no idea of what a database is for,
> besides being an SQL interface to their file systems. Exactly what MySQL
> was until very recently (and thanks to the work of third-parties, because
> the MySQL team itself couldn't care less about real features).
>
> -Roberto
>
> --
> +----| http://fslc.usu.edu USU Free Software & GNU/Linux Club |------+
>   Roberto Mello - Computer Science, USU - http://www.brasileiro.net
>        http://www.sdl.usu.edu - Space Dynamics Lab, Developer
> TAFB -> Text Above Fullquote Below
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org





^ permalink  raw  reply  [nested|flat] 4+ messages in thread

* Re: persistent connections, AOLserver (Was: maybe Offtopic :
  2001-11-30 10:23 Re: persistent connections, AOLserver (Was: maybe Offtopic : PostgreSQL & PHP ?) Steve Brett <steve.brett@e-mis.com>
@ 2001-12-03 19:07 ` Stephan Szabo <sszabo@megazone23.bigpanda.com>
  2001-12-17 02:01   ` Re: persistent connections, AOLserver (Was: maybe Offtopic : Marc Spitzer <marc@oscar.eng.cv.net>
  0 siblings, 1 reply; 4+ messages in thread

From: Stephan Szabo @ 2001-12-03 19:07 UTC (permalink / raw)
  To: Steve Brett <steve.brett@e-mis.com>; +Cc: pgsql-sql

On Fri, 30 Nov 2001, Steve Brett wrote:

> is there a limit/guide to the number of inserts you should use in a
> transaction block ?

Not a hard one, but I usually do thousands of rows (around 5k) per
transaction when I'm batch inserting, and I usually set it there so
I can watch its progress from another db session as the counts rise.





^ permalink  raw  reply  [nested|flat] 4+ messages in thread

* Re: persistent connections, AOLserver (Was: maybe Offtopic :
  2001-11-30 10:23 Re: persistent connections, AOLserver (Was: maybe Offtopic : PostgreSQL & PHP ?) Steve Brett <steve.brett@e-mis.com>
  2001-12-03 19:07 ` Re: persistent connections, AOLserver (Was: maybe Offtopic : Stephan Szabo <sszabo@megazone23.bigpanda.com>
@ 2001-12-17 02:01   ` Marc Spitzer <marc@oscar.eng.cv.net>
  2001-12-17 03:40     ` Re: persistent connections, AOLserver (Was: maybe Offtopic : Stephan Szabo <sszabo@megazone23.bigpanda.com>
  0 siblings, 1 reply; 4+ messages in thread

From: marc@oscar.eng.cv.net @ 2001-12-17 02:01 UTC (permalink / raw)
  To: pgsql-sql

In article <20011203110458.E93680-100000@megazone23.bigpanda.com>, 
Stephan Szabo wrote:
> On Fri, 30 Nov 2001, Steve Brett wrote:
> 
>> is there a limit/guide to the number of inserts you should use in a
>> transaction block ?
> 
> Not a hard one, but I usually do thousands of rows (around 5k) per
> transaction when I'm batch inserting, and I usually set it there so
> I can watch its progress from another db session as the counts rise.
> 

How do you set up the other session to watch?

marc


> 
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)



^ permalink  raw  reply  [nested|flat] 4+ messages in thread

* Re: persistent connections, AOLserver (Was: maybe Offtopic :
  2001-11-30 10:23 Re: persistent connections, AOLserver (Was: maybe Offtopic : PostgreSQL & PHP ?) Steve Brett <steve.brett@e-mis.com>
  2001-12-03 19:07 ` Re: persistent connections, AOLserver (Was: maybe Offtopic : Stephan Szabo <sszabo@megazone23.bigpanda.com>
  2001-12-17 02:01   ` Re: persistent connections, AOLserver (Was: maybe Offtopic : Marc Spitzer <marc@oscar.eng.cv.net>
@ 2001-12-17 03:40     ` Stephan Szabo <sszabo@megazone23.bigpanda.com>
  0 siblings, 0 replies; 4+ messages in thread

From: Stephan Szabo @ 2001-12-17 03:40 UTC (permalink / raw)
  To: Marc Spitzer <marc@oscar.eng.cv.net>; +Cc: pgsql-sql


On Mon, 17 Dec 2001, Marc Spitzer wrote:

> In article <20011203110458.E93680-100000@megazone23.bigpanda.com>,
> Stephan Szabo wrote:
> > On Fri, 30 Nov 2001, Steve Brett wrote:
> >
> >> is there a limit/guide to the number of inserts you should use in a
> >> transaction block ?
> >
> > Not a hard one, but I usually do thousands of rows (around 5k) per
> > transaction when I'm batch inserting, and I usually set it there so
> > I can watch its progress from another db session as the counts rise.
> >
>
> How do you set up the other session to watch?

Usually I just set up something to do throw an occasional query to the
server every so often (small script or something).  Something like a
select count(*) or select id order by id desc limit 1 type query.





^ permalink  raw  reply  [nested|flat] 4+ messages in thread


end of thread, other threads:[~2001-12-17 03:40 UTC | newest]

Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2001-11-30 10:23 Re: persistent connections, AOLserver (Was: maybe Offtopic : PostgreSQL & PHP ?) Steve Brett <steve.brett@e-mis.com>
2001-12-03 19:07 ` Re: persistent connections, AOLserver (Was: maybe Offtopic : Stephan Szabo <sszabo@megazone23.bigpanda.com>
2001-12-17 02:01   ` Re: persistent connections, AOLserver (Was: maybe Offtopic : Marc Spitzer <marc@oscar.eng.cv.net>
2001-12-17 03:40     ` Re: persistent connections, AOLserver (Was: maybe Offtopic : Stephan Szabo <sszabo@megazone23.bigpanda.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox