pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  
help / color / mirror / Atom feed
Trying to create multi db query in one large queries
5+ messages / 5 participants
[nested] [flat]

* Trying to create multi db query in one large queries
@ 2004-12-14 01:44  Hasnul Fadhly bin Hasan <hasnulfadhly.h@mimos.my>
  0 siblings, 1 reply; 5+ messages in thread

From: Hasnul Fadhly bin Hasan @ 2004-12-14 01:44 UTC (permalink / raw)
  To: pgsql-performance

Hi,

I am not sure if this is the place to ask this question, but since the 
question is trying to improve the performance.. i guess i am not that 
far off.

My question is if there is a query design that would query multiple 
server simultaneously.. would that improve the performance?

To make it clear.. let's say we have 3 db servers.  1 server is just 
designed to take the queries while the other 2 server is the ones that 
actually
holds the data.  let's say we have a query of 'select * from 
customer_data' and we change it to
select * from
(
dblink('db1','select * from customer_data where timestamp between 
timestamp \'01-01-2004\' and timestamp \'06-30-2004\'')
union
dblink('db2','select * from customer_data where timestamp between 
timestamp \'01-07-2004\' and timestamp \'12-31-2004\'')
)

Would the subquery above be done simultaneously by postgres before doing 
the end query? or would it just execute one at a time?

If it does execute simultaneously.. it's possible to create code to 
convert normal queries to distributed queries and requesting data from 
multiple
database to improve performance.  This would be advantageous for large 
amount of data.

Thanks,

Hasnul





^ permalink  raw  reply  [nested|flat] 5+ messages in thread

* Re: Trying to create multi db query in one large querie
@ 2004-12-14 04:07  Spiegelberg, Greg <gspiegelberg@cranel.com>
  0 siblings, 1 reply; 5+ messages in thread

From: Spiegelberg, Greg @ 2004-12-14 04:07 UTC (permalink / raw)
  To: Hasnul Fadhly bin Hasan <hasnulfadhly.h@mimos.my>; pgsql-performance

Hello,

My experience with dblink() is that each dblink() is executed serially, in
part I would guess, due to the plan for the query.  To have each query run
in parallel you would need to execute both dblink()'s simultaneously saving
each result into a table.  I'm not sure if the same table could be
specified.  Would depend on the constaint's I suppose.

#!/bin/sh
# Query 1
psql -d mydb -c "select * into mytable from dblink('db1','select * from
customer_data where timestamp between timestamp \'01-01-2004\' and timestamp
\'06-30-2004\'') as t1(c1 int, c2 text, ...);" & PID1=$!
# Query 2
psql -d mydb -c "select * into mytable from dblink('db2','select * from
customer_data where timestamp between timestamp \'01-07-2004\' and timestamp
\'12-31-2004\'') as t2(c1 int, c2 text, ...);" & PID2=$!
# wait
wait $PID1
wait $PID2
# Do more on mydb.mytable
...

Something like that so no guaranties.  I do remember testing with this a
while back and it is useful for JOIN's.

Greg


-----Original Message-----
From: Hasnul Fadhly bin Hasan
To: pgsql-performance@postgresql.org
Sent: 12/13/04 8:44 PM
Subject: [PERFORM] Trying to create multi db query in one large queries

Hi,

I am not sure if this is the place to ask this question, but since the 
question is trying to improve the performance.. i guess i am not that 
far off.

My question is if there is a query design that would query multiple 
server simultaneously.. would that improve the performance?

To make it clear.. let's say we have 3 db servers.  1 server is just 
designed to take the queries while the other 2 server is the ones that 
actually
holds the data.  let's say we have a query of 'select * from 
customer_data' and we change it to
select * from
(
dblink('db1','select * from customer_data where timestamp between 
timestamp \'01-01-2004\' and timestamp \'06-30-2004\'')
union
dblink('db2','select * from customer_data where timestamp between 
timestamp \'01-07-2004\' and timestamp \'12-31-2004\'')
)

Would the subquery above be done simultaneously by postgres before doing

the end query? or would it just execute one at a time?

If it does execute simultaneously.. it's possible to create code to 
convert normal queries to distributed queries and requesting data from 
multiple
database to improve performance.  This would be advantageous for large 
amount of data.

Thanks,

Hasnul



---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster



^ permalink  raw  reply  [nested|flat] 5+ messages in thread

* Re: Trying to create multi db query in one large querie
@ 2004-12-14 06:59  Joe Conway <mail@joeconway.com>
  parent: Spiegelberg, Greg <gspiegelberg@cranel.com>
  0 siblings, 0 replies; 5+ messages in thread

From: Joe Conway @ 2004-12-14 06:59 UTC (permalink / raw)
  To: Spiegelberg, Greg <gspiegelberg@cranel.com>; +Cc: Hasnul Fadhly bin Hasan <hasnulfadhly.h@mimos.my>; pgsql-performance

Spiegelberg, Greg wrote:
> 
> My experience with dblink() is that each dblink() is executed serially

Correct.

If you really want to do multiple queries simultaneously, you would need 
to write a function very similar to dblink_record, but using asynchonous 
libpq calls to both remote hosts. See:
   http://www.postgresql.org/docs/current/static/libpq-async.html

HTH,

Joe



^ permalink  raw  reply  [nested|flat] 5+ messages in thread

* Re: Trying to create multi db query in one large queries
@ 2004-12-14 07:18  Josh Berkus <josh@agliodbs.com>
  parent: Hasnul Fadhly bin Hasan <hasnulfadhly.h@mimos.my>
  0 siblings, 1 reply; 5+ messages in thread

From: Josh Berkus @ 2004-12-14 07:18 UTC (permalink / raw)
  To: pgsql-performance

Hasnul,

> My question is if there is a query design that would query multiple
> server simultaneously.. would that improve the performance?

Not without a vast amounts of infrastructure coding.   You're basically 
talking about what Oracle has spent the last 3 years and $100 million working 
on.

Would be nice, though.

-- 
Josh Berkus
Aglio Database Solutions
San Francisco



^ permalink  raw  reply  [nested|flat] 5+ messages in thread

* Re: Trying to create multi db query in one large queries
@ 2004-12-16 04:22  Christopher Browne <cbbrowne@acm.org>
  parent: Josh Berkus <josh@agliodbs.com>
  0 siblings, 0 replies; 5+ messages in thread

From: Christopher Browne @ 2004-12-16 04:22 UTC (permalink / raw)
  To: pgsql-performance

The world rejoiced as josh@agliodbs.com (Josh Berkus) wrote:
> Hasnul,
>
>> My question is if there is a query design that would query multiple
>> server simultaneously.. would that improve the performance?
>
> Not without a vast amounts of infrastructure coding.  You're
> basically talking about what Oracle has spent the last 3 years and
> $100 million working on.

I recall a presentation from folks from Empress Software
<http://www.empress.com/; back in about '94 or '95 offering this very
feature as part of the "base functionality" of their product.

I'm not sure it's quite fair to assess things as "more or less
preposterous" simply because they prove ludicrously expensive to
develop on a particular platform that happens to be targeted by even
more ludicrous quantities of development dollars...

On the other hand, it seems unlikely that "improved performance" would
be one of the merits of this approach...
-- 
let name="cbbrowne" and tld="gmail.com" in String.concat "@" [name;tld];;
http://www3.sympatico.ca/cbbrowne/
Rules of the  Evil Overlord #92.  "If I  ever talk to  the hero on the
phone, I  will  not taunt him.  Instead  I  will  say that  his dogged
perseverance has given me new insight on the futility  of my evil ways
and that if he leaves me alone for a few months of quiet contemplation
I will   likely  return to  the  path  of righteousness.   (Heroes are
incredibly gullible in this regard.) <http://www.eviloverlord.com/;



^ permalink  raw  reply  [nested|flat] 5+ messages in thread


end of thread, other threads:[~2004-12-16 04:22 UTC | newest]

Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2004-12-14 01:44 Trying to create multi db query in one large queries Hasnul Fadhly bin Hasan <hasnulfadhly.h@mimos.my>
2004-12-14 07:18 ` Josh Berkus <josh@agliodbs.com>
2004-12-16 04:22   ` Christopher Browne <cbbrowne@acm.org>
2004-12-14 04:07 Re: Trying to create multi db query in one large querie Spiegelberg, Greg <gspiegelberg@cranel.com>
2004-12-14 06:59 ` Re: Trying to create multi db query in one large querie Joe Conway <mail@joeconway.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox