Received: from makus.postgresql.org (makus.postgresql.org [98.129.198.125])
	by mail.postgresql.org (Postfix) with ESMTP id 609F91F02EEF
	for <pgsql-performance@postgresql.org>;
	Thu, 26 Apr 2012 03:49:38 -0300 (ADT)
Received: from mail-ey0-f174.google.com ([209.85.215.174])
	by makus.postgresql.org with esmtp (Exim 4.72)
	(envelope-from <yebhavinga@gmail.com>) id 1SNIWG-0006z2-UY
	for pgsql-performance@postgresql.org; Thu, 26 Apr 2012 06:49:37 +0000
Received: by eaaq12 with SMTP id q12so213105eaa.19
	for <pgsql-performance@postgresql.org>;
	Wed, 25 Apr 2012 23:49:23 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
	h=message-id:date:from:user-agent:mime-version:to:cc:subject
	:references:in-reply-to:content-type;
	bh=ZWRIeOVmciRjzU4TgeSlJthdC296Kn6evybbHigM/dI=;
	b=EenxfITdJ+flrMXPDa7LNmc0zdknvtmKwW3iT3xb87eBo3cWvuImDTrXuio/73vQiF
	SUkj3XGh0WIHA4b00ifUz3e0kNepaLvZ61KqSxy610hBHyo9XNpdnIkvMeGX3jywUXz8
	pqbxUz4yPYmTNYeunXEHGTgJ1j66KRGp4gtm9JkZMT0w+XTKWyzMNu+iEGHHwnP7irQM
	zEauJQyWSzjAn71CDoRTVAwoDD9ZXriIKmVBYm4QHTzdhbXZzUYjiWPlQdj6Fm78pTwS
	uHL6/AhSyqHthFArMISuE+5HGdVSEijnh2BPBAMvEH/8q+P7CQcJ19eEFWEYNRDbIQPp
	9akQ==
Received: by 10.14.98.72 with SMTP id u48mr1041107eef.81.1335422963671;
	Wed, 25 Apr 2012 23:49:23 -0700 (PDT)
Received: from [192.168.73.63] (5357DA75.cm-6-8d.dynamic.ziggo.nl.
	[83.87.218.117])
	by mx.google.com with ESMTPS id z47sm9710127een.5.2012.04.25.23.49.21
	(version=TLSv1/SSLv3 cipher=OTHER);
	Wed, 25 Apr 2012 23:49:22 -0700 (PDT)
Message-ID: <4F98EFE8.5090609@gmail.com>
Date: Thu, 26 Apr 2012 08:49:12 +0200
From: Yeb Havinga <yebhavinga@gmail.com>
User-Agent: Mozilla/5.0 (Windows NT 5.1;
	rv:11.0) Gecko/20120327 Thunderbird/11.0.1
MIME-Version: 1.0
To: Venki Ramachandran <venki_ramachandran@yahoo.com>
CC: Tom Lane <tgl@sss.pgh.pa.us>, Pavel Stehule <pavel.stehule@gmail.com>,
	Samuel Gendler <sgendler@ideasculptor.com>,
	"pgsql-performance@postgresql.org" <pgsql-performance@postgresql.org>
Subject: Re: Parallel Scaling of a pgplsql problem
References: <1335379923.47084.YahooMailNeo@web184514.mail.ne1.yahoo.com>
	<CAEV0TzAVQXf42n9vXu26JAe0-eQ9AgwwAErYFn0HMm8ScQaodA@mail.gmail.com>
	<1335388323.84458.YahooMailNeo@web184517.mail.ne1.yahoo.com>
	<CAFj8pRB7gvUocfW5Uv5-NVFtRPSE97AS6Sbk5gzrdMweCpNS_Q@mail.gmail.com>
	<1335390338.10325.YahooMailNeo@web184505.mail.ne1.yahoo.com>
	<5452.1335390728@sss.pgh.pa.us>
	<1335408018.29771.YahooMailNeo@web184514.mail.ne1.yahoo.com>
In-Reply-To: <1335408018.29771.YahooMailNeo@web184514.mail.ne1.yahoo.com>
Content-Type: multipart/alternative;
	boundary="------------010906040306070907060302"
X-Pg-Spam-Score: -2.6 (--)
X-Archive-Number: 201204/156
X-Sequence-Number: 46802

This is a multi-part message in MIME format.
--------------010906040306070907060302
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 2012-04-26 04:40, Venki Ramachandran wrote:
> Thanks Tom, clock_timestamp() worked. Appreciate it!!! and Sorry was 
> hurrying to get this done at work and hence did not read through.
>
> Can you comment on how you would solve the original problem? Even if I 
> can get  the 11 seconds down to 500 ms for one pair, running it for 
> 300k pairs will take multiple hours. How can one write a combination 
> of a bash script/pgplsql code so as to use all 8 cores of a server. I 
> am seeing that this is just executing in one session/process.

You want to compare a calculation on the cross product 'employee x 
employee'. If employee is partitioned into emp1, emp2, ... emp8, the 
cross product is equal to the union of emp1 x employee, emp2 x employee, 
.. emp8 x employee. Each of these 8 cross products on partitions can be 
executed in parallel. I'd look into dblink to execute each of the 8 
cross products in parallel, and then union all of those results.

http://www.postgresql.org/docs/9.1/static/contrib-dblink-connect.html

regards,
Yeb


--------------010906040306070907060302
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    On 2012-04-26 04:40, Venki Ramachandran wrote:
    <blockquote
      cite="mid:1335408018.29771.YahooMailNeo@web184514.mail.ne1.yahoo.com"
      type="cite">
      <div style="color: rgb(0, 0, 0); background-color: rgb(255, 255,
        255); font-family: tahoma,new york,times,serif; font-size:
        10pt;">
        <div><span>Thanks Tom, clock_timestamp() worked. Appreciate
            it!!! and Sorry was hurrying to get this done at work and
            hence did not read through.</span></div>
        <div><span><br>
          </span></div>
        <div><span>Can you comment on how you would solve the original
            problem? Even if I can get &nbsp;the 11 seconds down to 500 ms
            for one pair, running it for 300k pairs will take multiple
            hours. How can one write a combination of a bash
            script/pgplsql code so as to use all 8 cores of a server. I
            am seeing that this is just executing in one
            session/process.</span></div>
      </div>
    </blockquote>
    <br>
    You want to compare a calculation on the cross product 'employee x
    employee'. If employee is partitioned into emp1, emp2, ... emp8, the
    cross product is equal to the union of emp1 x employee, emp2 x
    employee, .. emp8 x employee. Each of these 8 cross products on
    partitions can be executed in parallel. I'd look into dblink to
    execute each of the 8 cross products in parallel, and then union all
    of those results.<br>
    <br>
<a class="moz-txt-link-freetext" href="http://www.postgresql.org/docs/9.1/static/contrib-dblink-connect.html">http://www.postgresql.org/docs/9.1/static/contrib-dblink-connect.html</a><br>
    <br>
    regards,<br>
    Yeb<br>
    <br>
  </body>
</html>

--------------010906040306070907060302--