Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes
To: "Eliot Gable" <egable+pgsql-performance@gmail.com>,
	pgsql-performance@postgresql.org
Subject: Re: Replacing Cursors with Temporary Tables
References: <n2ybf6923ed1004211316ib844d82bn7a63bd3292ca80a0@mail.gmail.com>
	<n2sb42b73151004220711mc2135db6heccf53ec1fb0136e@mail.gmail.com>
	<q2yb42b73151004220742gbb7ae4adyc3781cedf98e8559@mail.gmail.com>
	<m2gbf6923ed1004221357ua9ab54a3s92dfdbb59cfef832@mail.gmail.com>
	<n2qbf6923ed1004231342l61f766fax8fb2a482aa3b0faf@mail.gmail.com>
	<r2wb42b73151004231401r120510e8ne5ece4cdda768d5a@mail.gmail.com>
	<x2kbf6923ed1004231739qe92d06c2o557d4e31e66ba3d7@mail.gmail.com>
	<v2ybf6923ed1004231931gc02a4368wa00bbc0fdf8d4543@mail.gmail.com>
Date: Sat, 24 Apr 2010 11:26:58 +0200
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
From: "Pierre C" <lists@peufeu.com>
Message-ID: <op.vbntu8w9eorkce@immo.peufeu.com>
In-Reply-To: <v2ybf6923ed1004231931gc02a4368wa00bbc0fdf8d4543@mail.gmail.com>
User-Agent: Opera Mail/10.10 (Linux)


FYI, I had a query like this :

(complex search query ORDER BY foo LIMIT X)
LEFT JOIN objects_categories oc
LEFT JOIN categories c
GROUP BY ...
(more joins)
ORDER BY foo LIMIT X

Here, we do a search on "objects" (i'm not gonna give all the details,  
they're not interesting for the problem at hand).
Point is that these objects can belong to several categories, so I need to  
perform a GROUP BY with array_agg() somewhere unless I want the JOIN to  
return several rows per object, which is not what I want. This makes the  
query quite complicated...

I ended up rewriting it like this :

(complex search query ORDER BY foo LIMIT X)
LEFT JOIN
(SELECT .. FROM objects_categories oc
  LEFT JOIN categories c
  GROUP BY ...
) ON ...
(more joins)
ORDER BY foo LIMIT X

Basically moving the aggregates into a separate query. It is easier to  
handle.

I tried to process it like this, in a stored proc :

- do the (complex search query ORDER BY foo LIMIT X) alone and stuff it in  
a cursor
- extract the elements needed into arrays (mostly object_id)
- get the other information as separate queries like :

SELECT object_id, category_id, category_name
 FROM objects_categories JOIN categories ON ...
WHERE object_id =ANY( my_array );

and return the results into cursors, too.

Or like this (using 2 cursors) :

SELECT object_id, array_agg(category_id) FROM objects_categories WHERE  
object_id =ANY( my_array );

SELECT category_id, category_name, ...
 FROM categories WHERE category_id IN (
  SELECT category_id FROM objects_categories WHERE object_id =ANY( my_array  
));

I found it to be quite faster, and it also simplifies my PHP code. From  
PHP's point of view, it is simpler to get a cursor that returns the  
objects, and separate cursors that can be used to build an in-memory PHP  
hashtable of only the categories we're going to display. Also, it avoids  
retrieving lots of data multiple times, since many objects will belong to  
the same categories. With the second example, I can use my ORM to  
instantiate only one copy of each.

It would be quite useful if we could SELECT from a cursor, or JOIN a  
cursor to an existing table...