From: Pierre C <lists@peufeu.com>
To: Eliot Gable <egable+pgsql-performance@gmail.com>
To: pgsql-performance@postgresql.org
Subject: Re: Replacing Cursors with Temporary Tables
Date: Sat, 24 Apr 2010 11:26:58 +0200
Message-ID: <op.vbntu8w9eorkce@immo.peufeu.com> (raw)
In-Reply-To: <v2ybf6923ed1004231931gc02a4368wa00bbc0fdf8d4543@mail.gmail.com>
References: <n2ybf6923ed1004211316ib844d82bn7a63bd3292ca80a0@mail.gmail.com>
<n2sb42b73151004220711mc2135db6heccf53ec1fb0136e@mail.gmail.com>
<q2yb42b73151004220742gbb7ae4adyc3781cedf98e8559@mail.gmail.com>
<m2gbf6923ed1004221357ua9ab54a3s92dfdbb59cfef832@mail.gmail.com>
<n2qbf6923ed1004231342l61f766fax8fb2a482aa3b0faf@mail.gmail.com>
<r2wb42b73151004231401r120510e8ne5ece4cdda768d5a@mail.gmail.com>
<x2kbf6923ed1004231739qe92d06c2o557d4e31e66ba3d7@mail.gmail.com>
<v2ybf6923ed1004231931gc02a4368wa00bbc0fdf8d4543@mail.gmail.com>
FYI, I had a query like this :
(complex search query ORDER BY foo LIMIT X)
LEFT JOIN objects_categories oc
LEFT JOIN categories c
GROUP BY ...
(more joins)
ORDER BY foo LIMIT X
Here, we do a search on "objects" (i'm not gonna give all the details,
they're not interesting for the problem at hand).
Point is that these objects can belong to several categories, so I need to
perform a GROUP BY with array_agg() somewhere unless I want the JOIN to
return several rows per object, which is not what I want. This makes the
query quite complicated...
I ended up rewriting it like this :
(complex search query ORDER BY foo LIMIT X)
LEFT JOIN
(SELECT .. FROM objects_categories oc
LEFT JOIN categories c
GROUP BY ...
) ON ...
(more joins)
ORDER BY foo LIMIT X
Basically moving the aggregates into a separate query. It is easier to
handle.
I tried to process it like this, in a stored proc :
- do the (complex search query ORDER BY foo LIMIT X) alone and stuff it in
a cursor
- extract the elements needed into arrays (mostly object_id)
- get the other information as separate queries like :
SELECT object_id, category_id, category_name
FROM objects_categories JOIN categories ON ...
WHERE object_id =ANY( my_array );
and return the results into cursors, too.
Or like this (using 2 cursors) :
SELECT object_id, array_agg(category_id) FROM objects_categories WHERE
object_id =ANY( my_array );
SELECT category_id, category_name, ...
FROM categories WHERE category_id IN (
SELECT category_id FROM objects_categories WHERE object_id =ANY( my_array
));
I found it to be quite faster, and it also simplifies my PHP code. From
PHP's point of view, it is simpler to get a cursor that returns the
objects, and separate cursors that can be used to build an in-memory PHP
hashtable of only the categories we're going to display. Also, it avoids
retrieving lots of data multiple times, since many objects will belong to
the same categories. With the second example, I can use my ORM to
instantiate only one copy of each.
It would be quite useful if we could SELECT from a cursor, or JOIN a
cursor to an existing table...
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: lists@peufeu.com, egable+pgsql-performance@gmail.com
Subject: Re: Replacing Cursors with Temporary Tables
In-Reply-To: <op.vbntu8w9eorkce@immo.peufeu.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox