From: Justin Graf <justin@magwerks.com>
To: pgsql-sql@postgresql.org
Subject: Re: best paging strategies for large datasets?
Date: Wed, 12 May 2010 11:59:32 -0400
Message-ID: <4BEAD064.2020507@magwerks.com> (raw)
In-Reply-To: <20100512054144.GA8762@apartia.fr>
References: <20100512054144.GA8762@apartia.fr>
On 5/12/2010 1:41 AM, Louis-David Mitterrand wrote:
> Hi,
>
> I have a large dataset (page 1 at http://www.cruisefish.net/stat.md) and
> am in the process of developping a pager to let users leaf through it
> (30K rows).
>
That's not that big of a record set.
> Ideally I'd like to know when requesting any 'page' of data where I am
> within the dataset: how many pages are available each way, etc.
>
> Of course that can be done by doing a count(*) query before requesting a
> limit/offset subset. But the main query is already quite slow, so I'd
> like to minimize them.
>
What do you mean by quite slow??
On a 30K record table count() and query should speed should be a problem..
> But I am intrigued by window functions, especially the row_number() and
> ntile(int) ones.
>
> Adding "row_number() over (order by<reverse query>)" to my query will
> return the total number of rows in the first row, letting my deduce the
> number of pages remaining, etc. row_number() apparently adds very little
> cost to the main query.
>
That will get a sequential number, but you still don't know how many
records are in the table, limit and offset block that value.
I don't see how this helps?
Limit and Offset with Total Record count tell us where we are in the
record set and which page we are on.
RecordCount/Limit = Number of pages
CurrentPage = (offset%RecordCount)/Limit
to complicate things further what if the site allows user to change the
number of records displayed per page. The pager logic needs to figure
out how many records need to be return per page, and what the next and
previous iterations are. Without the total count records I don't see how
that is even possible.
I have written pagers in ASP and PHP
All legitimate Magwerks Corporation quotations are sent in a .PDF file attachment with a unique ID number generated by our proprietary quotation system. Quotations received via any other form of communication will not be honored.
CONFIDENTIALITY NOTICE: This e-mail, including attachments, may contain legally privileged, confidential or other information proprietary to Magwerks Corporation and is intended solely for the use of the individual to whom it addresses. If the reader of this e-mail is not the intended recipient or authorized agent, the reader is hereby notified that any unauthorized viewing, dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender by replying to this message and destroy all occurrences of this e-mail immediately.
Thank you.
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-sql@postgresql.org
Cc: justin@magwerks.com
Subject: Re: best paging strategies for large datasets?
In-Reply-To: <4BEAD064.2020507@magwerks.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox