From cestmirl@freeside.sk Fri Oct 7 09:24:18 2005 X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (av.hub.org [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 2894CDA48E for ; Fri, 7 Oct 2005 06:24:18 -0300 (ADT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 86454-07 for ; Fri, 7 Oct 2005 09:24:13 +0000 (GMT) Received: from ns2.dialtelecom.sk (ns2.dialtelecom.sk [217.67.16.3]) by svr1.postgresql.org (Postfix) with ESMTP id E6289D9001 for ; Fri, 7 Oct 2005 06:24:11 -0300 (ADT) Received: (qmail 2406 invoked by uid 0); 7 Oct 2005 09:23:48 -0000 Received: from unknown (HELO omikron.sk) (217.67.24.100) by ns2.dialtelecom.sk with SMTP; 7 Oct 2005 09:23:45 -0000 Received: (qmail 15740 invoked from network); 7 Oct 2005 09:23:45 -0000 Received: from unknown (HELO stratos) (158.195.31.19) by 0 with SMTP; 7 Oct 2005 09:23:45 -0000 Message-ID: <935801c5cb20$dcf93a00$131fc39e@stratos> From: "Cestmir Hybl" To: Subject: count(*) using index scan in "query often, update rarely" environment Date: Fri, 7 Oct 2005 11:24:05 +0200 Organization: NUSTEP s.r.o. MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_9355_01C5CB31.A0496DD0" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.062 required=5 tests=[AWL=-0.034, HTML_50_60=0.095, HTML_MESSAGE=0.001] X-Spam-Level: X-Archive-Number: 200510/134 X-Sequence-Number: 14902 This is a multi-part message in MIME format. ------=_NextPart_000_9355_01C5CB31.A0496DD0 Content-Type: text/plain; charset="iso-8859-2" Content-Transfer-Encoding: quoted-printable Hello all First of all, I do understand why pgsql with it's MVCC design has to = examine tuples to evaluate "count(*)" and "count(*) where (...)" queries = in environment with heavy concurrent updates. This kind of usage IMHO isn't the average one. There are many = circumstances with rather "query often, update rarely" character. Isn't it possible (and reasonable) for these environments to keep track = of whether there is a transaction in progress with update to given table = and if not, use an index scan (count(*) where) or cached value = (count(*)) to perform this kind of query? (sorry for disturbing if this was already discussed) Regards, Cestmir Hybl ------=_NextPart_000_9355_01C5CB31.A0496DD0 Content-Type: text/html; charset="iso-8859-2" Content-Transfer-Encoding: quoted-printable
Hello all
 
First of all, I do understand why pgsql = with it's=20 MVCC design has to examine tuples to evaluate "count(*)" and = "count(*)=20 where (...)" queries in environment with heavy concurrent = updates.
 
This kind of usage IMHO isn't the = average one.=20 There are many circumstances with rather "query often, update=20 rarely" character.
 
Isn't it possible (and reasonable) for = these=20 environments to keep track of whether there is a transaction in progress = with=20 update to given table and if not, use an index scan (count(*) where) or = cached=20 value (count(*)) to perform this kind of query?
 
(sorry for disturbing if this was = already=20 discussed)
 
Regards,
 
Cestmir Hybl
------=_NextPart_000_9355_01C5CB31.A0496DD0-- From depesz@gmail.com Fri Oct 7 09:54:28 2005 X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (av.hub.org [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id C012EDA53B for ; Fri, 7 Oct 2005 06:54:28 -0300 (ADT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 45227-10 for ; Fri, 7 Oct 2005 09:54:20 +0000 (GMT) Received: from qproxy.gmail.com (qproxy.gmail.com [72.14.204.202]) by svr1.postgresql.org (Postfix) with ESMTP id B4559DA521 for ; Fri, 7 Oct 2005 06:54:20 -0300 (ADT) Received: by qproxy.gmail.com with SMTP id z1so493320qbc for ; Fri, 07 Oct 2005 02:54:23 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:references; b=f9Ud/9vObec/4Xia9lDNm6b0ruhD6PZs7JD/zlvLtaR5QEOuVC0FL6lPBt7fAfYWtxu0Wzl6QjZCQpqhzoBP0hYmSLDn0XJW+5rAi7u7bXG6aicnh7MXwXiBy4JiFAjSbBLgnE4J0+pSkNCM6Uz+HrcFpYAXYkcf6c5ljaQZEfE= Received: by 10.64.209.15 with SMTP id h15mr241562qbg; Fri, 07 Oct 2005 02:54:23 -0700 (PDT) Received: by 10.64.195.4 with HTTP; Fri, 7 Oct 2005 02:54:23 -0700 (PDT) Message-ID: <9e4684ce0510070254v67e08bcfhb725715803a5de25@mail.gmail.com> Date: Fri, 7 Oct 2005 11:54:23 +0200 From: hubert depesz lubaczewski Reply-To: hubert depesz lubaczewski To: Cestmir Hybl Subject: Re: count(*) using index scan in "query often, update rarely" environment Cc: pgsql-performance@postgresql.org In-Reply-To: <935801c5cb20$dcf93a00$131fc39e@stratos> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_10913_26676785.1128678863681" References: <935801c5cb20$dcf93a00$131fc39e@stratos> X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=1.144 required=5 tests=[AWL=1.033, HTML_40_50=0.086, HTML_MESSAGE=0.001, RCVD_BY_IP=0.024] X-Spam-Level: * X-Archive-Number: 200510/135 X-Sequence-Number: 14903 ------=_Part_10913_26676785.1128678863681 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On 10/7/05, Cestmir Hybl wrote: > > Isn't it possible (and reasonable) for these environments to keep track o= f > whether there is a transaction in progress with update to given table and= if > not, use an index scan (count(*) where) or cached value (count(*)) to > perform this kind of query? > if i understand your problem correctly, then simple usage of triggers will do the job just fine. hubert ------=_Part_10913_26676785.1128678863681 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On 10/7/05, Cestmir Hybl <cestmirl@freeside.sk> wrote:
Isn't it possible (and reasonable) for= these=20 environments to keep track of whether there is a transaction in progress wi= th=20 update to given table and if not, use an index scan (count(*) where) or cac= hed=20 value (count(*)) to perform this kind of query?

if i understand your problem correctly, then simple usage of triggers will = do the job just fine.

hubert
------=_Part_10913_26676785.1128678863681-- From cestmirl@freeside.sk Fri Oct 7 10:14:17 2005 X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (av.hub.org [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 9A452D9DBF for ; Fri, 7 Oct 2005 07:14:17 -0300 (ADT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 19807-04 for ; Fri, 7 Oct 2005 10:14:12 +0000 (GMT) Received: from ns2.dialtelecom.sk (ns2.dialtelecom.sk [217.67.16.3]) by svr1.postgresql.org (Postfix) with ESMTP id A0151DA512 for ; Fri, 7 Oct 2005 07:14:07 -0300 (ADT) Received: (qmail 4334 invoked by uid 0); 7 Oct 2005 10:14:09 -0000 Received: from unknown (HELO omikron.sk) (217.67.24.100) by ns2.dialtelecom.sk with SMTP; 7 Oct 2005 10:14:09 -0000 Received: (qmail 27551 invoked from network); 7 Oct 2005 10:14:09 -0000 Received: from unknown (HELO stratos) (158.195.31.19) by 0 with SMTP; 7 Oct 2005 10:14:09 -0000 Message-ID: <9d5b01c5cb27$e7a1c970$131fc39e@stratos> From: "Cestmir Hybl" To: "hubert depesz lubaczewski" Cc: References: <935801c5cb20$dcf93a00$131fc39e@stratos> <9e4684ce0510070254v67e08bcfhb725715803a5de25@mail.gmail.com> Subject: Re: count(*) using index scan in "query often, update rarely" environment Date: Fri, 7 Oct 2005 12:14:29 +0200 Organization: NUSTEP s.r.o. MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_9D58_01C5CB38.AAF817C0" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.042 required=5 tests=[AWL=0.014, HTML_60_70=0.027, HTML_MESSAGE=0.001] X-Spam-Level: X-Archive-Number: 200510/136 X-Sequence-Number: 14904 This is a multi-part message in MIME format. ------=_NextPart_000_9D58_01C5CB38.AAF817C0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Yes, I can possibly use triggers to maintanin counts of several fixed = groups of records or total recordcount (but it's unpractical). No, I can't speed-up evaluation of generic "count(*) where ()" queries = this way. My question was rather about general performance of count() queries in = environment with infrequent updates. Cestmir ----- Original Message -----=20 From: hubert depesz lubaczewski=20 To: Cestmir Hybl=20 Cc: pgsql-performance@postgresql.org=20 Sent: Friday, October 07, 2005 11:54 AM Subject: Re: [PERFORM] count(*) using index scan in "query often, = update rarely" environment On 10/7/05, Cestmir Hybl wrote: Isn't it possible (and reasonable) for these environments to keep = track of whether there is a transaction in progress with update to given = table and if not, use an index scan (count(*) where) or cached value = (count(*)) to perform this kind of query? if i understand your problem correctly, then simple usage of triggers = will do the job just fine. hubert ------=_NextPart_000_9D58_01C5CB38.AAF817C0 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Yes, I can possibly use triggers to = maintanin=20 counts of several fixed groups of records or total recordcount (but it's = unpractical).
 
No, I can't speed-up evaluation of = generic=20 "count(*) where ()" queries this way.
 
My question was rather about general = performance of=20 count() queries in environment with infrequent updates.
 
Cestmir
----- Original Message -----
From:=20 hubert = depesz=20 lubaczewski
Cc: pgsql-performance@postgr= esql.org=20
Sent: Friday, October 07, 2005 = 11:54=20 AM
Subject: Re: [PERFORM] count(*) = using=20 index scan in "query often, update rarely" environment

On 10/7/05, Cestmir = Hybl <cestmirl@freeside.sk> = wrote:
Isn't it possible (and reasonable) = for these=20 environments to keep track of whether there is a transaction in = progress=20 with update to given table and if not, use an index scan (count(*) = where) or=20 cached value (count(*)) to perform this kind of query?

if i understand your problem = correctly, then=20 simple usage of triggers will do the job just=20 fine.

hubert
------=_NextPart_000_9D58_01C5CB38.AAF817C0-- From sgunderson@bigfoot.com Fri Oct 7 10:48:23 2005 X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (av.hub.org [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 9F77DD9AB9 for ; Fri, 7 Oct 2005 07:48:23 -0300 (ADT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 96807-09 for ; Fri, 7 Oct 2005 10:48:17 +0000 (GMT) Received: from cassarossa.samfundet.no (cassarossa.samfundet.no [129.241.93.19]) by svr1.postgresql.org (Postfix) with ESMTP id A7594D9001 for ; Fri, 7 Oct 2005 07:48:17 -0300 (ADT) Received: from trofast.sesse.net ([129.241.93.32]) by cassarossa.samfundet.no with esmtp (Exim 4.50) id 1ENply-0004nP-CK for pgsql-performance@postgresql.org; Fri, 07 Oct 2005 12:48:19 +0200 Received: from sesse by trofast.sesse.net with local (Exim 3.36 #1 (Debian)) id 1ENplw-0002QF-00 for ; Fri, 07 Oct 2005 12:48:16 +0200 Date: Fri, 7 Oct 2005 12:48:16 +0200 From: "Steinar H. Gunderson" To: pgsql-performance@postgresql.org Subject: Re: count(*) using index scan in "query often, update rarely" environment Message-ID: <20051007104816.GA8371@uio.no> Mail-Followup-To: pgsql-performance@postgresql.org References: <935801c5cb20$dcf93a00$131fc39e@stratos> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <935801c5cb20$dcf93a00$131fc39e@stratos> X-Operating-System: Linux 2.6.11.8 on a i686 X-Message-Flag: Outlook? --> http://www.mozilla.org/products/thunderbird/ User-Agent: Mutt/1.5.10i X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.012 required=5 tests=[AWL=0.012] X-Spam-Level: X-Archive-Number: 200510/137 X-Sequence-Number: 14905 On Fri, Oct 07, 2005 at 11:24:05AM +0200, Cestmir Hybl wrote: > Isn't it possible (and reasonable) for these environments to keep track of > whether there is a transaction in progress with update to given table and > if not, use an index scan (count(*) where) or cached value (count(*)) to > perform this kind of query? Even if there is no running update, there might still be dead rows in the table. In any case, of course, a new update could always be occurring while your counting query was still running. /* Steinar */ -- Homepage: http://www.sesse.net/ From cestmirl@freeside.sk Fri Oct 7 11:14:05 2005 X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (av.hub.org [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 24D6FDA552 for ; Fri, 7 Oct 2005 08:14:05 -0300 (ADT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 07543-02 for ; Fri, 7 Oct 2005 11:13:57 +0000 (GMT) Received: from ns2.dialtelecom.sk (ns2.dialtelecom.sk [217.67.16.3]) by svr1.postgresql.org (Postfix) with ESMTP id B8BCADA52C for ; Fri, 7 Oct 2005 08:13:57 -0300 (ADT) Received: (qmail 7004 invoked by uid 0); 7 Oct 2005 11:14:00 -0000 Received: from unknown (HELO omikron.sk) (217.67.24.100) by ns2.dialtelecom.sk with SMTP; 7 Oct 2005 11:14:00 -0000 Received: (qmail 7604 invoked from network); 7 Oct 2005 11:14:00 -0000 Received: from unknown (HELO stratos) (158.195.31.19) by 0 with SMTP; 7 Oct 2005 11:14:00 -0000 Message-ID: <9fbe01c5cb30$4407d580$131fc39e@stratos> From: "Cestmir Hybl" To: "Steinar H. Gunderson" , References: <935801c5cb20$dcf93a00$131fc39e@stratos> <20051007104816.GA8371@uio.no> Subject: Re: count(*) using index scan in "query often, update rarely" environment Date: Fri, 7 Oct 2005 13:14:20 +0200 Organization: NUSTEP s.r.o. MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="utf-8"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.024 required=5 tests=[AWL=0.024] X-Spam-Level: X-Archive-Number: 200510/138 X-Sequence-Number: 14906 collision: it's possible to either block updating transaction until index scan ends or discard index scan imediately and finish query using MVCC compliant scan dead rows: this sounds like more serious counter-argument, I don't know much about dead records management and whether it would be possible/worth to make indexes matching live records when there's no transaction in progress on that table ----- Original Message ----- From: "Steinar H. Gunderson" To: Sent: Friday, October 07, 2005 12:48 PM Subject: Re: [PERFORM] count(*) using index scan in "query often, update rarely" environment > On Fri, Oct 07, 2005 at 11:24:05AM +0200, Cestmir Hybl wrote: >> Isn't it possible (and reasonable) for these environments to keep track >> of >> whether there is a transaction in progress with update to given table and >> if not, use an index scan (count(*) where) or cached value (count(*)) to >> perform this kind of query? > > Even if there is no running update, there might still be dead rows in the > table. In any case, of course, a new update could always be occurring > while > your counting query was still running. > > /* Steinar */ > -- > Homepage: http://www.sesse.net/ > > ---------------------------(end of broadcast)--------------------------- > TIP 2: Don't 'kill -9' the postmaster From alvherre@alvh.no-ip.org Fri Oct 7 13:07:18 2005 X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (av.hub.org [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id D654FDA50B for ; Fri, 7 Oct 2005 10:07:18 -0300 (ADT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 01920-07 for ; Fri, 7 Oct 2005 13:07:10 +0000 (GMT) Received: from mx1.surnet.cl (mx2.surnet.cl [216.155.73.181]) by svr1.postgresql.org (Postfix) with ESMTP id F07D7DA56C for ; Fri, 7 Oct 2005 10:07:11 -0300 (ADT) Received: from unknown (HELO cluster.surnet.cl) ([216.155.73.165]) by mx1.surnet.cl with ESMTP; 07 Oct 2005 09:06:46 -0400 Received: from alvh.no-ip.org (200.85.218.198) by cluster.surnet.cl (7.0.043) (authenticated as alvherre@surnet.cl) id 433A601F002215BB; Fri, 7 Oct 2005 09:07:15 -0400 Received: by alvh.no-ip.org (Postfix, from userid 1000) id 1D5E0C2DC18; Fri, 7 Oct 2005 09:07:15 -0400 (CLT) Date: Fri, 7 Oct 2005 09:07:15 -0400 From: Alvaro Herrera To: Cestmir Hybl Cc: "Steinar H. Gunderson" , pgsql-performance@postgresql.org Subject: Re: count(*) using index scan in "query often, update rarely" environment Message-ID: <20051007130714.GB5175@surnet.cl> Mail-Followup-To: Cestmir Hybl , "Steinar H. Gunderson" , pgsql-performance@postgresql.org References: <935801c5cb20$dcf93a00$131fc39e@stratos> <20051007104816.GA8371@uio.no> <9fbe01c5cb30$4407d580$131fc39e@stratos> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <9fbe01c5cb30$4407d580$131fc39e@stratos> User-Agent: Mutt/1.5.10i X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=1.495 required=5 tests=[AWL=-0.305, DNS_FROM_RFC_ABUSE=0.374, DNS_FROM_RFC_POST=1.376, FORGED_RCVD_HELO=0.05] X-Spam-Level: * X-Archive-Number: 200510/140 X-Sequence-Number: 14908 On Fri, Oct 07, 2005 at 01:14:20PM +0200, Cestmir Hybl wrote: > collision: it's possible to either block updating transaction until > index scan ends or discard index scan imediately and finish query using > MVCC compliant scan You can't change from one scan method to a different one on the fly. There's no way to know which tuples have alreaady been returned. Our index access methods are designed to be very concurrent, and it works extremely well. One index scan being able to block an update would destroy that advantage. > dead rows: this sounds like more serious counter-argument, I don't know > much about dead records management and whether it would be > possible/worth to make indexes matching live records when there's no > transaction in progress on that table It's not possible, because a finishing transaction would have to clean up every index it has used, and also any index it hasn't used but has been modified by another transaction which couldn't clean up by itself but didn't do the work because the first one was looking at the index. It's easy to see that it's possible to create an unbounded number of transactions, each forcing the other to do some index cleanup. This is not acceptable. Plus, it would be very hard to implement, and a very wide door to bugs. -- Alvaro Herrera http://www.advogato.org/person/alvherre "Et put se mouve" (Galileo Galilei) From tgl@sss.pgh.pa.us Fri Oct 7 13:42:31 2005 X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (av.hub.org [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 2C734DA508 for ; Fri, 7 Oct 2005 10:42:31 -0300 (ADT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 03717-09 for ; Fri, 7 Oct 2005 13:42:27 +0000 (GMT) Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) by svr1.postgresql.org (Postfix) with ESMTP id 3127FDA56E for ; Fri, 7 Oct 2005 10:42:29 -0300 (ADT) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id j97DgWXt003255; Fri, 7 Oct 2005 09:42:32 -0400 (EDT) To: "Cestmir Hybl" Cc: pgsql-performance@postgresql.org Subject: Re: count(*) using index scan in "query often, update rarely" environment In-reply-to: <935801c5cb20$dcf93a00$131fc39e@stratos> References: <935801c5cb20$dcf93a00$131fc39e@stratos> Comments: In-reply-to "Cestmir Hybl" message dated "Fri, 07 Oct 2005 11:24:05 +0200" Date: Fri, 07 Oct 2005 09:42:32 -0400 Message-ID: <3254.1128692552@sss.pgh.pa.us> From: Tom Lane X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.006 required=5 tests=[AWL=0.006] X-Spam-Level: X-Archive-Number: 200510/142 X-Sequence-Number: 14910 "Cestmir Hybl" writes: > Isn't it possible (and reasonable) for these environments to keep track = > of whether there is a transaction in progress with update to given table = > and if not, use an index scan (count(*) where) or cached value = > (count(*)) to perform this kind of query? Please read the archives before bringing up such well-discussed issues. There's a workable-looking design in the archives (pghackers probably) for maintaining overall table counts in a separate table, with each transaction adding one row of "delta" information just before it commits. I haven't seen anything else that looks remotely attractive. regards, tom lane From merlin.moncure@rcsonline.com Fri Oct 7 13:50:30 2005 X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (av.hub.org [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 4413ADA580 for ; Fri, 7 Oct 2005 10:50:30 -0300 (ADT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 54950-05 for ; Fri, 7 Oct 2005 13:50:25 +0000 (GMT) Received: from Herge.rcsinc.local (mail.rcsonline.com [70.89.208.142]) by svr1.postgresql.org (Postfix) with ESMTP id 3FF1FDA586 for ; Fri, 7 Oct 2005 10:50:26 -0300 (ADT) Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: Re: count(*) using index scan in "query often, update rarely" environment Date: Fri, 7 Oct 2005 09:50:30 -0400 X-MimeOLE: Produced By Microsoft Exchange V6.5.7226.0 Message-ID: <6EE64EF3AB31D5448D0007DD34EEB3417DD541@Herge.rcsinc.local> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [PERFORM] count(*) using index scan in "query often, update rarely" environment Thread-Index: AcXLJX6DK1dmqs+BSDmqVVB22ekbRAAHXLuQ From: "Merlin Moncure" To: "hubert depesz lubaczewski" Cc: X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.04 required=5 tests=[AWL=-0.010, FORGED_RCVD_HELO=0.05] X-Spam-Level: X-Archive-Number: 200510/143 X-Sequence-Number: 14911 On 10/7/05, Cestmir Hybl wrote: Isn't it possible (and reasonable) for these environments to keep track of whether there is a transaction in progress with update to given table and if not, use an index scan (count(*) where) or cached value (count(*)) to perform this kind of query? ________________________________________ The answer to the first question is subtle. Basically, the PostgreSQL engine is designed for high concurrency. We are definitely on the right side of the cost/benefit tradeoff here. SQL server does not have MVCC (or at least until 2005 appears) so they are on the other side of the tradeoff. You can of course serialize the access yourself by materializing the count in a small table and use triggers or cleverly designed transactions. This is trickier than it might look however so check the archives for a thorough treatment of the topic. One interesting thing is that making count(*) over large swaths of data is frequently an indicator of a poorly normalized database. Is it possible to optimize the counting by laying out your data in a different way? Merlin From dev@archonet.com Fri Oct 7 14:15:23 2005 X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (av.hub.org [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 8A82FDA57A for ; Fri, 7 Oct 2005 11:15:23 -0300 (ADT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 52220-03 for ; Fri, 7 Oct 2005 14:15:16 +0000 (GMT) Received: from mail.metronet.co.uk (mail.metronet.co.uk [213.162.97.75]) by svr1.postgresql.org (Postfix) with ESMTP id C91E8DA571 for ; Fri, 7 Oct 2005 11:15:18 -0300 (ADT) Received: from mainbox.archonet.com (84-51-143-99.archon037.adsl.metronet.co.uk [84.51.143.99]) by smtp.metronet.co.uk (MetroNet Mail) with ESMTP id DC0CD416F35; Fri, 7 Oct 2005 15:15:10 +0100 (BST) Received: from localhost (localhost.localdomain [127.0.0.1]) by mainbox.archonet.com (Postfix) with ESMTP id 596A615EDA; Fri, 7 Oct 2005 15:11:20 +0100 (BST) Received: from mainbox.archonet.com ([127.0.0.1]) by localhost (mainbox [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 19964-05; Fri, 7 Oct 2005 15:11:07 +0100 (BST) Received: from [192.168.1.17] (client17.office.archonet.com [192.168.1.17]) by mainbox.archonet.com (Postfix) with ESMTP id 2D94915ED5; Fri, 7 Oct 2005 15:11:02 +0100 (BST) Message-ID: <434681F5.6000906@archonet.com> Date: Fri, 07 Oct 2005 15:11:01 +0100 From: Richard Huxton User-Agent: Mozilla Thunderbird 1.0.6 (X11/20050912) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Tom Lane Cc: pgsql-performance@postgresql.org Subject: Re: count(*) using index scan in "query often, update rarely" References: <935801c5cb20$dcf93a00$131fc39e@stratos> <3254.1128692552@sss.pgh.pa.us> In-Reply-To: <3254.1128692552@sss.pgh.pa.us> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.058 required=5 tests=[AWL=0.008, FORGED_RCVD_HELO=0.05] X-Spam-Level: X-Archive-Number: 200510/144 X-Sequence-Number: 14912 Tom Lane wrote: > > There's a workable-looking design in the archives (pghackers probably) > for maintaining overall table counts in a separate table, with each > transaction adding one row of "delta" information just before it > commits. I haven't seen anything else that looks remotely attractive. It might be useful if there was a way to trap certain queries and rewrite/replace them. That way more complex queries could be transparently redirected to a summary table etc. I'm guessing that the overhead to check every query would quickly destroy any gains though. -- Richard Huxton Archonet Ltd From depesz@gmail.com Sat Oct 8 10:44:09 2005 X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (av.hub.org [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 4F874DA590 for ; Sat, 8 Oct 2005 07:44:09 -0300 (ADT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 29970-06 for ; Sat, 8 Oct 2005 10:44:06 +0000 (GMT) Received: from qproxy.gmail.com (qproxy.gmail.com [72.14.204.204]) by svr1.postgresql.org (Postfix) with ESMTP id 8624ED9D1F for ; Sat, 8 Oct 2005 07:44:06 -0300 (ADT) Received: by qproxy.gmail.com with SMTP id z1so647899qbc for ; Sat, 08 Oct 2005 03:44:09 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:references; b=CC8+9nElxnYxl7kTrnoBtsrcp1P3BxtDGQ0a95nYXh1YuhaZOFOKqGQlZVSlHwJT3y7YVJ3rKsD1MCzFXnhIoelmNLRXumpup4IKMmnOsBh4Zlb6oTco/j/iacTrRy+didLZKuVCAFDV51PUqlT5PgNuNr0pkLji6Dx1fyLa3Qw= Received: by 10.64.196.3 with SMTP id t3mr1961096qbf; Sat, 08 Oct 2005 03:44:09 -0700 (PDT) Received: by 10.64.195.4 with HTTP; Sat, 8 Oct 2005 03:44:09 -0700 (PDT) Message-ID: <9e4684ce0510080344v1dfa36f7nea19ddb1ad38f652@mail.gmail.com> Date: Sat, 8 Oct 2005 12:44:09 +0200 From: hubert depesz lubaczewski Reply-To: hubert depesz lubaczewski To: Cestmir Hybl Subject: Re: count(*) using index scan in "query often, update rarely" environment Cc: pgsql-performance@postgresql.org In-Reply-To: <9d5b01c5cb27$e7a1c970$131fc39e@stratos> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_25997_11295934.1128768249494" References: <935801c5cb20$dcf93a00$131fc39e@stratos> <9e4684ce0510070254v67e08bcfhb725715803a5de25@mail.gmail.com> <9d5b01c5cb27$e7a1c970$131fc39e@stratos> X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=1.888 required=5 tests=[AWL=-0.055, HTML_40_50=0.086, HTML_MESSAGE=0.001, RCVD_BY_IP=0.024, RCVD_IN_BL_SPAMCOP_NET=1.832] X-Spam-Level: * X-Archive-Number: 200510/152 X-Sequence-Number: 14920 ------=_Part_25997_11295934.1128768249494 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On 10/7/05, Cestmir Hybl wrote: > > No, I can't speed-up evaluation of generic "count(*) where ()" queries > this way. > no you can't speed up generic where(), *but* you can check what are the mos= t common "where"'s (like usually i do where on one column like: select count(*) from table where some_particular_column =3D 'some value'; where you can simply make the trigger aware of the fact that it should coun= t based on value in some_particular_column. works good enough for me not to look for alternatives. depesz ------=_Part_25997_11295934.1128768249494 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On 10/7/05, Cestmir Hybl <cestmirl@freeside.sk> wrote:
No, I can't speed-up evaluation of gen= eric=20 "count(*) where ()" queries this way.

no you can't speed up generic where(), *but* you can check what are the most common "where"'s (like usually i do where on one column like= :
select count(*) from table where some_particular_column =3D 'some value'; where you can simply make the trigger aware of the fact that it should coun= t based on value in some_particular_column.
works good enough for me not to look for alternatives.

depesz


------=_Part_25997_11295934.1128768249494-- From mark@mark.mielke.cc Sat Oct 8 13:48:21 2005 X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (av.hub.org [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 8B4B4D7E8D for ; Sat, 8 Oct 2005 10:48:21 -0300 (ADT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 76922-10 for ; Sat, 8 Oct 2005 13:48:16 +0000 (GMT) Received: from mail.envyfinancial.com (mark.mielke.cc [206.248.142.186]) by svr1.postgresql.org (Postfix) with ESMTP id 2B57BD7E05 for ; Sat, 8 Oct 2005 10:48:19 -0300 (ADT) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.envyfinancial.com (Postfix) with ESMTP id 6F6C3234B8 for ; Sat, 8 Oct 2005 09:34:32 -0400 (EDT) Received: from mail.envyfinancial.com ([127.0.0.1]) by localhost (mark.mielke.cc [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 24033-07 for ; Sat, 8 Oct 2005 09:34:32 -0400 (EDT) Received: by mail.envyfinancial.com (Postfix, from userid 500) id 43EC0234B3; Sat, 8 Oct 2005 09:34:32 -0400 (EDT) Date: Sat, 8 Oct 2005 09:34:32 -0400 From: mark@mark.mielke.cc To: pgsql-performance@postgresql.org Subject: Re: count(*) using index scan in "query often, update rarely" environment Message-ID: <20051008133432.GC23913@mark.mielke.cc> References: <935801c5cb20$dcf93a00$131fc39e@stratos> <20051007104816.GA8371@uio.no> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20051007104816.GA8371@uio.no> User-Agent: Mutt/1.4.2.1i X-Virus-Scanned: amavisd-new at mail.envyfinancial.com X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.216 required=5 tests=[AWL=-0.012, FORGED_RCVD_HELO=0.05, NO_REAL_NAME=0.178] X-Spam-Level: X-Archive-Number: 200510/154 X-Sequence-Number: 14922 On Fri, Oct 07, 2005 at 12:48:16PM +0200, Steinar H. Gunderson wrote: > On Fri, Oct 07, 2005 at 11:24:05AM +0200, Cestmir Hybl wrote: > > Isn't it possible (and reasonable) for these environments to keep track of > > whether there is a transaction in progress with update to given table and > > if not, use an index scan (count(*) where) or cached value (count(*)) to > > perform this kind of query? > Even if there is no running update, there might still be dead rows in the > table. In any case, of course, a new update could always be occurring while > your counting query was still running. I don't see this being different from count(*) as it is today. Updating a count column is certainly clever. If using a trigger, perhaps it would allow the equivalent of: select count(*) from table for update; :-) Cheers, mark (not that this is necessarily a good thing!) -- mark@mielke.cc / markm@ncf.ca / markm@nortel.com __________________________ . . _ ._ . . .__ . . ._. .__ . . . .__ | Neighbourhood Coder |\/| |_| |_| |/ |_ |\/| | |_ | |/ |_ | | | | | | \ | \ |__ . | | .|. |__ |__ | \ |__ | Ottawa, Ontario, Canada One ring to rule them all, one ring to find them, one ring to bring them all and in the darkness bind them... http://mark.mielke.cc/