Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.72) (envelope-from ) id 1UhDib-0002U3-Qr for pgsql-performance@arkaria.postgresql.org; Tue, 28 May 2013 06:49:14 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.72) (envelope-from ) id 1UhDib-0000fC-95 for pgsql-performance@arkaria.postgresql.org; Tue, 28 May 2013 06:49:13 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtp (Exim 4.72) (envelope-from ) id 1UhDiZ-0000ev-O7 for pgsql-performance@postgresql.org; Tue, 28 May 2013 06:49:12 +0000 Received: from mail-la0-x22d.google.com ([2a00:1450:4010:c03::22d]) by magus.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1UhDiP-0003wj-EJ for pgsql-performance@postgresql.org; Tue, 28 May 2013 06:49:11 +0000 Received: by mail-la0-f45.google.com with SMTP id ec20so7149924lab.32 for ; Mon, 27 May 2013 23:49:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=references:mime-version:in-reply-to:content-type :content-transfer-encoding:message-id:cc:x-mailer:from:subject:date :to; bh=BalQ4+l7i7CQN9jBIpvxLRfweViwd0GhHnn2ME+z4H0=; b=NQ6e7ML1l3BE14tQiTqEY/YmjZKnxBOCFWjYpEJ3IXGFep2ArD9FgYcRtcLVE/rsjf cSoS6tbYiocZ/v0j3c4sCT1QZWoA4yK1C6AjH2P0/oNaiFcEWn7gJCL8pwGvcVKJyjTO 3oJdne0NPQgh4pg/jFVdcYLQbraD/nUf4uSfFN2IYvz7h8hQQ8kcf6wZa7CwX2n8SOFW CYBb1X4/mr+e3aTYd5IPajonjX1P8AjSvRmpTT8QCeyqSp9ZnVwg00otLmOP+wv+xtUr fNGmwqkruVU1yXppN8y7A+Zx3huH24gERnCEQYETNQwA8NfkWGyefVmBkGkKTIEEbIXg PvZQ== X-Received: by 10.112.33.227 with SMTP id u3mr6287426lbi.35.1369723739993; Mon, 27 May 2013 23:48:59 -0700 (PDT) Received: from [10.87.147.117] ([213.87.135.222]) by mx.google.com with ESMTPSA id c7sm1317508lag.3.2013.05.27.23.48.49 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 27 May 2013 23:48:58 -0700 (PDT) References: <6D8D2A94-C545-487F-BCB6-1E601F93E58C@gmail.com> Mime-Version: 1.0 (1.0) In-Reply-To: Content-Type: multipart/alternative; boundary=Apple-Mail-FF42C2CF-046C-4412-B5A0-9430FB7BF8C2 Content-Transfer-Encoding: 7bit Message-Id: <024E6C97-A56F-4BFD-AB73-6148C3CB80AA@gmail.com> Cc: "pgsql-performance@postgresql.org" X-Mailer: iPhone Mail (10B350) From: Evgeniy Shishkin Subject: Re: Slow SELECT by primary key? Postgres 9.1.2 Date: Tue, 28 May 2013 10:48:45 +0400 To: John Mudd X-Pg-Spam-Score: -2.0 (--) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-performance Precedence: bulk Sender: pgsql-performance-owner@postgresql.org --Apple-Mail-FF42C2CF-046C-4412-B5A0-9430FB7BF8C2 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable On 28.05.2013, at 2:17, John Mudd wrote: > Thanks again. >=20 > Well, I have two problems with using the CLUSTER option. It's only tempora= ry since any updates, depending how much free space is reserved per page, re= quires re-running the CLUSTER. And my primary concern is that it arbitrarily= gives an unfair advantage to the primary key SELECT. Still, it's easy to te= st so here are the results. The primary key still looses even with the CLUST= ER. Granted it is close but considering this is now an unfair comparison it s= till doesn't make sense to me. How can a search for a specific row that shou= ld be fairly straight forward take longer than a search that includes an ORD= ER BY clause? >=20 Well, you do just regular index scan because of LIMIT 1. And now it is just a matter of index size and table organization. I also don't understand why you consider CLUSTER unfair - the way you popula= ted the table was natural cluster over my_key. But it bothers me why my_key is always better. Can you please test it on dif= ferent values but the same rows? Because now it is two different tuples and y= ou count every io. >=20 > test=3D# CLUSTER test_select USING test_select_pkey ; > CLUSTER > test=3D# VACUUM ANALYZE test_select ; > VACUUM >=20 >> (stopped postgres; reset O/S cache; started postgres) >=20 >=20 > test=3D# explain analyze SELECT * FROM test_select WHERE key1 >=3D 500000 O= RDER BY key1, key2, key3, id LIMIT 1; > QUERY PLAN > --------------------------------------------------------------------------= ------------------------------------------------------------ > Limit (cost=3D0.00..0.08 rows=3D1 width=3D21) (actual time=3D19.430..19.= 431 rows=3D1 loops=3D1) > -> Index Scan using my_key on test_select (cost=3D0.00..41938.15 rows= =3D499992 width=3D21) (actual time=3D19.428..19.428 rows=3D1 loops=3D1) > Index Cond: (key1 >=3D 500000) > Total runtime: 19.526 ms >=20 >=20 >> (stopped postgres; reset O/S cache; started postgres) >=20 >=20 > test=3D# explain analyze SELECT * FROM test_select WHERE id =3D 500000; > QUERY PLAN = =20 > --------------------------------------------------------------------------= ------------------------------------------------------- > Index Scan using test_select_pkey on test_select (cost=3D0.00..8.36 rows= =3D1 width=3D21) (actual time=3D21.070..21.072 rows=3D1 loops=3D1) > Index Cond: (id =3D 500000) > Total runtime: 21.178 ms >=20 >=20 >=20 >=20 > On Mon, May 27, 2013 at 10:59 AM, Evgeny Shishkin w= rote: >>=20 >> On May 27, 2013, at 6:35 PM, John Mudd wrote: >>=20 >>> Thanks, that's easy enough to test. Didn't seem to help though. >>=20 >> Ok. And if you CLUSTER tables USING PK? >>=20 >>>=20 >>> test=3D# REINDEX index test_select_pkey; >>> REINDEX >>> test=3D# VACUUM ANALYZE test_select ; >>> VACUUM >>>=20 >>>=20 >>> (stopped postgres; reset O/S cache; started postgres) >>>=20 >>> test=3D# explain analyze SELECT * FROM test_select WHERE key1 >=3D 50000= 0 ORDER BY key1, key2, key3, id LIMIT 1; >>> QUERY PLAN= =20 >>> ------------------------------------------------------------------------= -------------------------------------------------------------- >>> Limit (cost=3D0.00..0.08 rows=3D1 width=3D21) (actual time=3D16.368..1= 6.369 rows=3D1 loops=3D1) >>> -> Index Scan using my_key on test_select (cost=3D0.00..41981.16 ro= ws=3D501333 width=3D21) (actual time=3D16.366..16.366 rows=3D1 loops=3D1) >>> Index Cond: (key1 >=3D 500000) >>> Total runtime: 16.444 ms >>>=20 >>>=20 >>> (stopped postgres; reset O/S cache; started postgres) >>>=20 >>> test=3D# explain analyze SELECT * FROM test_select WHERE id =3D 500000; >>> QUERY PLAN >>> ------------------------------------------------------------------------= --------------------------------------------------------- >>> Index Scan using test_select_pkey on test_select (cost=3D0.00..8.36 ro= ws=3D1 width=3D21) (actual time=3D23.072..23.074 rows=3D1 loops=3D1) >>> Index Cond: (id =3D 500000) >>> Total runtime: 23.192 ms >>>=20 >>>=20 >>>=20 >>>=20 >>> On Mon, May 27, 2013 at 10:21 AM, Evgeny Shishkin = wrote: >>>>=20 >>>> On May 27, 2013, at 6:02 PM, John Mudd wrote: >>>>=20 >>>> > Postgres 9.1.2 on Ubuntu 12.04 >>>> > >>>> > Any reason why a select by primary key would be slower than a select t= hat includes an ORDER BY? I was really hoping using the primary key would gi= ve me a boost. >>>> > >>>>=20 >>>> You created my_key after data loading, and PK was there all the time. >>>> If you REINDEX PK, i bet it will be as fast. >>>>=20 >>>> > I stopped the server and cleared the O/S cache using "sync; echo 3 > /= proc/sys/vm/drop_caches" between the runs. >>>> > >>>> > >>>> > >>>> > test=3D# VACUUM ANALYZE test_select; >>>> > VACUUM >>>> > >>>> > (stopped postgres; reset O/S cache; started postgres) >>>> > >>>> > test=3D# explain analyze SELECT * FROM test_select WHERE key1 >=3D 50= 0000 ORDER BY key1, key2, key3, id LIMIT 1; >>>> > QUERY P= LAN >>>> > ---------------------------------------------------------------------= ----------------------------------------------------------------- >>>> > Limit (cost=3D0.00..0.08 rows=3D1 width=3D21) (actual time=3D12.599= ..12.600 rows=3D1 loops=3D1) >>>> > -> Index Scan using my_key on test_select (cost=3D0.00..41895.49= rows=3D498724 width=3D21) (actual time=3D12.597..12.597 rows=3D1 loops=3D1)= >>>> > Index Cond: (key1 >=3D 500000) >>>> > Total runtime: 12.678 ms >>>> > >>>> > (stopped postgres; reset O/S cache; started postgres) >>>> > >>>> > test=3D# explain analyze SELECT * FROM test_select WHERE id =3D 50000= 0; >>>> > QUERY PLAN= >>>> > ---------------------------------------------------------------------= ------------------------------------------------------------ >>>> > Index Scan using test_select_pkey on test_select (cost=3D0.00..8.36= rows=3D1 width=3D21) (actual time=3D31.396..31.398 rows=3D1 loops=3D1) >>>> > Index Cond: (id =3D 500000) >>>> > Total runtime: 31.504 ms >>>> > >>>> > >>>> > >>>> > Schema: >>>> > >>>> > test=3D# \d test_select >>>> > Table "public.test_select" >>>> > Column | Type | Modifiers >>>> > --------+--------------+---------------------------------------------= ------------- >>>> > id | integer | not null default nextval('test_select_id_seq= '::regclass) >>>> > key1 | integer | >>>> > key2 | integer | >>>> > key3 | integer | >>>> > data | character(4) | >>>> > Indexes: >>>> > "test_select_pkey" PRIMARY KEY, btree (id) >>>> > "my_key" btree (key1, key2, key3, id) >>>> > >>>> > test=3D# >>>> > >>>> > >>>> > >>>> > Sample data: >>>> > >>>> > test=3D# SELECT * FROM test_select LIMIT 10; >>>> > id | key1 | key2 | key3 | data >>>> > ----+--------+--------+--------+------ >>>> > 1 | 984966 | 283954 | 772063 | x >>>> > 2 | 817668 | 393533 | 924888 | x >>>> > 3 | 751039 | 798753 | 454309 | x >>>> > 4 | 128505 | 329643 | 280553 | x >>>> > 5 | 105600 | 257225 | 710015 | x >>>> > 6 | 323891 | 615614 | 83206 | x >>>> > 7 | 194054 | 63506 | 353171 | x >>>> > 8 | 212068 | 881225 | 271804 | x >>>> > 9 | 644180 | 26693 | 200738 | x >>>> > 10 | 136586 | 498699 | 554417 | x >>>> > (10 rows) >>>> > >>>> > >>>> > >>>> > >>>> > Here's how I populated the table: >>>> > >>>> > import psycopg2 >>>> > >>>> > conn =3D psycopg2.connect('dbname=3Dtest') >>>> > >>>> > cur =3D conn.cursor() >>>> > >>>> > def random_int(): >>>> > n =3D 1000000 >>>> > return random.randint(0,n) >>>> > >>>> > def random_key(): >>>> > return random_int(), random_int(), random_int() >>>> > >>>> > def create_table(): >>>> > cur.execute(''' >>>> > DROP TABLE IF EXISTS test_select; >>>> > >>>> > CREATE TABLE test_select ( >>>> > id SERIAL PRIMARY KEY, >>>> > key1 INTEGER, >>>> > key2 INTEGER, >>>> > key3 INTEGER, >>>> > data char(4) >>>> > ); >>>> > ''') >>>> > conn.commit() >>>> > >>>> > n =3D 1000000 >>>> > for i in range(n): >>>> > cur.execute("INSERT INTO test_select(key1, key2, key3, data) V= ALUES(%s, %s, %s, 'x')", random_key()) >>>> > conn.commit() >>>> > >>>> > cur.execute('CREATE INDEX my_key ON test_select(key1, key2, key3,= id)') >>>> > conn.commit() >>>> > >>>> > create_table() >>>> > >=20 --Apple-Mail-FF42C2CF-046C-4412-B5A0-9430FB7BF8C2 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable




On 28.05.20= 13, at 2:17, John Mudd <johnbmudd@= gmail.com> wrote:

Thanks a= gain.

Well, I have t= wo problems with using the CLUSTER option. It's only temporary since any upd= ates, depending how much free space is reserved per page, requires re-runnin= g the CLUSTER. And my primary concern is that it arbitrarily gives an unfair= advantage to the primary key SELECT. Still, it's easy to test so here are t= he results. The primary key still looses even with the CLUSTER. Granted it i= s close but considering this is now an unfair comparison it still doesn't ma= ke sense to me. How can a search for a specific row that should be fairly st= raight forward take longer than a search that includes an ORDER BY clause?

<= /div>

Well, you do just regular index scan b= ecause of LIMIT 1.

And now it is just a matter of i= ndex size and table organization.

I also don't unde= rstand why you consider CLUSTER unfair - the way you populated the table was= natural cluster over my_key.

But it bothers me why= my_key is always better. Can you please test it on different values but the= same rows? Because now it is two different tuples and you count every io.

test=3D# CLUSTER test_select USING test_sele= ct_pkey ;
CLUSTER
tes= t=3D# VACUUM ANALYZE test_select ;
VACUUM

(stopped postgres; reset O/S cach= e; started postgres)

test= =3D# explain analyze SELECT * FROM test_select WHERE key1 >=3D 500000 ORD= ER BY key1, key2, key3, id LIMIT 1;
                    &= nbsp;                     &= nbsp;                   QUERY P= LAN
--------------------------------------------------------------= ------------------------------------------------------------------------
 Limit  (cost=3D0.00..0.08 rows=3D1 width=3D21) (actual= time=3D19.430..19.431 rows=3D1 loops=3D1)
   -> &nbs= p;Index Scan using my_key on test_select  (cost=3D0.00..41938.15 rows=3D= 499992 width=3D21) (actual time=3D19.428..19.428 rows=3D1 loops=3D1)
         Index Cond: (key1 >=3D= 500000)
 Total runtime: 19.526 ms


(= stopped postgres; reset O/S cache; started postgres)

test=3D# explain analyze SELECT * FRO= M test_select WHERE id =3D 500000;
        &nb= sp;                     &n= bsp;                     &= nbsp;      QUERY PLAN           &nbs= p;                     &nb= sp;                     &n= bsp;    
-----------------------------------------------------------------------= ----------------------------------------------------------
&= nbsp;Index Scan using test_select_pkey on test_select  (cost=3D0.00..8.= 36 rows=3D1 width=3D21) (actual time=3D21.070..21.072 rows=3D1 loops=3D1)=
   Index Cond: (id =3D 500000)
 T= otal runtime: 21.178 ms




On Mon, May 27, 2013= at 10:59 AM, Evgeny Shishkin <itparanoia@gmail.com> wrote:=

<= div class=3D"im">
On May 27, 2013, at 6:35 PM, John Mudd <johnbmudd@gmail.com> wr= ote:

Thanks, that's easy enough to test. Didn't see= m to help though.


Ok. And if you CLUSTE= R tables USING PK?


test=3D= # REINDEX index test_select_pkey;
REINDEX
test=3D# VACUUM ANALYZE test_select ;
VACUUM


(stopped postgres; reset O/S cache; started postgres)

test=3D# explain analyze SELECT * FROM test_select= WHERE key1 >=3D 500000 ORDER BY key1, key2, key3, id LIMIT 1;
=                      =                      = ;                   QUERY PLAN &= nbsp;                     &= nbsp;                     &= nbsp;                
-----------------------------------------------------------------------= ---------------------------------------------------------------
<= div> Limit  (cost=3D0.00..0.08 rows=3D1 width=3D21) (actual time=3D= 16.368..16.369 rows=3D1 loops=3D1)
   ->  Index Scan using my_key on test_select  (= cost=3D0.00..41981.16 rows=3D501333 width=3D21) (actual time=3D16.366..16.36= 6 rows=3D1 loops=3D1)
         Index Cond= : (key1 >=3D 500000)
 Total runtime: 16.444 ms


(stopped postgres; reset O/S cache; started postgres= )

test=3D# explain analyze SELECT * FROM test_s= elect WHERE id =3D 500000;
                    &= nbsp;                     &= nbsp;                QUERY PLAN
----------------------------------------------------------------------= -----------------------------------------------------------
 Index Scan u= sing test_select_pkey on test_select  (cost=3D0.00..8.36 rows=3D1 width= =3D21) (actual time=3D23.072..23.074 rows=3D1 loops=3D1)
   Index Cond: (id =3D 500000)
 Total runtime: 23.192 ms




On Mon, May 27, 2013 at 10:21 AM, Evgeny Shishkin <itparanoia@gmail.com= > wrote:

On May 27, 2013, at 6:02 PM, John Mudd <johnbmudd@gmail.com> wrote:

> Postgres 9.1.2 on Ubuntu 12.04
>
> Any reason why a select by primary key would be slower than a select th= at includes an ORDER BY? I was really hoping using the primary key would giv= e me a boost.
>

You created my_key after data loading, and PK was there all the time.<= br> If you REINDEX PK, i bet it will be as fast.

> I stopped the server and cleared the O/S cache using "sync; echo 3 >= /proc/sys/vm/drop_caches" between the runs.
>
>
>
> test=3D# VACUUM ANALYZE test_select;
> VACUUM
>
> (stopped postgres; reset O/S cache; started postgres)
>
> test=3D# explain analyze SELECT * FROM test_select WHERE key1 >=3D 5= 00000 ORDER BY key1, key2, key3, id LIMIT 1;
>                     &= nbsp;                     &= nbsp;                   QUERY P= LAN
> -----------------------------------------------------------------------= ---------------------------------------------------------------
>  Limit  (cost=3D0.00..0.08 rows=3D1 width=3D21) (actual time=3D= 12.599..12.600 rows=3D1 loops=3D1)
>    ->  Index Scan using my_key on test_select  (= cost=3D0.00..41895.49 rows=3D498724 width=3D21) (actual time=3D12.597..12.59= 7 rows=3D1 loops=3D1)
>          Index Cond: (key1 >=3D 500000)
= >  Total runtime: 12.678 ms
>
> (stopped postgres; reset O/S cache; started postgres)
>
> test=3D# explain analyze SELECT * FROM test_select WHERE id =3D 500000;=
>                     &= nbsp;                     &= nbsp;                QUERY PLAN
> -----------------------------------------------------------------------= ----------------------------------------------------------
>  Index Scan using test_select_pkey on test_select  (cost=3D0.= 00..8.36 rows=3D1 width=3D21) (actual time=3D31.396..31.398 rows=3D1 loops=3D= 1)
>    Index Cond: (id =3D 500000)
>  Total runtime: 31.504 ms
>
>
>
> Schema:
>
> test=3D# \d test_select
>                     &= nbsp;       Table "public.test_select"
>  Column |     Type     |       &= nbsp;                Modifiers
> --------+--------------+-----------------------------------------------= -----------
>  id     | integer      | not null default= nextval('test_select_id_seq'::regclass)
>  key1   | integer      |
>  key2   | integer      |
>  key3   | integer      |
>  data   | character(4) |
> Indexes:
>     "test_select_pkey" PRIMARY KEY, btree (id)
>     "my_key" btree (key1, key2, key3, id)
>
> test=3D#
>
>
>
> Sample data:
>
> test=3D# SELECT * FROM test_select LIMIT 10;
>  id |  key1  |  key2  |  key3  | dat= a
> ----+--------+--------+--------+------
>   1 | 984966 | 283954 | 772063 | x
>   2 | 817668 | 393533 | 924888 | x
>   3 | 751039 | 798753 | 454309 | x
>   4 | 128505 | 329643 | 280553 | x
>   5 | 105600 | 257225 | 710015 | x
>   6 | 323891 | 615614 |  83206 | x
>   7 | 194054 |  63506 | 353171 | x
>   8 | 212068 | 881225 | 271804 | x
>   9 | 644180 |  26693 | 200738 | x
>  10 | 136586 | 498699 | 554417 | x
> (10 rows)
>
>
>
>
> Here's how I populated the table:
>
> import psycopg2
>
> conn =3D psycopg2.connect('dbname=3Dtest')
>
> cur =3D conn.cursor()
>
> def random_int():
>     n =3D 1000000
>     return random.randint(0,n)
>
> def random_key():
>     return random_int(), random_int(), random_int()
>
> def create_table():
>     cur.execute('''
>             DROP TABLE IF EXISTS test_sel= ect;
>
>             CREATE TABLE test_select ( >                 id    = ;                  SERIAL PRIMA= RY KEY,
>                 key1   &nb= sp;                INTEGER,
>                 key2   &nb= sp;                INTEGER,
>                 key3   &nb= sp;                INTEGER,
>                 data   &nb= sp;                char(4)
>             );
>         ''')
>     conn.commit()
>
>     n =3D 1000000
>     for i in range(n):
>         cur.execute("INSERT INTO test_select(key1, k= ey2, key3, data) VALUES(%s, %s, %s, 'x')", random_key())
>     conn.commit()
>
>     cur.execute('CREATE INDEX my_key ON test_select(key1, key= 2, key3, id)')
>     conn.commit()
>
> create_table()
>




= --Apple-Mail-FF42C2CF-046C-4412-B5A0-9430FB7BF8C2--