Message-ID: <54B87820.3090101@catalyst.net.nz>
Date: Fri, 16 Jan 2015 15:32:00 +1300
From: Mark Kirkwood <mark.kirkwood@catalyst.net.nz>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:31.0) Gecko/20100101 Thunderbird/31.3.0
MIME-Version: 1.0
To: Josh Berkus <josh@agliodbs.com>,
 postgres performance list <pgsql-performance@postgresql.org>
Subject: Re: Strange choice of general index over partial index
References: <54B83F70.7080004@agliodbs.com> <54B85D32.7090106@catalyst.net.nz>
In-Reply-To: <54B85D32.7090106@catalyst.net.nz>
Content-Type: multipart/mixed; boundary="------------010201020208090103080004"
Precedence: bulk
Sender: pgsql-performance-owner@postgresql.org

This is a multi-part message in MIME format.
--------------010201020208090103080004
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

On 16/01/15 13:37, Mark Kirkwood wrote:
> On 16/01/15 11:30, Josh Berkus wrote:
>> This is an obfuscation and mock up, but:
>>
>> table files (
>>     id serial pk,
>>     filename text not null,
>>     state varchar(20) not null
>>     ... 18 more columns
>> )
>>
>> index file_state on (state)
>>     (35GB in size)
>> index file_in_flight_state (state) where state in (
>> 'waiting','assigning', 'processing' )
>>     (600MB in size)
>> ... 10 more indexes
>>
>> More important facts:
>> * state = 'done' 95% of the time.  thereform the partial index
>> represents only 5% of the table
>> * all indexes and the table are very bloated
>> * server has 128GB RAM
>> * Version 9.2.
>>
>> Given this setup, I would expect the planner to *always* choose
>> file_in_flight_state over file_state for this query:
>>
>> SELECT id, filename FROM files WHERE state = 'waiting';
>>
>> ... and yet it keeps selecting file_state based on extremely small
>> changes to the stats.   This is important because the same query, using
>> file_state, is 20X to 50X slower, because that index frequently gets
>> pushed out of memory.
>>
>> What am I missing?  Or is this potentially a planner bug for costing?
>>
>
> Are you seeing a bitmapscan access plan? If so see if disabling it gets
> you a plan on the files_in_flight index. I'm seeing this scenario with a
> fake/generated dataset a bit like yours in 9.2 (9.5 uses the
> files_in_flight w/o any coercing).
>

FWIW: For me 9.2 and 9.3 (default config) generate plans like:
state=# EXPLAIN ANALYZE
SELECT id, filename
FROM files
WHERE state = 'processing';
                                                             QUERY PLAN 

----------------------------------------------------------------------------------------------------------------------------------
  Bitmap Heap Scan on files  (cost=3102.02..89228.68 rows=164333 
width=15) (actual time=26.629..803.507 rows=166696 loops=1)
    Recheck Cond: ((state)::text = 'processing'::text)
    Rows Removed by Index Recheck: 7714304
    ->  Bitmap Index Scan on file_state  (cost=0.00..3060.93 rows=164333 
width=0) (actual time=25.682..25.682 rows=166696 loops=1)
          Index Cond: ((state)::text = 'processing'::text)
  Total runtime: 808.662 ms
(6 rows)


whereas 9.4 and 9.5 get:

                                                               QUERY 
PLAN

---------------------------------------------------------------------------------------------------------------------------------------
  Index Scan using file_in_flight on files  (cost=0.42..62857.39 
rows=158330 width=15) (actual time=0.055..202.732 rows=166696 loops=1)
    Index Cond: ((state)::text = 'processing'::text)
  Planning time: 24.203 ms
  Execution time: 208.926 ms
(4 rows)


This is with each version loading exactly the same dataset (generated by 
the attached scripty). Obviously this is a vast simplification of what 
Josh is looking at - but it is (hopefully) interesting that these later 
versions are doing so much better...

Cheers

Mark


--------------010201020208090103080004
Content-Type: application/x-perl;
 name="gendata.pl"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="gendata.pl"

#!/usr/bin/perl
#
# generate fake dataset for schema:
#
# CREATE TABLE files (
#    id SERIAL PRIMARY KEY,
#    filename TEXT NOT NULL,
#    state VARCHAR(20) NOT NULL,
#    filler TEXT
# );
#
#  to test query:
#
# SELECT id, filename
# FROM files
# WHERE state = 'waiting' | 'done';

use strict;
use warnings;

my $dir = '/data0/dump';
my @states = ('done','waiting','assigning','processing');
my $name;
my $state;
my $filler;
my $i;
my $stateidx;

my $rows = 10000000;
print "generate files\n";

open(FH, "> $dir/files.dat") || die ("cannot open files.dat: $!");

for ($i = 0; $i < $rows; $i++) {

    $name = 'file' . $i;
    if (rand() > 0.95) {
        $stateidx = 1 + rand(scalar(@states) -1 );
    } else {
        $stateidx = 0;
    }

    $state = $states[$stateidx];
    #$filler = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx';
    $filler = 'xxxxxxxxxxxxx';
    print FH "$i,$name,$state,$filler\n";
}
close (FH);

--------------010201020208090103080004
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

--------------010201020208090103080004--