pg.ddx.io  pgsql-admin@postgresql.org mailing list archive  
help / color / mirror / Atom feed
Index (re)-creation speed
4+ messages / 2 participants
[nested] [flat]

* Index (re)-creation speed
@ 2004-02-06 04:36 Steve Lane <slane@moyergroup.com>
  2004-02-06 06:30 ` Re: Index (re)-creation speed Tom Lane <tgl@sss.pgh.pa.us>
  0 siblings, 1 reply; 4+ messages in thread

From: Steve Lane @ 2004-02-06 04:36 UTC (permalink / raw)
  To: pgsql-admin

All:

Been looking at speed issues related to loading large data sets, and
experimenting with dropping and rebuilding indexes. I noticed some
interesting behavior on the table in question. Here's the table:

             Table "test_responses_2"
     Attribute     |       Type        | Modifier
-------------------+-------------------+----------
 id_response       | integer           |
 id_code           | integer           |
 id_survey         | integer           |
 id_topic          | integer           |
 id_item           | integer           |
 id_administration | integer           |
 response_int      | smallint          |
 response_txt      | character varying |
 rec_handle        | character varying |
Indices: id_administration_test_key,
         id_code_test_key,
         id_item_test_key,
         id_response_test_key,
         id_survey_test_key,
         id_topic_test_key


When I drop and rebuild the indexes, they take oddly varying amounts of time
to rebuild. I rebuilt them in the following order, with the following rough
times. I took a guess that the speed of the rebuild might be related to the
number of distinct values in the column -- this seems true in some cases but
not in others. Here are the times:

id_response       38 secs (86000 distinct)
id_topic          33 secs (6 distinct)
id_survey         13 secs (1 distinct)
id_code           39 secs (1444 distinct)
id_item           40 secs (65 distinct)
id_administration 13 secs (1 distinct)

Is there anything I should be learning from this?

-- sgl

=======================================================
Steve Lane

Vice President
The Moyer Group
14 North Peoria St Suite 2H
Chicago, IL 60607

Voice: (312) 433-2421       Email: slane@moyergroup.com
Fax:   (312) 850-3930       Web:   http://www.moyergroup.com
=======================================================




^ permalink  raw  reply  [nested|flat] 4+ messages in thread

* Re: Index (re)-creation speed
  2004-02-06 04:36 Index (re)-creation speed Steve Lane <slane@moyergroup.com>
@ 2004-02-06 06:30 ` Tom Lane <tgl@sss.pgh.pa.us>
  2004-02-06 07:03   ` Re: Index (re)-creation speed Steve Lane <slane@moyergroup.com>
  0 siblings, 1 reply; 4+ messages in thread

From: Tom Lane @ 2004-02-06 06:30 UTC (permalink / raw)
  To: Steve Lane <slane@moyergroup.com>; +Cc: pgsql-admin

Steve Lane <slane@moyergroup.com> writes:
> When I drop and rebuild the indexes, they take oddly varying amounts of time
> to rebuild. I rebuilt them in the following order, with the following rough
> times. I took a guess that the speed of the rebuild might be related to the
> number of distinct values in the column -- this seems true in some cases but
> not in others. Here are the times:

> id_response       38 secs (86000 distinct)
> id_topic          33 secs (6 distinct)
> id_survey         13 secs (1 distinct)
> id_code           39 secs (1444 distinct)
> id_item           40 secs (65 distinct)
> id_administration 13 secs (1 distinct)

How many rows altogether in this table?  What sort_mem are you using
(and have you tried altering it)?

When I read your previous message I was wondering about different
datatypes having different comparison costs, but since these are all
integer columns that's clearly not the issue here.  I think you may
be seeing something related to the number of initial runs created in the
sorting algorithm --- with only one distinct value, there'd always be
just one run and no need for any merge passes.

(Why are you bothering to index columns with only one or a few values in
them, anyway?  Such an index will be useless for searches ...)

			regards, tom lane



^ permalink  raw  reply  [nested|flat] 4+ messages in thread

* Re: Index (re)-creation speed
  2004-02-06 04:36 Index (re)-creation speed Steve Lane <slane@moyergroup.com>
  2004-02-06 06:30 ` Re: Index (re)-creation speed Tom Lane <tgl@sss.pgh.pa.us>
@ 2004-02-06 07:03   ` Steve Lane <slane@moyergroup.com>
  2004-02-06 14:50     ` Re: Index (re)-creation speed Tom Lane <tgl@sss.pgh.pa.us>
  0 siblings, 1 reply; 4+ messages in thread

From: Steve Lane @ 2004-02-06 07:03 UTC (permalink / raw)
  To: pgsql-admin

On 2/6/04 12:30 AM, "Tom Lane" <tgl@sss.pgh.pa.us> wrote:

> Steve Lane <slane@moyergroup.com> writes:
>> When I drop and rebuild the indexes, they take oddly varying amounts of time
>> to rebuild. I rebuilt them in the following order, with the following rough
>> times. I took a guess that the speed of the rebuild might be related to the
>> number of distinct values in the column -- this seems true in some cases but
>> not in others. Here are the times:
> 
>> id_response       38 secs (86000 distinct)
>> id_topic          33 secs (6 distinct)
>> id_survey         13 secs (1 distinct)
>> id_code           39 secs (1444 distinct)
>> id_item           40 secs (65 distinct)
>> id_administration 13 secs (1 distinct)
> 
> How many rows altogether in this table?  What sort_mem are you using
> (and have you tried altering it)?

Sorry, left out important info. I cleaned the table first (it's a temporary
copy), then imported one run of data, 86,000 rows.

I haven't checked sort_mem, will do so.
> 
> When I read your previous message I was wondering about different
> datatypes having different comparison costs, but since these are all
> integer columns that's clearly not the issue here.  I think you may
> be seeing something related to the number of initial runs created in the
> sorting algorithm --- with only one distinct value, there'd always be
> just one run and no need for any merge passes.
> 
> (Why are you bothering to index columns with only one or a few values in
> them, anyway?  Such an index will be useless for searches ...)

Again, I left out some useful details. Id_survey and id_administration will
be identical for a single run of data (which can generate 10-100K inserts).
And my tests were just one data set, imported into a cleaned table. Still,
the distinctness of these columns will be low -- on the order of one
distinct value per 10^4 rows. The others have levels of distinctness
proportional to what the above chart shows -- id_response is unique,
id_topic will have 4-8 values per 10^4 records, and so on.

And still, none of this explains to me why the index on id_item drove the
COPY cost up so apparently dramatically. I tried the COPY again both with
and without that one index and in both cases it caused COPY to go from 5 to
40+ seconds.

-- sgl




^ permalink  raw  reply  [nested|flat] 4+ messages in thread

* Re: Index (re)-creation speed
  2004-02-06 04:36 Index (re)-creation speed Steve Lane <slane@moyergroup.com>
  2004-02-06 06:30 ` Re: Index (re)-creation speed Tom Lane <tgl@sss.pgh.pa.us>
  2004-02-06 07:03   ` Re: Index (re)-creation speed Steve Lane <slane@moyergroup.com>
@ 2004-02-06 14:50     ` Tom Lane <tgl@sss.pgh.pa.us>
  0 siblings, 0 replies; 4+ messages in thread

From: Tom Lane @ 2004-02-06 14:50 UTC (permalink / raw)
  To: Steve Lane <slane@moyergroup.com>; +Cc: pgsql-admin

Steve Lane <slane@moyergroup.com> writes:
> And still, none of this explains to me why the index on id_item drove the
> COPY cost up so apparently dramatically. I tried the COPY again both with
> and without that one index and in both cases it caused COPY to go from 5 to
> 40+ seconds.

That doesn't make a lot of sense to me either, particularly seeing that
id_item seems to be an intermediate case as far as uniqueness goes.
Do you still get 30+ seconds to COPY if that is the *only* index?

If the data you're using isn't too sensitive, would you mind sending me
a copy (off-list)?  I'm interested to try profiling this behavior.

			regards, tom lane



^ permalink  raw  reply  [nested|flat] 4+ messages in thread


end of thread, other threads:[~2004-02-06 14:50 UTC | newest]

Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2004-02-06 04:36 Index (re)-creation speed Steve Lane <slane@moyergroup.com>
2004-02-06 06:30 ` Tom Lane <tgl@sss.pgh.pa.us>
2004-02-06 07:03   ` Steve Lane <slane@moyergroup.com>
2004-02-06 14:50     ` Tom Lane <tgl@sss.pgh.pa.us>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox