pg.ddx.io pgsql-performance@postgresql.org mailing list archivehelp / color / mirror / Atom feed
Slow loads when indexes added. 2+ messages / 1 participants [nested] [flat]
* Slow loads when indexes added. @ 2005-03-15 15:23 Stef <svb@ucs.co.za> 0 siblings, 0 replies; 2+ messages in thread From: Stef @ 2005-03-15 15:23 UTC (permalink / raw) To: pgsql-performance Hi all, I suspect this problem/bug has been dealt with already, but I couldn't find anything in the mail archives. I'm using postgres 7.3, and I managed to recreate the problem using the attached files. The database structure is in slow_structure.sql After creating the database, using this script, I ran run_before_load__fast.sql Then I created a load file using create_loadfile.sh (It creates a file called load.sql) I timed the loading of this file, and it loaded in 1 min 11.567 sec Then I recreated the database from slow_structure.sql, ran run_before_load__slow.sql, and then loaded the same load.sql and it took 3 min 51.293 sec which is about 6 times slower. I tried the same thing on postgres 8.0.0 to see if it does the same thing, but there it was consistently slow : 3 min 31.367 sec The only way I got the load.sql to load fast on postgres 8.0.0, was by not creating any of the foreign key constraints that point to the "main" table, and then enabling them afterwards. This gave me the fastest time overall : 1 min 4.911 sec My problem is that on the postgres 7.3.4 database I'm working with, a load process that used to take 40 minutes, now takes 4 hours, because of 3 rows data being loaded into a table (similar in setup to the "main" table in the example) before the indexes were created. (This happens automatically when you dump and re-import the database (7.3.4)) Is there a way to get it to load fast again on the 7.3 database without dropping the foreign key constraints (After running run_before_load_slow.sql) ? And, if someone knows off-hand, what's happening here? TIA Kind Regards Stefan Attachments:run_before_load__fast.sql (application/octet-stream, 236B) download run_before_load__slow.sql (application/octet-stream, 236B) download slow_structure.sql (application/octet-stream, 7.5K) download ^ permalink raw reply [nested|flat] 2+ messages in threadcreate_loadfile.sh (text/x-sh, 865B) download
#!/bin/bash for x in one two three four five six seven eight nine ten do for y in eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen twenty do for z in `seq 1 100` do echo "insert into main (c_text,d_text) values ('${x}','${y}');" >> load.sql echo "insert into a(id) values (currval('public.main_id_seq'::text));" >> load.sql echo "insert into b(id) values (currval('public.main_id_seq'::text));" >> load.sql echo "insert into e(id) values (currval('public.main_id_seq'::text));" >> load.sql echo "insert into f(id) values (currval('public.main_id_seq'::text));" >> load.sql echo "insert into g(id) values (currval('public.main_id_seq'::text));" >> load.sql echo "insert into h(id) values (currval('public.main_id_seq'::text));" >> load.sql done done done
* Re: Slow loads when indexes added. @ 2005-03-16 07:59 Stef <svb@ucs.co.za> 0 siblings, 0 replies; 2+ messages in thread From: Stef @ 2005-03-16 07:59 UTC (permalink / raw) To: andrew@pillette.com; +Cc: pgsql-performance andrew@pillette.com mentioned : => Try ANALYZE after loading the referenced tables, but before loading the main table I attached a new script for creating the load file... Analyze didn't help, it actually took longer to load. I set autocommit to off, and put a commit after every 100 inserts, chattr'd noatime atrribute off recursively on PGDATA, and set fsync to off, this improved the time from 3min 51sec to 2min 37 sec for the slow scenario. But I was already doing all these things in the app that used to take 40 minutes, but now takes four hours to load. Any other suggestions? Kind Regards Stefan ^ permalink raw reply [nested|flat] 2+ messages in thread
end of thread, other threads:[~2005-03-16 07:59 UTC | newest] Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2005-03-15 15:23 Slow loads when indexes added. Stef <svb@ucs.co.za> 2005-03-16 07:59 Re: Slow loads when indexes added. Stef <svb@ucs.co.za>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox