pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feed7.3.2 pg_restore very slow
9+ messages / 4 participants
[nested] [flat]
* 7.3.2 pg_restore very slow
@ 2006-08-06 16:54 Saranya Sivakumar <sarlavk@yahoo.com>
0 siblings, 1 reply; 9+ messages in thread
From: Saranya Sivakumar @ 2006-08-06 16:54 UTC (permalink / raw)
To: pgsql-performance; pgsqlnovice <pgsql-novice@postgresql.org>
Hi All,
I am trying to back up a full copy of one of our databases (14G) and restore it on another server. Both databases run 7.3.2 version. Though the restore completed successfully, it took 9 hours for the process to complete. The destination server runs Fedora Core 3 with 512 MB RAM and has 1 processor. I have also deferred referential intergrity checks during the restore. I tried to tune some parameters in the config file, but it still takes 9 hours.
I have tried this same procedure to restore a full copy, but using 8.1(pg_dump and pg_restore) on a different server and that process took only 2 hours for the same database. But we are unable to migrate to 8.1 at this point and stuck with 7.3.2.
I use a script to dump/restore. I can send the same if that information is needed.
Please give me some pointers on what else I should be looking at to reduce the restore time using 7.3.2 version.
Thanks,
Sincerely,
Saranya Sivakumar
---------------------------------
Yahoo! Messenger with Voice. Make PC-to-Phone Calls to the US (and 30+ countries) for 2ยข/min or less.
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: [PERFORM] 7.3.2 pg_restore very slow
@ 2006-08-07 09:55 Richard Huxton <dev@archonet.com>
parent: Saranya Sivakumar <sarlavk@yahoo.com>
0 siblings, 2 replies; 9+ messages in thread
From: Richard Huxton @ 2006-08-07 09:55 UTC (permalink / raw)
To: Saranya Sivakumar <sarlavk@yahoo.com>; +Cc: pgsql-performance; pgsqlnovice <pgsql-novice@postgresql.org>
Saranya Sivakumar wrote:
> Hi All,
>
> I am trying to back up a full copy of one of our databases (14G) and
> restore it on another server. Both databases run 7.3.2 version.
> Though the restore completed successfully, it took 9 hours for the
> process to complete. The destination server runs Fedora Core 3 with
> 512 MB RAM and has 1 processor. I have also deferred referential
> intergrity checks during the restore. I tried to tune some parameters
> in the config file, but it still takes 9 hours.
Firstly, you should upgrade to the most recent version of 7.3.x (7.3.15)
- that's a *lot* of bug-fixes you are missing
Then, I would temporarily disable fsync and increase sort_mem and
checkpoint_segments. What you're trying to do is make a single process
run as fast as possible, so allow it to grab more resources than you
normally would.
--
Richard Huxton
Archonet Ltd
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: [PERFORM] 7.3.2 pg_restore very slow
@ 2006-08-07 15:35 Saranya Sivakumar <sarlavk@yahoo.com>
parent: Richard Huxton <dev@archonet.com>
1 sibling, 0 replies; 9+ messages in thread
From: Saranya Sivakumar @ 2006-08-07 15:35 UTC (permalink / raw)
To: Richard Huxton <dev@archonet.com>; +Cc: pgsql-performance; pgsqlnovice <pgsql-novice@postgresql.org>
Hi Richard,
Thank you very much for the suggestions. As I said, we are stuck with 7.3.2 version for now. We have a Upgrade Project in place, but this backup is something we have to do immediately (we do not have enough time to test our application with 7.3.15 :( )
The checkpoint segments occur every 1.15 minutes with the default setting.
I tried tuning some parameters in the conf file, which took 4.5 hours for the restore.
sort_mem = 40960
shared_buffers = 3000
#checkpoint_segments = 3 (default)
#fsync = true --I will disable this and try
We can afford to have a downtime of only 1 to 1.5 hours.
I am going to increase the shared_buffers, sort_mem and disable fysnc as suggested by you, and try the restore process again.
I would appreciate any other suggestions/advice in this regard.
Thanks,
Saranya
Richard Huxton <dev@archonet.com> wrote:
Saranya Sivakumar wrote:
> Hi All,
>
> I am trying to back up a full copy of one of our databases (14G) and
> restore it on another server. Both databases run 7.3.2 version.
> Though the restore completed successfully, it took 9 hours for the
> process to complete. The destination server runs Fedora Core 3 with
> 512 MB RAM and has 1 processor. I have also deferred referential
> intergrity checks during the restore. I tried to tune some parameters
> in the config file, but it still takes 9 hours.
Firstly, you should upgrade to the most recent version of 7.3.x (7.3.15)
- that's a *lot* of bug-fixes you are missing
Then, I would temporarily disable fsync and increase sort_mem and
checkpoint_segments. What you're trying to do is make a single process
run as fast as possible, so allow it to grab more resources than you
normally would.
--
Richard Huxton
Archonet Ltd
---------------------------------
See the all-new, redesigned Yahoo.com. Check it out.
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: [PERFORM] 7.3.2 pg_restore very slow
@ 2006-08-07 16:08 Saranya Sivakumar <sarlavk@yahoo.com>
parent: Richard Huxton <dev@archonet.com>
1 sibling, 1 reply; 9+ messages in thread
From: Saranya Sivakumar @ 2006-08-07 16:08 UTC (permalink / raw)
To: pgsql-performance; pgsqlnovice <pgsql-novice@postgresql.org>
Hi All,
I tried to set shared_buffers= 10000, turned off fsync and reload the config file.
But I got the following error:
IpcMemoryCreate: shmget(key=5432001, size=85450752, 03600) failed: Invalid argument
This error usually means that PostgreSQL's request for a shared memory
segment exceeded your kernel's SHMMAX parameter. You can either
reduce the request size or reconfigure the kernel with larger SHMMAX.
To reduce the request size (currently 85450752 bytes), reduce
PostgreSQL's shared_buffers parameter (currently 10000) and/or
its max_connections parameter (currently 128).
If the request size is already small, it's possible that it is less than
your kernel's SHMMIN parameter, in which case raising the request size or
reconfiguring SHMMIN is called for.
The total RAM available on this machine is 512MB.
I am not sure how to set these parameters SHMMAX and SHMMIN.
Any help/advice would be greatly appreciated.
Thanks,
Saranya
Richard Huxton <dev@archonet.com> wrote:
Saranya Sivakumar wrote:
> Hi All,
>
> I am trying to back up a full copy of one of our databases (14G) and
> restore it on another server. Both databases run 7.3.2 version.
> Though the restore completed successfully, it took 9 hours for the
> process to complete. The destination server runs Fedora Core 3 with
> 512 MB RAM and has 1 processor. I have also deferred referential
> intergrity checks during the restore. I tried to tune some parameters
> in the config file, but it still takes 9 hours.
Firstly, you should upgrade to the most recent version of 7.3.x (7.3.15)
- that's a *lot* of bug-fixes you are missing
Then, I would temporarily disable fsync and increase sort_mem and
checkpoint_segments. What you're trying to do is make a single process
run as fast as possible, so allow it to grab more resources than you
normally would.
--
Richard Huxton
Archonet Ltd
---------------------------------
Do you Yahoo!?
Everyone is raving about the all-new Yahoo! Mail Beta.
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: [PERFORM] 7.3.2 pg_restore very slow
@ 2006-08-07 16:17 Richard Broersma Jr <rabroersma@yahoo.com>
parent: Saranya Sivakumar <sarlavk@yahoo.com>
0 siblings, 1 reply; 9+ messages in thread
From: Richard Broersma Jr @ 2006-08-07 16:17 UTC (permalink / raw)
To: Saranya Sivakumar <sarlavk@yahoo.com>; pgsql-performance; pgsqlnovice <pgsql-novice@postgresql.org>
> IpcMemoryCreate: shmget(key=5432001, size=85450752, 03600) failed: Invalid argument
> This error usually means that PostgreSQL's request for a shared memory
> segment exceeded your kernel's SHMMAX parameter. You can either
> reduce the request size or reconfigure the kernel with larger SHMMAX.
> To reduce the request size (currently 85450752 bytes), reduce
> PostgreSQL's shared_buffers parameter (currently 10000) and/or
> its max_connections parameter (currently 128).
> If the request size is already small, it's possible that it is less than
> your kernel's SHMMIN parameter, in which case raising the request size or
> reconfiguring SHMMIN is called for.
if you cat /proc/sys/kernel/shmmax
it will tell you what it is set to. It needs to be at least "85450752". The size that Postgresql
is trying to grab.
also shmall may need to be adjusted also.
> The total RAM available on this machine is 512MB.
>
> I am not sure how to set these parameters SHMMAX and SHMMIN.
> Any help/advice would be greatly appreciated.
http://www.postgresql.org/docs/8.1/interactive/kernel-resources.html
This will help you to set the kernel parameters.
Regards,
Richard Broersma Jr.
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: [PERFORM] 7.3.2 pg_restore very slow
@ 2006-08-07 17:28 Saranya Sivakumar <sarlavk@yahoo.com>
parent: Richard Broersma Jr <rabroersma@yahoo.com>
0 siblings, 2 replies; 9+ messages in thread
From: Saranya Sivakumar @ 2006-08-07 17:28 UTC (permalink / raw)
To: pgsqlnovice <pgsql-novice@postgresql.org>; pgsql-performance
Hi Richard,
Thank you very much for the information. The SHMMAX was set to 33554432, and that's why it failed to start the postmaster. Thanks for the link to the kernel resources article. I guess changing these parameters would require recompiling the kernel.
Is there any work around without changing these parameters to make maximum use of RAM?
We got a new server now with 2GB RAM, but it also has the same value for SHMMAX.
And I am trying the restore with the following conf
sort_mem = 40960 (changed from 1024)
shared_buffers = 3000 (changed from 64)
max_connections = 128 (changed from 32)
Thanks,
Saranya
Richard Broersma Jr <rabroersma@yahoo.com> wrote:
> IpcMemoryCreate: shmget(key=5432001, size=85450752, 03600) failed: Invalid argument
> This error usually means that PostgreSQL's request for a shared memory
> segment exceeded your kernel's SHMMAX parameter. You can either
> reduce the request size or reconfigure the kernel with larger SHMMAX.
> To reduce the request size (currently 85450752 bytes), reduce
> PostgreSQL's shared_buffers parameter (currently 10000) and/or
> its max_connections parameter (currently 128).
> If the request size is already small, it's possible that it is less than
> your kernel's SHMMIN parameter, in which case raising the request size or
> reconfiguring SHMMIN is called for.
if you cat /proc/sys/kernel/shmmax
it will tell you what it is set to. It needs to be at least "85450752". The size that Postgresql
is trying to grab.
also shmall may need to be adjusted also.
> The total RAM available on this machine is 512MB.
>
> I am not sure how to set these parameters SHMMAX and SHMMIN.
> Any help/advice would be greatly appreciated.
http://www.postgresql.org/docs/8.1/interactive/kernel-resources.html
This will help you to set the kernel parameters.
Regards,
Richard Broersma Jr.
---------------------------------
Groups are talking. We´re listening. Check out the handy changes to Yahoo! Groups.
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: [PERFORM] 7.3.2 pg_restore very slow
@ 2006-08-07 18:25 Markus Schaber <schabi@logix-tt.com>
parent: Saranya Sivakumar <sarlavk@yahoo.com>
1 sibling, 0 replies; 9+ messages in thread
From: Markus Schaber @ 2006-08-07 18:25 UTC (permalink / raw)
To: Saranya Sivakumar <sarlavk@yahoo.com>; +Cc: pgsqlnovice <pgsql-novice@postgresql.org>; pgsql-performance
Hi, Saranya,
Saranya Sivakumar wrote:
> Thank you very much for the information. The SHMMAX was set to 33554432,
> and that's why it failed to start the postmaster. Thanks for the link to
> the kernel resources article. I guess changing these parameters would
> require recompiling the kernel.
As stated on the
http://www.postgresql.org/docs/8.1/interactive/kernel-resources.html
page, those values can be changed via sysctl or echoing values into
/proc, under linux at least.
HTH,
Markus
--
Markus Schaber | Logical Tracking&Tracing International AG
Dipl. Inf. | Software Development GIS
Fight against software patents in EU! www.ffii.org www.nosoftwarepatents.org
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: [PERFORM] 7.3.2 pg_restore very slow
@ 2006-08-07 19:18 Richard Broersma Jr <rabroersma@yahoo.com>
parent: Saranya Sivakumar <sarlavk@yahoo.com>
1 sibling, 1 reply; 9+ messages in thread
From: Richard Broersma Jr @ 2006-08-07 19:18 UTC (permalink / raw)
To: Saranya Sivakumar <sarlavk@yahoo.com>; pgsqlnovice <pgsql-novice@postgresql.org>; pgsql-performance
> Thank you very much for the information. The SHMMAX was set to 33554432, and that's why it
> failed to start the postmaster. Thanks for the link to the kernel resources article. I guess
> changing these parameters would require recompiling the kernel.
>
> Is there any work around without changing these parameters to make maximum use of RAM?
>
> We got a new server now with 2GB RAM, but it also has the same value for SHMMAX.
>
> And I am trying the restore with the following conf
> sort_mem = 40960 (changed from 1024)
> shared_buffers = 3000 (changed from 64)
> max_connections = 128 (changed from 32)
This is one of the best links that I can give you in addition to the Postgresql Kernel resource
link.
http://www.powerpostgresql.com/PerfList
I am pretty much a beginner at resource tuning also. In fact, after googling for sources that
describe how to tune kernel parameters, the postgresql documents remains the best documents I've
found so far.
I would be interested if anyone else on the list knows of any resources or books that have an in
depth discussion on methods/strategies to tune kernel parameters to maximized usage of system
resources and at the same time allow for harmonious sharing between various programs/services.
Regards,
Richard Broersma Jr.
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: [PERFORM] 7.3.2 pg_restore very slow
@ 2006-08-07 20:07 Saranya Sivakumar <sarlavk@yahoo.com>
parent: Richard Broersma Jr <rabroersma@yahoo.com>
0 siblings, 0 replies; 9+ messages in thread
From: Saranya Sivakumar @ 2006-08-07 20:07 UTC (permalink / raw)
To: Richard Broersma Jr <rabroersma@yahoo.com>; pgsqlnovice <pgsql-novice@postgresql.org>; pgsql-performance
Hi All,
Thanks Richard for the additional link. The information is very useful.
The restore completed successfully in 2.5 hours in the new 2GB box, with the same configuration parameters. I think if I can tweak the parameters a little more, I should be able to get it down to the 1 hr down time that we can afford.
Thanks again for all the help.
Sincerely,
Saranya
Richard Broersma Jr <rabroersma@yahoo.com> wrote:
> Thank you very much for the information. The SHMMAX was set to 33554432, and that's why it
> failed to start the postmaster. Thanks for the link to the kernel resources article. I guess
> changing these parameters would require recompiling the kernel.
>
> Is there any work around without changing these parameters to make maximum use of RAM?
>
> We got a new server now with 2GB RAM, but it also has the same value for SHMMAX.
>
> And I am trying the restore with the following conf
> sort_mem = 40960 (changed from 1024)
> shared_buffers = 3000 (changed from 64)
> max_connections = 128 (changed from 32)
This is one of the best links that I can give you in addition to the Postgresql Kernel resource
link.
http://www.powerpostgresql.com/PerfList
I am pretty much a beginner at resource tuning also. In fact, after googling for sources that
describe how to tune kernel parameters, the postgresql documents remains the best documents I've
found so far.
I would be interested if anyone else on the list knows of any resources or books that have an in
depth discussion on methods/strategies to tune kernel parameters to maximized usage of system
resources and at the same time allow for harmonious sharing between various programs/services.
Regards,
Richard Broersma Jr.
---------------------------------
Want to be your own boss? Learn how on Yahoo! Small Business.
^ permalink raw reply [nested|flat] 9+ messages in thread
end of thread, other threads:[~2006-08-07 20:07 UTC | newest]
Thread overview: 9+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2006-08-06 16:54 7.3.2 pg_restore very slow Saranya Sivakumar <sarlavk@yahoo.com>
2006-08-07 09:55 ` Richard Huxton <dev@archonet.com>
2006-08-07 15:35 ` Saranya Sivakumar <sarlavk@yahoo.com>
2006-08-07 16:08 ` Saranya Sivakumar <sarlavk@yahoo.com>
2006-08-07 16:17 ` Richard Broersma Jr <rabroersma@yahoo.com>
2006-08-07 17:28 ` Saranya Sivakumar <sarlavk@yahoo.com>
2006-08-07 18:25 ` Markus Schaber <schabi@logix-tt.com>
2006-08-07 19:18 ` Richard Broersma Jr <rabroersma@yahoo.com>
2006-08-07 20:07 ` Saranya Sivakumar <sarlavk@yahoo.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox