pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedhow to (temporarily) disable/minimize benefits of disk block cache or postgresql shared buffer
6+ messages / 3 participants
[nested] [flat]
* how to (temporarily) disable/minimize benefits of disk block cache or postgresql shared buffer
@ 2010-07-01 09:41 Rajesh Kumar Mallah <mallah.rajesh@gmail.com>
0 siblings, 1 reply; 6+ messages in thread
From: Rajesh Kumar Mallah @ 2010-07-01 09:41 UTC (permalink / raw)
To: pgsql-performance
Hi,
this is not really a performance question , sorry if its bit irrelevant
to be posted here. We have a development environment and we want
to optimize the non-database parts of the application. The problem is
that subsequent run of queries are execute very fast and makes the
performance analysis a trivial problem. We want that the subsequent runs
of query should take similar times as the first run so that we can work
on the optimizing the calling patterns to the database.
regds
Rajesh Kumar Mallah.
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: how to (temporarily) disable/minimize benefits of disk block cache or postgresql shared buffer
@ 2010-07-01 16:37 Craig Ringer <craig@postnewspapers.com.au>
parent: Rajesh Kumar Mallah <mallah.rajesh@gmail.com>
0 siblings, 1 reply; 6+ messages in thread
From: Craig Ringer @ 2010-07-01 16:37 UTC (permalink / raw)
To: Rajesh Kumar Mallah <mallah.rajesh@gmail.com>; +Cc: pgsql-performance
On 01/07/10 17:41, Rajesh Kumar Mallah wrote:
> Hi,
>
> this is not really a performance question , sorry if its bit irrelevant
> to be posted here. We have a development environment and we want
> to optimize the non-database parts of the application. The problem is
> that subsequent run of queries are execute very fast and makes the
> performance analysis a trivial problem. We want that the subsequent runs
> of query should take similar times as the first run so that we can work
> on the optimizing the calling patterns to the database.
You can get rid of PostgreSQL's caches in shared_buffers by restarting
the PostgreSQL server. I don't know if there's any more convenient way.
Alternately, just set a really minimal shared_buffers that's just enough
for your connections so there's not much room for cached data.
If you are running a Linux server (as you didn't mention what you're
running on) you can drop the OS disk cache quite easily:
http://linux-mm.org/Drop_Caches
http://www.linuxinsight.com/proc_sys_vm_drop_caches.html
AFAIK for most other platforms you have to use a tool that gobbles
memory to force caches out. On Windows, most of those garbage tools that
claim to "free" memory do this - it's about the only time you'd ever
want to use one, since they do such horrid things to performance.
--
Craig Ringer
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: how to (temporarily) disable/minimize benefits of disk block cache or postgresql shared buffer
@ 2010-07-01 17:59 Rajesh Kumar Mallah <mallah.rajesh@gmail.com>
parent: Craig Ringer <craig@postnewspapers.com.au>
0 siblings, 1 reply; 6+ messages in thread
From: Rajesh Kumar Mallah @ 2010-07-01 17:59 UTC (permalink / raw)
To: Craig Ringer <craig@postnewspapers.com.au>; +Cc: pgsql-performance
On Thu, Jul 1, 2010 at 10:07 PM, Craig Ringer
<craig@postnewspapers.com.au>wrote:
> On 01/07/10 17:41, Rajesh Kumar Mallah wrote:
> > Hi,
> >
> > this is not really a performance question , sorry if its bit irrelevant
> > to be posted here. We have a development environment and we want
> > to optimize the non-database parts of the application. The problem is
> > that subsequent run of queries are execute very fast and makes the
> > performance analysis a trivial problem. We want that the subsequent runs
> > of query should take similar times as the first run so that we can work
> > on the optimizing the calling patterns to the database.
>
> You can get rid of PostgreSQL's caches in shared_buffers by restarting
> the PostgreSQL server. I don't know if there's any more convenient way.
> Alternately, just set a really minimal shared_buffers that's just enough
> for your connections so there's not much room for cached data.
>
> I had set it to 128kb
it does not really work , i even tried your next suggestion. I am in
virtualized
environment particularly OpenVz. where echo 3 > /proc/sys/vm/drop_caches
does not work inside the virtual container, i did it in the hardware node
but still does not give desired result.
regds
Rajesh Kumar Mallah.
> If you are running a Linux server (as you didn't mention what you're
> running on) you can drop the OS disk cache quite easily:
>
> http://linux-mm.org/Drop_Caches
> http://www.linuxinsight.com/proc_sys_vm_drop_caches.html
>
> AFAIK for most other platforms you have to use a tool that gobbles
> memory to force caches out. On Windows, most of those garbage tools that
> claim to "free" memory do this - it's about the only time you'd ever
> want to use one, since they do such horrid things to performance.
>
> --
> Craig Ringer
>
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: how to (temporarily) disable/minimize benefits of disk block cache or postgresql shared buffer
@ 2010-07-02 03:00 Craig Ringer <craig@postnewspapers.com.au>
parent: Rajesh Kumar Mallah <mallah.rajesh@gmail.com>
0 siblings, 1 reply; 6+ messages in thread
From: Craig Ringer @ 2010-07-02 03:00 UTC (permalink / raw)
To: Rajesh Kumar Mallah <mallah.rajesh@gmail.com>; +Cc: pgsql-performance
On 02/07/10 01:59, Rajesh Kumar Mallah wrote:
> I had set it to 128kb
> it does not really work , i even tried your next suggestion. I am in
> virtualized
> environment particularly OpenVz. where echo 3 > /proc/sys/vm/drop_caches
> does not work inside the virtual container, i did it in the hardware node
> but still does not give desired result.
Yeah, if you're in a weird virtualized environment like that you're
likely to have problems, because caching can be done at multiple levels.
In the case of OpenVZ, it's hard to know what the "guest" and what the
"host" even is sometimes, and I wouldn't trust it to respect things like
the Linux VM cache management.
You might have to fall back on the classic method: a program that tries
to allocate as much RAM as it can. On Linux this is EXTREMELY unsafe
unless you ensure you have vm overcommit disabled (see the postgresql
docs) because by default Linux systems will never fail a memory
allocation - instead they'll go looking for a big process to kill to
free some memory. In theory this should be your memory gobbler program,
but in reality the OOM killer isn't so predictable.
So: try turning vm overcommit off, then writing (or finding) a simple
program that keeps on malloc()ing memory until an allocation call fails.
That should force any caches out, freeing you for another cold run.
Note that this method won't just force out the obvious caches like
postgresql data files. It also forces out things like caches of running
binaries. Things will grind to an absolute crawl for a minute or two
before resuming normal speed, because *everything* has to come back from
disk at once. The same is true of using /proc/sys/vm/drop_caches to drop
all caches.
I guess, in the end, nothing really subtitutes for a good reboot.
--
Craig Ringer
Tech-related writing: http://soapyfrogs.blogspot.com/
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: how to (temporarily) disable/minimize benefits of disk block cache or postgresql shared buffer
@ 2010-07-02 19:22 Rajesh Kumar Mallah <mallah.rajesh@gmail.com>
parent: Craig Ringer <craig@postnewspapers.com.au>
0 siblings, 1 reply; 6+ messages in thread
From: Rajesh Kumar Mallah @ 2010-07-02 19:22 UTC (permalink / raw)
To: Craig Ringer <craig@postnewspapers.com.au>; +Cc: pgsql-performance
Dear Criag,
Thanks for thinking about it.I do not understand why u feel OpenVz is weird.
at the most its not very popular. But lets not get into that debate as its
not
the proper forum. From your reply i understand that there is not a easy and
clean way of doing it. Since performance related profiling requires multiple
iterations it is not feasible to reboot the machine. I think i will try to
profile
my code using new and unique input parameters in each iteration, this shall
roughly serve my purpose.
On Fri, Jul 2, 2010 at 8:30 AM, Craig Ringer <craig@postnewspapers.com.au>wrote:
> On 02/07/10 01:59, Rajesh Kumar Mallah wrote:
>
> > I had set it to 128kb
> > it does not really work , i even tried your next suggestion. I am in
> > virtualized
> > environment particularly OpenVz. where echo 3 > /proc/sys/vm/drop_caches
> > does not work inside the virtual container, i did it in the hardware node
> > but still does not give desired result.
>
> Yeah, if you're in a weird virtualized environment like that you're
> likely to have problems, because caching can be done at multiple levels.
> In the case of OpenVZ, it's hard to know what the "guest" and what the
> "host" even is sometimes, and I wouldn't trust it to respect things like
> the Linux VM cache management.
>
> You might have to fall back on the classic method: a program that tries
> to allocate as much RAM as it can. On Linux this is EXTREMELY unsafe
> unless you ensure you have vm overcommit disabled (see the postgresql
> docs) because by default Linux systems will never fail a memory
> allocation - instead they'll go looking for a big process to kill to
> free some memory. In theory this should be your memory gobbler program,
> but in reality the OOM killer isn't so predictable.
>
> So: try turning vm overcommit off, then writing (or finding) a simple
> program that keeps on malloc()ing memory until an allocation call fails.
> That should force any caches out, freeing you for another cold run.
>
> Note that this method won't just force out the obvious caches like
> postgresql data files. It also forces out things like caches of running
> binaries. Things will grind to an absolute crawl for a minute or two
> before resuming normal speed, because *everything* has to come back from
> disk at once. The same is true of using /proc/sys/vm/drop_caches to drop
> all caches.
>
> I guess, in the end, nothing really subtitutes for a good reboot.
>
> --
> Craig Ringer
>
> Tech-related writing: http://soapyfrogs.blogspot.com/
>
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: how to (temporarily) disable/minimize benefits of disk block cache or postgresql shared buffer
@ 2010-07-05 09:55 Matthew Wakeling <matthew@flymine.org>
parent: Rajesh Kumar Mallah <mallah.rajesh@gmail.com>
0 siblings, 0 replies; 6+ messages in thread
From: Matthew Wakeling @ 2010-07-05 09:55 UTC (permalink / raw)
To: Rajesh Kumar Mallah <mallah.rajesh@gmail.com>; +Cc: Craig Ringer <craig@postnewspapers.com.au>; pgsql-performance
> On Fri, Jul 2, 2010 at 8:30 AM, Craig Ringer <craig@postnewspapers.com.au>wrote:
>> Yeah, if you're in a weird virtualized environment like that you're
>> likely to have problems...
On Sat, 3 Jul 2010, Rajesh Kumar Mallah wrote:
> Thanks for thinking about it.I do not understand why u feel OpenVz is weird.
> at the most its not very popular.
It's not OpenVz that is wierd, but virtualisation in general. If you are
running in a virtual machine, then all sorts of things will not run as
well as expected.
Matthew
--
Contrary to popular belief, Unix is user friendly. It just happens to be
very selective about who its friends are. -- Kyle Hearn
^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2010-07-05 09:55 UTC | newest]
Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2010-07-01 09:41 how to (temporarily) disable/minimize benefits of disk block cache or postgresql shared buffer Rajesh Kumar Mallah <mallah.rajesh@gmail.com>
2010-07-01 16:37 ` Re: how to (temporarily) disable/minimize benefits of disk block cache or postgresql shared buffer Craig Ringer <craig@postnewspapers.com.au>
2010-07-01 17:59 ` Rajesh Kumar Mallah <mallah.rajesh@gmail.com>
2010-07-02 03:00 ` Craig Ringer <craig@postnewspapers.com.au>
2010-07-02 19:22 ` Rajesh Kumar Mallah <mallah.rajesh@gmail.com>
2010-07-05 09:55 ` Matthew Wakeling <matthew@flymine.org>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox