pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedPostgreSQL Cache
7+ messages / 5 participants
[nested] [flat]
* PostgreSQL Cache
@ 2008-09-29 04:43 Matthew Pulis <mpulis@gmail.com>
2008-09-29 06:55 ` Re: [GENERAL] PostgreSQL Cache Greg Smith <gsmith@gregsmith.com>
2008-09-29 07:01 ` Re: [GENERAL] PostgreSQL Cache Joris Dobbelsteen <joris@familiedobbelsteen.nl>
0 siblings, 2 replies; 7+ messages in thread
From: Matthew Pulis @ 2008-09-29 04:43 UTC (permalink / raw)
To: pgsql-admin; pgsql-general@postgresql.org
Hi,
I need to perform some timed testing, thus need to make sure that disk cache
does not affect me. Is clearing the OS (Ubuntu) disk cache, ( by running:
sudo echo 3 | sudo tee /proc/sys/vm/drop_caches ) enough to do this? If not
can you please point me to some site please since all I am finding is such
command.
Thanks and regards
Matthew
--
Matthew Pulis
URL : http://www.solutions-lab.net
MSN : pulis_matthew[@]hotmail.com
ICQ : 145951110
Skype : solutions-lab.net
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: [GENERAL] PostgreSQL Cache
2008-09-29 04:43 PostgreSQL Cache Matthew Pulis <mpulis@gmail.com>
@ 2008-09-29 06:55 ` Greg Smith <gsmith@gregsmith.com>
2008-09-29 07:51 ` Re: [GENERAL] PostgreSQL Cache Oleg Bartunov <oleg@sai.msu.su>
2008-09-29 11:26 ` Re: PostgreSQL Cache Sam Mason <sam@samason.me.uk>
1 sibling, 2 replies; 7+ messages in thread
From: Greg Smith @ 2008-09-29 06:55 UTC (permalink / raw)
To: Matthew Pulis <mpulis@gmail.com>; +Cc: pgsql-admin; pgsql-general@postgresql.org
On Mon, 29 Sep 2008, Matthew Pulis wrote:
> I need to perform some timed testing, thus need to make sure that disk cache
> does not affect me. Is clearing the OS (Ubuntu) disk cache, ( by running:
> sudo echo 3 | sudo tee /proc/sys/vm/drop_caches ) enough to do this?
What you should do is:
1) Shutdown the database server (pg_ctl, sudo service postgresql stop,
etc.)
2) sync
3) sudo echo 3 > /proc/sys/vm/drop_caches
4) Start the database server
That will clear both the database and OS cache with a minimum of junk left
behind in the process; clearing the cache without a sync is a bad idea.
Note that all of this will still leave behind whatever cache is in your
disk controller card or on the disk themselves available. There are some
other techniques you could consider. Add a setp 2.5 that generates a
bunch of data unused by the test, then sync again, and you've turned most
of that into useless caching.
Ideally, your test should be running against a large enough data set that
the dozens or couple of hundred megabytes that might be in those will only
add a bit of noise to whatever you're testing. If you're not running a
larger test or going through tasts to make the caches clear, the only easy
way to make things more clear is to reboot the whole server.
--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: [GENERAL] PostgreSQL Cache
2008-09-29 04:43 PostgreSQL Cache Matthew Pulis <mpulis@gmail.com>
2008-09-29 06:55 ` Re: [GENERAL] PostgreSQL Cache Greg Smith <gsmith@gregsmith.com>
@ 2008-09-29 07:51 ` Oleg Bartunov <oleg@sai.msu.su>
1 sibling, 0 replies; 7+ messages in thread
From: Oleg Bartunov @ 2008-09-29 07:51 UTC (permalink / raw)
To: Greg Smith <gsmith@gregsmith.com>; +Cc: Matthew Pulis <mpulis@gmail.com>; pgsql-admin; pgsql-general@postgresql.org
A while ago I wrote a script based on Dave Plonka work
http://net.doit.wisc.edu/~plonka/fincore/
My script monitors system buffers and shared buffers
(if pg_buffercache installed) and I found it's almost useless to
check system buffers, since I got rather ridiculous numbers.
> I use it to investigate OS cacheing of PostgreSQL files and was
> surprized on 24 Gb server, total cache was about 30 Gb. How this is
> possible ?
I can send script and perl module if you want to play with.
Oleg
On Mon, 29 Sep 2008, Greg Smith wrote:
> On Mon, 29 Sep 2008, Matthew Pulis wrote:
>
>> I need to perform some timed testing, thus need to make sure that disk
>> cache
>> does not affect me. Is clearing the OS (Ubuntu) disk cache, ( by running:
>> sudo echo 3 | sudo tee /proc/sys/vm/drop_caches ) enough to do this?
>
> What you should do is:
>
> 1) Shutdown the database server (pg_ctl, sudo service postgresql stop, etc.)
> 2) sync
> 3) sudo echo 3 > /proc/sys/vm/drop_caches
> 4) Start the database server
>
> That will clear both the database and OS cache with a minimum of junk left
> behind in the process; clearing the cache without a sync is a bad idea.
>
> Note that all of this will still leave behind whatever cache is in your disk
> controller card or on the disk themselves available. There are some other
> techniques you could consider. Add a setp 2.5 that generates a bunch of data
> unused by the test, then sync again, and you've turned most of that into
> useless caching.
>
> Ideally, your test should be running against a large enough data set that the
> dozens or couple of hundred megabytes that might be in those will only add a
> bit of noise to whatever you're testing. If you're not running a larger test
> or going through tasts to make the caches clear, the only easy way to make
> things more clear is to reboot the whole server.
>
> --
> * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
>
>
Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: PostgreSQL Cache
2008-09-29 04:43 PostgreSQL Cache Matthew Pulis <mpulis@gmail.com>
2008-09-29 06:55 ` Re: [GENERAL] PostgreSQL Cache Greg Smith <gsmith@gregsmith.com>
@ 2008-09-29 11:26 ` Sam Mason <sam@samason.me.uk>
2008-09-29 21:53 ` Re: PostgreSQL Cache Greg Smith <gsmith@gregsmith.com>
1 sibling, 1 reply; 7+ messages in thread
From: Sam Mason @ 2008-09-29 11:26 UTC (permalink / raw)
To: pgsql-general@postgresql.org
On Mon, Sep 29, 2008 at 02:55:52AM -0400, Greg Smith wrote:
> On Mon, 29 Sep 2008, Matthew Pulis wrote:
> >I need to perform some timed testing, thus need to make sure that
> >disk cache does not affect me. Is clearing the OS (Ubuntu) disk
> >cache, (by running: sudo echo 3 | sudo tee /proc/sys/vm/drop_caches)
> >enough to do this?
>
> 3) sudo echo 3 > /proc/sys/vm/drop_caches
I'm not sure about the rest, but shouldn't this be:
echo 3 | sudo tee /proc/sys/vm/drop_caches
the OP's "sudo echo 3" seems redundant; "echo" doesn't need to be run
with higher privileges, only the writing process needs that. As an
aside, it would be nicer if there was a more appropriately program than
tee but I've yet to find one.
Sam
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: PostgreSQL Cache
2008-09-29 04:43 PostgreSQL Cache Matthew Pulis <mpulis@gmail.com>
2008-09-29 06:55 ` Re: [GENERAL] PostgreSQL Cache Greg Smith <gsmith@gregsmith.com>
2008-09-29 11:26 ` Re: PostgreSQL Cache Sam Mason <sam@samason.me.uk>
@ 2008-09-29 21:53 ` Greg Smith <gsmith@gregsmith.com>
2008-09-30 12:04 ` Re: PostgreSQL Cache Sam Mason <sam@samason.me.uk>
0 siblings, 1 reply; 7+ messages in thread
From: Greg Smith @ 2008-09-29 21:53 UTC (permalink / raw)
To: Sam Mason <sam@samason.me.uk>; +Cc: pgsql-general@postgresql.org
On Mon, 29 Sep 2008, Sam Mason wrote:
> On Mon, Sep 29, 2008 at 02:55:52AM -0400, Greg Smith wrote:
>> 3) sudo echo 3 > /proc/sys/vm/drop_caches
>
> I'm not sure about the rest, but shouldn't this be:
> echo 3 | sudo tee /proc/sys/vm/drop_caches
I couldn't think of any reason to actually include the tee in there and
just optimized displaying the "3" out as script noise.
> As an aside, it would be nicer if there was a more appropriately program
> than tee but I've yet to find one.
What are you trying to accomplish here that tee isn't quite right for?
--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: PostgreSQL Cache
2008-09-29 04:43 PostgreSQL Cache Matthew Pulis <mpulis@gmail.com>
2008-09-29 06:55 ` Re: [GENERAL] PostgreSQL Cache Greg Smith <gsmith@gregsmith.com>
2008-09-29 11:26 ` Re: PostgreSQL Cache Sam Mason <sam@samason.me.uk>
2008-09-29 21:53 ` Re: PostgreSQL Cache Greg Smith <gsmith@gregsmith.com>
@ 2008-09-30 12:04 ` Sam Mason <sam@samason.me.uk>
0 siblings, 0 replies; 7+ messages in thread
From: Sam Mason @ 2008-09-30 12:04 UTC (permalink / raw)
To: pgsql-general@postgresql.org
On Mon, Sep 29, 2008 at 05:53:02PM -0400, Greg Smith wrote:
> On Mon, 29 Sep 2008, Sam Mason wrote:
> > echo 3 | sudo tee /proc/sys/vm/drop_caches
> >As an aside, it would be nicer if there was a more appropriate program
> >than tee but I've yet to find one.
>
> What are you trying to accomplish here that tee isn't quite right for?
tee works for how it's being used for here; it's just wired into my
brain for tasks other than writing to files with different privileges.
I ended up with a similar invocation when I was solving a similar
problem a couple of months ago (not sure how I'd not hit it before then)
and kept searching for something more natural.
I've just done a search and using tee here appears to be somewhat
standard, so I guess that answers my question. Sorry for the noise!
Sam
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: [GENERAL] PostgreSQL Cache
2008-09-29 04:43 PostgreSQL Cache Matthew Pulis <mpulis@gmail.com>
@ 2008-09-29 07:01 ` Joris Dobbelsteen <joris@familiedobbelsteen.nl>
1 sibling, 0 replies; 7+ messages in thread
From: Joris Dobbelsteen @ 2008-09-29 07:01 UTC (permalink / raw)
To: Matthew Pulis <mpulis@gmail.com>; +Cc: pgsql-admin; pgsql-general@postgresql.org
Matthew Pulis wrote:
> Hi,
>
> I need to perform some timed testing, thus need to make sure that disk
> cache does not affect me. Is clearing the OS (Ubuntu) disk cache, ( by
> running: sudo echo 3 | sudo tee /proc/sys/vm/drop_caches ) enough to
> do this? If not can you please point me to some site please since all
> I am finding is such command.
Look for methodologies for doing performance tests. A problem is that
the disk cache is an essential part that makes up for postgresql
performance. Also do not forget about overhead and inaccuracies that you
will affect your results.
In general performance tests are a rather large simulation of how your
application would use the database. It should be large enough for many
effects (such as initial cache state) to be neglected. It only provides
an average for the performance on your system configuration.
If you run it a few times more, you can compute the variation. It
provides some insight how stable your system is in handling the workload.
- Joris
^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2008-09-30 12:04 UTC | newest]
Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2008-09-29 04:43 PostgreSQL Cache Matthew Pulis <mpulis@gmail.com>
2008-09-29 06:55 ` Greg Smith <gsmith@gregsmith.com>
2008-09-29 07:51 ` Oleg Bartunov <oleg@sai.msu.su>
2008-09-29 11:26 ` Sam Mason <sam@samason.me.uk>
2008-09-29 21:53 ` Greg Smith <gsmith@gregsmith.com>
2008-09-30 12:04 ` Sam Mason <sam@samason.me.uk>
2008-09-29 07:01 ` Joris Dobbelsteen <joris@familiedobbelsteen.nl>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox