pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Greg Stark <gsstark@mit.edu>
To: Greg Stark <gsstark@MIT.EDU>
Cc: pgsql-hackers@postgresql.org
Subject: Re: [PATCHES] ARC Memory Usage analysis
Date: 25 Oct 2004 17:30:52 -0400
Message-ID: <87hdoiv62b.fsf@stark.xeocode.com> (raw)
In-Reply-To: <87mzyavhxc.fsf@stark.xeocode.com>
References: <1098471059.20926.49.camel@localhost.localdomain>
	<41796115.2020501@Yahoo.com>
	<20041022200955.GC4382@it.is.rice.edu>
	<417D1D01.3090401@Yahoo.com>
	<20651.1098720192@sss.pgh.pa.us>
	<87mzyavhxc.fsf@stark.xeocode.com>

> However I wonder about another approach entirely. If postgres timed how long
> reads took it shouldn't find it very hard to distinguish between a cached
> buffer being copied and an actual i/o operation. It should be able to track
> the percentage of time that buffers requested are in the kernel's cache and
> use that directly instead of the estimated cache size.

I tested this with a program that times seeking to random locations in a file.
It's pretty easy to spot the break point. There are very few fetches that take
between 50us and 1700us, probably they come from the drive's onboard cache.

The 1700us bound probably would be lower for high end server equipment with
10k RPM drives and RAID arrays. But I doubt it will ever come close to the
100us edge, not without features like cache ram that Postgres would be better
off considering to be part of "effective_cache" anyways.

So I would suggest using something like 100us as the threshold for determining
whether a buffer fetch came from cache.

Here are two graphs, one showing a nice curve showing how disk seek times are
distributed. It's neat to look at for that alone:


$ dd bs=1M count=1024 if=/dev/urandom of=/tmp/big
$ ./a.out 10000 /tmp/big > /tmp/l
$ gnuplot
gnuplot> set terminal png
gnuplot> set output "/tmp/plot1.png"
gnuplot> plot '/tmp/l2' with points pointtype 1 pointsize 1
gnuplot> set output "/tmp/plot2.png"
gnuplot> plot [0:2000] [0:1000] '/tmp/l2' with points pointtype 1 pointsize 1



-- 
greg

Attachments:

plot1.png (image/png, 3.0K) download view image
plot2.png (image/png, 2.4K) download view image
timing.c (text/x-csrc, 1.1K) download #include <unistd.h> #include <sys/types.h> #include <sys/stat.h> #include <sys/time.h> #include <fcntl.h> #include <time.h> #include <stdlib.h> #include <stdio.h> int main(int argc, char *argv[]) { int rep = atoi(argv[1]); int i; char *filename; int fd; struct stat statbuf; off_t filesize; unsigned blocksize; void *blockbuf; filename = argv[2]; fd = open(filename, O_RDONLY); fstat(fd, &statbuf); filesize = statbuf.st_size; blocksize = statbuf.st_blksize; blockbuf = malloc(blocksize); srandom(getpid()^clock()); for (i=0;i<rep;i++) { struct timeval timeval1,timeval2; off_t offset = random()%filesize / blocksize * blocksize; gettimeofday(&timeval1,NULL); lseek(fd, offset, SEEK_SET); read(fd, blockbuf, blocksize); gettimeofday(&timeval2,NULL); /* printf("Read (%d at %ld) took %ld us\n", blocksize, offset, timeval2.tv_usec-timeval1.tv_usec + (timeval2.tv_sec-timeval1.tv_sec)*1000000);*/ printf("%ld\n", timeval2.tv_usec-timeval1.tv_usec + (timeval2.tv_sec-timeval1.tv_sec)*1000000); } return 0; }

view thread (32+ messages)  latest in thread

Message-ID: <87hdoiv62b.fsf@stark.xeocode.com>
Permalink:  ../87hdoiv62b.fsf@stark.xeocode.com/
Also on:    postgresql.org/message-id/87hdoiv62b.fsf@stark.xeocode.com

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-performance@postgresql.org
  Cc: gsstark@mit.edu, pgsql-hackers@postgresql.org
  Subject: Re: [PATCHES] ARC Memory Usage analysis
  In-Reply-To: <87hdoiv62b.fsf@stark.xeocode.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox