pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedPostgreSQL to host e-mail?
8+ messages / 7 participants
[nested] [flat]
* PostgreSQL to host e-mail?
@ 2007-01-04 18:00 Charles A. Landemaine <landemaine@gmail.com>
2007-01-04 18:04 ` PostgreSQL to host e-mail? Charles A. Landemaine <landemaine@gmail.com>
2007-01-04 18:08 ` Re: PostgreSQL to host e-mail? Frank Wiles <frank@wiles.org>
2007-01-04 18:11 ` Re: PostgreSQL to host e-mail? Joshua D. Drake <jd@commandprompt.com>
2007-01-05 03:10 ` Re: PostgreSQL to host e-mail? Grega Bremec <gregab@p0f.net>
0 siblings, 4 replies; 8+ messages in thread
From: Charles A. Landemaine @ 2007-01-04 18:00 UTC (permalink / raw)
To: pgsql-performance
I'm building an e-mail service that has two requirements: It should
index messages on the fly to have lightening search results, and it
should be able to handle large amounts of space. The server is going
to be dedicated only for e-mail with 250GB of storage in Raid-5. I'd
like to know how PostgreSQL could handle such a large amount of data.
How much RAM would I need? I expect my users to have a 10GB quota per
e-mail account.
Thanks for your advice,
--
Charles A. Landemaine.
^ permalink raw reply [nested|flat] 8+ messages in thread
* PostgreSQL to host e-mail?
2007-01-04 18:00 PostgreSQL to host e-mail? Charles A. Landemaine <landemaine@gmail.com>
@ 2007-01-04 18:04 ` Charles A. Landemaine <landemaine@gmail.com>
3 siblings, 0 replies; 8+ messages in thread
From: Charles A. Landemaine @ 2007-01-04 18:04 UTC (permalink / raw)
To: pgsql-performance
I'm building an e-mail service that has two requirements: It should
index messages on the fly to have lightening search results, and it
should be able to handle large amounts of space. The server is going
to be dedicated only for e-mail with 250GB of storage in Raid-5. I'd
like to know how PostgreSQL could handle such a large amount of data.
How much RAM would I need? I expect my users to have a 10GB quota per
e-mail account.
Thanks for your advice,
--
Charles A. Landemaine.
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: PostgreSQL to host e-mail?
2007-01-04 18:00 PostgreSQL to host e-mail? Charles A. Landemaine <landemaine@gmail.com>
@ 2007-01-04 18:08 ` Frank Wiles <frank@wiles.org>
2007-01-04 18:42 ` Re: PostgreSQL to host e-mail? Matthew Schumacher <matt.s@aptalaska.net>
3 siblings, 1 reply; 8+ messages in thread
From: Frank Wiles @ 2007-01-04 18:08 UTC (permalink / raw)
To: Charles A. Landemaine <landemaine@gmail.com>; +Cc: pgsql-performance
On Thu, 4 Jan 2007 15:00:05 -0300
"Charles A. Landemaine" <landemaine@gmail.com> wrote:
> I'm building an e-mail service that has two requirements: It should
> index messages on the fly to have lightening search results, and it
> should be able to handle large amounts of space. The server is going
> to be dedicated only for e-mail with 250GB of storage in Raid-5. I'd
> like to know how PostgreSQL could handle such a large amount of data.
> How much RAM would I need? I expect my users to have a 10GB quota per
> e-mail account.
Well this is a bit like asking "what's the top speed for a van that
can carry 8 people", it really isn't enough information to be able
to give you a good answer.
It depends on everything from the data model you use to represent
your E-mail messages, to configuration, to hardware speeds, etc.
In general, you want as much RAM as you can afford for the project,
the more the better. I'd say 2-4GB is the minimum. And RAID-5 isn't
very good for database work in general, you'll get better performance
from RAID 1+0.
---------------------------------
Frank Wiles <frank@wiles.org>
http://www.wiles.org
---------------------------------
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: PostgreSQL to host e-mail?
2007-01-04 18:00 PostgreSQL to host e-mail? Charles A. Landemaine <landemaine@gmail.com>
2007-01-04 18:08 ` Re: PostgreSQL to host e-mail? Frank Wiles <frank@wiles.org>
@ 2007-01-04 18:42 ` Matthew Schumacher <matt.s@aptalaska.net>
0 siblings, 0 replies; 8+ messages in thread
From: Matthew Schumacher @ 2007-01-04 18:42 UTC (permalink / raw)
To: pgsql-performance
Frank Wiles wrote:
> On Thu, 4 Jan 2007 15:00:05 -0300
> "Charles A. Landemaine" <landemaine@gmail.com> wrote:
>
>> I'm building an e-mail service that has two requirements: It should
>> index messages on the fly to have lightening search results, and it
>> should be able to handle large amounts of space. The server is going
>> to be dedicated only for e-mail with 250GB of storage in Raid-5. I'd
>> like to know how PostgreSQL could handle such a large amount of data.
>> How much RAM would I need? I expect my users to have a 10GB quota per
>> e-mail account.
>
I wouldn't do it this way, I would use cyrus. It stores the messages in
plain text each in it's own file (like maildir) but then it also indexes
the headers in bdb format and also can index the search database in bdb
format. The result is a very simple mail store that can perform
searches very fast. The only problem is that the search index in only
updated periodically.
If you need more information then look at the cyrus-users list as this
is WAY off topic.
schu
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: PostgreSQL to host e-mail?
2007-01-04 18:00 PostgreSQL to host e-mail? Charles A. Landemaine <landemaine@gmail.com>
@ 2007-01-04 18:11 ` Joshua D. Drake <jd@commandprompt.com>
3 siblings, 0 replies; 8+ messages in thread
From: Joshua D. Drake @ 2007-01-04 18:11 UTC (permalink / raw)
To: Charles A. Landemaine <landemaine@gmail.com>; +Cc: pgsql-performance
On Thu, 2007-01-04 at 15:00 -0300, Charles A. Landemaine wrote:
> I'm building an e-mail service that has two requirements: It should
> index messages on the fly to have lightening search results, and it
> should be able to handle large amounts of space. The server is going
> to be dedicated only for e-mail with 250GB of storage in Raid-5.
Well Raid 5 is likely a mistake. Consider RAID 10.
> I'd
> like to know how PostgreSQL could handle such a large amount of data.
250GB is not really that much data for PostgreSQL I have customers with
much larger data sets.
> How much RAM would I need?
Lots... which is about all I can tell you without more information. How
many customers? Are you using table partitioning? How will you be
searching? Full text or regex?
Joshua D. Drake
> I expect my users to have a 10GB quota per
> e-mail account.
> Thanks for your advice,
>
--
=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: PostgreSQL to host e-mail?
2007-01-04 18:00 PostgreSQL to host e-mail? Charles A. Landemaine <landemaine@gmail.com>
@ 2007-01-05 03:10 ` Grega Bremec <gregab@p0f.net>
2007-01-05 18:15 ` Re: PostgreSQL to host e-mail? Reid Thompson <Reid.Thompson@ateb.com>
3 siblings, 1 reply; 8+ messages in thread
From: Grega Bremec @ 2007-01-05 03:10 UTC (permalink / raw)
To: Charles A. Landemaine <landemaine@gmail.com>; +Cc: pgsql-performance
-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160
Charles A. Landemaine wrote:
| I'm building an e-mail service that has two requirements: It should
| index messages on the fly to have lightening search results, and it
| should be able to handle large amounts of space. The server is going
| to be dedicated only for e-mail with 250GB of storage in Raid-5. I'd
| like to know how PostgreSQL could handle such a large amount of data.
| How much RAM would I need? I expect my users to have a 10GB quota per
| e-mail account.
| Thanks for your advice,
|
Hello, Charles.
I'll second people's suggestions to stay away from RAID5; the kind of
workload a mail storage will have is one that is approximately an even mix
of writes (in database terms, INSERTs, UPDATEs and DELETEs) and reads, and
we all know RAID5 is a loser when it comes to writing a lot, at least when
you're building arrays with less than 10-15 drives. I'd suggest you go for
RAID10 for the database cluster and an extra drive for WAL.
Another point of interest I'd like to mention is one particular aspect of
the workflow of an e-mail user: we will typically touch the main inbox a lot
and leave most of the other folders pretty much intact for most of the time.
This suggests per-inbox quota might be useful, maybe in addition to the
overall quota, because then you can calculate your database working set more
easily, based on usage statistics for a typical account. Namely, if the
maximum size of an inbox is x MB, with y% average utilization, and you plan
for z users, of which w% will be typically active in one day, your database
working set will be somewhere in the general area of (x * y%) * (z * w%) MB.
Add to that the size of the indexes you create, and you have a very
approximate idea of the amount of RAM you need to place in your machines to
keep your performance from becoming I/O-bound.
The main reason I'm writing this mail though, is to suggest you take a look
at Oryx, http://www.oryx.com/; They used to have this product called
Mailstore, which was designed to be a mail store using PostgreSQL as a
backend, and has since evolved to a bit more than just that, it seems.
Perhaps it could be of help to you while building your system, and I'm sure
the people at Oryx will be glad to hear from you while, and after you've
built your system.
Kind regards,
- --
~ Grega Bremec
~ gregab at p0f dot net
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
iD8DBQFFncGcfu4IwuB3+XoRA9Y9AJ0WA+0aooVvGMOpQXGStzkRNVDCjwCeNdfs
CArTFwo6geR1oRBFDzFRY/U=
=Y1Lf
-----END PGP SIGNATURE-----
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: PostgreSQL to host e-mail?
2007-01-04 18:00 PostgreSQL to host e-mail? Charles A. Landemaine <landemaine@gmail.com>
2007-01-05 03:10 ` Re: PostgreSQL to host e-mail? Grega Bremec <gregab@p0f.net>
@ 2007-01-05 18:15 ` Reid Thompson <Reid.Thompson@ateb.com>
2007-01-10 20:18 ` Re: PostgreSQL to host e-mail? Jim C. Nasby <jim@nasby.net>
0 siblings, 1 reply; 8+ messages in thread
From: Reid Thompson @ 2007-01-05 18:15 UTC (permalink / raw)
To: Grega Bremec <gregab@p0f.net>; +Cc: Charles A. Landemaine <landemaine@gmail.com>; pgsql-performance
On Fri, 2007-01-05 at 04:10 +0100, Grega Bremec wrote:
> he main reason I'm writing this mail though, is to suggest you take a
> look
> at Oryx, http://www.oryx.com/; They used to have this product called
> Mailstore, which was designed to be a mail store using PostgreSQL as a
> backend, and has since evolved to a bit more than just that, it seems.
> Perhaps it could be of help to you while building your system, and I'm
> sure
> the people at Oryx will be glad to hear from you while, and after
> you've
> built your system.
>
> Kind regards,
> --
> ~ Grega Bremec
re above...
http://www.archiveopteryx.org/1.10.html
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: PostgreSQL to host e-mail?
2007-01-04 18:00 PostgreSQL to host e-mail? Charles A. Landemaine <landemaine@gmail.com>
2007-01-05 03:10 ` Re: PostgreSQL to host e-mail? Grega Bremec <gregab@p0f.net>
2007-01-05 18:15 ` Re: PostgreSQL to host e-mail? Reid Thompson <Reid.Thompson@ateb.com>
@ 2007-01-10 20:18 ` Jim C. Nasby <jim@nasby.net>
0 siblings, 0 replies; 8+ messages in thread
From: Jim C. Nasby @ 2007-01-10 20:18 UTC (permalink / raw)
To: Reid Thompson <Reid.Thompson@ateb.com>; +Cc: Grega Bremec <gregab@p0f.net>; Charles A. Landemaine <landemaine@gmail.com>; pgsql-performance
On Fri, Jan 05, 2007 at 01:15:44PM -0500, Reid Thompson wrote:
> On Fri, 2007-01-05 at 04:10 +0100, Grega Bremec wrote:
> > he main reason I'm writing this mail though, is to suggest you take a
> > look
> > at Oryx, http://www.oryx.com/; They used to have this product called
> > Mailstore, which was designed to be a mail store using PostgreSQL as a
> > backend, and has since evolved to a bit more than just that, it seems.
> > Perhaps it could be of help to you while building your system, and I'm
> > sure
> > the people at Oryx will be glad to hear from you while, and after
> > you've
> > built your system.
> >
> > Kind regards,
> > --
> > ~ Grega Bremec
> re above...
> http://www.archiveopteryx.org/1.10.html
You should also look at http://dbmail.org/ , which runs on several
databases (PostgreSQL included).
--
Jim Nasby jim@nasby.net
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
^ permalink raw reply [nested|flat] 8+ messages in thread
end of thread, other threads:[~2007-01-10 20:18 UTC | newest]
Thread overview: 8+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2007-01-04 18:00 PostgreSQL to host e-mail? Charles A. Landemaine <landemaine@gmail.com>
2007-01-04 18:04 ` Charles A. Landemaine <landemaine@gmail.com>
2007-01-04 18:08 ` Frank Wiles <frank@wiles.org>
2007-01-04 18:42 ` Matthew Schumacher <matt.s@aptalaska.net>
2007-01-04 18:11 ` Joshua D. Drake <jd@commandprompt.com>
2007-01-05 03:10 ` Grega Bremec <gregab@p0f.net>
2007-01-05 18:15 ` Reid Thompson <Reid.Thompson@ateb.com>
2007-01-10 20:18 ` Jim C. Nasby <jim@nasby.net>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox