pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedInfinite Cache
9+ messages / 7 participants
[nested] [flat]
* Infinite Cache
@ 2011-07-01 14:43 Anthony Presley <anthony@resolution.com>
0 siblings, 2 replies; 9+ messages in thread
From: Anthony Presley @ 2011-07-01 14:43 UTC (permalink / raw)
To: pgsql-performance
All:
Was curious if there was some sort of Open Source version of Infinite Cache,
and/or a memcache layer that can be "dropped" in front of PostgreSQL without
application changes (which seems to be the "key" piece of Infinite Cache),
or is this something that EnterpriseDB owns and you have to buy their
version of the software to use?
I'm fine with piecing together a few different OS projects, but would prefer
to not modify the app too much.
Thanks!
--
Anthony Presley
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Infinite Cache
@ 2011-07-01 15:54 Greg Smith <greg@2ndQuadrant.com>
parent: Anthony Presley <anthony@resolution.com>
1 sibling, 0 replies; 9+ messages in thread
From: Greg Smith @ 2011-07-01 15:54 UTC (permalink / raw)
To: pgsql-performance
On 07/01/2011 10:43 AM, Anthony Presley wrote:
> Was curious if there was some sort of Open Source version of Infinite
> Cache, and/or a memcache layer that can be "dropped" in front of
> PostgreSQL without application changes (which seems to be the "key"
> piece of Infinite Cache), or is this something that EnterpriseDB owns
> and you have to buy their version of the software to use?
The best solution available for this class of problem is pgmemcache:
http://pgfoundry.org/projects/pgmemcache/
There's not too much documentation about it around, but you'll find an
intro talk at http://projects.2ndquadrant.com/char10 I found helpful
when Hannu presented it. It does take some work to utilize, including
application code changes. The hardest part of which is usually making
sure the key hashing scheme it uses to identify re-usable queries is
useful to you. And that isn't always the case.
This approach scales better than "Infinite Cache" because you can move
the whole mess onto another server optimized to be a caching system.
Those systems have a very different set of trade-offs and
correspondingly economics than a database server must have. The cache
system can be a cheap box with a bunch of RAM, that's it. And the read
traffic it avoids passing to the server really doesn't touch the
database at all, which is way better than going to the database but
being serviced quickly.
Everyone would prefer performance improvements that don't involve any
modification of their application. The unfortunate reality of database
design is that any server tuning can only provide a modest gain; if you
make things twice as fast you've done a great job. Whereas when doing
application redesign for better performance, I aim for a 10X speedup and
often do much better than that.
--
Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD
Comprehensive and Customized PostgreSQL Training Classes:
http://www.2ndquadrant.us/postgresql-training/
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Infinite Cache
@ 2011-07-01 22:37 Jim Nasby <jim@nasby.net>
parent: Anthony Presley <anthony@resolution.com>
1 sibling, 3 replies; 9+ messages in thread
From: Jim Nasby @ 2011-07-01 22:37 UTC (permalink / raw)
To: Anthony Presley <anthony@resolution.com>; +Cc: pgsql-performance
On Jul 1, 2011, at 9:43 AM, Anthony Presley wrote:
> Was curious if there was some sort of Open Source version of Infinite Cache, and/or a memcache layer that can be "dropped" in front of PostgreSQL without application changes (which seems to be the "key" piece of Infinite Cache), or is this something that EnterpriseDB owns and you have to buy their version of the software to use?
There had been some talk at one point about getting the backend-changes to support Infinite Cache into mainline Postgres. If that ever happened you could build your own version of it.
BTW, thanks to the compression feature of IC I've heard it can actually be beneficial to run it on the same server.
--
Jim C. Nasby, Database Architect jim@nasby.net
512.569.9461 (cell) http://jim.nasby.net
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Infinite Cache
@ 2011-07-02 08:20 Dave Page <dpage@pgadmin.org>
parent: Jim Nasby <jim@nasby.net>
2 siblings, 0 replies; 9+ messages in thread
From: Dave Page @ 2011-07-02 08:20 UTC (permalink / raw)
To: Jim Nasby <jim@nasby.net>; +Cc: Anthony Presley <anthony@resolution.com>; pgsql-performance
On Fri, Jul 1, 2011 at 11:37 PM, Jim Nasby <jim@nasby.net> wrote:
> On Jul 1, 2011, at 9:43 AM, Anthony Presley wrote:
>> Was curious if there was some sort of Open Source version of Infinite Cache, and/or a memcache layer that can be "dropped" in front of PostgreSQL without application changes (which seems to be the "key" piece of Infinite Cache), or is this something that EnterpriseDB owns and you have to buy their version of the software to use?
>
> There had been some talk at one point about getting the backend-changes to support Infinite Cache into mainline Postgres. If that ever happened you could build your own version of it.
>
> BTW, thanks to the compression feature of IC I've heard it can actually be beneficial to run it on the same server.
Correct.
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Infinite Cache
@ 2011-07-03 11:21 Magnus Hagander <magnus@hagander.net>
parent: Jim Nasby <jim@nasby.net>
2 siblings, 1 reply; 9+ messages in thread
From: Magnus Hagander @ 2011-07-03 11:21 UTC (permalink / raw)
To: Jim Nasby <jim@nasby.net>; +Cc: Anthony Presley <anthony@resolution.com>; pgsql-performance
On Sat, Jul 2, 2011 at 00:37, Jim Nasby <jim@nasby.net> wrote:
> On Jul 1, 2011, at 9:43 AM, Anthony Presley wrote:
>> Was curious if there was some sort of Open Source version of Infinite Cache, and/or a memcache layer that can be "dropped" in front of PostgreSQL without application changes (which seems to be the "key" piece of Infinite Cache), or is this something that EnterpriseDB owns and you have to buy their version of the software to use?
>
> There had been some talk at one point about getting the backend-changes to support Infinite Cache into mainline Postgres. If that ever happened you could build your own version of it.
I was at one point told that *all* of infinite cache would be
submitted to the community, but it was in need of some cleanup first.
But by now I think that decision has been changed - I certainly hope
it didn't take years to clean up ;) So I wouldn't hold my breath for
that one.
--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Infinite Cache
@ 2011-07-03 18:18 Greg Smith <greg@2ndQuadrant.com>
parent: Jim Nasby <jim@nasby.net>
2 siblings, 0 replies; 9+ messages in thread
From: Greg Smith @ 2011-07-03 18:18 UTC (permalink / raw)
To: pgsql-performance
On 07/01/2011 06:37 PM, Jim Nasby wrote:
> BTW, thanks to the compression feature of IC I've heard it can
> actually be beneficial to run it on the same server.
Sure, its implementation works in a way that helps improve performance
on the database server. My point was that I'd be shocked if it were
even possible to double performance if you use it. Whereas putting a
pgmemcache server in front of the database can do much better than that,
on a system that reads the same things many times per update. "Infinite
Cache" is a useful technology and the fact that it work transparently
the application is a nice benefit of EDB's commercial product. But it's
usually the case that if you really want to do the best possible
implementation of an approach, optimizing very specifically for your
application is what's needed.
--
Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD
Comprehensive and Customized PostgreSQL Training Classes:
http://www.2ndquadrant.us/postgresql-training/
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Infinite Cache
@ 2011-07-05 13:35 Shaun Thomas <sthomas@peak6.com>
parent: Magnus Hagander <magnus@hagander.net>
0 siblings, 1 reply; 9+ messages in thread
From: Shaun Thomas @ 2011-07-05 13:35 UTC (permalink / raw)
To: Magnus Hagander <magnus@hagander.net>; +Cc: Jim Nasby <jim@nasby.net>; Anthony Presley <anthony@resolution.com>; pgsql-performance
On 07/03/2011 06:21 AM, Magnus Hagander wrote:
> I was at one point told that *all* of infinite cache would be
> submitted to the community, but it was in need of some cleanup
> first.
I'm not sure what kind of cleanup would be involved, but we had some
problems with index corruption that wasn't fixed until a February patch
was applied. My guess is that earlier versions of Infinite Cache weren't
all that careful with verifying certain edge cases during connection
renegotiation or timeout scenarios. It only seemed to pop up once every
few billion queries, but that's all it takes to bring down a heavy OLTP
system.
I'd say it's probably safe enough these days. But it's also one of those
exclusive selling points they're using right now to garner EDB
customers. So I doubt it'll be released any time *soon*, though may make
it eventually.
--
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd. | Suite 800 | Chicago IL, 60604
312-676-8870
sthomas@peak6.com
______________________________________________
See http://www.peak6.com/email_disclaimer.php
for terms and conditions related to this email
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Infinite Cache
@ 2011-07-08 13:34 Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
parent: Shaun Thomas <sthomas@peak6.com>
0 siblings, 1 reply; 9+ messages in thread
From: Heikki Linnakangas @ 2011-07-08 13:34 UTC (permalink / raw)
To: sthomas@peak6.com; +Cc: Magnus Hagander <magnus@hagander.net>; Jim Nasby <jim@nasby.net>; Anthony Presley <anthony@resolution.com>; pgsql-performance
On 05.07.2011 16:35, Shaun Thomas wrote:
> I'd say it's probably safe enough these days. But it's also one of those
> exclusive selling points they're using right now to garner EDB
> customers. So I doubt it'll be released any time *soon*, though may make
> it eventually.
I doubt the community would want it even if it was open sourced. As an
open source project, what would probably make more sense is a similar
caching mechanism built into the kernel, somewhere between the
filesystem cache and user-space. That way any you could use it with any
application that benefits from the kind of large cache that Infinite
Cache provides.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Infinite Cache
@ 2011-07-08 13:41 Magnus Hagander <magnus@hagander.net>
parent: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
0 siblings, 0 replies; 9+ messages in thread
From: Magnus Hagander @ 2011-07-08 13:41 UTC (permalink / raw)
To: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>; +Cc: sthomas@peak6.com, Jim Nasby <jim@nasby.net>; Anthony Presley <anthony@resolution.com>; pgsql-performance
On Fri, Jul 8, 2011 at 15:34, Heikki Linnakangas
<heikki.linnakangas@enterprisedb.com> wrote:
> On 05.07.2011 16:35, Shaun Thomas wrote:
>>
>> I'd say it's probably safe enough these days. But it's also one of those
>> exclusive selling points they're using right now to garner EDB
>> customers. So I doubt it'll be released any time *soon*, though may make
>> it eventually.
>
> I doubt the community would want it even if it was open sourced. As an open
> source project, what would probably make more sense is a similar caching
> mechanism built into the kernel, somewhere between the filesystem cache and
> user-space. That way any you could use it with any application that benefits
> from the kind of large cache that Infinite Cache provides.
Don't underestimate how much easier it is to get something into an
environment as part of the database than as an extra module or app
that you have to patch your kernel with. Unless you can get it all the
way into the baseline kernel of course, but that's not going to be
easy...
--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/
^ permalink raw reply [nested|flat] 9+ messages in thread
end of thread, other threads:[~2011-07-08 13:41 UTC | newest]
Thread overview: 9+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2011-07-01 14:43 Infinite Cache Anthony Presley <anthony@resolution.com>
2011-07-01 15:54 ` Greg Smith <greg@2ndQuadrant.com>
2011-07-01 22:37 ` Jim Nasby <jim@nasby.net>
2011-07-02 08:20 ` Dave Page <dpage@pgadmin.org>
2011-07-03 11:21 ` Magnus Hagander <magnus@hagander.net>
2011-07-05 13:35 ` Shaun Thomas <sthomas@peak6.com>
2011-07-08 13:34 ` Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2011-07-08 13:41 ` Magnus Hagander <magnus@hagander.net>
2011-07-03 18:18 ` Greg Smith <greg@2ndQuadrant.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox