pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedPrefetch - OffTopic
3+ messages / 2 participants
[nested] [flat]
* Prefetch - OffTopic
@ 2005-05-10 19:17 Mohan, Ross <RMohan@arbinet.com>
2005-05-10 20:14 ` Re: Prefetch - OffTopic Chris Browne <cbbrowne@acm.org>
0 siblings, 1 reply; 3+ messages in thread
From: Mohan, Ross @ 2005-05-10 19:17 UTC (permalink / raw)
To: pgsql-performance
for time-series and "insane fast", nothing beats kdB, I believe
www.kx.com
Not trying to Quisling-out PG here, just hoping to respond to Mr. Olson....
-----Original Message-----
From: pgsql-performance-owner@postgresql.org [mailto:pgsql-performance-owner@postgresql.org] On Behalf Of Tom Lane
Sent: Tuesday, May 10, 2005 2:54 PM
To: Greg Stark
Cc: molson@oceanconsulting.com; pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Prefetch
Greg Stark <gsstark@mit.edu> writes:
> Actually forcing things to use indexes is the wrong direction to go if
> you're trying to process lots of data and want to stream it off disk
> as rapidly as possible. I would think about whether you can structure
> your data such that you can use sequential scans.
Agreed.
> In your application that might be hard. It sounds like you would need
> more or less one table per stock ticker which would really be hard to
> manage.
Actually, in a previous lifetime I used to do pretty much the same stuff Matt is working on. The reason I suggested parallelizing is that what you want is usually not so much the 200day moving average of FOO, as the 200day moving averages of a whole bunch of things. If your input table contains time-ordered data for all those things, then a seqscan works out pretty well.
> One thing you might look into is using the CLUSTER command. But
> postgres doesn't maintain the cluster ordering so it would require
> periodically rerunning it.
If the desired sort order is time-based, it falls out pretty much for free in this application, because historical data doesn't change -- you are only interested in appending at the right.
In said previous lifetime, we used Postgres for tracking our actual transactions, but we built a custom file format for storing the individual tick data. That's not stuff you need transactional semantics for; the historical data is what it is. Besides, you need to compress it as much as you can because there's always too much of it. Machines are faster and disk space cheaper than they were at the time, but I'd still question the wisdom of using a Postgres row for each daily bar, let alone finer-grain data.
regards, tom lane
---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: Prefetch - OffTopic
2005-05-10 19:17 Prefetch - OffTopic Mohan, Ross <RMohan@arbinet.com>
@ 2005-05-10 20:14 ` Chris Browne <cbbrowne@acm.org>
0 siblings, 0 replies; 3+ messages in thread
From: Chris Browne @ 2005-05-10 20:14 UTC (permalink / raw)
To: pgsql-performance
RMohan@arbinet.com ("Mohan, Ross") writes:
> for time-series and "insane fast", nothing beats kdB, I believe
>
> www.kx.com
... Which is well and fine if you're prepared to require that all of
the staff that interact with data are skilled APL hackers. Skilled
enough that they're all ready to leap into Whitney's ASCII-based
variant, K.
--
(format nil "~S@~S" "cbbrowne" "acm.org")
http://www.ntlug.org/~cbbrowne/functional.html
Rules of the Evil Overlord #78. "I will not tell my Legions of Terror
"And he must be taken alive!" The command will be: ``And try to take
him alive if it is reasonably practical.''"
<http://www.eviloverlord.com/;
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: Prefetch - OffTopic
@ 2005-05-10 21:28 Mohan, Ross <RMohan@arbinet.com>
0 siblings, 0 replies; 3+ messages in thread
From: Mohan, Ross @ 2005-05-10 21:28 UTC (permalink / raw)
To: pgsql-performance
Yes, that would be a sufficient (although not necessary) condition for being well and fine with kdB.
Last time I used APL was.....pre-Gregorian, so yea, that's scary to me, too.
( Of course, one can use C/ODBC or Java/JDBC to reach kdB; once there, one uses SQL92, or
proprietary kSQL. )
-----Original Message-----
From: pgsql-performance-owner@postgresql.org [mailto:pgsql-performance-owner@postgresql.org] On Behalf Of Chris Browne
Sent: Tuesday, May 10, 2005 4:14 PM
To: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Prefetch - OffTopic
RMohan@arbinet.com ("Mohan, Ross") writes:
> for time-series and "insane fast", nothing beats kdB, I believe
>
> www.kx.com
... Which is well and fine if you're prepared to require that all of the staff that interact with data are skilled APL hackers. Skilled enough that they're all ready to leap into Whitney's ASCII-based variant, K.
--
(format nil "~S@~S" "cbbrowne" "acm.org") http://www.ntlug.org/~cbbrowne/functional.html
Rules of the Evil Overlord #78. "I will not tell my Legions of Terror "And he must be taken alive!" The command will be: ``And try to take him alive if it is reasonably practical.''" <http://www.eviloverlord.com/;
---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2005-05-10 21:28 UTC | newest]
Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2005-05-10 19:17 Prefetch - OffTopic Mohan, Ross <RMohan@arbinet.com>
2005-05-10 20:14 ` Chris Browne <cbbrowne@acm.org>
2005-05-10 21:28 Re: Prefetch - OffTopic Mohan, Ross <RMohan@arbinet.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox