pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feednested query on last n rows of huge table
3+ messages / 2 participants
[nested] [flat]
* nested query on last n rows of huge table
@ 2006-02-24 14:13 jcfischer <jcfischer.lists@gmail.com>
0 siblings, 2 replies; 3+ messages in thread
From: jcfischer @ 2006-02-24 14:13 UTC (permalink / raw)
To: pgsql-performance
Hi list
I'm fairly new to Postgres so bear with me. Googling and searching the
list, I didn't find anything that resembled my problem.
I have a large table with ca. 10 million inserts per day (fairly simple
data: timestam, a couple of id's and a varchar message)
I run a query every couple of minutes that looks at the new entries
since the last run and retrieves them for further processing (using a
WHERE eventtime > '2006-02-24 14:00:00' ) to limit to the most recent
entries
These queries run around 40-50 seconds (largely due to some LIKE %msg%
threwn in for good measure). Postgres performs a seq table scan on
those queries :-(
My idea is to limit the search to only the last n entries because I
found that a
SELECT * from table ORDER eventtime DESC limit 1000
is very fast. Because the inserts are in chronolgical order, I can
store the sequential id of the highest row from the last query and
subtract that from the current high row count to determine that number.
Is there a way to limit the expensive query to only those last 1000 (or
whatever) results?
I have tried to nest SELECTS but my SQL-fu is to limited to get
anything through the SQL processor :-)
thanks
Jens-Christian Fischer
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: nested query on last n rows of huge table
@ 2006-02-24 14:26 jcfischer <jcfischer.lists@gmail.com>
parent: jcfischer <jcfischer.lists@gmail.com>
1 sibling, 0 replies; 3+ messages in thread
From: jcfischer @ 2006-02-24 14:26 UTC (permalink / raw)
To: pgsql-performance
sorry: Postgres 8.0.2 server. The EXPLAIN ANALYZE for the query looks
like this:
explain analyze select
syslog.logs.eventtime,assets.hosts.name,syslog.processes.name as
process
from
syslog.logs,assets.hosts,assets.ipaddrs,assets.macaddrs,syslog.processes
where msg like '%session opened for user root%'
and syslog.logs.assets_ipaddr_id = assets.ipaddrs.id
and assets.ipaddrs.macaddr_id = assets.macaddrs.id
and assets.macaddrs.host_id = assets.hosts.id
and syslog.processes.id = syslog.logs.process_id and
eventtime > timestamp '2006-02-24 15:05:00'
Nested Loop (cost=0.00..328832.34 rows=2 width=254) (actual
time=49389.924..49494.665 rows=45 loops=1)
-> Nested Loop (cost=0.00..328826.32 rows=1 width=90) (actual
time=49365.709..49434.500 rows=45 loops=1)
-> Nested Loop (cost=0.00..328820.30 rows=1 width=90) (actual
time=49327.211..49360.043 rows=45 loops=1)
-> Nested Loop (cost=0.00..328814.27 rows=1 width=90)
(actual time=49327.183..49344.281 rows=45 loops=1)
-> Seq Scan on logs (cost=0.00..328809.04 rows=1
width=16) (actual time=49314.928..49331.451 rows=45 loops=1)
Filter: (((msg)::text ~~ '%session opened for
user root%'::text) AND (eventtime > '2006-02-24 15:05:00'::timestamp
without time zone))
-> Index Scan using "pk_syslog.processes" on
processes (cost=0.00..5.21 rows=1 width=82) (actual time=0.278..0.280
rows=1 loops=45)
Index Cond: (processes.id =
"outer".process_id)
-> Index Scan using "pk_assets.ipaddrs" on ipaddrs
(cost=0.00..6.01 rows=1 width=8) (actual time=0.344..0.346 rows=1
loops=45)
Index Cond: ("outer".assets_ipaddr_id = ipaddrs.id)
-> Index Scan using "pk_assets.macaddrs" on macaddrs
(cost=0.00..6.01 rows=1 width=8) (actual time=1.648..1.650 rows=1
loops=45)
Index Cond: ("outer".macaddr_id = macaddrs.id)
-> Index Scan using "pk_assets.hosts" on hosts (cost=0.00..6.01
rows=1 width=172) (actual time=1.330..1.331 rows=1 loops=45)
Index Cond: ("outer".host_id = hosts.id)
Total runtime: 49494.830 ms
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: nested query on last n rows of huge table
@ 2006-03-01 03:44 Michael Glaesemann <grzm@myrealbox.com>
parent: jcfischer <jcfischer.lists@gmail.com>
1 sibling, 0 replies; 3+ messages in thread
From: Michael Glaesemann @ 2006-03-01 03:44 UTC (permalink / raw)
To: jcfischer <jcfischer.lists@gmail.com>; +Cc: pgsql-performance
On Feb 24, 2006, at 23:13 , jcfischer wrote:
> Is there a way to limit the expensive query to only those last 1000
> (or
> whatever) results?
>
> I have tried to nest SELECTS but my SQL-fu is to limited to get
> anything through the SQL processor :-)
The basics of a subquery are:
SELECT <expensive query>
FROM (
SELECT *
FROM table
ORDER eventtime DESC
LIMIT 1000
) as most_recent_1000
Don't know enough about the other parts, but hopefully this can get
you started. :)
Michael Glaesemann
grzm myrealbox com
^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2006-03-01 03:44 UTC | newest]
Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2006-02-24 14:13 nested query on last n rows of huge table jcfischer <jcfischer.lists@gmail.com>
2006-02-24 14:26 ` jcfischer <jcfischer.lists@gmail.com>
2006-03-01 03:44 ` Michael Glaesemann <grzm@myrealbox.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox