Received: from maia.hub.org (maia-5.hub.org [200.46.204.29]) by mail.postgresql.org (Postfix) with ESMTP id 1E4D6B5DBF9 for ; Fri, 7 Oct 2011 12:12:20 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.29]) (amavisd-maia, port 10024) with ESMTP id 02901-01 for ; Fri, 7 Oct 2011 15:12:13 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0-rc2 Received: from smtp.babel.it (ns1.babel.it [89.97.242.210]) by mail.postgresql.org (Postfix) with ESMTP id 7E39FB5DBC4 for ; Fri, 7 Oct 2011 12:12:12 -0300 (ADT) Received: from smtp.babel.it (catfish.babel.it [10.0.0.10]) by smtp.babel.it (Postfix) with SMTP id E467C7737C; Fri, 7 Oct 2011 17:12:08 +0200 (CEST) X-DKIM: OpenDKIM Filter v2.1.3 smtp.babel.it E467C7737C Authentication-Results: smtp.babel.it; dkim=permerror (verification error: empty key record; insecure key) header.i=@babel.it header.b=KFVeBody; dkim-adsp=none (insecure policy) Received: from [10.11.48.31] (unknown [2.196.62.55]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: gmancuso@babel.it) by smtp.babel.it (Postfix) with ESMTPSA id 2E8444D70D; Fri, 7 Oct 2011 17:12:07 +0200 (CEST) X-DKIM: OpenDKIM Filter v2.1.3 smtp.babel.it 2E8444D70D DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=babel.it; s=babeldkim; t=1318000328; bh=sMRD5text45Q2D90e8ki/owD1afRPWcquOyNXw4lt9E=; h=Message-ID:Date:From:MIME-Version:To:CC:Subject:References: In-Reply-To:Content-Type; b=KFVeBody9oH4yQtqe/HLWsP+TjqlBaVcM5ox7UBS/Snv1ES345agPVDe957hyoH/e Aoz3w74wu1W2sv+or6FO3lqSKL8Eccf+RLDSaAjXbLvkUE+HFWDvbgs4o+sfNaGK24 4uZdxxbDXoUfJKgXjErOu++2xgAls0IwScgNGeFA= Message-ID: <4E8F16C6.7070606@babel.it> Date: Fri, 07 Oct 2011 17:12:06 +0200 From: Giovanni Mancuso User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:7.0) Gecko/20110917 Thunderbird/7.0 MIME-Version: 1.0 To: Guillaume Cottenceau CC: pgsql-performance@postgresql.org Subject: Re: Performance problem with a table with 38928077 record References: <4E8ECEB7.4080800@babel.it> In-Reply-To: X-Enigmail-Version: 1.3.2 Content-Type: multipart/mixed; boundary="------------020801040903090007080109" X-PMX-Version: 5.6.0.2009776 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-2.403 tagged_above=-10 required=5 tests=BAYES_00=-1.9, HTML_MESSAGE=0.001, RP_MATCHES_RCVD=-0.504 X-Spam-Level: X-Archive-Number: 201110/72 X-Sequence-Number: 45294 This is a multi-part message in MIME format. --------------020801040903090007080109 Content-Type: multipart/alternative; boundary="------------090505080204000902060703" --------------090505080204000902060703 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Il 07/10/2011 12:24, Guillaume Cottenceau ha scritto: > Giovanni Mancuso writes: > >> select count(*) from dm_object_perm; >> count >> ---------- >> 38'928'077 >> (1 row) > [...] > >> If i run "explain analyze select * from dm_object_perm;" it goes on for many >> hours. > Almost 39 million records is not small, especially if you run on > poor hardware[1], poor configuration[2], poor database optimization[3], > bloat[4], or a combination of these. > > [1] you could tell what hardware you use My Memory: # cat /proc/meminfo total: used: free: shared: buffers: cached: Mem: 4022861824 2201972736 1820889088 0 8044544 1983741952 Swap: 8589926400 199303168 8390623232 MemTotal: 3928576 kB MemFree: 1778212 kB MemShared: 0 kB Buffers: 7856 kB Cached: 1897356 kB SwapCached: 39892 kB Active: 1330076 kB ActiveAnon: 554472 kB ActiveCache: 775604 kB Inact_dirty: 539124 kB Inact_laundry: 55348 kB Inact_clean: 36504 kB Inact_target: 392208 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 3928576 kB LowFree: 1778212 kB SwapTotal: 8388600 kB SwapFree: 8193968 kB CommitLimit: 10352888 kB Committed_AS: 1713308 kB HugePages_Total: 0 HugePages_Free: 0 Hugepagesize: 2048 kB My CPU: # egrep 'processor|model name|cpu MHz|cache size|flags' /proc/cpuinfo processor : 0 model name : Dual Core AMD Opteron(tm) Processor 275 cpu MHz : 2193.798 cache size : 1024 KB flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext lm 3dnowext 3dnow processor : 1 model name : Dual Core AMD Opteron(tm) Processor 275 cpu MHz : 2193.798 cache size : 1024 KB flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext lm 3dnowext 3dnow processor : 2 model name : Dual Core AMD Opteron(tm) Processor 275 cpu MHz : 2193.798 cache size : 1024 KB flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext lm 3dnowext 3dnow processor : 3 model name : Dual Core AMD Opteron(tm) Processor 275 cpu MHz : 2193.798 cache size : 1024 KB flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext lm 3dnowext 3dnow > [2] you could report if your DB configuration is tuned/good max_connections = 50 shared_buffers = 512MB temp_buffers = 128MB max_prepared_transactions = 55 max_fsm_pages = 153600 vacuum_cost_delay = 0 vacuum_cost_page_hit = 1 vacuum_cost_page_miss = 10 vacuum_cost_page_dirty = 20 vacuum_cost_limit = 200 effective_cache_size = 256MB autovacuum = on autovacuum_naptime = 1min autovacuum_vacuum_threshold = 500 autovacuum_analyze_threshold = 250 autovacuum_vacuum_scale_factor = 0.2 autovacuum_analyze_scale_factor = 0.1 autovacuum_freeze_max_age = 200000000 autovacuum_vacuum_cost_delay = -1 autovacuum_vacuum_cost_limit = -1 vacuum_freeze_min_age = 100000000 > [3] you could report if the DB is regularly analyzed/vacuumed > [4] you could try a VACUUM FULL or CLUSTER and/or REINDEX on your > large table(s) if you suspect answer to [3] is "no" - > warning, these block some/all DB operations while running, > and they will probably run for long in your situation I run VACUUM yesterday. > >> If i try to execute a left join: "SELECT dm_object.id FROM dm_object LEFT JOIN >> dm_object_perm ON dm_object.id = dm_object_perm.object_id;" my db is unusable. > EXPLAIN on this query would probably tell you PG has quite some > work to do to produce the result. > > >> how can I fix this? > I'm wondering if your DB design (storing almost all "object x > account" combinations in object_perm) is optimal. > -- *Giovanni Mancuso* System Architect Babel S.r.l. - http://www.babel.it *T:* 06.9826.9600 *M:* 3406580739 *F:* 06.9826.9680 P.zza S.Benedetto da Norcia, 33 - 00040 Pomezia (Roma) ------------------------------------------------------------------------ CONFIDENZIALE: Questo messaggio ed i suoi allegati sono di carattere confidenziale per i destinatari in indirizzo. E' vietato l'inoltro non autorizzato a destinatari diversi da quelli indicati nel messaggio originale. Se ricevuto per errore, l'uso del contenuto e' proibito; si prega di comunicarlo al mittente e cancellarlo immediatamente. --------------090505080204000902060703 Content-Type: multipart/related; boundary="------------080402070201060709070108" --------------080402070201060709070108 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Il 07/10/2011 12:24, Guillaume Cottenceau ha scritto:
Giovanni Mancuso <gmancuso 'at' babel.it> writes:

select count(*) from dm_object_perm;
  count  
----------
 38'928'077
(1 row)
[...]

If i run "explain analyze select * from dm_object_perm;" it goes on for many
hours.
Almost 39 million records is not small, especially if you run on
poor hardware[1], poor configuration[2], poor database optimization[3],
bloat[4], or a combination of these.

[1] you could tell what hardware you use
My Memory:
# cat /proc/meminfo
        total:    used:    free:  shared: buffers:  cached:
Mem:  4022861824 2201972736 1820889088        0  8044544 1983741952
Swap: 8589926400 199303168 8390623232
MemTotal:      3928576 kB
MemFree:       1778212 kB
MemShared:           0 kB
Buffers:          7856 kB
Cached:        1897356 kB
SwapCached:      39892 kB
Active:        1330076 kB
ActiveAnon:     554472 kB
ActiveCache:    775604 kB
Inact_dirty:    539124 kB
Inact_laundry:   55348 kB
Inact_clean:     36504 kB
Inact_target:   392208 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      3928576 kB
LowFree:       1778212 kB
SwapTotal:     8388600 kB
SwapFree:      8193968 kB
CommitLimit:  10352888 kB
Committed_AS:  1713308 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

My CPU:
# egrep 'processor|model name|cpu MHz|cache size|flags' /proc/cpuinfo
processor       : 0
model name      : Dual Core AMD Opteron(tm) Processor 275
cpu MHz         : 2193.798
cache size      : 1024 KB
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext lm 3dnowext 3dnow
processor       : 1
model name      : Dual Core AMD Opteron(tm) Processor 275
cpu MHz         : 2193.798
cache size      : 1024 KB
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext lm 3dnowext 3dnow
processor       : 2
model name      : Dual Core AMD Opteron(tm) Processor 275
cpu MHz         : 2193.798
cache size      : 1024 KB
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext lm 3dnowext 3dnow
processor       : 3
model name      : Dual Core AMD Opteron(tm) Processor 275
cpu MHz         : 2193.798
cache size      : 1024 KB
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext lm 3dnowext 3dnow


[2] you could report if your DB configuration is tuned/good
max_connections = 50
shared_buffers = 512MB
temp_buffers = 128MB
max_prepared_transactions = 55
max_fsm_pages = 153600
vacuum_cost_delay = 0
vacuum_cost_page_hit = 1
vacuum_cost_page_miss = 10
vacuum_cost_page_dirty = 20
vacuum_cost_limit = 200
effective_cache_size = 256MB
autovacuum = on
autovacuum_naptime = 1min
autovacuum_vacuum_threshold = 500
autovacuum_analyze_threshold = 250
autovacuum_vacuum_scale_factor = 0.2
autovacuum_analyze_scale_factor = 0.1
autovacuum_freeze_max_age = 200000000
autovacuum_vacuum_cost_delay = -1
autovacuum_vacuum_cost_limit = -1
vacuum_freeze_min_age = 100000000

[3] you could report if the DB is regularly analyzed/vacuumed
[4] you could try a VACUUM FULL or CLUSTER and/or REINDEX on your
    large table(s) if you suspect answer to [3] is "no" -
    warning, these block some/all DB operations while running,
    and they will probably run for long in your situation
I run VACUUM yesterday.

If i try to execute a left join: "SELECT dm_object.id FROM dm_object LEFT JOIN
dm_object_perm ON dm_object.id = dm_object_perm.object_id;" my db is unusable.
EXPLAIN on this query would probably tell you PG has quite some
work to do to produce the result.


how can I fix this?
I'm wondering if your DB design (storing almost all "object x
account" combinations in object_perm) is optimal.



--
Documento senza titolo
Giovanni Mancuso
System Architect
Babel S.r.l. - http://www.babel.it
T: 06.9826.9600 M: 3406580739 F: 06.9826.9680
P.zza S.Benedetto da Norcia, 33 - 00040 Pomezia (Roma)

CONFIDENZIALE: Questo messaggio ed i suoi allegati sono di carattere confidenziale per i destinatari in indirizzo.
E' vietato l'inoltro non autorizzato a destinatari diversi da quelli indicati nel messaggio originale.
Se ricevuto per errore, l'uso del contenuto e' proibito; si prega di comunicarlo al mittente e cancellarlo immediatamente.
--------------080402070201060709070108 Content-Type: image/gif; name="logo.gif" Content-Transfer-Encoding: base64 Content-ID: Content-Disposition: inline; filename="logo.gif" R0lGODlhdwAqANUgAFNRVMbFxvObEIyLjfrXn/jJf/zkv/WvQPnQj/3rz/7473Bucf3y3/a2 UPvdr/jDcCgmKfHw8Tc0OKmoqX99f+Li4vShIPe8YPSoMEVDRpqZm2JgY9TT1Le2txoXG/KU AP///wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACH/C1hNUCBEYXRh WE1QPD94cGFja2V0IGJlZ2luPSLvu78iIGlkPSJXNU0wTXBDZWhpSHpyZVN6TlRjemtjOWQi Pz4gPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iQWRvYmUg WE1QIENvcmUgNS4wLWMwNjAgNjEuMTM0Nzc3LCAyMDEwLzAyLzEyLTE3OjMyOjAwICAgICAg ICAiPiA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjIt cmRmLXN5bnRheC1ucyMiPiA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIiB4bWxuczp4 bXA9Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC8iIHhtbG5zOnhtcE1NPSJodHRwOi8v bnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0iaHR0cDovL25zLmFkb2Jl LmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1wOkNyZWF0b3JUb29sPSJBZG9i ZSBQaG90b3Nob3AgQ1M1IFdpbmRvd3MiIHhtcE1NOkluc3RhbmNlSUQ9InhtcC5paWQ6RDk1 QTA4OUZDQkE3MTFERkJCQ0RCNkY0NzdEMjZCRkUiIHhtcE1NOkRvY3VtZW50SUQ9InhtcC5k aWQ6RDk1QTA4QTBDQkE3MTFERkJCQ0RCNkY0NzdEMjZCRkUiPiA8eG1wTU06RGVyaXZlZEZy b20gc3RSZWY6aW5zdGFuY2VJRD0ieG1wLmlpZDpEOTVBMDg5RENCQTcxMURGQkJDREI2RjQ3 N0QyNkJGRSIgc3RSZWY6ZG9jdW1lbnRJRD0ieG1wLmRpZDpEOTVBMDg5RUNCQTcxMURGQkJD REI2RjQ3N0QyNkJGRSIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRGPiA8L3g6eG1w bWV0YT4gPD94cGFja2V0IGVuZD0iciI/PgH//v38+/r5+Pf29fTz8vHw7+7t7Ovq6ejn5uXk 4+Lh4N/e3dzb2tnY19bV1NPS0dDPzs3My8rJyMfGxcTDwsHAv769vLu6ubi3trW0s7KxsK+u rayrqqmop6alpKOioaCfnp2cm5qZmJeWlZSTkpGQj46NjIuKiYiHhoWEg4KBgH9+fXx7enl4 d3Z1dHNycXBvbm1sa2ppaGdmZWRjYmFgX15dXFtaWVhXVlVUU1JRUE9OTUxLSklIR0ZFRENC QUA/Pj08Ozo5ODc2NTQzMjEwLy4tLCsqKSgnJiUkIyIhIB8eHRwbGhkYFxYVFBMSERAPDg0M CwoJCAcGBQQDAgEAACH5BAEAACAALAAAAAB3ACoAAAb/wINw+HCAjsikcslsOp/QqHS6TDQ+ 2KygoKB6v+CwWGkQZLWGsXrNFjMw5yyhTa/bkwpz/DO/+/9gBHtYAgyAh4hNgoNYB4mPiApX jFlpkJd1D3qUWBcHZwIHBQmYpVQFqKgPn4NbjBh9prJTCgQWe4uMB4azvU4EWwVxCHFbkx8C Rr7LSLmwmwZxDyAKcaTMvgnFwFi5WBYOD2fT2L6soIvee8nlvtGDDQ0CxK1c7b4XlATxxQ0E Xfd83dpj4YEBBQYSJuQVkNmeA5YaSgTxjpCyiRMrYriGcaI2ZAzDeBhJMsOACFMqjNQwBQDJ kRIWoHzi8uVIAE8GeMDpBEtE/zE2SW6YotODhJZBjc5sUtMmzyY6nzLBcIHNyABHKiwYOQUC yQlSXA44EkHDVZoexlKJ+uRBSKAesB4JwDXKBKNbpToRm+Ts3rRe2Dq58BOuhgCIN3jIkCTB qCUZ0qr0wEGJAgSxQLhcgDiATggVkCgocFGzB86d5SJBgCCJ4CYWHK1JCqEyEjMClND1gHKz EnoRm77skGQSR+EkkyxqfeQ1EyzMx4zMAKC619pIbuVOonjBkQ4jl4IAjsSlhOoAvHoAe2QS Q/PoqyuHjsQ5mSwcRcZFEsHl0CMMFBDSZEGptVpmfCFBgVGiIfBTgopkZl8SFbEDl2ogLKiX EgvSBsIFhCCA5wFaBkoxIRIVYWEPGCsh1kGH3jURgVfskTVSjUxs1tkAXjH2F2qd2QaVUQMU WSSGBCEAEBVJGRVaE3dBsMRWPjJFG4ZLIHdTTkmVCMcgBzyQSiqZJRFfdQtMIN4SFADAkhIc VPckE23Gt4EGczZR55kUPDHBmdXhKA4nhFyQX0emMEDoBw28hags+VAS3aO+MLBJHGVS2osD rRyqaS/qEOLpp7IkMBAopZE6y2iXNlKYqrKEM8QQF6QKax1BAAA7 --------------080402070201060709070108-- --------------090505080204000902060703-- --------------020801040903090007080109 Content-Type: text/x-vcard; charset=utf-8; name="gmancuso.vcf" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="gmancuso.vcf" begin:vcard fn:Giovanni Mancuso n:Mancuso;Giovanni org:Babel S.r.l adr:;;P.zza S.Benedetto da Norcia, 33;Pomezia;ROMA;00040;Italy email;internet:gmancuso@babel.it title:System Architect tel;work:+39.06.91801075 tel;fax:+39.06.91612446 tel;cell:+393406580739 url:http://www.babel.it version:2.1 end:vcard --------------020801040903090007080109--