pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedPostgreSQL database size is not reasonable
6+ messages / 4 participants
[nested] [flat]
* PostgreSQL database size is not reasonable
@ 2017-12-12 15:15 Mariel Cherkassky <mariel.cherkassky@gmail.com>
0 siblings, 1 reply; 6+ messages in thread
From: Mariel Cherkassky @ 2017-12-12 15:15 UTC (permalink / raw)
To: pgsql-performance
In my postgresql 9.6 instance I have 1 production database. When I query
the size of all databases :
combit=> Select
pg_database.datname,pg_size_pretty(pg_database_size(pg_database.datname))
as size from pg_database;
datname | size -----------+---------
template0 | 7265 kB
combit | 285 GB
postgres | 7959 kB
template1 | 7983 kB
repmgr | 8135 kB(5 rows)
When I check what are the big tables in my database (includes indexes) :
combit=> SELECT nspname || '.' || relname AS "relation",
combit-> pg_size_pretty(pg_total_relation_size(C.oid)) AS "total_size"
combit-> FROM pg_class C
combit-> LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
combit-> WHERE nspname NOT IN ('pg_catalog', 'information_schema')
combit-> AND C.relkind <> 'i'
combit-> AND nspname !~ '^pg_toast'
combit-> ORDER BY pg_total_relation_size(C.oid) DESC
combit-> LIMIT 20;
relation | total_size -----------------------------+------------
rep.ps_rf_inst_prod | 48 GB
rep.nap_inter_x5 | 46 GB
rep.man_x5 | 16 GB
rep.tc_fint_x5 | 9695 MB
rep.nap_ip_debit_x5 | 7645 MB
rep.ip__billing | 5458 MB
rep.ps_rd | 3417 MB
rep.nap_ip_discount | 3147 MB
rep.custo_x5 | 2154 MB
rep.ip_service_discou_x5 | 1836 MB
rep.tc_sub_rate__x5 | 294 MB
The total sum is not more than 120G.
When I check the fs directly :
[/data/base] : du -sk * | sort -n7284 133227868 133237892
18156 166694298713364 16400
[/data/base] :
16400 is the oid of the combit database. As you can see the size of combit
on the fs is about 298G.
I checked for dead tuples in the biggest tables :
combit=>select relname,n_dead_tup,last_autoanalyze,last_analyze,last_autovacuum,last_vacuum
from pg_stat_user_tables order by n_live_tup desc limit4;
-[ RECORD 1 ]----+------------------------------
relname | ps_rf_inst_prod
n_dead_tup | 0
last_autoanalyze | 2017-12-04 09:00:16.585295+02
last_analyze | 2017-12-05 16:08:31.218621+02
last_autovacuum |
last_vacuum |
-[ RECORD 2 ]----+------------------------------
relname | man_x5
n_dead_tup | 0
last_autoanalyze | 2017-12-05 06:02:07.189184+02
last_analyze | 2017-12-05 16:12:58.130519+02
last_autovacuum |
last_vacuum |
-[ RECORD 3 ]----+------------------------------
relname | tc_fint_x5
n_dead_tup | 0
last_autoanalyze | 2017-12-05 06:04:06.698422+02
last_analyze |
last_autovacuum |
last_vacuum |
-[ RECORD 4 ]----+------------------------------
relname | nap_inter_x5
n_dead_tup | 0
last_autoanalyze | 2017-12-04 08:54:16.764392+02
last_analyze | 2017-12-05 16:10:23.411266+02
last_autovacuum |
last_vacuum |
I run vacuum full on all 5 top tables 2 hours ago and it didnt free alot of
space...
On this database the only operations that happen are truncate , insert and
select. So how can it be that I had dead tuples on some of my tables ? If I
only run truncate,select,insert query tuples shouldnt be created..
And the bigger question, Where are the missing 180G ?
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: PostgreSQL database size is not reasonable
@ 2017-12-12 15:21 David G. Johnston <david.g.johnston@gmail.com>
parent: Mariel Cherkassky <mariel.cherkassky@gmail.com>
0 siblings, 2 replies; 6+ messages in thread
From: David G. Johnston @ 2017-12-12 15:21 UTC (permalink / raw)
To: Mariel Cherkassky <mariel.cherkassky@gmail.com>; +Cc: pgsql-performance
On Tue, Dec 12, 2017 at 8:15 AM, Mariel Cherkassky <
mariel.cherkassky@gmail.com> wrote:
> A
> nd the bigger question, Where are the missing 180G ?
>
> In the toaster probably...
https://www.postgresql.org/docs/current/static/storage-toast.html
Basically large data values are store in another table different than both
the main table and indexes.
David J.
^ permalink raw reply [nested|flat] 6+ messages in thread
* RE: PostgreSQL database size is not reasonable
@ 2017-12-12 15:44 Craig McIlwee <craigm@q-free.com>
parent: David G. Johnston <david.g.johnston@gmail.com>
1 sibling, 0 replies; 6+ messages in thread
From: Craig McIlwee @ 2017-12-12 15:44 UTC (permalink / raw)
To: David G. Johnston <david.g.johnston@gmail.com>; Mariel Cherkassky <mariel.cherkassky@gmail.com>; +Cc: pgsql-performance
On Tue, Dec 12, 2017 at 8:15 AM, Mariel Cherkassky <mariel.cherkassky@gmail.com<mailto:mariel.cherkassky@gmail.com>> wrote:
And the bigger question, Where are the missing 180G ?
In the toaster probably...
https://www.postgresql.org/docs/current/static/storage-toast.html
Basically large data values are store in another table different than both the main table and indexes.
David J.
The query also says C.relkind <> 'i' which means it’s excluding indexes. Also note that pg_catalog is excluded but LOB data would be stored in pg_catalog.pg_largeobject. That could account for some overlooked space as well.
Craig
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: PostgreSQL database size is not reasonable
@ 2017-12-12 15:49 Tom Lane <tgl@sss.pgh.pa.us>
parent: David G. Johnston <david.g.johnston@gmail.com>
1 sibling, 1 reply; 6+ messages in thread
From: Tom Lane @ 2017-12-12 15:49 UTC (permalink / raw)
To: David G. Johnston <david.g.johnston@gmail.com>; +Cc: Mariel Cherkassky <mariel.cherkassky@gmail.com>; pgsql-performance
"David G. Johnston" <david.g.johnston@gmail.com> writes:
> On Tue, Dec 12, 2017 at 8:15 AM, Mariel Cherkassky <
> mariel.cherkassky@gmail.com> wrote:
>> And the bigger question, Where are the missing 180G ?
> In the toaster probably...
pg_total_relation_size should have counted the toast tables,
as well as the indexes, if memory serves.
What I'm wondering about is the system catalogs, which Mariel's
query explicitly excluded. 180G would be awful darn large for
those, but maybe there's a bloat problem somewhere.
Otherwise, try to identify the largest individual files in the
database directory ...
regards, tom lane
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: PostgreSQL database size is not reasonable
@ 2017-12-12 16:22 Mariel Cherkassky <mariel.cherkassky@gmail.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 6+ messages in thread
From: Mariel Cherkassky @ 2017-12-12 16:22 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: David G. Johnston <david.g.johnston@gmail.com>; pgsql-performance
The system catalogs located in the global directory but the global
directory isnt so big(500K). As I mentioned, the base directory is huge and
the directory 16400 is the biggest inside. I checked some big files inside
the directory 16400 (which represents the commbit database) and for some
there *isnt an object that match* and for some there are. So, How can I
continue ?
2017-12-12 17:49 GMT+02:00 Tom Lane <tgl@sss.pgh.pa.us>:
> "David G. Johnston" <david.g.johnston@gmail.com> writes:
> > On Tue, Dec 12, 2017 at 8:15 AM, Mariel Cherkassky <
> > mariel.cherkassky@gmail.com> wrote:
> >> And the bigger question, Where are the missing 180G ?
>
> > In the toaster probably...
>
> pg_total_relation_size should have counted the toast tables,
> as well as the indexes, if memory serves.
>
> What I'm wondering about is the system catalogs, which Mariel's
> query explicitly excluded. 180G would be awful darn large for
> those, but maybe there's a bloat problem somewhere.
>
> Otherwise, try to identify the largest individual files in the
> database directory ...
>
> regards, tom lane
>
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: PostgreSQL database size is not reasonable
@ 2017-12-12 17:59 Tom Lane <tgl@sss.pgh.pa.us>
parent: Mariel Cherkassky <mariel.cherkassky@gmail.com>
0 siblings, 0 replies; 6+ messages in thread
From: Tom Lane @ 2017-12-12 17:59 UTC (permalink / raw)
To: Mariel Cherkassky <mariel.cherkassky@gmail.com>; +Cc: David G. Johnston <david.g.johnston@gmail.com>; pgsql-performance
Mariel Cherkassky <mariel.cherkassky@gmail.com> writes:
> The system catalogs located in the global directory but the global
> directory isnt so big(500K).
You're operating under false assumptions. Only catalogs marked
relisshared are in that directory, other ones are in the per-database
directories.
Somebody mentioned pg_largeobject upthread --- that would definitely
be a candidate to be big, if you're using large objects at all.
regards, tom lane
^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2017-12-12 17:59 UTC | newest]
Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2017-12-12 15:15 PostgreSQL database size is not reasonable Mariel Cherkassky <mariel.cherkassky@gmail.com>
2017-12-12 15:21 ` David G. Johnston <david.g.johnston@gmail.com>
2017-12-12 15:44 ` Craig McIlwee <craigm@q-free.com>
2017-12-12 15:49 ` Tom Lane <tgl@sss.pgh.pa.us>
2017-12-12 16:22 ` Mariel Cherkassky <mariel.cherkassky@gmail.com>
2017-12-12 17:59 ` Tom Lane <tgl@sss.pgh.pa.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox