pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedCompression of text columns
5+ messages / 4 participants
[nested] [flat]
* Compression of text columns
@ 2005-10-10 12:17 Stef <svb@ucs.co.za>
0 siblings, 2 replies; 5+ messages in thread
From: Stef @ 2005-10-10 12:17 UTC (permalink / raw)
To: pgsql-performance
I have a table in the databases I work with,
that contains two text columns with XML data
stored inside them.
This table is by far the biggest table in the databases,
and the text columns use up the most space.
I saw that the default storage type for text columns is
"EXTENDED" which, according to the documentation, uses up extra
space to make possible substring functioning faster.
Suppose that the data in those columns are only really ever
_used_ once, but may be needed in future for viewing purposes mostly,
and I cannot really change the underlying structure of the table,
what can I possibly do to maximally reduce the amount of disk space
used by the table on disk. (There are no indexes on these two columns.)
I've thought about compression using something like :
ztext http://www.mahalito.net/~harley/sw/postgres/
but I have to change the table structure a lot and I've already
encountered problems unzipping the data again.
The other problem with this solution, is that database dumps almost double
in size, because of double compression.
Any suggestions much appreciated
TIA
Stefan
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Compression of text columns
@ 2005-10-10 12:27 Tino Wildenhain <tino@wildenhain.de>
parent: Stef <svb@ucs.co.za>
1 sibling, 1 reply; 5+ messages in thread
From: Tino Wildenhain @ 2005-10-10 12:27 UTC (permalink / raw)
To: Stef <svb@ucs.co.za>; +Cc: pgsql-performance
Stef schrieb:
> I have a table in the databases I work with,
> that contains two text columns with XML data
> stored inside them.
>
> This table is by far the biggest table in the databases,
> and the text columns use up the most space.
> I saw that the default storage type for text columns is
> "EXTENDED" which, according to the documentation, uses up extra
> space to make possible substring functioning faster.
>
> Suppose that the data in those columns are only really ever
> _used_ once, but may be needed in future for viewing purposes mostly,
> and I cannot really change the underlying structure of the table,
> what can I possibly do to maximally reduce the amount of disk space
> used by the table on disk. (There are no indexes on these two columns.)
> I've thought about compression using something like :
> ztext http://www.mahalito.net/~harley/sw/postgres/
>
> but I have to change the table structure a lot and I've already
> encountered problems unzipping the data again.
> The other problem with this solution, is that database dumps almost double
> in size, because of double compression.
>
> Any suggestions much appreciated
Well, text columns are automatically compressed via the toast mechanism.
This is handled transparently for you.
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Compression of text columns
@ 2005-10-10 12:57 Stef <svb@ucs.co.za>
parent: Tino Wildenhain <tino@wildenhain.de>
0 siblings, 1 reply; 5+ messages in thread
From: Stef @ 2005-10-10 12:57 UTC (permalink / raw)
To: Tino Wildenhain <tino@wildenhain.de>; +Cc: pgsql-performance
Tino Wildenhain mentioned :
=> Well, text columns are automatically compressed via the toast mechanism.
=> This is handled transparently for you.
OK, I misread the documentation, and I forgot to mention that
I'm using postgres 7.3 and 8.0
It's actually the EXTERNAL storage type that is larger, not EXTENDED.
What kind of compression is used in the EXTERNAL storage type?
Is there any way to achieve better compression?
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Compression of text columns
@ 2005-10-10 14:38 Tom Lane <tgl@sss.pgh.pa.us>
parent: Stef <svb@ucs.co.za>
1 sibling, 0 replies; 5+ messages in thread
From: Tom Lane @ 2005-10-10 14:38 UTC (permalink / raw)
To: Stef <svb@ucs.co.za>; +Cc: pgsql-performance
Stef <svb@ucs.co.za> writes:
> I saw that the default storage type for text columns is
> "EXTENDED" which, according to the documentation, uses up extra
> space to make possible substring functioning faster.
You misread it. EXTENDED does compression by default on long strings.
EXTERNAL is the one that suppresses compression.
regards, tom lane
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Compression of text columns
@ 2005-10-11 08:02 Simon Riggs <simon@2ndquadrant.com>
parent: Stef <svb@ucs.co.za>
0 siblings, 0 replies; 5+ messages in thread
From: Simon Riggs @ 2005-10-11 08:02 UTC (permalink / raw)
To: Stef <svb@ucs.co.za>; +Cc: Tino Wildenhain <tino@wildenhain.de>; pgsql-performance
On Mon, 2005-10-10 at 14:57 +0200, Stef wrote:
> Is there any way to achieve better compression?
You can use XML schema aware compression techniques, but PostgreSQL
doesn't know about those. You have to do it yourself, or translate the
XML into an infoset-preserving form that will still allow XPath and
friends.
Best Regards, Simon Riggs
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2005-10-11 08:02 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2005-10-10 12:17 Compression of text columns Stef <svb@ucs.co.za>
2005-10-10 12:27 ` Tino Wildenhain <tino@wildenhain.de>
2005-10-10 12:57 ` Stef <svb@ucs.co.za>
2005-10-11 08:02 ` Simon Riggs <simon@2ndquadrant.com>
2005-10-10 14:38 ` Tom Lane <tgl@sss.pgh.pa.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox