From: Justin Pryzby <pryzby@telsasoft.com>
To: Heikki Linnakangas <hlinnaka@iki.fi>
Cc: Ashwin Agrawal <aagrawal@pivotal.io>
Cc: Ashutosh Sharma <ashu.coek88@gmail.com>
Cc: PostgreSQL mailing lists <pgsql-hackers@postgresql.org>
Subject: Re: Zedstore - compressed in-core columnar storage
Date: Sun, 18 Aug 2019 14:35:33 -0500
Message-ID: <20190818193533.GL11185@telsasoft.com> (raw)
In-Reply-To: <ffee5017-90e9-238a-6e9e-ac1697d007a0@iki.fi>
References: <CAH2-WznPCX1Kp+SKoPTZ5VH9y=OEA_L+SZ6GO2DLeZasnRCGWQ@mail.gmail.com>
<22678.1555355981@sss.pgh.pa.us>
<CAH2-WzkTUkLfWAxYZZLeoUxiJkHnnrnWBZVcJdfnSsRxWK0YVA@mail.gmail.com>
<c4f3f315-bee3-3d7f-1284-5db2c528fbc6@iki.fi>
<CALfoeiuuLe12PuQ=zvM_L7B5qegBqGHYENHUGbLOsjAnG=mi4A@mail.gmail.com>
<5ceff1bd.1c69fb81.704ee.a5b1SMTPIN_ADDED_BROKEN@mx.google.com>
<CALfoeivDkvyyw68GsUo5NX67P+i5WUvpbbJ14BOFB0M=4RWAzQ@mail.gmail.com>
<CAE9k0P=1MhCSMeBJytPe34Zm-xUbRJupHrk2jqkT=9VFYM-BiQ@mail.gmail.com>
<CALfoeivAGExe37Q1yVJyYPm+7k1AUS2_QW_cFCNamDA5vU_Few@mail.gmail.com>
<ffee5017-90e9-238a-6e9e-ac1697d007a0@iki.fi>
On Thu, Aug 15, 2019 at 01:05:49PM +0300, Heikki Linnakangas wrote:
> We've continued hacking on Zedstore, here's a new patch version against
> current PostgreSQL master (commit f1bf619acdf). If you want to follow the
> development in real-time, we're working on this branch:
> https://github.com/greenplum-db/postgres/tree/zedstore
Thanks for persuing this. It's an exciting development and I started
looking at how we'd put it to use. I imagine we'd use it in favour of ZFS
tablespaces, which I hope to retire.
I've just done very brief experiment so far. Some thoughts:
. I was missing a way to check for compression ratio; it looks like zedstore
with lz4 gets ~4.6x for our largest customer's largest table. zfs using
compress=gzip-1 gives 6x compression across all their partitioned tables,
and I'm surprised it beats zedstore .
. What do you think about pg_restore --no-tableam; similar to
--no-tablespaces, it would allow restoring a table to a different AM:
PGOPTIONS='-c default_table_access_method=zedstore' pg_restore --no-tableam ./pg_dump.dat -d postgres
Otherwise, the dump says "SET default_table_access_method=heap", which
overrides any value from PGOPTIONS and precludes restoring to new AM.
. It occured to me that indices won't be compressed. That's no fault of
zedstore, but it may mean that some sites would need to retain their ZFS
tablespace, and suggests the possibility of an separate, future project
(I wonder if there's some way a new meta-AM could "enable" compression of
other index AMs, to avoid the need to implement zbtree, zhash, zgin, ...).
. it'd be nice if there was an ALTER TABLE SET ACCESS METHOD, to allow
migrating data. Otherwise I think the alternative is:
begin; lock t;
CREATE TABLE new_t LIKE (t INCLUDING ALL) USING (zedstore);
INSERT INTO new_t SELECT * FROM t;
for index; do CREATE INDEX...; done
DROP t; RENAME new_t (and all its indices). attach/inherit, etc.
commit;
. Speaking of which, I think LIKE needs a new option for ACCESS METHOD, which
is otherwise lost.
Cheers,
Justin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: pryzby@telsasoft.com, hlinnaka@iki.fi, aagrawal@pivotal.io, ashu.coek88@gmail.com
Subject: Re: Zedstore - compressed in-core columnar storage
In-Reply-To: <20190818193533.GL11185@telsasoft.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox