Date: Mon, 15 Apr 2019 08:34:24 -0400
From: Stephen Frost <sfrost@snowman.net>
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: Robert Haas <robertmhaas@gmail.com>,
	Alvaro Herrera <alvherre@2ndquadrant.com>,
	Konstantin Knizhnik <k.knizhnik@postgrespro.ru>,
	Ashwin Agrawal <aagrawal@pivotal.io>,
	PostgreSQL mailing lists <pgsql-hackers@postgresql.org>
Subject: Re: Zedstore - compressed in-core columnar storage
Message-ID: <20190415123424.GD6197@tamriel.snowman.net>
References: <bf4b8f5f-90c9-025d-013c-87cf998204ad@postgrespro.ru>
 <20190409155121.GA4320@alvherre.pgsql>
 <CA+TgmoardtfM+0nfj=g1qrH5khuHP6Oi5ZS546CLx_iQUG620g@mail.gmail.com>
 <20190414162210.6djspkmbspbd4els@development>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="EeavCsNdDeHREbkS"
Content-Disposition: inline
In-Reply-To: <20190414162210.6djspkmbspbd4els@development>
User-Agent: Mutt/1.5.24 (2015-08-30)
Precedence: bulk


--EeavCsNdDeHREbkS
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Greetings,

* Tomas Vondra (tomas.vondra@2ndquadrant.com) wrote:
> On Tue, Apr 09, 2019 at 02:29:09PM -0400, Robert Haas wrote:
> >On Tue, Apr 9, 2019 at 11:51 AM Alvaro Herrera <alvherre@2ndquadrant.com=
> wrote:
> >>This is not surprising, considering that columnar store is precisely the
> >>reason for starting the work on table AMs.
> >>
> >>We should certainly look into integrating some sort of columnar storage
> >>in mainline.  Not sure which of zedstore or VOPS is the best candidate,
> >>or maybe we'll have some other proposal.  My feeling is that having more
> >>than one is not useful; if there are optimizations to one that can be
> >>borrowed from the other, let's do that instead of duplicating effort.
> >
> >I think that conclusion may be premature.  There seem to be a bunch of
> >different ways of doing columnar storage, so I don't know how we can
> >be sure that one size will fit all, or that the first thing we accept
> >will be the best thing.
> >
> >Of course, we probably do not want to accept a ton of storage manager
> >implementations is core.  I think if people propose implementations
> >that are poor quality, or missing important features, or don't have
> >significantly different use cases from the ones we've already got,
> >it's reasonable to reject those.  But I wouldn't be prepared to say
> >that if we have two significantly different column store that are both
> >awesome code with a complete feature set and significantly disjoint
> >use cases, we should reject the second one just because it is also a
> >column store.  I think that won't get out of control because few
> >people will be able to produce really high-quality implementations.
> >
> >This stuff is hard, which I think is also why we only have 6.5 index
> >AMs in core after many, many years.  And our standards have gone up
> >over the years - not all of those would pass muster if they were
> >proposed today.
>=20
> It's not clear to me whether you're arguing for not having any such
> implementation in core, or having multiple ones? I think we should aim
> to have at least one in-core implementation, even if it's not the best
> possible one for all sizes. It's not like our rowstore is the best
> possible implementation for all cases either.
>=20
> I think having a colstore in core is important not just for adoption,
> but also for testing and development of the executor / planner bits.

Agreed.

> If we have multiple candidates with sufficient code quality, then we may
> consider including both. I don't think it's very likely to happen in the
> same release, considering how much work it will require. And I have no
> idea if zedstore or VOPS are / will be the only candidates - it's way
> too early at this point.

Definitely, but having as many different indexes as we have is certainly
a good thing and we should be looking to a future where we have multiple
in-core options for row and column-oriented storage.

> FWIW I personally plan to focus primarily on the features that aim to
> be included in core, and that applies to colstores too.

Yeah, same here.

Thanks!

Stephen

--EeavCsNdDeHREbkS
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCgAGBQJctHpQAAoJEO1sijiDR2RVfN4P/ienI4Gm6ZZ3OfW/29YjeoPA
Mj0IHw+Zk5YOoMXXsnX9FYUmB0c6Mq5HuU8vts6VwrBKKgZHUUvNOWiF5U/jJyKP
9lnZDwP96ADtW0x7LV2zWpt/H3fw535cy3pw2cu0tkJ2bj+oNliKxRV4yiPyKhzw
NO6elVfdDEqpZqenHTPhCtnsgOx++Qmz37+TS+5sfaETAW/Tgan4/xdVFg+hJ3l7
VtHrHlvzna3Zs3ZARjGwXMx/aqFVIkobLYjvnMrZ7xRiKJaW1cTmlpc/RwHrOxXw
lOP2b6TlnIOWPUG+25/4h1gD5UwGrXTYsLQiDYUiOyLWX0b5F1XV6nlFYnjmlvDN
lpzX2HWj9dgRwH9prEkDj0Nw01a+HFpe0IK2BrQBFxjTe9hnH+Gmfk3Dq+wxZ8oR
ApNOPM/pp10EFSInqG5b21ZfsCKEpIp2jVHyB+FIqktfc3+BfV/N4tkxjtcSTNYe
8BkdX+IZaNq9TvA7coWbRVC4w/u35AKdk1oPQPgOhuEn0DBbXsCb4wX9wjyuKPKm
/uXNBD0sOXz+w3yHUPVx08s+tLTM0uAbDpEGuTMZe8TwxBIpyFFR7lzARDXHIFaZ
c2gxkE6n827YUDm3SWfyqGh7OCaGBiTplriJmwYGXsWEvhFvNSJEftaZW69DeloC
kY9gbuVGvonTPh0Pof0Y
=Ggdp
-----END PGP SIGNATURE-----

--EeavCsNdDeHREbkS--