Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1hG7PY-0004Cl-GQ for pgsql-hackers@arkaria.postgresql.org; Mon, 15 Apr 2019 19:37:00 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1hG7PX-0002Ub-8n for pgsql-hackers@arkaria.postgresql.org; Mon, 15 Apr 2019 19:36:59 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1hG7PW-0002UU-Sy for pgsql-hackers@lists.postgresql.org; Mon, 15 Apr 2019 19:36:59 +0000 Received: from sss.pgh.pa.us ([66.207.139.130]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1hG7PU-0004ZU-3K for pgsql-hackers@postgresql.org; Mon, 15 Apr 2019 19:36:57 +0000 Received: from sss1.sss.pgh.pa.us (localhost [127.0.0.1]) by sss.pgh.pa.us (8.14.4/8.14.4) with ESMTP id x3FJaiBb023413; Mon, 15 Apr 2019 15:36:44 -0400 From: Tom Lane To: Andres Freund cc: Robert Haas , Stephen Frost , Tomas Vondra , Alvaro Herrera , Konstantin Knizhnik , Ashwin Agrawal , PostgreSQL mailing lists Subject: Re: Zedstore - compressed in-core columnar storage In-reply-to: <20190415192004.od3paxoo5rlogmm4@alap3.anarazel.de> References: <20190409155121.GA4320@alvherre.pgsql> <20190414162210.6djspkmbspbd4els@development> <20190415123424.GD6197@tamriel.snowman.net> <30261.1555341038@sss.pgh.pa.us> <20789.1555353343@sss.pgh.pa.us> <20190415192004.od3paxoo5rlogmm4@alap3.anarazel.de> Comments: In-reply-to Andres Freund message dated "Mon, 15 Apr 2019 12:20:04 -0700" MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <23411.1555357004.1@sss.pgh.pa.us> Date: Mon, 15 Apr 2019 15:36:44 -0400 Message-ID: <23412.1555357004@sss.pgh.pa.us> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk Andres Freund writes: > We also have at times pretty explicitly resisted making crucial pieces > of infrastructure usable outside of core. E.g. because it's legitimately > hard (grammar extensibility), or because we'd some concerns around > stability and the exact approach (WAL - the generic stuff is usable for > anything that wants to even be somewhat efficient, some xlog > integration). So there's several types of extensions that one > realistically cannot do out of core, by our choice. Well, the grammar issue comes from a pretty specific technical problem: bison grammars don't cope with run-time extension, and moving off of bison would cost a lot of work, and probably more than just work (i.e., probable loss of ability to detect grammar ambiguity). WAL extensibility likewise has some technical issues that are hard to surmount (how do you find the code for replaying an extension WAL record, when you can't read catalogs). I think we could fix the latter, it's just that no one has yet troubled to expend the effort. Similarly, things like the planner's hard-wired handling of physical-tlist optimization are certainly a problem for column stores, but I think the way to solve that is to provide an actual extension capability, not merely replace one hard-wired behavior with two. As a counterpoint to my gripe about SP-GiST being a time sink, I do not think I'll regret the time I spent a few months ago on implementing "planner support function" hooks. I'm all in favor of adding flexibility like that. regards, tom lane