Subject: Columnar format export in Postgres

From: Sushrut Shivaswamy <[email-masked]>
Date: Wed, 12 Jun 2024 22:26:30 +0530

Hey Postgres team,

I have been working on adding support for columnar format export to
Postgres to speed up analytics queries.
I've created an extension that achieves this functionality here
<https://github.com/sushrut141/pg_analytica>.

I"m looking to improve the performance of this extension to enable drop-in
analytics support for Postgres. Some immediate improvements I have in mind
are:
 - Reduce memory consumption when exporting table data to columnar format
 - Create a native planner / execution hook that can read columnar data
with vectorised operations.

It would be very helpful if you could take a look and suggest improvements
to the extension.
Hopefully, this extension can be shipped by default with postgres at some
point in the future.

Thanks,
Sushrut

---

From: Sutou Kouhei <[email-masked]>
Date: Thu, 13 Jun 2024 09:21:34 +0900 (JST)

Hi,

In <[email-masked]>
  "Columnar format export in Postgres" on Wed, 12 Jun 2024 22:26:30 +0530,

> Postgres to speed up analytics queries.

FYI: I'm proposing making COPY format extendable:

* https://www.postgresql.org/message-id/flat/[email-masked]
* https://commitfest.postgresql.org/48/4681/

If it's accepted, we can implement extensions for COPY
FORMAT arrow and COPY FORMAT parquet. With these extensions,
we can use file_fdw to read Apache Arrow and Apache Parquet
file because file_fdw is based on COPY FROM:
https://www.postgresql.org/docs/current/file-fdw.html

If you're interested in this proposal, you can review the
latest proposed patch set to proceed this proposal.

>  - Reduce memory consumption when exporting table data to columnar format

The above COPY support will help this.

Thanks,
-- 
kou