public inbox for [email protected]help / color / mirror / Atom feed
Parquet support 5+ messages / 4 participants [nested] [flat]
* Parquet support @ 2022-11-23 18:49 Christopher Bader <[email protected]> 0 siblings, 1 reply; 5+ messages in thread From: Christopher Bader @ 2022-11-23 18:49 UTC (permalink / raw) To: [email protected] Does psycopg support parquet as an input format? Thanks, Christopher Bader Staff Data Scientist Zscaler [email protected] ^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Parquet support @ 2022-11-23 18:56 Daniele Varrazzo <[email protected]> parent: Christopher Bader <[email protected]> 0 siblings, 1 reply; 5+ messages in thread From: Daniele Varrazzo @ 2022-11-23 18:56 UTC (permalink / raw) To: Christopher Bader <[email protected]>; +Cc: [email protected] On Wed, 23 Nov 2022 at 19:49, Christopher Bader <[email protected]> wrote: > > Does psycopg support parquet as an input format? No, not yet. I had some conversation in the past around parquet input/output: it is a major project which I would like to either develop or see developed, but at the moment I don't have the several months required to do the former, and nobody has volunteered for the latter. Cheers -- Daniele ^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Parquet support @ 2022-11-23 19:56 Vladimir Ryabtsev <[email protected]> parent: Daniele Varrazzo <[email protected]> 0 siblings, 1 reply; 5+ messages in thread From: Vladimir Ryabtsev @ 2022-11-23 19:56 UTC (permalink / raw) To: Daniele Varrazzo <[email protected]>; +Cc: Christopher Bader <[email protected]>; [email protected] Just curious folks, what are your thoughts about the scope of that potential support? What is the use case? Is it loading data from Parquet to Postgres (and back)? Why is the combination with Python modules like pyarrow not enough? Regards, --VR On Wed, 23 Nov 2022 at 10:56, Daniele Varrazzo <[email protected]> wrote: > On Wed, 23 Nov 2022 at 19:49, Christopher Bader <[email protected]> > wrote: > > > > Does psycopg support parquet as an input format? > > No, not yet. > > I had some conversation in the past around parquet input/output: it is > a major project which I would like to either develop or see developed, > but at the moment I don't have the several months required to do the > former, and nobody has volunteered for the latter. > > Cheers > > -- Daniele > > > ^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Parquet support @ 2022-11-23 20:00 Daniele Varrazzo <[email protected]> parent: Vladimir Ryabtsev <[email protected]> 0 siblings, 1 reply; 5+ messages in thread From: Daniele Varrazzo @ 2022-11-23 20:00 UTC (permalink / raw) To: Vladimir Ryabtsev <[email protected]>; +Cc: Christopher Bader <[email protected]>; [email protected] On Wed, 23 Nov 2022 at 20:56, Vladimir Ryabtsev <[email protected]> wrote: > > Just curious folks, what are your thoughts about the scope of that potential support? What is the use case? Is it loading data from Parquet to Postgres (and back)? Why is the combination with Python modules like pyarrow not enough? I am not an expert, but I understand that Python-Postgres roundtrip goes via generating and parsing CSV files, whereas there is some performance gain to be had by creating native arrow data. -- Daniele ^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Parquet support @ 2022-11-23 20:18 Brian M Hamlin <[email protected]> parent: Daniele Varrazzo <[email protected]> 0 siblings, 0 replies; 5+ messages in thread From: Brian M Hamlin @ 2022-11-23 20:18 UTC (permalink / raw) To: [email protected] Hi - desktop linux user/maker here in California -- The engineering stakes are high in the clouds these days. There are some important efforts underway to make "cloud-native" ways for python, python installation, python data and python communication tools. In my corners of the world (remote sensing, urban planning) that means DASK and xarray. As a desktop linux distribution, we/OSGeoLive ship both, and enthusiastically so.. the "cloud-native" data storage formats ZARR and parquet, not so much. My best understanding is xarray is a happy medium between "what only runs on cloud" and "the powerful Linux I can run myself on standard equipment today" .. I support a python ecosystem that individual people can run entirely locally, and can interoperate well with standard networking and data formats. Not every python environment is doing that.. change happens interested to see the common and useful Python discussion here, regarding Postgresql, PostGIS and cloudy interoperability. --Brian M Hamlin / MAPLABS / OSGeoLive PSC On 11/23/22 12:00, Daniele Varrazzo wrote: > On Wed, 23 Nov 2022 at 20:56, Vladimir Ryabtsev <[email protected]> wrote: >> Just curious folks, what are your thoughts about the scope of that potential support? What is the use case? Is it loading data from Parquet to Postgres (and back)? Why is the combination with Python modules like pyarrow not enough? > I am not an expert, but I understand that Python-Postgres roundtrip > goes via generating and parsing CSV files, whereas there is some > performance gain to be had by creating native arrow data. > > -- Daniele > > ^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2022-11-23 20:18 UTC | newest] Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2022-11-23 18:49 Parquet support Christopher Bader <[email protected]> 2022-11-23 18:56 ` Daniele Varrazzo <[email protected]> 2022-11-23 19:56 ` Vladimir Ryabtsev <[email protected]> 2022-11-23 20:00 ` Daniele Varrazzo <[email protected]> 2022-11-23 20:18 ` Brian M Hamlin <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox