public inbox for [email protected]
help / color / mirror / Atom feedFrom: Brian M Hamlin <[email protected]>
To: [email protected]
Subject: Re: Parquet support
Date: Wed, 23 Nov 2022 12:18:43 -0800
Message-ID: <[email protected]> (raw)
In-Reply-To: <CA+mi_8b-c8Q4zpkd=X+mqRivscEoFmn_f1UUh-wr3UjEdD+sXg@mail.gmail.com>
References: <CAKvpR7MaATqwiDYG1YyixGnUrqyjjq1z2aO3T-a6Rh82vTvcWg@mail.gmail.com>
<CA+mi_8buxHRjeabOPWfPpg1RKvRpC2tm1Ass7uttgFvLRoRH7Q@mail.gmail.com>
<CAMqTPqnvLxiUcfOXLwBP3NsKo1OfEE8E-YhSHFUJV0GX2M9eEw@mail.gmail.com>
<CA+mi_8b-c8Q4zpkd=X+mqRivscEoFmn_f1UUh-wr3UjEdD+sXg@mail.gmail.com>
Hi - desktop linux user/maker here in California --
The engineering stakes are high in the clouds these days. There are
some important efforts underway to make "cloud-native" ways for python,
python installation, python data and python communication tools. In my
corners of the world (remote sensing, urban planning) that means DASK
and xarray. As a desktop linux distribution, we/OSGeoLive ship both,
and enthusiastically so.. the "cloud-native" data storage formats ZARR
and parquet, not so much. My best understanding is xarray is a happy
medium between "what only runs on cloud" and "the powerful Linux I can
run myself on standard equipment today" ..
I support a python ecosystem that individual people can run entirely
locally, and can interoperate well with standard networking and data
formats. Not every python environment is doing that.. change happens
interested to see the common and useful Python discussion here,
regarding Postgresql, PostGIS and cloudy interoperability.
--Brian M Hamlin / MAPLABS / OSGeoLive PSC
On 11/23/22 12:00, Daniele Varrazzo wrote:
> On Wed, 23 Nov 2022 at 20:56, Vladimir Ryabtsev <[email protected]> wrote:
>> Just curious folks, what are your thoughts about the scope of that potential support? What is the use case? Is it loading data from Parquet to Postgres (and back)? Why is the combination with Python modules like pyarrow not enough?
> I am not an expert, but I understand that Python-Postgres roundtrip
> goes via generating and parsing CSV files, whereas there is some
> performance gain to be had by creating native arrow data.
>
> -- Daniele
>
>
view thread (5+ messages)
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected]
Subject: Re: Parquet support
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox