agora inbox for pgsql-sql@postgresql.org
help / color / mirror / Atom feedFrom: Jiří Fejfar <jurafejfar@gmail.com>
To: Mark Teper <mark.teper@gmail.com>
To: pgsql-sql@lists.postgresql.org
Subject: Re: plpython transforms vs. arrays
Date: Fri, 3 May 2019 06:53:43 +0200
Message-ID: <2dbe9d47-2a23-274e-ca46-6896d4635845@gmail.com> (raw)
In-Reply-To: <CAMwSyGra+i1oL0pKgCv1WeE6GeBNhJJZTJ0bGxLNqJ0SFeEsVw@mail.gmail.com>
References: <CAMwSyGra+i1oL0pKgCv1WeE6GeBNhJJZTJ0bGxLNqJ0SFeEsVw@mail.gmail.com>
Dear Mark,
I am also looking around to find way how to do effectively (in
parallel?) computations (with potentially large and sometimes sparse)
matrices directly in PostgreSQL. Some time ago I have found this
experimental extension https://github.com/PandaPost/panda_post which
"allow you to represent Python NumPy/Pandas objects in Postgres". There
is also http://madlib.apache.org/ but is seems too much "heavyweight"
for my use-case.
Now I do not have much time to spent with this, but I hope in late
summer it will be better. I am looking forward what will be your
conclusions.
Good luck, Jiří.
On 5/2/19 7:51 PM, Mark Teper wrote:
>
> Hi,
>
> I'm trying to build some numerical processing algorithms on Postgres
> Array data types. Using PL/Python I can get access to the numpy
> libraries but the performance is not great. A guess is that there is
> a lot of overhead going from Postgres -> Python List -> Numpy and back
> again.
>
> I'd like to test if that's the issue, and potentially fix by creating
> a C extension to convert directly from Postgres Array types to Numpy
> Array types. I _think_ I have the C side somewhat working, but I
> can't get Postgres to use the transform.
>
> What I have:
>
> ---
>
> CREATE FUNCTION arr_to_np(val internal) RETURNS internal LANGUAGE C AS
> 'MODULE_PATHNAME', 'arr_to_np';
>
> CREATE FUNCTION np_to_arr(val internal) RETURNS real[] LANGUAGE C AS
> 'MODULE_PATHNAME', 'np_to_arr';
>
> CREATE TRANSFORM FOR real[] LANGUAGE plpythonu (
>
> FROM SQL WITH FUNCTION arr_to_np(internal),
>
> TO SQL WITH FUNCTION np_to_arr(internal)
>
> );
>
> CREATE FUNCTION fn (a integer[]) RETURNS integer
>
> TRANSFORM FOR TYPE real[]
>
> AS $$ return a $$ LANGUAGE plpythonu;
>
> ----
>
> The problem is this produces an error that transforms for type "real"
> doesn't work. It doesn't seem to allow for transforms on array's as
> opposed to underlying types. Is it possible to tell it to apply the
> transform to the array?
>
> Thanks,
>
> Mark
>
view thread (3+ messages) latest in thread
Message-ID: <2dbe9d47-2a23-274e-ca46-6896d4635845@gmail.com>
Permalink: ../2dbe9d47-2a23-274e-ca46-6896d4635845@gmail.com/
Also on: postgresql.org/message-id/2dbe9d47-2a23-274e-ca46-6896d4635845@gmail.com
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-sql@postgresql.org
Cc: jurafejfar@gmail.com, mark.teper@gmail.com, pgsql-sql@lists.postgresql.org
Subject: Re: plpython transforms vs. arrays
In-Reply-To: <2dbe9d47-2a23-274e-ca46-6896d4635845@gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox