From: Laurenz Albe <laurenz.albe@cybertec.at>
To: Lars Aksel Opsahl <Lars.Opsahl@nibio.no>
To: pgsql-performance@lists.postgresql.org <pgsql-performance@lists.postgresql.org>
Subject: Re: How to run in parallel in Postgres
Date: Thu, 05 Dec 2019 17:42:06 +0100
Message-ID: <65371f0819be9dc954fdece32e9df30d1c5aa39e.camel@cybertec.at> (raw)
In-Reply-To: <HE1P189MB026622ECFF2400B1932E4A2D9D5D0@HE1P189MB0266.EURP189.PROD.OUTLOOK.COM>
References: <HE1P189MB026622ECFF2400B1932E4A2D9D5D0@HE1P189MB0266.EURP189.PROD.OUTLOOK.COM>
On Thu, 2019-12-05 at 12:10 +0000, Lars Aksel Opsahl wrote:
> have a function that prepares data, so the big job can be run it in parallel.
>
> Today I have solved this by using "Gnu parallel" like this.
> psql testdb -c"\! psql -t -q -o /tmp/run_cmd.sql testdb -c\"SELECT find_overlap_gap_make_run_cmd('sl_lop.overlap_gap_input_t1','geom',4258,'sl_lop.overlap_gap_input_t1_res',50);\"; parallel -j 4
> psql testdb -c :::: /tmp/run_cmd.sql" 2>> /tmp/analyze.log;
>
> The problem here is that I depend on external code which may not be installed.
>
> Since Postgres now supports parallel I was wondering if it's easy to trigger parallel dynamically created SQL calls.
>
> If you look at https://github.com/larsop/find-overlap-and-gap/blob/master/src/test/sql/regress/find_overlap_and_gap... you see that
> find_overlap_gap_make_run_cmd generates as set of 28 sql calls.
>
> So is it in a simple way possible to use Postgres parallel functionality to call this 28 functions i parallel so I don't have dependent on externally install programs ?
>
> When this 28 sql calls are done, the find_overlap_gap_make_run_cmd may continue to the next step of work. So the function that triggers parallel calls wait for them complete and then may start on
> the next step of work.
You cannot run several queries in parallel in a PostgreSQL function.
You may want to have a look at PL/Proxy which might be used for things like that.
Yours,
Laurenz Albe
--
Cybertec | https://www.cybertec-postgresql.com
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: laurenz.albe@cybertec.at, Lars.Opsahl@nibio.no, pgsql-performance@lists.postgresql.org
Subject: Re: How to run in parallel in Postgres
In-Reply-To: <65371f0819be9dc954fdece32e9df30d1c5aa39e.camel@cybertec.at>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox