MIME-Version: 1.0
References: 
 <CA+i5JwaC9Z-rjWiyo7LpncdK=CuQcKzuVNMnAhPW1oi=8c9Sug@mail.gmail.com>
 <CA+i5JwZ1_gu0g-X+5P8hVhyc1YS7F4nhrcEQdZKADOp+7o+1SQ@mail.gmail.com>
 <CAJexoSJy1yXa9vQuoN0U8ykf0w29yWJWw9aODak6sHsx+wzjNw@mail.gmail.com>
In-Reply-To: 
 <CAJexoSJy1yXa9vQuoN0U8ykf0w29yWJWw9aODak6sHsx+wzjNw@mail.gmail.com>
From: Shaozhong SHI <shishaozhong@gmail.com>
Date: Thu, 28 Apr 2022 18:46:01 +0100
Message-ID: 
 <CA+i5Jwa9_+WkJj+PGz55VrKF9-gAeeRExPEzeQhNiPp0_dtj=Q@mail.gmail.com>
Subject: Re: parallel-processing multiple similar query tasks - any example?
To: Steve Midgley <science@misuse.org>
Cc: pgsql-sql <pgsql-sql@lists.postgresql.org>
Content-Type: multipart/alternative; boundary="00000000000071b6e605ddba80a9"
Archived-At: 
 <https://www.postgresql.org/message-id/CA%2Bi5Jwa9_%2BWkJj%2BPGz55VrKF9-gAeeRExPEzeQhNiPp0_dtj%3DQ%40mail.gmail.com>
Precedence: bulk

--00000000000071b6e605ddba80a9
Content-Type: text/plain; charset="UTF-8"

On Thu, 28 Apr 2022 at 18:15, Steve Midgley <science@misuse.org> wrote:

>
>
> On Wed, Apr 27, 2022 at 4:34 PM Shaozhong SHI <shishaozhong@gmail.com>
> wrote:
>
>>
>>
>>
>> multiple similar query tasks are as follows:
>>
>> select * from a_table where country ='UK'
>> select * from a_table where country='France'
>> and so on
>>
>> How best to parallel-processing such types of multiple similar query
>> tasks?
>>
>>
>> This depends on how you are engaging with the queries when they return
> results. Let's assume you are running them from a programming environment
> with an ORM layer. In that case you can run each query in a separate thread
> and connection pipe, and the queries will run async just fine. If you are
> running at the command line using psql, you can just open multiple shells
> and run each query from a different shell terminal.
>
> Postgres is very good at async queries, so your challenge is really
> figuring out how you will use the results of each query and setting up the
> environment sending the queries to perform asynchronously.
>
> Steve
>

Hi, Steve,

That is very useful.

All we want to do is to process a large amount of data.

I found loops of recursive queries are very time consuming and will not
finish on time.

Measures like indexing are simply not adequate to address the problem.

I am thinking of making use of Linux capability to fire off concurrent
processors.

So long as it is efficient, we can always work out how to ask it to return
results.

Regards,

David

--00000000000071b6e605ddba80a9
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">=
<div dir=3D"ltr" class=3D"gmail_attr">On Thu, 28 Apr 2022 at 18:15, Steve M=
idgley &lt;<a href=3D"mailto:science@misuse.org">science@misuse.org</a>&gt;=
 wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px =
0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir=
=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote"><div dir=
=3D"ltr" class=3D"gmail_attr">On Wed, Apr 27, 2022 at 4:34 PM Shaozhong SHI=
 &lt;<a href=3D"mailto:shishaozhong@gmail.com" target=3D"_blank">shishaozho=
ng@gmail.com</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" styl=
e=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);paddin=
g-left:1ex"><div dir=3D"ltr"><div class=3D"gmail_quote"><div dir=3D"ltr" cl=
ass=3D"gmail_attr"><br></div><br><br><div dir=3D"ltr">multiple similar quer=
y tasks are as follows:<div><br></div><div>select * from a_table where coun=
try =3D&#39;UK&#39;</div><div>select * from a_table where country=3D&#39;Fr=
ance&#39;</div><div>and so on</div><div><br></div><div>How best to parallel=
-processing such types of multiple similar query tasks?</div><div><br></div=
><div><br></div></div></div></div></blockquote><div>This depends on how you=
 are engaging with the queries when they return results. Let&#39;s assume y=
ou are running them from a programming environment with an ORM layer. In th=
at case you can run each query in a separate thread and connection pipe, an=
d the queries will run async just fine. If you are running at the command l=
ine using psql, you can just open multiple shells and run each query from a=
 different shell terminal.</div><div><br></div><div>Postgres is very good a=
t async queries, so your challenge is really figuring out how you will use =
the results of each query and setting up the environment sending the querie=
s to perform asynchronously.</div><div><br></div><div>Steve</div></div></di=
v></blockquote><div><br></div><div>Hi, Steve,</div><div><br></div><div>That=
 is very useful.</div><div><br></div><div>All we want to do is to process a=
 large amount of data.</div><div><br></div><div>I found loops of recursive =
queries are very time consuming and will not finish on time.</div><div><br>=
</div><div>Measures like indexing are simply not adequate to address the pr=
oblem.</div><div><br></div><div>I am thinking of making use of Linux capabi=
lity to fire off concurrent processors.</div><div><br></div><div>So long as=
 it is efficient, we can always work out how to ask it to return results.</=
div><div><br></div><div>Regards,</div><div><br></div><div>David=C2=A0</div>=
</div></div>

--00000000000071b6e605ddba80a9--