public inbox for [email protected]
help / color / mirror / Atom feedFrom: Erik Brandsberg <[email protected]>
To: Shaozhong SHI <[email protected]>
Cc: Steve Midgley <[email protected]>
Cc: pgsql-sql <[email protected]>
Subject: Re: parallel-processing multiple similar query tasks - any example?
Date: Thu, 28 Apr 2022 14:22:25 -0400
Message-ID: <CAFcck8Hk9g6JwFD5GQsFdJ2920jbj8yct-Jr7EZhzUcgVWQw3g@mail.gmail.com> (raw)
In-Reply-To: <CA+i5Jwa9_+WkJj+PGz55VrKF9-gAeeRExPEzeQhNiPp0_dtj=Q@mail.gmail.com>
References: <CA+i5JwaC9Z-rjWiyo7LpncdK=CuQcKzuVNMnAhPW1oi=8c9Sug@mail.gmail.com>
<CA+i5JwZ1_gu0g-X+5P8hVhyc1YS7F4nhrcEQdZKADOp+7o+1SQ@mail.gmail.com>
<CAJexoSJy1yXa9vQuoN0U8ykf0w29yWJWw9aODak6sHsx+wzjNw@mail.gmail.com>
<CA+i5Jwa9_+WkJj+PGz55VrKF9-gAeeRExPEzeQhNiPp0_dtj=Q@mail.gmail.com>
None of this discussion is really specific to postgres.
On Thu, Apr 28, 2022 at 1:46 PM Shaozhong SHI <[email protected]>
wrote:
>
>
> On Thu, 28 Apr 2022 at 18:15, Steve Midgley <[email protected]> wrote:
>
>>
>>
>> On Wed, Apr 27, 2022 at 4:34 PM Shaozhong SHI <[email protected]>
>> wrote:
>>
>>>
>>>
>>>
>>> multiple similar query tasks are as follows:
>>>
>>> select * from a_table where country ='UK'
>>> select * from a_table where country='France'
>>> and so on
>>>
>>> How best to parallel-processing such types of multiple similar query
>>> tasks?
>>>
>>>
>>> This depends on how you are engaging with the queries when they return
>> results. Let's assume you are running them from a programming environment
>> with an ORM layer. In that case you can run each query in a separate thread
>> and connection pipe, and the queries will run async just fine. If you are
>> running at the command line using psql, you can just open multiple shells
>> and run each query from a different shell terminal.
>>
>> Postgres is very good at async queries, so your challenge is really
>> figuring out how you will use the results of each query and setting up the
>> environment sending the queries to perform asynchronously.
>>
>> Steve
>>
>
> Hi, Steve,
>
> That is very useful.
>
> All we want to do is to process a large amount of data.
>
> I found loops of recursive queries are very time consuming and will not
> finish on time.
>
> Measures like indexing are simply not adequate to address the problem.
>
> I am thinking of making use of Linux capability to fire off concurrent
> processors.
>
> So long as it is efficient, we can always work out how to ask it to return
> results.
>
> Regards,
>
> David
>
view thread (4+ messages)
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected]
Subject: Re: parallel-processing multiple similar query tasks - any example?
In-Reply-To: <CAFcck8Hk9g6JwFD5GQsFdJ2920jbj8yct-Jr7EZhzUcgVWQw3g@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox