Subject: parallel-processing multiple similar query tasks - any example?

From: Shaozhong SHI <[email-masked]>
Date: Thu, 28 Apr 2022 00:33:49 +0100

multiple similar query tasks are as follows:

select * from a_table where country ='UK'
select * from a_table where country='France'
and so on

How best to parallel-processing such types of multiple similar query tasks?

Any example available?

Regards,

David

---

From: Steve Midgley <[email-masked]>
Date: Thu, 28 Apr 2022 10:14:47 -0700

On Wed, Apr 27, 2022 at 4:34 PM Shaozhong SHI <[email-masked]>
wrote:

> This depends on how you are engaging with the queries when they return
results. Let's assume you are running them from a programming environment
with an ORM layer. In that case you can run each query in a separate thread
and connection pipe, and the queries will run async just fine. If you are
running at the command line using psql, you can just open multiple shells
and run each query from a different shell terminal.

Postgres is very good at async queries, so your challenge is really
figuring out how you will use the results of each query and setting up the
environment sending the queries to perform asynchronously.

Steve

---

From: Shaozhong SHI <[email-masked]>
Date: Thu, 28 Apr 2022 18:46:01 +0100

> environment sending the queries to perform asynchronously.

Hi, Steve,

That is very useful.

All we want to do is to process a large amount of data.

I found loops of recursive queries are very time consuming and will not
finish on time.

Measures like indexing are simply not adequate to address the problem.

I am thinking of making use of Linux capability to fire off concurrent
processors.

So long as it is efficient, we can always work out how to ask it to return
results.

Regards,

David

---

From: Erik Brandsberg <[email-masked]>
Date: Thu, 28 Apr 2022 14:22:25 -0400

None of this discussion is really specific to postgres.