Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nk8mt-0005BZ-Bd for pgsql-sql@arkaria.postgresql.org; Thu, 28 Apr 2022 18:22:47 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1nk8mr-0005AE-I0 for pgsql-sql@arkaria.postgresql.org; Thu, 28 Apr 2022 18:22:45 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nk8mr-00059x-5Y for pgsql-sql@lists.postgresql.org; Thu, 28 Apr 2022 18:22:45 +0000 Received: from mail-lf1-x12e.google.com ([2a00:1450:4864:20::12e]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1nk8mk-0008CO-Fw for pgsql-sql@lists.postgresql.org; Thu, 28 Apr 2022 18:22:43 +0000 Received: by mail-lf1-x12e.google.com with SMTP id bq30so10124374lfb.3 for ; Thu, 28 Apr 2022 11:22:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heimdalldata.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=AjN96ES3XaPgbscJMPR3q3FMmNziMzgugB8O41UA9C0=; b=jjkwq+GsA3u5EKGczITwTBW+IIqVk99wmqc7WEgJPOWAmdWnU/cJbMMJrxjcZN3mo4 kiMnNbT15APcy4Dru+mtzBQDpdFDRVUwNkb4WlzK4mnoM1e+CTPw4gbs4Ff0iSmmZVLt S/5+McetvK33F/sQGvt5zx8Wy/RUSg6GXlE9c= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=AjN96ES3XaPgbscJMPR3q3FMmNziMzgugB8O41UA9C0=; b=3T5i7Bq+frDP+Uq9PwRh4wrcFHsSIlEsVF9V9Y/ViUtjiSaMRoo41uTFj19LSzrqup oIB8YGPYlihOR+15n4O7/EotrL17I7cC/5qPUxa/LGyTG5r+vHoK17wNomIF3Pz68hvp ey7ecdeLSIizw+A8z/fFPD0V/Bn57kmybQ12Y2MpHOMqnHoEztDciwhZ31ejuW/63tEx OBX3cwlnmMhsTKugudexS4qrGRSYZCYg3HVpM3PKMws13s4A0dpMLbeVQB2wrzEbiRiq 3OOv2njQPvZKeJC5oO+BTgkJyFeciIPGRq9ziFyQQHrmqeTwIDXwmJ1lP3XG6796+ETG Ij6g== X-Gm-Message-State: AOAM533xmfahGA0xFwQBifmDsWVIWtGVEAORB33aI0K1VJPEKwEeC/8i HfqKq4mugYWRj3Klg+hittGvGyZo/Tth3Ctcn2PpCw== X-Google-Smtp-Source: ABdhPJwaNklHywGAI923imF95PyjT4C3Je3okye1cGP5QtefiYhw3UPPFRC4/qvfI/qDosS7E3qFAzvAOLaNFwEJRow= X-Received: by 2002:a05:6512:38c5:b0:471:c17e:59a9 with SMTP id p5-20020a05651238c500b00471c17e59a9mr25175470lft.137.1651170156863; Thu, 28 Apr 2022 11:22:36 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Erik Brandsberg Date: Thu, 28 Apr 2022 14:22:25 -0400 Message-ID: Subject: Re: parallel-processing multiple similar query tasks - any example? To: Shaozhong SHI Cc: Steve Midgley , pgsql-sql Content-Type: multipart/alternative; boundary="000000000000a9f22205ddbb0223" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --000000000000a9f22205ddbb0223 Content-Type: text/plain; charset="UTF-8" None of this discussion is really specific to postgres. On Thu, Apr 28, 2022 at 1:46 PM Shaozhong SHI wrote: > > > On Thu, 28 Apr 2022 at 18:15, Steve Midgley wrote: > >> >> >> On Wed, Apr 27, 2022 at 4:34 PM Shaozhong SHI >> wrote: >> >>> >>> >>> >>> multiple similar query tasks are as follows: >>> >>> select * from a_table where country ='UK' >>> select * from a_table where country='France' >>> and so on >>> >>> How best to parallel-processing such types of multiple similar query >>> tasks? >>> >>> >>> This depends on how you are engaging with the queries when they return >> results. Let's assume you are running them from a programming environment >> with an ORM layer. In that case you can run each query in a separate thread >> and connection pipe, and the queries will run async just fine. If you are >> running at the command line using psql, you can just open multiple shells >> and run each query from a different shell terminal. >> >> Postgres is very good at async queries, so your challenge is really >> figuring out how you will use the results of each query and setting up the >> environment sending the queries to perform asynchronously. >> >> Steve >> > > Hi, Steve, > > That is very useful. > > All we want to do is to process a large amount of data. > > I found loops of recursive queries are very time consuming and will not > finish on time. > > Measures like indexing are simply not adequate to address the problem. > > I am thinking of making use of Linux capability to fire off concurrent > processors. > > So long as it is efficient, we can always work out how to ask it to return > results. > > Regards, > > David > --000000000000a9f22205ddbb0223 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
None of this discussion is really specific to postgres.=C2= =A0=C2=A0

On Thu, Apr 28, 2022 at 1:46 PM Shaozhong SHI <shishaozhong@gmail.com> wrote:


On Thu, 28 Apr 2022 at 18:15, Steve Midgley <science@misuse.org> wrote:=


On Wed, Apr 27, 2022 at 4:34 PM Shaozhong SHI <shishaozhong@gmai= l.com> wrote:



multiple similar query tasks= are as follows:

select * from a_table where country =3D= 'UK'
select * from a_table where country=3D'France= 9;
and so on

How best to parallel-proces= sing such types of multiple similar query tasks?

<= br>
This depends on how you are en= gaging with the queries when they return results. Let's assume you are = running them from a programming environment with an ORM layer. In that case= you can run each query in a separate thread and connection pipe, and the q= ueries will run async just fine. If you are running at the command line usi= ng psql, you can just open multiple shells and run each query from a differ= ent shell terminal.

Postgres is very good at async= queries, so your challenge is really figuring out how you will use the res= ults of each query and setting up the environment sending the queries to pe= rform asynchronously.

Steve

Hi, Steve,

That is ver= y useful.

All we want to do is to process a large = amount of data.

I found loops of recursive queries= are very time consuming and will not finish on time.

<= div>Measures like indexing are simply not adequate to address the problem.<= /div>

I am thinking of making use of Linux capability to= fire off concurrent processors.

So long as it is = efficient, we can always work out how to ask it to return results.

Regards,

David=C2=A0
<= /div>
--000000000000a9f22205ddbb0223--