Received: from malur.postgresql.org ([217.196.149.56])
	by arkaria.postgresql.org with esmtps
 (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256)
	(Exim 4.92)
	(envelope-from <psycopg-owner+archive@lists.postgresql.org>)
	id 1l8mL0-0006Es-9Z
	for psycopg@arkaria.postgresql.org; Sun, 07 Feb 2021 15:51:02 +0000
Received: from localhost ([127.0.0.1] helo=malur.postgresql.org)
	by malur.postgresql.org with esmtp (Exim 4.92)
	(envelope-from <psycopg-owner+archive@lists.postgresql.org>)
	id 1l8mKy-0006C2-R7
	for psycopg@arkaria.postgresql.org; Sun, 07 Feb 2021 15:51:00 +0000
Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29])
	by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256)
	(Exim 4.92)
	(envelope-from <daniele.varrazzo@gmail.com>)
	id 1l8mKy-0006Bv-JM
	for psycopg@lists.postgresql.org; Sun, 07 Feb 2021 15:51:00 +0000
Received: from mail-lf1-x12b.google.com ([2a00:1450:4864:20::12b])
	by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128)
	(Exim 4.92)
	(envelope-from <daniele.varrazzo@gmail.com>)
	id 1l8mKr-0002zh-FI
	for psycopg@postgresql.org; Sun, 07 Feb 2021 15:50:59 +0000
Received: by mail-lf1-x12b.google.com with SMTP id h7so18454329lfc.6
        for <psycopg@postgresql.org>; Sun, 07 Feb 2021 07:50:53 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20161025;
        h=mime-version:from:date:message-id:subject:to;
        bh=Fm6j2lFoV6qV0cUp6N5+5ENiAjiNFtQQShZui7wou8o=;
        b=j8//ob2+YHufBGk+h55Ry0RfoMo/j1Cl2CEIrCKg9ZjtOPgtA4H53meGVbGigMPUvT
         EH7N8pxRDlqfq73+T1pNf0/J0LPJKhgE3y0LWNvwjbmTcaPdKDVJMMI2ur4nt0MXPjBW
         o356x8wM0jkz/E2ZtzHknRnmDPxzPdpk+Nem/vhuKS6hEtooa9/HDZi6v6pN/ew42HAD
         U7Vs6jvQGUHpX3fXiZasJNH9SfmzUWXnUqMHY80l0o6+asW8DUFtaWRImT9HWnYLHmdl
         kBJq4mXnepBRofzFY7Jou44PxTDpjZFxmDx6jp0fOAf1P4Q9mUoLWSuFefxAwYFky6SN
         CF1w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:mime-version:from:date:message-id:subject:to;
        bh=Fm6j2lFoV6qV0cUp6N5+5ENiAjiNFtQQShZui7wou8o=;
        b=h315mHEDMbillpeLqe9lNE0AOpi8dh6MmySaI2+cWrm4479vRjPG9N+le9b3zlY62e
         zwOGE35Crs9EHEqvUkZU3BOZf4LkVyCg71U/58932CuXBcTz6XYdhXeR867jyAgXCyDK
         OuNx7t57jcAE0Tx1wf7suScfTg+cXsHzCiu3xULxmpg8/u7xOJuFPTq0a47aFmG94CFB
         Gkligg0t6Q+HngrS9RRxW3FRYgjRzCNBilux6tyHkklrFUbVgFwNAsl4HttMA6YAlD+V
         /Of2LaEB4sWqwqY0V9swjq7FgAnIChwexs70GPU8MdF81LfCas284iNUcm58/SfcbyGG
         bV/Q==
X-Gm-Message-State: AOAM530vNRZ56O+qdiYwaI6l3p/WzrXKk4JvF9Ul/rTlB0HTiJ6MPSSK
	tKkE9KUq1nmiatnRFicNhDqdxvq8u4HSp0DBx5Zng1aVCDPSp6f9
X-Google-Smtp-Source: 
 ABdhPJz0d2USNucqicfYcdq5aic00CQOvB7IXQ+JVCDM2zzpUGv6tPPtHc/oMQWPSzVo8mBMo4Tn7qXNpN/FNnCvavQ=
X-Received: by 2002:a19:c194:: with SMTP id
 r142mr8299998lff.102.1612713051160;
 Sun, 07 Feb 2021 07:50:51 -0800 (PST)
MIME-Version: 1.0
From: Daniele Varrazzo <daniele.varrazzo@gmail.com>
Date: Sun, 7 Feb 2021 16:50:39 +0100
Message-ID: 
 <CA+mi_8YkA=c6cDSRPtdDfaTjZHr48vh6Hqkz=iYuUpD6g-egyA@mail.gmail.com>
Subject: libpq pipeline/batch mode
To: psycopg@postgresql.org
Content-Type: text/plain; charset="UTF-8"
List-Id: <psycopg.lists.postgresql.org>
List-Help: <https://lists.postgresql.org/manage/>
List-Subscribe: <https://lists.postgresql.org/manage/>
List-Post: <mailto:psycopg@lists.postgresql.org>
List-Owner: <mailto:psycopg-owner@lists.postgresql.org>
List-Archive: <https://www.postgresql.org/list/psycopg>
Precedence: bulk

Hello,

I am pleased to see that the pipeline/batch mode is progressing and it
might be released in pg14: https://commitfest.postgresql.org/30/2724/

psycopg3 uses only async functions, even for seemingly blocking calls,
so I think it is in a good position to make use of it. I don't know
yet if it should be a special operation mode, or the normal way to
work, replacing the lock to coordinate cursor interaction with the
connection with a pair of send/receive queues, if we could make
execute only send and fetch receive (ending up giving the async fetch
a proper I/O use, to reconnect to the recent conversation about it).

However I am getting stretched pretty thinly with different directions
of work: main tasks in my queue are:

- connection pool
- binary numeric/datetime adapters
- server-side cursors
- jsonb binary protocol (something to design server-side too)

so I doubt I will be able to hack on a branch of psycopg and a branch
of the libpq to study and design the pipeline possibilities too.

If anyone would like to take up this research task, or even would like
to collaborate on any other task listed above (or on the project
board: https://github.com/psycopg/psycopg3/projects/1) I would be
grateful!

Cheers!

-- Daniele