MIME-Version: 1.0
References: 
 <CA+mi_8aZQXzB8Kr4V0nps3aMQ74N72uWC=pZF0P1H7AW4GLasA@mail.gmail.com>
In-Reply-To: 
 <CA+mi_8aZQXzB8Kr4V0nps3aMQ74N72uWC=pZF0P1H7AW4GLasA@mail.gmail.com>
From: Vladimir Ryabtsev <greatvovan@gmail.com>
Date: Mon, 18 Jan 2021 10:20:12 -0800
Message-ID: 
 <CAMqTPqkx0rx2a79rV7UOhs1CxXZG+vG6F_qV2QWR4+9Bd+1Ocg@mail.gmail.com>
Subject: Re: Designing a better connection pool for psycopg3
To: Daniele Varrazzo <daniele.varrazzo@gmail.com>
Cc: psycopg@postgresql.org
Content-Type: multipart/alternative; boundary="000000000000841a5d05b930c60b"
Precedence: bulk

--000000000000841a5d05b930c60b
Content-Type: text/plain; charset="UTF-8"

> Connection configuration
In my opinion it is not only static vs dynamic,
I have a feeling that subclassing gives more freedom for improper
implementation (resulting in problems). Also it depends on internal
API (not necessarily public), which puts more obligations on the
maintainer to keep it compatible with old releases. In this case
I like the configuring approach better
(and it does not exclude subclassing as well).

> Exclusive connections?
The feature itself seems useful, but what are the benefits of getting it
from the pool (e.g. checking state, reconnection, configuration)
as opposed to creating one manually?
Is such a connection counted in regard to 'minconn', 'maxconn'?

> What to do in case of connection failure?
> A first step towards a sane behaviour could be to die early, on startup,
if the connection is not working when the first pool population is done
Agree.
BTW, at which moment will the driver be trying to connect? At pool
initialization or on first "getconn()"?

> What if the database is missing in action during the program lifetime?
Following connection attempts may be repeated, with an exponential backoff,
until dying after a few minutes of fruitless attempts
Yes, I would prefer exponential backoff, with a timeout prior to "panic"
callback.
Preferably, both configurable.

> Connections usage pattern
I don't see cases where an application should bother about it,
choose whatever is better for your implementation. And yes, stack
looks reasonable.

> 'minconn', Proposed default: 4
Many services and applications do not really need many connections
but would like to benefit from automatic reconnections. So the default
is questionable, as many people will leave it unchanged.

> "get_info()"
One thing that comes into my head every time I use connection pools
is how many are really needed for my application. Usually it is adjusted
experimentally or using some performance tests. Can you collect and
provide some stats to get more insights on that? I see you want to
expose some in "get_info()", which exactly? E.g. number of times
the max limit was hit, current number of connections acquired?

> 'max_lifetime_sec'
Why is it necessary at all? Does a connection get rotten with time?
As I understand, you are planning to check a connection state in
background. As long as it is OK, what is the need to replace it?

> num_workers, default: 3
Assuming every worker does the same job, is there really the amount
of work for three workers?

"get_maintenance_task()"
This API seems to be too advanced, in what cases you see a sane use case
for it?

Thank you,
Vladimir


On Mon, 18 Jan 2021 at 04:04, Daniele Varrazzo <daniele.varrazzo@gmail.com>
wrote:

> Hello,
>
> I've been gathering a few ideas about the connection pool I would like
> to provide with psycopg3. I would be happy if you would like to take a
> look and give some feedback.
>
> https://www.psycopg.org/articles/2021/01/17/pool-design/
>
> Thank you very much,
>
> -- Daniele
>
>
>

--000000000000841a5d05b930c60b
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">&gt; Connection configuration<div>In my opinion it is not =
only static vs dynamic,<br><div>I have a feeling that subclassing gives mor=
e freedom for improper</div><div>implementation (resulting in problems). Al=
so it depends on internal</div><div>API (not necessarily=C2=A0public), whic=
h puts more obligations on the</div><div>maintainer to keep it compatible w=
ith old releases. In this case</div><div>I like the configuring approach be=
tter</div><div>(and it does not exclude subclassing as well).</div><div><br=
></div>&gt; Exclusive connections?<div>The feature itself seems useful, but=
 what are the benefits of getting it</div><div>from the pool (e.g. checking=
 state, reconnection, configuration)</div><div>as opposed to creating one m=
anually?</div><div>Is such a connection=C2=A0counted in regard to &#39;minc=
onn&#39;, &#39;maxconn&#39;?</div><div><br></div>&gt; What to do in case of=
 connection failure?<br>&gt; A first step towards a sane behaviour could be=
 to die early, on startup, if the connection is not working when the first =
pool population is done<br>Agree.</div><div>BTW, at which moment will the d=
river be trying to connect? At pool</div><div>initialization or on first &q=
uot;getconn()&quot;?</div><div><br>&gt; What if the database is missing in =
action during the program lifetime? Following connection attempts may be re=
peated, with an exponential backoff, until dying after a few minutes of fru=
itless attempts<br>Yes, I would prefer exponential backoff, with a timeout =
prior to &quot;panic&quot; callback.<br>Preferably, both configurable.<br><=
br>&gt; Connections usage pattern<br>I don&#39;t see cases where an applica=
tion should bother about it,<br>choose whatever is better for your implemen=
tation. And=C2=A0yes, stack<br>looks reasonable.</div><div><br>&gt; &#39;mi=
nconn&#39;, Proposed default: 4<br>Many services and applications do not re=
ally need many connections<br>but would like to benefit from automatic reco=
nnections. So the default<br>is questionable, as many people will leave it =
unchanged.<br><br>&gt; &quot;get_info()&quot;<br>One thing that comes into =
my head every time I use connection pools<br>is how many are really needed =
for my application. Usually it is adjusted<br>experimentally or using some =
performance tests. Can you collect and<br>provide some stats to get more in=
sights on that? I see you want to<br>expose some in &quot;get_info()&quot;,=
 which exactly? E.g. number of times<br>the max limit was hit, current numb=
er of connections acquired?</div><div><br>&gt; &#39;max_lifetime_sec&#39;<b=
r>Why is it necessary at all? Does a connection get rotten with time?<br>As=
 I understand, you are planning to check a connection state in<br>backgroun=
d. As long as it is OK, what is the need to replace it?<br><br>&gt; num_wor=
kers, default: 3<br>Assuming every worker does the same job, is there reall=
y the amount<br>of work for three workers?<br><br>&quot;get_maintenance_tas=
k()&quot;<br>This API seems to be too advanced, in what cases you see a san=
e use case for it?<br></div><div><br></div><div>Thank you,</div><div>Vladim=
ir</div><div><br></div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr=
" class=3D"gmail_attr">On Mon, 18 Jan 2021 at 04:04, Daniele Varrazzo &lt;<=
a href=3D"mailto:daniele.varrazzo@gmail.com">daniele.varrazzo@gmail.com</a>=
&gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px =
0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hell=
o,<br>
<br>
I&#39;ve been gathering a few ideas about the connection pool I would like<=
br>
to provide with psycopg3. I would be happy if you would like to take a<br>
look and give some feedback.<br>
<br>
<a href=3D"https://www.psycopg.org/articles/2021/01/17/pool-design/" rel=3D=
"noreferrer" target=3D"_blank">https://www.psycopg.org/articles/2021/01/17/=
pool-design/</a><br>
<br>
Thank you very much,<br>
<br>
-- Daniele<br>
<br>
<br>
</blockquote></div>

--000000000000841a5d05b930c60b--