Subject: Re: Latest developments in psycopg3
To: Daniele Varrazzo <daniele.varrazzo@gmail.com>
Cc: psycopg@postgresql.org
References: 
 <CA+mi_8btbC_-0KheQ3WYZW81fS4B0MyPyuALihmqHTvqq=6VyA@mail.gmail.com>
 <4eb4e86a-c813-bb8f-ae93-4f1b9fd46b6b@dalibo.com>
 <CA+mi_8YZnEuO=GAJiWnv_0Bx7_6DsL9jUcwhMxtXGqeSBgSp0A@mail.gmail.com>
From: Denis Laxalde <denis.laxalde@dalibo.com>
Message-ID: <f8b65955-50fb-a561-6cd0-7d98861103d1@dalibo.com>
Date: Mon, 8 Feb 2021 15:16:27 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.6.0
MIME-Version: 1.0
In-Reply-To: 
 <CA+mi_8YZnEuO=GAJiWnv_0Bx7_6DsL9jUcwhMxtXGqeSBgSp0A@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
Precedence: bulk

Daniele Varrazzo a écrit :
> On Mon, 8 Feb 2021 at 12:14, Denis Laxalde <denis.laxalde@dalibo.com> wrote:
>> - It seems to me that the conn-info developments work fine (also looked
>> at the commits, which look fine as well to me): what needs to be done
>> here? Any help needed?
> 
> I can't remember what is missing: probably the interface is complete,
> it is missing test coverage and documentation. I will review if there
> are methods missing: its purpose is to expose pretty much all the info
> about the connection that the libpq can give, correctly decoded from
> bytes. Complete tests and docs are definitely to be added.

Ok, good to know. (Let me know if help is needed.)

>> - There appears to be no card about dictcursor on the project board.
>> What's the plan (and priority) for that (if any)?
> 
> I forgot to add a point about them, sorry: I added one now. There is
> no firm plan for me, but probably a conversation to have, because I
> have my own usage patterns for psycopg2, when I am its user, and I
> want to be mindful of other use cases too.
> 
> In short: yes, I would like to provide alternative record factories.
> We can review if the best way is to create cursor subclasses
> out-of-the-box or mixins. In psycopg2 there are:
> 
> - DictCursor (returns a hybrid object between a sequence and a dictionary)
> - RealDictCursor (returns a straight subclass of a dictionary, so it's
> not a legit dbapi cursor as it doesn't expose a sequence)
> - NamedTupleCursor (what it says on the tin).
> 
> Personally I use the NamedTupeCursor most often, as I prefer the dot
> notation to access attribute. Named tuples also have a _dict() method
> when a dict is needed. If psycopg3 was only my pet project, I would
> personally have NamedTuples as the default and only thing supported...
> But that's hardly the case.

Personally, I use DictCursor because it provides access to rows without 
depending on SELECT order and then pass items to a custom dataclass as 
**kwargs.


> 1) We can provide a feature to select the type of cursor that is not
> much dissimilar from psycopg2
> 2) Do we need DictCursor/RealDictCursor? ISTM that one of the two
> would be sufficient? Possibly an object inheriting from a Python dict
> instead of emulating it so that e.g. json.dumps()

Inheriting from dict is strange because we only need read access. So 
rather collections.abc.Mapping, I think.

> 3) Or, conversely, we could have a Row object providing both attribute
> and getattr access, both as index and name:
> 
>     row.myattr
>     row["myattr"]
>     row[1]

+1 (this would also fit with collections.abc.Mapping)

> 
> 4) There are reports of the *DictCursor being slow: the objects in
> psycopg2 have hardly been profiled. I would look well at the
> performance of the objects created.
> 
> I am curious about why people use DictCursor: what are its advantages?
> Should we provide such objects only to guarantee a smoother upgrade
> path or there are use cases where they are genuinely better than
> NamedTuples or a would-be Row object (JSON does come to mind).

I seem to recall that namedtuple was not good for performances as either 
(see this article https://lwn.net/Articles/731423/ for instance). I tend 
to avoid using them and often write custom classes instead (which is 
very handy with py3.7's dataclass).

So, for me, no use case that wouldn't be covered by "would-be Row object".


>> I'm also curious about the "strictly typed" thing you wrote above. What
>> do you have in mind more specifically?
> 
> The codebase uses mypy --strict, so it is extensively type-annotated
> and of course all the public interfaces are type-annotated too, to
> collaborate with projects using static typing. I am also curious about
> whether it would be possible to provide types out of the query
> results, but I doubt it is.

That does not sounds feasible/easy. It's the price to pay where 
interfacing with external data sources (e.g. the same is true when one 
retrieves data from a JSON API) and type checking is probably only 
possible when "decoding" results.


> Which makes me think... 5) PydanticCursor? Or, as a more general case,
> implement all the cursor flavours not as subclasses but using a
> rowfactory function, and allow people to use their own (e.g. asking a
> cursor to create a Pydantic model provided by the user, so that they
> can have psycopg3 collaborate with e.g. FastAPI strictly typed
> models). Or a Python dataclass...

Providing a plugable API for that would be really nice, indeed.