MIME-Version: 1.0
References: 
 <CA+mi_8aAuORkdXZ9bG_GpU34iPCGktMO83ktn+ODjL+Y=Y=e3Q@mail.gmail.com>
 <25a137a0-9ad3-bb9f-b008-263dcc81a645@aklaver.com>
 <CA+mi_8YoZ6_0NM4TbfqE8SQY8-b=RgaBH-jUVxzbWQdp+4oZGQ@mail.gmail.com>
 <331f273f-fc13-8d80-b209-c94c0960268b@aklaver.com>
 <CAMqTPq=G1E=b32JxNo6dGk1-9qGHSCP7Puo5FCqf68nakiBo_Q@mail.gmail.com>
In-Reply-To: 
 <CAMqTPq=G1E=b32JxNo6dGk1-9qGHSCP7Puo5FCqf68nakiBo_Q@mail.gmail.com>
From: Daniele Varrazzo <daniele.varrazzo@gmail.com>
Date: Tue, 22 Dec 2020 22:36:48 +0000
Message-ID: 
 <CA+mi_8YhQVo8xouWnc6hCR6RvE1THRB0Wz0OzHhMHsuvdLNYnw@mail.gmail.com>
Subject: Re: psycopg3, prepared statements
To: Vladimir Ryabtsev <greatvovan@gmail.com>
Cc: Adrian Klaver <adrian.klaver@aklaver.com>, psycopg@postgresql.org
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Tue, 22 Dec 2020 at 05:39, Vladimir Ryabtsev <greatvovan@gmail.com> wrote:
>
> > it's a moment in your
> > program's lifetime where you know the query to run and the arguments:
> > most likely now you want to run the query too, not only to prepare it.
>
> I am just thinking about type issues in some edge cases...
> What if the data is such that in the first row it has some small integers,
> but on subsequent rows it has larger numbers that do not fit into "integer"?
> There are probably other cases similar to this, including "None" values
> on the moment you want to capture types... Maybe it is more reliable to
> oblige the user, who knows their data better, to explicitly supply the types
> for preparation... Will any type inference work in case of types mismatch?
> If yes, to what extent?

The only case I have thought where this can happen is with None vs.
not None, in which case the oid info would be missing. Oid selection
is based only on the Python type, not on the specific values (for
instance Python datetime is always passed as timestamptz, never
timestamp, even if the tezinfo is missing, and it relies on Postgres
cast): I have avoided so far to make a choice based on the values
exactly to avoid to find ourselves in this type of situations.

I don't have in mind a case where a missing value would make a
difference but that doesn't mean that they don't exist. What I think
is that if in a certain context a NULL is acceptable and Python can
pass either a None or e.g. a date, preparing witn unknown or date oid
shouldn't make a difference. Of course it is possible to trigger
ambiguities: trivially the query `SELECT %s` could be prepared with a
date and then you can throw it a string that will make it fail. But if
a placeholder is part of an expression or target for an insert I
believe (but will test too) that preparing with unknown types will be
equivalent.

I am interested in real cases in which a prepared statement is
definitely undesirable. My current idea is to make the feature
automatic but with the possibility to disable it at global,
connection, statement level. If there is evidence that the feature is
dangerous we can leave it disabled by default and leave the
possibility to enable it in the same contexts.

-- Daniele


>
> Generally I think that automatic preparation is a great idea, but if it is achieved
> by sacrificing reliability to any extent, I would prefer to have such a feature
> switched off by default.
>
> > - if more than `connection.prepared_number` queries are prepared, the
> > one used least recently is deallocated and evicted from the cache
> > (proposed default: 100).
>
> Why do you need such logic? Why not just keep some limited number of
> prepared statements? Is it a problem if a PS is in cache but rarely used?
>
> --
> Vlad
>
>
> On Mon, 21 Dec 2020 at 13:55, Adrian Klaver <adrian.klaver@aklaver.com> wrote:
>>
>> On 12/21/20 12:26 PM, Daniele Varrazzo wrote:
>> > On Mon, 21 Dec 2020 at 16:02, Adrian Klaver <adrian.klaver@aklaver.com> wrote:
>> >
>>
>> >
>> > I mistakenly replied Adrian privately. Following, his reply.
>> >
>>
>> > I think so. Personally, I prefer the two step approach as I am becoming
>> > less and less enthusiastic about hidden 'magic'. To that end a
>> > global(maybe connection) setting that would disable prepare would be nice.
>> >
>> > ----
>> >
>> > To which, 1) thank you very much, Adrian, for the plpython pointer,
>> > I'll take a look at it.
>> >
>> > 2) About disabling the automatic prepare: the mechanism I have in mind
>> > is to set prepare_threshold to None on the connection instance;
>>
>> So the above is something the user would have to do on each connection?
>>
>> > however we could make sure to have the default attribute defined on
>> > the class: this way who really hates the idea of prepared statements
>> > can be cheeky and set `psycopg3.Connection.prepare_threshold = None`
>>
>> I could get behind that. This all may be premature optimization on my
>> part. As long as there is some way to turn it off at some level I could
>> live with it.
>>
>> > instead of `myconn.prepare_threshold`... More seriously, if there is a
>> > large base of people who think that something can go wrong with
>> > prepared statement we can either provide a better interface to control
>> > it globally or to have the feature opt-in.
>> >
>> >
>> > -- Daniele
>> >
>>
>>
>> --
>> Adrian Klaver
>> adrian.klaver@aklaver.com
>>
>>