Subject: Re: Integration of Psycopg with XTA

From: Christian Ferrari <[email-masked]>
Date: Tue, 2 Oct 2018 20:07:50 +0000 (UTC)

Hi Federico,I have been able to test _raw_pgconn using SWIG, as documented here https://sourceforge.net/p/swig/mailman/message/27919299/ , the trick to use PyCapsule and SWIG together is to specifically map the type with a directive like this one:
%typemap(in) PGconn * {
  $1 = (PGconn *) PyCapsule_GetPointer($input, "psycopg2.connection._raw_pgconn");
}

This is interesting because PyCapsule is for sure a "clean" way to pass the pointer from Psycopg to XTA.

A couple of questions to understand how to implement it as a re-usable deliverable.
In current branch, _raw_pgconn is a property, it's not a method, I have to retrieve it as below:
pgconn = rm1._raw_pgconn
xar1 = PostgresqlXaResource(pgconn, "PostgreSQL", "dbname=testdb")

what do you think about a "getter method":
pgconn = rm1._raw_pgconn()
the name of the method does not really matter to me.
_raw_pgconn() create a Capsule object that should be destroyed.What do you think about a factory to create and destroy a PyCapsule that contains the PGconn pointer?
Kind RegardsCh.F.

-------------------------------------------------------------
Good design can't stop bad implementation

    Il giovedì 27 settembre 2018, 22:11:35 CEST, Federico Di Gregorio <[email-masked]> ha scritto:  
 
 On 09/27/2018 09:57 PM, Christian Ferrari wrote:
> Thanks, for the fix, now the function works, but unfortunately it's a 
> dead end in the SWIG realm...Here's the statement: 
> http://www.swig.org/Doc2.0/SWIGDocumentation.html#Python_nn18
> [...]
> *However, the inverse operation is not possible, i.e., you can't build a 
> SWIG pointer object from a raw integer value.*
> [...]
> 
> PyCapsule https://docs.python.org/2/c-api/capsule.html, suggested by 
> Federico seems a more viable approach:
> https://bfroehle.com/2011/07/18/python-capsules/
> 
> If Psycopg was able to retrieve a PyCapsule, XTA should be able to use 
> it and the Python interpreter would be a gateway between two C/C++ 
> distinct modules (Psycopg and XTA).
> I'm going to perform some experiments in the XTA wrapper before coming 
> back to this mail thread.

If you want the capsule, just pull from my report:

https://github.com/fogzot/psycopg2/tree/feature-expose-pgconn

and use the feature-expose-pgconn branch.

federico

-- 
Federico Di Gregorio                        [email-masked]
DNDG srl                                                  http://dndg.it
  Everything will be OK at the end. If it's not OK, it's not the end.
                                                              -- Unknown

---

From: Federico Di Gregorio <[email-masked]>
Date: Wed, 3 Oct 2018 14:28:00 +0200

On 10/02/2018 10:07 PM, Christian Ferrari wrote:
> Hi Federico,
> I have been able to test _raw_pgconn using SWIG, as documented here 
> https://sourceforge.net/p/swig/mailman/message/27919299/ , the trick to 
> use PyCapsule and SWIG together is to specifically map the type with a 
> directive like this one:
> 
> %typemap(in) PGconn * {
> _raw_pgconn() create a Capsule object that should be destroyed.

Right now the getter creates a new capsule at every invocation so 
transforming it into a method should be straightforward. I'll do it 
later today so that you can test.

 > What do you think about a factory to create and destroy a PyCapsule that
 > contains the PGconn pointer?

Mm. I don't understand what you need here? Can you show an example of usage?

federico

-- 
Federico Di Gregorio                         [email-masked]
DNDG srl                                                  http://dndg.it
  Credo fermamente che da qualche parte, in una scatola ci sia un gatto
   che non è vivo ne morto. Credo anche che se i fisici non si sbrigano
   a dargli una scatoletta, ben presto sarà solo morto.
                               -- adattato da una frase di Sam Black Crow

---

From: Daniele Varrazzo <[email-masked]>
Date: Wed, 3 Oct 2018 13:37:56 +0100

> > the pointer from Psycopg to XTA.

Fog, maybe it's worth adding the capsules for PGconn and PGresult to
the objects interface and document them? Can do that for psycopg 2.8.

-- Daniele

---

From: Federico Di Gregorio <[email-masked]>
Date: Wed, 3 Oct 2018 14:44:39 +0200

On 10/03/2018 02:37 PM, Daniele Varrazzo wrote:
> On Wed, Oct 3, 2018 at 1:28 PM Federico Di Gregorio<[email-masked]>  wrote:
> 

If it works I'd add it (and document it as an "internal" extension point 
used by other C-level modules) for 2.8, yes.

By PGresult you mean adding it on the cursor, or the connection?

federico

-- 
Federico Di Gregorio                         [email-masked]
DNDG srl                                                  http://dndg.it
  Io non sono romantica. La candelina sul tavolo mi vede e si spegne.
                                                       -- sisterconfusion

---

From: Daniele Varrazzo <[email-masked]>
Date: Wed, 3 Oct 2018 13:52:43 +0100

> By PGresult you mean adding it on the cursor, or the connection?

Yes, i'd say a _pgconn attribute or _get_pgconn() method on the
connection, a _pgresult thing on the cursor.

By the way a read-only attribute as per your implementation seems
appropriate to me. Christian, why do you think a function would be
better?

-- Daniele

---

From: Federico Di Gregorio <[email-masked]>
Date: Wed, 3 Oct 2018 15:03:52 +0200

On 10/03/2018 02:52 PM, Daniele Varrazzo wrote:
> On Wed, Oct 3, 2018 at 1:44 PM Federico Di Gregorio<[email-masked]>  wrote:

I guess the capsule is meant to be consumed by the other API and using a 
method makes is explicit that a new object is instantiated every time. I 
really don't care about this, let's do whatever is more Pythonic.

federico

-- 
Federico Di Gregorio                         [email-masked]
DNDG srl                                                  http://dndg.it
  I did appreciate the irony that I was whining about encoding issues on
   a mailing list that was unable to show those chars, too.
                                  -- Antti S. Lankila to mono-devel-list@

---

From: Christian Ferrari <[email-masked]>
Date: Wed, 3 Oct 2018 20:50:08 +0000 (UTC)

> really don't care about this, let's do whatever is more Pythonic.

Dear Daniele & Federico,I'm not a "pythonist" at all, I can't provide suggestions about "Python style".In oo context, typically I expect to call a method when I need to retrieve something, that's why I proposed a method instead of a property.Anyway, my obscure question was related to garbage collection: who is in charge of destroying the capsule and how it must be destroyed?Looking at this example:
pgconn = xx._get_pgconn()do_something(pgconn)pgconn = yy._get_pgconn()
will the first capsule be automatically destroyed by the Python runtime when pgconn is assigned to another one (yy....)? Or some explicit statement must be executed?
If everything is automatically managed by the Python runtime, I will be very happy. If "do_something()" must release the capsule to avoid memory leaks, I have to implement it.

I will be very happy to test a "release candidate" version of the feature and link next LIXA/XTA release with psycopg 2.8.
K.R.Ch.F.

---

From: Federico Di Gregorio <[email-masked]>
Date: Thu, 4 Oct 2018 11:32:35 +0200

On 10/03/2018 10:50 PM, Christian Ferrari wrote:
[snip]
> Dear Daniele & Federico,
> I'm not a "pythonist" at all, I can't provide suggestions about "Python 
> style".
> In oo context, typically I expect to call a method when I need to 
> retrieve something, that's why I proposed a method instead of a property.
> Anyway, my obscure question was related to garbage collection: who is in 
> charge of destroying the capsule and how it must be destroyed?
> Looking at this example:
> 
> pgconn = xx._get_pgconn()
> do_something(pgconn)
> pgconn = yy._get_pgconn()
> 
> will the first capsule be automatically destroyed by the Python runtime 
> when pgconn is assigned to another one (yy....)? Or some explicit 
> statement must be executed?
> If everything is automatically managed by the Python runtime, I will be 
> very happy. If "do_something()" must release the capsule to avoid memory 
> leaks, I have to implement it.

The capsule is a standard reference-counted Python object. If the API 
just use the pointer inside it and does not keep a reference to the 
Python object itself, it will be free'd by the gc when the reference 
count goes to 0. In your example, the first capsule is garbage collected 
when you assign the second one to the same variable "pgconn".

@Daniele, I see that a capsule has an implicit state, "invalid" if the 
pointer inside it is null. Does it makes sense to reuse the same capsule 
and keep a reference to it to be able to invalidate it when the 
connection is closed?

federico

-- 
Federico Di Gregorio                         [email-masked]
DNDG srl                                                  http://dndg.it
  The only thing I see is if you are pumping so much data into the
   database all the time when do you expect to look at it?
                                                         -- Charlie Clark

---

From: Daniele Varrazzo <[email-masked]>
Date: Thu, 4 Oct 2018 11:13:43 +0100

> and keep a reference to it to be able to invalidate it when the

I can take a look at it but I suspect it's a lot of work for a feature
very rarely used.

-- Daniele

---

From: Federico Di Gregorio <[email-masked]>
Date: Thu, 4 Oct 2018 12:24:54 +0200

On 10/04/2018 12:13 PM, Daniele Varrazzo wrote:
> On Thu, Oct 4, 2018 at 10:33 AM Federico Di Gregorio<[email-masked]>  wrote:

Nah, I can do it. I was just asking for your opinion. :D

federico

-- 
Federico Di Gregorio                         [email-masked]
DNDG srl                                                  http://dndg.it
                              Best friends are often failed lovers. -- Me

---

From: Daniele Varrazzo <[email-masked]>
Date: Thu, 4 Oct 2018 11:32:38 +0100

> Nah, I can do it. I was just asking for your opinion. :D

I don't know much about the capsule object, its life cycle etc. Yes,
not a huge deal saving it in the connection state and returning always
the same. I just wonder if it's worth the effort.

-- Daniele