public inbox for [email protected]help / color / mirror / Atom feed
Re: psycopg2 conn.poll() hangs when used with sshtunnel 11+ messages / 2 participants [nested] [flat]
* Re: psycopg2 conn.poll() hangs when used with sshtunnel @ 2018-10-08 06:49 Akshay Joshi <[email protected]> 0 siblings, 1 reply; 11+ messages in thread From: Akshay Joshi @ 2018-10-08 06:49 UTC (permalink / raw) To: psycopg Hi Team As one issue(*https://github.com/psycopg/psycopg2/issues/781 <https://github.com/psycopg/psycopg2/issues/781>*) has already been created for conn.poll() function hangs when executes COPY command in async mode. Similarly conn.poll() functions hangs when we run wrong query in async mode using sshtunnel <https://pypi.org/project/sshtunnel/; v0.1.3. Can someone please suggest what went wrong when used sshtunnel, or the pointers/reason where poll() method hangs. On Fri, Sep 28, 2018 at 5:43 PM Akshay Joshi <[email protected]> wrote: > Hello > > I am using psycopg2 2.7.4 to connect to the PostgreSQL database server > using asynchronoussupport. It is working absolutely fine. I have used > sshtunnel <https://pypi.org/project/sshtunnel/; v0.1.3 in pgAdmin4. When > I connect the PostgreSQL database server using ssh tunnel and run the > valid/correct query it works fine, but when I run any wrong query(invalid > column of table) my application gets hang on conn.poll() function. > > Please refer the code how we use conn.poll() with timeouts > https://git.postgresql.org/gitweb/?p=pgadmin4.git;a=blob;f=web/pgadmin/utils/driver/psycopg2/connect... > > Can anyone please provide some pointers, suggestions? > > -- > *Akshay Joshi* > > *Sr. Software Architect * > > > > *Phone: +91 20-3058-9517Mobile: +91 976-788-8246* > -- *Akshay Joshi* *Sr. Software Architect * *Phone: +91 20-3058-9517Mobile: +91 976-788-8246* ^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: psycopg2 conn.poll() hangs when used with sshtunnel @ 2018-10-30 12:36 Akshay Joshi <[email protected]> parent: Akshay Joshi <[email protected]> 0 siblings, 1 reply; 11+ messages in thread From: Akshay Joshi @ 2018-10-30 12:36 UTC (permalink / raw) To: psycopg Hi Team I have created one sample application to replicate the issue. Attached is the sample application, please read "README.txt" file to proceed further. Can someone please suggest why poll() function hangs when run wrong query with sshtunnel v0.1.4. On Mon, Oct 8, 2018 at 12:19 PM Akshay Joshi <[email protected]> wrote: > Hi Team > > As one issue(*https://github.com/psycopg/psycopg2/issues/781 > <https://github.com/psycopg/psycopg2/issues/781>*) has already been > created for conn.poll() function hangs when executes COPY command in async > mode. Similarly conn.poll() functions hangs when we run wrong query in > async mode using sshtunnel <https://pypi.org/project/sshtunnel/; v0.1.3. > > Can someone please suggest what went wrong when used sshtunnel, or the > pointers/reason where poll() method hangs. > > On Fri, Sep 28, 2018 at 5:43 PM Akshay Joshi < > [email protected]> wrote: > >> Hello >> >> I am using psycopg2 2.7.4 to connect to the PostgreSQL database server >> using asynchronoussupport. It is working absolutely fine. I have used >> sshtunnel <https://pypi.org/project/sshtunnel/; v0.1.3 in pgAdmin4. When >> I connect the PostgreSQL database server using ssh tunnel and run the >> valid/correct query it works fine, but when I run any wrong query(invalid >> column of table) my application gets hang on conn.poll() function. >> >> Please refer the code how we use conn.poll() with timeouts >> https://git.postgresql.org/gitweb/?p=pgadmin4.git;a=blob;f=web/pgadmin/utils/driver/psycopg2/connect... >> >> Can anyone please provide some pointers, suggestions? >> >> -- >> *Akshay Joshi* >> >> *Sr. Software Architect * >> >> >> >> *Phone: +91 20-3058-9517Mobile: +91 976-788-8246* >> > > > -- > *Akshay Joshi* > > *Sr. Software Architect * > > > > *Phone: +91 20-3058-9517Mobile: +91 976-788-8246* > -- *Akshay Joshi* *Sr. Software Architect * *Phone: +91 20-3058-9517Mobile: +91 976-788-8246* Attachments: [application/zip] Hang_Sample_Application.zip (158.1K, ../../CANxoLDesBkV9=qpg22-9nG=JYgvUuo3Xm+iDxSWwLVReTiEC7g@mail.gmail.com/3-Hang_Sample_Application.zip) download ^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: psycopg2 conn.poll() hangs when used with sshtunnel @ 2018-10-30 12:52 Daniele Varrazzo <[email protected]> parent: Akshay Joshi <[email protected]> 0 siblings, 1 reply; 11+ messages in thread From: Daniele Varrazzo @ 2018-10-30 12:52 UTC (permalink / raw) To: Akshay Joshi <[email protected]>; +Cc: psycopg Taking a look. Thank you for the test. On Tue, Oct 30, 2018 at 12:36 PM Akshay Joshi <[email protected]> wrote: > Hi Team > > I have created one sample application to replicate the issue. Attached is > the sample application, please read "README.txt" file to proceed further. > Can someone please suggest why poll() function hangs when run wrong query > with sshtunnel v0.1.4. > > On Mon, Oct 8, 2018 at 12:19 PM Akshay Joshi < > [email protected]> wrote: > >> Hi Team >> >> As one issue(*https://github.com/psycopg/psycopg2/issues/781 >> <https://github.com/psycopg/psycopg2/issues/781>*) has already been >> created for conn.poll() function hangs when executes COPY command in async >> mode. Similarly conn.poll() functions hangs when we run wrong query in >> async mode using sshtunnel <https://pypi.org/project/sshtunnel/; v0.1.3. >> >> Can someone please suggest what went wrong when used sshtunnel, or the >> pointers/reason where poll() method hangs. >> >> On Fri, Sep 28, 2018 at 5:43 PM Akshay Joshi < >> [email protected]> wrote: >> >>> Hello >>> >>> I am using psycopg2 2.7.4 to connect to the PostgreSQL database server >>> using asynchronoussupport. It is working absolutely fine. I have used >>> sshtunnel <https://pypi.org/project/sshtunnel/; v0.1.3 in pgAdmin4. >>> When I connect the PostgreSQL database server using ssh tunnel and run the >>> valid/correct query it works fine, but when I run any wrong query(invalid >>> column of table) my application gets hang on conn.poll() function. >>> >>> Please refer the code how we use conn.poll() with timeouts >>> https://git.postgresql.org/gitweb/?p=pgadmin4.git;a=blob;f=web/pgadmin/utils/driver/psycopg2/connect... >>> >>> Can anyone please provide some pointers, suggestions? >>> >>> -- >>> *Akshay Joshi* >>> >>> *Sr. Software Architect * >>> >>> >>> >>> *Phone: +91 20-3058-9517Mobile: +91 976-788-8246* >>> >> >> >> -- >> *Akshay Joshi* >> >> *Sr. Software Architect * >> >> >> >> *Phone: +91 20-3058-9517Mobile: +91 976-788-8246* >> > > > -- > *Akshay Joshi* > > *Sr. Software Architect * > > > > *Phone: +91 20-3058-9517Mobile: +91 976-788-8246* > ^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: psycopg2 conn.poll() hangs when used with sshtunnel @ 2018-10-30 13:47 Daniele Varrazzo <[email protected]> parent: Daniele Varrazzo <[email protected]> 0 siblings, 1 reply; 11+ messages in thread From: Daniele Varrazzo @ 2018-10-30 13:47 UTC (permalink / raw) To: Akshay Joshi <[email protected]>; +Cc: psycopg It seems to me that the problem is in pqpath.c pq_get_last_result() function. The loop assumes PQgetResult won't block, which seems always the case with the direct connection, but PQisBusy() actually returns 1 connecting through the tunnel. So at a first read, the simplification given by pq_get_last_result() is broken: the results should be returned one by one in the normal loop going through PQisBusy(), PQconsumeInput(), and the polling machinery. It seems a chunky refactoring but as it is now I don't like pq_get_last_result() anymore :( -- Daniele On Tue, Oct 30, 2018 at 12:52 PM Daniele Varrazzo < [email protected]> wrote: > Taking a look. Thank you for the test. > > On Tue, Oct 30, 2018 at 12:36 PM Akshay Joshi < > [email protected]> wrote: > >> Hi Team >> >> I have created one sample application to replicate the issue. Attached is >> the sample application, please read "README.txt" file to proceed further. >> Can someone please suggest why poll() function hangs when run wrong query >> with sshtunnel v0.1.4. >> >> On Mon, Oct 8, 2018 at 12:19 PM Akshay Joshi < >> [email protected]> wrote: >> >>> Hi Team >>> >>> As one issue(*https://github.com/psycopg/psycopg2/issues/781 >>> <https://github.com/psycopg/psycopg2/issues/781>*) has already been >>> created for conn.poll() function hangs when executes COPY command in async >>> mode. Similarly conn.poll() functions hangs when we run wrong query in >>> async mode using sshtunnel <https://pypi.org/project/sshtunnel/; v0.1.3. >>> >>> Can someone please suggest what went wrong when used sshtunnel, or the >>> pointers/reason where poll() method hangs. >>> >>> On Fri, Sep 28, 2018 at 5:43 PM Akshay Joshi < >>> [email protected]> wrote: >>> >>>> Hello >>>> >>>> I am using psycopg2 2.7.4 to connect to the PostgreSQL database server >>>> using asynchronoussupport. It is working absolutely fine. I have used >>>> sshtunnel <https://pypi.org/project/sshtunnel/; v0.1.3 in pgAdmin4. >>>> When I connect the PostgreSQL database server using ssh tunnel and run the >>>> valid/correct query it works fine, but when I run any wrong query(invalid >>>> column of table) my application gets hang on conn.poll() function. >>>> >>>> Please refer the code how we use conn.poll() with timeouts >>>> https://git.postgresql.org/gitweb/?p=pgadmin4.git;a=blob;f=web/pgadmin/utils/driver/psycopg2/connect... >>>> >>>> Can anyone please provide some pointers, suggestions? >>>> >>>> -- >>>> *Akshay Joshi* >>>> >>>> *Sr. Software Architect * >>>> >>>> >>>> >>>> *Phone: +91 20-3058-9517Mobile: +91 976-788-8246* >>>> >>> >>> >>> -- >>> *Akshay Joshi* >>> >>> *Sr. Software Architect * >>> >>> >>> >>> *Phone: +91 20-3058-9517Mobile: +91 976-788-8246* >>> >> >> >> -- >> *Akshay Joshi* >> >> *Sr. Software Architect * >> >> >> >> *Phone: +91 20-3058-9517Mobile: +91 976-788-8246* >> > ^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: psycopg2 conn.poll() hangs when used with sshtunnel @ 2018-10-30 13:54 Akshay Joshi <[email protected]> parent: Daniele Varrazzo <[email protected]> 0 siblings, 2 replies; 11+ messages in thread From: Akshay Joshi @ 2018-10-30 13:54 UTC (permalink / raw) To: Daniele Varrazzo <[email protected]>; +Cc: psycopg Hi On Tue, 30 Oct 2018, 19:17 Daniele Varrazzo <[email protected] wrote: > It seems to me that the problem is in pqpath.c pq_get_last_result() > function. The loop assumes PQgetResult won't block, which seems always the > case with the direct connection, but PQisBusy() actually returns 1 > connecting through the tunnel. > > So at a first read, the simplification given by pq_get_last_result() is > broken: the results should be returned one by one in the normal loop going > through PQisBusy(), PQconsumeInput(), and the polling machinery. It seems a > chunky refactoring but as it is now I don't like pq_get_last_result() > anymore :( > What changes should I made to fix this? Or it should be fixed in psycopg2 itself. > > -- Daniele > > > On Tue, Oct 30, 2018 at 12:52 PM Daniele Varrazzo < > [email protected]> wrote: > >> Taking a look. Thank you for the test. >> >> On Tue, Oct 30, 2018 at 12:36 PM Akshay Joshi < >> [email protected]> wrote: >> >>> Hi Team >>> >>> I have created one sample application to replicate the issue. Attached >>> is the sample application, please read "README.txt" file to proceed further. >>> Can someone please suggest why poll() function hangs when run wrong >>> query with sshtunnel v0.1.4. >>> >>> On Mon, Oct 8, 2018 at 12:19 PM Akshay Joshi < >>> [email protected]> wrote: >>> >>>> Hi Team >>>> >>>> As one issue(*https://github.com/psycopg/psycopg2/issues/781 >>>> <https://github.com/psycopg/psycopg2/issues/781>*) has already been >>>> created for conn.poll() function hangs when executes COPY command in async >>>> mode. Similarly conn.poll() functions hangs when we run wrong query in >>>> async mode using sshtunnel <https://pypi.org/project/sshtunnel/; >>>> v0.1.3. >>>> >>>> Can someone please suggest what went wrong when used sshtunnel, or the >>>> pointers/reason where poll() method hangs. >>>> >>>> On Fri, Sep 28, 2018 at 5:43 PM Akshay Joshi < >>>> [email protected]> wrote: >>>> >>>>> Hello >>>>> >>>>> I am using psycopg2 2.7.4 to connect to the PostgreSQL database >>>>> server using asynchronoussupport. It is working absolutely fine. I >>>>> have used sshtunnel <https://pypi.org/project/sshtunnel/; v0.1.3 in >>>>> pgAdmin4. When I connect the PostgreSQL database server using ssh >>>>> tunnel and run the valid/correct query it works fine, but when I run any >>>>> wrong query(invalid column of table) my application gets hang on >>>>> conn.poll() function. >>>>> >>>>> Please refer the code how we use conn.poll() with timeouts >>>>> https://git.postgresql.org/gitweb/?p=pgadmin4.git;a=blob;f=web/pgadmin/utils/driver/psycopg2/connect... >>>>> >>>>> Can anyone please provide some pointers, suggestions? >>>>> >>>>> -- >>>>> *Akshay Joshi* >>>>> >>>>> *Sr. Software Architect * >>>>> >>>>> >>>>> >>>>> *Phone: +91 20-3058-9517Mobile: +91 976-788-8246* >>>>> >>>> >>>> >>>> -- >>>> *Akshay Joshi* >>>> >>>> *Sr. Software Architect * >>>> >>>> >>>> >>>> *Phone: +91 20-3058-9517Mobile: +91 976-788-8246* >>>> >>> >>> >>> -- >>> *Akshay Joshi* >>> >>> *Sr. Software Architect * >>> >>> >>> >>> *Phone: +91 20-3058-9517Mobile: +91 976-788-8246* >>> >> ^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: psycopg2 conn.poll() hangs when used with sshtunnel @ 2018-10-30 14:26 Daniele Varrazzo <[email protected]> parent: Akshay Joshi <[email protected]> 1 sibling, 1 reply; 11+ messages in thread From: Daniele Varrazzo @ 2018-10-30 14:26 UTC (permalink / raw) To: Akshay Joshi <[email protected]>; +Cc: psycopg On Tue, Oct 30, 2018 at 1:55 PM Akshay Joshi <[email protected]> wrote: > Hi > > On Tue, 30 Oct 2018, 19:17 Daniele Varrazzo <[email protected] > wrote: > >> It seems to me that the problem is in pqpath.c pq_get_last_result() >> function. The loop assumes PQgetResult won't block, which seems always the >> case with the direct connection, but PQisBusy() actually returns 1 >> connecting through the tunnel. >> >> So at a first read, the simplification given by pq_get_last_result() is >> broken: the results should be returned one by one in the normal loop going >> through PQisBusy(), PQconsumeInput(), and the polling machinery. It seems a >> chunky refactoring but as it is now I don't like pq_get_last_result() >> anymore :( >> > > What changes should I made to fix this? Or it should be fixed in > psycopg2 itself. > It is a problem in psycopg. I have a quick fix I can release with the next bugfix if it shows no regression ( https://github.com/psycopg/psycopg2/issues/801). A better fix requires a non trivial rewrite of the async and green paths ( https://github.com/psycopg/psycopg2/issues/802): I'd like to do it but it may happen later if the band aid holds. I was just about to prepare the 2.7.6 release, so if the quick fix is easy and doesn't cause regressions it will be released soon. -- Daniele ^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: psycopg2 conn.poll() hangs when used with sshtunnel @ 2018-10-30 14:50 Akshay Joshi <[email protected]> parent: Daniele Varrazzo <[email protected]> 0 siblings, 0 replies; 11+ messages in thread From: Akshay Joshi @ 2018-10-30 14:50 UTC (permalink / raw) To: Daniele Varrazzo <[email protected]>; +Cc: psycopg Thanks Daniele On Tue, 30 Oct 2018, 19:56 Daniele Varrazzo <[email protected] wrote: > On Tue, Oct 30, 2018 at 1:55 PM Akshay Joshi < > [email protected]> wrote: > >> Hi >> >> On Tue, 30 Oct 2018, 19:17 Daniele Varrazzo <[email protected] >> wrote: >> >>> It seems to me that the problem is in pqpath.c pq_get_last_result() >>> function. The loop assumes PQgetResult won't block, which seems always the >>> case with the direct connection, but PQisBusy() actually returns 1 >>> connecting through the tunnel. >>> >>> So at a first read, the simplification given by pq_get_last_result() is >>> broken: the results should be returned one by one in the normal loop going >>> through PQisBusy(), PQconsumeInput(), and the polling machinery. It seems a >>> chunky refactoring but as it is now I don't like pq_get_last_result() >>> anymore :( >>> >> >> What changes should I made to fix this? Or it should be fixed in >> psycopg2 itself. >> > > It is a problem in psycopg. I have a quick fix I can release with the next > bugfix if it shows no regression ( > https://github.com/psycopg/psycopg2/issues/801). A better fix requires a > non trivial rewrite of the async and green paths ( > https://github.com/psycopg/psycopg2/issues/802): I'd like to do it but it > may happen later if the band aid holds. > > I was just about to prepare the 2.7.6 release, so if the quick fix is easy > and doesn't cause regressions it will be released soon. > > -- Daniele > > ^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: psycopg2 conn.poll() hangs when used with sshtunnel @ 2018-10-30 15:36 Daniele Varrazzo <[email protected]> parent: Akshay Joshi <[email protected]> 1 sibling, 1 reply; 11+ messages in thread From: Daniele Varrazzo @ 2018-10-30 15:36 UTC (permalink / raw) To: Akshay Joshi <[email protected]>; +Cc: psycopg On Tue, Oct 30, 2018 at 1:55 PM Akshay Joshi <[email protected]> wrote: > What changes should I made to fix this? Or it should be fixed in > psycopg2 itself. > Actually, I have another observation. I was surprised to see that the whole test suite passed running the connection through an ssh tunnel: there are definitely failing queries in the test suite. So I made other tests and it seems the bad condition only happens using the `sshtunnel` module: If I open a tunnel manually with something like: ssh -L 36421:localhost:5432 -N localhost and point your script to port 36421 everything works ok. Conversely, I don't seem able to run the test suite through the tunnel open by the sshtunnel module. It doesn't hang, but the tunnel breaks in a test with the following reported on the sshtunnel side: 2018-10-30 15:24:25,550| ERROR | Socket exception: Bad file descriptor (9) 2018-10-30 15:24:25,551| ERROR | Could not establish connection from ('127.0.0.1', 33743) to remote side of the tunnel and the following tests fail to run as the connection is broken. Con-conversely, the patch I had in mind to fix #801, which very brutally is: ``` @@ -1136,6 +1136,13 @@ pq_get_last_result(connectionObject *conn) || status == PGRES_COPY_IN) { break; } + if (PQisBusy(conn->pgconn)) { + /* This happens connecting through ssl tunnel + * TODO: just kill this function. The loop should happen within + * the async/green machinery. */ + Dprintf("pq_get_last_result: we are busy"); + break; + } } return result; ``` doesn't work, and failing queries leave the connection in an inconsistent state. So, wrapping up, I think there is something you can do on your side, checking why the sshtunnel module behaves differently from a normal ssh tunnel, and if you use the latter (or configure sshtunnel to behave so) you shouldn't hit the problem. On our side I don't think we can fix #801 with a quick band aid, and we should rather do #802, but as things stands now I don't trust sshtunnel to do the right thing, and have less of an urgency to do so. Please let us know if you understand what is the difference between the module and the tunnel via `ssh -L`, thank you! -- Daniele > ^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: psycopg2 conn.poll() hangs when used with sshtunnel @ 2018-10-30 15:52 Akshay Joshi <[email protected]> parent: Daniele Varrazzo <[email protected]> 0 siblings, 1 reply; 11+ messages in thread From: Akshay Joshi @ 2018-10-30 15:52 UTC (permalink / raw) To: Daniele Varrazzo <[email protected]>; +Cc: psycopg On Tue, 30 Oct 2018, 21:06 Daniele Varrazzo <[email protected] wrote: > On Tue, Oct 30, 2018 at 1:55 PM Akshay Joshi < > [email protected]> wrote: > > >> What changes should I made to fix this? Or it should be fixed in >> psycopg2 itself. >> > > Actually, I have another observation. I was surprised to see that the > whole test suite passed running the connection through an ssh tunnel: there > are definitely failing queries in the test suite. So I made other tests and > it seems the bad condition only happens using the `sshtunnel` module: If I > open a tunnel manually with something like: > > ssh -L 36421:localhost:5432 -N localhost > > and point your script to port 36421 everything works ok. > > > Conversely, I don't seem able to run the test suite through the tunnel > open by the sshtunnel module. It doesn't hang, but the tunnel breaks in a > test with the following reported on the sshtunnel side: > > 2018-10-30 15:24:25,550| ERROR | Socket exception: Bad file > descriptor (9) > 2018-10-30 15:24:25,551| ERROR | Could not establish connection from > ('127.0.0.1', 33743) to remote side of the tunnel > > and the following tests fail to run as the connection is broken. > > > Con-conversely, the patch I had in mind to fix #801, which very brutally > is: > > ``` > @@ -1136,6 +1136,13 @@ pq_get_last_result(connectionObject *conn) > || status == PGRES_COPY_IN) { > break; > } > + if (PQisBusy(conn->pgconn)) { > + /* This happens connecting through ssl tunnel > + * TODO: just kill this function. The loop should happen > within > + * the async/green machinery. */ > + Dprintf("pq_get_last_result: we are busy"); > + break; > + } > } > > return result; > ``` > > doesn't work, and failing queries leave the connection in an inconsistent > state. > > > So, wrapping up, I think there is something you can do on your side, > checking why the sshtunnel module behaves differently from a normal ssh > tunnel, and if you use the latter (or configure sshtunnel to behave so) you > shouldn't hit the problem. On our side I don't think we can fix #801 with a > quick band aid, and we should rather do #802, but as things stands now I > don't trust sshtunnel to do the right thing, and have less of an urgency to > do so. > > Please let us know if you understand what is the difference between the > module and the tunnel via `ssh -L`, thank you! > I have send the sample application and created an issue to sshtunnel git hub. Not sure where is the problem, as poll() function hangs so i thought its issue in psycopg2. > > -- Daniele > >> ^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: psycopg2 conn.poll() hangs when used with sshtunnel @ 2018-10-30 15:57 Akshay Joshi <[email protected]> parent: Akshay Joshi <[email protected]> 0 siblings, 1 reply; 11+ messages in thread From: Akshay Joshi @ 2018-10-30 15:57 UTC (permalink / raw) To: Daniele Varrazzo <[email protected]>; +Cc: psycopg I have one more observation when i run this application one out of 10 times it works. I have logged one more issue as COPY command hangs, so thought it is the problem in psycopg2 On Tue, 30 Oct 2018, 21:22 Akshay Joshi <[email protected] wrote: > > > On Tue, 30 Oct 2018, 21:06 Daniele Varrazzo <[email protected] > wrote: > >> On Tue, Oct 30, 2018 at 1:55 PM Akshay Joshi < >> [email protected]> wrote: >> >> >>> What changes should I made to fix this? Or it should be fixed in >>> psycopg2 itself. >>> >> >> Actually, I have another observation. I was surprised to see that the >> whole test suite passed running the connection through an ssh tunnel: there >> are definitely failing queries in the test suite. So I made other tests and >> it seems the bad condition only happens using the `sshtunnel` module: If I >> open a tunnel manually with something like: >> >> ssh -L 36421:localhost:5432 -N localhost >> >> and point your script to port 36421 everything works ok. >> >> >> Conversely, I don't seem able to run the test suite through the tunnel >> open by the sshtunnel module. It doesn't hang, but the tunnel breaks in a >> test with the following reported on the sshtunnel side: >> >> 2018-10-30 15:24:25,550| ERROR | Socket exception: Bad file >> descriptor (9) >> 2018-10-30 15:24:25,551| ERROR | Could not establish connection >> from ('127.0.0.1', 33743) to remote side of the tunnel >> >> and the following tests fail to run as the connection is broken. >> >> >> Con-conversely, the patch I had in mind to fix #801, which very brutally >> is: >> >> ``` >> @@ -1136,6 +1136,13 @@ pq_get_last_result(connectionObject *conn) >> || status == PGRES_COPY_IN) { >> break; >> } >> + if (PQisBusy(conn->pgconn)) { >> + /* This happens connecting through ssl tunnel >> + * TODO: just kill this function. The loop should happen >> within >> + * the async/green machinery. */ >> + Dprintf("pq_get_last_result: we are busy"); >> + break; >> + } >> } >> >> return result; >> ``` >> >> doesn't work, and failing queries leave the connection in an inconsistent >> state. >> >> >> So, wrapping up, I think there is something you can do on your side, >> checking why the sshtunnel module behaves differently from a normal ssh >> tunnel, and if you use the latter (or configure sshtunnel to behave so) you >> shouldn't hit the problem. On our side I don't think we can fix #801 with a >> quick band aid, and we should rather do #802, but as things stands now I >> don't trust sshtunnel to do the right thing, and have less of an urgency to >> do so. >> >> Please let us know if you understand what is the difference between the >> module and the tunnel via `ssh -L`, thank you! >> > > I have send the sample application and created an issue to sshtunnel > git hub. Not sure where is the problem, as poll() function hangs so i > thought its issue in psycopg2. > >> >> -- Daniele >> >>> ^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: psycopg2 conn.poll() hangs when used with sshtunnel @ 2018-10-30 16:07 Daniele Varrazzo <[email protected]> parent: Akshay Joshi <[email protected]> 0 siblings, 0 replies; 11+ messages in thread From: Daniele Varrazzo @ 2018-10-30 16:07 UTC (permalink / raw) To: Akshay Joshi <[email protected]>; +Cc: psycopg On Tue, Oct 30, 2018 at 3:57 PM Akshay Joshi <[email protected]> wrote: > > I have one more observation when i run this application one out of 10 times it works. I have logged one more issue as COPY command hangs, so thought it is the problem in psycopg2 The problem with COPY was a different one: we just didn't manage the state resulting by running COPY through execute, and ended up looping forever on fake results returned by the libpq. The condition was detected and handled for COPY BOTH (resulting by replication statements) but not for the normal COPY TO/FROM. See <https://github.com/psycopg/psycopg2/issues/781;. This is a different story but, as described in <https://github.com/pahaz/sshtunnel/issues/135; it seems that even if we are doing something not entirely legit with isBusy/consumeInput/getResult, it only becomes a problem through the sshtunnel module, not through ssh -L. -- Daniele ^ permalink raw reply [nested|flat] 11+ messages in thread
end of thread, other threads:[~2018-10-30 16:07 UTC | newest] Thread overview: 11+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2018-10-08 06:49 Re: psycopg2 conn.poll() hangs when used with sshtunnel Akshay Joshi <[email protected]> 2018-10-30 12:36 ` Akshay Joshi <[email protected]> 2018-10-30 12:52 ` Daniele Varrazzo <[email protected]> 2018-10-30 13:47 ` Daniele Varrazzo <[email protected]> 2018-10-30 13:54 ` Akshay Joshi <[email protected]> 2018-10-30 14:26 ` Daniele Varrazzo <[email protected]> 2018-10-30 14:50 ` Akshay Joshi <[email protected]> 2018-10-30 15:36 ` Daniele Varrazzo <[email protected]> 2018-10-30 15:52 ` Akshay Joshi <[email protected]> 2018-10-30 15:57 ` Akshay Joshi <[email protected]> 2018-10-30 16:07 ` Daniele Varrazzo <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox