public inbox for [email protected]
help / color / mirror / Atom feedpgAdmin 4 commit: Ensure we pick up the messages from the current query
17+ messages / 4 participants
[nested] [flat]
* pgAdmin 4 commit: Ensure we pick up the messages from the current query
@ 2018-02-26 14:19 Dave Page <[email protected]>
2018-02-26 16:18 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
0 siblings, 1 reply; 17+ messages in thread
From: Dave Page @ 2018-02-26 14:19 UTC (permalink / raw)
To: pgadmin-hackers
Ensure we pick up the messages from the current query and not a previous one. Fixes #3094
Branch
------
master
Details
-------
https://git.postgresql.org/gitweb?p=pgadmin4.git;a=commitdiff;h=08b3ccc01a4d57e8ea3657f8882a53dcd1b9...
Author: Khushboo Vashi <[email protected]>
Modified Files
--------------
web/pgadmin/utils/driver/abstract.py | 1 +
web/pgadmin/utils/driver/psycopg2/__init__.py | 64 +++++++++------------------
2 files changed, 21 insertions(+), 44 deletions(-)
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query
2018-02-26 14:19 pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
@ 2018-02-26 16:18 ` Murtuza Zabuawala <[email protected]>
2018-02-26 16:23 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
0 siblings, 1 reply; 17+ messages in thread
From: Murtuza Zabuawala @ 2018-02-26 16:18 UTC (permalink / raw)
To: Dave Page <[email protected]>; Khushboo Vashi <[email protected]>; +Cc: pgadmin-hackers
Hi Khushboo/Dave,
With given commit, I'm again seeing the issue raised in
https://redmine.postgresql.org/issues/1523 :(
--
Regards,
Murtuza Zabuawala
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Mon, Feb 26, 2018 at 7:49 PM, Dave Page <[email protected]> wrote:
> Ensure we pick up the messages from the current query and not a previous
> one. Fixes #3094
>
> Branch
> ------
> master
>
> Details
> -------
> https://git.postgresql.org/gitweb?p=pgadmin4.git;a=commitdiff;h=
> 08b3ccc01a4d57e8ea3657f8882a53dcd1b99386
> Author: Khushboo Vashi <[email protected]>
>
> Modified Files
> --------------
> web/pgadmin/utils/driver/abstract.py | 1 +
> web/pgadmin/utils/driver/psycopg2/__init__.py | 64
> +++++++++------------------
> 2 files changed, 21 insertions(+), 44 deletions(-)
>
>
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query
2018-02-26 14:19 pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-02-26 16:18 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
@ 2018-02-26 16:23 ` Murtuza Zabuawala <[email protected]>
2018-02-26 16:32 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
0 siblings, 1 reply; 17+ messages in thread
From: Murtuza Zabuawala @ 2018-02-26 16:23 UTC (permalink / raw)
To: Dave Page <[email protected]>; Khushboo Vashi <[email protected]>; +Cc: pgadmin-hackers
Sent bit early,
You can run 'VACUUM FULL VERBOSE' in query tool and verify the populated
messages (pgAdmin3 vs. pgAdmin4).
On Mon, Feb 26, 2018 at 9:48 PM, Murtuza Zabuawala <
[email protected]> wrote:
> Hi Khushboo/Dave,
>
> With given commit, I'm again seeing the issue raised in
> https://redmine.postgresql.org/issues/1523 :(
>
>
>
>
> --
> Regards,
> Murtuza Zabuawala
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
>
> On Mon, Feb 26, 2018 at 7:49 PM, Dave Page <[email protected]> wrote:
>
>> Ensure we pick up the messages from the current query and not a previous
>> one. Fixes #3094
>>
>> Branch
>> ------
>> master
>>
>> Details
>> -------
>> https://git.postgresql.org/gitweb?p=pgadmin4.git;a=commitdif
>> f;h=08b3ccc01a4d57e8ea3657f8882a53dcd1b99386
>> Author: Khushboo Vashi <[email protected]>
>>
>> Modified Files
>> --------------
>> web/pgadmin/utils/driver/abstract.py | 1 +
>> web/pgadmin/utils/driver/psycopg2/__init__.py | 64
>> +++++++++------------------
>> 2 files changed, 21 insertions(+), 44 deletions(-)
>>
>>
>
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query
2018-02-26 14:19 pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-02-26 16:18 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
2018-02-26 16:23 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
@ 2018-02-26 16:32 ` Dave Page <[email protected]>
2018-02-26 17:34 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
2018-02-28 11:49 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
0 siblings, 2 replies; 17+ messages in thread
From: Dave Page @ 2018-02-26 16:32 UTC (permalink / raw)
To: Murtuza Zabuawala <[email protected]>; +Cc: Khushboo Vashi <[email protected]>; pgadmin-hackers
Argh, I ran some tests, but didn't spot any lost messages in the tests I
ran. I'll revert the patch.
Khushboo;
Please look at the following:
- Fix the patch so it doesn't drop messages.
- Add regression tests to make sure it doesn't break in the future. This
may require creating one or more functions the spew out a whole lot of
notices, and then running a couple of queries and checking the output.
- Check the messages panel on the history tab. I just noticed it seems to
only be showing an even smaller subset of the messages.
Thanks.
On Mon, Feb 26, 2018 at 4:23 PM, Murtuza Zabuawala <
[email protected]> wrote:
> Sent bit early,
>
> You can run 'VACUUM FULL VERBOSE' in query tool and verify the populated
> messages (pgAdmin3 vs. pgAdmin4).
>
>
> On Mon, Feb 26, 2018 at 9:48 PM, Murtuza Zabuawala <murtuza.zabuawala@
> enterprisedb.com> wrote:
>
>> Hi Khushboo/Dave,
>>
>> With given commit, I'm again seeing the issue raised in
>> https://redmine.postgresql.org/issues/1523 :(
>>
>>
>>
>>
>> --
>> Regards,
>> Murtuza Zabuawala
>> EnterpriseDB: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>>
>>
>> On Mon, Feb 26, 2018 at 7:49 PM, Dave Page <[email protected]> wrote:
>>
>>> Ensure we pick up the messages from the current query and not a previous
>>> one. Fixes #3094
>>>
>>> Branch
>>> ------
>>> master
>>>
>>> Details
>>> -------
>>> https://git.postgresql.org/gitweb?p=pgadmin4.git;a=commitdif
>>> f;h=08b3ccc01a4d57e8ea3657f8882a53dcd1b99386
>>> Author: Khushboo Vashi <[email protected]>
>>>
>>> Modified Files
>>> --------------
>>> web/pgadmin/utils/driver/abstract.py | 1 +
>>> web/pgadmin/utils/driver/psycopg2/__init__.py | 64
>>> +++++++++------------------
>>> 2 files changed, 21 insertions(+), 44 deletions(-)
>>>
>>>
>>
>
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query
2018-02-26 14:19 pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-02-26 16:18 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
2018-02-26 16:23 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
2018-02-26 16:32 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
@ 2018-02-26 17:34 ` Murtuza Zabuawala <[email protected]>
1 sibling, 0 replies; 17+ messages in thread
From: Murtuza Zabuawala @ 2018-02-26 17:34 UTC (permalink / raw)
To: Dave Page <[email protected]>; +Cc: Khushboo Vashi <[email protected]>; pgadmin-hackers
FYI, This is what I'm receiving(attachments) when I'm running vacuum full
verbose on my PG10.
On Mon, Feb 26, 2018 at 10:02 PM, Dave Page <[email protected]> wrote:
> Argh, I ran some tests, but didn't spot any lost messages in the tests I
> ran. I'll revert the patch.
>
> Khushboo;
>
> Please look at the following:
>
> - Fix the patch so it doesn't drop messages.
> - Add regression tests to make sure it doesn't break in the future. This
> may require creating one or more functions the spew out a whole lot of
> notices, and then running a couple of queries and checking the output.
> - Check the messages panel on the history tab. I just noticed it seems to
> only be showing an even smaller subset of the messages.
>
> Thanks.
>
> On Mon, Feb 26, 2018 at 4:23 PM, Murtuza Zabuawala <murtuza.zabuawala@
> enterprisedb.com> wrote:
>
>> Sent bit early,
>>
>> You can run 'VACUUM FULL VERBOSE' in query tool and verify the populated
>> messages (pgAdmin3 vs. pgAdmin4).
>>
>>
>> On Mon, Feb 26, 2018 at 9:48 PM, Murtuza Zabuawala <
>> [email protected]> wrote:
>>
>>> Hi Khushboo/Dave,
>>>
>>> With given commit, I'm again seeing the issue raised in
>>> https://redmine.postgresql.org/issues/1523 :(
>>>
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Murtuza Zabuawala
>>> EnterpriseDB: http://www.enterprisedb.com
>>> The Enterprise PostgreSQL Company
>>>
>>>
>>> On Mon, Feb 26, 2018 at 7:49 PM, Dave Page <[email protected]> wrote:
>>>
>>>> Ensure we pick up the messages from the current query and not a
>>>> previous one. Fixes #3094
>>>>
>>>> Branch
>>>> ------
>>>> master
>>>>
>>>> Details
>>>> -------
>>>> https://git.postgresql.org/gitweb?p=pgadmin4.git;a=commitdif
>>>> f;h=08b3ccc01a4d57e8ea3657f8882a53dcd1b99386
>>>> Author: Khushboo Vashi <[email protected]>
>>>>
>>>> Modified Files
>>>> --------------
>>>> web/pgadmin/utils/driver/abstract.py | 1 +
>>>> web/pgadmin/utils/driver/psycopg2/__init__.py | 64
>>>> +++++++++------------------
>>>> 2 files changed, 21 insertions(+), 44 deletions(-)
>>>>
>>>>
>>>
>>
>
>
> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
INFO: vacuuming "public.mytable"
INFO: "mytable": found 0 removable, 1000001 nonremovable row versions in 4425 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.17 s, system: 0.17 s, elapsed: 0.41 s.
INFO: vacuuming "pg_catalog.pg_statistic"
INFO: "pg_statistic": found 48 removable, 394 nonremovable row versions in 18 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_type"
INFO: "pg_type": found 5 removable, 379 nonremovable row versions in 9 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_policy"
INFO: "pg_policy": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_authid"
INFO: "pg_authid": found 0 removable, 6 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_user_mapping"
INFO: "pg_user_mapping": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_subscription"
INFO: "pg_subscription": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_attribute"
INFO: "pg_attribute": found 22 removable, 2614 nonremovable row versions in 51 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_proc"
INFO: "pg_proc": found 0 removable, 2902 nonremovable row versions in 74 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_attrdef"
INFO: "pg_attrdef": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_constraint"
INFO: "pg_constraint": found 0 removable, 2 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_inherits"
INFO: "pg_inherits": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_index"
INFO: "pg_index": found 17 removable, 134 nonremovable row versions in 4 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_operator"
INFO: "pg_operator": found 0 removable, 787 nonremovable row versions in 15 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_opfamily"
INFO: "pg_opfamily": found 0 removable, 115 nonremovable row versions in 2 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_opclass"
INFO: "pg_opclass": found 0 removable, 133 nonremovable row versions in 3 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_am"
INFO: "pg_am": found 0 removable, 6 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_amop"
INFO: "pg_amop": found 0 removable, 709 nonremovable row versions in 6 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_amproc"
INFO: "pg_amproc": found 0 removable, 411 nonremovable row versions in 4 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_language"
INFO: "pg_language": found 0 removable, 4 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_largeobject_metadata"
INFO: "pg_largeobject_metadata": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_aggregate"
INFO: "pg_aggregate": found 0 removable, 138 nonremovable row versions in 2 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_statistic_ext"
INFO: "pg_statistic_ext": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_rewrite"
INFO: "pg_rewrite": found 0 removable, 121 nonremovable row versions in 12 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_trigger"
INFO: "pg_trigger": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_description"
INFO: "pg_description": found 0 removable, 4325 nonremovable row versions in 37 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_cast"
INFO: "pg_cast": found 0 removable, 219 nonremovable row versions in 2 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_enum"
INFO: "pg_enum": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_namespace"
INFO: "pg_namespace": found 0 removable, 6 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_conversion"
INFO: "pg_conversion": found 0 removable, 132 nonremovable row versions in 3 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_depend"
INFO: "pg_depend": found 28 removable, 7342 nonremovable row versions in 55 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_database"
INFO: "pg_database": found 0 removable, 3 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_tablespace"
INFO: "pg_tablespace": found 0 removable, 2 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_pltemplate"
INFO: "pg_pltemplate": found 0 removable, 8 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_auth_members"
INFO: "pg_auth_members": found 0 removable, 3 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_shdepend"
INFO: "pg_shdepend": found 0 removable, 6 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_shdescription"
INFO: "pg_shdescription": found 0 removable, 3 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_ts_config"
INFO: "pg_ts_config": found 0 removable, 16 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_ts_config_map"
INFO: "pg_ts_config_map": found 0 removable, 304 nonremovable row versions in 2 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_ts_dict"
INFO: "pg_ts_dict": found 0 removable, 16 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_ts_parser"
INFO: "pg_ts_parser": found 0 removable, 1 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_ts_template"
INFO: "pg_ts_template": found 0 removable, 5 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_extension"
INFO: "pg_extension": found 0 removable, 2 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_foreign_data_wrapper"
INFO: "pg_foreign_data_wrapper": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_foreign_server"
INFO: "pg_foreign_server": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_foreign_table"
INFO: "pg_foreign_table": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_replication_origin"
INFO: "pg_replication_origin": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_default_acl"
INFO: "pg_default_acl": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_init_privs"
INFO: "pg_init_privs": found 0 removable, 149 nonremovable row versions in 2 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_seclabel"
INFO: "pg_seclabel": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_shseclabel"
INFO: "pg_shseclabel": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_collation"
INFO: "pg_collation": found 0 removable, 415 nonremovable row versions in 14 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_partitioned_table"
INFO: "pg_partitioned_table": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_range"
INFO: "pg_range": found 0 removable, 6 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_transform"
INFO: "pg_transform": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_sequence"
INFO: "pg_sequence": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_publication"
INFO: "pg_publication": found 0 removable, 1 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_publication_rel"
INFO: "pg_publication_rel": found 0 removable, 1 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_subscription_rel"
INFO: "pg_subscription_rel": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "information_schema.sql_parts"
INFO: "sql_parts": found 0 removable, 9 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "information_schema.sql_languages"
INFO: "sql_languages": found 0 removable, 4 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "information_schema.sql_features"
INFO: "sql_features": found 0 removable, 671 nonremovable row versions in 7 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "information_schema.sql_implementation_info"
INFO: "sql_implementation_info": found 0 removable, 12 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "information_schema.sql_packages"
INFO: "sql_packages": found 0 removable, 10 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "information_schema.sql_sizing"
INFO: "sql_sizing": found 0 removable, 23 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "information_schema.sql_sizing_profiles"
INFO: "sql_sizing_profiles": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_class"
INFO: "pg_class": found 18 removable, 343 nonremovable row versions in 10 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_event_trigger"
INFO: "pg_event_trigger": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_db_role_setting"
INFO: "pg_db_role_setting": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_largeobject"
INFO: "pg_largeobject": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
Query returned successfully with no result in 720 ms.
INFO: vacuuming "public.mytable"
INFO: vacuuming "pg_catalog.pg_extension"
INFO: "pg_extension": found 0 removable, 2 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_foreign_data_wrapper"
INFO: "pg_foreign_data_wrapper": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_foreign_server"
INFO: "pg_foreign_server": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_foreign_table"
INFO: "pg_foreign_table": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_replication_origin"
INFO: "pg_replication_origin": found 6 removable, 0 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_default_acl"
INFO: "pg_default_acl": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_init_privs"
INFO: "pg_init_privs": found 0 removable, 149 nonremovable row versions in 2 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_seclabel"
INFO: "pg_seclabel": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_shseclabel"
INFO: "pg_shseclabel": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_collation"
INFO: "pg_collation": found 0 removable, 415 nonremovable row versions in 14 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_partitioned_table"
INFO: "pg_partitioned_table": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_range"
INFO: "pg_range": found 0 removable, 6 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_transform"
INFO: "pg_transform": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_sequence"
INFO: "pg_sequence": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_publication"
INFO: "pg_publication": found 0 removable, 1 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_publication_rel"
INFO: "pg_publication_rel": found 0 removable, 1 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_subscription_rel"
INFO: "pg_subscription_rel": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_largeobject"
INFO: "pg_largeobject": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "information_schema.sql_parts"
INFO: "sql_parts": found 0 removable, 9 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "information_schema.sql_languages"
INFO: "sql_languages": found 0 removable, 4 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "information_schema.sql_features"
INFO: "sql_features": found 0 removable, 671 nonremovable row versions in 7 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "information_schema.sql_implementation_info"
INFO: "sql_implementation_info": found 0 removable, 12 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "information_schema.sql_packages"
INFO: "sql_packages": found 0 removable, 10 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "information_schema.sql_sizing"
INFO: "sql_sizing": found 0 removable, 23 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "information_schema.sql_sizing_profiles"
INFO: "sql_sizing_profiles": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
VACUUM
Query returned successfully in 958 msec.
Attachments:
[text/plain] pg3.txt (15.8K, 3-pg3.txt)
download | inline:
INFO: vacuuming "public.mytable"
INFO: "mytable": found 0 removable, 1000001 nonremovable row versions in 4425 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.17 s, system: 0.17 s, elapsed: 0.41 s.
INFO: vacuuming "pg_catalog.pg_statistic"
INFO: "pg_statistic": found 48 removable, 394 nonremovable row versions in 18 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_type"
INFO: "pg_type": found 5 removable, 379 nonremovable row versions in 9 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_policy"
INFO: "pg_policy": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_authid"
INFO: "pg_authid": found 0 removable, 6 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_user_mapping"
INFO: "pg_user_mapping": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_subscription"
INFO: "pg_subscription": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_attribute"
INFO: "pg_attribute": found 22 removable, 2614 nonremovable row versions in 51 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_proc"
INFO: "pg_proc": found 0 removable, 2902 nonremovable row versions in 74 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_attrdef"
INFO: "pg_attrdef": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_constraint"
INFO: "pg_constraint": found 0 removable, 2 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_inherits"
INFO: "pg_inherits": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_index"
INFO: "pg_index": found 17 removable, 134 nonremovable row versions in 4 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_operator"
INFO: "pg_operator": found 0 removable, 787 nonremovable row versions in 15 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_opfamily"
INFO: "pg_opfamily": found 0 removable, 115 nonremovable row versions in 2 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_opclass"
INFO: "pg_opclass": found 0 removable, 133 nonremovable row versions in 3 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_am"
INFO: "pg_am": found 0 removable, 6 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_amop"
INFO: "pg_amop": found 0 removable, 709 nonremovable row versions in 6 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_amproc"
INFO: "pg_amproc": found 0 removable, 411 nonremovable row versions in 4 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_language"
INFO: "pg_language": found 0 removable, 4 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_largeobject_metadata"
INFO: "pg_largeobject_metadata": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_aggregate"
INFO: "pg_aggregate": found 0 removable, 138 nonremovable row versions in 2 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_statistic_ext"
INFO: "pg_statistic_ext": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_rewrite"
INFO: "pg_rewrite": found 0 removable, 121 nonremovable row versions in 12 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_trigger"
INFO: "pg_trigger": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_description"
INFO: "pg_description": found 0 removable, 4325 nonremovable row versions in 37 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_cast"
INFO: "pg_cast": found 0 removable, 219 nonremovable row versions in 2 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_enum"
INFO: "pg_enum": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_namespace"
INFO: "pg_namespace": found 0 removable, 6 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_conversion"
INFO: "pg_conversion": found 0 removable, 132 nonremovable row versions in 3 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_depend"
INFO: "pg_depend": found 28 removable, 7342 nonremovable row versions in 55 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_database"
INFO: "pg_database": found 0 removable, 3 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_tablespace"
INFO: "pg_tablespace": found 0 removable, 2 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_pltemplate"
INFO: "pg_pltemplate": found 0 removable, 8 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_auth_members"
INFO: "pg_auth_members": found 0 removable, 3 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_shdepend"
INFO: "pg_shdepend": found 0 removable, 6 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_shdescription"
INFO: "pg_shdescription": found 0 removable, 3 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_ts_config"
INFO: "pg_ts_config": found 0 removable, 16 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_ts_config_map"
INFO: "pg_ts_config_map": found 0 removable, 304 nonremovable row versions in 2 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_ts_dict"
INFO: "pg_ts_dict": found 0 removable, 16 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_ts_parser"
INFO: "pg_ts_parser": found 0 removable, 1 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_ts_template"
INFO: "pg_ts_template": found 0 removable, 5 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_extension"
INFO: "pg_extension": found 0 removable, 2 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_foreign_data_wrapper"
INFO: "pg_foreign_data_wrapper": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_foreign_server"
INFO: "pg_foreign_server": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_foreign_table"
INFO: "pg_foreign_table": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_replication_origin"
INFO: "pg_replication_origin": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_default_acl"
INFO: "pg_default_acl": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_init_privs"
INFO: "pg_init_privs": found 0 removable, 149 nonremovable row versions in 2 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_seclabel"
INFO: "pg_seclabel": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_shseclabel"
INFO: "pg_shseclabel": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_collation"
INFO: "pg_collation": found 0 removable, 415 nonremovable row versions in 14 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_partitioned_table"
INFO: "pg_partitioned_table": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_range"
INFO: "pg_range": found 0 removable, 6 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_transform"
INFO: "pg_transform": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_sequence"
INFO: "pg_sequence": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_publication"
INFO: "pg_publication": found 0 removable, 1 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_publication_rel"
INFO: "pg_publication_rel": found 0 removable, 1 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_subscription_rel"
INFO: "pg_subscription_rel": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "information_schema.sql_parts"
INFO: "sql_parts": found 0 removable, 9 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "information_schema.sql_languages"
INFO: "sql_languages": found 0 removable, 4 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "information_schema.sql_features"
INFO: "sql_features": found 0 removable, 671 nonremovable row versions in 7 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "information_schema.sql_implementation_info"
INFO: "sql_implementation_info": found 0 removable, 12 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "information_schema.sql_packages"
INFO: "sql_packages": found 0 removable, 10 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "information_schema.sql_sizing"
INFO: "sql_sizing": found 0 removable, 23 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "information_schema.sql_sizing_profiles"
INFO: "sql_sizing_profiles": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_class"
INFO: "pg_class": found 18 removable, 343 nonremovable row versions in 10 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_event_trigger"
INFO: "pg_event_trigger": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_db_role_setting"
INFO: "pg_db_role_setting": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_largeobject"
INFO: "pg_largeobject": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
Query returned successfully with no result in 720 ms.
[text/plain] pg4.txt (5.8K, 4-pg4.txt)
download | inline:
INFO: vacuuming "public.mytable"
INFO: vacuuming "pg_catalog.pg_extension"
INFO: "pg_extension": found 0 removable, 2 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_foreign_data_wrapper"
INFO: "pg_foreign_data_wrapper": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_foreign_server"
INFO: "pg_foreign_server": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_foreign_table"
INFO: "pg_foreign_table": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_replication_origin"
INFO: "pg_replication_origin": found 6 removable, 0 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_default_acl"
INFO: "pg_default_acl": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_init_privs"
INFO: "pg_init_privs": found 0 removable, 149 nonremovable row versions in 2 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_seclabel"
INFO: "pg_seclabel": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_shseclabel"
INFO: "pg_shseclabel": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_collation"
INFO: "pg_collation": found 0 removable, 415 nonremovable row versions in 14 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_partitioned_table"
INFO: "pg_partitioned_table": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_range"
INFO: "pg_range": found 0 removable, 6 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_transform"
INFO: "pg_transform": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_sequence"
INFO: "pg_sequence": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_publication"
INFO: "pg_publication": found 0 removable, 1 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_publication_rel"
INFO: "pg_publication_rel": found 0 removable, 1 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_subscription_rel"
INFO: "pg_subscription_rel": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_catalog.pg_largeobject"
INFO: "pg_largeobject": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "information_schema.sql_parts"
INFO: "sql_parts": found 0 removable, 9 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "information_schema.sql_languages"
INFO: "sql_languages": found 0 removable, 4 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "information_schema.sql_features"
INFO: "sql_features": found 0 removable, 671 nonremovable row versions in 7 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "information_schema.sql_implementation_info"
INFO: "sql_implementation_info": found 0 removable, 12 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "information_schema.sql_packages"
INFO: "sql_packages": found 0 removable, 10 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "information_schema.sql_sizing"
INFO: "sql_sizing": found 0 removable, 23 nonremovable row versions in 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "information_schema.sql_sizing_profiles"
INFO: "sql_sizing_profiles": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
VACUUM
Query returned successfully in 958 msec.
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query
2018-02-26 14:19 pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-02-26 16:18 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
2018-02-26 16:23 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
2018-02-26 16:32 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
@ 2018-02-28 11:49 ` Khushboo Vashi <[email protected]>
2018-02-28 15:25 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Joao De Almeida Pereira <[email protected]>
1 sibling, 1 reply; 17+ messages in thread
From: Khushboo Vashi @ 2018-02-28 11:49 UTC (permalink / raw)
To: Dave Page <[email protected]>; +Cc: Murtuza Zabuawala <[email protected]>; pgadmin-hackers
On Mon, Feb 26, 2018 at 10:02 PM, Dave Page <[email protected]> wrote:
> Argh, I ran some tests, but didn't spot any lost messages in the tests I
> ran. I'll revert the patch.
>
> Khushboo;
>
> Please look at the following:
>
> - Fix the patch so it doesn't drop messages.
>
Fixed.
By default, the notice attribute of the connection object of psycopg 2 only
stores 50 notices. Once it reaches to 50 it starts from 1 again.
To fix this I have changed the notice attribute from list to deque to
append more messages. Currently I have kept the maximum limit at a time of
the notice attribute is 100000 (in a single poll).
> - Add regression tests to make sure it doesn't break in the future. This
> may require creating one or more functions the spew out a whole lot of
> notices, and then running a couple of queries and checking the output.
>
Added. With this regression test, the current code is failing which has
been taken care in this patch.
> - Check the messages panel on the history tab. I just noticed it seems to
> only be showing an even smaller subset of the messages.
>
Tested and no issues found.
>
>
Thanks.
>
> On Mon, Feb 26, 2018 at 4:23 PM, Murtuza Zabuawala <murtuza.zabuawala@
> enterprisedb.com> wrote:
>
>> Sent bit early,
>>
>> You can run 'VACUUM FULL VERBOSE' in query tool and verify the populated
>> messages (pgAdmin3 vs. pgAdmin4).
>>
>>
>> On Mon, Feb 26, 2018 at 9:48 PM, Murtuza Zabuawala <
>> [email protected]> wrote:
>>
>>> Hi Khushboo/Dave,
>>>
>>> With given commit, I'm again seeing the issue raised in
>>> https://redmine.postgresql.org/issues/1523 :(
>>>
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Murtuza Zabuawala
>>> EnterpriseDB: http://www.enterprisedb.com
>>> The Enterprise PostgreSQL Company
>>>
>>>
>>> On Mon, Feb 26, 2018 at 7:49 PM, Dave Page <[email protected]> wrote:
>>>
>>>> Ensure we pick up the messages from the current query and not a
>>>> previous one. Fixes #3094
>>>>
>>>> Branch
>>>> ------
>>>> master
>>>>
>>>> Details
>>>> -------
>>>> https://git.postgresql.org/gitweb?p=pgadmin4.git;a=commitdif
>>>> f;h=08b3ccc01a4d57e8ea3657f8882a53dcd1b99386
>>>> Author: Khushboo Vashi <[email protected]>
>>>>
>>>> Modified Files
>>>> --------------
>>>> web/pgadmin/utils/driver/abstract.py | 1 +
>>>> web/pgadmin/utils/driver/psycopg2/__init__.py | 64
>>>> +++++++++------------------
>>>> 2 files changed, 21 insertions(+), 44 deletions(-)
>>>>
>>>>
>>>
>>
>
>
> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
Attachments:
[text/x-patch] RM_3094_ver1.patch (10.7K, 3-RM_3094_ver1.patch)
download | inline diff:
diff --git a/web/pgadmin/feature_tests/keyboard_shortcut_test.py b/web/pgadmin/feature_tests/keyboard_shortcut_test.py
index b83457c..f751211 100644
--- a/web/pgadmin/feature_tests/keyboard_shortcut_test.py
+++ b/web/pgadmin/feature_tests/keyboard_shortcut_test.py
@@ -62,7 +62,9 @@ class KeyboardShortcutFeatureTest(BaseFeatureTest):
).key_down(
key_combo[2]
).key_up(
- Keys.ALT
+ key_combo[0]
+ ).key_up(
+ key_combo[1]
).perform()
print("Executing shortcut: " + self.new_shortcuts[s]['locator'] + "...", file=sys.stderr, end="")
diff --git a/web/pgadmin/tools/sqleditor/tests/test_poll_query_tool.py b/web/pgadmin/tools/sqleditor/tests/test_poll_query_tool.py
new file mode 100644
index 0000000..61655e7
--- /dev/null
+++ b/web/pgadmin/tools/sqleditor/tests/test_poll_query_tool.py
@@ -0,0 +1,109 @@
+##########################################################################
+#
+# pgAdmin 4 - PostgreSQL Tools
+#
+# Copyright (C) 2013 - 2018, The pgAdmin Development Team
+# This software is released under the PostgreSQL Licence
+#
+##########################################################################
+
+import json
+
+from pgadmin.browser.server_groups.servers.databases.tests import utils as \
+ database_utils
+from pgadmin.utils.route import BaseTestGenerator
+from regression import parent_node_dict
+from regression.python_test_utils import test_utils as utils
+
+
+class TestPollQueryTool(BaseTestGenerator):
+ """ This class will test the query tool polling. """
+ scenarios = [
+ ('When query tool polling returns messages with result data-set',
+ dict(
+ sql="""
+DROP TABLE IF EXISTS test_for_notices;
+
+DO $$
+BEGIN
+ RAISE NOTICE 'Hello, world!';
+END $$;
+
+SELECT 'CHECKING POLLING';
+ """,
+ expected_message='NOTICE: table "test_for_notices" ' +
+ """does not exist, skipping
+NOTICE: Hello, world!
+""",
+ expected_result='CHECKING POLLING'
+ )),
+ ('When query tool polling returns a large subset of messages'
+ ' with result data-set',
+ dict(
+ sql="""
+DO $$
+BEGIN
+ FOR i in 1..1000 LOOP
+ RAISE NOTICE 'Count is %', i;
+ END LOOP;
+END $$;
+
+SELECT 'CHECKING POLLING FOR LONG MESSAGES';
+""",
+ expected_message="\n".join(["NOTICE: Count is {0}".format(i)
+ for i in range(1, 1001)]) + "\n",
+ expected_result='CHECKING POLLING FOR LONG MESSAGES'
+ )),
+ ('When query tool polling returns no messages'
+ ' with result data-set',
+ dict(
+ sql="SELECT 'CHECKING POLLING WITHOUT MESSAGES';",
+ expected_message=None,
+ expected_result='CHECKING POLLING WITHOUT MESSAGES'
+ ))
+ ]
+
+ def runTest(self):
+ """ This function will check messages return by query tool polling. """
+ database_info = parent_node_dict["database"][-1]
+ server_id = database_info["server_id"]
+
+ db_id = database_info["db_id"]
+ db_con = database_utils.connect_database(self,
+ utils.SERVER_GROUP,
+ server_id,
+ db_id)
+ if not db_con["info"] == "Database connected.":
+ raise Exception("Could not connect to the database.")
+
+ # Initialize query tool
+ url = '/datagrid/initialize/query_tool/{0}/{1}/{2}'.format(
+ utils.SERVER_GROUP, server_id, db_id)
+ response = self.tester.post(url)
+ self.assertEquals(response.status_code, 200)
+
+ response_data = json.loads(response.data.decode('utf-8'))
+ trans_id = response_data['data']['gridTransId']
+
+ # Start query tool transaction
+ url = '/sqleditor/query_tool/start/{0}'.format(trans_id)
+ response = self.tester.post(url, data=json.dumps({"sql": self.sql}),
+ content_type='html/json')
+
+ self.assertEquals(response.status_code, 200)
+
+ # Query tool polling
+ url = '/sqleditor/poll/{0}'.format(trans_id)
+ response = self.tester.get(url)
+ self.assertEquals(response.status_code, 200)
+ response_data = json.loads(response.data.decode('utf-8'))
+
+ # Check the returned messages
+ self.assertEquals(self.expected_message,
+ response_data['data']['additional_messages'])
+ # Check the output
+ self.assertEquals(self.expected_result,
+ response_data['data']['result'][0][0])
+
+ # Disconnect the database
+ database_utils.disconnect_database(self, server_id, db_id)
diff --git a/web/pgadmin/utils/driver/abstract.py b/web/pgadmin/utils/driver/abstract.py
index 32e1c97..271bfec 100644
--- a/web/pgadmin/utils/driver/abstract.py
+++ b/web/pgadmin/utils/driver/abstract.py
@@ -168,6 +168,8 @@ class BaseConnection(object):
ASYNC_WRITE_TIMEOUT = 3
ASYNC_NOT_CONNECTED = 4
ASYNC_EXECUTION_ABORTED = 5
+ ASYNC_TIMEOUT = 0.2
+ ASYNC_NOTICE_MAXLENGTH = 100000
@abstractmethod
def connect(self, **kwargs):
diff --git a/web/pgadmin/utils/driver/psycopg2/__init__.py b/web/pgadmin/utils/driver/psycopg2/__init__.py
index 81442e4..4305aa5 100644
--- a/web/pgadmin/utils/driver/psycopg2/__init__.py
+++ b/web/pgadmin/utils/driver/psycopg2/__init__.py
@@ -37,6 +37,7 @@ from .cursor import DictCursor
from .typecast import register_global_typecasters, \
register_string_typecasters, register_binary_typecasters, \
register_array_to_string_typecasters, ALL_JSON_TYPES
+from collections import deque
if sys.version_info < (3,):
@@ -110,7 +111,7 @@ class Connection(BaseConnection):
- This method is used to wait for asynchronous connection. This is a
blocking call.
- * _wait_timeout(conn, time)
+ * _wait_timeout(conn)
- This method is used to wait for asynchronous connection with timeout.
This is a non blocking call.
@@ -310,6 +311,9 @@ class Connection(BaseConnection):
)
return False, msg
+ # Overwrite connection notice attr to support
+ # more than 50 notices at a time
+ pg_conn.notices = deque([], self.ASYNC_NOTICE_MAXLENGTH)
self.conn = pg_conn
self.wasConnected = True
try:
@@ -1208,6 +1212,7 @@ Failed to reset the connection to the server due to following error:
)
return False, msg
+ pg_conn.notices = deque([], self.ASYNC_NOTICE_MAXLENGTH)
self.conn = pg_conn
self.__backend_pid = pg_conn.get_backend_pid()
@@ -1261,51 +1266,27 @@ Failed to reset the connection to the server due to following error:
Args:
conn: connection object
- time: wait time
"""
-
- state = conn.poll()
- if state == psycopg2.extensions.POLL_OK:
- return self.ASYNC_OK
- elif state == psycopg2.extensions.POLL_WRITE:
- # Wait for the given time and then check the return status
- # If three empty lists are returned then the time-out is reached.
- timeout_status = select.select([], [conn.fileno()], [], 0)
- if timeout_status == ([], [], []):
- return self.ASYNC_WRITE_TIMEOUT
-
- # poll again to check the state if it is still POLL_WRITE
- # then return ASYNC_WRITE_TIMEOUT else return ASYNC_OK.
+ while 1:
state = conn.poll()
- if state == psycopg2.extensions.POLL_WRITE:
- return self.ASYNC_WRITE_TIMEOUT
- return self.ASYNC_OK
- elif state == psycopg2.extensions.POLL_READ:
- # Wait for the given time and then check the return status
- # If three empty lists are returned then the time-out is reached.
- timeout_status = select.select([conn.fileno()], [], [], 0)
- if timeout_status == ([], [], []):
- return self.ASYNC_READ_TIMEOUT
-
- # select.select timeout option works only if we provide
- # empty [] [] [] file descriptor in select.select() function
- # and that also works only on UNIX based system, it do not support
- # Windows Hence we have wrote our own pooling mechanism to read
- # data fast each call conn.poll() reads chunks of data from
- # connection object more we poll more we read data from connection
- cnt = 0
- while cnt < 1000:
- # poll again to check the state if it is still POLL_READ
- # then return ASYNC_READ_TIMEOUT else return ASYNC_OK.
- state = conn.poll()
- if state == psycopg2.extensions.POLL_OK:
- return self.ASYNC_OK
- cnt += 1
- return self.ASYNC_READ_TIMEOUT
- else:
- raise psycopg2.OperationalError(
- "poll() returned %s from _wait_timeout function" % state
- )
+ if state == psycopg2.extensions.POLL_OK:
+ return self.ASYNC_OK
+ elif state == psycopg2.extensions.POLL_WRITE:
+ # Wait for the given time and then check the return status
+ # If three empty lists are returned then the time-out is reached.
+ timeout_status = select.select([], [conn.fileno()], [], self.ASYNC_TIMEOUT)
+ if timeout_status == ([], [], []):
+ return self.ASYNC_WRITE_TIMEOUT
+ elif state == psycopg2.extensions.POLL_READ:
+ # Wait for the given time and then check the return status
+ # If three empty lists are returned then the time-out is reached.
+ timeout_status = select.select([conn.fileno()], [], [], self.ASYNC_TIMEOUT)
+ if timeout_status == ([], [], []):
+ return self.ASYNC_READ_TIMEOUT
+ else:
+ raise psycopg2.OperationalError(
+ "poll() returned %s from _wait_timeout function" % state
+ )
def poll(self, formatted_exception_msg=False, no_result=False):
"""
@@ -1347,8 +1328,8 @@ Failed to reset the connection to the server due to following error:
is_error = True
if self.conn.notices and self.__notices is not None:
- while self.conn.notices:
- self.__notices.append(self.conn.notices.pop(0)[:])
+ self.__notices.extend(self.conn.notices)
+ self.conn.notices.clear()
# We also need to fetch notices before we return from function in case
# of any Exception, To avoid code duplication we will return after
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query
2018-02-26 14:19 pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-02-26 16:18 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
2018-02-26 16:23 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
2018-02-26 16:32 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-02-28 11:49 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
@ 2018-02-28 15:25 ` Joao De Almeida Pereira <[email protected]>
2018-03-01 05:56 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
0 siblings, 1 reply; 17+ messages in thread
From: Joao De Almeida Pereira @ 2018-02-28 15:25 UTC (permalink / raw)
To: Khushboo Vashi <[email protected]>; +Cc: Dave Page <[email protected]>; Murtuza Zabuawala <[email protected]>; pgadmin-hackers
Hello Khushboo,
After reviewing the patch I have the gut feeling that we do not have enough
test coverage on this issue, specially due to the intricate while loop and
conditions around the polling.
I think that this deserve Unit tests around it, When I say Unit Test I am
not talking about executing queries against the database, but do some
stubbing of the database so that we can control the flow that we want.
It is a temptation to try to always do a Feature Test to test what we want
because it is "easier" to write and ultimately it is what users see, but
while 1 Feature Test runs we can run 200 Unit Tests that give us much more
confidence that the code is doing what we expect it to do.
This being said, I run the tests on the CI Pipeline and all tests pass.
Running pycodestyle fails due to some line sizes on the
psycopg2/__init__py. I believe that it is not what you changed, but since
you were changing the file it can be fixed it is just:
pgadmin/utils/driver/psycopg2/__init__.py:1276: [E501] line too long (81 >
79 characters)
pgadmin/utils/driver/psycopg2/__init__.py:1277: [E501] line too long (91 >
79 characters)
pgadmin/utils/driver/psycopg2/__init__.py:1282: [E501] line too long (81 >
79 characters)
pgadmin/utils/driver/psycopg2/__init__.py:1283: [E501] line too long (91 >
79 characters)
4 E501 line too long (81 > 79 characters)
Thanks
Joao
On Wed, Feb 28, 2018 at 6:49 AM Khushboo Vashi <
[email protected]> wrote:
> On Mon, Feb 26, 2018 at 10:02 PM, Dave Page <[email protected]> wrote:
>
>> Argh, I ran some tests, but didn't spot any lost messages in the tests I
>> ran. I'll revert the patch.
>>
>> Khushboo;
>>
>> Please look at the following:
>>
>> - Fix the patch so it doesn't drop messages.
>>
> Fixed.
> By default, the notice attribute of the connection object of psycopg 2
> only stores 50 notices. Once it reaches to 50 it starts from 1 again.
> To fix this I have changed the notice attribute from list to deque to
> append more messages. Currently I have kept the maximum limit at a time of
> the notice attribute is 100000 (in a single poll).
>
>> - Add regression tests to make sure it doesn't break in the future. This
>> may require creating one or more functions the spew out a whole lot of
>> notices, and then running a couple of queries and checking the output.
>>
> Added. With this regression test, the current code is failing which has
> been taken care in this patch.
>
>> - Check the messages panel on the history tab. I just noticed it seems to
>> only be showing an even smaller subset of the messages.
>>
> Tested and no issues found.
>
>>
>>
> Thanks.
>>
>> On Mon, Feb 26, 2018 at 4:23 PM, Murtuza Zabuawala <
>> [email protected]> wrote:
>>
>>> Sent bit early,
>>>
>>> You can run 'VACUUM FULL VERBOSE' in query tool and verify the
>>> populated messages (pgAdmin3 vs. pgAdmin4).
>>>
>>>
>>> On Mon, Feb 26, 2018 at 9:48 PM, Murtuza Zabuawala <
>>> [email protected]> wrote:
>>>
>>>> Hi Khushboo/Dave,
>>>>
>>>> With given commit, I'm again seeing the issue raised in
>>>> https://redmine.postgresql.org/issues/1523 :(
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Murtuza Zabuawala
>>>> EnterpriseDB: http://www.enterprisedb.com
>>>> The Enterprise PostgreSQL Company
>>>>
>>>>
>>>> On Mon, Feb 26, 2018 at 7:49 PM, Dave Page <[email protected]> wrote:
>>>>
>>>>> Ensure we pick up the messages from the current query and not a
>>>>> previous one. Fixes #3094
>>>>>
>>>>> Branch
>>>>> ------
>>>>> master
>>>>>
>>>>> Details
>>>>> -------
>>>>>
>>>>> https://git.postgresql.org/gitweb?p=pgadmin4.git;a=commitdiff;h=08b3ccc01a4d57e8ea3657f8882a53dcd1b9...
>>>>> Author: Khushboo Vashi <[email protected]>
>>>>>
>>>>> Modified Files
>>>>> --------------
>>>>> web/pgadmin/utils/driver/abstract.py | 1 +
>>>>> web/pgadmin/utils/driver/psycopg2/__init__.py | 64
>>>>> +++++++++------------------
>>>>> 2 files changed, 21 insertions(+), 44 deletions(-)
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> Dave Page
>> Blog: http://pgsnake.blogspot.com
>> Twitter: @pgsnake
>>
>> EnterpriseDB UK: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>>
>
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query
2018-02-26 14:19 pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-02-26 16:18 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
2018-02-26 16:23 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
2018-02-26 16:32 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-02-28 11:49 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
2018-02-28 15:25 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Joao De Almeida Pereira <[email protected]>
@ 2018-03-01 05:56 ` Khushboo Vashi <[email protected]>
2018-03-01 15:21 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Joao De Almeida Pereira <[email protected]>
2018-03-02 13:25 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
0 siblings, 2 replies; 17+ messages in thread
From: Khushboo Vashi @ 2018-03-01 05:56 UTC (permalink / raw)
To: Joao De Almeida Pereira <[email protected]>; +Cc: Dave Page <[email protected]>; Murtuza Zabuawala <[email protected]>; pgadmin-hackers
Hi Joao,
Thanks for reviewing.
On Wed, Feb 28, 2018 at 8:55 PM, Joao De Almeida Pereira <
[email protected]> wrote:
> Hello Khushboo,
> After reviewing the patch I have the gut feeling that we do not have
> enough test coverage on this issue, specially due to the intricate while
> loop and conditions around the polling.
> I think that this deserve Unit tests around it, When I say Unit Test I am
> not talking about executing queries against the database, but do some
> stubbing of the database so that we can control the flow that we want.
>
You are right. It needs more unit testing. I have checked below scenarios:
1. Returns 2 notices with data output
2. Returns 1000 notices with data output
3. No notices with data output
By running above, I have checked, each time returned notices are accurate,
no old notices are getting appended, it does not affect with the amount of
messages (few, none or more). Also, with the updated patch, I have made
sure that all these queries run with the single transaction id (same
connection).
So, please let me know if you think I can add more things to this.
>
>
It is a temptation to try to always do a Feature Test to test what we want
> because it is "easier" to write and ultimately it is what users see, but
> while 1 Feature Test runs we can run 200 Unit Tests that give us much more
> confidence that the code is doing what we expect it to do.
>
> Right, so added regression tests instead of feature tests.
This being said, I run the tests on the CI Pipeline and all tests pass.
> Running pycodestyle fails due to some line sizes on the
> psycopg2/__init__py. I believe that it is not what you changed, but since
> you were changing the file it can be fixed it is just:
>
> pgadmin/utils/driver/psycopg2/__init__.py:1276: [E501] line too long (81
> > 79 characters)
> pgadmin/utils/driver/psycopg2/__init__.py:1277: [E501] line too long (91
> > 79 characters)
> pgadmin/utils/driver/psycopg2/__init__.py:1282: [E501] line too long (81
> > 79 characters)
> pgadmin/utils/driver/psycopg2/__init__.py:1283: [E501] line too long (91
> > 79 characters)
> 4 E501 line too long (81 > 79 characters)
>
> Fixed. Thanks for pointing out.
>
> Thanks
> Joao
>
>
> On Wed, Feb 28, 2018 at 6:49 AM Khushboo Vashi <
> [email protected]> wrote:
>
>> On Mon, Feb 26, 2018 at 10:02 PM, Dave Page <[email protected]> wrote:
>>
>>> Argh, I ran some tests, but didn't spot any lost messages in the tests I
>>> ran. I'll revert the patch.
>>>
>>> Khushboo;
>>>
>>> Please look at the following:
>>>
>>> - Fix the patch so it doesn't drop messages.
>>>
>> Fixed.
>> By default, the notice attribute of the connection object of psycopg 2
>> only stores 50 notices. Once it reaches to 50 it starts from 1 again.
>> To fix this I have changed the notice attribute from list to deque to
>> append more messages. Currently I have kept the maximum limit at a time of
>> the notice attribute is 100000 (in a single poll).
>>
>>> - Add regression tests to make sure it doesn't break in the future. This
>>> may require creating one or more functions the spew out a whole lot of
>>> notices, and then running a couple of queries and checking the output.
>>>
>> Added. With this regression test, the current code is failing which has
>> been taken care in this patch.
>>
>>> - Check the messages panel on the history tab. I just noticed it seems
>>> to only be showing an even smaller subset of the messages.
>>>
>> Tested and no issues found.
>>
>>>
>>>
>> Thanks.
>>>
>>> On Mon, Feb 26, 2018 at 4:23 PM, Murtuza Zabuawala <murtuza.zabuawala@
>>> enterprisedb.com> wrote:
>>>
>>>> Sent bit early,
>>>>
>>>> You can run 'VACUUM FULL VERBOSE' in query tool and verify the
>>>> populated messages (pgAdmin3 vs. pgAdmin4).
>>>>
>>>>
>>>> On Mon, Feb 26, 2018 at 9:48 PM, Murtuza Zabuawala <murtuza.zabuawala@
>>>> enterprisedb.com> wrote:
>>>>
>>>>> Hi Khushboo/Dave,
>>>>>
>>>>> With given commit, I'm again seeing the issue raised in
>>>>> https://redmine.postgresql.org/issues/1523 :(
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Murtuza Zabuawala
>>>>> EnterpriseDB: http://www.enterprisedb.com
>>>>> The Enterprise PostgreSQL Company
>>>>>
>>>>>
>>>>> On Mon, Feb 26, 2018 at 7:49 PM, Dave Page <[email protected]> wrote:
>>>>>
>>>>>> Ensure we pick up the messages from the current query and not a
>>>>>> previous one. Fixes #3094
>>>>>>
>>>>>> Branch
>>>>>> ------
>>>>>> master
>>>>>>
>>>>>> Details
>>>>>> -------
>>>>>> https://git.postgresql.org/gitweb?p=pgadmin4.git;a=commitdiff;h=
>>>>>> 08b3ccc01a4d57e8ea3657f8882a53dcd1b99386
>>>>>> Author: Khushboo Vashi <[email protected]>
>>>>>>
>>>>>> Modified Files
>>>>>> --------------
>>>>>> web/pgadmin/utils/driver/abstract.py | 1 +
>>>>>> web/pgadmin/utils/driver/psycopg2/__init__.py | 64
>>>>>> +++++++++------------------
>>>>>> 2 files changed, 21 insertions(+), 44 deletions(-)
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Dave Page
>>> Blog: http://pgsnake.blogspot.com
>>> Twitter: @pgsnake
>>>
>>> EnterpriseDB UK: http://www.enterprisedb.com
>>> The Enterprise PostgreSQL Company
>>>
>>
Attachments:
[text/x-patch] RM_3094_ver2.patch (11.3K, 3-RM_3094_ver2.patch)
download | inline diff:
diff --git a/web/pgadmin/feature_tests/keyboard_shortcut_test.py b/web/pgadmin/feature_tests/keyboard_shortcut_test.py
index b83457c..bbae194 100644
--- a/web/pgadmin/feature_tests/keyboard_shortcut_test.py
+++ b/web/pgadmin/feature_tests/keyboard_shortcut_test.py
@@ -62,10 +62,13 @@ class KeyboardShortcutFeatureTest(BaseFeatureTest):
).key_down(
key_combo[2]
).key_up(
- Keys.ALT
+ key_combo[0]
+ ).key_up(
+ key_combo[1]
).perform()
- print("Executing shortcut: " + self.new_shortcuts[s]['locator'] + "...", file=sys.stderr, end="")
+ print("Executing shortcut: " + self.new_shortcuts[s]['locator'] +
+ "...", file=sys.stderr, end="")
self.wait.until(
EC.presence_of_element_located(
diff --git a/web/pgadmin/tools/sqleditor/tests/test_poll_query_tool.py b/web/pgadmin/tools/sqleditor/tests/test_poll_query_tool.py
new file mode 100644
index 0000000..e47129b
--- /dev/null
+++ b/web/pgadmin/tools/sqleditor/tests/test_poll_query_tool.py
@@ -0,0 +1,112 @@
+##########################################################################
+#
+# pgAdmin 4 - PostgreSQL Tools
+#
+# Copyright (C) 2013 - 2018, The pgAdmin Development Team
+# This software is released under the PostgreSQL Licence
+#
+##########################################################################
+
+import json
+
+from pgadmin.browser.server_groups.servers.databases.tests import utils as \
+ database_utils
+from pgadmin.utils.route import BaseTestGenerator
+from regression import parent_node_dict
+from regression.python_test_utils import test_utils as utils
+
+
+class TestPollQueryTool(BaseTestGenerator):
+ """ This class will test the query tool polling. """
+ scenarios = [
+ ('When query tool polling returns messages with result data-set',
+ dict(
+ sql=[
+ """
+DROP TABLE IF EXISTS test_for_notices;
+
+DO $$
+BEGIN
+ RAISE NOTICE 'Hello, world!';
+END $$;
+
+SELECT 'CHECKING POLLING';
+""",
+ """
+DO $$
+BEGIN
+ FOR i in 1..1000 LOOP
+ RAISE NOTICE 'Count is %', i;
+ END LOOP;
+END $$;
+
+SELECT 'CHECKING POLLING FOR LONG MESSAGES';
+""",
+ "SELECT 'CHECKING POLLING WITHOUT MESSAGES';"
+ ],
+ expected_message=['NOTICE: table "test_for_notices" ' +
+ """does not exist, skipping
+NOTICE: Hello, world!
+""",
+ "\n".join(["NOTICE: Count is {0}".format(i)
+ for i in range(1, 1001)]) + "\n",
+ None],
+ expected_result=['CHECKING POLLING',
+ 'CHECKING POLLING FOR LONG MESSAGES',
+ 'CHECKING POLLING WITHOUT MESSAGES'],
+ print_messages=['2 NOTICES WITH DATASET',
+ '1000 NOTICES WITH DATASET',
+ 'NO NOTICE WITH DATASET'
+ ]
+ ))
+ ]
+
+ def runTest(self):
+ """ This function will check messages return by query tool polling. """
+ database_info = parent_node_dict["database"][-1]
+ self.server_id = database_info["server_id"]
+
+ self.db_id = database_info["db_id"]
+ db_con = database_utils.connect_database(self,
+ utils.SERVER_GROUP,
+ self.server_id,
+ self.db_id)
+ if not db_con["info"] == "Database connected.":
+ raise Exception("Could not connect to the database.")
+
+ # Initialize query tool
+ url = '/datagrid/initialize/query_tool/{0}/{1}/{2}'.format(
+ utils.SERVER_GROUP, self.server_id, self.db_id)
+ response = self.tester.post(url)
+ self.assertEquals(response.status_code, 200)
+
+ response_data = json.loads(response.data.decode('utf-8'))
+ self.trans_id = response_data['data']['gridTransId']
+
+ cnt = 0
+ for s in self.sql:
+ print("Executing and polling with: " + self.print_messages[cnt])
+ # Start query tool transaction
+ url = '/sqleditor/query_tool/start/{0}'.format(self.trans_id)
+ response = self.tester.post(url, data=json.dumps({"sql": s}),
+ content_type='html/json')
+
+ self.assertEquals(response.status_code, 200)
+
+ # Query tool polling
+ url = '/sqleditor/poll/{0}'.format(self.trans_id)
+ response = self.tester.get(url)
+ self.assertEquals(response.status_code, 200)
+ response_data = json.loads(response.data.decode('utf-8'))
+
+ # Check the returned messages
+ self.assertEquals(self.expected_message[cnt],
+ response_data['data']['additional_messages'])
+ # Check the output
+ self.assertEquals(self.expected_result[cnt],
+ response_data['data']['result'][0][0])
+
+ cnt += 1
+
+ # Disconnect the database
+ database_utils.disconnect_database(self, self.server_id, self.db_id)
diff --git a/web/pgadmin/utils/driver/abstract.py b/web/pgadmin/utils/driver/abstract.py
index 32e1c97..271bfec 100644
--- a/web/pgadmin/utils/driver/abstract.py
+++ b/web/pgadmin/utils/driver/abstract.py
@@ -168,6 +168,8 @@ class BaseConnection(object):
ASYNC_WRITE_TIMEOUT = 3
ASYNC_NOT_CONNECTED = 4
ASYNC_EXECUTION_ABORTED = 5
+ ASYNC_TIMEOUT = 0.2
+ ASYNC_NOTICE_MAXLENGTH = 100000
@abstractmethod
def connect(self, **kwargs):
diff --git a/web/pgadmin/utils/driver/psycopg2/__init__.py b/web/pgadmin/utils/driver/psycopg2/__init__.py
index 81442e4..941a694 100644
--- a/web/pgadmin/utils/driver/psycopg2/__init__.py
+++ b/web/pgadmin/utils/driver/psycopg2/__init__.py
@@ -37,6 +37,7 @@ from .cursor import DictCursor
from .typecast import register_global_typecasters, \
register_string_typecasters, register_binary_typecasters, \
register_array_to_string_typecasters, ALL_JSON_TYPES
+from collections import deque
if sys.version_info < (3,):
@@ -110,7 +111,7 @@ class Connection(BaseConnection):
- This method is used to wait for asynchronous connection. This is a
blocking call.
- * _wait_timeout(conn, time)
+ * _wait_timeout(conn)
- This method is used to wait for asynchronous connection with timeout.
This is a non blocking call.
@@ -310,6 +311,9 @@ class Connection(BaseConnection):
)
return False, msg
+ # Overwrite connection notice attr to support
+ # more than 50 notices at a time
+ pg_conn.notices = deque([], self.ASYNC_NOTICE_MAXLENGTH)
self.conn = pg_conn
self.wasConnected = True
try:
@@ -1208,6 +1212,7 @@ Failed to reset the connection to the server due to following error:
)
return False, msg
+ pg_conn.notices = deque([], self.ASYNC_NOTICE_MAXLENGTH)
self.conn = pg_conn
self.__backend_pid = pg_conn.get_backend_pid()
@@ -1261,51 +1266,31 @@ Failed to reset the connection to the server due to following error:
Args:
conn: connection object
- time: wait time
"""
-
- state = conn.poll()
- if state == psycopg2.extensions.POLL_OK:
- return self.ASYNC_OK
- elif state == psycopg2.extensions.POLL_WRITE:
- # Wait for the given time and then check the return status
- # If three empty lists are returned then the time-out is reached.
- timeout_status = select.select([], [conn.fileno()], [], 0)
- if timeout_status == ([], [], []):
- return self.ASYNC_WRITE_TIMEOUT
-
- # poll again to check the state if it is still POLL_WRITE
- # then return ASYNC_WRITE_TIMEOUT else return ASYNC_OK.
+ while 1:
state = conn.poll()
- if state == psycopg2.extensions.POLL_WRITE:
- return self.ASYNC_WRITE_TIMEOUT
- return self.ASYNC_OK
- elif state == psycopg2.extensions.POLL_READ:
- # Wait for the given time and then check the return status
- # If three empty lists are returned then the time-out is reached.
- timeout_status = select.select([conn.fileno()], [], [], 0)
- if timeout_status == ([], [], []):
- return self.ASYNC_READ_TIMEOUT
-
- # select.select timeout option works only if we provide
- # empty [] [] [] file descriptor in select.select() function
- # and that also works only on UNIX based system, it do not support
- # Windows Hence we have wrote our own pooling mechanism to read
- # data fast each call conn.poll() reads chunks of data from
- # connection object more we poll more we read data from connection
- cnt = 0
- while cnt < 1000:
- # poll again to check the state if it is still POLL_READ
- # then return ASYNC_READ_TIMEOUT else return ASYNC_OK.
- state = conn.poll()
- if state == psycopg2.extensions.POLL_OK:
- return self.ASYNC_OK
- cnt += 1
- return self.ASYNC_READ_TIMEOUT
- else:
- raise psycopg2.OperationalError(
- "poll() returned %s from _wait_timeout function" % state
- )
+ if state == psycopg2.extensions.POLL_OK:
+ return self.ASYNC_OK
+ elif state == psycopg2.extensions.POLL_WRITE:
+ # Wait for the given time and then check the return status
+ # If three empty lists are returned then the time-out is
+ # reached.
+ timeout_status = select.select([], [conn.fileno()], [],
+ self.ASYNC_TIMEOUT)
+ if timeout_status == ([], [], []):
+ return self.ASYNC_WRITE_TIMEOUT
+ elif state == psycopg2.extensions.POLL_READ:
+ # Wait for the given time and then check the return status
+ # If three empty lists are returned then the time-out is
+ # reached.
+ timeout_status = select.select([conn.fileno()], [], [],
+ self.ASYNC_TIMEOUT)
+ if timeout_status == ([], [], []):
+ return self.ASYNC_READ_TIMEOUT
+ else:
+ raise psycopg2.OperationalError(
+ "poll() returned %s from _wait_timeout function" % state
+ )
def poll(self, formatted_exception_msg=False, no_result=False):
"""
@@ -1347,8 +1332,8 @@ Failed to reset the connection to the server due to following error:
is_error = True
if self.conn.notices and self.__notices is not None:
- while self.conn.notices:
- self.__notices.append(self.conn.notices.pop(0)[:])
+ self.__notices.extend(self.conn.notices)
+ self.conn.notices.clear()
# We also need to fetch notices before we return from function in case
# of any Exception, To avoid code duplication we will return after
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query
2018-02-26 14:19 pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-02-26 16:18 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
2018-02-26 16:23 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
2018-02-26 16:32 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-02-28 11:49 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
2018-02-28 15:25 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Joao De Almeida Pereira <[email protected]>
2018-03-01 05:56 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
@ 2018-03-01 15:21 ` Joao De Almeida Pereira <[email protected]>
2018-03-02 13:31 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
1 sibling, 1 reply; 17+ messages in thread
From: Joao De Almeida Pereira @ 2018-03-01 15:21 UTC (permalink / raw)
To: Khushboo Vashi <[email protected]>; +Cc: Dave Page <[email protected]>; Murtuza Zabuawala <[email protected]>; pgadmin-hackers
Hello Khushboo,
The patch runs successfully in our CI with all tests passing.
I see the test that you created, and I do not understand why we need to
create tests that do HTTP requests in order to check something that is
executed against the database. What I was talking about in my previous
email was having tests that tested the function by itself.
(Copied from: https://jfiaffe.files.wordpress.com/2014/09/tests-pyramid.png)
This is the Testing Pyramid, there are a bunch of different drawings of it
and ways to explain it, but in broad stokes what is means is that we should
have the majority of tests around a Unit (that are some disagreements in
the community of what a Unit is) and a very small amount of Manual testing.
What Unit usually means is piece of code, it can be a function, it can be a
class or it can even be a module, but is something self contained. In
pgAdmin's case the majority of our tests go around the Integration Layer
because we are using HTTP requests in order to executing queries in the
database, so basically we are doing tests end to end in the backend, and
the cost time.
I do not want to held this patch back because of this, and I say this
because I have minimal confidence with the tests that you created, that
they would catch the majority of the problems, and hope that the majority
of the code is exercised by it.
Nevertheless I would like to challenge all the Hackers to think about
testing in a different way. The tests in our code are used to give us
confidence that the work we did is working as expected, this also makes it
much easier to refactor out bad patterns or very complicated ones into
something simple. A signal that our code is more complicated then it should
is when we need to test some behavior and we end up with a Stubbing Hell or
we need to test it End 2 End because it is to hard to isolate the part we
want to test. In the other hand we should not test all functions and every
class, because we might be coupling our tests to much to the implementation
and that will have the contrary effect, and we will not be able to refactor
and simply our code.
Like everything in life there need to be a balance.
Thanks
Joao
On Thu, Mar 1, 2018 at 12:56 AM Khushboo Vashi <
[email protected]> wrote:
> Hi Joao,
>
> Thanks for reviewing.
>
> On Wed, Feb 28, 2018 at 8:55 PM, Joao De Almeida Pereira <
> [email protected]> wrote:
>
>> Hello Khushboo,
>> After reviewing the patch I have the gut feeling that we do not have
>> enough test coverage on this issue, specially due to the intricate while
>> loop and conditions around the polling.
>> I think that this deserve Unit tests around it, When I say Unit Test I am
>> not talking about executing queries against the database, but do some
>> stubbing of the database so that we can control the flow that we want.
>>
> You are right. It needs more unit testing. I have checked below scenarios:
> 1. Returns 2 notices with data output
> 2. Returns 1000 notices with data output
> 3. No notices with data output
>
> By running above, I have checked, each time returned notices are accurate,
> no old notices are getting appended, it does not affect with the amount of
> messages (few, none or more). Also, with the updated patch, I have made
> sure that all these queries run with the single transaction id (same
> connection).
>
> So, please let me know if you think I can add more things to this.
>
>>
>>
> It is a temptation to try to always do a Feature Test to test what we want
>> because it is "easier" to write and ultimately it is what users see, but
>> while 1 Feature Test runs we can run 200 Unit Tests that give us much more
>> confidence that the code is doing what we expect it to do.
>>
>> Right, so added regression tests instead of feature tests.
>
> This being said, I run the tests on the CI Pipeline and all tests pass.
>> Running pycodestyle fails due to some line sizes on the
>> psycopg2/__init__py. I believe that it is not what you changed, but since
>> you were changing the file it can be fixed it is just:
>>
>> pgadmin/utils/driver/psycopg2/__init__.py:1276: [E501] line too long (81
>> > 79 characters)
>> pgadmin/utils/driver/psycopg2/__init__.py:1277: [E501] line too long (91
>> > 79 characters)
>> pgadmin/utils/driver/psycopg2/__init__.py:1282: [E501] line too long (81
>> > 79 characters)
>> pgadmin/utils/driver/psycopg2/__init__.py:1283: [E501] line too long (91
>> > 79 characters)
>> 4 E501 line too long (81 > 79 characters)
>>
>> Fixed. Thanks for pointing out.
>
>>
>> Thanks
>> Joao
>>
>>
>> On Wed, Feb 28, 2018 at 6:49 AM Khushboo Vashi <
>> [email protected]> wrote:
>>
>>> On Mon, Feb 26, 2018 at 10:02 PM, Dave Page <[email protected]> wrote:
>>>
>>>> Argh, I ran some tests, but didn't spot any lost messages in the tests
>>>> I ran. I'll revert the patch.
>>>>
>>>> Khushboo;
>>>>
>>>> Please look at the following:
>>>>
>>>> - Fix the patch so it doesn't drop messages.
>>>>
>>> Fixed.
>>> By default, the notice attribute of the connection object of psycopg 2
>>> only stores 50 notices. Once it reaches to 50 it starts from 1 again.
>>> To fix this I have changed the notice attribute from list to deque to
>>> append more messages. Currently I have kept the maximum limit at a time of
>>> the notice attribute is 100000 (in a single poll).
>>>
>>>> - Add regression tests to make sure it doesn't break in the future.
>>>> This may require creating one or more functions the spew out a whole lot of
>>>> notices, and then running a couple of queries and checking the output.
>>>>
>>> Added. With this regression test, the current code is failing which has
>>> been taken care in this patch.
>>>
>>>> - Check the messages panel on the history tab. I just noticed it seems
>>>> to only be showing an even smaller subset of the messages.
>>>>
>>> Tested and no issues found.
>>>
>>>>
>>>>
>>> Thanks.
>>>>
>>>> On Mon, Feb 26, 2018 at 4:23 PM, Murtuza Zabuawala <
>>>> [email protected]> wrote:
>>>>
>>>>> Sent bit early,
>>>>>
>>>>> You can run 'VACUUM FULL VERBOSE' in query tool and verify the
>>>>> populated messages (pgAdmin3 vs. pgAdmin4).
>>>>>
>>>>>
>>>>> On Mon, Feb 26, 2018 at 9:48 PM, Murtuza Zabuawala <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Hi Khushboo/Dave,
>>>>>>
>>>>>> With given commit, I'm again seeing the issue raised in
>>>>>> https://redmine.postgresql.org/issues/1523 :(
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Murtuza Zabuawala
>>>>>> EnterpriseDB: http://www.enterprisedb.com
>>>>>> The Enterprise PostgreSQL Company
>>>>>>
>>>>>>
>>>>>> On Mon, Feb 26, 2018 at 7:49 PM, Dave Page <[email protected]> wrote:
>>>>>>
>>>>>>> Ensure we pick up the messages from the current query and not a
>>>>>>> previous one. Fixes #3094
>>>>>>>
>>>>>>> Branch
>>>>>>> ------
>>>>>>> master
>>>>>>>
>>>>>>> Details
>>>>>>> -------
>>>>>>>
>>>>>>> https://git.postgresql.org/gitweb?p=pgadmin4.git;a=commitdiff;h=08b3ccc01a4d57e8ea3657f8882a53dcd1b9...
>>>>>>> Author: Khushboo Vashi <[email protected]>
>>>>>>>
>>>>>>> Modified Files
>>>>>>> --------------
>>>>>>> web/pgadmin/utils/driver/abstract.py | 1 +
>>>>>>> web/pgadmin/utils/driver/psycopg2/__init__.py | 64
>>>>>>> +++++++++------------------
>>>>>>> 2 files changed, 21 insertions(+), 44 deletions(-)
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Dave Page
>>>> Blog: http://pgsnake.blogspot.com
>>>> Twitter: @pgsnake
>>>>
>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>> The Enterprise PostgreSQL Company
>>>>
>>>
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query
2018-02-26 14:19 pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-02-26 16:18 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
2018-02-26 16:23 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
2018-02-26 16:32 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-02-28 11:49 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
2018-02-28 15:25 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Joao De Almeida Pereira <[email protected]>
2018-03-01 05:56 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
2018-03-01 15:21 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Joao De Almeida Pereira <[email protected]>
@ 2018-03-02 13:31 ` Dave Page <[email protected]>
2018-03-02 18:42 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Joao De Almeida Pereira <[email protected]>
0 siblings, 1 reply; 17+ messages in thread
From: Dave Page @ 2018-03-02 13:31 UTC (permalink / raw)
To: Joao De Almeida Pereira <[email protected]>; +Cc: Khushboo Vashi <[email protected]>; Murtuza Zabuawala <[email protected]>; pgadmin-hackers
On Thu, Mar 1, 2018 at 3:21 PM, Joao De Almeida Pereira <
[email protected]> wrote:
> Hello Khushboo,
> The patch runs successfully in our CI with all tests passing.
>
> I see the test that you created, and I do not understand why we need to
> create tests that do HTTP requests in order to check something that is
> executed against the database. What I was talking about in my previous
> email was having tests that tested the function by itself.
>
>
> (Copied from: https://jfiaffe.files.wordpress.com/2014/09/tests-
> pyramid.png)
>
> This is the Testing Pyramid, there are a bunch of different drawings of it
> and ways to explain it, but in broad stokes what is means is that we should
> have the majority of tests around a Unit (that are some disagreements in
> the community of what a Unit is) and a very small amount of Manual testing.
> What Unit usually means is piece of code, it can be a function, it can be a
> class or it can even be a module, but is something self contained. In
> pgAdmin's case the majority of our tests go around the Integration Layer
> because we are using HTTP requests in order to executing queries in the
> database, so basically we are doing tests end to end in the backend, and
> the cost time.
>
> I do not want to held this patch back because of this, and I say this
> because I have minimal confidence with the tests that you created, that
> they would catch the majority of the problems, and hope that the majority
> of the code is exercised by it.
>
> Nevertheless I would like to challenge all the Hackers to think about
> testing in a different way. The tests in our code are used to give us
> confidence that the work we did is working as expected, this also makes it
> much easier to refactor out bad patterns or very complicated ones into
> something simple. A signal that our code is more complicated then it should
> is when we need to test some behavior and we end up with a Stubbing Hell or
> we need to test it End 2 End because it is to hard to isolate the part we
> want to test. In the other hand we should not test all functions and every
> class, because we might be coupling our tests to much to the implementation
> and that will have the contrary effect, and we will not be able to refactor
> and simply our code.
> Like everything in life there need to be a balance.
>
I think that is very good advice, and I would also like to encourage all
the developers to think this way. However, I would also caution against
underestimating the importance of our feature tests. It is obviously
important to have confidence that our code is robust and functions
correctly at the unit level, but that doesn't mean that it all works
together as expected to provide the user functionality we desire. I think
the feature tests are critical in this regard; they protect us against
class of bug that might otherwise be missed without manual testing that we
strive to eliminate entirely. For the EDBers on the team, think of the
feature tests in terms of what we normally call integration testing;
ensuring that not only do the individual pieces work as they should, but
that they work together as they should.
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query
2018-02-26 14:19 pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-02-26 16:18 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
2018-02-26 16:23 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
2018-02-26 16:32 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-02-28 11:49 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
2018-02-28 15:25 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Joao De Almeida Pereira <[email protected]>
2018-03-01 05:56 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
2018-03-01 15:21 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Joao De Almeida Pereira <[email protected]>
2018-03-02 13:31 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
@ 2018-03-02 18:42 ` Joao De Almeida Pereira <[email protected]>
0 siblings, 0 replies; 17+ messages in thread
From: Joao De Almeida Pereira @ 2018-03-02 18:42 UTC (permalink / raw)
To: Dave Page <[email protected]>; +Cc: Khushboo Vashi <[email protected]>; Murtuza Zabuawala <[email protected]>; pgadmin-hackers
Hello Dave,
Very well said, I didn't intended to say that Feature Tests are not
important, what I was saying is that when we can do faster and more
thorough tests we should take it. All the tests that we do are important to
ensure that the software we produce is of good quality.
Also the higher you go in the pyramid the more volatile the test are and
more variables need to be taken into account when doing them.
Thanks
Joao
On Fri, Mar 2, 2018 at 8:31 AM Dave Page <[email protected]> wrote:
> On Thu, Mar 1, 2018 at 3:21 PM, Joao De Almeida Pereira <
> [email protected]> wrote:
>
>> Hello Khushboo,
>>
> The patch runs successfully in our CI with all tests passing.
>>
>> I see the test that you created, and I do not understand why we need to
>> create tests that do HTTP requests in order to check something that is
>> executed against the database. What I was talking about in my previous
>> email was having tests that tested the function by itself.
>>
>>
>> (Copied from:
>> https://jfiaffe.files.wordpress.com/2014/09/tests-pyramid.png)
>>
>> This is the Testing Pyramid, there are a bunch of different drawings of
>> it and ways to explain it, but in broad stokes what is means is that we
>> should have the majority of tests around a Unit (that are some
>> disagreements in the community of what a Unit is) and a very small amount
>> of Manual testing. What Unit usually means is piece of code, it can be a
>> function, it can be a class or it can even be a module, but is something
>> self contained. In pgAdmin's case the majority of our tests go around the
>> Integration Layer because we are using HTTP requests in order to executing
>> queries in the database, so basically we are doing tests end to end in the
>> backend, and the cost time.
>>
>> I do not want to held this patch back because of this, and I say this
>> because I have minimal confidence with the tests that you created, that
>> they would catch the majority of the problems, and hope that the majority
>> of the code is exercised by it.
>>
>> Nevertheless I would like to challenge all the Hackers to think about
>> testing in a different way. The tests in our code are used to give us
>> confidence that the work we did is working as expected, this also makes it
>> much easier to refactor out bad patterns or very complicated ones into
>> something simple. A signal that our code is more complicated then it should
>> is when we need to test some behavior and we end up with a Stubbing Hell or
>> we need to test it End 2 End because it is to hard to isolate the part we
>> want to test. In the other hand we should not test all functions and every
>> class, because we might be coupling our tests to much to the implementation
>> and that will have the contrary effect, and we will not be able to refactor
>> and simply our code.
>> Like everything in life there need to be a balance.
>>
>
> I think that is very good advice, and I would also like to encourage all
> the developers to think this way. However, I would also caution against
> underestimating the importance of our feature tests. It is obviously
> important to have confidence that our code is robust and functions
> correctly at the unit level, but that doesn't mean that it all works
> together as expected to provide the user functionality we desire. I think
> the feature tests are critical in this regard; they protect us against
> class of bug that might otherwise be missed without manual testing that we
> strive to eliminate entirely. For the EDBers on the team, think of the
> feature tests in terms of what we normally call integration testing;
> ensuring that not only do the individual pieces work as they should, but
> that they work together as they should.
>
> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query
2018-02-26 14:19 pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-02-26 16:18 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
2018-02-26 16:23 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
2018-02-26 16:32 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-02-28 11:49 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
2018-02-28 15:25 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Joao De Almeida Pereira <[email protected]>
2018-03-01 05:56 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
@ 2018-03-02 13:25 ` Dave Page <[email protected]>
2018-03-05 04:51 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
1 sibling, 1 reply; 17+ messages in thread
From: Dave Page @ 2018-03-02 13:25 UTC (permalink / raw)
To: Khushboo Vashi <[email protected]>; +Cc: Joao De Almeida Pereira <[email protected]>; Murtuza Zabuawala <[email protected]>; pgadmin-hackers
Could you rebase this please? It no longer applies.
Thanks.
On Thu, Mar 1, 2018 at 5:56 AM, Khushboo Vashi <
[email protected]> wrote:
> Hi Joao,
>
> Thanks for reviewing.
>
> On Wed, Feb 28, 2018 at 8:55 PM, Joao De Almeida Pereira <
> [email protected]> wrote:
>
>> Hello Khushboo,
>> After reviewing the patch I have the gut feeling that we do not have
>> enough test coverage on this issue, specially due to the intricate while
>> loop and conditions around the polling.
>> I think that this deserve Unit tests around it, When I say Unit Test I am
>> not talking about executing queries against the database, but do some
>> stubbing of the database so that we can control the flow that we want.
>>
> You are right. It needs more unit testing. I have checked below scenarios:
> 1. Returns 2 notices with data output
> 2. Returns 1000 notices with data output
> 3. No notices with data output
>
> By running above, I have checked, each time returned notices are accurate,
> no old notices are getting appended, it does not affect with the amount of
> messages (few, none or more). Also, with the updated patch, I have made
> sure that all these queries run with the single transaction id (same
> connection).
>
> So, please let me know if you think I can add more things to this.
>
>>
>>
> It is a temptation to try to always do a Feature Test to test what we want
>> because it is "easier" to write and ultimately it is what users see, but
>> while 1 Feature Test runs we can run 200 Unit Tests that give us much more
>> confidence that the code is doing what we expect it to do.
>>
>> Right, so added regression tests instead of feature tests.
>
> This being said, I run the tests on the CI Pipeline and all tests pass.
>> Running pycodestyle fails due to some line sizes on the
>> psycopg2/__init__py. I believe that it is not what you changed, but since
>> you were changing the file it can be fixed it is just:
>>
>> pgadmin/utils/driver/psycopg2/__init__.py:1276: [E501] line too long (81
>> > 79 characters)
>> pgadmin/utils/driver/psycopg2/__init__.py:1277: [E501] line too long (91
>> > 79 characters)
>> pgadmin/utils/driver/psycopg2/__init__.py:1282: [E501] line too long (81
>> > 79 characters)
>> pgadmin/utils/driver/psycopg2/__init__.py:1283: [E501] line too long (91
>> > 79 characters)
>> 4 E501 line too long (81 > 79 characters)
>>
>> Fixed. Thanks for pointing out.
>
>>
>> Thanks
>> Joao
>>
>>
>> On Wed, Feb 28, 2018 at 6:49 AM Khushboo Vashi <
>> [email protected]> wrote:
>>
>>> On Mon, Feb 26, 2018 at 10:02 PM, Dave Page <[email protected]> wrote:
>>>
>>>> Argh, I ran some tests, but didn't spot any lost messages in the tests
>>>> I ran. I'll revert the patch.
>>>>
>>>> Khushboo;
>>>>
>>>> Please look at the following:
>>>>
>>>> - Fix the patch so it doesn't drop messages.
>>>>
>>> Fixed.
>>> By default, the notice attribute of the connection object of psycopg 2
>>> only stores 50 notices. Once it reaches to 50 it starts from 1 again.
>>> To fix this I have changed the notice attribute from list to deque to
>>> append more messages. Currently I have kept the maximum limit at a time of
>>> the notice attribute is 100000 (in a single poll).
>>>
>>>> - Add regression tests to make sure it doesn't break in the future.
>>>> This may require creating one or more functions the spew out a whole lot of
>>>> notices, and then running a couple of queries and checking the output.
>>>>
>>> Added. With this regression test, the current code is failing which has
>>> been taken care in this patch.
>>>
>>>> - Check the messages panel on the history tab. I just noticed it seems
>>>> to only be showing an even smaller subset of the messages.
>>>>
>>> Tested and no issues found.
>>>
>>>>
>>>>
>>> Thanks.
>>>>
>>>> On Mon, Feb 26, 2018 at 4:23 PM, Murtuza Zabuawala <
>>>> [email protected]> wrote:
>>>>
>>>>> Sent bit early,
>>>>>
>>>>> You can run 'VACUUM FULL VERBOSE' in query tool and verify the
>>>>> populated messages (pgAdmin3 vs. pgAdmin4).
>>>>>
>>>>>
>>>>> On Mon, Feb 26, 2018 at 9:48 PM, Murtuza Zabuawala <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Hi Khushboo/Dave,
>>>>>>
>>>>>> With given commit, I'm again seeing the issue raised in
>>>>>> https://redmine.postgresql.org/issues/1523 :(
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Murtuza Zabuawala
>>>>>> EnterpriseDB: http://www.enterprisedb.com
>>>>>> The Enterprise PostgreSQL Company
>>>>>>
>>>>>>
>>>>>> On Mon, Feb 26, 2018 at 7:49 PM, Dave Page <[email protected]> wrote:
>>>>>>
>>>>>>> Ensure we pick up the messages from the current query and not a
>>>>>>> previous one. Fixes #3094
>>>>>>>
>>>>>>> Branch
>>>>>>> ------
>>>>>>> master
>>>>>>>
>>>>>>> Details
>>>>>>> -------
>>>>>>> https://git.postgresql.org/gitweb?p=pgadmin4.git;a=commitdif
>>>>>>> f;h=08b3ccc01a4d57e8ea3657f8882a53dcd1b99386
>>>>>>> Author: Khushboo Vashi <[email protected]>
>>>>>>>
>>>>>>> Modified Files
>>>>>>> --------------
>>>>>>> web/pgadmin/utils/driver/abstract.py | 1 +
>>>>>>> web/pgadmin/utils/driver/psycopg2/__init__.py | 64
>>>>>>> +++++++++------------------
>>>>>>> 2 files changed, 21 insertions(+), 44 deletions(-)
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Dave Page
>>>> Blog: http://pgsnake.blogspot.com
>>>> Twitter: @pgsnake
>>>>
>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>> The Enterprise PostgreSQL Company
>>>>
>>>
>
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query
2018-02-26 14:19 pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-02-26 16:18 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
2018-02-26 16:23 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
2018-02-26 16:32 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-02-28 11:49 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
2018-02-28 15:25 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Joao De Almeida Pereira <[email protected]>
2018-03-01 05:56 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
2018-03-02 13:25 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
@ 2018-03-05 04:51 ` Khushboo Vashi <[email protected]>
2018-03-05 15:12 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Joao De Almeida Pereira <[email protected]>
0 siblings, 1 reply; 17+ messages in thread
From: Khushboo Vashi @ 2018-03-05 04:51 UTC (permalink / raw)
To: Dave Page <[email protected]>; +Cc: Joao De Almeida Pereira <[email protected]>; Murtuza Zabuawala <[email protected]>; pgadmin-hackers
On Fri, Mar 2, 2018 at 6:55 PM, Dave Page <[email protected]> wrote:
> Could you rebase this please? It no longer applies.
>
> Please find the attached updated patch.
> Thanks.
>
> On Thu, Mar 1, 2018 at 5:56 AM, Khushboo Vashi <
> [email protected]> wrote:
>
>> Hi Joao,
>>
>> Thanks for reviewing.
>>
>> On Wed, Feb 28, 2018 at 8:55 PM, Joao De Almeida Pereira <
>> [email protected]> wrote:
>>
>>> Hello Khushboo,
>>> After reviewing the patch I have the gut feeling that we do not have
>>> enough test coverage on this issue, specially due to the intricate while
>>> loop and conditions around the polling.
>>> I think that this deserve Unit tests around it, When I say Unit Test I
>>> am not talking about executing queries against the database, but do some
>>> stubbing of the database so that we can control the flow that we want.
>>>
>> You are right. It needs more unit testing. I have checked below scenarios:
>> 1. Returns 2 notices with data output
>> 2. Returns 1000 notices with data output
>> 3. No notices with data output
>>
>> By running above, I have checked, each time returned notices are
>> accurate, no old notices are getting appended, it does not affect with the
>> amount of messages (few, none or more). Also, with the updated patch, I
>> have made sure that all these queries run with the single transaction id
>> (same connection).
>>
>> So, please let me know if you think I can add more things to this.
>>
>>>
>>>
>> It is a temptation to try to always do a Feature Test to test what we
>>> want because it is "easier" to write and ultimately it is what users see,
>>> but while 1 Feature Test runs we can run 200 Unit Tests that give us much
>>> more confidence that the code is doing what we expect it to do.
>>>
>>> Right, so added regression tests instead of feature tests.
>>
>> This being said, I run the tests on the CI Pipeline and all tests pass.
>>> Running pycodestyle fails due to some line sizes on the
>>> psycopg2/__init__py. I believe that it is not what you changed, but since
>>> you were changing the file it can be fixed it is just:
>>>
>>> pgadmin/utils/driver/psycopg2/__init__.py:1276: [E501] line too long
>>> (81 > 79 characters)
>>> pgadmin/utils/driver/psycopg2/__init__.py:1277: [E501] line too long
>>> (91 > 79 characters)
>>> pgadmin/utils/driver/psycopg2/__init__.py:1282: [E501] line too long
>>> (81 > 79 characters)
>>> pgadmin/utils/driver/psycopg2/__init__.py:1283: [E501] line too long
>>> (91 > 79 characters)
>>> 4 E501 line too long (81 > 79 characters)
>>>
>>> Fixed. Thanks for pointing out.
>>
>>>
>>> Thanks
>>> Joao
>>>
>>>
>>> On Wed, Feb 28, 2018 at 6:49 AM Khushboo Vashi <
>>> [email protected]> wrote:
>>>
>>>> On Mon, Feb 26, 2018 at 10:02 PM, Dave Page <[email protected]> wrote:
>>>>
>>>>> Argh, I ran some tests, but didn't spot any lost messages in the tests
>>>>> I ran. I'll revert the patch.
>>>>>
>>>>> Khushboo;
>>>>>
>>>>> Please look at the following:
>>>>>
>>>>> - Fix the patch so it doesn't drop messages.
>>>>>
>>>> Fixed.
>>>> By default, the notice attribute of the connection object of psycopg 2
>>>> only stores 50 notices. Once it reaches to 50 it starts from 1 again.
>>>> To fix this I have changed the notice attribute from list to deque to
>>>> append more messages. Currently I have kept the maximum limit at a time of
>>>> the notice attribute is 100000 (in a single poll).
>>>>
>>>>> - Add regression tests to make sure it doesn't break in the future.
>>>>> This may require creating one or more functions the spew out a whole lot of
>>>>> notices, and then running a couple of queries and checking the output.
>>>>>
>>>> Added. With this regression test, the current code is failing which has
>>>> been taken care in this patch.
>>>>
>>>>> - Check the messages panel on the history tab. I just noticed it seems
>>>>> to only be showing an even smaller subset of the messages.
>>>>>
>>>> Tested and no issues found.
>>>>
>>>>>
>>>>>
>>>> Thanks.
>>>>>
>>>>> On Mon, Feb 26, 2018 at 4:23 PM, Murtuza Zabuawala <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Sent bit early,
>>>>>>
>>>>>> You can run 'VACUUM FULL VERBOSE' in query tool and verify the
>>>>>> populated messages (pgAdmin3 vs. pgAdmin4).
>>>>>>
>>>>>>
>>>>>> On Mon, Feb 26, 2018 at 9:48 PM, Murtuza Zabuawala <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> Hi Khushboo/Dave,
>>>>>>>
>>>>>>> With given commit, I'm again seeing the issue raised in
>>>>>>> https://redmine.postgresql.org/issues/1523 :(
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Murtuza Zabuawala
>>>>>>> EnterpriseDB: http://www.enterprisedb.com
>>>>>>> The Enterprise PostgreSQL Company
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Feb 26, 2018 at 7:49 PM, Dave Page <[email protected]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Ensure we pick up the messages from the current query and not a
>>>>>>>> previous one. Fixes #3094
>>>>>>>>
>>>>>>>> Branch
>>>>>>>> ------
>>>>>>>> master
>>>>>>>>
>>>>>>>> Details
>>>>>>>> -------
>>>>>>>> https://git.postgresql.org/gitweb?p=pgadmin4.git;a=commitdif
>>>>>>>> f;h=08b3ccc01a4d57e8ea3657f8882a53dcd1b99386
>>>>>>>> Author: Khushboo Vashi <[email protected]>
>>>>>>>>
>>>>>>>> Modified Files
>>>>>>>> --------------
>>>>>>>> web/pgadmin/utils/driver/abstract.py | 1 +
>>>>>>>> web/pgadmin/utils/driver/psycopg2/__init__.py | 64
>>>>>>>> +++++++++------------------
>>>>>>>> 2 files changed, 21 insertions(+), 44 deletions(-)
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Dave Page
>>>>> Blog: http://pgsnake.blogspot.com
>>>>> Twitter: @pgsnake
>>>>>
>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>> The Enterprise PostgreSQL Company
>>>>>
>>>>
>>
>
>
> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
Attachments:
[text/x-patch] RM_3094_ver3.patch (11.1K, 3-RM_3094_ver3.patch)
download | inline diff:
diff --git a/web/pgadmin/feature_tests/keyboard_shortcut_test.py b/web/pgadmin/feature_tests/keyboard_shortcut_test.py
index 443eff6..bbae194 100644
--- a/web/pgadmin/feature_tests/keyboard_shortcut_test.py
+++ b/web/pgadmin/feature_tests/keyboard_shortcut_test.py
@@ -62,7 +62,9 @@ class KeyboardShortcutFeatureTest(BaseFeatureTest):
).key_down(
key_combo[2]
).key_up(
- Keys.ALT
+ key_combo[0]
+ ).key_up(
+ key_combo[1]
).perform()
print("Executing shortcut: " + self.new_shortcuts[s]['locator'] +
diff --git a/web/pgadmin/tools/sqleditor/tests/test_poll_query_tool.py b/web/pgadmin/tools/sqleditor/tests/test_poll_query_tool.py
new file mode 100644
index 0000000..e47129b
--- /dev/null
+++ b/web/pgadmin/tools/sqleditor/tests/test_poll_query_tool.py
@@ -0,0 +1,112 @@
+##########################################################################
+#
+# pgAdmin 4 - PostgreSQL Tools
+#
+# Copyright (C) 2013 - 2018, The pgAdmin Development Team
+# This software is released under the PostgreSQL Licence
+#
+##########################################################################
+
+import json
+
+from pgadmin.browser.server_groups.servers.databases.tests import utils as \
+ database_utils
+from pgadmin.utils.route import BaseTestGenerator
+from regression import parent_node_dict
+from regression.python_test_utils import test_utils as utils
+
+
+class TestPollQueryTool(BaseTestGenerator):
+ """ This class will test the query tool polling. """
+ scenarios = [
+ ('When query tool polling returns messages with result data-set',
+ dict(
+ sql=[
+ """
+DROP TABLE IF EXISTS test_for_notices;
+
+DO $$
+BEGIN
+ RAISE NOTICE 'Hello, world!';
+END $$;
+
+SELECT 'CHECKING POLLING';
+""",
+ """
+DO $$
+BEGIN
+ FOR i in 1..1000 LOOP
+ RAISE NOTICE 'Count is %', i;
+ END LOOP;
+END $$;
+
+SELECT 'CHECKING POLLING FOR LONG MESSAGES';
+""",
+ "SELECT 'CHECKING POLLING WITHOUT MESSAGES';"
+ ],
+ expected_message=['NOTICE: table "test_for_notices" ' +
+ """does not exist, skipping
+NOTICE: Hello, world!
+""",
+ "\n".join(["NOTICE: Count is {0}".format(i)
+ for i in range(1, 1001)]) + "\n",
+ None],
+ expected_result=['CHECKING POLLING',
+ 'CHECKING POLLING FOR LONG MESSAGES',
+ 'CHECKING POLLING WITHOUT MESSAGES'],
+ print_messages=['2 NOTICES WITH DATASET',
+ '1000 NOTICES WITH DATASET',
+ 'NO NOTICE WITH DATASET'
+ ]
+ ))
+ ]
+
+ def runTest(self):
+ """ This function will check messages return by query tool polling. """
+ database_info = parent_node_dict["database"][-1]
+ self.server_id = database_info["server_id"]
+
+ self.db_id = database_info["db_id"]
+ db_con = database_utils.connect_database(self,
+ utils.SERVER_GROUP,
+ self.server_id,
+ self.db_id)
+ if not db_con["info"] == "Database connected.":
+ raise Exception("Could not connect to the database.")
+
+ # Initialize query tool
+ url = '/datagrid/initialize/query_tool/{0}/{1}/{2}'.format(
+ utils.SERVER_GROUP, self.server_id, self.db_id)
+ response = self.tester.post(url)
+ self.assertEquals(response.status_code, 200)
+
+ response_data = json.loads(response.data.decode('utf-8'))
+ self.trans_id = response_data['data']['gridTransId']
+
+ cnt = 0
+ for s in self.sql:
+ print("Executing and polling with: " + self.print_messages[cnt])
+ # Start query tool transaction
+ url = '/sqleditor/query_tool/start/{0}'.format(self.trans_id)
+ response = self.tester.post(url, data=json.dumps({"sql": s}),
+ content_type='html/json')
+
+ self.assertEquals(response.status_code, 200)
+
+ # Query tool polling
+ url = '/sqleditor/poll/{0}'.format(self.trans_id)
+ response = self.tester.get(url)
+ self.assertEquals(response.status_code, 200)
+ response_data = json.loads(response.data.decode('utf-8'))
+
+ # Check the returned messages
+ self.assertEquals(self.expected_message[cnt],
+ response_data['data']['additional_messages'])
+ # Check the output
+ self.assertEquals(self.expected_result[cnt],
+ response_data['data']['result'][0][0])
+
+ cnt += 1
+
+ # Disconnect the database
+ database_utils.disconnect_database(self, self.server_id, self.db_id)
diff --git a/web/pgadmin/utils/driver/abstract.py b/web/pgadmin/utils/driver/abstract.py
index 32e1c97..271bfec 100644
--- a/web/pgadmin/utils/driver/abstract.py
+++ b/web/pgadmin/utils/driver/abstract.py
@@ -168,6 +168,8 @@ class BaseConnection(object):
ASYNC_WRITE_TIMEOUT = 3
ASYNC_NOT_CONNECTED = 4
ASYNC_EXECUTION_ABORTED = 5
+ ASYNC_TIMEOUT = 0.2
+ ASYNC_NOTICE_MAXLENGTH = 100000
@abstractmethod
def connect(self, **kwargs):
diff --git a/web/pgadmin/utils/driver/psycopg2/__init__.py b/web/pgadmin/utils/driver/psycopg2/__init__.py
index 81442e4..941a694 100644
--- a/web/pgadmin/utils/driver/psycopg2/__init__.py
+++ b/web/pgadmin/utils/driver/psycopg2/__init__.py
@@ -37,6 +37,7 @@ from .cursor import DictCursor
from .typecast import register_global_typecasters, \
register_string_typecasters, register_binary_typecasters, \
register_array_to_string_typecasters, ALL_JSON_TYPES
+from collections import deque
if sys.version_info < (3,):
@@ -110,7 +111,7 @@ class Connection(BaseConnection):
- This method is used to wait for asynchronous connection. This is a
blocking call.
- * _wait_timeout(conn, time)
+ * _wait_timeout(conn)
- This method is used to wait for asynchronous connection with timeout.
This is a non blocking call.
@@ -310,6 +311,9 @@ class Connection(BaseConnection):
)
return False, msg
+ # Overwrite connection notice attr to support
+ # more than 50 notices at a time
+ pg_conn.notices = deque([], self.ASYNC_NOTICE_MAXLENGTH)
self.conn = pg_conn
self.wasConnected = True
try:
@@ -1208,6 +1212,7 @@ Failed to reset the connection to the server due to following error:
)
return False, msg
+ pg_conn.notices = deque([], self.ASYNC_NOTICE_MAXLENGTH)
self.conn = pg_conn
self.__backend_pid = pg_conn.get_backend_pid()
@@ -1261,51 +1266,31 @@ Failed to reset the connection to the server due to following error:
Args:
conn: connection object
- time: wait time
"""
-
- state = conn.poll()
- if state == psycopg2.extensions.POLL_OK:
- return self.ASYNC_OK
- elif state == psycopg2.extensions.POLL_WRITE:
- # Wait for the given time and then check the return status
- # If three empty lists are returned then the time-out is reached.
- timeout_status = select.select([], [conn.fileno()], [], 0)
- if timeout_status == ([], [], []):
- return self.ASYNC_WRITE_TIMEOUT
-
- # poll again to check the state if it is still POLL_WRITE
- # then return ASYNC_WRITE_TIMEOUT else return ASYNC_OK.
+ while 1:
state = conn.poll()
- if state == psycopg2.extensions.POLL_WRITE:
- return self.ASYNC_WRITE_TIMEOUT
- return self.ASYNC_OK
- elif state == psycopg2.extensions.POLL_READ:
- # Wait for the given time and then check the return status
- # If three empty lists are returned then the time-out is reached.
- timeout_status = select.select([conn.fileno()], [], [], 0)
- if timeout_status == ([], [], []):
- return self.ASYNC_READ_TIMEOUT
-
- # select.select timeout option works only if we provide
- # empty [] [] [] file descriptor in select.select() function
- # and that also works only on UNIX based system, it do not support
- # Windows Hence we have wrote our own pooling mechanism to read
- # data fast each call conn.poll() reads chunks of data from
- # connection object more we poll more we read data from connection
- cnt = 0
- while cnt < 1000:
- # poll again to check the state if it is still POLL_READ
- # then return ASYNC_READ_TIMEOUT else return ASYNC_OK.
- state = conn.poll()
- if state == psycopg2.extensions.POLL_OK:
- return self.ASYNC_OK
- cnt += 1
- return self.ASYNC_READ_TIMEOUT
- else:
- raise psycopg2.OperationalError(
- "poll() returned %s from _wait_timeout function" % state
- )
+ if state == psycopg2.extensions.POLL_OK:
+ return self.ASYNC_OK
+ elif state == psycopg2.extensions.POLL_WRITE:
+ # Wait for the given time and then check the return status
+ # If three empty lists are returned then the time-out is
+ # reached.
+ timeout_status = select.select([], [conn.fileno()], [],
+ self.ASYNC_TIMEOUT)
+ if timeout_status == ([], [], []):
+ return self.ASYNC_WRITE_TIMEOUT
+ elif state == psycopg2.extensions.POLL_READ:
+ # Wait for the given time and then check the return status
+ # If three empty lists are returned then the time-out is
+ # reached.
+ timeout_status = select.select([conn.fileno()], [], [],
+ self.ASYNC_TIMEOUT)
+ if timeout_status == ([], [], []):
+ return self.ASYNC_READ_TIMEOUT
+ else:
+ raise psycopg2.OperationalError(
+ "poll() returned %s from _wait_timeout function" % state
+ )
def poll(self, formatted_exception_msg=False, no_result=False):
"""
@@ -1347,8 +1332,8 @@ Failed to reset the connection to the server due to following error:
is_error = True
if self.conn.notices and self.__notices is not None:
- while self.conn.notices:
- self.__notices.append(self.conn.notices.pop(0)[:])
+ self.__notices.extend(self.conn.notices)
+ self.conn.notices.clear()
# We also need to fetch notices before we return from function in case
# of any Exception, To avoid code duplication we will return after
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query
2018-02-26 14:19 pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-02-26 16:18 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
2018-02-26 16:23 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
2018-02-26 16:32 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-02-28 11:49 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
2018-02-28 15:25 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Joao De Almeida Pereira <[email protected]>
2018-03-01 05:56 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
2018-03-02 13:25 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-03-05 04:51 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
@ 2018-03-05 15:12 ` Joao De Almeida Pereira <[email protected]>
2018-03-06 10:09 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
0 siblings, 1 reply; 17+ messages in thread
From: Joao De Almeida Pereira @ 2018-03-05 15:12 UTC (permalink / raw)
To: Khushboo Vashi <[email protected]>; +Cc: Dave Page <[email protected]>; Murtuza Zabuawala <[email protected]>; pgadmin-hackers
Hello Khushboo,
Looks like we are almost doen, just missing one PEP-8 issue:
→ pycodestyle --config=.pycodestyle pgadmin/tools
pgadmin/tools/sqleditor/tests/test_poll_query_tool.py:46: [E126]
continuation line over-indented for hanging indent
1 E126 continuation line over-indented for hanging indent
1
The tests run successfully in our CI pipeline.
Thanks
Joao
On Sun, Mar 4, 2018 at 11:51 PM Khushboo Vashi <
[email protected]> wrote:
> On Fri, Mar 2, 2018 at 6:55 PM, Dave Page <[email protected]> wrote:
>
>> Could you rebase this please? It no longer applies.
>>
>> Please find the attached updated patch.
>
>> Thanks.
>>
>> On Thu, Mar 1, 2018 at 5:56 AM, Khushboo Vashi <
>> [email protected]> wrote:
>>
>>> Hi Joao,
>>>
>>> Thanks for reviewing.
>>>
>>> On Wed, Feb 28, 2018 at 8:55 PM, Joao De Almeida Pereira <
>>> [email protected]> wrote:
>>>
>>>> Hello Khushboo,
>>>> After reviewing the patch I have the gut feeling that we do not have
>>>> enough test coverage on this issue, specially due to the intricate while
>>>> loop and conditions around the polling.
>>>> I think that this deserve Unit tests around it, When I say Unit Test I
>>>> am not talking about executing queries against the database, but do some
>>>> stubbing of the database so that we can control the flow that we want.
>>>>
>>> You are right. It needs more unit testing. I have checked below
>>> scenarios:
>>> 1. Returns 2 notices with data output
>>> 2. Returns 1000 notices with data output
>>> 3. No notices with data output
>>>
>>> By running above, I have checked, each time returned notices are
>>> accurate, no old notices are getting appended, it does not affect with the
>>> amount of messages (few, none or more). Also, with the updated patch, I
>>> have made sure that all these queries run with the single transaction id
>>> (same connection).
>>>
>>> So, please let me know if you think I can add more things to this.
>>>
>>>>
>>>>
>>> It is a temptation to try to always do a Feature Test to test what we
>>>> want because it is "easier" to write and ultimately it is what users see,
>>>> but while 1 Feature Test runs we can run 200 Unit Tests that give us much
>>>> more confidence that the code is doing what we expect it to do.
>>>>
>>>> Right, so added regression tests instead of feature tests.
>>>
>>> This being said, I run the tests on the CI Pipeline and all tests pass.
>>>> Running pycodestyle fails due to some line sizes on the
>>>> psycopg2/__init__py. I believe that it is not what you changed, but since
>>>> you were changing the file it can be fixed it is just:
>>>>
>>>> pgadmin/utils/driver/psycopg2/__init__.py:1276: [E501] line too long
>>>> (81 > 79 characters)
>>>> pgadmin/utils/driver/psycopg2/__init__.py:1277: [E501] line too long
>>>> (91 > 79 characters)
>>>> pgadmin/utils/driver/psycopg2/__init__.py:1282: [E501] line too long
>>>> (81 > 79 characters)
>>>> pgadmin/utils/driver/psycopg2/__init__.py:1283: [E501] line too long
>>>> (91 > 79 characters)
>>>> 4 E501 line too long (81 > 79 characters)
>>>>
>>>> Fixed. Thanks for pointing out.
>>>
>>>>
>>>> Thanks
>>>> Joao
>>>>
>>>>
>>>> On Wed, Feb 28, 2018 at 6:49 AM Khushboo Vashi <
>>>> [email protected]> wrote:
>>>>
>>>>> On Mon, Feb 26, 2018 at 10:02 PM, Dave Page <[email protected]> wrote:
>>>>>
>>>>>> Argh, I ran some tests, but didn't spot any lost messages in the
>>>>>> tests I ran. I'll revert the patch.
>>>>>>
>>>>>> Khushboo;
>>>>>>
>>>>>> Please look at the following:
>>>>>>
>>>>>> - Fix the patch so it doesn't drop messages.
>>>>>>
>>>>> Fixed.
>>>>> By default, the notice attribute of the connection object of psycopg 2
>>>>> only stores 50 notices. Once it reaches to 50 it starts from 1 again.
>>>>> To fix this I have changed the notice attribute from list to deque to
>>>>> append more messages. Currently I have kept the maximum limit at a time of
>>>>> the notice attribute is 100000 (in a single poll).
>>>>>
>>>>>> - Add regression tests to make sure it doesn't break in the future.
>>>>>> This may require creating one or more functions the spew out a whole lot of
>>>>>> notices, and then running a couple of queries and checking the output.
>>>>>>
>>>>> Added. With this regression test, the current code is failing which
>>>>> has been taken care in this patch.
>>>>>
>>>>>> - Check the messages panel on the history tab. I just noticed it
>>>>>> seems to only be showing an even smaller subset of the messages.
>>>>>>
>>>>> Tested and no issues found.
>>>>>
>>>>>>
>>>>>>
>>>>> Thanks.
>>>>>>
>>>>>> On Mon, Feb 26, 2018 at 4:23 PM, Murtuza Zabuawala <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> Sent bit early,
>>>>>>>
>>>>>>> You can run 'VACUUM FULL VERBOSE' in query tool and verify the
>>>>>>> populated messages (pgAdmin3 vs. pgAdmin4).
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Feb 26, 2018 at 9:48 PM, Murtuza Zabuawala <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> Hi Khushboo/Dave,
>>>>>>>>
>>>>>>>> With given commit, I'm again seeing the issue raised in
>>>>>>>> https://redmine.postgresql.org/issues/1523 :(
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards,
>>>>>>>> Murtuza Zabuawala
>>>>>>>> EnterpriseDB: http://www.enterprisedb.com
>>>>>>>> The Enterprise PostgreSQL Company
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Feb 26, 2018 at 7:49 PM, Dave Page <[email protected]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Ensure we pick up the messages from the current query and not a
>>>>>>>>> previous one. Fixes #3094
>>>>>>>>>
>>>>>>>>> Branch
>>>>>>>>> ------
>>>>>>>>> master
>>>>>>>>>
>>>>>>>>> Details
>>>>>>>>> -------
>>>>>>>>>
>>>>>>>>> https://git.postgresql.org/gitweb?p=pgadmin4.git;a=commitdiff;h=08b3ccc01a4d57e8ea3657f8882a53dcd1b9...
>>>>>>>>> Author: Khushboo Vashi <[email protected]>
>>>>>>>>>
>>>>>>>>> Modified Files
>>>>>>>>> --------------
>>>>>>>>> web/pgadmin/utils/driver/abstract.py | 1 +
>>>>>>>>> web/pgadmin/utils/driver/psycopg2/__init__.py | 64
>>>>>>>>> +++++++++------------------
>>>>>>>>> 2 files changed, 21 insertions(+), 44 deletions(-)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Dave Page
>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>> Twitter: @pgsnake
>>>>>>
>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>> The Enterprise PostgreSQL Company
>>>>>>
>>>>>
>>>
>>
>>
>> --
>> Dave Page
>> Blog: http://pgsnake.blogspot.com
>> Twitter: @pgsnake
>>
>> EnterpriseDB UK: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>>
>
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query
2018-02-26 14:19 pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-02-26 16:18 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
2018-02-26 16:23 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
2018-02-26 16:32 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-02-28 11:49 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
2018-02-28 15:25 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Joao De Almeida Pereira <[email protected]>
2018-03-01 05:56 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
2018-03-02 13:25 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-03-05 04:51 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
2018-03-05 15:12 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Joao De Almeida Pereira <[email protected]>
@ 2018-03-06 10:09 ` Khushboo Vashi <[email protected]>
2018-03-06 14:57 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Joao De Almeida Pereira <[email protected]>
0 siblings, 1 reply; 17+ messages in thread
From: Khushboo Vashi @ 2018-03-06 10:09 UTC (permalink / raw)
To: Joao De Almeida Pereira <[email protected]>; +Cc: Dave Page <[email protected]>; Murtuza Zabuawala <[email protected]>; pgadmin-hackers
On Mon, Mar 5, 2018 at 8:42 PM, Joao De Almeida Pereira <
[email protected]> wrote:
> Hello Khushboo,
> Looks like we are almost doen, just missing one PEP-8 issue:
> → pycodestyle --config=.pycodestyle pgadmin/tools
> pgadmin/tools/sqleditor/tests/test_poll_query_tool.py:46: [E126]
> continuation line over-indented for hanging indent
> 1 E126 continuation line over-indented for hanging indent
> 1
>
>
> Thanks Joao.
Please find the attached updated patch.
> The tests run successfully in our CI pipeline.
>
> Thanks
> Joao
>
> Thanks,
Khushboo
> On Sun, Mar 4, 2018 at 11:51 PM Khushboo Vashi <
> [email protected]> wrote:
>
>> On Fri, Mar 2, 2018 at 6:55 PM, Dave Page <[email protected]> wrote:
>>
>>> Could you rebase this please? It no longer applies.
>>>
>>> Please find the attached updated patch.
>>
>>> Thanks.
>>>
>>> On Thu, Mar 1, 2018 at 5:56 AM, Khushboo Vashi <
>>> [email protected]> wrote:
>>>
>>>> Hi Joao,
>>>>
>>>> Thanks for reviewing.
>>>>
>>>> On Wed, Feb 28, 2018 at 8:55 PM, Joao De Almeida Pereira <
>>>> [email protected]> wrote:
>>>>
>>>>> Hello Khushboo,
>>>>> After reviewing the patch I have the gut feeling that we do not have
>>>>> enough test coverage on this issue, specially due to the intricate while
>>>>> loop and conditions around the polling.
>>>>> I think that this deserve Unit tests around it, When I say Unit Test I
>>>>> am not talking about executing queries against the database, but do some
>>>>> stubbing of the database so that we can control the flow that we want.
>>>>>
>>>> You are right. It needs more unit testing. I have checked below
>>>> scenarios:
>>>> 1. Returns 2 notices with data output
>>>> 2. Returns 1000 notices with data output
>>>> 3. No notices with data output
>>>>
>>>> By running above, I have checked, each time returned notices are
>>>> accurate, no old notices are getting appended, it does not affect with the
>>>> amount of messages (few, none or more). Also, with the updated patch, I
>>>> have made sure that all these queries run with the single transaction id
>>>> (same connection).
>>>>
>>>> So, please let me know if you think I can add more things to this.
>>>>
>>>>>
>>>>>
>>>> It is a temptation to try to always do a Feature Test to test what we
>>>>> want because it is "easier" to write and ultimately it is what users see,
>>>>> but while 1 Feature Test runs we can run 200 Unit Tests that give us much
>>>>> more confidence that the code is doing what we expect it to do.
>>>>>
>>>>> Right, so added regression tests instead of feature tests.
>>>>
>>>> This being said, I run the tests on the CI Pipeline and all tests pass.
>>>>> Running pycodestyle fails due to some line sizes on the
>>>>> psycopg2/__init__py. I believe that it is not what you changed, but since
>>>>> you were changing the file it can be fixed it is just:
>>>>>
>>>>> pgadmin/utils/driver/psycopg2/__init__.py:1276: [E501] line too long
>>>>> (81 > 79 characters)
>>>>> pgadmin/utils/driver/psycopg2/__init__.py:1277: [E501] line too long
>>>>> (91 > 79 characters)
>>>>> pgadmin/utils/driver/psycopg2/__init__.py:1282: [E501] line too long
>>>>> (81 > 79 characters)
>>>>> pgadmin/utils/driver/psycopg2/__init__.py:1283: [E501] line too long
>>>>> (91 > 79 characters)
>>>>> 4 E501 line too long (81 > 79 characters)
>>>>>
>>>>> Fixed. Thanks for pointing out.
>>>>
>>>>>
>>>>> Thanks
>>>>> Joao
>>>>>
>>>>>
>>>>> On Wed, Feb 28, 2018 at 6:49 AM Khushboo Vashi <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> On Mon, Feb 26, 2018 at 10:02 PM, Dave Page <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> Argh, I ran some tests, but didn't spot any lost messages in the
>>>>>>> tests I ran. I'll revert the patch.
>>>>>>>
>>>>>>> Khushboo;
>>>>>>>
>>>>>>> Please look at the following:
>>>>>>>
>>>>>>> - Fix the patch so it doesn't drop messages.
>>>>>>>
>>>>>> Fixed.
>>>>>> By default, the notice attribute of the connection object of psycopg
>>>>>> 2 only stores 50 notices. Once it reaches to 50 it starts from 1 again.
>>>>>> To fix this I have changed the notice attribute from list to deque to
>>>>>> append more messages. Currently I have kept the maximum limit at a time of
>>>>>> the notice attribute is 100000 (in a single poll).
>>>>>>
>>>>>>> - Add regression tests to make sure it doesn't break in the future.
>>>>>>> This may require creating one or more functions the spew out a whole lot of
>>>>>>> notices, and then running a couple of queries and checking the output.
>>>>>>>
>>>>>> Added. With this regression test, the current code is failing which
>>>>>> has been taken care in this patch.
>>>>>>
>>>>>>> - Check the messages panel on the history tab. I just noticed it
>>>>>>> seems to only be showing an even smaller subset of the messages.
>>>>>>>
>>>>>> Tested and no issues found.
>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Thanks.
>>>>>>>
>>>>>>> On Mon, Feb 26, 2018 at 4:23 PM, Murtuza Zabuawala <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> Sent bit early,
>>>>>>>>
>>>>>>>> You can run 'VACUUM FULL VERBOSE' in query tool and verify the
>>>>>>>> populated messages (pgAdmin3 vs. pgAdmin4).
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Feb 26, 2018 at 9:48 PM, Murtuza Zabuawala <
>>>>>>>> [email protected]> wrote:
>>>>>>>>
>>>>>>>>> Hi Khushboo/Dave,
>>>>>>>>>
>>>>>>>>> With given commit, I'm again seeing the issue raised in
>>>>>>>>> https://redmine.postgresql.org/issues/1523 :(
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards,
>>>>>>>>> Murtuza Zabuawala
>>>>>>>>> EnterpriseDB: http://www.enterprisedb.com
>>>>>>>>> The Enterprise PostgreSQL Company
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Feb 26, 2018 at 7:49 PM, Dave Page <[email protected]>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Ensure we pick up the messages from the current query and not a
>>>>>>>>>> previous one. Fixes #3094
>>>>>>>>>>
>>>>>>>>>> Branch
>>>>>>>>>> ------
>>>>>>>>>> master
>>>>>>>>>>
>>>>>>>>>> Details
>>>>>>>>>> -------
>>>>>>>>>> https://git.postgresql.org/gitweb?p=pgadmin4.git;a=commitdiff;h=
>>>>>>>>>> 08b3ccc01a4d57e8ea3657f8882a53dcd1b99386
>>>>>>>>>> Author: Khushboo Vashi <[email protected]>
>>>>>>>>>>
>>>>>>>>>> Modified Files
>>>>>>>>>> --------------
>>>>>>>>>> web/pgadmin/utils/driver/abstract.py | 1 +
>>>>>>>>>> web/pgadmin/utils/driver/psycopg2/__init__.py | 64
>>>>>>>>>> +++++++++------------------
>>>>>>>>>> 2 files changed, 21 insertions(+), 44 deletions(-)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Dave Page
>>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>>> Twitter: @pgsnake
>>>>>>>
>>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>>> The Enterprise PostgreSQL Company
>>>>>>>
>>>>>>
>>>>
>>>
>>>
>>> --
>>> Dave Page
>>> Blog: http://pgsnake.blogspot.com
>>> Twitter: @pgsnake
>>>
>>> EnterpriseDB UK: http://www.enterprisedb.com
>>> The Enterprise PostgreSQL Company
>>>
>>
Attachments:
[text/x-patch] RM_3094_ver4.patch (11.1K, 3-RM_3094_ver4.patch)
download | inline diff:
diff --git a/web/pgadmin/feature_tests/keyboard_shortcut_test.py b/web/pgadmin/feature_tests/keyboard_shortcut_test.py
index 443eff6..bbae194 100644
--- a/web/pgadmin/feature_tests/keyboard_shortcut_test.py
+++ b/web/pgadmin/feature_tests/keyboard_shortcut_test.py
@@ -62,7 +62,9 @@ class KeyboardShortcutFeatureTest(BaseFeatureTest):
).key_down(
key_combo[2]
).key_up(
- Keys.ALT
+ key_combo[0]
+ ).key_up(
+ key_combo[1]
).perform()
print("Executing shortcut: " + self.new_shortcuts[s]['locator'] +
diff --git a/web/pgadmin/tools/sqleditor/tests/test_poll_query_tool.py b/web/pgadmin/tools/sqleditor/tests/test_poll_query_tool.py
new file mode 100644
index 0000000..275ed9c
--- /dev/null
+++ b/web/pgadmin/tools/sqleditor/tests/test_poll_query_tool.py
@@ -0,0 +1,112 @@
+##########################################################################
+#
+# pgAdmin 4 - PostgreSQL Tools
+#
+# Copyright (C) 2013 - 2018, The pgAdmin Development Team
+# This software is released under the PostgreSQL Licence
+#
+##########################################################################
+
+import json
+
+from pgadmin.browser.server_groups.servers.databases.tests import utils as \
+ database_utils
+from pgadmin.utils.route import BaseTestGenerator
+from regression import parent_node_dict
+from regression.python_test_utils import test_utils as utils
+
+
+class TestPollQueryTool(BaseTestGenerator):
+ """ This class will test the query tool polling. """
+ scenarios = [
+ ('When query tool polling returns messages with result data-set',
+ dict(
+ sql=[
+ """
+DROP TABLE IF EXISTS test_for_notices;
+
+DO $$
+BEGIN
+ RAISE NOTICE 'Hello, world!';
+END $$;
+
+SELECT 'CHECKING POLLING';
+""",
+ """
+DO $$
+BEGIN
+ FOR i in 1..1000 LOOP
+ RAISE NOTICE 'Count is %', i;
+ END LOOP;
+END $$;
+
+SELECT 'CHECKING POLLING FOR LONG MESSAGES';
+""",
+ "SELECT 'CHECKING POLLING WITHOUT MESSAGES';"
+ ],
+ expected_message=['NOTICE: table "test_for_notices" ' +
+ """does not exist, skipping
+NOTICE: Hello, world!
+""",
+ "\n".join(["NOTICE: Count is {0}".format(i)
+ for i in range(1, 1001)]) + "\n",
+ None],
+ expected_result=['CHECKING POLLING',
+ 'CHECKING POLLING FOR LONG MESSAGES',
+ 'CHECKING POLLING WITHOUT MESSAGES'],
+ print_messages=['2 NOTICES WITH DATASET',
+ '1000 NOTICES WITH DATASET',
+ 'NO NOTICE WITH DATASET'
+ ]
+ ))
+ ]
+
+ def runTest(self):
+ """ This function will check messages return by query tool polling. """
+ database_info = parent_node_dict["database"][-1]
+ self.server_id = database_info["server_id"]
+
+ self.db_id = database_info["db_id"]
+ db_con = database_utils.connect_database(self,
+ utils.SERVER_GROUP,
+ self.server_id,
+ self.db_id)
+ if not db_con["info"] == "Database connected.":
+ raise Exception("Could not connect to the database.")
+
+ # Initialize query tool
+ url = '/datagrid/initialize/query_tool/{0}/{1}/{2}'.format(
+ utils.SERVER_GROUP, self.server_id, self.db_id)
+ response = self.tester.post(url)
+ self.assertEquals(response.status_code, 200)
+
+ response_data = json.loads(response.data.decode('utf-8'))
+ self.trans_id = response_data['data']['gridTransId']
+
+ cnt = 0
+ for s in self.sql:
+ print("Executing and polling with: " + self.print_messages[cnt])
+ # Start query tool transaction
+ url = '/sqleditor/query_tool/start/{0}'.format(self.trans_id)
+ response = self.tester.post(url, data=json.dumps({"sql": s}),
+ content_type='html/json')
+
+ self.assertEquals(response.status_code, 200)
+
+ # Query tool polling
+ url = '/sqleditor/poll/{0}'.format(self.trans_id)
+ response = self.tester.get(url)
+ self.assertEquals(response.status_code, 200)
+ response_data = json.loads(response.data.decode('utf-8'))
+
+ # Check the returned messages
+ self.assertEquals(self.expected_message[cnt],
+ response_data['data']['additional_messages'])
+ # Check the output
+ self.assertEquals(self.expected_result[cnt],
+ response_data['data']['result'][0][0])
+
+ cnt += 1
+
+ # Disconnect the database
+ database_utils.disconnect_database(self, self.server_id, self.db_id)
diff --git a/web/pgadmin/utils/driver/abstract.py b/web/pgadmin/utils/driver/abstract.py
index 32e1c97..271bfec 100644
--- a/web/pgadmin/utils/driver/abstract.py
+++ b/web/pgadmin/utils/driver/abstract.py
@@ -168,6 +168,8 @@ class BaseConnection(object):
ASYNC_WRITE_TIMEOUT = 3
ASYNC_NOT_CONNECTED = 4
ASYNC_EXECUTION_ABORTED = 5
+ ASYNC_TIMEOUT = 0.2
+ ASYNC_NOTICE_MAXLENGTH = 100000
@abstractmethod
def connect(self, **kwargs):
diff --git a/web/pgadmin/utils/driver/psycopg2/__init__.py b/web/pgadmin/utils/driver/psycopg2/__init__.py
index 81442e4..941a694 100644
--- a/web/pgadmin/utils/driver/psycopg2/__init__.py
+++ b/web/pgadmin/utils/driver/psycopg2/__init__.py
@@ -37,6 +37,7 @@ from .cursor import DictCursor
from .typecast import register_global_typecasters, \
register_string_typecasters, register_binary_typecasters, \
register_array_to_string_typecasters, ALL_JSON_TYPES
+from collections import deque
if sys.version_info < (3,):
@@ -110,7 +111,7 @@ class Connection(BaseConnection):
- This method is used to wait for asynchronous connection. This is a
blocking call.
- * _wait_timeout(conn, time)
+ * _wait_timeout(conn)
- This method is used to wait for asynchronous connection with timeout.
This is a non blocking call.
@@ -310,6 +311,9 @@ class Connection(BaseConnection):
)
return False, msg
+ # Overwrite connection notice attr to support
+ # more than 50 notices at a time
+ pg_conn.notices = deque([], self.ASYNC_NOTICE_MAXLENGTH)
self.conn = pg_conn
self.wasConnected = True
try:
@@ -1208,6 +1212,7 @@ Failed to reset the connection to the server due to following error:
)
return False, msg
+ pg_conn.notices = deque([], self.ASYNC_NOTICE_MAXLENGTH)
self.conn = pg_conn
self.__backend_pid = pg_conn.get_backend_pid()
@@ -1261,51 +1266,31 @@ Failed to reset the connection to the server due to following error:
Args:
conn: connection object
- time: wait time
"""
-
- state = conn.poll()
- if state == psycopg2.extensions.POLL_OK:
- return self.ASYNC_OK
- elif state == psycopg2.extensions.POLL_WRITE:
- # Wait for the given time and then check the return status
- # If three empty lists are returned then the time-out is reached.
- timeout_status = select.select([], [conn.fileno()], [], 0)
- if timeout_status == ([], [], []):
- return self.ASYNC_WRITE_TIMEOUT
-
- # poll again to check the state if it is still POLL_WRITE
- # then return ASYNC_WRITE_TIMEOUT else return ASYNC_OK.
+ while 1:
state = conn.poll()
- if state == psycopg2.extensions.POLL_WRITE:
- return self.ASYNC_WRITE_TIMEOUT
- return self.ASYNC_OK
- elif state == psycopg2.extensions.POLL_READ:
- # Wait for the given time and then check the return status
- # If three empty lists are returned then the time-out is reached.
- timeout_status = select.select([conn.fileno()], [], [], 0)
- if timeout_status == ([], [], []):
- return self.ASYNC_READ_TIMEOUT
-
- # select.select timeout option works only if we provide
- # empty [] [] [] file descriptor in select.select() function
- # and that also works only on UNIX based system, it do not support
- # Windows Hence we have wrote our own pooling mechanism to read
- # data fast each call conn.poll() reads chunks of data from
- # connection object more we poll more we read data from connection
- cnt = 0
- while cnt < 1000:
- # poll again to check the state if it is still POLL_READ
- # then return ASYNC_READ_TIMEOUT else return ASYNC_OK.
- state = conn.poll()
- if state == psycopg2.extensions.POLL_OK:
- return self.ASYNC_OK
- cnt += 1
- return self.ASYNC_READ_TIMEOUT
- else:
- raise psycopg2.OperationalError(
- "poll() returned %s from _wait_timeout function" % state
- )
+ if state == psycopg2.extensions.POLL_OK:
+ return self.ASYNC_OK
+ elif state == psycopg2.extensions.POLL_WRITE:
+ # Wait for the given time and then check the return status
+ # If three empty lists are returned then the time-out is
+ # reached.
+ timeout_status = select.select([], [conn.fileno()], [],
+ self.ASYNC_TIMEOUT)
+ if timeout_status == ([], [], []):
+ return self.ASYNC_WRITE_TIMEOUT
+ elif state == psycopg2.extensions.POLL_READ:
+ # Wait for the given time and then check the return status
+ # If three empty lists are returned then the time-out is
+ # reached.
+ timeout_status = select.select([conn.fileno()], [], [],
+ self.ASYNC_TIMEOUT)
+ if timeout_status == ([], [], []):
+ return self.ASYNC_READ_TIMEOUT
+ else:
+ raise psycopg2.OperationalError(
+ "poll() returned %s from _wait_timeout function" % state
+ )
def poll(self, formatted_exception_msg=False, no_result=False):
"""
@@ -1347,8 +1332,8 @@ Failed to reset the connection to the server due to following error:
is_error = True
if self.conn.notices and self.__notices is not None:
- while self.conn.notices:
- self.__notices.append(self.conn.notices.pop(0)[:])
+ self.__notices.extend(self.conn.notices)
+ self.conn.notices.clear()
# We also need to fetch notices before we return from function in case
# of any Exception, To avoid code duplication we will return after
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query
2018-02-26 14:19 pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-02-26 16:18 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
2018-02-26 16:23 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
2018-02-26 16:32 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-02-28 11:49 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
2018-02-28 15:25 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Joao De Almeida Pereira <[email protected]>
2018-03-01 05:56 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
2018-03-02 13:25 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-03-05 04:51 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
2018-03-05 15:12 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Joao De Almeida Pereira <[email protected]>
2018-03-06 10:09 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
@ 2018-03-06 14:57 ` Joao De Almeida Pereira <[email protected]>
2018-03-07 13:38 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
0 siblings, 1 reply; 17+ messages in thread
From: Joao De Almeida Pereira @ 2018-03-06 14:57 UTC (permalink / raw)
To: Khushboo Vashi <[email protected]>; +Cc: Dave Page <[email protected]>; Murtuza Zabuawala <[email protected]>; pgadmin-hackers
Hello Khushboo,
Thanks for the changes here. Now everything looks good, and the tests all
pass.
Thanks
Joao
On Tue, Mar 6, 2018 at 5:09 AM Khushboo Vashi <
[email protected]> wrote:
> On Mon, Mar 5, 2018 at 8:42 PM, Joao De Almeida Pereira <
> [email protected]> wrote:
>
>> Hello Khushboo,
>>
> Looks like we are almost doen, just missing one PEP-8 issue:
>> → pycodestyle --config=.pycodestyle pgadmin/tools
>> pgadmin/tools/sqleditor/tests/test_poll_query_tool.py:46: [E126]
>> continuation line over-indented for hanging indent
>> 1 E126 continuation line over-indented for hanging indent
>> 1
>>
>>
>> Thanks Joao.
> Please find the attached updated patch.
>
>> The tests run successfully in our CI pipeline.
>>
>> Thanks
>> Joao
>>
>> Thanks,
> Khushboo
>
>> On Sun, Mar 4, 2018 at 11:51 PM Khushboo Vashi <
>> [email protected]> wrote:
>>
>>> On Fri, Mar 2, 2018 at 6:55 PM, Dave Page <[email protected]> wrote:
>>>
>>>> Could you rebase this please? It no longer applies.
>>>>
>>>> Please find the attached updated patch.
>>>
>>>> Thanks.
>>>>
>>>> On Thu, Mar 1, 2018 at 5:56 AM, Khushboo Vashi <
>>>> [email protected]> wrote:
>>>>
>>>>> Hi Joao,
>>>>>
>>>>> Thanks for reviewing.
>>>>>
>>>>> On Wed, Feb 28, 2018 at 8:55 PM, Joao De Almeida Pereira <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Hello Khushboo,
>>>>>> After reviewing the patch I have the gut feeling that we do not have
>>>>>> enough test coverage on this issue, specially due to the intricate while
>>>>>> loop and conditions around the polling.
>>>>>> I think that this deserve Unit tests around it, When I say Unit Test
>>>>>> I am not talking about executing queries against the database, but do some
>>>>>> stubbing of the database so that we can control the flow that we want.
>>>>>>
>>>>> You are right. It needs more unit testing. I have checked below
>>>>> scenarios:
>>>>> 1. Returns 2 notices with data output
>>>>> 2. Returns 1000 notices with data output
>>>>> 3. No notices with data output
>>>>>
>>>>> By running above, I have checked, each time returned notices are
>>>>> accurate, no old notices are getting appended, it does not affect with the
>>>>> amount of messages (few, none or more). Also, with the updated patch, I
>>>>> have made sure that all these queries run with the single transaction id
>>>>> (same connection).
>>>>>
>>>>> So, please let me know if you think I can add more things to this.
>>>>>
>>>>>>
>>>>>>
>>>>> It is a temptation to try to always do a Feature Test to test what we
>>>>>> want because it is "easier" to write and ultimately it is what users see,
>>>>>> but while 1 Feature Test runs we can run 200 Unit Tests that give us much
>>>>>> more confidence that the code is doing what we expect it to do.
>>>>>>
>>>>>> Right, so added regression tests instead of feature tests.
>>>>>
>>>>> This being said, I run the tests on the CI Pipeline and all tests
>>>>>> pass. Running pycodestyle fails due to some line sizes on the
>>>>>> psycopg2/__init__py. I believe that it is not what you changed, but since
>>>>>> you were changing the file it can be fixed it is just:
>>>>>>
>>>>>> pgadmin/utils/driver/psycopg2/__init__.py:1276: [E501] line too long
>>>>>> (81 > 79 characters)
>>>>>> pgadmin/utils/driver/psycopg2/__init__.py:1277: [E501] line too long
>>>>>> (91 > 79 characters)
>>>>>> pgadmin/utils/driver/psycopg2/__init__.py:1282: [E501] line too long
>>>>>> (81 > 79 characters)
>>>>>> pgadmin/utils/driver/psycopg2/__init__.py:1283: [E501] line too long
>>>>>> (91 > 79 characters)
>>>>>> 4 E501 line too long (81 > 79 characters)
>>>>>>
>>>>>> Fixed. Thanks for pointing out.
>>>>>
>>>>>>
>>>>>> Thanks
>>>>>> Joao
>>>>>>
>>>>>>
>>>>>> On Wed, Feb 28, 2018 at 6:49 AM Khushboo Vashi <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> On Mon, Feb 26, 2018 at 10:02 PM, Dave Page <[email protected]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Argh, I ran some tests, but didn't spot any lost messages in the
>>>>>>>> tests I ran. I'll revert the patch.
>>>>>>>>
>>>>>>>> Khushboo;
>>>>>>>>
>>>>>>>> Please look at the following:
>>>>>>>>
>>>>>>>> - Fix the patch so it doesn't drop messages.
>>>>>>>>
>>>>>>> Fixed.
>>>>>>> By default, the notice attribute of the connection object of psycopg
>>>>>>> 2 only stores 50 notices. Once it reaches to 50 it starts from 1 again.
>>>>>>> To fix this I have changed the notice attribute from list to deque
>>>>>>> to append more messages. Currently I have kept the maximum limit at a time
>>>>>>> of the notice attribute is 100000 (in a single poll).
>>>>>>>
>>>>>>>> - Add regression tests to make sure it doesn't break in the future.
>>>>>>>> This may require creating one or more functions the spew out a whole lot of
>>>>>>>> notices, and then running a couple of queries and checking the output.
>>>>>>>>
>>>>>>> Added. With this regression test, the current code is failing which
>>>>>>> has been taken care in this patch.
>>>>>>>
>>>>>>>> - Check the messages panel on the history tab. I just noticed it
>>>>>>>> seems to only be showing an even smaller subset of the messages.
>>>>>>>>
>>>>>>> Tested and no issues found.
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> On Mon, Feb 26, 2018 at 4:23 PM, Murtuza Zabuawala <
>>>>>>>> [email protected]> wrote:
>>>>>>>>
>>>>>>>>> Sent bit early,
>>>>>>>>>
>>>>>>>>> You can run 'VACUUM FULL VERBOSE' in query tool and verify the
>>>>>>>>> populated messages (pgAdmin3 vs. pgAdmin4).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Feb 26, 2018 at 9:48 PM, Murtuza Zabuawala <
>>>>>>>>> [email protected]> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Khushboo/Dave,
>>>>>>>>>>
>>>>>>>>>> With given commit, I'm again seeing the issue raised in
>>>>>>>>>> https://redmine.postgresql.org/issues/1523 :(
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards,
>>>>>>>>>> Murtuza Zabuawala
>>>>>>>>>> EnterpriseDB: http://www.enterprisedb.com
>>>>>>>>>> The Enterprise PostgreSQL Company
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Feb 26, 2018 at 7:49 PM, Dave Page <[email protected]>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Ensure we pick up the messages from the current query and not a
>>>>>>>>>>> previous one. Fixes #3094
>>>>>>>>>>>
>>>>>>>>>>> Branch
>>>>>>>>>>> ------
>>>>>>>>>>> master
>>>>>>>>>>>
>>>>>>>>>>> Details
>>>>>>>>>>> -------
>>>>>>>>>>>
>>>>>>>>>>> https://git.postgresql.org/gitweb?p=pgadmin4.git;a=commitdiff;h=08b3ccc01a4d57e8ea3657f8882a53dcd1b9...
>>>>>>>>>>> Author: Khushboo Vashi <[email protected]>
>>>>>>>>>>>
>>>>>>>>>>> Modified Files
>>>>>>>>>>> --------------
>>>>>>>>>>> web/pgadmin/utils/driver/abstract.py | 1 +
>>>>>>>>>>> web/pgadmin/utils/driver/psycopg2/__init__.py | 64
>>>>>>>>>>> +++++++++------------------
>>>>>>>>>>> 2 files changed, 21 insertions(+), 44 deletions(-)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Dave Page
>>>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>>>> Twitter: @pgsnake
>>>>>>>>
>>>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>>>> The Enterprise PostgreSQL Company
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Dave Page
>>>> Blog: http://pgsnake.blogspot.com
>>>> Twitter: @pgsnake
>>>>
>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>> The Enterprise PostgreSQL Company
>>>>
>>>
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query
2018-02-26 14:19 pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-02-26 16:18 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
2018-02-26 16:23 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Murtuza Zabuawala <[email protected]>
2018-02-26 16:32 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-02-28 11:49 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
2018-02-28 15:25 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Joao De Almeida Pereira <[email protected]>
2018-03-01 05:56 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
2018-03-02 13:25 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-03-05 04:51 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
2018-03-05 15:12 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Joao De Almeida Pereira <[email protected]>
2018-03-06 10:09 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Khushboo Vashi <[email protected]>
2018-03-06 14:57 ` Re: pgAdmin 4 commit: Ensure we pick up the messages from the current query Joao De Almeida Pereira <[email protected]>
@ 2018-03-07 13:38 ` Dave Page <[email protected]>
0 siblings, 0 replies; 17+ messages in thread
From: Dave Page @ 2018-03-07 13:38 UTC (permalink / raw)
To: Joao De Almeida Pereira <[email protected]>; +Cc: Khushboo Vashi <[email protected]>; Murtuza Zabuawala <[email protected]>; pgadmin-hackers
Thanks - patch applied.
On Tue, Mar 6, 2018 at 2:57 PM, Joao De Almeida Pereira <
[email protected]> wrote:
> Hello Khushboo,
> Thanks for the changes here. Now everything looks good, and the tests all
> pass.
>
> Thanks
> Joao
>
> On Tue, Mar 6, 2018 at 5:09 AM Khushboo Vashi <
> [email protected]> wrote:
>
>> On Mon, Mar 5, 2018 at 8:42 PM, Joao De Almeida Pereira <
>> [email protected]> wrote:
>>
>>> Hello Khushboo,
>>>
>> Looks like we are almost doen, just missing one PEP-8 issue:
>>> → pycodestyle --config=.pycodestyle pgadmin/tools
>>> pgadmin/tools/sqleditor/tests/test_poll_query_tool.py:46: [E126]
>>> continuation line over-indented for hanging indent
>>> 1 E126 continuation line over-indented for hanging indent
>>> 1
>>>
>>>
>>> Thanks Joao.
>> Please find the attached updated patch.
>>
>>> The tests run successfully in our CI pipeline.
>>>
>>> Thanks
>>> Joao
>>>
>>> Thanks,
>> Khushboo
>>
>>> On Sun, Mar 4, 2018 at 11:51 PM Khushboo Vashi <
>>> [email protected]> wrote:
>>>
>>>> On Fri, Mar 2, 2018 at 6:55 PM, Dave Page <[email protected]> wrote:
>>>>
>>>>> Could you rebase this please? It no longer applies.
>>>>>
>>>>> Please find the attached updated patch.
>>>>
>>>>> Thanks.
>>>>>
>>>>> On Thu, Mar 1, 2018 at 5:56 AM, Khushboo Vashi <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Hi Joao,
>>>>>>
>>>>>> Thanks for reviewing.
>>>>>>
>>>>>> On Wed, Feb 28, 2018 at 8:55 PM, Joao De Almeida Pereira <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> Hello Khushboo,
>>>>>>> After reviewing the patch I have the gut feeling that we do not have
>>>>>>> enough test coverage on this issue, specially due to the intricate while
>>>>>>> loop and conditions around the polling.
>>>>>>> I think that this deserve Unit tests around it, When I say Unit Test
>>>>>>> I am not talking about executing queries against the database, but do some
>>>>>>> stubbing of the database so that we can control the flow that we want.
>>>>>>>
>>>>>> You are right. It needs more unit testing. I have checked below
>>>>>> scenarios:
>>>>>> 1. Returns 2 notices with data output
>>>>>> 2. Returns 1000 notices with data output
>>>>>> 3. No notices with data output
>>>>>>
>>>>>> By running above, I have checked, each time returned notices are
>>>>>> accurate, no old notices are getting appended, it does not affect with the
>>>>>> amount of messages (few, none or more). Also, with the updated patch, I
>>>>>> have made sure that all these queries run with the single transaction id
>>>>>> (same connection).
>>>>>>
>>>>>> So, please let me know if you think I can add more things to this.
>>>>>>
>>>>>>>
>>>>>>>
>>>>>> It is a temptation to try to always do a Feature Test to test what we
>>>>>>> want because it is "easier" to write and ultimately it is what users see,
>>>>>>> but while 1 Feature Test runs we can run 200 Unit Tests that give us much
>>>>>>> more confidence that the code is doing what we expect it to do.
>>>>>>>
>>>>>>> Right, so added regression tests instead of feature tests.
>>>>>>
>>>>>> This being said, I run the tests on the CI Pipeline and all tests
>>>>>>> pass. Running pycodestyle fails due to some line sizes on the
>>>>>>> psycopg2/__init__py. I believe that it is not what you changed, but since
>>>>>>> you were changing the file it can be fixed it is just:
>>>>>>>
>>>>>>> pgadmin/utils/driver/psycopg2/__init__.py:1276: [E501] line too
>>>>>>> long (81 > 79 characters)
>>>>>>> pgadmin/utils/driver/psycopg2/__init__.py:1277: [E501] line too
>>>>>>> long (91 > 79 characters)
>>>>>>> pgadmin/utils/driver/psycopg2/__init__.py:1282: [E501] line too
>>>>>>> long (81 > 79 characters)
>>>>>>> pgadmin/utils/driver/psycopg2/__init__.py:1283: [E501] line too
>>>>>>> long (91 > 79 characters)
>>>>>>> 4 E501 line too long (81 > 79 characters)
>>>>>>>
>>>>>>> Fixed. Thanks for pointing out.
>>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>> Joao
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Feb 28, 2018 at 6:49 AM Khushboo Vashi <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> On Mon, Feb 26, 2018 at 10:02 PM, Dave Page <[email protected]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Argh, I ran some tests, but didn't spot any lost messages in the
>>>>>>>>> tests I ran. I'll revert the patch.
>>>>>>>>>
>>>>>>>>> Khushboo;
>>>>>>>>>
>>>>>>>>> Please look at the following:
>>>>>>>>>
>>>>>>>>> - Fix the patch so it doesn't drop messages.
>>>>>>>>>
>>>>>>>> Fixed.
>>>>>>>> By default, the notice attribute of the connection object of
>>>>>>>> psycopg 2 only stores 50 notices. Once it reaches to 50 it starts from 1
>>>>>>>> again.
>>>>>>>> To fix this I have changed the notice attribute from list to deque
>>>>>>>> to append more messages. Currently I have kept the maximum limit at a time
>>>>>>>> of the notice attribute is 100000 (in a single poll).
>>>>>>>>
>>>>>>>>> - Add regression tests to make sure it doesn't break in the
>>>>>>>>> future. This may require creating one or more functions the spew out a
>>>>>>>>> whole lot of notices, and then running a couple of queries and checking the
>>>>>>>>> output.
>>>>>>>>>
>>>>>>>> Added. With this regression test, the current code is failing which
>>>>>>>> has been taken care in this patch.
>>>>>>>>
>>>>>>>>> - Check the messages panel on the history tab. I just noticed it
>>>>>>>>> seems to only be showing an even smaller subset of the messages.
>>>>>>>>>
>>>>>>>> Tested and no issues found.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> On Mon, Feb 26, 2018 at 4:23 PM, Murtuza Zabuawala <
>>>>>>>>> [email protected]> wrote:
>>>>>>>>>
>>>>>>>>>> Sent bit early,
>>>>>>>>>>
>>>>>>>>>> You can run 'VACUUM FULL VERBOSE' in query tool and verify the
>>>>>>>>>> populated messages (pgAdmin3 vs. pgAdmin4).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Feb 26, 2018 at 9:48 PM, Murtuza Zabuawala <
>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Khushboo/Dave,
>>>>>>>>>>>
>>>>>>>>>>> With given commit, I'm again seeing the issue raised in
>>>>>>>>>>> https://redmine.postgresql.org/issues/1523 :(
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Regards,
>>>>>>>>>>> Murtuza Zabuawala
>>>>>>>>>>> EnterpriseDB: http://www.enterprisedb.com
>>>>>>>>>>> The Enterprise PostgreSQL Company
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Feb 26, 2018 at 7:49 PM, Dave Page <[email protected]>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Ensure we pick up the messages from the current query and not a
>>>>>>>>>>>> previous one. Fixes #3094
>>>>>>>>>>>>
>>>>>>>>>>>> Branch
>>>>>>>>>>>> ------
>>>>>>>>>>>> master
>>>>>>>>>>>>
>>>>>>>>>>>> Details
>>>>>>>>>>>> -------
>>>>>>>>>>>> https://git.postgresql.org/gitweb?p=pgadmin4.git;a=
>>>>>>>>>>>> commitdiff;h=08b3ccc01a4d57e8ea3657f8882a53dcd1b99386
>>>>>>>>>>>> Author: Khushboo Vashi <[email protected]>
>>>>>>>>>>>>
>>>>>>>>>>>> Modified Files
>>>>>>>>>>>> --------------
>>>>>>>>>>>> web/pgadmin/utils/driver/abstract.py | 1 +
>>>>>>>>>>>> web/pgadmin/utils/driver/psycopg2/__init__.py | 64
>>>>>>>>>>>> +++++++++------------------
>>>>>>>>>>>> 2 files changed, 21 insertions(+), 44 deletions(-)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Dave Page
>>>>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>>>>> Twitter: @pgsnake
>>>>>>>>>
>>>>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>>>>> The Enterprise PostgreSQL Company
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Dave Page
>>>>> Blog: http://pgsnake.blogspot.com
>>>>> Twitter: @pgsnake
>>>>>
>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>> The Enterprise PostgreSQL Company
>>>>>
>>>>
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 17+ messages in thread
end of thread, other threads:[~2018-03-07 13:38 UTC | newest]
Thread overview: 17+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2018-02-26 14:19 pgAdmin 4 commit: Ensure we pick up the messages from the current query Dave Page <[email protected]>
2018-02-26 16:18 ` Murtuza Zabuawala <[email protected]>
2018-02-26 16:23 ` Murtuza Zabuawala <[email protected]>
2018-02-26 16:32 ` Dave Page <[email protected]>
2018-02-26 17:34 ` Murtuza Zabuawala <[email protected]>
2018-02-28 11:49 ` Khushboo Vashi <[email protected]>
2018-02-28 15:25 ` Joao De Almeida Pereira <[email protected]>
2018-03-01 05:56 ` Khushboo Vashi <[email protected]>
2018-03-01 15:21 ` Joao De Almeida Pereira <[email protected]>
2018-03-02 13:31 ` Dave Page <[email protected]>
2018-03-02 18:42 ` Joao De Almeida Pereira <[email protected]>
2018-03-02 13:25 ` Dave Page <[email protected]>
2018-03-05 04:51 ` Khushboo Vashi <[email protected]>
2018-03-05 15:12 ` Joao De Almeida Pereira <[email protected]>
2018-03-06 10:09 ` Khushboo Vashi <[email protected]>
2018-03-06 14:57 ` Joao De Almeida Pereira <[email protected]>
2018-03-07 13:38 ` Dave Page <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox