Subject: pgAdmin 4 commit: Support EXPLAIN on Greenplum. Fixes #3097

From: Dave Page <[email-masked]>
Date: Fri, 09 Feb 2018 11:54:45 +0000

Support EXPLAIN on Greenplum. Fixes #3097

 - Extract SQLEditor.execute and SQLEditor._poll into their own files and add test around them
 - Extract SQLEditor backend functions that start executing query to their own files and add tests around it
 - Move the Explain SQL from the front-end and now pass the Explain plan parameters as a JSON object in the start query call.
 - Extract the compile_template_name into a function that can be used by the different places that try to select the version of the template and the server type

Branch
------
master

Details
-------
https://git.postgresql.org/gitweb?p=pgadmin4.git;a=commitdiff;h=e16a95275336529a734bf0066889e39cc8ef0662
Author: Joao Pedro De Almeida Pereira <[email-masked]>

Modified Files
--------------
.../databases/schemas/tables/tests/test_utils.py   |    0
web/pgadmin/static/js/sqleditor/execute_query.js   |  287 ++++
.../js/sqleditor/is_new_transaction_required.js    |   14 +
.../static/js/sqleditor/query_tool_actions.js      |   33 +-
web/pgadmin/tools/sqleditor/__init__.py            |  396 +----
web/pgadmin/tools/sqleditor/static/js/sqleditor.js |  227 +--
.../sqleditor/sql/10_plus/explain_plan.sql         |   23 +
.../sqleditor/sql/9.2_plus/explain_plan.sql        |   20 +
.../sqleditor/sql/default/explain_plan.sql         |   17 +
.../sqleditor/sql/gpdb_5.0_plus/explain_plan.sql   |    5 +
web/pgadmin/tools/sqleditor/tests/__init__.py      |    8 +
.../sqleditor/tests/test_explain_plan_templates.py |  152 ++
.../test_extract_sql_from_network_parameters.py    |   59 +
.../tools/sqleditor/tests/test_start_query_tool.py |   38 +
web/pgadmin/tools/sqleditor/utils/__init__.py      |   14 +
.../sqleditor/utils/apply_explain_plan_wrapper.py  |   24 +
.../tools/sqleditor/utils/constant_definition.py   |   32 +
.../tools/sqleditor/utils/is_begin_required.py     |  169 ++
.../tools/sqleditor/utils/start_running_query.py   |  172 ++
.../tools/sqleditor/utils/tests/__init__.py        |    8 +
.../utils/tests/test_apply_explain_plan_wrapper.py |  121 ++
.../utils/tests/test_start_running_query.py        |  445 +++++
.../utils/update_session_grid_transaction.py       |   18 +
web/pgadmin/utils/compile_template_name.py         |   17 +
.../utils/tests/test_compile_template_name.py      |   34 +
web/pgadmin/utils/versioned_template_loader.py     |    2 +-
web/regression/javascript/fake_endpoints.js        |    6 +-
.../javascript/sqleditor/execute_query_spec.js     | 1702 ++++++++++++++++++++
.../sqleditor/is_new_transaction_required_spec.js  |   65 +
.../sqleditor/query_tool_actions_spec.js           |  141 +-
30 files changed, 3670 insertions(+), 579 deletions(-)

---

From: Khushboo Vashi <[email-masked]>
Date: Thu, 8 Mar 2018 10:11:13 +0530

Hi Joao,

In the test_start_running_query.py, 2 static methods
(is_begin_required_for_sql_query
and is_rollback_statement_required)
of StartRunningQuery class were used directly without @patch. Due to this,
in all the cases, the original value of them doesn't restore.

To fix this, I have sent the patch in another thread, to restore its
original state, but I wonder if we can use these methods with @patch.

Thanks,
Khushboo

On Fri, Feb 9, 2018 at 5:24 PM, Dave Page <[email-masked]> wrote:

>  - Move the Explain SQL from the front-end and now pass the Explain plan
> parameters as a JSON object in the start query call.
>  - Extract the compile_template_name into a function that can be used by
> the different places that try to select the version of the template and the
> server type
>
> Branch
> ------
> master
>
> Details
> -------

---

From: Joao De Almeida Pereira <[email-masked]>
Date: Thu, 08 Mar 2018 14:22:45 +0000

Hello Khushboo,
Completely forgot about this python "feature".......
Attached is the fix.

Just as a side question, does anyone else feel the pain of wanting to run a
single test using a IDE or the command line and not being able to?
We an HandRolled the loader, and that as some implications. Did anyone try
to use a different launcher like pytest or nose instead of the current
runner?
I understand that testscenarios is one of the problems we have if we want
to move away from this way of running tests.
Any suggestion?

Thanks
Joao

On Wed, Mar 7, 2018 at 11:41 PM Khushboo Vashi <
[email-masked]> wrote:

> Hi Joao,
>
> In the test_start_running_query.py, 2 static methods (is_begin_required_for_sql_query
> of StartRunningQuery class were used directly without @patch. Due to
> this, in all the cases, the original value of them doesn't restore.
>

---

From: Khushboo Vashi <[email-masked]>
Date: Fri, 9 Mar 2018 08:35:16 +0530

Hi Joao,

On Thu, Mar 8, 2018 at 7:52 PM, Joao De Almeida Pereira <
[email-masked]> wrote:

> Patch looks good to me and also passed the test cases on PY 2.7 and PY
3.5.

> Just as a side question, does anyone else feel the pain of wanting to run
> a single test using a IDE or the command line and not being able to?
>
While development, I run module wise test cases but yes as you said can't
run a single test case without doing some manual work.

Khushboo

>> (is_begin_required_for_sql_query and is_rollback_statement_required)

---

From: Dave Page <[email-masked]>
Date: Fri, 9 Mar 2018 13:31:19 +0000

> Completely forgot about this python "feature".......

Thanks, applied.

> a single test using a IDE or the command line and not being able to?

Not really - the Python and JS tests are so quick I don't really care (and
with the Python ones, I can execute for a single module for even more
speed).

What I would *really* like, is the ability to run individual feature tests.
That would be very valuable and save me a ton of time.

-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

---

From: Joao De Almeida Pereira <[email-masked]>
Date: Fri, 09 Mar 2018 15:54:31 +0000

Hello,
Definitely running single tests is something that would be great, specially
if you are TDDing something waiting 30-40 seconds to get feedback is a
little cumbersome when the test you are concerned with take less then a
second.

In the process:
1. Write a test
2. Make the test pass
3. Refactor

in between each step you run the test more then 1 time, and depending on
the refactoring you might need to run it several times. So imagine waiting
30 seconds per run to get results. To run a subset of tests is a pain
because you need to be always changing the way you run the tests.....

I believe we could archive a better granularity and choosing what test to
run if we used a runner like pytest or nose to do it. What was the reason
behind handrolling a test runner script? I am asking this because in a
previous job I decided to handroll a unittest loader script and that was
something that I regretted every time I had to touch it, and eventually was
in the process of changing it to pytest.

I looked into pytest to replace the current the current runtest, and the
major problem I found was the testscenarios integration(See Note 1). It can
be done but we would need to change all the test functions to receive the
scenario variables through arguments on the function. Also didn't dug much
into setting all the variables that we need there and all the environment.
The other issue that I do not like very much about pytest is the fact that
you loose the unittest assertion that is not so bad because there are some
neat libraries like: https://github.com/grappa-py/grappa,
https://github.com/ActivisionGameScience/assertpy,
https://github.com/dgilland/verify. Personally I really like the syntax of
Grapa, but the Veridfy one is pretty similar to Jasmine too.

What are your thoughts?

Note 1: As an example of what our functions would have to look like you can
see:
https://github.com/OriMenashe/pytest-scenario/blob/master/tests/test_parametrize.py
As a example this class:

class ServersWithServiceIDAddTestCase(BaseTestGenerator):
    """ This class will add the servers under default server group. """

    scenarios = [
        # Fetch the default url for server object
        (
            'Default Server Node url', dict(
                url='/browser/server/obj/'
            )
        )
    ]

    def setUp(self):
        pass

    def runTest(self):
        """ This function will add the server under default server group."""
        url = "{0}{1}/".format(self.url, utils.SERVER_GROUP)
        # Add service name in the config
        self.server['service'] = "TestDB"
        response = self.tester.post(
            url,
            data=json.dumps(self.server),
            content_type='html/json'
        )
        self.assertEquals(response.status_code, 200)
        response_data = json.loads(response.data.decode('utf-8'))
        self.server_id = response_data['node']['_id']

    def tearDown(self):
        """This function delete the server from SQLite """
        utils.delete_server_with_api(self.tester, self.server_id)

Would have to look changed to:

class ServersWithServiceIDAddTestCase(object):
    """ This class will add the servers under default server group. """

    scenarios = [
        # Fetch the default url for server object
        (
            'Default Server Node url', dict(
                url='/browser/server/obj/'
            )
        )
    ]

    def setUp(self):
        pass

    def runTest(self, url):
        """ This function will add the server under default server group."""
        url = "{0}{1}/".format(url, utils.SERVER_GROUP)
        # Add service name in the config
        self.server['service'] = "TestDB"
        response = self.tester.post(
            url,
            data=json.dumps(self.server),
            content_type='html/json'
        )
        self.assertEquals(response.status_code, 200)
        response_data = json.loads(response.data.decode('utf-8'))
        self.server_id = response_data['node']['_id']

    def tearDown(self):
        """This function delete the server from SQLite """
        utils.delete_server_with_api(self.tester, self.server_id)

Thanks
Joao

On Fri, Mar 9, 2018 at 8:31 AM Dave Page <[email-masked]> wrote:

> What I would *really* like, is the ability to run individual feature
> tests. That would be very valuable and save me a ton of time.
>
>
>
>> We an HandRolled the loader, and that as some implications. Did anyone
>> try to use a different launcher like pytest or nose instead of the current
>> runner?
>>>>  - Move the Explain SQL from the front-end and now pass the Explain
>>>> plan parameters as a JSON object in the start query call.
>>>>  - Extract the compile_template_name into a function that can be used
>>>> by the different places that try to select the version of the template and
>>>> the server type
>>>>
>>>> Branch
>>>> ------
>>>> master
>>>>
>>>> Details
>>>> -------
>>>>

---

From: Dave Page <[email-masked]>
Date: Fri, 9 Mar 2018 16:04:30 +0000

Hi

On Fri, Mar 9, 2018 at 3:54 PM, Joao De Almeida Pereira <
[email-masked]> wrote:

> Hello,
> Definitely running single tests is something that would be great,
> specially if you are TDDing something waiting 30-40 seconds to get feedback
> is a little cumbersome when the test you are concerned with take less then
> a second.
>
> In the process:
> 1. Write a test
> 2. Make the test pass
> 3. Refactor
>

Sure, makes sense for development. As I spend 99% of my time reviewing and
testing these days, I was just relaying my pain points :-)

> in the process of changing it to pytest.

Pure newbie-ism. I have no objections to changing to something else, if it
reduces our tech debt.

> neat libraries like: https://github.com/grappa-py/grappa, https://
> github.com/ActivisionGameScience/assertpy, https://github.com/dgilland/
> verify. Personally I really like the syntax of Grapa, but the Veridfy one
> is pretty similar to Jasmine too.
>
> What are your thoughts?
>

Huh, I also really like the grappa syntax. It's nice and readable.

>
>
>
> Note 1: As an example of what our functions would have to look like you
> can see: https://github.com/OriMenashe/pytest-scenario/
> blob/master/tests/test_parametrize.py
> As a example this class:
>

Without a diff, it's hard to be sure, but it looks like the only change was
BaseTestGenerator to object on the first line?

>>> Just as a side question, does anyone else feel the pain of wanting to
>>> run a single test using a IDE or the command line and not being able to?
>>>
>>
>> Not really - the Python and JS tests are so quick I don't really care
>> (and with the Python ones, I can execute for a single module for even more
>> speed).
>>
>>> I understand that testscenarios is one of the problems we have if we
>>> want to move away from this way of running tests.
>>> Any suggestion?
>>>
>>> Thanks
>>> Joao
>>>

---

From: Joao De Almeida Pereira <[email-masked]>
Date: Fri, 09 Mar 2018 16:29:01 +0000

On Fri, Mar 9, 2018 at 11:04 AM Dave Page <[email-masked]> wrote:

>> The other issue that I do not like very much about pytest is the fact
>> that you loose the unittest assertion that is not so bad because there are
>> some neat libraries like: https://github.com/grappa-py/grappa,
>> https://github.com/dgilland/verify. Personally I really like the syntax
>> of Grapa, but the Veridfy one is pretty similar to Jasmine too.
>>
>> What are your thoughts?
>>
>
>> can see:
> Without a diff, it's hard to be sure, but it looks like the only change
> was BaseTestGenerator to object on the first line?
>
See the function signature, that is the cumbersome issue

---

From: Dave Page <[email-masked]>
Date: Fri, 9 Mar 2018 16:30:44 +0000

On Fri, Mar 9, 2018 at 4:29 PM, Joao De Almeida Pereira <
[email-masked]> wrote:

>> Sure, makes sense for development. As I spend 99% of my time reviewing
>> and testing these days, I was just relaying my pain points :-)
>>
>>
>>>
>>> I believe we could archive a better granularity and choosing what test
>>> to run if we used a runner like pytest or nose to do it. What was the
>>> reason behind handrolling a test runner script? I am asking this because in
>>> a previous job I decided to handroll a unittest loader script and that was
>> Pure newbie-ism. I have no objections to changing to something else, if
>> it reduces our tech debt.
>>
>>
>>>
>>> some neat libraries like: https://github.com/grappa-py/grappa, https://
>>> verify. Personally I really like the syntax of Grapa, but the Veridfy
>>> one is pretty similar to Jasmine too.
>>>
>>> What are your thoughts?
>>>
>>

Ahh, yes.

>>>>>>>  - Extract the compile_template_name into a function that can be
>>>>>>> used by the different places that try to select the version of the template
>>>>>>> and the server type
>>>>>>>
>>>>>>> Branch
>>>>>>> ------
>>>>>>> master
>>>>>>>
>>>>>>> Details
>>>>>>> -------