Subject: pgAdmin 4 commit: Support for external tables in GPDB. Fixes #3168

From: Dave Page <[email-masked]>
Date: Fri, 02 Mar 2018 16:49:29 +0000

Support for external tables in GPDB. Fixes #3168

Branch
------
master

Details
-------
https://git.postgresql.org/gitweb?p=pgadmin4.git;a=commitdiff;h=427314cfdfeb96c3a7835eab5a4d638903bc6dc3
Author: Joao Pedro De Almeida Pereira <[email-masked]>

Modified Files
--------------
.../servers/databases/external_tables/__init__.py  | 275 +++++++++++++
.../databases/external_tables/actions/__init__.py  |   0
.../external_tables/actions/get_all_nodes.py       |   4 +
.../databases/external_tables/mapping_utils.py     | 165 ++++++++
.../databases/external_tables/properties.py        |  78 ++++
.../external_tables/reverse_engineer_ddl.py        |  69 ++++
.../static/img/coll-external_table.svg             |   1 +
.../external_tables/static/img/external_table.svg  |   1 +
.../templates/sql/gpdb_5.0_plus/create.sql         |  60 +++
.../templates/sql/gpdb_5.0_plus/get_columns.sql    |  12 +
.../sql/gpdb_5.0_plus/get_table_information.sql    |  22 ++
.../templates/sql/gpdb_5.0_plus/list.sql           |   6 +
.../templates/sql/gpdb_5.0_plus/node.sql           |   5 +
.../databases/external_tables/tests/__init__.py    |   0
.../tests/test_external_tables_module.py           |  99 +++++
.../tests/test_external_tables_view.py             | 428 +++++++++++++++++++++
.../external_tables/tests/test_mapping_utils.py    | 375 ++++++++++++++++++
.../external_tables/tests/test_properties.py       | 156 ++++++++
.../tests/test_reverse_engineer_ddl.py             | 261 +++++++++++++
.../tests/test_sql_template_create_integration.py  |   0
.../templates/table/sql/gpdb_5.0_plus/nodes.sql    |   1 +
web/pgadmin/static/bundle/browser.js               |   1 +
.../databases/external_tables/external_tables.js   |  88 +++++
.../servers/databases/external_tables/index.js     |  18 +
web/pgadmin/tools/sqleditor/__init__.py            |   1 +
.../external_tables/external_tables_spec.js        |  56 +++
web/webpack.config.js                              |   9 +
web/webpack.shim.js                                |   3 +
28 files changed, 2194 insertions(+)

---

From: Neel Patel <[email-masked]>
Date: Tue, 6 Mar 2018 12:59:07 +0530

Hi Joao,

I ran the testsuite in windows 10 with Python 3.4 and it fails for external
tables. Linux it is working fine. Let me know if I miss anything.

Please check the below logs.

python runtests.py --pkg browser --exclude feature_tests

########

======================================================================
ERROR: runTest
(pgadmin.browser.server_groups.servers.databases.schemas.tables.tests.test_template_create.TestTemplateCreate)
When rendering GreenPlum 5.3 template, when no distribution is present,
when no primary key is present, it returns "DISTRIBUTED RANDOMLY"
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\servers\databases\schemas\tables\tests\test_template_create.py",
line 99, in runTest
    self.template_path, **self.input_parameters)
  File
"C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\flask\templating.py",
line 133, in render_template
    return
_render(ctx.app.jinja_env.get_or_select_template(template_name_or_list),
  File
"C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\jinja2\environment.py",
line 830, in get_or_select_template
    return self.get_template(template_name_or_list, parent, globals)
  File
"C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\jinja2\environment.py",
line 791, in get_template
    return self._load_template(name, self.make_globals(globals))
  File
"C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\jinja2\environment.py",
line 765, in _load_template
    template = self.loader.load(self, name, globals)
  File
"C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\jinja2\loaders.py",
line 113, in load
    source, filename, uptodate = self.get_source(environment, name)
  File
"C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\flask\templating.py",
line 57, in get_source
    return self._get_source_fast(environment, template)
  File
"C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\flask\templating.py",
line 85, in _get_source_fast
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: table\sql\gpdb_5.0_plus\create.sql

======================================================================
ERROR: runTest
(pgadmin.browser.server_groups.servers.databases.schemas.tables.tests.test_template_create.TestTemplateCreate)
When rendering GreenPlum 5.3 template, when no distribution is present,
when primary key is present, it returns "DISTRIBUTED BY (attr_primary_key)"
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\servers\databases\schemas\tables\tests\test_template_create.py",
line 99, in runTest
    self.template_path, **self.input_parameters)
  File
"C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\flask\templating.py",
line 133, in render_template
    return
_render(ctx.app.jinja_env.get_or_select_template(template_name_or_list),
  File
"C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\jinja2\environment.py",
line 830, in get_or_select_template
    return self.get_template(template_name_or_list, parent, globals)
  File
"C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\jinja2\environment.py",
line 791, in get_template
    return self._load_template(name, self.make_globals(globals))
  File
"C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\jinja2\environment.py",
line 765, in _load_template
    template = self.loader.load(self, name, globals)
  File
"C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\jinja2\loaders.py",
line 113, in load
    source, filename, uptodate = self.get_source(environment, name)
  File
"C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\flask\templating.py",
line 57, in get_source
    return self._get_source_fast(environment, template)
  File
"C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\flask\templating.py",
line 85, in _get_source_fast
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: table\sql\gpdb_5.0_plus\create.sql

======================================================================
ERROR: runTest
(pgadmin.browser.server_groups.servers.databases.schemas.tables.tests.test_template_create.TestTemplateCreate)
When rendering GreenPlum 5.3 template, when distribution is present, it
returns "DISTRIBUTED BY (attr1, attr2, attr4)"
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\servers\databases\schemas\tables\tests\test_template_create.py",
line 99, in runTest
    self.template_path, **self.input_parameters)
  File
"C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\flask\templating.py",
line 133, in render_template
    return
_render(ctx.app.jinja_env.get_or_select_template(template_name_or_list),
  File
"C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\jinja2\environment.py",
line 830, in get_or_select_template
    return self.get_template(template_name_or_list, parent, globals)
  File
"C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\jinja2\environment.py",
line 791, in get_template
    return self._load_template(name, self.make_globals(globals))
  File
"C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\jinja2\environment.py",
line 765, in _load_template
    template = self.loader.load(self, name, globals)
  File
"C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\jinja2\loaders.py",
line 113, in load
    source, filename, uptodate = self.get_source(environment, name)
  File
"C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\flask\templating.py",
line 57, in get_source
    return self._get_source_fast(environment, template)
  File
"C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\flask\templating.py",
line 85, in _get_source_fast
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: table\sql\gpdb_5.0_plus\create.sql

======================================================================
ERROR: runTest
(pgadmin.browser.tests.test_change_password.ChangePasswordTestCase)
TestCase for Changing Valid_Password
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"C:\Projects\pgadmin4\web\pgadmin\browser\tests\test_change_password.py",
line 91, in runTest
    user_id = json.loads(response.data.decode('utf-8'))['id']
KeyError: 'id'

======================================================================
FAIL: runTest
(pgadmin.browser.server_groups.servers.databases.external_tables.tests.test_external_tables_view.TestExternalTablesView)
#nodes When retrieving the nodes and the database does not have external
tables, it return no child nodes and status 200
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python34\Lib\unittest\mock.py", line 1142, in patched
    return func(*args, **keywargs)
  File
"C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\servers\databases\external_tables\tests\test_external_tables_view.py",
line 314, in runTest
    self.__test_nodes()
  File "C:\Python34\Lib\unittest\mock.py", line 1142, in patched
    return func(*args, **keywargs)
  File
"C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\servers\databases\external_tables\tests\test_external_tables_view.py",
line 367, in __test_nodes
    self.expect_render_template_called_with
  File "C:\Python34\Lib\unittest\mock.py", line 777, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: Expected call: render_template('sql/#gpdb#80323#/list.sql')
Actual call: render_template('sql/#gpdb#80323#\\list.sql')

======================================================================
FAIL: runTest
(pgadmin.browser.server_groups.servers.databases.external_tables.tests.test_external_tables_view.TestExternalTablesView)
#nodes When retrieving the nodes and an error happens while executing the
query, it return an internal server error and status 500
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python34\Lib\unittest\mock.py", line 1142, in patched
    return func(*args, **keywargs)
  File
"C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\servers\databases\external_tables\tests\test_external_tables_view.py",
line 314, in runTest
    self.__test_nodes()
  File "C:\Python34\Lib\unittest\mock.py", line 1142, in patched
    return func(*args, **keywargs)
  File
"C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\servers\databases\external_tables\tests\test_external_tables_view.py",
line 367, in __test_nodes
    self.expect_render_template_called_with
  File "C:\Python34\Lib\unittest\mock.py", line 777, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: Expected call: render_template('sql/#gpdb#80323#/list.sql')
Actual call: render_template('sql/#gpdb#80323#\\list.sql')

======================================================================
FAIL: runTest
(pgadmin.browser.server_groups.servers.databases.external_tables.tests.test_external_tables_view.TestExternalTablesView)
#nodes When retrieving the nodes and the database has 2 external tables, it
return 2 child nodes and status 200
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python34\Lib\unittest\mock.py", line 1142, in patched
    return func(*args, **keywargs)
  File
"C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\servers\databases\external_tables\tests\test_external_tables_view.py",
line 314, in runTest
    self.__test_nodes()
  File "C:\Python34\Lib\unittest\mock.py", line 1142, in patched
    return func(*args, **keywargs)
  File
"C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\servers\databases\external_tables\tests\test_external_tables_view.py",
line 367, in __test_nodes
    self.expect_render_template_called_with
  File "C:\Python34\Lib\unittest\mock.py", line 777, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: Expected call: render_template('sql/#gpdb#80323#/list.sql')
Actual call: render_template('sql/#gpdb#80323#\\list.sql')

======================================================================
FAIL: runTest
(pgadmin.browser.server_groups.servers.databases.external_tables.tests.test_external_tables_view.TestExternalTablesView)
#node When retrieving the information about 1 external table and an error
happens while executing the query, it return an internal server error and
status 500
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python34\Lib\unittest\mock.py", line 1142, in patched
    return func(*args, **keywargs)
  File
"C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\servers\databases\external_tables\tests\test_external_tables_view.py",
line 316, in runTest
    self.__test_node()
  File "C:\Python34\Lib\unittest\mock.py", line 1142, in patched
    return func(*args, **keywargs)
  File
"C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\servers\databases\external_tables\tests\test_external_tables_view.py",
line 393, in __test_node
    **self.expect_render_template_called_with
  File "C:\Python34\Lib\unittest\mock.py", line 777, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: Expected call: render_template(external_table_id=11,
template_name_or_list='sql/#gpdb#80323#/node.sql')
Actual call: render_template(external_table_id=11,
template_name_or_list='sql/#gpdb#80323#\\node.sql')

======================================================================
FAIL: runTest
(pgadmin.browser.server_groups.servers.databases.external_tables.tests.test_external_tables_view.TestExternalTablesView)
#node When retrieving the information about 1 external table and table does
not exist, it return an error message and status 404
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python34\Lib\unittest\mock.py", line 1142, in patched
    return func(*args, **keywargs)
  File
"C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\servers\databases\external_tables\tests\test_external_tables_view.py",
line 316, in runTest
    self.__test_node()
  File "C:\Python34\Lib\unittest\mock.py", line 1142, in patched
    return func(*args, **keywargs)
  File
"C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\servers\databases\external_tables\tests\test_external_tables_view.py",
line 393, in __test_node
    **self.expect_render_template_called_with
  File "C:\Python34\Lib\unittest\mock.py", line 777, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: Expected call: render_template(external_table_id=11,
template_name_or_list='sql/#gpdb#80323#/node.sql')
Actual call: render_template(external_table_id=11,
template_name_or_list='sql/#gpdb#80323#\\node.sql')

======================================================================
FAIL: runTest
(pgadmin.browser.server_groups.servers.databases.external_tables.tests.test_external_tables_view.TestExternalTablesView)
#nodes When retrieving the information about 1 external table and the table
exists, it return external node information and status 200
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python34\Lib\unittest\mock.py", line 1142, in patched
    return func(*args, **keywargs)
  File
"C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\servers\databases\external_tables\tests\test_external_tables_view.py",
line 316, in runTest
    self.__test_node()
  File "C:\Python34\Lib\unittest\mock.py", line 1142, in patched
    return func(*args, **keywargs)
  File
"C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\servers\databases\external_tables\tests\test_external_tables_view.py",
line 393, in __test_node
    **self.expect_render_template_called_with
  File "C:\Python34\Lib\unittest\mock.py", line 777, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: Expected call: render_template(external_table_id=11,
template_name_or_list='sql/#gpdb#80323#/node.sql')
Actual call: render_template(external_table_id=11,
template_name_or_list='sql/#gpdb#80323#\\node.sql')

======================================================================
FAIL: runTest
(pgadmin.browser.server_groups.servers.databases.external_tables.tests.test_external_tables_view.TestExternalTablesView)
#properties When retrieving the properties of a external table and the
table exists, it return the properties and status 200
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python34\Lib\unittest\mock.py", line 1142, in patched
    return func(*args, **keywargs)
  File
"C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\servers\databases\external_tables\tests\test_external_tables_view.py",
line 320, in runTest
    self.__test_properties()
  File "C:\Python34\Lib\unittest\mock.py", line 1142, in patched
    return func(*args, **keywargs)
  File
"C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\servers\databases\external_tables\tests\test_external_tables_view.py",
line 419, in __test_properties
    **self.expect_render_template_called_with
  File "C:\Python34\Lib\unittest\mock.py", line 777, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: Expected call:
render_template(template_name_or_list='sql/#gpdb#80323#/get_table_information.sql',
table_oid=11)
Actual call:
render_template(template_name_or_list='sql/#gpdb#80323#\\get_table_information.sql',
table_oid=11)

----------------------------------------------------------------------
Ran 256 tests in 77.123s

FAILED (failures=7, errors=4, skipped=25)

======================================================================
Test Result Summary
======================================================================

PostgreSQL 9.6:

        234 tests passed
        3 tests failed:
                TestExternalTablesView (#node When retrieving the
information about 1 external table and table does not exist, it return an
error message and status 404,
                                        #node When retrieving the
information about 1 external table and an error happens while executing the
query, it return an internal server error and status 500,
                                        #nodes When retrieving the nodes
and the database has 2 external tables, it return 2 child nodes and status
200,
                                        #nodes When retrieving the
information about 1 external table and the table exists, it return external
node information and status 200,
                                        #nodes When retrieving the nodes
and an error happens while executing the query, it return an internal
server error and status 500,
                                        #properties When retrieving the
properties of a external table and the table exists, it return the
properties and status 200,
                                        #nodes When retrieving the nodes
and the database does not have external tables, it return no child nodes
and status 200)
                ChangePasswordTestCase (TestCase for Changing
Valid_Password)
                TestTemplateCreate (When rendering GreenPlum 5.3 template,
when no distribution is present, when no primary key is present, it returns
"DISTRIBUTED RANDOMLY",
                                        When rendering GreenPlum 5.3
template, when distribution is present, it returns "DISTRIBUTED BY (attr1,
attr2, attr4)",
                                        When rendering GreenPlum 5.3
template, when no distribution is present, when primary key is present, it
returns "DISTRIBUTED BY (attr_primary_key)")
        19 tests skipped:
                ResourceGroupsGetTestCase (Get resource groups)
                TableSpaceDeleteTestCase (Check Tablespace Node)
                ResourceGroupsDeleteTestCase (Delete resource groups)
                SynonymDeleteTestCase (Fetch synonym Node URL)
                TestSSLConnection (Test for SSL connection)
                PackageAddTestCase (Fetch Package Node URL)
                SynonymPutTestCase (Fetch synonym Node URL)
                TablespaceGetTestCase (Check Tablespace Node)
                SynonymAddTestCase (Default Node URL)
                PackagePutTestCase (Fetch Package Node URL)
                TableAddTestCase (Create Range partitioned table with 2
partitions,
                                        Create List partitioned table with
2 partitions)
                PackageGetTestCase (Fetch Package Node URL)
                PackageDeleteTestCase (Fetch Package Node URL)
                TableSpaceAddTestCase (Check Tablespace Node)
                ResourceGroupsAddTestCase (Add resource groups)
                SynonymGetTestCase (Fetch synonym Node URL)
                ResourceGroupsPutTestCase (Put resource groups)
                TableSpaceUpdateTestCase (Check Tablespace Node)
                TableUpdateTestCase (Attach partition to existing range
partitioned table,
                                        Detach partition from existing
range partitioned table,
                                        Create partitions of existing range
partitioned table,
                                        Detach partition from existing list
partitioned table,
                                        Create partitions of existing list
partitioned table,
                                        Attach partition to existing list
partitioned table)

======================================================================

########

Thanks,
Neel Patel

---

From: Joao De Almeida Pereira <[email-masked]>
Date: Tue, 06 Mar 2018 14:55:49 +0000

Hello Neel,

You can find attached the corrections of the path's needed for windows. The
fix should correct TestExternalTablesView and TestTemplateCreate but for
the ChangePasswordTestCase I need more information to help you out. We need
to understand what is the response that the endpoint /user_management/user
is returning.

Thanks
Joao

On Tue, Mar 6, 2018 at 2:29 AM Neel Patel <[email-masked]>
wrote:

> Hi Joao,
>
> I ran the testsuite in windows 10 with Python 3.4 and it fails for
> external tables. Linux it is working fine. Let me know if I miss anything.
>
> #nodes When retrieving the nodes and the database has 2 external tables,
> it return 2 child nodes and status 200
> #node When retrieving the information about 1 external table and table
> does not exist, it return an error message and status 404
> #nodes When retrieving the information about 1 external table and the
> table exists, it return external node information and status 200
>                                         Create partitions of existing
> range partitioned table,
> list partitioned table,
diff --git a/web/pgadmin/browser/server_groups/servers/databases/external_tables/tests/test_external_tables_view.py b/web/pgadmin/browser/server_groups/servers/databases/external_tables/tests/test_external_tables_view.py
index acedc61b..3f08e40e 100644
--- a/web/pgadmin/browser/server_groups/servers/databases/external_tables/tests/test_external_tables_view.py
+++ b/web/pgadmin/browser/server_groups/servers/databases/external_tables/tests/test_external_tables_view.py
@@ -7,6 +7,7 @@
 #
 ##########################################################################
 
+import os
 import sys
 
 from pgadmin.browser.server_groups.servers.databases.external_tables import \
@@ -68,7 +69,9 @@ class TestExternalTablesView(BaseTestGenerator):
              connection=MagicMock(execute_2darray=MagicMock()),
              execute_2darray_return_value=(True, dict(rows=[])),
 
-             expect_render_template_called_with='sql/#gpdb#80323#/list.sql',
+             expect_render_template_called_with=os.path.join('sql',
+                                                             '#gpdb#80323#'
+                                                             , 'list.sql'),
              expected_make_json_response_called_with=dict(
                  data=[],
                  status=200
@@ -90,7 +93,9 @@ class TestExternalTablesView(BaseTestGenerator):
              connection=MagicMock(execute_2darray=MagicMock()),
              execute_2darray_return_value=(False, 'Some error message'),
 
-             expect_render_template_called_with='sql/#gpdb#80323#/list.sql',
+             expect_render_template_called_with=os.path.join('sql',
+                                                             '#gpdb#80323#',
+                                                             'list.sql'),
              expected_internal_server_error_called_with=dict(
                  errormsg='Some error message'
              ),
@@ -122,7 +127,9 @@ class TestExternalTablesView(BaseTestGenerator):
                  ]
              )),
 
-             expect_render_template_called_with='sql/#gpdb#80323#/list.sql',
+             expect_render_template_called_with=os.path.join('sql',
+                                                             '#gpdb#80323#',
+                                                             'list.sql'),
              expected_make_json_response_called_with=dict(
                  data=[
                      {
@@ -167,7 +174,9 @@ class TestExternalTablesView(BaseTestGenerator):
              execute_2darray_return_value=(False, 'Some error message'),
 
              expect_render_template_called_with=dict(
-                 template_name_or_list='sql/#gpdb#80323#/node.sql',
+                 template_name_or_list=os.path.join('sql',
+                                                    '#gpdb#80323#',
+                                                    'node.sql'),
                  external_table_id=11
              ),
              expected_internal_server_error_called_with=dict(
@@ -192,7 +201,9 @@ class TestExternalTablesView(BaseTestGenerator):
              execute_2darray_return_value=(True, dict(rows=[])),
 
              expect_render_template_called_with=dict(
-                 template_name_or_list='sql/#gpdb#80323#/node.sql',
+                 template_name_or_list=os.path.join('sql',
+                                                    '#gpdb#80323#',
+                                                    'node.sql'),
                  external_table_id=11
              ),
              expected_make_json_response_called_with=dict(
@@ -229,7 +240,9 @@ class TestExternalTablesView(BaseTestGenerator):
              )),
 
              expect_render_template_called_with=dict(
-                 template_name_or_list='sql/#gpdb#80323#/node.sql',
+                 template_name_or_list=os.path.join('sql',
+                                                    '#gpdb#80323#',
+                                                    'node.sql'),
                  external_table_id=11
              ),
              expected_make_json_response_called_with=dict(
@@ -283,8 +296,11 @@ class TestExternalTablesView(BaseTestGenerator):
              )),
 
              expect_render_template_called_with=dict(
-                 template_name_or_list='sql/#gpdb#80323#/'
-                                       'get_table_information.sql',
+                 template_name_or_list=os.path.join(
+                     'sql',
+                     '#gpdb#80323#',
+                     'get_table_information.sql'
+                 ),
                  table_oid=11
              ),
              expected_make_response_called_with=dict(
diff --git a/web/pgadmin/browser/server_groups/servers/databases/schemas/tables/tests/test_template_create.py b/web/pgadmin/browser/server_groups/servers/databases/schemas/tables/tests/test_template_create.py
index 25d58925..cd765aaf 100644
--- a/web/pgadmin/browser/server_groups/servers/databases/schemas/tables/tests/test_template_create.py
+++ b/web/pgadmin/browser/server_groups/servers/databases/schemas/tables/tests/test_template_create.py
@@ -117,20 +117,26 @@ class FakeApp(Flask):
         self.jinja_env.filters['qtTypeIdent'] = driver.qtTypeIdent
         self.jinja_loader = ChoiceLoader([
             FileSystemLoader(
-                os.path.dirname(
-                    os.path.realpath(__file__)) + '/../templates/'
+                os.path.join(os.path.dirname(
+                    os.path.realpath(__file__)
+                ), os.pardir, 'templates')
             ),
             FileSystemLoader(
-                os.path.dirname(
-                    os.path.realpath(__file__)) + '/../../templates/'
+                os.path.join(
+                    os.path.dirname(
+                        os.path.realpath(__file__)
+                    ), os.pardir, os.pardir, 'templates')
             ),
             FileSystemLoader(
-                os.path.dirname(
-                    os.path.realpath(__file__)) + '/../../types/templates/'
+                os.path.join(os.path.dirname(
+                    os.path.realpath(__file__))
+                    , os.pardir, os.pardir, 'types', 'templates')
             ),
             FileSystemLoader(
-                os.path.dirname(
-                    os.path.realpath(__file__)) + '/../../../../templates/'
+                os.path.join(os.path.dirname(
+                    os.path.realpath(__file__))
+                    , os.pardir, os.pardir, os.pardir, os.pardir,
+                    'templates')
             ),
         ]
         )

---

From: Murtuza Zabuawala <[email-masked]>
Date: Mon, 12 Mar 2018 19:52:15 +0530

Hi Dave,

Joao's patch is pending, would you please do the needful?
I also encounter similar issue on Windows while running tests.

--
Regards,
Murtuza Zabuawala
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, Mar 6, 2018 at 8:25 PM, Joao De Almeida Pereira <
[email-masked]> wrote:

> Hello Neel,
>
> You can find attached the corrections of the path's needed for windows.
> The fix should correct TestExternalTablesView and TestTemplateCreate but
> for the ChangePasswordTestCase I need more information to help you out. We
> need to understand what is the response that the endpoint
> /user_management/user is returning.
>
> Thanks
> Joao
>
>> ERROR: runTest (pgadmin.browser.server_groups.servers.databases.
>> schemas.tables.tests.test_template_create.TestTemplateCreate)
>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\
>> servers\databases\schemas\tables\tests\test_template_create.py", line
>> 99, in runTest
>>   File "C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\flask\templating.py",
>>     return _render(ctx.app.jinja_env.get_or_select_template(template_
>> name_or_list),
>>   File "C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\jinja2\environment.py",
>>   File "C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\jinja2\environment.py",
>>   File "C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\jinja2\environment.py",
>>   File "C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\jinja2\loaders.py",
>> line 113, in load
>>   File "C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\flask\templating.py",
>> line 57, in get_source
>>   File "C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\flask\templating.py",
>> ERROR: runTest (pgadmin.browser.server_groups.servers.databases.
>> schemas.tables.tests.test_template_create.TestTemplateCreate)
>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\
>> servers\databases\schemas\tables\tests\test_template_create.py", line
>> 99, in runTest
>>   File "C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\flask\templating.py",
>>     return _render(ctx.app.jinja_env.get_or_select_template(template_
>> name_or_list),
>>   File "C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\jinja2\environment.py",
>>   File "C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\jinja2\environment.py",
>>   File "C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\jinja2\environment.py",
>>   File "C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\jinja2\loaders.py",
>> line 113, in load
>>   File "C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\flask\templating.py",
>> line 57, in get_source
>>   File "C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\flask\templating.py",
>> ERROR: runTest (pgadmin.browser.server_groups.servers.databases.
>> schemas.tables.tests.test_template_create.TestTemplateCreate)
>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\
>> servers\databases\schemas\tables\tests\test_template_create.py", line
>> 99, in runTest
>>   File "C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\flask\templating.py",
>>     return _render(ctx.app.jinja_env.get_or_select_template(template_
>> name_or_list),
>>   File "C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\jinja2\environment.py",
>>   File "C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\jinja2\environment.py",
>>   File "C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\jinja2\environment.py",
>>   File "C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\jinja2\loaders.py",
>> line 113, in load
>>   File "C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\flask\templating.py",
>> line 57, in get_source
>>   File "C:\Projects\venv_pgadmin4_py_3_4\lib\site-packages\flask\templating.py",
>> ERROR: runTest (pgadmin.browser.tests.test_change_password.
>> ChangePasswordTestCase)
>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\tests\test_change_password.py",
>> line 91, in runTest
>> FAIL: runTest (pgadmin.browser.server_groups.servers.databases.
>> external_tables.tests.test_external_tables_view.TestExternalTablesView)
>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\
>> servers\databases\external_tables\tests\test_external_tables_view.py",
>> line 314, in runTest
>>     self.__test_nodes()
>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\
>> servers\databases\external_tables\tests\test_external_tables_view.py",
>> FAIL: runTest (pgadmin.browser.server_groups.servers.databases.
>> external_tables.tests.test_external_tables_view.TestExternalTablesView)
>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\
>> servers\databases\external_tables\tests\test_external_tables_view.py",
>> line 314, in runTest
>>     self.__test_nodes()
>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\
>> servers\databases\external_tables\tests\test_external_tables_view.py",
>> FAIL: runTest (pgadmin.browser.server_groups.servers.databases.
>> external_tables.tests.test_external_tables_view.TestExternalTablesView)
>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\
>> servers\databases\external_tables\tests\test_external_tables_view.py",
>> line 314, in runTest
>>     self.__test_nodes()
>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\
>> servers\databases\external_tables\tests\test_external_tables_view.py",
>> FAIL: runTest (pgadmin.browser.server_groups.servers.databases.
>> external_tables.tests.test_external_tables_view.TestExternalTablesView)
>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\
>> servers\databases\external_tables\tests\test_external_tables_view.py",
>> line 316, in runTest
>>     self.__test_node()
>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\
>> servers\databases\external_tables\tests\test_external_tables_view.py",
>> line 393, in __test_node
>> FAIL: runTest (pgadmin.browser.server_groups.servers.databases.
>> external_tables.tests.test_external_tables_view.TestExternalTablesView)
>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\
>> servers\databases\external_tables\tests\test_external_tables_view.py",
>> line 316, in runTest
>>     self.__test_node()
>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\
>> servers\databases\external_tables\tests\test_external_tables_view.py",
>> line 393, in __test_node
>> FAIL: runTest (pgadmin.browser.server_groups.servers.databases.
>> external_tables.tests.test_external_tables_view.TestExternalTablesView)
>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\
>> servers\databases\external_tables\tests\test_external_tables_view.py",
>> line 316, in runTest
>>     self.__test_node()
>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\
>> servers\databases\external_tables\tests\test_external_tables_view.py",
>> line 393, in __test_node
>> FAIL: runTest (pgadmin.browser.server_groups.servers.databases.
>> external_tables.tests.test_external_tables_view.TestExternalTablesView)
>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\
>> servers\databases\external_tables\tests\test_external_tables_view.py",
>> line 320, in runTest
>>     self.__test_properties()
>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\
>> servers\databases\external_tables\tests\test_external_tables_view.py",
>> AssertionError: Expected call: render_template(template_name_
>> or_list='sql/#gpdb#80323#/get_table_information.sql', table_oid=11)
>> Actual call: render_template(template_name_or_list='sql/#gpdb#80323#\\get_table_information.sql',
>> table_oid=11)
>>
>>                 TestTemplateCreate (When rendering GreenPlum 5.3
>> template, when no distribution is present, when no primary key is present,
>> it returns "DISTRIBUTED RANDOMLY",
>>                                         Create List partitioned table
>> with 2 partitions)

---

From: Murtuza Zabuawala <[email-masked]>
Date: Mon, 12 Mar 2018 20:10:22 +0530

Hi Joao,

Can we make tests to skip if db is not greenplum?
Like we are doing for resource group
"..web/pgadmin/browser/server_groups/servers/resource_groups/tests".

On Mon, Mar 12, 2018 at 7:52 PM, Murtuza Zabuawala <
[email-masked]> wrote:

>>> ERROR: runTest (pgadmin.browser.server_groups.servers.databases.schemas.
>>> tables.tests.test_template_create.TestTemplateCreate)
>>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\serv
>>> ers\databases\schemas\tables\tests\test_template_create.py", line 99,
>>> in runTest
>>>     return _render(ctx.app.jinja_env.get_or_select_template(template_na
>>> me_or_list),
>>> ERROR: runTest (pgadmin.browser.server_groups.servers.databases.schemas.
>>> tables.tests.test_template_create.TestTemplateCreate)
>>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\serv
>>> ers\databases\schemas\tables\tests\test_template_create.py", line 99,
>>> in runTest
>>>     return _render(ctx.app.jinja_env.get_or_select_template(template_na
>>> me_or_list),
>>> ERROR: runTest (pgadmin.browser.server_groups.servers.databases.schemas.
>>> tables.tests.test_template_create.TestTemplateCreate)
>>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\serv
>>> ers\databases\schemas\tables\tests\test_template_create.py", line 99,
>>> in runTest
>>>     return _render(ctx.app.jinja_env.get_or_select_template(template_na
>>> me_or_list),
>>> ERROR: runTest (pgadmin.browser.tests.test_ch
>>> ange_password.ChangePasswordTestCase)
>>> FAIL: runTest (pgadmin.browser.server_groups.servers.databases.external_
>>> tables.tests.test_external_tables_view.TestExternalTablesView)
>>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\serv
>>> ers\databases\external_tables\tests\test_external_tables_view.py", line
>>> 314, in runTest
>>>     self.__test_nodes()
>>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\serv
>>> ers\databases\external_tables\tests\test_external_tables_view.py", line
>>> 367, in __test_nodes
>>> FAIL: runTest (pgadmin.browser.server_groups.servers.databases.external_
>>> tables.tests.test_external_tables_view.TestExternalTablesView)
>>> #nodes When retrieving the nodes and an error happens while executing
>>> the query, it return an internal server error and status 500
>>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\serv
>>> ers\databases\external_tables\tests\test_external_tables_view.py", line
>>> 314, in runTest
>>>     self.__test_nodes()
>>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\serv
>>> ers\databases\external_tables\tests\test_external_tables_view.py", line
>>> 367, in __test_nodes
>>> FAIL: runTest (pgadmin.browser.server_groups.servers.databases.external_
>>> tables.tests.test_external_tables_view.TestExternalTablesView)
>>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\serv
>>> ers\databases\external_tables\tests\test_external_tables_view.py", line
>>> 314, in runTest
>>>     self.__test_nodes()
>>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\serv
>>> ers\databases\external_tables\tests\test_external_tables_view.py", line
>>> 367, in __test_nodes
>>> FAIL: runTest (pgadmin.browser.server_groups.servers.databases.external_
>>> tables.tests.test_external_tables_view.TestExternalTablesView)
>>> #node When retrieving the information about 1 external table and an
>>> error happens while executing the query, it return an internal server error
>>> and status 500
>>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\serv
>>> ers\databases\external_tables\tests\test_external_tables_view.py", line
>>> 316, in runTest
>>>     self.__test_node()
>>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\serv
>>> ers\databases\external_tables\tests\test_external_tables_view.py", line
>>> 393, in __test_node
>>> FAIL: runTest (pgadmin.browser.server_groups.servers.databases.external_
>>> tables.tests.test_external_tables_view.TestExternalTablesView)
>>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\serv
>>> ers\databases\external_tables\tests\test_external_tables_view.py", line
>>> 316, in runTest
>>>     self.__test_node()
>>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\serv
>>> ers\databases\external_tables\tests\test_external_tables_view.py", line
>>> 393, in __test_node
>>> FAIL: runTest (pgadmin.browser.server_groups.servers.databases.external_
>>> tables.tests.test_external_tables_view.TestExternalTablesView)
>>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\serv
>>> ers\databases\external_tables\tests\test_external_tables_view.py", line
>>> 316, in runTest
>>>     self.__test_node()
>>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\serv
>>> ers\databases\external_tables\tests\test_external_tables_view.py", line
>>> 393, in __test_node
>>> FAIL: runTest (pgadmin.browser.server_groups.servers.databases.external_
>>> tables.tests.test_external_tables_view.TestExternalTablesView)
>>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\serv
>>> ers\databases\external_tables\tests\test_external_tables_view.py", line
>>> 320, in runTest
>>>     self.__test_properties()
>>>   File "C:\Projects\pgadmin4\web\pgadmin\browser\server_groups\serv
>>> ers\databases\external_tables\tests\test_external_tables_view.py", line
>>> 419, in __test_properties
>>> Actual call: render_template(template_name_
>>> or_list='sql/#gpdb#80323#\\get_table_information.sql', table_oid=11)
>>>
>>>                                         Attach partition to existing
>>> list partitioned table)
>>>
>>>> https://git.postgresql.org/gitweb?p=pgadmin4.git;a=commitdif
>>>> f;h=427314cfdfeb96c3a7835eab5a4d638903bc6dc3

---

From: Joao De Almeida Pereira <[email-masked]>
Date: Mon, 12 Mar 2018 21:15:32 +0000

Hello Murtuza,
Yes I think it makes sense

Thanks
Joao

On Mon, Mar 12, 2018 at 10:40 AM Murtuza Zabuawala <
[email-masked]> wrote:

>>> The fix should correct TestExternalTablesView and TestTemplateCreate
>>> but for the ChangePasswordTestCase I need more information to help you out.
>>> We need to understand what is the response that the endpoint
>>>> #nodes When retrieving the nodes and the database does not have
>>>> external tables, it return no child nodes and status 200
>>>> render_template('sql/#gpdb#80323#/list.sql')
>>>> render_template('sql/#gpdb#80323#/list.sql')
>>>> #nodes When retrieving the nodes and the database has 2 external
>>>> tables, it return 2 child nodes and status 200
>>>> render_template('sql/#gpdb#80323#/list.sql')
>>>> nodes and the database has 2 external tables, it return 2 child nodes and
>>>> status 200,
>>>> nodes and an error happens while executing the query, it return an internal
>>>> nodes and the database does not have external tables, it return no child
>>>> nodes and status 200)