Subject: feature test timeouts

From: George Gelashvili <[email-masked]>
Date: Fri, 26 May 2017 18:26:38 -0400

Hello Hackers,

We are running tests including feature tests in Circle CI, and have been
seeing some flakiness due to what we believe are insufficient wait-for
timeouts.

This shouldn't increase the overall test time, but only allow waiting
longer for an element to appear, if needed.

We've also split out the app start timeout since that seemed to need more
time on CI.

Patch attached.

Thanks!
Joao, Matt, and George
diff --git a/web/regression/feature_utils/pgadmin_page.py b/web/regression/feature_utils/pgadmin_page.py
index b1dfa182..b9abe7ca 100644
--- a/web/regression/feature_utils/pgadmin_page.py
+++ b/web/regression/feature_utils/pgadmin_page.py
@@ -26,7 +26,8 @@ class PgadminPage:
     def __init__(self, driver, app_config):
         self.driver = driver
         self.app_config = app_config
-        self.timeout = 10
+        self.timeout = 30
+        self.app_start_timeout = 60
 
     def reset_layout(self):
         self.click_element(self.find_by_partial_link_text("File"))
@@ -178,8 +179,10 @@ class PgadminPage:
                 driver.refresh()
                 return False
 
-        self._wait_for("app to start", page_shows_app)
+        self._wait_for("app to start", page_shows_app, self.app_start_timeout)
 
-    def _wait_for(self, waiting_for_message, condition_met_function):
-        return WebDriverWait(self.driver, self.timeout, 0.01).until(condition_met_function,
+    def _wait_for(self, waiting_for_message, condition_met_function, timeout = None):
+        if timeout is None:
+            timeout = self.timeout
+        return WebDriverWait(self.driver, timeout, 0.01).until(condition_met_function,
                                                                     "Timed out waiting for " + waiting_for_message)

-- 
Sent via pgadmin-hackers mailing list ([email-masked])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgadmin-hackers

---

From: Dave Page <[email-masked]>
Date: Sat, 27 May 2017 14:17:19 -0400

Thanks, patch applied.

On Fri, May 26, 2017 at 6:26 PM, George Gelashvili
<[email-masked]> wrote:
> This shouldn't increase the overall test time, but only allow waiting longer
> for an element to appear, if needed.
>

-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

---

From: Murtuza Zabuawala <[email-masked]>
Date: Mon, 29 May 2017 11:53:57 +0530

Hi Dave,

With latest pull, PG data-type test fails on my system,

======================================================================
ERROR: runTest (pgadmin.feature_tests.pg_datatype_validation_test.
PGDataypeFeatureTest)
Test checks for PG data-types output
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/edb/Documents/projects/pgadmin4/web/pgadmin/
feature_tests/pg_datatype_validation_test.py", line 44, in runTest
    self._check_datatype()
  File "/Users/edb/Documents/projects/pgadmin4/web/pgadmin/
feature_tests/pg_datatype_validation_test.py", line 97, in _check_datatype
    self.page.fill_codemirror_area_with(query)
  File "/Users/edb/Documents/projects/pgadmin4/web/regression/feature_utils/pgadmin_page.py",
line 122, in fill_codemirror_area_with
    "//pre[contains(@class,'CodeMirror-line')]/../../../*[
contains(@class,'CodeMirror-code')]").click()
  File "/Users/edb/Documents/projects/pgadmin4/web/regression/feature_utils/pgadmin_page.py",
line 87, in find_by_xpath
    return self.wait_for_element(lambda driver:
driver.find_element_by_xpath(xpath))
  File "/Users/edb/Documents/projects/pgadmin4/web/regression/feature_utils/pgadmin_page.py",
line 152, in wait_for_element
    return self._wait_for("element to exist", element_if_it_exists)
  File "/Users/edb/Documents/projects/pgadmin4/web/regression/feature_utils/pgadmin_page.py",
line 188, in _wait_for
    "Timed out waiting for " + waiting_for_message)
  File "/Users/edb/Documents/projects/venv_pg4/lib/python3.
5/site-packages/selenium/webdriver/support/wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: Timed out waiting for
element to exist

----------------------------------------------------------------------
Ran 6 tests in 170.026s

FAILED (errors=1)

======================================================================
Test Result Summary
======================================================================

PG_95:

5 tests passed
1 test failed:
PGDataypeFeatureTest (Test checks for PG data-types output)
0 tests skipped

======================================================================

Please check output in file: /Users/edb/Documents/projects/
pgadmin4/web/regression/regression.log

--
Regards,
Murtuza Zabuawala
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

---

From: Dave Page <[email-masked]>
Date: Tue, 30 May 2017 09:45:48 +0100

Hi

OK. Do you know why? What's it waiting for? It was working for me (I
always run tests multiple times before committing them, to try to
ensure they're reliable).

On Mon, May 29, 2017 at 7:23 AM, Murtuza Zabuawala
<[email-masked]> wrote:
> ERROR: runTest
> (pgadmin.feature_tests.pg_datatype_validation_test.PGDataypeFeatureTest)
>   File
> "/Users/edb/Documents/projects/pgadmin4/web/pgadmin/feature_tests/pg_datatype_validation_test.py",
> line 44, in runTest
>     self._check_datatype()
>   File
> "/Users/edb/Documents/projects/pgadmin4/web/pgadmin/feature_tests/pg_datatype_validation_test.py",
> line 97, in _check_datatype
>   File
> "/Users/edb/Documents/projects/pgadmin4/web/regression/feature_utils/pgadmin_page.py",
> "//pre[contains(@class,'CodeMirror-line')]/../../../*[contains(@class,'CodeMirror-code')]").click()
>   File
> "/Users/edb/Documents/projects/pgadmin4/web/regression/feature_utils/pgadmin_page.py",
>   File
> "/Users/edb/Documents/projects/pgadmin4/web/regression/feature_utils/pgadmin_page.py",
>   File
> "/Users/edb/Documents/projects/pgadmin4/web/regression/feature_utils/pgadmin_page.py",
> line 188, in _wait_for
>   File
> "/Users/edb/Documents/projects/venv_pg4/lib/python3.5/site-packages/selenium/webdriver/support/wait.py",
> line 80, in until
> Please check output in file:
> /Users/edb/Documents/projects/pgadmin4/web/regression/regression.log
>
>
> --
> Regards,
> Murtuza Zabuawala

---

From: Murtuza Zabuawala <[email-masked]>
Date: Tue, 30 May 2017 16:13:51 +0530

Hi Dave,

I didn't look into the code for it but every time I run the tests it stays
on below screen and fails.

@Khushboo,
Any suggestion?

On Tue, May 30, 2017 at 2:15 PM, Dave Page <[email-masked]> wrote:

> > "/Users/edb/Documents/projects/pgadmin4/web/pgadmin/
> > "/Users/edb/Documents/projects/pgadmin4/web/pgadmin/
> > "/Users/edb/Documents/projects/venv_pg4/lib/python3.

---

From: Khushboo Vashi <[email-masked]>
Date: Tue, 30 May 2017 16:22:42 +0530

Hi Murtuza,

> on below screen and fails.
Harshal is also getting the same issue with his feature test case. The
problem we found that, in the bound time the driver is not able to find the
code-mirror. Me and Harshal looked into this yesterday but couldn't find
the solution.

BTW, I can not reproduce this issue at my end but me and Harshal will look
into this issue on his machine.

Thanks,
Khushboo

>> > "/Users/edb/Documents/projects/venv_pg4/lib/python3.5/site-
>> packages/selenium/webdriver/support/wait.py",
>> > line 80, in until

---

From: George Gelashvili <[email-masked]>
Date: Mon, 5 Jun 2017 16:58:53 -0400

We fixed the codemirror-filling helper function method thing in
`pgadmin_page`. This should resolve the PG datatype feature test flakiness
and make for a nice buttery crust.

It looked like selenium was unable to find an element within the sqleditor
iframe, so we made a workaround that re-locates the iframe and codemirror
element each iteration of the wait.

Thanks
Joao & George

On Tue, May 30, 2017 at 6:52 AM, Khushboo Vashi <
[email-masked]> wrote:

> Hi Murtuza,
>
> On Tue, May 30, 2017 at 4:13 PM, Murtuza Zabuawala <murtuza.zabuawala@
> enterprisedb.com> wrote:
>
>> Hi Dave,
>>
>> I didn't look into the code for it but every time I run the tests it
>> stays on below screen and fails.
>>
>> @Khushboo,
>> Any suggestion?
>>
>>
>>> > "/Users/edb/Documents/projects/venv_pg4/lib/python3.5/site-p
>>> ackages/selenium/webdriver/support/wait.py",
>>> > line 80, in until
diff --git a/web/pgadmin/feature_tests/pg_datatype_validation_test.py b/web/pgadmin/feature_tests/pg_datatype_validation_test.py
index fdc65a58..69b12f30 100644
--- a/web/pgadmin/feature_tests/pg_datatype_validation_test.py
+++ b/web/pgadmin/feature_tests/pg_datatype_validation_test.py
@@ -6,7 +6,6 @@
 # This software is released under the PostgreSQL Licence
 #
 ##########################################################################
-
 from selenium.webdriver import ActionChains
 from selenium.common.exceptions import TimeoutException
 from selenium.webdriver.support.ui import WebDriverWait
@@ -90,30 +89,28 @@ class PGDataypeFeatureTest(BaseFeatureTest):
 
         self.page.driver.find_element_by_link_text("Tools").click()
         self.page.find_by_partial_link_text("Query Tool").click()
+
+        self.page.fill_codemirror_area_with(query)
+        self.page.find_by_id("btn-flash").click()
         wait = WebDriverWait(self.page.driver, 5)
-        element = wait.until(EC.frame_to_be_available_and_switch_to_it((By.TAG_NAME, 'iframe')))
-
-        if element:
-            self.page.fill_codemirror_area_with(query)
-            self.page.find_by_id("btn-flash").click()
-            wait.until(EC.presence_of_element_located(
-                (By.XPATH, "//*[@id='0']//*[@id='datagrid']/div[5]/div/div[1]/div[2]/span")))
-
-            # For every sample data-type value, check the expected output.
-            cnt = 2
-            for val in expected_output:
-                try:
-                    source_code = self.page.find_by_xpath(
-                        "//*[@id='0']//*[@id='datagrid']/div[5]/div/div[1]/div[" + str(cnt) + "]/span"
-                    ).get_attribute('innerHTML')
-
-                    PGDataypeFeatureTest.check_result(
-                        source_code,
-                        expected_output[cnt - 2]
-                    )
-                    cnt += 1
-                except TimeoutException:
-                    assert False, "{0} does not match with {1}".format(val, expected_output[cnt])
+        wait.until(EC.presence_of_element_located(
+            (By.XPATH, "//*[@id='0']//*[@id='datagrid']/div[5]/div/div[1]/div[2]/span")))
+
+        # For every sample data-type value, check the expected output.
+        cnt = 2
+        for val in expected_output:
+            try:
+                source_code = self.page.find_by_xpath(
+                    "//*[@id='0']//*[@id='datagrid']/div[5]/div/div[1]/div[" + str(cnt) + "]/span"
+                ).get_attribute('innerHTML')
+
+                PGDataypeFeatureTest.check_result(
+                    source_code,
+                    expected_output[cnt - 2]
+                )
+                cnt += 1
+            except TimeoutException:
+                assert False, "{0} does not match with {1}".format(val, expected_output[cnt])
 
     @staticmethod
     def check_result(source_code, string_to_find):
diff --git a/web/pgadmin/feature_tests/xss_checks_panels_and_query_tool_test.py b/web/pgadmin/feature_tests/xss_checks_panels_and_query_tool_test.py
index 526d4e76..d59e8ac3 100644
--- a/web/pgadmin/feature_tests/xss_checks_panels_and_query_tool_test.py
+++ b/web/pgadmin/feature_tests/xss_checks_panels_and_query_tool_test.py
@@ -149,8 +149,6 @@ class CheckForXssFeatureTest(BaseFeatureTest):
     def _check_xss_in_query_tool(self):
         self.page.driver.find_element_by_link_text("Tools").click()
         self.page.find_by_partial_link_text("Query Tool").click()
-        time.sleep(3)
-        self.page.driver.switch_to.frame(self.page.driver.find_element_by_tag_name('iframe'))
         self.page.fill_codemirror_area_with("select '<img src=\"x\" onerror=\"console.log(1)\">'")
         time.sleep(1)
         self.page.find_by_id("btn-flash").click()
diff --git a/web/regression/feature_utils/pgadmin_page.py b/web/regression/feature_utils/pgadmin_page.py
index b9abe7ca..30f710eb 100644
--- a/web/regression/feature_utils/pgadmin_page.py
+++ b/web/regression/feature_utils/pgadmin_page.py
@@ -26,7 +26,7 @@ class PgadminPage:
     def __init__(self, driver, app_config):
         self.driver = driver
         self.app_config = app_config
-        self.timeout = 30
+        self.timeout = 20
         self.app_start_timeout = 60
 
     def reset_layout(self):
@@ -118,14 +118,27 @@ class PgadminPage:
         # For long text, if we try to execute send_keys and perform back to back, then the actions are
         # not executed properly as the driver can send only 50 to 60 characters. To avoid this, sleep
         # on the basis of content length.
-        self.find_by_xpath(
-            "//pre[contains(@class,'CodeMirror-line')]/../../../*[contains(@class,'CodeMirror-code')]").click()
+        def find_codemirror(driver):
+            try:
+                driver.switch_to.default_content()
+                driver.switch_to_frame(driver.find_element_by_tag_name("iframe"))
+                element = driver.find_element_by_xpath(
+                    "//pre[contains(@class,'CodeMirror-line')]/../../../*[contains(@class,'CodeMirror-code')]")
+                if element.is_displayed() and element.is_enabled():
+                    return element
+            except (NoSuchElementException, WebDriverException):
+                return False
+
+        WebDriverWait(self.driver, timeout=self.timeout, poll_frequency=0.01).\
+            until(find_codemirror, "Timed out waiting for codemirror to appear").\
+            click()
+        time.sleep(1)
+
         action = ActionChains(self.driver)
         action.send_keys(field_content)
+        action.perform()
         sleep_time = math.ceil(len(field_content) / 50)
         time.sleep(sleep_time)
-        action.perform()
-        time.sleep(1)
 
     def click_tab(self, tab_name):
         self.find_by_xpath("//*[contains(@class,'wcTabTop')]//*[contains(@class,'wcPanelTab') "

---

From: Murtuza Zabuawala <[email-masked]>
Date: Tue, 6 Jun 2017 15:00:27 +0530

That fixed the issue for me, PG data-type feature test working now,

On Tue, Jun 6, 2017 at 2:28 AM, George Gelashvili <[email-masked]>
wrote:

>> BTW, I can not reproduce this issue at my end but me and Harshal will
>> look into this issue on his machine.
>>
>>
>> Thanks,
>> Khushboo
>>
>>
>>>
>>> --
>>> Regards,
>>> Murtuza Zabuawala
>>>> >> > We've also split out the app start timeout since that seemed to
>>>> need
>>>> >> > more
>>>> >> > time on CI.
>>>> >> >
>>>> >> > Patch attached.
>>>> >> >
>>>> >> > Thanks!
>>>> >> > Joao, Matt, and George
>>>> >> >
>>>> >> >
>>>> >> > --

---

From: Dave Page <[email-masked]>
Date: Wed, 7 Jun 2017 14:27:02 +0100

Thanks, patch applied.

Mmmmmm, pie....