Subject: [pgAdmin4][Debugger]: Initial Patch

From: Neel Patel <[email-masked]>
Date: Tue, 5 Apr 2016 17:10:50 +0530

Hi,

Please find attached debugger v1 patch. To test this patch we need to apply
the "function" module patch submitted by Khushboo.

For the debugger functionality, direct and indirect debugging is working as
per pgadmin3 except below TODOs.

Below are the TODOs which will be submitted in next patch along with the
fixed review comments.

   - For the direct debugging, 'NULL' and 'Expression' column value should
   be validated before pressing the debug button.
   - User can deposit the value of input arguments and local variable but
   need to be
   validated against respective data types.
   - Currently we have tested for plpgsql functions only, We need to test
   against EDB SPL functions and trigger functions.
   - For the direct debugging, once the execution is completed, only
   "Continue/Restart" button will be enabled and user should allow to restart
   the debugging with same function and previous values.
   - Values entered by the user in input dialog during direct debugging
   should be saved.

Currently we have tested this patch with PostgreSQL v9.5.

Do review it and let us know for the comments.

Thanks,
Neel Patel

-- 
Sent via pgadmin-hackers mailing list ([email-masked])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgadmin-hackers

---

From: Dave Page <[email-masked]>
Date: Tue, 5 Apr 2016 17:29:08 +0100

Hi

On Tue, Apr 5, 2016 at 12:40 PM, Neel Patel <[email-masked]> wrote:
> For the direct debugging, 'NULL' and 'Expression' column value should be
> validated before pressing the debug button.
> User can deposit the value of input arguments and local variable but need to
> be
> Currently we have tested for plpgsql functions only, We need to test against
> EDB SPL functions and trigger functions.
> For the direct debugging, once the execution is completed, only
> Values entered by the user in input dialog during direct debugging should be
> saved.
>
>

I just get a blank tab when I try to debug something. I do see the
params dialogue, but then nothing. Note:

- I have Akshay's Query Tool patch applied, and am running from git master

- PostgreSQL 9.4, with the debugger plugin in the public schema.

- Various functions in the pem schema tested, both with and without parameters.

- Both direct and indirect debugging have the same issue.

- No errors seen in the console, except when closing the debugger tab:

2016-04-05 17:27:37,720: INFO werkzeug: 127.0.0.1 - - [05/Apr/2016
17:27:37] "GET /debugger/close/8569717 HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/dpage/git/pgadmin4-demo/web/pgadmin/tools/debugger/__init__.py",
line 397, in close
    session_id=obj['session_id'])
KeyError: 'session_id'

Thanks.

-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

---

From: Neel Patel <[email-masked]>
Date: Wed, 6 Apr 2016 11:38:40 +0530

Hi Dave,

We will fix the error "KeyError: 'session_id'" but regarding the blank tab,
we have tried with PostgreSQL 9.4 with both the version of python (2.7 &
3.4) but we are not able to reproduce the issue.

Below are some of the questions to debug the issue.

   - Is it only reproduce on PostgreSQL 9.4 ? Is it working on PostgreSQL
   9.5 ?
   - Is it possible to know the debugger version ?
   - The issue is reproduce on specific functions ? If yes, can you please
   give definition of the function ?
   - Are you able to see toolbar buttons ? Or nothing on the screen ? Only
   blank panel ?
   - Can you please share request/response of last request URL sent to
   server in browser ?

Thanks,
Neel Patel

On Tue, Apr 5, 2016 at 9:59 PM, Dave Page <[email-masked]> wrote:

>   File
> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1836, in __call__
>   File
> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1820, in wsgi_app
>   File
> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>   File
> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1817, in wsgi_app
>   File
> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>   File
> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>   File
> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>   File
> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>   File
> "/Users/dpage/git/pgadmin4-demo/web/pgadmin/tools/debugger/__init__.py",
> line 397, in close

---

From: Neel Patel <[email-masked]>
Date: Wed, 6 Apr 2016 11:50:29 +0530

One more additional information.

Follow the below steps and send us pgAdmin4.log file.

   - Browse till function which we want to debug. Do not click on the
   "Debug" context menu.
   - Stop the server
   - Enable the "Debug" option in the config file under "web" folder.
   - Start the server.
   - Now click on the "Debug" context menu and add requirement input. After
   clicking on the debug button on the input dialog send us the pgAdmin4.log
   file.

Thanks,
Neel Patel

On Wed, Apr 6, 2016 at 11:38 AM, Neel Patel <[email-masked]>
wrote:

> Hi Dave,
>
> We will fix the error "KeyError: 'session_id'" but regarding the blank
> tab, we have tried with PostgreSQL 9.4 with both the version of python (2.7
> & 3.4) but we are not able to reproduce the issue.
>
>    - The issue is reproduce on specific functions ? If yes, can you
>    please give definition of the function ?
>    - Are you able to see toolbar buttons ? Or nothing on the screen ?
>    Only blank panel ?

---

From: Neel Patel <[email-masked]>
Date: Wed, 6 Apr 2016 12:07:07 +0530

Hi Dave,

Updated above information.
We want this pgadmin4 log information to debug the debugger blank tab open
issue.

Thanks,
Neel Patel

On Wed, Apr 6, 2016 at 11:50 AM, Neel Patel <[email-masked]>
wrote:

>             Stop the web server.

>              Set the Log setting to "Debug" and save the log to files and
send the updated debug log information.

>                Start the Web Server.

>
>    - Now click on the "Debug" context menu and add requirement input.
>    After clicking on the debug button on the input dialog send us the
>    pgAdmin4.log file.
>
> Thanks,
> Neel Patel
>
>>    - Is it only reproduce on PostgreSQL 9.4 ? Is it working on
>>    PostgreSQL 9.5 ?

---

From: Neel Patel <[email-masked]>
Date: Wed, 6 Apr 2016 13:27:24 +0530

Hi Dave,

I got the root cause. Issue will reproduce when we applied Akshay's query
tool patch.
Issue was once the panel was initialized, HTML file was not getting loaded
in to the panel due to the change in frame.js.

Both query tool and debugger patch file contains the change in "frame.js"
file. Once you applied the query tool patch and after that debugger then
debugger patch should fail but i don't know why you do not get any error.

*Solution:-*

   - Open "frame.js"  file and add the below line in "setTimeout" function
   which is required by debugger.

 pgBrowser.Events.trigger('pgadmin-browser:frame:urlloaded:' + that.name,
frame, that.url, self);

Do let us know if you still face the same issue.

Thanks,
Neel Patel

---

From: Dave Page <[email-masked]>
Date: Wed, 6 Apr 2016 09:07:08 +0100

On Wed, Apr 6, 2016 at 8:57 AM, Neel Patel <[email-masked]> wrote:
> Solution:-
> Open "frame.js"  file and add the below line in "setTimeout" function which
> is required by debugger.
>
>                  pgBrowser.Events.trigger('pgadmin-browser:frame:urlloaded:'
> + that.name, frame, that.url, self);
>

Ahh, good catch - that got it :-)

Thanks!

---

From: Neel Patel <[email-masked]>
Date: Thu, 14 Apr 2016 18:22:17 +0530

Hi,

Please find attached v2 patch file of the debugger which fixes the below
issues which was not present in the first patch.
In this patch, we have added new table in sqlite database to store the
functions arguments value user has given during debugging.
After applying this patch, user needs to execute "setup.py" to create the
new table in pgadmin4.db file.

   - In direct debugging, when user debug the function then arguments
   values will be stored in the sqlite database so when user debug the same
   function again then previous values will be filled in the user input dialog.
   - Once the execution is completed then user will be able to do the debug
   of the same function again by pressing the "Continue/Restart" button.
   - User can debug the "procedure" which is supported in PPAS database.
   - Replaced the "Glyphicon" with the "font-awesome" icons.

Below are the TODOs

   - Validate the input arguments values changed by user while depositing
   the value during debugging.
   - Need to implement the code folding in the codemirror editor area.
   - As per the Ashesh's suggestion, need to add debug logs information so
   that we can get the state of the debug function. Also need to add "arrow"
   next to breakpoint in the gutters as per the pgadmin3.
   - Need to add "Debug package initializer" in the user input dialog for
   the direct debugging.
   - Last but not least "Review comments" :)

Below functionalities are implemented but testing are pending.

   - Trigger functions need to test with the debugger.
   - Functions are tested with data types (like text, integer etc.)  but it
   needs to be tested with all the data types for direct debugging.
   - Functions/Procedures need to test with PPAS 9.2 and earlier version
   where debugger version is different.

Can you please review the patch file and let us know the comments.

Thanks,
Neel Patel

---

From: Dave Page <[email-masked]>
Date: Thu, 14 Apr 2016 21:39:21 +0100

Hi

On Thu, Apr 14, 2016 at 1:52 PM, Neel Patel <[email-masked]> wrote:
> In direct debugging, when user debug the function then arguments values will
> be stored in the sqlite database so when user debug the same function again
> then previous values will be filled in the user input dialog.
> Once the execution is completed then user will be able to do the debug of
> the same function again by pressing the "Continue/Restart" button.
> User can debug the "procedure" which is supported in PPAS database.
> Replaced the "Glyphicon" with the "font-awesome" icons.

Very cool! Committed, understanding that there are still improvements
to be made.

> Below are the TODOs
>
> Validate the input arguments values changed by user while depositing the
> value during debugging.
> Need to implement the code folding in the codemirror editor area.
> As per the Ashesh's suggestion, need to add debug logs information so that
> we can get the state of the debug function. Also need to add "arrow" next to
> breakpoint in the gutters as per the pgadmin3.
> Need to add "Debug package initializer" in the user input dialog for the
> direct debugging.
> Last but not least "Review comments" :)

Here you go :-)

- Ensure all messages are gettext enabled.

- Constructs like the following won't work, because Jinja will
evaluate the string " + err.errormsg + " before it ever gets evaluated
as JS by the browser.

Alertify.alert("{{ _('" + err.errormsg + "') }}");

- Please adjust the button bar to use the same styling as the button
bar on the Properties tab.

- Let's make the stack pane tab part of the tabset at the bottom of
the debugger, and ensure docking etc. works so tabs can be split off
and arranged around the main source window.

- Stepping is quite slow. What's causing that? I wonder if we really
need to have all the one line SQL templates - they're probably quite
expensive to process.

- Is backend_running.sql required? I've removed both versions as I
can't find any references to them. Are any other templates not
required?

Will log any other issues that come up in further work.

> Trigger functions need to test with the debugger.
> Functions are tested with data types (like text, integer etc.)  but it needs
> to be tested with all the data types for direct debugging.
> Functions/Procedures need to test with PPAS 9.2 and earlier version where
> debugger version is different.

Thanks!

---

From: Neel Patel <[email-masked]>
Date: Mon, 18 Apr 2016 14:25:19 +0530

Hi Dave,

Please find inline comments with all the fixes.
Attached is the updated patch file. Do review it and let me know for any
comments.

If you find any issues, let me know .Now, Working on the remaining TODOs as
mentioned in below email.

Thanks,
Neel Patel

> - Ensure all messages are gettext enabled.

Fixed.

> Alertify.alert("{{ _('" + err.errormsg + "') }}");

Fixed.

> bar on the Properties tab.

Fixed

> and arranged around the main source window.

Fixed. Now stack pane will be displayed along with another panel at bottom
and also docking has been introduced for all the panels so tabs will be
arranged around main debugger panel.

> need to have all the one line SQL templates - they're probably quite
Fixed. This is due to polling timeout was high (1 second) and we are
getting delay in getting the results. Now polling timeout has reduced to to
200ms.

> can't find any references to them. Are any other templates not
Ok. No other templates.

---

From: Dave Page <[email-masked]>
Date: Mon, 18 Apr 2016 13:37:11 +0100

Hi

On Monday, April 18, 2016, Neel Patel <[email-masked]> wrote:

> If you find any issues, let me know .Now, Working on the remaining TODOs
> as mentioned in below email.
>

Thanks - committed with some minor tweaks. One problem partly still remains
though - you've partially copied the toolbar styling. Please use the actual
classes used by the Properties panel. I've already updated the query tool
in that way. Whilst your version looks much closer, it's missing the
minimum button widths, and duplicates CSS unnecessarily.

Thanks.

>
> On Fri, Apr 15, 2016 at 2:09 AM, Dave Page <[email-masked]
> <javascript:_e(%7B%7D,'cvml',[email-masked]');>> wrote:
>
>> Hi
>>
>> On Thu, Apr 14, 2016 at 1:52 PM, Neel Patel <[email-masked]
>> <javascript:_e(%7B%7D,'cvml',[email-masked]');>> wrote:
>> > Hi,
>> >

---

From: Neel Patel <[email-masked]>
Date: Tue, 19 Apr 2016 11:59:57 +0530

Hi Dave,

Please find the attached patch file with below fix.

   - Remove the duplicate CSS and used the actual class for the debugger
   button toolbar.
   - As per the Ashesh's comment, previously we used 2 wcDocker instance to
   arrange the multiple tabs to main debugger panel. Now with this patch file,
   we have used only 1 wcDocker instance.

Do review it and let us know for comments.

Thanks,
Neel  Patel

On Mon, Apr 18, 2016 at 6:07 PM, Dave Page <[email-masked]> wrote:

> Thanks - committed with some minor tweaks. One problem partly still
> remains though - you've partially copied the toolbar styling. Please use
> the actual classes used by the Properties panel. I've already updated the
> query tool in that way. Whilst your version looks much closer, it's missing
> the minimum button widths, and duplicates CSS unnecessarily.
>
> Thanks.
>
>
>>
>> Fixed. Now stack pane will be displayed along with another panel at
>> bottom and also docking has been introduced for all the panels so tabs will
>> be arranged around main debugger panel.
>>
>>
>>>

---

From: Dave Page <[email-masked]>
Date: Wed, 20 Apr 2016 09:02:15 -0400

Thanks - applied!

On Tuesday, April 19, 2016, Neel Patel <[email-masked]> wrote:

>    - As per the Ashesh's comment, previously we used 2 wcDocker instance
>    to arrange the multiple tabs to main debugger panel. Now with this patch
>    file, we have used only 1 wcDocker instance.
>
> On Mon, Apr 18, 2016 at 6:07 PM, Dave Page <[email-masked]
>> On Monday, April 18, 2016, Neel Patel <[email-masked]

---

From: Neel Patel <[email-masked]>
Date: Mon, 25 Apr 2016 19:18:33 +0530

Hi,

Please find attached patch file with below fixes.

   - Removed the "lineWrapping" option from the codemirror textarea because
   it was creating issue in the code folding.
   - Handle the values while depositing during debugging.
   - Properly handle the Array values while saving it to sqlite database
   and displayed in input dialog.
   - SQL code folding was not supported in codemirror so added the same.
   Currently we have added support for below keywords.
         IF-END IF
         LOOP-END LOOP
         BEGIN - END
         CASE- END CASE
   We will add more SQL folding support later ( e.g. Fold multiline query
   inside function, Folding of create function etc.)

           As query tool also requires the use of code folding so below is
the usage for it.
              - Define below require javascript files in modules.
                 'codemirror/addon/fold/foldgutter',
                 'codemirror/addon/fold/foldcode',
                 'codemirror/addon/fold/pgadmin-sqlfoldcode'

              - Remove "lineWrapping" option from the codemirror textarea.

              - Add the below options while creating the Codemirror
textarea.

                      *foldOptions*: {
                        widget: "\u2026"
                      },
                      *foldGutter*: {
                        rangeFinder:
CodeMirror.fold.combine(CodeMirror.pgadminBeginRangeFinder,
CodeMirror.pgadminIfRangeFinder,

 CodeMirror.pgadminLoopRangeFinder, CodeMirror.pgadminCaseRangeFinder)
                     },
                     *gutters*: ["CodeMirror-foldgutter"]

Thanks,
Neel Patel

On Wed, Apr 20, 2016 at 6:32 PM, Dave Page <[email-masked]> wrote:

>>>> Attached is the updated patch file. Do review it and let me know for
>>>> any comments.
>>>>
>>>> If you find any issues, let me know .Now, Working on the remaining
>>>> TODOs as mentioned in below email.
>>>>
>>>

---

From: Akshay Joshi <[email-masked]>
Date: Fri, 6 May 2016 17:50:13 +0530

On Mon, Apr 25, 2016 at 7:18 PM, Neel Patel <[email-masked]>
wrote:

>    - Removed the "lineWrapping" option from the codemirror textarea
>    because it was creating issue in the code folding.

    Thanks patch applied.

>>>    - Remove the duplicate CSS and used the actual class for the
>>>    debugger button toolbar.
>>>    - As per the Ashesh's comment, previously we used 2 wcDocker
>>>    instance to arrange the multiple tabs to main debugger panel. Now with this
>>>    patch file, we have used only 1 wcDocker instance.
>>>

-- 
*Akshay Joshi*
*Principal Software Engineer *

*Phone: +91 20-3058-9517Mobile: +91 976-788-8246*