Subject: [pgAdmin4][Patch]: File Manager & Backform FileControl

From: Surinder Kumar <[email-masked]>
Date: Wed, 27 Apr 2016 18:18:12 +0530

Hi,

The patch is for *File Manager and Backform FileControl*.

*File Manager:* It allows user to store their files at one place. The user
can set path to this directory in preferences which file manger will use.

It will be used by various modules of pgAdmin4 such as
*import data from file, query tool, backup & recovery tool* so on.. It
allows user to perform various operations such as:

   - File deletion
   - File upload
   - Create new directory
   - File rename
   - File download and
   - List files & folders in list and grid view.

*FileControl: *This control is used with *File Manager. *It can be used for:

   - File Selection
   - Folder Selection
   - File Creation
   - Storage Manager Dialog.( it supports all operations listed above).

Usage*:*

schema:[{
  id: 'select_file', label: '{{ _('Select File')}}',
  type: 'text', *control*: *Backform.FileControl*,
  *dialog_type: 'select_file'*, *supp_types: ['*', 'pdf', 'text', 'patch',
'py']*
}

Where:

   - *control* - It should be *Backform.FileControl*
   - *dialog_type *- the type of dialog to use.
   - options can be:  *select_file, select_folder, create_file &
   storage_dialog*.
   - *supp_types(*not mandatory) - the types of files allowed to show and
   upload.

Please review the patch and let me know for the review comments.

Thanks
Surinder Kumar

-- 
Sent via pgadmin-hackers mailing list ([email-masked])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgadmin-hackers

---

From: Neel Patel <[email-masked]>
Date: Wed, 27 Apr 2016 21:15:04 +0530

Hi Surinder,

I have applied the patch file and below are some observations.

   - When we create the new folder and hit the enter then we are getting
   the below error.

                  TypeError: argument of type 'NoneType' is not iterable

   - I have just added the new control in "Import" dialog and observed that
   new control is getting added in new tab called "General". I think it should
   not create the new "General" tab.
   - "Select" and "Cancel" buttons font is different then other dialogs.

Thanks,
Neel Patel

On Wed, Apr 27, 2016 at 6:18 PM, Surinder Kumar <
[email-masked]> wrote:

> *File Manager:* It allows user to store their files at one place. The
> user can set path to this directory in preferences which file manger will
> use.
>
>   *dialog_type: 'select_file'*, *supp_types: ['*', 'pdf', 'text',
> 'patch', 'py']*
> }
>
> Where:
>

---

From: Neel Patel <[email-masked]>
Date: Thu, 28 Apr 2016 18:39:08 +0530

Hi Surinder,

Please find additional comments as below.

1.  In Windows OS, when we try to open the dialog with default preference
value then it gives error as below.

2016-04-28 13:07:44,716: ERROR  pgadmin:        Exception on
/file_manager/filemanager/2530270/ [POST]
Traceback (most recent call last):
  File "C:\Projects\venv_python_2_7\Lib\site-packages\flask\app.py", line
1817, in wsgi_app
    response = self.full_dispatch_request()
  File "C:\Projects\venv_python_2_7\Lib\site-packages\flask\app.py", line
1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "C:\Projects\venv_python_2_7\Lib\site-packages\flask\app.py", line
1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "C:\Projects\venv_python_2_7\Lib\site-packages\flask\app.py", line
1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "C:\Projects\venv_python_2_7\Lib\site-packages\flask\app.py", line
1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "C:\Projects\venv_python_2_7\Lib\site-packages\flask_login.py", line
758, in decorated_view
    return func(*args, **kwargs)
  File "C:\Projects\pgadmin4\web\pgadmin\misc\file_manager\__init__.py",
line 629, in file_manager
    return getattr(myFilemanager, mode)(**kwargs)
  File "C:\Projects\pgadmin4\web\pgadmin\misc\file_manager\__init__.py",
line 421, in getfolder
    filelist = self.list_filesystem(dir, path, trans_data, file_type)
  File "C:\Projects\pgadmin4\web\pgadmin\misc\file_manager\__init__.py",
line 331, in list_filesystem
    for f in sorted(os.listdir(orig_path)):
WindowsError: [Error 161] The specified path is invalid: '//*.*'

2. Folder having write only permission should not allow to read the
directory. Same way read only permission folder should not allow to create
the folder.     Currently it gives below error.

   PermissionError: [Errno 13] Permission denied:
'/home/neel/Projects/pgadmin4_file_manager/write_only_premission/'

3. In Windows OS, when we open the file manager dialog, no directories
listed though we have directories in the folder. Due to that we are not
able to
    test all the functionality in windows. Once we fix this issue then we
test all the functionality in windows OS.

   We debug the issue and found that issue is with the json response which
includes the HTML tags.

4. In Windows OS, file manager dialog have issue with layout. We are
getting some margin at top before buttons.

5. When we give any special characters to folder name that it gives error
saying "Folder does not exist".
    Same is applicable when we give dialog_type to create_file

6. In create_file mode, after giving the filename it should display the
name of the file to text control.

7. In create_file mode, when user gives the filename and click on the any
blank area of dialog then name is getting cleared.

8. When we upload the new file through file manager then it is not showing
in the list though we have selected "All files" options.

9. When we rename the folder in "Table mode" then font-family gets changed.
It is not happened in "Grid" mode.

10. When we try to download the file then it goes into loop and opens up
20-30 dialogs at the same time.

11. "Rename" button is enabled even though we have not selected the
files/folder to rename and when we click on the "Rename" button it gives
below
     error.

    TypeError: orig_value is undefined
12. "Delete" button is enabled even though we have not selected the
files/folder to delete and when we click on the "Delete" button it gives
below error.
     TypeError: path is undefined

Do let us know if you have queries.

Thanks,
Neel Patel

On Wed, Apr 27, 2016 at 9:15 PM, Neel Patel <[email-masked]>
wrote:

>    - I have just added the new control in "Import" dialog and observed
>    that new control is getting added in new tab called "General". I think it
>    should not create the new "General" tab.
>>    - *supp_types(*not mandatory) - the types of files allowed to show
>>    and upload.
>>

---

From: Murtuza Zabuawala <[email-masked]>
Date: Tue, 3 May 2016 12:45:10 +0530

Hi,

Adding one more issue:

When we apply disabled attribute on control, only text input gets disable, we are still able to open file manager by clicking on ellipsis button which should also be disabled.

Regards,
Murtuza

> On 27-Apr-2016, at 6:18 pm, Surinder Kumar <[email-masked]> wrote:
> 
> Hi,
> 
> The patch is for File Manager and Backform FileControl.
> 
> File Manager: It allows user to store their files at one place. The user can set path to this directory in preferences which file manger will use.
> 
> import data from file, query tool, backup & recovery tool so on.. It allows user to perform various operations such as: 
> File deletion
> File upload
> Create new directory
> File rename
> File download and 
> List files & folders in list and grid view.
> FileControl: This control is used with File Manager. It can be used for:
> File Selection            
> Folder Selection      
> File Creation             
> Storage Manager Dialog.( it supports all operations listed above).
> 
> Usage:
> 
> schema:[{
>   type: 'text', control: Backform.FileControl,
>   dialog_type: 'select_file', supp_types: ['*', 'pdf', 'text', 'patch', 'py']
> }
> 
> Where:
> control - It should be Backform.FileControl
> dialog_type - the type of dialog to use.
> options can be:  select_file, select_folder, create_file & storage_dialog.
> supp_types(not mandatory) - the types of files allowed to show and upload.
> <FileManager_and_Backform_FileControl.patch>
> --

---

From: Surinder Kumar <[email-masked]>
Date: Wed, 4 May 2016 18:35:47 +0530

Hi

PFA updated patch with resolved review comments.

On Thu, Apr 28, 2016 at 6:39 PM, Neel Patel <[email-masked]>
wrote:

>   File "C:\Projects\venv_python_2_7\Lib\site-packages\flask_login.py",
> line 758, in decorated_view
> Now the default path will always be the users directory. I have used *os.path.expanduser('~')
*which prints user directory. It works for all operating systems.

> '/home/neel/Projects/pgadmin4_file_manager/write_only_premission/'
Fixed.

>    We debug the issue and found that issue is with the json response which
Fixed. It still have issues in very older browsers. I will fix it.

> getting some margin at top before buttons.
Fixed.

>     Same is applicable when we give dialog_type to create_file
Fixed.

> name of the file to text control.
Fixed.

> blank area of dialog then name is getting cleared.
Fixed.

> in the list though we have selected "All files" options.
Fixed.

>
> 9. When we rename the folder in "Table mode" then font-family gets
> changed. It is not happened in "Grid" mode.
>
Fixed.

> 20-30 dialogs at the same time.
Fixed.

>     TypeError: orig_value is undefined
Fixed.

>      TypeError: path is undefined
Fixed.

---

From: Surinder Kumar <[email-masked]>
Date: Tue, 10 May 2016 23:43:56 +0530

Hi,

Please find updated path with couple of review comments given by Ashesh
Vashi.
1. Add variable *STORAGE_DIR* in configuration settings which decides the
directory to list when *File manager* is opens, previously
we have provided this option in preferences which is removed due to
security issues.
2. List all available Volumes by default when *STORAGE_DIR *is set to empty
for *Windows* platform.
3. Also, added support to allow user to provide
*Dialog title and custom name for primary(OK) button,*

Usage:

1. Set variable *STORAGE_DIR *to some value in config.py.
2. Import it in your module as following:
*   import config*
*   config.**STORAGE_DIR*

*Usage*: Backform FilleControl

schema:[{
  id: 'select_file', label: '{{ _('Select File')}}',
  type: 'text', *control*: *Backform.FileControl*,
  *dialog_type: '*select_file*'*, *supp_types: [*'*', 'pdf', 'text',
'patch', 'py'*],*
*  dialog_title: '*Title to appear on dialog*", *
*  btn_primary: '*Save File*'*
}]

Please review the patch.

On Wed, May 4, 2016 at 6:35 PM, Surinder Kumar <
[email-masked]> wrote:

>>    We debug the issue and found that issue is with the json response
>> which includes the HTML tags.
>>
>> 8. When we upload the new file through file manager then it is not
>> showing in the list though we have selected "All files" options.
>>
> Fixed.
>
>>
>>>    - When we create the new folder and hit the enter then we are
>>>    getting the below error.
>>>

---

From: Surinder Kumar <[email-masked]>
Date: Thu, 12 May 2016 00:18:27 +0530

Hi,

PFA updated patch with following issues fixed:
1. not listing files based on default file type selected.
2. load css file using 'get_own_stylesheets' method instead of loading from
'base.html'.
3. setting custom dialog title and button not working.

On Wed, May 11, 2016 at 3:03 PM, Surinder Kumar <
[email-masked]> wrote:

>
> ---------- Forwarded message ----------
> From: Surinder Kumar <[email-masked]>
> Date: Tue, May 10, 2016 at 11:43 PM
> Subject: Re: [pgadmin-hackers][pgAdmin4][Patch]: File Manager & Backform
> FileControl
> To: Neel Patel <[email-masked]>
> Cc: pgadmin-hackers <[email-masked]>
>
>
> Hi,
>
> 2. List all available Volumes by default when *STORAGE_DIR *is set to
> empty for *Windows* platform.
>>> 1.  In Windows OS, when we try to open the dialog with default
>>> preference value then it gives error as below.
>>>
>>>   File "C:\Projects\venv_python_2_7\Lib\site-packages\flask\app.py",
>>> line 1817, in wsgi_app
>>>   File "C:\Projects\venv_python_2_7\Lib\site-packages\flask\app.py",
>>> line 1477, in full_dispatch_request
>>>   File "C:\Projects\venv_python_2_7\Lib\site-packages\flask\app.py",
>>> line 1381, in handle_user_exception
>>>   File "C:\Projects\venv_python_2_7\Lib\site-packages\flask\app.py",
>>> line 1475, in full_dispatch_request
>>>   File "C:\Projects\venv_python_2_7\Lib\site-packages\flask\app.py",
>>> line 1461, in dispatch_request
>>>     test all the functionality in windows. Once we fix this issue then
>>> we test all the functionality in windows OS.
>>>
>>> 5. When we give any special characters to folder name that it gives
>>> error saying "Folder does not exist".
>>> 7. In create_file mode, when user gives the filename and click on the
>>> any blank area of dialog then name is getting cleared.
>>>
>> Fixed.
>>
>>>
>>> On Wed, Apr 27, 2016 at 9:15 PM, Neel Patel <[email-masked]
>>> > wrote:
>>>
>>>> Hi Surinder,
>>>>
>>>>> *import data from file, query tool, backup & recovery tool* so on..
>>>>> It allows user to perform various operations such as:
>>>>>
>>>>>    - File deletion
>>>>>    - File upload
>>>>>    - Create new directory
>>>>>    - File rename
>>>>>    - File download and

---

From: Ashesh Vashi <[email-masked]>
Date: Fri, 13 May 2016 00:24:21 +0530

Hi Surinder,

Thanks - it was committed!

As discussed personally, there are few improvement needed.

* Do not show non applicable buttons, whenever no files/directory available
in the storage root directory.
* In crate mode, use dimmer with the overwrite warning, so that - it does
not allow to deselect the current selection.
* The default selection for the file type selection combo box should be
other than 'All Files'.
* File type selection combo box should support file type description.
  i.e.
Query/SQL File (*.sql)
CSV File (*.csv)
All Files (*.* | *)

Please put that in the TODO file, we will handle that in future.

I must apologies for a mistake for not giving you the credit for the work,
I forgot to set the author while committing it.
I tried setting it again to set the author without any success. :-(

Thanks for the patch.
I will be more careful next time.

--

Thanks & Regards,

Ashesh Vashi
EnterpriseDB INDIA: Enterprise PostgreSQL Company
<http://www.enterprisedb.com>

*http://www.linkedin.com/in/asheshvashi*
<http://www.linkedin.com/in/asheshvashi>

On Thu, May 12, 2016 at 12:18 AM, Surinder Kumar <
[email-masked]> wrote:

> 2. load css file using 'get_own_stylesheets' method instead of loading
> from 'base.html'.
>> 1. Add variable *STORAGE_DIR* in configuration settings which decides
>> the directory to list when *File manager* is opens, previously
>>> On Thu, Apr 28, 2016 at 6:39 PM, Neel Patel <[email-masked]
>>> > wrote:
>>>
>>>> Hi Surinder,
>>>>
>>>>   File
>>>> "C:\Projects\pgadmin4\web\pgadmin\misc\file_manager\__init__.py", line 629,
>>>> in file_manager
>>>>   File
>>>> "C:\Projects\pgadmin4\web\pgadmin\misc\file_manager\__init__.py", line 421,
>>>> in getfolder
>>>>   File
>>>> "C:\Projects\pgadmin4\web\pgadmin\misc\file_manager\__init__.py", line 331,
>>>> in list_filesystem
>>>> 10. When we try to download the file then it goes into loop and opens
>>>> up 20-30 dialogs at the same time.
>>>>
>>> Fixed.
>>>
>>>>
>>>> On Wed, Apr 27, 2016 at 9:15 PM, Neel Patel <
>>>> [email-masked]> wrote:
>>>>
>>>>> Hi Surinder,
>>>>>
>>>>>    - I have just added the new control in "Import" dialog and
>>>>>    observed that new control is getting added in new tab called "General". I
>>>>>    think it should not create the new "General" tab.
>>>>>> *File Manager:* It allows user to store their files at one place.
>>>>>> The user can set path to this directory in preferences which file manger
>>>>>> will use.
>>>>>>
>>>>>>    - *supp_types(*not mandatory) - the types of files allowed to
>>>>>>    show and upload.
>>>>>>

---

From: Surinder Kumar <[email-masked]>
Date: Fri, 13 May 2016 16:33:55 +0530

Hi,

Please find the patch for the following improvements:
1. FileManager won't show Volumes on Windows when STORAGE_DIR is set to
None.

On Fri, May 13, 2016 at 12:24 AM, Ashesh Vashi <
[email-masked]> wrote:

> * Do not show non applicable buttons, whenever no files/directory
> available in the storage root directory.
>
Done

> not allow to deselect the current selection.
Done

> * The default selection for the file type selection combo box should be
Done

> Please put that in the TODO file, we will handle that in future.
I have added it into TODO file.

>>>> On Thu, Apr 28, 2016 at 6:39 PM, Neel Patel <
>>>>> 6. In create_file mode, after giving the filename it should display
>>>>> the name of the file to text control.
>>>>>
>>>> Fixed.
>>>>
>>>>>

---

From: Surinder Kumar <[email-masked]>
Date: Fri, 20 May 2016 16:33:31 +0530

Hi,

Please find patch with following issues/changes fixed:

1. Rename file/folder not working in list view.
2. Selection folder only should not allow to select file.
3. Ui changes in *table list* to match with the pgadmin color styles.
4. Custom *STORAGE_DIR* path don't list files/folders when opened in window
system.
5. Code cleanup, formatting & did proper commenting.
6. In file selection mode, the value typed in text box not setting into
backbone model.
7. Removed create file code, it should be handled on module using it.
8. Written a common function to make ajax request, instead of writing
individual functions for same task.

*Issue to be fixed*: When FileControl is used with 'nested' type, it
redirects to some url when it is opened up.
Once it gets fixed. I will send patch for this issue.

Thanks
Surinder Kumar

On Sat, May 14, 2016 at 12:40 AM, Ashesh Vashi <
[email-masked]> wrote:

> Thanks - committed with some tweaks!
>
> --
>
> Thanks & Regards,
>
> Ashesh Vashi
> On Fri, May 13, 2016 at 7:36 PM, Surinder Kumar <
>> PFA patch with changes:
>> 1. Line of code shouldn't exceed greater than 80.
>> 2. Can't consider boolean as string while comparing.
>>
>> On Fri, May 13, 2016 at 4:33 PM, Surinder Kumar <
>>>> * In crate mode, use dimmer with the overwrite warning, so that - it
>>>> does not allow to deselect the current selection.
>>>>
>>> Done
>>>
>>>> I must apologies for a mistake for not giving you the credit for the
>>>> work, I forgot to set the author while committing it.
>>>>>> Subject: Re: [pgadmin-hackers][pgAdmin4][Patch]: File Manager &
>>>>>> Backform FileControl
>>>>>> Please find updated path with couple of review comments given by
>>>>>> Ashesh Vashi.
>>>>>> 1. Add variable *STORAGE_DIR* in configuration settings which
>>>>>> decides the directory to list when *File manager* is opens,
>>>>>> previously
>>>>>> 2. List all available Volumes by default when *STORAGE_DIR *is set
>>>>>> to empty for *Windows* platform.
>>>>>>>> "C:\Projects\venv_python_2_7\Lib\site-packages\flask\app.py", line 1817, in
>>>>>>>> wsgi_app
>>>>>>>> "C:\Projects\venv_python_2_7\Lib\site-packages\flask\app.py", line 1477, in
>>>>>>>> full_dispatch_request
>>>>>>>> "C:\Projects\venv_python_2_7\Lib\site-packages\flask\app.py", line 1381, in
>>>>>>>> handle_user_exception
>>>>>>>> "C:\Projects\venv_python_2_7\Lib\site-packages\flask\app.py", line 1475, in
>>>>>>>> full_dispatch_request
>>>>>>>> "C:\Projects\venv_python_2_7\Lib\site-packages\flask\app.py", line 1461, in
>>>>>>>> dispatch_request
>>>>>>>> "C:\Projects\venv_python_2_7\Lib\site-packages\flask_login.py", line 758,
>>>>>>>> in decorated_view
>>>>>>>> Now the default path will always be the users directory. I have
>>>>>>> used *os.path.expanduser('~') *which prints user directory. It
>>>>>>> works for all operating systems.
>>>>>>>
>>>>>>>>
>>>>>>>> 3. In Windows OS, when we open the file manager dialog, no
>>>>>>>> directories listed though we have directories in the folder. Due to that we
>>>>>>>> are not able to
>>>>>>>>     test all the functionality in windows. Once we fix this issue
>>>>>>>> then we test all the functionality in windows OS.
>>>>>>>>
>>>>>>>>    We debug the issue and found that issue is with the json
>>>>>>>> response which includes the HTML tags.
>>>>>>>>
>>>>>>>> 4. In Windows OS, file manager dialog have issue with layout. We
>>>>>>>> are getting some margin at top before buttons.
>>>>>>>>
>>>>>>> Fixed.
>>>>>>>
>>>>>>>>
>>>>>>>> 7. In create_file mode, when user gives the filename and click on
>>>>>>>> the any blank area of dialog then name is getting cleared.
>>>>>>>>
>>>>>>> Fixed.
>>>>>>>
>>>>>>>>
>>>>>>>> 10. When we try to download the file then it goes into loop and
>>>>>>>> opens up 20-30 dialogs at the same time.
>>>>>>>>
>>>>>>> Fixed.
>>>>>>>
>>>>>>>>
>>>>>>>>>> *File Manager:* It allows user to store their files at one
>>>>>>>>>> place. The user can set path to this directory in preferences which file
>>>>>>>>>> manger will use.
>>>>>>>>>>
>>>>>>>>>> *import data from file, query tool, backup & recovery tool* so
>>>>>>>>>> on.. It allows user to perform various operations such as:
>>>>>>>>>>
>>>>>>>>>>    - File deletion
>>>>>>>>>>    - File upload
>>>>>>>>>>    - Create new directory
>>>>>>>>>>    - File rename
>>>>>>>>>>    - File download and
>>>>>>>>>>   *dialog_type: 'select_file'*, *supp_types: ['*', 'pdf',
>>>>>>>>>> 'text', 'patch', 'py']*
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> Where:
>>>>>>>>>>

---

From: Ashesh Vashi <[email-masked]>
Date: Sat, 21 May 2016 15:03:29 +0530

Thanks - committed with small tweak.

--

Thanks & Regards,

Ashesh Vashi

On Fri, May 20, 2016 at 4:33 PM, Surinder Kumar <
[email-masked]> wrote:

> 4. Custom *STORAGE_DIR* path don't list files/folders when opened in
> window system.
>>>>> * The default selection for the file type selection combo box should
>>>>> be other than 'All Files'.
>>>>>
>>>> Done
>>>>
>>>>>> 2. load css file using 'get_own_stylesheets' method instead of
>>>>>> loading from 'base.html'.
>>>>>>>   *dialog_type: '*select_file*'*, *supp_types: [*'*', 'pdf',
>>>>>>> 'text', 'patch', 'py'*],*
>>>>>>>>> 2. Folder having write only permission should not allow to read
>>>>>>>>> the directory. Same way read only permission folder should not allow to
>>>>>>>>> create the folder.     Currently it gives below error.
>>>>>>>>>
>>>>>>>>> 5. When we give any special characters to folder name that it
>>>>>>>>> gives error saying "Folder does not exist".
>>>>>>>>> 6. In create_file mode, after giving the filename it should
>>>>>>>>> display the name of the file to text control.
>>>>>>>>>
>>>>>>>> Fixed.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> 11. "Rename" button is enabled even though we have not selected
>>>>>>>>> the files/folder to rename and when we click on the "Rename" button it
>>>>>>>>> gives below
>>>>>>>>>      error.
>>>>>>>>>
>>>>>>>>> 12. "Delete" button is enabled even though we have not selected
>>>>>>>>> the files/folder to delete and when we click on the "Delete" button it
>>>>>>>>> gives below error.
>>>>>>>>>>>    - options can be:  *select_file, select_folder, create_file
>>>>>>>>>>>    & storage_dialog*.
>>>>>>>>>>>    - *supp_types(*not mandatory) - the types of files allowed
>>>>>>>>>>>    to show and upload.
>>>>>>>>>>>