Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1gaJfy-000396-JE for pgadmin-hackers@arkaria.postgresql.org; Fri, 21 Dec 2018 12:13:11 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1gaJfw-0001RF-AS for pgadmin-hackers@arkaria.postgresql.org; Fri, 21 Dec 2018 12:13:08 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1gaJfv-0001R8-VO for pgadmin-hackers@lists.postgresql.org; Fri, 21 Dec 2018 12:13:08 +0000 Received: from mail-qk1-x72e.google.com ([2607:f8b0:4864:20::72e]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1gaJfn-0006nu-R8 for pgadmin-hackers@postgresql.org; Fri, 21 Dec 2018 12:13:07 +0000 Received: by mail-qk1-x72e.google.com with SMTP id r71so2906976qkr.10 for ; Fri, 21 Dec 2018 04:12:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=enterprisedb-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=KPxJhmU+AOM39uKEirvdpYHCZWYBg/9IzuGwESksTAQ=; b=Nkwz9UsFHMzg26rjEPuwaA3KQ5wHXEAYOI9qsOa60kRcFhfX+DVhmEYuOM/Xs7cqcB ZKUXG382HcYB1amsMeolKhNTux6f5eo2i7NvqEmLhJiRiQPJLHWxkzPe0pmN5oydErEh Ik8TGltj4aYgcQt5cvqVdg+/MtE4L+Hi3CyU5fEG1hU8Uv6xohFe6OS/DumlfgeJ6gWM zx2cOPo8QqRX8wyaaw24LaIuqzFRWaLWCnSfG6Azkw0/LYJZ5sh7n1q6uA5utx+oxRxP JzOXUVlAuzTdwTuhfvXuhleusOKIXe4vkCfWPAoBxGvk4S8JuHpPk0Es/zU6VzE0sL2U tJJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=KPxJhmU+AOM39uKEirvdpYHCZWYBg/9IzuGwESksTAQ=; b=nccio9joii7fZZuNefLC1C04ElowYIKW7UQRtO8sFSq75CA/PA2qA9fcfPj83O5zMP KBpytjlL6WXUcX/VKdJSq2/XocH39hCZQeC24L3VrdsQUrvEUaU5eTQL+q6fNzj1OA/X DmM7RB4Aar4crnBTVs4St9rK4zVpGxYHYKX1OPA36dyhRlUj8/RYYWS/j+/JIWmHIhmm xFMcrkhCq1vZ0ebMgWcTkQu0HE6RrjAha6GV/xaNle38AlhNtConBp9PbFEa9oopOUOq zrY8I2N0poqtSWLRKgtROUhTEIVbMmeANMwrXlsYXmL22dvd7XYUYWLTzU1BZCedhmQW dfFQ== X-Gm-Message-State: AJcUukdu/nMEpQmcXm1QlUn7LWqiiIIN+FPb6txcM4sGExhD97IDogVq i9H6/KtHVa99mXQBSL8NqfKUCHZmQ1piluj9viTedGfXVTA= X-Google-Smtp-Source: ALg8bN5tfwsXvmkjyc7IR+N83ROa7a4Y5z/l/TJi7PoLKCr70rh5j0jQIF8RyDkYXqciJsVSs8LsSQ0URV2HDaDNHDo= X-Received: by 2002:a37:90c3:: with SMTP id s186mr1954191qkd.339.1545394376945; Fri, 21 Dec 2018 04:12:56 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Akshay Joshi Date: Fri, 21 Dec 2018 04:12:45 -0800 Message-ID: Subject: Re: [pgAdmin4][Patch] - RM 3780 pgAdmin4 lacks ability to specify NULL values in CSV export To: Dave Page Cc: pgadmin-hackers Content-Type: multipart/alternative; boundary="000000000000e049d3057d872a4c" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --000000000000e049d3057d872a4c Content-Type: text/plain; charset="UTF-8" Hi Dave Can you please attached the updated patch with your changes. I'll try to fix the exception. On Fri, Dec 21, 2018 at 3:57 AM Dave Page wrote: > Hi > > Here's an updated patch as I've tweaked some of the wording. The > screenshot probably isn't the right resolution, but as we're replacing them > anyway it doesn't seem overly important. Feel free to fix if you like :-) > > With quoting set to either All or Strings, everything looks good. With it > set to None, I still get an exception (below). The query I'm using is this: > > SELECT NULL::text, 1234::int, 'Foo bar'::text, E'Foo\nBar'::text > > Field separator: , > Quote character: " > Replace null's with: NULL > > Steps: > > 1) Run pgAdmin in Desktop mode. I'm running from within PyuCharms, using > the venv detailed below. > 2) Open the Query Tool on a PostgreSQL 9.6.10 database, running on MacOS > 10.14.1 > 3) Run the above query, wit quoting set to All and check the result in the > grid. > 4) Download the CSV file and check. > 5) Open Preferences and set quoting to Strings. > 6) Download the CSV file and check. > 7) Open Preferences and set quoting to None. > 8) Download the CSV file *exception occurs*. > > > System info: > > (pgadmin4) dpage@hal:*~/git/pgadmin4*$ python --version > > Python 3.6.7 > > (pgadmin4) dpage@hal:*~/git/pgadmin4*$ pip freeze > > alabaster==0.7.11 > > alembic==1.0.0 > > asn1crypto==0.24.0 > > Babel==2.6.0 > > bcrypt==3.1.4 > > blinker==1.4 > > certifi==2018.8.24 > > cffi==1.11.5 > > chardet==3.0.4 > > chromedriver-installer==0.0.6 > > click==6.7 > > cryptography==2.3 > > docutils==0.14 > > extras==1.0.0 > > fixtures==3.0.0 > > Flask==0.12.4 > > Flask-BabelEx==0.9.3 > > Flask-Gravatar==0.5.0 > > Flask-HTMLmin==1.3.2 > > Flask-Login==0.3.2 > > Flask-Mail==0.9.1 > > Flask-Migrate==2.1.1 > > Flask-Paranoid==0.2.0 > > Flask-Principal==0.4.0 > > Flask-Security==3.0.0 > > Flask-SQLAlchemy==2.3.2 > > Flask-WTF==0.14.2 > > html5lib==1.0.1 > > htmlmin==0.1.12 > > idna==2.7 > > imagesize==1.1.0 > > itsdangerous==0.24 > > Jinja2==2.10 > > linecache2==1.0.0 > > Mako==1.0.7 > > MarkupSafe==1.0 > > packaging==18.0 > > paramiko==2.4.1 > > passlib==1.7.1 > > pbr==3.1.1 > > psutil==5.4.8 > > psycopg2==2.7.5 > > pyasn1==0.4.4 > > pycodestyle==2.3.1 > > pycparser==2.18 > > pycrypto==2.6.1 > > Pygments==2.2.0 > > PyNaCl==1.2.1 > > pyparsing==2.2.2 > > pyperclip==1.6.4 > > pyrsistent==0.14.2 > > python-dateutil==2.7.3 > > python-editor==1.0.3 > > python-mimeparse==1.6.0 > > pytz==2018.3 > > requests==2.19.1 > > selenium==3.14.1 > > simplejson==3.13.2 > > six==1.11.0 > > snowballstemmer==1.2.1 > > speaklater==1.3 > > Sphinx==1.8.2 > > sphinxcontrib-websupport==1.1.0 > > SQLAlchemy==1.2.10 > > sqlparse==0.2.4 > > sshtunnel==0.1.4 > > testscenarios==0.5.0 > > testtools==2.3.0 > > traceback2==1.4.0 > > unittest2==1.1.0 > > urllib3==1.23 > > webencodings==0.5.1 > > Werkzeug==0.14.1 > > WTForms==2.1 > > Exception: > > 2018-12-21 11:47:28,995: SQL pgadmin: Execute (with server cursor) for > server #2 - CONN:8760231 (Query-id: 8649354): > SELECT NULL::text, 1234::int, 'Foo bar'::text, E'Foo\nBar'::text > 2018-12-21 11:47:29,001: INFO werkzeug: 127.0.0.1 - - [21/Dec/2018 > 11:47:29] "GET > /sqleditor/query_tool/download/2133388?query=SELECT%20NULL%3A%3Atext%2C%201234%3A%3Aint%2C%20%27Foo%20bar%27%3A%3Atext%2C%20E%27Foo%5CnBar%27%3A%3Atext&filename=data-1545392848979.csv > HTTP/1.1" 500 - > 2018-12-21 11:47:29,003: ERROR werkzeug: Error on request: > Traceback (most recent call last): > File > "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.6/site-packages/werkzeug/serving.py", > line 270, in run_wsgi > execute(self.server.app) > File > "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.6/site-packages/werkzeug/serving.py", > line 260, in execute > for data in application_iter: > File > "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.6/site-packages/werkzeug/wsgi.py", > line 870, in __next__ > return self._next() > File > "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.6/site-packages/werkzeug/wrappers.py", > line 82, in _iter_encoded > for item in iterable: > File > "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py", > line 848, in gen > csv_writer.writerows(results) > File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line 761, in > writerows > return self.writer.writerows(map(self._dict_to_list, rowdicts)) > File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line 268, in > writerows > self.writerow(row) > File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line 261, in > writerow > row = [self.strategy.prepare(field, only=only) for field in row] > File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line 261, in > > row = [self.strategy.prepare(field, only=only) for field in row] > File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line 142, in > prepare > raise Error('No escapechar is set') > _csv.Error: No escapechar is set > > On Thu, Dec 20, 2018 at 1:05 PM Akshay Joshi < > akshay.joshi@enterprisedb.com> wrote: > >> Hi Dave >> >> On Thu, Dec 20, 2018 at 5:12 PM Akshay Joshi < >> akshay.joshi@enterprisedb.com> wrote: >> >>> >>> >>> On Thu, Dec 20, 2018 at 4:48 PM Dave Page wrote: >>> >>>> Hi >>>> >>>> On Thu, Dec 20, 2018 at 10:09 AM Akshay Joshi < >>>> akshay.joshi@enterprisedb.com> wrote: >>>> >>>>> Hi Dave >>>>> >>>>> On Thu, Dec 20, 2018 at 3:08 PM Dave Page wrote: >>>>> >>>>>> Hi >>>>>> >>>>>> When testing with quoting set to None, quote = " and delimiter = , I >>>>>> get the following exception when I try to download: >>>>>> >>>>>> 2018-12-20 09:34:02,547: SQL pgadmin: Execute (with server cursor) >>>>>> for server #2 - CONN:354106 (Query-id: 4121147): >>>>>> SELECT NULL::text, 1234::int, 'Foo bar'::text, E'Foo\nBar'::text >>>>>> 2018-12-20 09:34:02,570: INFO werkzeug: 127.0.0.1 - - [20/Dec/2018 >>>>>> 09:34:02] "GET >>>>>> /sqleditor/query_tool/download/5610522?query=SELECT%20NULL%3A%3Atext%2C%201234%3A%3Aint%2C%20%27Foo%20bar%27%3A%3Atext%2C%20E%27Foo%5CnBar%27%3A%3Atext&filename=data-1545298442530.csv >>>>>> HTTP/1.1" 500 - >>>>>> 2018-12-20 09:34:02,572: ERROR werkzeug: Error on request: >>>>>> Traceback (most recent call last): >>>>>> File >>>>>> "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.6/site-packages/werkzeug/serving.py", >>>>>> line 270, in run_wsgi >>>>>> execute(self.server.app) >>>>>> File >>>>>> "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.6/site-packages/werkzeug/serving.py", >>>>>> line 260, in execute >>>>>> for data in application_iter: >>>>>> File >>>>>> "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.6/site-packages/werkzeug/wsgi.py", >>>>>> line 870, in __next__ >>>>>> return self._next() >>>>>> File >>>>>> "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.6/site-packages/werkzeug/wrappers.py", >>>>>> line 82, in _iter_encoded >>>>>> for item in iterable: >>>>>> File >>>>>> "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py", >>>>>> line 820, in gen >>>>>> csv_writer.writerows(results) >>>>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line >>>>>> 748, in writerows >>>>>> return self.writer.writerows(map(self._dict_to_list, rowdicts)) >>>>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line >>>>>> 256, in writerows >>>>>> self.writerow(row) >>>>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line >>>>>> 249, in writerow >>>>>> row = [self.strategy.prepare(field, only=only) for field in row] >>>>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line >>>>>> 249, in >>>>>> row = [self.strategy.prepare(field, only=only) for field in row] >>>>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line >>>>>> 136, in prepare >>>>>> raise Error('No escapechar is set') >>>>>> _csv.Error: No escapechar is set >>>>>> >>>>> >>>>> Not able to reproduce the above issue. I have tested it with the >>>>> same setting as you mentioned. Please refer all the attached screenshots. >>>>> Please specify the steps if they are different. >>>>> >>>>>> >>>>>> When I have quoting set to All, the first column is returned as "" >>>>>> >>>>>> dpage@hal:*~/Downloads*$ more data-1545298598112.csv >>>>>> >>>>>> "text","int4","text-2","text-3" >>>>>> >>>>>> "","1234","Foo bar","Foo >>>>>> >>>>>> Bar" >>>>>> >>>>>> Isn't the point for it to be NULL? >>>>>> >>>>> >>>>> while quoting is set to ALL, all the data types has been quoted, >>>>> so I thought null values should be replaced by "" instead of blank. But if >>>>> you think null values shouldn't be quoted even if user select quote ALL, >>>>> I'll fix it and resend the patch. >>>>> >>>> >>>> So how would you distinguish NULL from an empty string? Isn't that >>>> exactly what the bug is about? >>>> >>>> I still think we need a "Replace NULLs with" config option, and >>>> regardless of quoting settings we always replace NULL values with whatever >>>> that is set to - for which the user could then choose options like: >>>> >>>> NULL >>>> "NULL" >>>> "" >>>> '' >>>> >>>> >>>> We would never quote the NULL replacement value - if the user wanted it >>>> to be quoted, they would include the quotes in the configured string. >>>> >>> >>> >>> OK, Will work on it and send the modified patch again. >>> >> >> Attached is the modified patch as per your suggestion. >> >>> >>>> >>>>> >>>>>> On Tue, Dec 18, 2018 at 11:13 AM Akshay Joshi < >>>>>> akshay.joshi@enterprisedb.com> wrote: >>>>>> >>>>>>> Hi Dave >>>>>>> >>>>>>> Attached is the modified patch to fix review comments. >>>>>>> >>>>>>> On Tue, Dec 18, 2018 at 3:00 PM Akshay Joshi < >>>>>>> akshay.joshi@enterprisedb.com> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Dec 18, 2018 at 2:49 PM Dave Page >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi >>>>>>>>> >>>>>>>>> On Tue, Dec 18, 2018 at 3:45 AM Akshay Joshi < >>>>>>>>> akshay.joshi@enterprisedb.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Hackers, >>>>>>>>>> >>>>>>>>>> Attached is the patch to fix RM #3780 pgAdmin4 lacks ability to >>>>>>>>>> specify NULL values in CSV export. >>>>>>>>>> >>>>>>>>>> Please review it. >>>>>>>>>> >>>>>>>>> >>>>>>>>> A few points; >>>>>>>>> >>>>>>>>> - You've included code from backports.csv, but per the licence you >>>>>>>>> need to include a description of the changes made. >>>>>>>>> >>>>>>>> >>>>>>>> Sure. In that case I'll copy the complete file and will do my >>>>>>>> changes which is of two lines only. With my patch I have remove all the >>>>>>>> unwanted code from backport.csv. >>>>>>>> >>>>>>>>> >>>>>>>>> - Shouldn't backports.csv be removed from requirements.txt, or is >>>>>>>>> it used elsewhere? >>>>>>>>> >>>>>>>> >>>>>>>> Yes. Will do that. >>>>>>>> >>>>>>>>> >>>>>>>>> - If the previous point is true, then I'm fairly sure there is >>>>>>>>> code in one or more of the many package build scripts that adds an >>>>>>>>> __init__.py file to backports.csv in the venv that's created. >>>>>>>>> >>>>>>>> >>>>>>>> I'll remove that code as well. >>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Dave Page >>>>>>>>> Blog: http://pgsnake.blogspot.com >>>>>>>>> Twitter: @pgsnake >>>>>>>>> >>>>>>>>> EnterpriseDB UK: http://www.enterprisedb.com >>>>>>>>> The Enterprise PostgreSQL Company >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Akshay Joshi* >>>>>>>> >>>>>>>> *Sr. Software Architect * >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> *Phone: +91 20-3058-9517Mobile: +91 976-788-8246* >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Akshay Joshi* >>>>>>> >>>>>>> *Sr. Software Architect * >>>>>>> >>>>>>> >>>>>>> >>>>>>> *Phone: +91 20-3058-9517Mobile: +91 976-788-8246* >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Dave Page >>>>>> Blog: http://pgsnake.blogspot.com >>>>>> Twitter: @pgsnake >>>>>> >>>>>> EnterpriseDB UK: http://www.enterprisedb.com >>>>>> The Enterprise PostgreSQL Company >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Akshay Joshi* >>>>> >>>>> *Sr. Software Architect * >>>>> >>>>> >>>>> >>>>> *Phone: +91 20-3058-9517Mobile: +91 976-788-8246* >>>>> >>>> >>>> >>>> -- >>>> Dave Page >>>> Blog: http://pgsnake.blogspot.com >>>> Twitter: @pgsnake >>>> >>>> EnterpriseDB UK: http://www.enterprisedb.com >>>> The Enterprise PostgreSQL Company >>>> >>> >>> >>> -- >>> *Akshay Joshi* >>> >>> *Sr. Software Architect * >>> >>> >>> >>> *Phone: +91 20-3058-9517Mobile: +91 976-788-8246* >>> >> >> >> -- >> *Akshay Joshi* >> >> *Sr. Software Architect * >> >> >> >> *Phone: +91 20-3058-9517Mobile: +91 976-788-8246* >> > > > -- > Dave Page > Blog: http://pgsnake.blogspot.com > Twitter: @pgsnake > > EnterpriseDB UK: http://www.enterprisedb.com > The Enterprise PostgreSQL Company > -- *Akshay Joshi* *Sr. Software Architect * *Phone: +91 20-3058-9517Mobile: +91 976-788-8246* --000000000000e049d3057d872a4c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Dave

Can you please = attached the updated patch with your changes. I'll try to fix the excep= tion.

On Fri= , Dec 21, 2018 at 3:57 AM Dave Page <dpage@pgadmin.org> wrote:
Hi

Here's an updated patch as I've t= weaked some of the wording. The screenshot probably isn't the right res= olution, but as we're replacing them anyway it doesn't seem overly = important. Feel free to fix if you like :-)

With q= uoting set to either All or Strings, everything looks good. With it set to = None, I still get an exception (below). The query I'm using is this:

SELECT NULL::text, 1234::int, 'Foo bar'::tex= t, E'Foo\nBar'::text

Field separator: = ,
Quote character: "
Replace null's with: NULL=

Steps:

1) Run pgAdmin in= Desktop mode. I'm running from within PyuCharms, using the venv detail= ed below.
2) Open the Query Tool on a PostgreSQL 9.6.10 database,= running on MacOS 10.14.1
3) Run the above query, wit quoting set= to All and check the result in the grid.
4) Download the CSV fil= e and check.
5) Open Preferences and set quoting to Strings.
6) Download the CSV file and check.
7) Open Preferences and= set quoting to None.
8) Download the CSV file *exception occurs*= .


System info:

=

(pgadmin4) <= /span>dpage@hal:~/git/pgadmin4$ python --vers= ion

Python 3.6.7=

(pgadmin4) <= /span>dpage@hal:~/git/pgadmin4$ pip freeze

alabaster=3D= =3D0.7.11

alembic=3D= =3D1.0.0

asn1crypto= =3D=3D0.24.0

Babel=3D=3D2= .6.0

bcrypt=3D=3D= 3.1.4

blinker=3D= =3D1.4

certifi=3D= =3D2018.8.24

cffi=3D=3D1.= 11.5

chardet=3D= =3D3.0.4

chromedriver= -installer=3D=3D0.0.6

click=3D=3D6= .7

cryptography= =3D=3D2.3

docutils=3D= =3D0.14

extras=3D=3D= 1.0.0

fixtures=3D= =3D3.0.0

Flask=3D=3D0= .12.4

Flask-BabelE= x=3D=3D0.9.3

Flask-Gravat= ar=3D=3D0.5.0

Flask-HTMLmi= n=3D=3D1.3.2

Flask-Login= =3D=3D0.3.2

Flask-Mail= =3D=3D0.9.1

Flask-Migrat= e=3D=3D2.1.1

Flask-Parano= id=3D=3D0.2.0

Flask-Princi= pal=3D=3D0.4.0

Flask-Securi= ty=3D=3D3.0.0

Flask-SQLAlc= hemy=3D=3D2.3.2

Flask-WTF=3D= =3D0.14.2

html5lib=3D= =3D1.0.1

htmlmin=3D= =3D0.1.12

idna=3D=3D2.= 7

imagesize=3D= =3D1.1.0

itsdangerous= =3D=3D0.24

Jinja2=3D=3D= 2.10

linecache2= =3D=3D1.0.0

Mako=3D=3D1.= 0.7

MarkupSafe= =3D=3D1.0

packaging=3D= =3D18.0

paramiko=3D= =3D2.4.1

passlib=3D= =3D1.7.1

pbr=3D=3D3.1= .1

psutil=3D=3D= 5.4.8

psycopg2=3D= =3D2.7.5

pyasn1=3D=3D= 0.4.4

pycodestyle= =3D=3D2.3.1

pycparser=3D= =3D2.18

pycrypto=3D= =3D2.6.1

Pygments=3D= =3D2.2.0

PyNaCl=3D=3D= 1.2.1

pyparsing=3D= =3D2.2.2

pyperclip=3D= =3D1.6.4

pyrsistent= =3D=3D0.14.2

python-dateu= til=3D=3D2.7.3

python-edito= r=3D=3D1.0.3

python-mimep= arse=3D=3D1.6.0

pytz=3D=3D20= 18.3

requests=3D= =3D2.19.1

selenium=3D= =3D3.14.1

simplejson= =3D=3D3.13.2

six=3D=3D1.1= 1.0

snowballstem= mer=3D=3D1.2.1

speaklater= =3D=3D1.3

Sphinx=3D=3D= 1.8.2

sphinxcontri= b-websupport=3D=3D1.1.0

SQLAlchemy= =3D=3D1.2.10

sqlparse=3D= =3D0.2.4

sshtunnel=3D= =3D0.1.4

testscenario= s=3D=3D0.5.0

testtools=3D= =3D2.3.0

traceback2= =3D=3D1.4.0

unittest2=3D= =3D1.1.0

urllib3=3D= =3D1.23

webencodings= =3D=3D0.5.1

Werkzeug=3D= =3D0.14.1

WTForms=3D= =3D2.1


Exception:

2018-12-21 11:47:28,995: SQL pgadmin: Exe= cute (with server cursor) for server #2 - CONN:8760231 (Query-id: 8649354):=
SELECT NULL::text, 1234::int, 'Foo bar'::text, E'Foo= \nBar'::text
2018-12-21 11:47:29,001: INFO werkzeug: 127.0.0.1 - - [21/Dec/2018 11:47:29] "GET /sqleditor/query_tool/d= ownload/2133388?query=3DSELECT%20NULL%3A%3Atext%2C%201234%3A%3Aint%2C%20%27= Foo%20bar%27%3A%3Atext%2C%20E%27Foo%5CnBar%27%3A%3Atext&filename=3Ddata= -1545392848979.csv HTTP/1.1" 500 -
2018-12-21 11:47:29,003: = ERROR werkzeug: Error on request:
Traceback (most re= cent call last):
=C2=A0 File "/Users/dpage/.virtualenvs/pgad= min4/lib/python3.6/site-packages/werkzeug/serving.py", line 270, in ru= n_wsgi
=C2=A0 =C2=A0 execute(self.server.app)
=C2=A0 Fi= le "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.6/site-packages/wer= kzeug/serving.py", line 260, in execute
=C2=A0 =C2=A0 for da= ta in application_iter:
=C2=A0 File "/Users/dpage/.virtualen= vs/pgadmin4/lib/python3.6/site-packages/werkzeug/wsgi.py", line 870, i= n __next__
=C2=A0 =C2=A0 return self._next()
=C2=A0 Fil= e "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.6/site-packages/werk= zeug/wrappers.py", line 82, in _iter_encoded
=C2=A0 =C2=A0 f= or item in iterable:
=C2=A0 File "/Users/dpage/git/pgadmin4/= web/pgadmin/utils/driver/psycopg2/connection.py", line 848, in gen
=C2=A0 =C2=A0 csv_writer.writerows(results)
=C2=A0 File &q= uot;/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line 761, in = writerows
=C2=A0 =C2=A0 return self.writer.writerows(map(self._di= ct_to_list, rowdicts))
=C2=A0 File "/Users/dpage/git/pgadmin= 4/web/pgadmin/utils/csv.py", line 268, in writerows
=C2=A0 = =C2=A0 self.writerow(row)
=C2=A0 File "/Users/dpage/git/pgad= min4/web/pgadmin/utils/csv.py", line 261, in writerow
=C2=A0= =C2=A0 row =3D [self.strategy.prepare(field, only=3Donly) for field in row= ]
=C2=A0 File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/c= sv.py", line 261, in <listcomp>
=C2=A0 =C2=A0 row =3D = [self.strategy.prepare(field, only=3Donly) for field in row]
=C2= =A0 File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", li= ne 142, in prepare
=C2=A0 =C2=A0 raise Error('No escapechar i= s set')
_csv.Error: No escapechar is set

On Thu, Dec 20, 2018 at 1:05 PM Aksh= ay Joshi <akshay.joshi@enterprisedb.com> wrote:
Hi=C2= =A0Dave

On Thu, Dec 20= , 2018 at 5:12 PM Akshay Joshi <akshay.joshi@enterprisedb.com> wrote:
=


On= Thu, Dec 20, 2018 at 4:48 PM Dave Page <dpage@pgadmin.org> wrote:
Hi

On Thu, Dec 20, 2018 at 10:09 AM Aksha= y Joshi <akshay.joshi@enterprisedb.com> wrote:
Hi Dave<= /div>
On Thu, Dec 20, 2018 a= t 3:08 PM Dave Page <dpage@pgadmin.org> wrote:
Hi

When testing with quoting set to None, quote =3D " and delimiter =3D= , I get the following exception when I try to download:

2018-12-20 09:34:02,547: SQL pgadmin: Execute (wit= h server cursor) for server #2 - CONN:354106 (Query-id: 4121147):
SELECT NULL::text, 1234::int, 'Foo bar'::text, E'Foo\nBar'= ::text
2018-12-20 09:34:02,570: INFO werkzeug: 127.0= .0.1 - - [20/Dec/2018 09:34:02] "GET /sqleditor/query_tool/download/56= 10522?query=3DSELECT%20NULL%3A%3Atext%2C%201234%3A%3Aint%2C%20%27Foo%20bar%= 27%3A%3Atext%2C%20E%27Foo%5CnBar%27%3A%3Atext&filename=3Ddata-154529844= 2530.csv HTTP/1.1" 500 -
2018-12-20 09:34:02,572: ERROR werkzeug: Error on request:
Traceback (most recent call = last):
=C2=A0 File "/Users/dpage/.virtualenvs/pgadmin4/lib/p= ython3.6/site-packages/werkzeug/serving.py", line 270, in run_wsgi
=C2=A0 =C2=A0 execute(self.server.app)
=C2=A0 File "/= Users/dpage/.virtualenvs/pgadmin4/lib/python3.6/site-packages/werkzeug/serv= ing.py", line 260, in execute
=C2=A0 =C2=A0 for data in appl= ication_iter:
=C2=A0 File "/Users/dpage/.virtualenvs/pgadmin= 4/lib/python3.6/site-packages/werkzeug/wsgi.py", line 870, in __next__=
=C2=A0 =C2=A0 return self._next()
=C2=A0 File "/U= sers/dpage/.virtualenvs/pgadmin4/lib/python3.6/site-packages/werkzeug/wrapp= ers.py", line 82, in _iter_encoded
=C2=A0 =C2=A0 for item in= iterable:
=C2=A0 File "/Users/dpage/git/pgadmin4/web/pgadmi= n/utils/driver/psycopg2/connection.py", line 820, in gen
=C2= =A0 =C2=A0 csv_writer.writerows(results)
=C2=A0 File "/Users= /dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line 748, in writerows<= /div>
=C2=A0 =C2=A0 return self.writer.writerows(map(self._dict_to_list= , rowdicts))
=C2=A0 File "/Users/dpage/git/pgadmin4/web/pgad= min/utils/csv.py", line 256, in writerows
=C2=A0 =C2=A0 self= .writerow(row)
=C2=A0 File "/Users/dpage/git/pgadmin4/web/pg= admin/utils/csv.py", line 249, in writerow
=C2=A0 =C2=A0 row= =3D [self.strategy.prepare(field, only=3Donly) for field in row]
=C2=A0 File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py"= , line 249, in <listcomp>
=C2=A0 =C2=A0 row =3D [self.strat= egy.prepare(field, only=3Donly) for field in row]
=C2=A0 File &qu= ot;/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line 136, in p= repare
=C2=A0 =C2=A0 raise Error('No escapechar is set')<= /div>
_csv.Error: No escapechar is set

=C2=A0 =C2=A0 Not able to reproduce the above issue= . I have tested it with the same setting as you mentioned. Please refer all= the attached screenshots. Please specify the steps if they are different.<= /div>

When I have quoting set to All, the = first column is returned as ""

dpage@hal:~/Downloads$ more data-1545298598112.csv=C2=A0

"text","int4",&quo= t;text-2","text-3"

"","1234","Fo= o bar","Foo

Bar"


Isn't the point for it= to be NULL?

= =C2=A0 =C2=A0 while quoting is set to ALL, all the data types has been quot= ed, so I thought null values should be replaced by "" instead of = blank. But if you think null values shouldn't be quoted even if user se= lect quote ALL, I'll fix it and resend the patch.

So how would you distinguish NULL from an empt= y string? Isn't that exactly what the bug is about?

I still think we need a "Replace NULLs with" config option,= and regardless of quoting settings we always replace NULL values with what= ever that is set to=C2=A0 - for which the user could then choose options li= ke:

NULL
"NULL"
&quo= t;"
''
<empty string>

=
We would never quote the NULL replacement value - if the user wa= nted it to be quoted, they would include the quotes in the configured strin= g.


=C2=A0 = =C2=A0OK, Will work on it and send the modified patch again.=C2=A0

=C2=A0 =C2=A0 =C2=A0 Attached is = the modified patch as per your suggestion.=C2=A0
=C2=A0

On Tue, De= c 18, 2018 at 11:13 AM Akshay Joshi <akshay.joshi@enterprisedb.com> wrote= :
Hi=C2=A0Dave

Attached is the modified patch to fix re= view comments.

O= n Tue, Dec 18, 2018 at 3:00 PM Akshay Joshi <akshay.joshi@enterprisedb.com&g= t; wrote:


On Tue, Dec 18, 2018 at 2:49 PM Dave Page <dpage@pgadmin.org> wrote:
Hi
On Tue, Dec 18, 2018 at 3= :45 AM Akshay Joshi <akshay.joshi@enterprisedb.com> wrote:
Hi Hackers,

Attached is the patch to fix RM #3780=C2= =A0pgAdmin4 lacks ability to specify NULL values in CSV export.

Please review it.

A few points;

- You've included code = from backports.csv, but per the licence you need to include a description o= f the changes made.

=C2= =A0 =C2=A0 =C2=A0 Sure. In that case I'll copy the complete file and wi= ll do my changes which is of two lines only. With my patch I have remove al= l the unwanted code from backport.csv.=C2=A0 =C2=A0 =C2=A0=C2=A0

- Shouldn't backports.csv be removed = from requirements.txt, or is it used elsewhere?

=C2=A0 =C2=A0 =C2=A0Yes. Will do that.=C2=A0

- If the previous point is true, then = I'm fairly sure there is code in one or more of the many package build = scripts that adds an __init__.py file to backports.csv in the venv that'= ;s created.

=C2=A0 =C2=A0= I'll remove that code as well.=C2=A0=C2=A0
=C2=A0
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
=
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


--
Akshay Joshi<= /div>
Sr. Softwar= e Architect

Phone: +91 20-3058-9517
Mobile: +91 976-788-8246


--
Akshay Joshi
Sr. Software Architect
=

<= div>Phone: +91 20-305= 8-9517
Mobile: +91 976-788-8246


--
Dave Page
Blog: http://pgsnake.blogspot.com=
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterpr= ise PostgreSQL Company


--
Akshay Joshi
Sr. Software Archi= tect

<= b>

Phone: +91 20-3058-9517
Mobile: +91 976-788-8246


--
Dave PageBlog: http://pgs= nake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.= com
The Enterprise PostgreSQL Company


--
<= div>Akshay Joshi
Sr. Software Architect


Phone: +91 20-3058-9517
Mobile: +91 976-788-8246=


--
Akshay Joshi
Sr. Software Architect <= br>


=
Pho= ne: +91 20-3058-9517
Mobile: +91 976-788-8246


--
Dave Page
Blog: <= a href=3D"http://pgsnake.blogspot.com" target=3D"_blank">http://pgsnake.blo= gspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.comThe Enterprise PostgreSQL Company


--
Akshay Joshi
=
Sr. Software Arc= hitect


Phone: +91 20-3058-9517
Mobile: +91 976-788-8246<= /div>
--000000000000e049d3057d872a4c--