Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1gaJR1-000235-2q for pgadmin-hackers@arkaria.postgresql.org; Fri, 21 Dec 2018 11:57:43 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1gaJQz-0005A9-Hw for pgadmin-hackers@arkaria.postgresql.org; Fri, 21 Dec 2018 11:57:41 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1gaJQz-0005A2-9E for pgadmin-hackers@lists.postgresql.org; Fri, 21 Dec 2018 11:57:41 +0000 Received: from mail-wm1-x342.google.com ([2a00:1450:4864:20::342]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1gaJQq-0006Se-8M for pgadmin-hackers@postgresql.org; Fri, 21 Dec 2018 11:57:39 +0000 Received: by mail-wm1-x342.google.com with SMTP id r24so14074429wmh.0 for ; Fri, 21 Dec 2018 03:57:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pgadmin-org.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=wX847JZiTxMy7swg4u6tRv2g/00Y7OTk1APee4fVzeg=; b=Mm5dmXm3vurDn2uZn72J6unHsctc+HICFAi2xd/j12jt/V7L4GlsPi+VrdKMEd40Fl 4g1kQFPPf7yQ1M1Zezj5bmZxtwt0c/PYtZYLHQCHSQS151+tsRdC6AZ6MWglJndl6KEf aCEePHyfOKzmPfP1xgyjAeOmYkfIyfbYt74N1hyB7G0eZLzOICQrTkowEWBqMjqpWATj cX8vHR6h6RfePUJaNdS4dJHCbV/qr9Xi+6fv9WTW5bAkcIiF9s3xRNrBwJNkH7wlGs3T Hx9wUj+HK0SBiIuwZgyc+QgblSABxVYLC27gIvCOYMF8ATloeQb61GvcpCIyMhrv+tCw t4/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=wX847JZiTxMy7swg4u6tRv2g/00Y7OTk1APee4fVzeg=; b=E/nP18tF1SI0jZ2VbucPC7TNNPycoPPH3kudoRLrZ43x8P8CF23YX0/b6OVkMEZro0 2CTD7Ua8P/QOKP+NRajcXJzMa07I2Hdqtk3eQ3fcOttm8c91s8weks7I2e5CSC+bgPzp sUnYUMl0hwAbhK9NX1JK3lP9qG9uMVrB6RvRLw7Z0SeEYs3mKRhg6sMQRQ0kwlLEZcOQ fjGM7DTncrPd+KfBuIQLOHYf22yxdtJnkwlegVOuplJpwvEq4stXItAori4MKiiL2hHj eQ+7I201eO3QT1EegOhslEmcE5FLUhTb68oEOXWifDTZlVUO/MnSXT5+kaJND3Y6QJpF aybA== X-Gm-Message-State: AA+aEWYo4RqcFTHqiwUYsjcneItXV9GmdHEH57krGtyMjvGw/ND0mztD AXegxvHfja2CvHVsJrWAhC8bhWNu/6DJNdE76hLH7nSOKw9XZg== X-Google-Smtp-Source: ALg8bN7CMTompLTl3C4XbWcB3HQ3b0dx62apO6mnaI0+pjYR5UWAdSl2tMG4Djkwh/U79tREW1VatH3eVps1PsbFoY4= X-Received: by 2002:a1c:7719:: with SMTP id t25mr2754729wmi.7.1545393448627; Fri, 21 Dec 2018 03:57:28 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Dave Page Date: Fri, 21 Dec 2018 11:57:17 +0000 Message-ID: Subject: Re: [pgAdmin4][Patch] - RM 3780 pgAdmin4 lacks ability to specify NULL values in CSV export To: Akshay Joshi Cc: pgadmin-hackers Content-Type: multipart/alternative; boundary="0000000000008b4632057d86f374" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --0000000000008b4632057d86f374 Content-Type: text/plain; charset="UTF-8" Hi Here's an updated patch as I've tweaked some of the wording. The screenshot probably isn't the right resolution, but as we're replacing them anyway it doesn't seem overly important. Feel free to fix if you like :-) With quoting set to either All or Strings, everything looks good. With it set to None, I still get an exception (below). The query I'm using is this: SELECT NULL::text, 1234::int, 'Foo bar'::text, E'Foo\nBar'::text Field separator: , Quote character: " Replace null's with: NULL Steps: 1) Run pgAdmin in Desktop mode. I'm running from within PyuCharms, using the venv detailed below. 2) Open the Query Tool on a PostgreSQL 9.6.10 database, running on MacOS 10.14.1 3) Run the above query, wit quoting set to All and check the result in the grid. 4) Download the CSV file and check. 5) Open Preferences and set quoting to Strings. 6) Download the CSV file and check. 7) Open Preferences and set quoting to None. 8) Download the CSV file *exception occurs*. System info: (pgadmin4) dpage@hal:*~/git/pgadmin4*$ python --version Python 3.6.7 (pgadmin4) dpage@hal:*~/git/pgadmin4*$ pip freeze alabaster==0.7.11 alembic==1.0.0 asn1crypto==0.24.0 Babel==2.6.0 bcrypt==3.1.4 blinker==1.4 certifi==2018.8.24 cffi==1.11.5 chardet==3.0.4 chromedriver-installer==0.0.6 click==6.7 cryptography==2.3 docutils==0.14 extras==1.0.0 fixtures==3.0.0 Flask==0.12.4 Flask-BabelEx==0.9.3 Flask-Gravatar==0.5.0 Flask-HTMLmin==1.3.2 Flask-Login==0.3.2 Flask-Mail==0.9.1 Flask-Migrate==2.1.1 Flask-Paranoid==0.2.0 Flask-Principal==0.4.0 Flask-Security==3.0.0 Flask-SQLAlchemy==2.3.2 Flask-WTF==0.14.2 html5lib==1.0.1 htmlmin==0.1.12 idna==2.7 imagesize==1.1.0 itsdangerous==0.24 Jinja2==2.10 linecache2==1.0.0 Mako==1.0.7 MarkupSafe==1.0 packaging==18.0 paramiko==2.4.1 passlib==1.7.1 pbr==3.1.1 psutil==5.4.8 psycopg2==2.7.5 pyasn1==0.4.4 pycodestyle==2.3.1 pycparser==2.18 pycrypto==2.6.1 Pygments==2.2.0 PyNaCl==1.2.1 pyparsing==2.2.2 pyperclip==1.6.4 pyrsistent==0.14.2 python-dateutil==2.7.3 python-editor==1.0.3 python-mimeparse==1.6.0 pytz==2018.3 requests==2.19.1 selenium==3.14.1 simplejson==3.13.2 six==1.11.0 snowballstemmer==1.2.1 speaklater==1.3 Sphinx==1.8.2 sphinxcontrib-websupport==1.1.0 SQLAlchemy==1.2.10 sqlparse==0.2.4 sshtunnel==0.1.4 testscenarios==0.5.0 testtools==2.3.0 traceback2==1.4.0 unittest2==1.1.0 urllib3==1.23 webencodings==0.5.1 Werkzeug==0.14.1 WTForms==2.1 Exception: 2018-12-21 11:47:28,995: SQL pgadmin: Execute (with server cursor) for server #2 - CONN:8760231 (Query-id: 8649354): SELECT NULL::text, 1234::int, 'Foo bar'::text, E'Foo\nBar'::text 2018-12-21 11:47:29,001: INFO werkzeug: 127.0.0.1 - - [21/Dec/2018 11:47:29] "GET /sqleditor/query_tool/download/2133388?query=SELECT%20NULL%3A%3Atext%2C%201234%3A%3Aint%2C%20%27Foo%20bar%27%3A%3Atext%2C%20E%27Foo%5CnBar%27%3A%3Atext&filename=data-1545392848979.csv HTTP/1.1" 500 - 2018-12-21 11:47:29,003: ERROR werkzeug: Error on request: Traceback (most recent call last): File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.6/site-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.6/site-packages/werkzeug/serving.py", line 260, in execute for data in application_iter: File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.6/site-packages/werkzeug/wsgi.py", line 870, in __next__ return self._next() File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.6/site-packages/werkzeug/wrappers.py", line 82, in _iter_encoded for item in iterable: File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py", line 848, in gen csv_writer.writerows(results) File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line 761, in writerows return self.writer.writerows(map(self._dict_to_list, rowdicts)) File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line 268, in writerows self.writerow(row) File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line 261, in writerow row = [self.strategy.prepare(field, only=only) for field in row] File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line 261, in row = [self.strategy.prepare(field, only=only) for field in row] File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line 142, in prepare raise Error('No escapechar is set') _csv.Error: No escapechar is set On Thu, Dec 20, 2018 at 1:05 PM Akshay Joshi wrote: > Hi Dave > > On Thu, Dec 20, 2018 at 5:12 PM Akshay Joshi < > akshay.joshi@enterprisedb.com> wrote: > >> >> >> On Thu, Dec 20, 2018 at 4:48 PM Dave Page wrote: >> >>> Hi >>> >>> On Thu, Dec 20, 2018 at 10:09 AM Akshay Joshi < >>> akshay.joshi@enterprisedb.com> wrote: >>> >>>> Hi Dave >>>> >>>> On Thu, Dec 20, 2018 at 3:08 PM Dave Page wrote: >>>> >>>>> Hi >>>>> >>>>> When testing with quoting set to None, quote = " and delimiter = , I >>>>> get the following exception when I try to download: >>>>> >>>>> 2018-12-20 09:34:02,547: SQL pgadmin: Execute (with server cursor) >>>>> for server #2 - CONN:354106 (Query-id: 4121147): >>>>> SELECT NULL::text, 1234::int, 'Foo bar'::text, E'Foo\nBar'::text >>>>> 2018-12-20 09:34:02,570: INFO werkzeug: 127.0.0.1 - - [20/Dec/2018 >>>>> 09:34:02] "GET >>>>> /sqleditor/query_tool/download/5610522?query=SELECT%20NULL%3A%3Atext%2C%201234%3A%3Aint%2C%20%27Foo%20bar%27%3A%3Atext%2C%20E%27Foo%5CnBar%27%3A%3Atext&filename=data-1545298442530.csv >>>>> HTTP/1.1" 500 - >>>>> 2018-12-20 09:34:02,572: ERROR werkzeug: Error on request: >>>>> Traceback (most recent call last): >>>>> File >>>>> "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.6/site-packages/werkzeug/serving.py", >>>>> line 270, in run_wsgi >>>>> execute(self.server.app) >>>>> File >>>>> "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.6/site-packages/werkzeug/serving.py", >>>>> line 260, in execute >>>>> for data in application_iter: >>>>> File >>>>> "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.6/site-packages/werkzeug/wsgi.py", >>>>> line 870, in __next__ >>>>> return self._next() >>>>> File >>>>> "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.6/site-packages/werkzeug/wrappers.py", >>>>> line 82, in _iter_encoded >>>>> for item in iterable: >>>>> File >>>>> "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py", >>>>> line 820, in gen >>>>> csv_writer.writerows(results) >>>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line 748, >>>>> in writerows >>>>> return self.writer.writerows(map(self._dict_to_list, rowdicts)) >>>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line 256, >>>>> in writerows >>>>> self.writerow(row) >>>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line 249, >>>>> in writerow >>>>> row = [self.strategy.prepare(field, only=only) for field in row] >>>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line 249, >>>>> in >>>>> row = [self.strategy.prepare(field, only=only) for field in row] >>>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line 136, >>>>> in prepare >>>>> raise Error('No escapechar is set') >>>>> _csv.Error: No escapechar is set >>>>> >>>> >>>> Not able to reproduce the above issue. I have tested it with the >>>> same setting as you mentioned. Please refer all the attached screenshots. >>>> Please specify the steps if they are different. >>>> >>>>> >>>>> When I have quoting set to All, the first column is returned as "" >>>>> >>>>> dpage@hal:*~/Downloads*$ more data-1545298598112.csv >>>>> >>>>> "text","int4","text-2","text-3" >>>>> >>>>> "","1234","Foo bar","Foo >>>>> >>>>> Bar" >>>>> >>>>> Isn't the point for it to be NULL? >>>>> >>>> >>>> while quoting is set to ALL, all the data types has been quoted, so >>>> I thought null values should be replaced by "" instead of blank. But if you >>>> think null values shouldn't be quoted even if user select quote ALL, I'll >>>> fix it and resend the patch. >>>> >>> >>> So how would you distinguish NULL from an empty string? Isn't that >>> exactly what the bug is about? >>> >>> I still think we need a "Replace NULLs with" config option, and >>> regardless of quoting settings we always replace NULL values with whatever >>> that is set to - for which the user could then choose options like: >>> >>> NULL >>> "NULL" >>> "" >>> '' >>> >>> >>> We would never quote the NULL replacement value - if the user wanted it >>> to be quoted, they would include the quotes in the configured string. >>> >> >> >> OK, Will work on it and send the modified patch again. >> > > Attached is the modified patch as per your suggestion. > >> >>> >>>> >>>>> On Tue, Dec 18, 2018 at 11:13 AM Akshay Joshi < >>>>> akshay.joshi@enterprisedb.com> wrote: >>>>> >>>>>> Hi Dave >>>>>> >>>>>> Attached is the modified patch to fix review comments. >>>>>> >>>>>> On Tue, Dec 18, 2018 at 3:00 PM Akshay Joshi < >>>>>> akshay.joshi@enterprisedb.com> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Dec 18, 2018 at 2:49 PM Dave Page wrote: >>>>>>> >>>>>>>> Hi >>>>>>>> >>>>>>>> On Tue, Dec 18, 2018 at 3:45 AM Akshay Joshi < >>>>>>>> akshay.joshi@enterprisedb.com> wrote: >>>>>>>> >>>>>>>>> Hi Hackers, >>>>>>>>> >>>>>>>>> Attached is the patch to fix RM #3780 pgAdmin4 lacks ability to >>>>>>>>> specify NULL values in CSV export. >>>>>>>>> >>>>>>>>> Please review it. >>>>>>>>> >>>>>>>> >>>>>>>> A few points; >>>>>>>> >>>>>>>> - You've included code from backports.csv, but per the licence you >>>>>>>> need to include a description of the changes made. >>>>>>>> >>>>>>> >>>>>>> Sure. In that case I'll copy the complete file and will do my >>>>>>> changes which is of two lines only. With my patch I have remove all the >>>>>>> unwanted code from backport.csv. >>>>>>> >>>>>>>> >>>>>>>> - Shouldn't backports.csv be removed from requirements.txt, or is >>>>>>>> it used elsewhere? >>>>>>>> >>>>>>> >>>>>>> Yes. Will do that. >>>>>>> >>>>>>>> >>>>>>>> - If the previous point is true, then I'm fairly sure there is code >>>>>>>> in one or more of the many package build scripts that adds an __init__.py >>>>>>>> file to backports.csv in the venv that's created. >>>>>>>> >>>>>>> >>>>>>> I'll remove that code as well. >>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Dave Page >>>>>>>> Blog: http://pgsnake.blogspot.com >>>>>>>> Twitter: @pgsnake >>>>>>>> >>>>>>>> EnterpriseDB UK: http://www.enterprisedb.com >>>>>>>> The Enterprise PostgreSQL Company >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Akshay Joshi* >>>>>>> >>>>>>> *Sr. Software Architect * >>>>>>> >>>>>>> >>>>>>> >>>>>>> *Phone: +91 20-3058-9517Mobile: +91 976-788-8246* >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Akshay Joshi* >>>>>> >>>>>> *Sr. Software Architect * >>>>>> >>>>>> >>>>>> >>>>>> *Phone: +91 20-3058-9517Mobile: +91 976-788-8246* >>>>>> >>>>> >>>>> >>>>> -- >>>>> Dave Page >>>>> Blog: http://pgsnake.blogspot.com >>>>> Twitter: @pgsnake >>>>> >>>>> EnterpriseDB UK: http://www.enterprisedb.com >>>>> The Enterprise PostgreSQL Company >>>>> >>>> >>>> >>>> -- >>>> *Akshay Joshi* >>>> >>>> *Sr. Software Architect * >>>> >>>> >>>> >>>> *Phone: +91 20-3058-9517Mobile: +91 976-788-8246* >>>> >>> >>> >>> -- >>> Dave Page >>> Blog: http://pgsnake.blogspot.com >>> Twitter: @pgsnake >>> >>> EnterpriseDB UK: http://www.enterprisedb.com >>> The Enterprise PostgreSQL Company >>> >> >> >> -- >> *Akshay Joshi* >> >> *Sr. Software Architect * >> >> >> >> *Phone: +91 20-3058-9517Mobile: +91 976-788-8246* >> > > > -- > *Akshay Joshi* > > *Sr. Software Architect * > > > > *Phone: +91 20-3058-9517Mobile: +91 976-788-8246* > -- Dave Page Blog: http://pgsnake.blogspot.com Twitter: @pgsnake EnterpriseDB UK: http://www.enterprisedb.com The Enterprise PostgreSQL Company --0000000000008b4632057d86f374 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi
=
Here's an updated patch as I've tweaked some of the = wording. The screenshot probably isn't the right resolution, but as we&= #39;re replacing them anyway it doesn't seem overly important. Feel fre= e to fix if you like :-)

With quoting set to eithe= r All or Strings, everything looks good. With it set to None, I still get a= n exception (below). The query I'm using is this:

<= div>SELECT NULL::text, 1234::int, 'Foo bar'::text, E'Foo\nBar&#= 39;::text

Field separator: ,
Quote c= haracter: "
Replace null's with: NULL

Steps:

1) Run pgAdmin in Desktop mode. I= 9;m running from within PyuCharms, using the venv detailed below.
2) Open the Query Tool on a PostgreSQL 9.6.10 database, running on MacOS 1= 0.14.1
3) Run the above query, wit quoting set to All and check t= he result in the grid.
4) Download the CSV file and check.
<= div>5) Open Preferences and set quoting to Strings.
6) Download t= he CSV file and check.
7) Open Preferences and set quoting to Non= e.
8) Download the CSV file *exception occurs*.


System info:

(pgadmin4) dpage@= hal:~/git/pgadmi= n4$ python --version

Python 3.6.7

(pgadmin4) dpage@= hal:~/git/pgadmi= n4$ pip freeze

alabaster=3D=3D0.7.11

alembic=3D=3D1.0.0

asn1crypto=3D=3D0.24.0

Babel=3D=3D2.6.0

bcrypt=3D=3D3.1.4

blinker=3D=3D1.4

certifi=3D=3D2018.8.24

cffi=3D=3D1.11.5

chardet=3D=3D3.0.4

chromedriver-installer=3D=3D0.0.6

click=3D=3D6.7

cryptography=3D=3D2.3

docutils=3D=3D0.14

extras=3D=3D1.0.0

fixtures=3D=3D3.0.0

Flask=3D=3D0.12.4

Flask-BabelEx=3D=3D0.9.3

Flask-Gravatar=3D=3D0.5.0

Flask-HTMLmin=3D=3D1.3.2

Flask-Login=3D=3D0.3.2

Flask-Mail=3D=3D0.9.1

Flask-Migrate=3D=3D2.1.1

Flask-Paranoid=3D=3D0.2.0

Flask-Principal=3D=3D0.4.0

Flask-Security=3D=3D3.0.0

Flask-SQLAlchemy=3D=3D2.3.2

Flask-WTF=3D=3D0.14.2

html5lib=3D=3D1.0.1

htmlmin=3D=3D0.1.12

idna=3D=3D2.7

imagesize=3D=3D1.1.0

itsdangerous=3D=3D0.24

Jinja2=3D=3D2.10

linecache2=3D=3D1.0.0

Mako=3D=3D1.0.7

MarkupSafe=3D=3D1.0

packaging=3D=3D18.0

paramiko=3D=3D2.4.1

passlib=3D=3D1.7.1

pbr=3D=3D3.1.1

psutil=3D=3D5.4.8

psycopg2=3D=3D2.7.5

pyasn1=3D=3D0.4.4

pycodestyle=3D=3D2.3.1

pycparser=3D=3D2.18

pycrypto=3D=3D2.6.1

Pygments=3D=3D2.2.0

PyNaCl=3D=3D1.2.1

pyparsing=3D=3D2.2.2

pyperclip=3D=3D1.6.4

pyrsistent=3D=3D0.14.2

python-dateutil=3D=3D2.7.3

python-editor=3D=3D1.0.3

python-mimeparse=3D=3D1.6.0

pytz=3D=3D2018.3

requests=3D=3D2.19.1

selenium=3D=3D3.14.1

simplejson=3D=3D3.13.2

six=3D=3D1.11.0

snowballstemmer=3D=3D1.2.1

speaklater=3D=3D1.3

Sphinx=3D=3D1.8.2

sphinxcontrib-websupport=3D=3D1.1.0

SQLAlchemy=3D=3D1.2.10

sqlparse=3D=3D0.2.4

sshtunnel=3D=3D0.1.4

testscenarios=3D=3D0.5.0

testtools=3D=3D2.3.0

traceback2=3D=3D1.4.0

unittest2=3D=3D1.1.0

urllib3=3D=3D1.23

webencodings=3D=3D0.5.1

Werkzeug=3D=3D0.14.1

WTForms=3D=3D2.1


Exc= eption:

2018-12-21 11:47:28,995: SQL pgadmin: Execute (with server cursor) for server #2 - CONN:8760231 (Query-id: 86= 49354):
SELECT NULL::text, 1234::int, 'Foo bar'::text, E&= #39;Foo\nBar'::text
2018-12-21 11:47:29,001: INFO werkzeug: 127.0.0.1 - - [21/Dec/2018 11:47:29] "GET /sqleditor/query_tool/dow= nload/2133388?query=3DSELECT%20NULL%3A%3Atext%2C%201234%3A%3Aint%2C%20%27Fo= o%20bar%27%3A%3Atext%2C%20E%27Foo%5CnBar%27%3A%3Atext&filename=3Ddata-1= 545392848979.csv HTTP/1.1" 500 -
2018-12-21 11:47:29,003: ER= ROR werkzeug: Error on request:
Traceback (most recent call la= st):
=C2=A0 File "/Users/dpage/.virtualenvs/pgadmin4/lib/pyt= hon3.6/site-packages/werkzeug/serving.py", line 270, in run_wsgi
=
=C2=A0 =C2=A0 execute(self.server.app)
=C2=A0 File "/Us= ers/dpage/.virtualenvs/pgadmin4/lib/python3.6/site-packages/werkzeug/servin= g.py", line 260, in execute
=C2=A0 =C2=A0 for data in applic= ation_iter:
=C2=A0 File "/Users/dpage/.virtualenvs/pgadmin4/= lib/python3.6/site-packages/werkzeug/wsgi.py", line 870, in __next__
=C2=A0 =C2=A0 return self._next()
=C2=A0 File "/Use= rs/dpage/.virtualenvs/pgadmin4/lib/python3.6/site-packages/werkzeug/wrapper= s.py", line 82, in _iter_encoded
=C2=A0 =C2=A0 for item in i= terable:
=C2=A0 File "/Users/dpage/git/pgadmin4/web/pgadmin/= utils/driver/psycopg2/connection.py", line 848, in gen
=C2= =A0 =C2=A0 csv_writer.writerows(results)
=C2=A0 File "/Users= /dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line 761, in writerows<= /div>
=C2=A0 =C2=A0 return self.writer.writerows(map(self._dict_to_list= , rowdicts))
=C2=A0 File "/Users/dpage/git/pgadmin4/web/pgad= min/utils/csv.py", line 268, in writerows
=C2=A0 =C2=A0 self= .writerow(row)
=C2=A0 File "/Users/dpage/git/pgadmin4/web/pg= admin/utils/csv.py", line 261, in writerow
=C2=A0 =C2=A0 row= =3D [self.strategy.prepare(field, only=3Donly) for field in row]
=C2=A0 File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py"= , line 261, in <listcomp>
=C2=A0 =C2=A0 row =3D [self.strat= egy.prepare(field, only=3Donly) for field in row]
=C2=A0 File &qu= ot;/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line 142, in p= repare
=C2=A0 =C2=A0 raise Error('No escapechar is set')<= /div>
_csv.Error: No escapechar is set

On Thu, Dec 20, 2018 at 1:05 PM Akshay Joshi <= ;akshay.joshi@enterprisedb= .com> wrote:
Hi=C2=A0Dave

On Thu, Dec 20, 2018 at 5:12 PM Akshay Joshi &= lt;aksha= y.joshi@enterprisedb.com> wrote:


On Thu, Dec 20, 2018 at 4:48 PM Da= ve Page <dpage@pg= admin.org> wrote:
Hi

On Thu, Dec 20, 2018 at 10:09 AM Akshay Joshi <akshay.joshi@enterprised= b.com> wrote:
Hi Dave

On Thu, Dec 20, 2018 at 3:08 PM Dave Page <dpage@pgadmin.org>= ; wrote:
Hi

When testing with quoting = set to None, quote =3D " and delimiter =3D , I get the following excep= tion when I try to download:

2018-12-20 09:34= :02,547: SQL pgadmin: Execute (with server cursor) for server = #2 - CONN:354106 (Query-id: 4121147):
SELECT NULL::text, 1234::in= t, 'Foo bar'::text, E'Foo\nBar'::text
2018-12-20 = 09:34:02,570: INFO werkzeug: 127.0.0.1 - - [20/Dec/2018 09:34:= 02] "GET /sqleditor/query_tool/download/5610522?query=3DSELECT%20NULL%= 3A%3Atext%2C%201234%3A%3Aint%2C%20%27Foo%20bar%27%3A%3Atext%2C%20E%27Foo%5C= nBar%27%3A%3Atext&filename=3Ddata-1545298442530.csv HTTP/1.1" 500 = -
2018-12-20 09:34:02,572: ERROR werkzeug: Error on = request:
Traceback (most recent call last):
=C2=A0 File= "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.6/site-packages/werkz= eug/serving.py", line 270, in run_wsgi
=C2=A0 =C2=A0 execute= (self.server.app)
=C2=A0 File "/Users/dpage/.virtualenvs/pga= dmin4/lib/python3.6/site-packages/werkzeug/serving.py", line 260, in e= xecute
=C2=A0 =C2=A0 for data in application_iter:
=C2= =A0 File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.6/site-packag= es/werkzeug/wsgi.py", line 870, in __next__
=C2=A0 =C2=A0 re= turn self._next()
=C2=A0 File "/Users/dpage/.virtualenvs/pga= dmin4/lib/python3.6/site-packages/werkzeug/wrappers.py", line 82, in _= iter_encoded
=C2=A0 =C2=A0 for item in iterable:
=C2=A0= File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/con= nection.py", line 820, in gen
=C2=A0 =C2=A0 csv_writer.write= rows(results)
=C2=A0 File "/Users/dpage/git/pgadmin4/web/pga= dmin/utils/csv.py", line 748, in writerows
=C2=A0 =C2=A0 ret= urn self.writer.writerows(map(self._dict_to_list, rowdicts))
=C2= =A0 File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", li= ne 256, in writerows
=C2=A0 =C2=A0 self.writerow(row)
= =C2=A0 File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py",= line 249, in writerow
=C2=A0 =C2=A0 row =3D [self.strategy.prepa= re(field, only=3Donly) for field in row]
=C2=A0 File "/Users= /dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line 249, in <listco= mp>
=C2=A0 =C2=A0 row =3D [self.strategy.prepare(field, only= =3Donly) for field in row]
=C2=A0 File "/Users/dpage/git/pga= dmin4/web/pgadmin/utils/csv.py", line 136, in prepare
=C2=A0= =C2=A0 raise Error('No escapechar is set')
_csv.Error: N= o escapechar is set

=C2=A0 =C2=A0 Not able to reproduce the above issue. I have tested it with= the same setting as you mentioned. Please refer all the attached screensho= ts. Please specify the steps if they are different.

When I have quoting set to All, the first column is return= ed as ""

dpage@hal:~/Downloads$ more data-1= 545298598112.csv=C2=A0

"text",&q= uot;int4","text-2","text-3"

"","= 1234","Foo bar","Foo

Bar"


Isn't= the point for it to be NULL?

=C2=A0 =C2=A0 while quoting is set to ALL, all the data ty= pes has been quoted, so I thought null values should be replaced by "&= quot; instead of blank. But if you think null values shouldn't be quote= d even if user select quote ALL, I'll fix it and resend the patch.

So how would you distinguish = NULL from an empty string? Isn't that exactly what the bug is about?

I still think we need a "Replace NULLs with&quo= t; config option, and regardless of quoting settings we always replace NULL= values with whatever that is set to=C2=A0 - for which the user could then = choose options like:

NULL
"NULL&quo= t;
""
''
<empty string&g= t;

We would never quote the NULL replacement value= - if the user wanted it to be quoted, they would include the quotes in the= configured string.


<= /div>
=C2=A0 =C2=A0OK, Will work on it and send the modified patch agai= n.=C2=A0

=C2=A0 =C2=A0 = =C2=A0 Attached is the modified patch as per your suggestion.=C2=A0
=C2=A0

On Tue, Dec 18, 2018 at 11:13 AM Akshay Joshi <akshay.joshi@enterprised= b.com> wrote:
Hi=C2=A0Dave

Attached is the modif= ied patch to fix review comments.

On Tue, Dec 18, 2018 at 3:00 PM Akshay Joshi <akshay.joshi@en= terprisedb.com> wrote:


On Tue, Dec 18, 2018 at 2:49 PM Dave Page &l= t;dpage@pgadmin.org<= /a>> wrote:
<= div dir=3D"ltr">Hi

Hi Hackers,

Attached is the pa= tch to fix RM #3780=C2=A0pgAdmin4 lacks ability to specify NULL values in <= span id=3D"gmail-m_-2894014154183785825gmail-m_-455675873228126996gmail-m_-= 3511925126658346770gmail-m_361423959471310265gmail-m_-1300974772564471919gm= ail-m_5044164911799542113gmail-m_8445853776182989198gmail-m_-70630813957611= 07541gmail-m_-3640237263547570362:1ww.4">CSV export.

<= /div>
Please review it.

A few points;

- You've included code from b= ackports.csv, but per the licence you need to include a description of the = changes made.

=C2=A0 =C2= =A0 =C2=A0 Sure. In that case I'll copy the complete file and will do m= y changes which is of two lines only. With my patch I have remove all the u= nwanted code from backport.csv.=C2=A0 =C2=A0 =C2=A0=C2=A0

- Shouldn't backports.csv be removed from re= quirements.txt, or is it used elsewhere?

=C2=A0 =C2=A0 =C2=A0Yes. Will do that.=C2=A0

- If the previous point is true, then I'm= fairly sure there is code in one or more of the many package build scripts= that adds an __init__.py file to backports.csv in the venv that's crea= ted.

=C2=A0 =C2=A0 I'= ll remove that code as well.=C2=A0=C2=A0
=C2= =A0
--
Dave Page
Blo= g: http://pgsnake= .blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


--
Akshay Joshi
Sr. Software Architect
<= /font>

=

Phone: +91 20-3058= -9517
Mobile: +91 976-788-8246
=


--
Akshay Josh= i
Sr. Software Architect


= Phone: +91 20-3058-9517
Mobile: +91 976-78= 8-8246


--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


--
Akshay Joshi
<= b>Sr. Software Architect
<= /div>

=

Phone: +91 20-3058-9517Mobile: +91 976-788-8246
<= /div>


--
Dave Page
Blog: http://pgsnake.blogspot.com
Twi= tter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise Pos= tgreSQL Company


--
= Akshay Joshi
Sr. Software Architect

=

<= /font>
Phon= e: +91 20-3058-9517
Mobile: +91 976-788-8246


--
Akshay Joshi
Sr. Software Architect
<= div>

Phone: +91 20-3058-9517
Mobi= le: +91 976-788-8246


--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @p= gsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL = Company
--0000000000008b4632057d86f374--