Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1gZun9-0004hH-US for pgadmin-hackers@arkaria.postgresql.org; Thu, 20 Dec 2018 09:38:56 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1gZun7-0004fP-M2 for pgadmin-hackers@arkaria.postgresql.org; Thu, 20 Dec 2018 09:38:53 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1gZun7-0004fI-7y for pgadmin-hackers@lists.postgresql.org; Thu, 20 Dec 2018 09:38:53 +0000 Received: from mail-wr1-x442.google.com ([2a00:1450:4864:20::442]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1gZun3-0006Dg-0O for pgadmin-hackers@postgresql.org; Thu, 20 Dec 2018 09:38:51 +0000 Received: by mail-wr1-x442.google.com with SMTP id z5so970634wrt.11 for ; Thu, 20 Dec 2018 01:38:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pgadmin-org.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=9XsfHXHaruJWHrgJj+Tr+uiIapifAB2wrYhFArhn4Vs=; b=k3tAxCfkHZoXVQA49YSqSYLL2o6IopRmx3Weda/D5EzB40GLIJNc+eQjO/wQC8QOoV lfVMs0SJULnEXG/wJl2j+A++YlHlRsNKbqgBaw0hHwNZkYy7ZFrWOLYJWXckhsKh3YJb sqL2s19KzYbpAC5SC49lcqb8giWaBHPFNwbEZxtIfbfTU8b6JfVKhn69RiJp5LyGBVHZ kezt5BSxhtpQ5dZavvwNWWiMqd6zcT8RJU8wVfoxJHjITDbrE58Sd1dENNdoiBrr8hfX tnUcSAaEf3Ptfw7XkU2Cf63XzbuBYVnvOy0l2Aln5JiI4RPRMaWkAnoznbHvwsBG1/MX ZzWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=9XsfHXHaruJWHrgJj+Tr+uiIapifAB2wrYhFArhn4Vs=; b=PCnYUD1Oy2hNK5RnL9WnuO5i0PTJa43G+x0g21TDRuaSQAlo1DS03e21u9RN/ms9GC Bt1IGNrW8pCnqWc9LT2TSHqAZfe2NiQGamsfTdF9HDyb6Rl7OI4zWPqMLakgpEacebRJ 3b+rqdw70cARbO0GcdFjJpnk60fxdYfypBNz7ybVTJ0iYCZdG63XjLiItHwXWjvy3vSD yuVEm0bZknMpPWW2tacsAUSJ+HBCNTon/3WnALh5axRx6gIboZ0rrxucmxB2Z6qJLIOE KbeysrIxXkhWW/dLHVvHfIuR6K0fMTvIcogfo+J4qu1/5912i8IB3FlWnPoCETVj5JxN k4SQ== X-Gm-Message-State: AA+aEWa+fJ1xmvdGxy3aEFtU6Btoqsh2VZx/kKwuQXIzFleZElJVzy8L 62kK0GETAk+E/HgOqjsRHoeBGk3OOsGb7plg27kF7A== X-Google-Smtp-Source: AFSGD/UfAuSjVRQSJSkEAGa+NDSrPARIE/YH5vFc8QeUXMhWBC2W82Eu68pH0T9aXxztTt/uSD5e1G7VfqmvTegW5CE= X-Received: by 2002:adf:8264:: with SMTP id 91mr21011709wrb.312.1545298727140; Thu, 20 Dec 2018 01:38:47 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Dave Page Date: Thu, 20 Dec 2018 09:38:35 +0000 Message-ID: Subject: Re: [pgAdmin4][Patch] - RM 3780 pgAdmin4 lacks ability to specify NULL values in CSV export To: Akshay Joshi Cc: pgadmin-hackers Content-Type: multipart/alternative; boundary="000000000000b41831057d70e599" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --000000000000b41831057d70e599 Content-Type: text/plain; charset="UTF-8" Hi When testing with quoting set to None, quote = " and delimiter = , I get the following exception when I try to download: 2018-12-20 09:34:02,547: SQL pgadmin: Execute (with server cursor) for server #2 - CONN:354106 (Query-id: 4121147): SELECT NULL::text, 1234::int, 'Foo bar'::text, E'Foo\nBar'::text 2018-12-20 09:34:02,570: INFO werkzeug: 127.0.0.1 - - [20/Dec/2018 09:34:02] "GET /sqleditor/query_tool/download/5610522?query=SELECT%20NULL%3A%3Atext%2C%201234%3A%3Aint%2C%20%27Foo%20bar%27%3A%3Atext%2C%20E%27Foo%5CnBar%27%3A%3Atext&filename=data-1545298442530.csv HTTP/1.1" 500 - 2018-12-20 09:34:02,572: ERROR werkzeug: Error on request: Traceback (most recent call last): File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.6/site-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.6/site-packages/werkzeug/serving.py", line 260, in execute for data in application_iter: File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.6/site-packages/werkzeug/wsgi.py", line 870, in __next__ return self._next() File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.6/site-packages/werkzeug/wrappers.py", line 82, in _iter_encoded for item in iterable: File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py", line 820, in gen csv_writer.writerows(results) File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line 748, in writerows return self.writer.writerows(map(self._dict_to_list, rowdicts)) File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line 256, in writerows self.writerow(row) File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line 249, in writerow row = [self.strategy.prepare(field, only=only) for field in row] File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line 249, in row = [self.strategy.prepare(field, only=only) for field in row] File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line 136, in prepare raise Error('No escapechar is set') _csv.Error: No escapechar is set When I have quoting set to All, the first column is returned as "" dpage@hal:*~/Downloads*$ more data-1545298598112.csv "text","int4","text-2","text-3" "","1234","Foo bar","Foo Bar" Isn't the point for it to be NULL? On Tue, Dec 18, 2018 at 11:13 AM Akshay Joshi wrote: > Hi Dave > > Attached is the modified patch to fix review comments. > > On Tue, Dec 18, 2018 at 3:00 PM Akshay Joshi < > akshay.joshi@enterprisedb.com> wrote: > >> >> >> On Tue, Dec 18, 2018 at 2:49 PM Dave Page wrote: >> >>> Hi >>> >>> On Tue, Dec 18, 2018 at 3:45 AM Akshay Joshi < >>> akshay.joshi@enterprisedb.com> wrote: >>> >>>> Hi Hackers, >>>> >>>> Attached is the patch to fix RM #3780 pgAdmin4 lacks ability to specify >>>> NULL values in CSV export. >>>> >>>> Please review it. >>>> >>> >>> A few points; >>> >>> - You've included code from backports.csv, but per the licence you need >>> to include a description of the changes made. >>> >> >> Sure. In that case I'll copy the complete file and will do my >> changes which is of two lines only. With my patch I have remove all the >> unwanted code from backport.csv. >> >>> >>> - Shouldn't backports.csv be removed from requirements.txt, or is it >>> used elsewhere? >>> >> >> Yes. Will do that. >> >>> >>> - If the previous point is true, then I'm fairly sure there is code in >>> one or more of the many package build scripts that adds an __init__.py file >>> to backports.csv in the venv that's created. >>> >> >> I'll remove that code as well. >> >>> >>> -- >>> Dave Page >>> Blog: http://pgsnake.blogspot.com >>> Twitter: @pgsnake >>> >>> EnterpriseDB UK: http://www.enterprisedb.com >>> The Enterprise PostgreSQL Company >>> >> >> >> -- >> *Akshay Joshi* >> >> *Sr. Software Architect * >> >> >> >> *Phone: +91 20-3058-9517Mobile: +91 976-788-8246* >> > > > -- > *Akshay Joshi* > > *Sr. Software Architect * > > > > *Phone: +91 20-3058-9517Mobile: +91 976-788-8246* > -- Dave Page Blog: http://pgsnake.blogspot.com Twitter: @pgsnake EnterpriseDB UK: http://www.enterprisedb.com The Enterprise PostgreSQL Company --000000000000b41831057d70e599 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi

When testing with q= uoting set to None, quote =3D " and delimiter =3D , I get the followin= g exception when I try to download:

2018-12-2= 0 09:34:02,547: SQL pgadmin: Execute (with server cursor) for server #2 = - CONN:354106 (Query-id: 4121147):
SELECT NULL::text, 1234::int, = 'Foo bar'::text, E'Foo\nBar'::text
2018-12-20 09:= 34:02,570: INFO werkzeug: 127.0.0.1 - - [20/Dec/2018 09:34:02] "GET= /sqleditor/query_tool/download/5610522?query=3DSELECT%20NULL%3A%3Atext%2C%= 201234%3A%3Aint%2C%20%27Foo%20bar%27%3A%3Atext%2C%20E%27Foo%5CnBar%27%3A%3A= text&filename=3Ddata-1545298442530.csv HTTP/1.1" 500 -
2= 018-12-20 09:34:02,572: ERROR werkze= ug: Error on request:
Trac= eback (most recent call last):
=C2=A0 File "/Users/dpage/.vi= rtualenvs/pgadmin4/lib/python3.6/site-packages/werkzeug/serving.py", l= ine 270, in run_wsgi
=C2=A0 =C2=A0 execute(self.server.app)
=
=C2=A0 File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.6/sit= e-packages/werkzeug/serving.py", line 260, in execute
=C2=A0= =C2=A0 for data in application_iter:
=C2=A0 File "/Users/dp= age/.virtualenvs/pgadmin4/lib/python3.6/site-packages/werkzeug/wsgi.py"= ;, line 870, in __next__
=C2=A0 =C2=A0 return self._next()
<= div>=C2=A0 File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.6/site= -packages/werkzeug/wrappers.py", line 82, in _iter_encoded
= =C2=A0 =C2=A0 for item in iterable:
=C2=A0 File "/Users/dpag= e/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py", line = 820, in gen
=C2=A0 =C2=A0 csv_writer.writerows(results)
=C2=A0 File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/csv.py"= , line 748, in writerows
=C2=A0 =C2=A0 return self.writer.writero= ws(map(self._dict_to_list, rowdicts))
=C2=A0 File "/Users/dp= age/git/pgadmin4/web/pgadmin/utils/csv.py", line 256, in writerows
=C2=A0 =C2=A0 self.writerow(row)
=C2=A0 File "/Users/= dpage/git/pgadmin4/web/pgadmin/utils/csv.py", line 249, in writerow
=C2=A0 =C2=A0 row =3D [self.strategy.prepare(field, only=3Donly) fo= r field in row]
=C2=A0 File "/Users/dpage/git/pgadmin4/web/p= gadmin/utils/csv.py", line 249, in <listcomp>
=C2=A0 = =C2=A0 row =3D [self.strategy.prepare(field, only=3Donly) for field in row]=
=C2=A0 File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/cs= v.py", line 136, in prepare
=C2=A0 =C2=A0 raise Error('N= o escapechar is set')
_csv.Error: No escapechar is set
<= div>
When I have quoting set to All, the first column is retu= rned as ""

dpage@hal:~/Downloads$ more d= ata-1545298598112.csv=C2=A0

"text","int4","text-2","text-3&qu= ot;

"","1234","Foo bar","Foo

Bar"


Isn't the point for it to be NULL?

On Tue, Dec 18, 2018 at 11:13 AM Akshay Joshi= <akshay.joshi@enterpri= sedb.com> wrote:
Hi=C2=A0Dave

Attached is the mo= dified patch to fix review comments.

On Tue, Dec 18, 2018 at 3:00 PM Akshay Joshi <akshay.joshi@= enterprisedb.com> wrote:


On Tue, Dec 18, 2018 at 2:49 PM Dave Page= <dpage@pgadmin.o= rg> wrote:
Hi

On= Tue, Dec 18, 2018 at 3:45 AM Akshay Joshi <akshay.joshi@enterprisedb.com>= ; wrote:
Hi Hackers,

Attached is the p= atch to fix RM #3780=C2=A0pgAdmin4 lacks ability to specify NULL values in = CSV export.

Please review it.

A few points;

- You've includ= ed code from backports.csv, but per the licence you need to include a descr= iption of the changes made.

=C2=A0 =C2=A0 =C2=A0 Sure. In that case I'll copy the complete file = and will do my changes which is of two lines only. With my patch I have rem= ove all the unwanted code from backport.csv.=C2=A0 =C2=A0 =C2=A0=C2=A0

- Shouldn't backports.csv be re= moved from requirements.txt, or is it used elsewhere?

=C2=A0 =C2=A0 =C2=A0Yes. Will do that.=C2=A0

- If the previous point is true,= then I'm fairly sure there is code in one or more of the many package = build scripts that adds an __init__.py file to backports.csv in the venv th= at's created.

=C2=A0 = =C2=A0 I'll remove that code as well.=C2=A0=C2=A0
=C2=A0
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

Ent= erpriseDB UK: htt= p://www.enterprisedb.com
The Enterprise PostgreSQL Company
=


--
<= b>Akshay Joshi
Sr. Software Architect

<= div>
Phone= : +91 20-3058-9517
Mobile: +91 976-788-8246


--
Sr. Software Architect


Phone: +91 20-3058-9517
Mobil= e: +91 976-788-8246


--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @p= gsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL = Company
--000000000000b41831057d70e599--