Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1evUXx-0001wC-OM for pgadmin-hackers@arkaria.postgresql.org; Mon, 12 Mar 2018 20:59:53 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1evUXv-0005Yb-OB for pgadmin-hackers@arkaria.postgresql.org; Mon, 12 Mar 2018 20:59:51 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1evUXv-0005YQ-Hs for pgadmin-hackers@lists.postgresql.org; Mon, 12 Mar 2018 20:59:51 +0000 Received: from mail-wm0-x243.google.com ([2a00:1450:400c:c09::243]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1evUXq-00005R-SK for pgadmin-hackers@postgresql.org; Mon, 12 Mar 2018 20:59:50 +0000 Received: by mail-wm0-x243.google.com with SMTP id s206so15205002wme.0 for ; Mon, 12 Mar 2018 13:59:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pgadmin-org.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=mO6dZHX42C33ojTnmPd//YQXWC0y1n/s2fekxaI4mM8=; b=SHjkzm9Z8P/dqBUTT0rfg5fV7kVlvzf8Z79/VJu8GPuImqr1jZVBLfAqa4pgPvUF1j NQdKz6Xj/Fo6Ac/Lob2lHTU3DkuCTYzGFG9j2svb60cYSqB6TLDV6ZRh1hbafXbV6Ar5 Yh5AvzxLcso1Cro6GrMDg5RRKk2GhR0RAbedW79BSgPhyWNj210nXUIKVQA0ndeSUn87 6ZgTU7qg7Xe7ua0DQgGhZIFwD0Fgtq8ILp4CWshRDcSaYzfP/Ac6yz+4rs4dL63RSx87 zuWGgCeyj+v+a9kkjLSJpfEl+etrwm/VkAzvLQ6kQGdBbcqGjzUsYYNUo2snYl1FnPvl SSVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=mO6dZHX42C33ojTnmPd//YQXWC0y1n/s2fekxaI4mM8=; b=blGF6BXkYWdr/UIrbn5gsgDacyUeSdE0+bp1iZ5lRBkJCARMUy3pddtpqsJbXVshZ5 PHpHdyO7j5S8vTtM2LtcdmrcuykknIkGUlR1drRfdWEDiceh4WxniIPjoz9MVFr9Ddsp FDRTSoWIldxfluOczFuFjj1m+OUCQvm/mow0bmeRQDhGjx48MTN/f8FVuRbBv5+YOdts 3V4ag3MRjFRImEhRxwevPQUgkDu3MJOfFhRuMbm3dAd5OFbUVxIWXc/CDGswuSTK/EFv Isfdn/W5mW0wmqJdcE25706qcCPhPL7iTtp4EqwHFenpFTQ2OtTp1BAX0byOuzlGyNdP LAIQ== X-Gm-Message-State: AElRT7GYM/VOwFPR7AStVTg2bdiu9L2nzHxdWLTrqhEcuzTd9TzLbuJa CI4ZnmstPWej1WDSxOMqMqL536Oe5G/PK6JLcDoZjQ== X-Google-Smtp-Source: AG47ELuRTBrxAaQkEZXlBqJDE9xipqTAaJ0XzOoGYEwBlDbqui7w3E3SX1G27UzBByQVppWrTR1omoBGcGu1He3AkmE= X-Received: by 10.28.190.18 with SMTP id o18mr7032248wmf.86.1520888385826; Mon, 12 Mar 2018 13:59:45 -0700 (PDT) MIME-Version: 1.0 Received: by 10.28.109.7 with HTTP; Mon, 12 Mar 2018 13:59:45 -0700 (PDT) In-Reply-To: References: From: Dave Page Date: Mon, 12 Mar 2018 16:59:45 -0400 Message-ID: Subject: =?UTF-8?Q?Re=3A_=5BpgAdmin4=5D=5BPatch=5D=3A_RM_=232963_=2D_Backup_database=2C_R?= =?UTF-8?Q?estore_database_and_Maintenance_Database_failed_for_=C3=A9_objec?= =?UTF-8?Q?t=2E?= To: Khushboo Vashi Cc: pgadmin-hackers Content-Type: multipart/alternative; boundary="f403043c3bf0fafd8405673d6bb3" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --f403043c3bf0fafd8405673d6bb3 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable So I was trying to test this, and every time I try to run a backup, I'm getting the following, with or without your patch: (sqlite3.ProgrammingError) You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory =3D str). It is highly recommended that you instead just switch your application to Unicode strings. [SQL: u'INSERT INTO process (pid, user_id, command, "desc", arguments, logdir, start_time, end_time, exit_code, acknowledge) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)'] [parameters: (180312205250107339, 1, u'/Library/PostgreSQL/9.4/bin/pg_dump', 'ccopy_reg\n_reconstructor\np0\n(cpgadmin.tools.backup\nBackupMessage\np1\n= c__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS\'cmd\'\np6\nV --file "/Users/dpage/foo.dmp" --host "127.0.0.1" --port "5432" --username "postgres" --no-password --verbose --format=3Dc --blobs "\xe9"\np7\nsS\'backup_type\'\np8\nI3\nsS\'database\'\np9\nV\xe9\np10\nsS\'= bfile\'\np11\nS\'foo.dmp\'\np12\nsS\'sid\'\np13\nI1\nsb.', u'--file,/Users/dpage/foo.dmp,--host,127.0.0.1,--port,5432,--username,postg= res,--no-password,--verbose,--format=3Dc,--blobs,\xe9', '/var/lib/pgadmin/sessions/process_logs/180312205250107339', None, None, None, None)] Any thoughts as to what's going on? I wasn't getting this on my other laptop, and I can't think what else we would have changed to cause this. On Mon, Mar 12, 2018 at 2:00 AM, Khushboo Vashi < khushboo.vashi@enterprisedb.com> wrote: > > Hi Dave, > > On Fri, Mar 9, 2018 at 9:09 PM, Dave Page wrote: > >> Hi >> >> On Fri, Mar 9, 2018 at 3:32 PM, Dave Page wrote: >> >>> Hi >>> >>> On Fri, Mar 9, 2018 at 3:54 AM, Khushboo Vashi < >>> khushboo.vashi@enterprisedb.com> wrote: >>> >>>> Hi, >>>> >>>> Please find the attached patch to fix below issues: >>>> >>>> 1. #2963 - Backup database, Restore database and Maintenance Database >>>> failed for =C3=A9 object >>>> 2. #3157 - Process viewer doesn't show complete command executed. >>>> >>>> Test cases are not included for these fixes as we don't have test case= s >>>> for these modules (backup, restore, maintenance). >>>> I will create one separate RM for the same which will cover this. >>>> >>> >>> Interesting that you fix these together, as together they also exhibit >>> another bug :-). Backing up the =C3=A9 database displays the following >>> command: >>> >>> /usr/local/pgsql/bin/pg_dump --file "/Users/dpage/foo.bak" --host >>> "localhost" --port "5432" --username "postgres" --no-password --verbose >>> --format=3Dc --blobs "é" >>> >> >> I can reproduce this issue only with notification dialogue (which I have > fixed in the attached patch) not with the details dialogue. Please refer > the screenshots for the same. > >> Also, what tests can we add for backup/restore? We have nothing at all a= t >> the moment, and it is pretty troublesome. I'd like to ensure that we can >> backup and restore a database correctly, and ensure that the displayed >> commands are what we expect and that we get valid output from >> pg_dump/pg_restore (though, it may change from PG version to PG version,= so >> maybe we should just check for something small and generic). I guess thi= s >> might need some config parameters for the tests to specify the pg_* util= ity >> paths for each server. >> >> I'd suggest maybe having a feature test that opens the prefs, sets the >> appropriate path, then runs a backup, waits for it to finish, checks the >> process monitor output, then restores the same backup to a new database, >> checking the process monitor output again, and then checking that the >> restored database contains at least one object from the original databas= e >> (we don't need to check all of pg_dump/pg_restore, just that something >> expected was restored). We should use a (partial) database name and back= up >> filename from the advanced test config file, and I think both should >> default to some interesting non-ASCII strings to ensure quoting works. >> > I was thinking of writing the unit test cases for the processes.py file a= s > all the major functionalities for backup/restore/maintenance jobs done by > this file, but by this we can not achieve the front-end string validation > esp for non-ASCII strings. > So, I am thinking of writing feature tests (as you have suggested) first > and after that if needed I will write unit test cases. > >> >> -- >> Dave Page >> Blog: http://pgsnake.blogspot.com >> Twitter: @pgsnake >> >> EnterpriseDB UK: http://www.enterprisedb.com >> The Enterprise PostgreSQL Company >> > > --=20 Dave Page Blog: http://pgsnake.blogspot.com Twitter: @pgsnake EnterpriseDB UK: http://www.enterprisedb.com The Enterprise PostgreSQL Company --f403043c3bf0fafd8405673d6bb3 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
So I was trying to test this, and every time I try to run = a backup, I'm getting the following, with or without your patch:
(sqlite3.ProgrammingError) You must not use 8-bit bytestrings = unless you use a text_factory that can interpret 8-bit bytestrings (like te= xt_factory =3D str). It is highly recommended that you instead just switch = your application to Unicode strings. [SQL: u'INSERT INTO process (pid, = user_id, command, "desc", arguments, logdir, start_time, end_time= , exit_code, acknowledge) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)'] [para= meters: (180312205250107339, 1, u'/Library/PostgreSQL/9.4/bin/pg_dump&#= 39;, 'ccopy_reg\n_reconstructor\np0\n(cpgadmin.tools.backup\nBackupMess= age\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS\'cmd\'\np6\nV= --file "/Users/dpage/foo.dmp" --host "127.0.0.1" --por= t "5432" --username "postgres" --no-password --verbose = --format=3Dc --blobs "\xe9"\np7\nsS\'backup_type\'\np8\nI= 3\nsS\'database\'\np9\nV\xe9\np10\nsS\'bfile\'\np11\nS\'= ;foo.dmp\'\np12\nsS\'sid\'\np13\nI1\nsb.', u'--file,/Us= ers/dpage/foo.dmp,--host,127.0.0.1,--port,5432,--username,postgres,--no-pas= sword,--verbose,--format=3Dc,--blobs,\xe9', '/var/lib/pgadmin/sessi= ons/process_logs/180312205250107339', None, None, None, None)]

Any thoughts as to what's going on? I wasn't g= etting this on my other laptop, and I can't think what else we would ha= ve changed to cause this.

On Mon, Mar 12, 2018 at 2:00 AM, Khushboo Vashi <khushboo.vashi@enterprisedb.com> wrote:

Hi Da= ve,

On Fri, Mar 9, 2018 at 9:09 PM, Dave Page <dpage@pgadmin.org&= gt; wrote:
Hi
=

On Fri, Mar= 9, 2018 at 3:32 PM, Dave Page <dpage@pgadmin.org> wrote:
Hi

On Fri, Mar 9, 2018 at 3:54 AM, Khu= shboo Vashi <khushboo.vashi@enterprisedb.com> wrote:
Hi,
Please find the attached patch to fix below issues:=

1. #2963 - Backup database, Restore database and = Maintenance Database failed for =C3=A9 object
2. #3157 - Process = viewer doesn't show complete command executed.

Test cases are not included for these fixes as we don't have test case= s for these modules (backup, restore, maintenance).
I will create= one separate RM for the same which will cover this.

Interesting that you fix these together, as to= gether they also exhibit another bug :-). Backing up the =C3=A9=C2=A0database displays the following command:

/usr/local/pgsql/bin/pg_du= mp --file "/Users/dpage/foo.bak" --host "localhost" --p= ort "5432" --username "postgres" --no-password --verbos= e --format=3Dc --blobs "&#233;"=C2=A0

I can reproduce this issue only with notification dialogue (which= I have fixed in the attached patch) not with the details dialogue. Please = refer the screenshots for the same.
Also, what tests can we add for back= up/restore? We have nothing at all at the moment, and it is pretty troubles= ome. I'd like to ensure that we can backup and restore a database corre= ctly, and ensure that the displayed commands are what we expect and that we= get valid output from pg_dump/pg_restore (though, it may change from PG ve= rsion to PG version, so maybe we should just check for something small and = generic). I guess this might need some config parameters for the tests to s= pecify the pg_* utility paths for each server.=C2=A0

I'd suggest maybe having a feature test that opens the prefs, sets t= he appropriate path, then runs a backup, waits for it to finish, checks the= process monitor output, then restores the same backup to a new database, c= hecking the process monitor output again, and then checking that the restor= ed database contains at least one object from the original database (we don= 't need to check all of pg_dump/pg_restore, just that something expecte= d was restored). We should use a (partial) database name and backup filenam= e from the advanced test config file, and I think both should default to so= me interesting non-ASCII strings to ensure quoting works.
=
I was thinking of writing the unit test case= s for the processes.py file as all the major functionalities for backup/res= tore/maintenance jobs done by this file, but by this we can not achieve the= front-end string validation esp for non-ASCII strings.
So, I am = thinking of writing feature tests (as you have suggested) first and after t= hat if needed I will write unit test cases.

--
D= ave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB U= K: http://www.ent= erprisedb.com
The Enterprise PostgreSQL Company




--
Dave Page
Blog: = http://pgsnake.bl= ogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com<= br>The Enterprise PostgreSQL Company
--f403043c3bf0fafd8405673d6bb3--