Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1evaMw-0003wd-Hi for pgadmin-hackers@arkaria.postgresql.org; Tue, 13 Mar 2018 03:12:55 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1evaMt-0001hr-Jj for pgadmin-hackers@arkaria.postgresql.org; Tue, 13 Mar 2018 03:12:51 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1evaMs-0001hS-Om for pgadmin-hackers@lists.postgresql.org; Tue, 13 Mar 2018 03:12:51 +0000 Received: from mail-ot0-x230.google.com ([2607:f8b0:4003:c0f::230]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1evaMk-0004YD-WF for pgadmin-hackers@postgresql.org; Tue, 13 Mar 2018 03:12:49 +0000 Received: by mail-ot0-x230.google.com with SMTP id l12so17511521otj.7 for ; Mon, 12 Mar 2018 20:12:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=enterprisedb-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=t0r6Ugw/For6REnQJ7cw+Zdtqf6yIKWG/XUu2aWroTM=; b=V3O5mN7s5vu+oPg4y2itskqN32Z4RKLtkwFvSjC++CKRWO4TaFWV7BEBcqonWc5PsW 71nZv3A9JMkD55dV0OxIZnE6bbMBSsfpgFJLUC3TNn9Bv5FLATInv+DS/C9yQ7Rc2IUP mtO8BJgzSI9uXQrL86EOG2bPAqdSfL0826TvnOQ8myecCA2NaD1IrGzRJJ1fYuTZNQRN UENLG/NfnCFqGOWhV5T/TLbJ4bkzW3MV+FBaibgKGkPqMW/UEKTC4AobDb3nQ+0qys/v 0HdOGitRckqo9ze/5WleS7go+5SyVyVMmrEdpkK1fEr/faPYsGrdpkyht8qhHBz1RXNZ eQ2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=t0r6Ugw/For6REnQJ7cw+Zdtqf6yIKWG/XUu2aWroTM=; b=iqcxpyhyVhH2gTercdIApzlxIjG61nI8H89MAQqgtMkDjI4ltlPls8ldgx9LfxOnMV hoBhsUX9sNKwP/v5ZgsfFLKNml1MY0et2O10nG9e1SvzYRLLK/mF+HdrFnyr01Z+CD/L yD4VpK9lSX+yztnH048dd2WQcHfqMwQHvRgi6E7Dras6CGcUV/5OoyNhfQXqWbbRkTqT 4xCTbFQ/5/RoHyTcoVx97IVKis0g9k5mziCzbHarGqbCMtW7RnSgL0wCT6MCMVQVs0Sg 5Lr5yycr6r9BzMHYGODOWXPnc/Ncphc+TVj4hXjGCLGcnJdCHhg2RCri+C2I3BpD59WQ y4BA== X-Gm-Message-State: AElRT7GX8eaLalJvrp986naN2FEZLrH9yMN8cxStH5GXUYl6dbHg91SB tSPhR02r335mI42U+u4Mj13/Ipy2zZDy7jiYkZ3JBg== X-Google-Smtp-Source: AG47ELsR871BR+m4jhz9D7c/LsEoo5zhi1JKGDOJTnQiK08mGtKU5bX863GNcd5jmdZNhL7ugLHS9zWTpy89QOxjv7s= X-Received: by 10.157.64.189 with SMTP id n58mr6588041ote.215.1520910761449; Mon, 12 Mar 2018 20:12:41 -0700 (PDT) MIME-Version: 1.0 Received: by 10.74.203.154 with HTTP; Mon, 12 Mar 2018 20:12:40 -0700 (PDT) In-Reply-To: References: From: Khushboo Vashi Date: Tue, 13 Mar 2018 08:42:40 +0530 Message-ID: Subject: =?UTF-8?Q?Re=3A_=5BpgAdmin4=5D=5BPatch=5D=3A_RM_=232963_=2D_Backup_database=2C_R?= =?UTF-8?Q?estore_database_and_Maintenance_Database_failed_for_=C3=A9_objec?= =?UTF-8?Q?t=2E?= To: Dave Page Cc: pgadmin-hackers Content-Type: multipart/alternative; boundary="94eb2c1c0d90abf2ae056742a174" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --94eb2c1c0d90abf2ae056742a174 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Mar 13, 2018 at 2:29 AM, Dave Page wrote: > So I was trying to test this, and every time I try to run a backup, I'm > getting the following, with or without your patch: > > (sqlite3.ProgrammingError) You must not use 8-bit bytestrings unless you > use a text_factory that can interpret 8-bit bytestrings (like text_factor= y > =3D str). It is highly recommended that you instead just switch your > application to Unicode strings. [SQL: u'INSERT INTO process (pid, user_id= , > command, "desc", arguments, logdir, start_time, end_time, exit_code, > acknowledge) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)'] [parameters: > (180312205250107339, 1, u'/Library/PostgreSQL/9.4/bin/pg_dump', > 'ccopy_reg\n_reconstructor\np0\n(cpgadmin.tools.backup\ > nBackupMessage\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS\'cmd\'\n= p6\nV > --file "/Users/dpage/foo.dmp" --host "127.0.0.1" --port "5432" --username > "postgres" --no-password --verbose --format=3Dc --blobs > "\xe9"\np7\nsS\'backup_type\'\np8\nI3\nsS\'database\'\np9\ > nV\xe9\np10\nsS\'bfile\'\np11\nS\'foo.dmp\'\np12\nsS\'sid\'\np13\nI1\nsb.= ', > u'--file,/Users/dpage/foo.dmp,--host,127.0.0.1,--port,5432,- > -username,postgres,--no-password,--verbose,--format=3Dc,--blobs,\xe9', > '/var/lib/pgadmin/sessions/process_logs/180312205250107339', None, None, > None, None)] > > Any thoughts as to what's going on? I wasn't getting this on my other > laptop, and I can't think what else we would have changed to cause this. > > Deleting all the records from the process table from SQLITE will solve this problem. There were few issues related to encoding-decoding in my old patches, you may have applied those. > On Mon, Mar 12, 2018 at 2:00 AM, Khushboo Vashi < > khushboo.vashi@enterprisedb.com> wrote: > >> >> Hi Dave, >> >> On Fri, Mar 9, 2018 at 9:09 PM, Dave Page wrote: >> >>> Hi >>> >>> On Fri, Mar 9, 2018 at 3:32 PM, Dave Page wrote: >>> >>>> Hi >>>> >>>> On Fri, Mar 9, 2018 at 3:54 AM, Khushboo Vashi < >>>> khushboo.vashi@enterprisedb.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> Please find the attached patch to fix below issues: >>>>> >>>>> 1. #2963 - Backup database, Restore database and Maintenance Database >>>>> failed for =C3=A9 object >>>>> 2. #3157 - Process viewer doesn't show complete command executed. >>>>> >>>>> Test cases are not included for these fixes as we don't have test >>>>> cases for these modules (backup, restore, maintenance). >>>>> I will create one separate RM for the same which will cover this. >>>>> >>>> >>>> Interesting that you fix these together, as together they also exhibit >>>> another bug :-). Backing up the =C3=A9 database displays the following >>>> command: >>>> >>>> /usr/local/pgsql/bin/pg_dump --file "/Users/dpage/foo.bak" --host >>>> "localhost" --port "5432" --username "postgres" --no-password --verbos= e >>>> --format=3Dc --blobs "é" >>>> >>> >>> I can reproduce this issue only with notification dialogue (which I hav= e >> fixed in the attached patch) not with the details dialogue. Please refer >> the screenshots for the same. >> >>> Also, what tests can we add for backup/restore? We have nothing at all >>> at the moment, and it is pretty troublesome. I'd like to ensure that we= can >>> backup and restore a database correctly, and ensure that the displayed >>> commands are what we expect and that we get valid output from >>> pg_dump/pg_restore (though, it may change from PG version to PG version= , so >>> maybe we should just check for something small and generic). I guess th= is >>> might need some config parameters for the tests to specify the pg_* uti= lity >>> paths for each server. >>> >>> I'd suggest maybe having a feature test that opens the prefs, sets the >>> appropriate path, then runs a backup, waits for it to finish, checks th= e >>> process monitor output, then restores the same backup to a new database= , >>> checking the process monitor output again, and then checking that the >>> restored database contains at least one object from the original databa= se >>> (we don't need to check all of pg_dump/pg_restore, just that something >>> expected was restored). We should use a (partial) database name and bac= kup >>> filename from the advanced test config file, and I think both should >>> default to some interesting non-ASCII strings to ensure quoting works. >>> >> I was thinking of writing the unit test cases for the processes.py file >> as all the major functionalities for backup/restore/maintenance jobs don= e >> by this file, but by this we can not achieve the front-end string >> validation esp for non-ASCII strings. >> So, I am thinking of writing feature tests (as you have suggested) first >> and after that if needed I will write unit test cases. >> >>> >>> -- >>> Dave Page >>> Blog: http://pgsnake.blogspot.com >>> Twitter: @pgsnake >>> >>> EnterpriseDB UK: http://www.enterprisedb.com >>> The Enterprise PostgreSQL Company >>> >> >> > > > -- > Dave Page > Blog: http://pgsnake.blogspot.com > Twitter: @pgsnake > > EnterpriseDB UK: http://www.enterprisedb.com > The Enterprise PostgreSQL Company > --94eb2c1c0d90abf2ae056742a174 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Tue, Mar 13, 2018 at 2:29 AM, Dave Page <dpage@pgadmin.org><= /span> wrote:
So I was t= rying to test this, and every time I try to run a backup, I'm getting t= he following, with or without your patch:

(sqlite3.Progr= ammingError) You must not use 8-bit bytestrings unless you use a text_facto= ry that can interpret 8-bit bytestrings (like text_factory =3D str). It is = highly recommended that you instead just switch your application to Unicode= strings. [SQL: u'INSERT INTO process (pid, user_id, command, "des= c", arguments, logdir, start_time, end_time, exit_code, acknowledge) V= ALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)'] [parameters: (180312205250107339= , 1, u'/Library/PostgreSQL/9.4/bin/pg_dump', 'ccopy_reg\n_= reconstructor\np0\n(cpgadmin.tools.backup\nBackupMessage\np1\nc__= builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS\'cmd\'\np6\nV -= -file "/Users/dpage/foo.dmp" --host "127.0.0.1" --port = "5432" --username "postgres" --no-password --verbose --= format=3Dc --blobs "\xe9"\np7\nsS\'backup_type\'\np8= \nI3\nsS\'database\'\np9\nV\xe9\np10\nsS\'bfile\'\np11= \nS\'foo.dmp\'\np12\nsS\'sid\'\np13\nI1\nsb.'= , u'--file,/Users/dpage/foo.dmp,--host,127.0.0.1,--port,5432,--username,postgres,--no-password,--verbose,--format=3Dc,--blobs,= \xe9', '/var/lib/pgadmin/sessions/process_logs/1803122052= 50107339', None, None, None, None)]

Any th= oughts as to what's going on? I wasn't getting this on my other lap= top, and I can't think what else we would have changed to cause this.

Deleting all the records from the= process table from SQLITE will solve this problem.
There were fe= w issues related to encoding-decoding in my old patches, you may have appli= ed those.
On Mon, Ma= r 12, 2018 at 2:00 AM, Khushboo Vashi <khushboo.vashi@enterp= risedb.com> wrote:

Hi Dave,

On Fri, Mar 9, 2018 at 9:09= PM, Dave Page <dpage@pgadmin.org> wrote:
Hi

On Fri, Mar 9, 2018 at 3:32 PM, Dave Page <dpa= ge@pgadmin.org> wrote:
Hi

On Fri, Mar 9, 2018 at 3:54 AM, Khushboo Vashi <khushboo= .vashi@enterprisedb.com> wrote:
Hi,

Please find = the attached patch to fix below issues:

1. #2963 -= Backup database, Restore database and Maintenance Database failed for =C3= =A9 object
2. #3157 - Process viewer doesn't show complete co= mmand executed.

Test cases are not included for th= ese fixes as we don't have test cases for these modules (backup, restor= e, maintenance).
I will create one separate RM for the same which= will cover this.

Intere= sting that you fix these together, as together they also exhibit another bu= g :-). Backing up the =C3=A9=C2=A0database= displays the following command:

/usr/local/pgsql/bin/pg_dump --file "/Users/dpage/foo.ba= k" --host "localhost" --port "5432" --username &qu= ot;postgres" --no-password --verbose --format=3Dc --blobs "&#= 233;"=C2=A0

=
I can reproduce this issu= e only with notification dialogue (which I have fixed in the attached patch= ) not with the details dialogue. Please refer the screenshots for the same.=
Also, w= hat tests can we add for backup/restore? We have nothing at all at the mome= nt, and it is pretty troublesome. I'd like to ensure that we can backup= and restore a database correctly, and ensure that the displayed commands a= re what we expect and that we get valid output from pg_dump/pg_restore (tho= ugh, it may change from PG version to PG version, so maybe we should just c= heck for something small and generic). I guess this might need some config = parameters for the tests to specify the pg_* utility paths for each server.= =C2=A0

I'd suggest maybe having a feature test= that opens the prefs, sets the appropriate path, then runs a backup, waits= for it to finish, checks the process monitor output, then restores the sam= e backup to a new database, checking the process monitor output again, and = then checking that the restored database contains at least one object from = the original database (we don't need to check all of pg_dump/pg_restore= , just that something expected was restored). We should use a (partial) dat= abase name and backup filename from the advanced test config file, and I th= ink both should default to some interesting non-ASCII strings to ensure quo= ting works.
I was thinking = of writing the unit test cases for the processes.py file as all the major f= unctionalities for backup/restore/maintenance jobs done by this file, but b= y this we can not achieve the front-end string validation esp for non-ASCII= strings.
So, I am thinking of writing feature tests (as you have= suggested) first and after that if needed I will write unit test cases.

--
Dave Page
Blog: http://pgsnake.blogspot.comTwitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise= PostgreSQL Company




--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

Enterprise= DB UK: http://www= .enterprisedb.com
The Enterprise PostgreSQL Company

--94eb2c1c0d90abf2ae056742a174--