Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lYPBR-0005FE-I6 for pgadmin-hackers@arkaria.postgresql.org; Mon, 19 Apr 2021 08:23:06 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1lYPBQ-0006nE-FJ for pgadmin-hackers@arkaria.postgresql.org; Mon, 19 Apr 2021 08:23:04 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lYPBP-0006md-LQ for pgadmin-hackers@lists.postgresql.org; Mon, 19 Apr 2021 08:23:04 +0000 Received: from mail-ed1-x533.google.com ([2a00:1450:4864:20::533]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1lYPBM-0002Sh-5w for pgadmin-hackers@postgresql.org; Mon, 19 Apr 2021 08:23:03 +0000 Received: by mail-ed1-x533.google.com with SMTP id g17so38881572edm.6 for ; Mon, 19 Apr 2021 01:22:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pgadmin.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=FeJHIwkhL96HlzWQKgJYAJFpcMAhYu/0aPPdN7WAH6Q=; b=BmuLL6rwC0KNXhorHGdh92KdbBmnsKcy4Bx5QLxADBxyz1dFlDvvy2tnFfCWv+c3uW o1LExkoDbMpB3ftnRfUTcanHQMiKnRSm//OaNcI1EI5vw2YHbgZwJrZwNhSHbTUvNvdA 5KBc0k5jp+uVBqaifHN/w0g1VeV6X89nZF1VQ+fz/zgZGXisjncPR80xd+G8Ktd/dt5Y MURqkIY+RERYMpNHyj7YMnOSHEM6WvyU2MLrb3grn116zYVqsGBNEfuJH2cFcHYklxHh RGYmYKIlPEKiWbXdZ5TvrvrxFYayralelIRm70nvouIVWgt7Q+jxRcgjxxQcc0l07zxH psCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=FeJHIwkhL96HlzWQKgJYAJFpcMAhYu/0aPPdN7WAH6Q=; b=HO9i2jiTZoNUp1dLrCMlGMYTXefkGeQV1FZa1pkxC7LMDJb2Kh4cs8JiHqVaDZc3zH lpTOvxG5NutmgB07F/GmeO3Zx+CFk29h615+DgppYl6KUOSqW5QzNF73XSUFyejVZV9f 11slqmDlP7TBtqndAiUwCfUeRh2i91Y3QnQbdxJ41Cr2mZKyktYforDITJ73lDRf42nC ZdwvNQfX9T6XDZLS30DKVVbQZboxmYElLVybNAG6paGeZlhLb4/FgWYN+SAkLY7G5MGZ /3xi2x4HM5YR8i/XLQPl1y5tSYEABP2CYBskpwJ0QWKQ/GAaeZsou7jf65JtwXMZButl RlKw== X-Gm-Message-State: AOAM532KmjSq+HPQ73Ql7RixVwC1RzzBYvzSTk9nP2t3LBEAEuE36NfG moxpmaJB+3qzpYaEFsMdxe9ZMe8y1wU+UL3jo/NiXw== X-Google-Smtp-Source: ABdhPJyh4nmaFwbDuNA98S4swEE5EBCy5KR99+r4no7gM6a92Kxikx5b7sA+YMKRFe/1FUyG2OH/c1N03mFFhk9eugY= X-Received: by 2002:a05:6402:382:: with SMTP id o2mr17797087edv.370.1618820578241; Mon, 19 Apr 2021 01:22:58 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Dave Page Date: Mon, 19 Apr 2021 09:22:44 +0100 Message-ID: Subject: Re: CLI for Schema Diff To: Steve Chavez Cc: pgadmin-hackers Content-Type: multipart/alternative; boundary="000000000000857f9005c04f0996" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --000000000000857f9005c04f0996 Content-Type: text/plain; charset="UTF-8" Hi On Mon, Apr 19, 2021 at 6:47 AM Steve Chavez wrote: > Hi Dave, > > Thanks for the thoughtful reply. I got sidetracked but now I'm able to > continue work on the CLI. > > > 1) Agree a standard for command line syntax. > > 3) Make the code modular and extensible; perhaps using a Python module > per CLI module. > > Cool, I agree with the standard. From my understanding, the "core" CLI > module would be in charge of registering the servers, and thus it'll be > needed by all the other CLI modules right? > 'needed' in the sense that the CLI would be quite incomplete without at least the ability to add/delete servers and users I think. I don't think it would be absolutely required from a technical perspective. BTW; I'm open to better names than 'core'. 'pgadmin' might be one option. > Regarding the grant wizard, I'm not familiar with the module so I can't be > of much help in implementing a CLI mode for it. > > However, I can definitely help with both core and schema diff. Would you > agree if I scope my contribution to only those 2 modules? > Oh, definitely. I certainly wasn't suggesting you write everything! Your help on the CLI framework, and core/schema diff modules would certainly be appreciated. > > > 2) Figure out how to avoid the hacks in your existing code to do things > like: > > Absolutely. I've been meaning to speed up the CLI start-up time, my > implementation currently loads many unneeded modules. I'll also avoid the > hack to hide stdout. > > > 4) Allow the use of saved connections as well as arbitrary connection > strings. > > Agree. I think saved connections will come with the "core" CLI module > implementation(the register-server command). > Right, that's my thinking. Thanks! > > On Thu, 1 Apr 2021 at 03:44, Dave Page wrote: > >> Hi >> >> On Thu, Apr 1, 2021 at 12:38 AM Steve Chavez wrote: >> >>> > but is this something you'd be interested in working on to become a >>> more fully featured and production quality CLI? >>> >>> Yes, absolutely! >>> >> >> Cool! >> >> >>> What would be the next step? I'm all for doing a CLI that covers pgAdmin >>> quality standards. >>> >> >> I think there are a few things we'd need to do: >> >> 1) Agree a standard for command line syntax. CLIs can easily become very >> messy - I've spent time designing such standards in the past >> (unfortunately, I can't just share that), but some thoughts might be: >> >> - have a consistent >> syntax. For example; >> >> pgacli --output-format=json schema-diff generate-diff >> --source=... --target=... >> pgacli core register-server --host=... --maintenance-db=... ... >> pgacli grant-wizard grant-acl --select --type=tables >> --objects=schema.tb_* >> >> (I've assumed a module name of 'core' for things directly related to >> the heart of pgAdmin) >> >> - Actions are in the form "verb-noun" >> - Every common option has both a short and a long form >> - Uncommon options have a long form only >> - Support various help options, showing appropriate info; >> >> pgacli -h >> pgacli --help >> pgacli core -h >> pgacli schema-diff --help >> >> 2) Figure out how to avoid the hacks in your existing code to do things >> like: >> >> - Hide initialisation output >> - Setup the dummy SQLite database >> >> 3) Make the code modular and extensible; perhaps using a Python module >> per CLI module. >> >> 4) Allow the use of saved connections as well as arbitrary connection >> strings. >> >> I think 2 is the hardest issue. It may be that we add a flag to the main >> application code that tells it it's running under the CLI, so that it can >> avoid doing things it doesn't need to in those cases. One example is the >> module loading; we might want to skip that entirely under the CLI, and have >> the CLI module that's being called load just the modules it needs. >> >> >>> >>> On Wed, 31 Mar 2021 at 03:08, Dave Page wrote: >>> >>>> Hi >>>> >>>> On Tue, Mar 30, 2021 at 3:36 PM Steve Chavez wrote: >>>> >>>>> Hey all, >>>>> >>>>> In case anyone is interested, I've managed to enable a CLI mode for >>>>> the Schema Diff on this repo: >>>>> >>>>> https://github.com/steve-chavez/pgadmin4/blob/cli/web/cli.py >>>>> >>>>> It basically works by using a Flask test client that interacts with >>>>> the Schema Diff endpoints. >>>>> It's a single isolated file, I haven't patched any of the existing >>>>> modules. >>>>> >>>>> For a quickstart, there's also a docker image that can be used like: >>>>> >>>>> docker run supabase/pgadmin-schema-diff \ >>>>> 'postgres://postgres@host/diff_source' \ >>>>> 'postgres://postgres@host/diff_target' \ >>>>> > diff.sql >>>>> ## the stderr output shows the same messages as the Schema Diff GUI: >>>>> >>>>> Starting schema diff... >>>>> Comparision started......0% >>>>> Comparing FTS Dictionaries ...35% >>>>> Comparing Functions ...50% >>>>> Comparing Trigger Functions ...60% >>>>> Comparing Sequences ...70% >>>>> Comparing Tables ...80% >>>>> Comparing Views ...90% >>>>> Done. >>>>> >>>>> >>>> That's an interesting approach! Obviously the code is just a proof of >>>> concept at the moment (redirecting stdout is masking errors for example), >>>> but is this something you'd be interested in working on to become a more >>>> fully featured and production quality CLI? >>>> >>>> -- >>>> Dave Page >>>> Blog: http://pgsnake.blogspot.com >>>> Twitter: @pgsnake >>>> >>>> EDB: http://www.enterprisedb.com >>>> >>>> >> >> -- >> Dave Page >> Blog: http://pgsnake.blogspot.com >> Twitter: @pgsnake >> >> EDB: http://www.enterprisedb.com >> >> -- Dave Page Blog: https://pgsnake.blogspot.com Twitter: @pgsnake EDB: https://www.enterprisedb.com --000000000000857f9005c04f0996 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi

On Mon, Apr 19, 2021 at 6:47 AM Steve C= havez <steve@supabase.io> wr= ote:
Hi Dave,

Thanks for = the thoughtful reply. I got sidetracked but now I'm able to continue wo= rk on the CLI.

> 1) Agree a standard for command line syntax.
= > 3) Make the code modular and extensible; perhaps using a Python module= per CLI module.

Cool, I agree with the standard. From my understand= ing, the "core" CLI module would be in charge of registering the = servers, and thus it'll be needed by all the other CLI modules right?

'needed' in the sense that= the CLI would be quite incomplete without at least the ability to add/dele= te servers and users I think. I don't think it would be absolutely requ= ired from a technical perspective.

BTW; I'm op= en to better names than 'core'. 'pgadmin' might be one opti= on.
=C2=A0
Regarding the g= rant wizard, I'm not familiar with the module so I can't be of much= help in implementing a CLI mode for it.

However, I can definitely h= elp with both core and schema diff. Would you agree if I scope my contribut= ion to only those 2 modules?

Oh, = definitely. I certainly wasn't suggesting you write everything! Your he= lp on the CLI framework, and core/schema diff modules would certainly be ap= preciated.
=C2=A0

>= 2) Figure out how to avoid the hacks in your existing code to do things li= ke:

Absolutely. I've been meaning to speed up the CLI start-up t= ime, my implementation currently loads many unneeded modules. I'll also= avoid the hack to hide stdout.

> 4) Allow the use of saved conne= ctions as well as arbitrary connection strings.

Agree. I think saved= connections will come with the "core" CLI module implementation(= the register-server command).

Rig= ht, that's my thinking.

Thanks!
=C2= =A0

On Thu, 1 Apr 2021 at 03:44, Dav= e Page <dpage@pga= dmin.org> wrote:
Hi<= /div>
O= n Thu, Apr 1, 2021 at 12:38 AM Steve Chavez <steve@supabase.io> wrote:
> but is this something you'd be intere= sted in working on to become a more fully featured and production quality C= LI?=C2=A0

Yes, absolutely!

Cool!
=C2=A0
What would be the next step? I'm all for doing a CLI that covers pgAdm= in quality standards.

I think t= here are a few things we'd need to do:

1) Agre= e a standard for command line syntax. CLIs can easily become very messy - I= 've spent time designing such standards in the past (unfortunately, I c= an't just share that), but some thoughts might be:

=
=C2=A0 - have a consistent <command> <command options> <= ;module> <action> <action options> syntax. For example;

=C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0pgacli --output-forma= t=3Djson schema-diff generate-diff --source=3D... --target=3D...
= =C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0pgacli=C2=A0core register-server --host=3D= ... --maintenance-db=3D... ...
=C2=A0 =C2=A0 =C2=A0 =C2=A0=C2= =A0pgacli=C2=A0grant-wizard grant-acl --select --type=3Dtables --objects=3D= schema.tb_*

=C2=A0 =C2=A0 (I've assumed a = module name of 'core' for things directly related to the heart of p= gAdmin)

=C2=A0 =C2=A0 - Actions are in the form &q= uot;verb-noun"
=C2=A0 =C2=A0 - Every common option has both = a short and a long form
=C2=A0 =C2=A0 - Uncommon options have a l= ong form only
=C2=A0 =C2=A0 - Support various help options, showi= ng appropriate info;

=C2=A0 =C2=A0 pgacli -h
=
=C2=A0 =C2=A0 pgacli --help
=C2=A0 =C2=A0 pgacli core -h
=C2=A0 =C2=A0 pgacli=C2=A0schema-diff --help

2) Figure out how to avoid the hacks in your existing code to do things l= ike:

=C2=A0 =C2=A0 - Hide initialisation output
=C2=A0 =C2=A0 - Setup the dummy SQLite database

3) Make the code modular and extensible; perhaps using a Python modul= e per CLI module.

4) Allow the use of saved connec= tions as well as arbitrary connection strings.

I t= hink 2 is the hardest issue. It may be that we add a flag to the main appli= cation code that tells it it's running under the CLI, so that it can av= oid doing things it doesn't need to in those cases. One example is the = module loading; we might want to skip that entirely under the CLI, and have= the CLI module that's being called load just the modules it needs.
=C2=A0

On Wed, 31 Mar 2021 at 03:08, Dave Page <<= a href=3D"mailto:dpage@pgadmin.org" target=3D"_blank">dpage@pgadmin.org= > wrote:
Hi

On Tue, Mar 3= 0, 2021 at 3:36 PM Steve Chavez <steve@supabase.io> wrote:
=
Hey all,

In case anyone is interested, I've ma= naged to enable a CLI mode for the Schema Diff on this repo:

=

It basically works by using a Flask= test client that interacts with the Schema Diff endpoints.
It= 9;s a single isolated file, I haven't patched any of the existing modul= es.

For a quickstart, there's also a docker im= age that can be used like:

docker run su=
pabase/pgadmin-schema-diff \
  'postgre=
s://postgres@host/diff_source' \
  '=
;postgres://postgres@host/diff_target' \
  > diff.sql

## the=
 stderr output shows the same messages as the Schema Diff GUI:
Sta=
rting schema diff...
Comparision started...=
...0%
Comparing FTS Dictionaries ...35%
Comparing Functions ...50%
Comparing Trigger Functions ...60%
Comparing Sequences ...70%
Comparing T=
ables ...80%
Comparing Views ...90%
Done.
That's an interesting approach! Obviously the code is just = a proof of concept at the moment (redirecting stdout is masking errors for = example), but is this something you'd be interested in working on to be= come a more fully featured and production quality CLI?=C2=A0

--


--


--
<= /div> --000000000000857f9005c04f0996--