public inbox for [email protected]help / color / mirror / Atom feed
CLI for Schema Diff 7+ messages / 2 participants [nested] [flat]
* CLI for Schema Diff @ 2021-03-10 18:42 Steve Chavez <[email protected]> 2021-03-29 23:41 ` Re: CLI for Schema Diff Steve Chavez <[email protected]> 0 siblings, 1 reply; 7+ messages in thread From: Steve Chavez @ 2021-03-10 18:42 UTC (permalink / raw) To: pgadmin-hackers Hey all, The Schema Diff tool is amazing, works much better than all the current OSS alternatives. However, it's only possible to use it from the GUI and thus it cannot be used programmatically, which would be really useful for CI/CD environments(deployment automation). Would you guys be open to make the Schema Diff work through a CLI? Best regards, Steve ^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: CLI for Schema Diff 2021-03-10 18:42 CLI for Schema Diff Steve Chavez <[email protected]> @ 2021-03-29 23:41 ` Steve Chavez <[email protected]> 2021-03-31 08:08 ` Re: CLI for Schema Diff Dave Page <[email protected]> 0 siblings, 1 reply; 7+ messages in thread From: Steve Chavez @ 2021-03-29 23:41 UTC (permalink / raw) To: pgadmin-hackers Hey all, In case anyone is interested, I've managed to enable a CLI mode for the Schema Diff on this repo: https://github.com/steve-chavez/pgadmin4/blob/cli/web/cli.py It basically works by using a Flask test client that interacts with the Schema Diff endpoints. It's a single isolated file, I haven't patched any of the existing modules. For a quickstart, there's also a docker image that can be used like: docker run supabase/pgadmin-schema-diff \ 'postgres://postgres@host/diff_source' \ 'postgres://postgres@host/diff_target' \ > diff.sql ## the stderr output shows the same messages as the Schema Diff GUI: Starting schema diff... Comparision started......0% Comparing FTS Dictionaries ...35% Comparing Functions ...50% Comparing Trigger Functions ...60% Comparing Sequences ...70% Comparing Tables ...80% Comparing Views ...90% Done. On Wed, 10 Mar 2021 at 13:42, Steve Chavez <[email protected]> wrote: > Hey all, > > The Schema Diff tool is amazing, works much better than all the current > OSS alternatives. > > However, it's only possible to use it from the GUI and thus it cannot be > used programmatically, which would be really useful for CI/CD > environments(deployment automation). > > Would you guys be open to make the Schema Diff work through a CLI? > > Best regards, > Steve > ^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: CLI for Schema Diff 2021-03-10 18:42 CLI for Schema Diff Steve Chavez <[email protected]> 2021-03-29 23:41 ` Re: CLI for Schema Diff Steve Chavez <[email protected]> @ 2021-03-31 08:08 ` Dave Page <[email protected]> 2021-03-31 23:37 ` Re: CLI for Schema Diff Steve Chavez <[email protected]> 0 siblings, 1 reply; 7+ messages in thread From: Dave Page @ 2021-03-31 08:08 UTC (permalink / raw) To: Steve Chavez <[email protected]>; +Cc: pgadmin-hackers Hi On Tue, Mar 30, 2021 at 3:36 PM Steve Chavez <[email protected]> wrote: > Hey all, > > In case anyone is interested, I've managed to enable a CLI mode for the > Schema Diff on this repo: > > https://github.com/steve-chavez/pgadmin4/blob/cli/web/cli.py > > It basically works by using a Flask test client that interacts with the > Schema Diff endpoints. > It's a single isolated file, I haven't patched any of the existing modules. > > For a quickstart, there's also a docker image that can be used like: > > docker run supabase/pgadmin-schema-diff \ > 'postgres://postgres@host/diff_source' \ > 'postgres://postgres@host/diff_target' \ > > diff.sql > ## the stderr output shows the same messages as the Schema Diff GUI: > > Starting schema diff... > Comparision started......0% > Comparing FTS Dictionaries ...35% > Comparing Functions ...50% > Comparing Trigger Functions ...60% > Comparing Sequences ...70% > Comparing Tables ...80% > Comparing Views ...90% > Done. > > That's an interesting approach! Obviously the code is just a proof of concept at the moment (redirecting stdout is masking errors for example), but is this something you'd be interested in working on to become a more fully featured and production quality CLI? -- Dave Page Blog: http://pgsnake.blogspot.com Twitter: @pgsnake EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: CLI for Schema Diff 2021-03-10 18:42 CLI for Schema Diff Steve Chavez <[email protected]> 2021-03-29 23:41 ` Re: CLI for Schema Diff Steve Chavez <[email protected]> 2021-03-31 08:08 ` Re: CLI for Schema Diff Dave Page <[email protected]> @ 2021-03-31 23:37 ` Steve Chavez <[email protected]> 2021-04-01 08:44 ` Re: CLI for Schema Diff Dave Page <[email protected]> 0 siblings, 1 reply; 7+ messages in thread From: Steve Chavez @ 2021-03-31 23:37 UTC (permalink / raw) To: Dave Page <[email protected]>; pgadmin-hackers > but is this something you'd be interested in working on to become a more fully featured and production quality CLI? Yes, absolutely! What would be the next step? I'm all for doing a CLI that covers pgAdmin quality standards. On Wed, 31 Mar 2021 at 03:08, Dave Page <[email protected]> wrote: > Hi > > On Tue, Mar 30, 2021 at 3:36 PM Steve Chavez <[email protected]> wrote: > >> Hey all, >> >> In case anyone is interested, I've managed to enable a CLI mode for the >> Schema Diff on this repo: >> >> https://github.com/steve-chavez/pgadmin4/blob/cli/web/cli.py >> >> It basically works by using a Flask test client that interacts with the >> Schema Diff endpoints. >> It's a single isolated file, I haven't patched any of the existing >> modules. >> >> For a quickstart, there's also a docker image that can be used like: >> >> docker run supabase/pgadmin-schema-diff \ >> 'postgres://postgres@host/diff_source' \ >> 'postgres://postgres@host/diff_target' \ >> > diff.sql >> ## the stderr output shows the same messages as the Schema Diff GUI: >> >> Starting schema diff... >> Comparision started......0% >> Comparing FTS Dictionaries ...35% >> Comparing Functions ...50% >> Comparing Trigger Functions ...60% >> Comparing Sequences ...70% >> Comparing Tables ...80% >> Comparing Views ...90% >> Done. >> >> > That's an interesting approach! Obviously the code is just a proof of > concept at the moment (redirecting stdout is masking errors for example), > but is this something you'd be interested in working on to become a more > fully featured and production quality CLI? > > -- > Dave Page > Blog: http://pgsnake.blogspot.com > Twitter: @pgsnake > > EDB: http://www.enterprisedb.com > > ^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: CLI for Schema Diff 2021-03-10 18:42 CLI for Schema Diff Steve Chavez <[email protected]> 2021-03-29 23:41 ` Re: CLI for Schema Diff Steve Chavez <[email protected]> 2021-03-31 08:08 ` Re: CLI for Schema Diff Dave Page <[email protected]> 2021-03-31 23:37 ` Re: CLI for Schema Diff Steve Chavez <[email protected]> @ 2021-04-01 08:44 ` Dave Page <[email protected]> 2021-04-19 05:47 ` Re: CLI for Schema Diff Steve Chavez <[email protected]> 0 siblings, 1 reply; 7+ messages in thread From: Dave Page @ 2021-04-01 08:44 UTC (permalink / raw) To: Steve Chavez <[email protected]>; +Cc: pgadmin-hackers Hi On Thu, Apr 1, 2021 at 12:38 AM Steve Chavez <[email protected]> wrote: > > but is this something you'd be interested in working on to become a more > fully featured and production quality CLI? > > Yes, absolutely! > Cool! > What would be the next step? I'm all for doing a CLI that covers pgAdmin > quality standards. > I think there are a few things we'd need to do: 1) Agree a standard for command line syntax. CLIs can easily become very messy - I've spent time designing such standards in the past (unfortunately, I can't just share that), but some thoughts might be: - have a consistent <command> <command options> <module> <action> <action options> syntax. For example; pgacli --output-format=json schema-diff generate-diff --source=... --target=... pgacli core register-server --host=... --maintenance-db=... ... pgacli grant-wizard grant-acl --select --type=tables --objects=schema.tb_* (I've assumed a module name of 'core' for things directly related to the heart of pgAdmin) - Actions are in the form "verb-noun" - Every common option has both a short and a long form - Uncommon options have a long form only - Support various help options, showing appropriate info; pgacli -h pgacli --help pgacli core -h pgacli schema-diff --help 2) Figure out how to avoid the hacks in your existing code to do things like: - Hide initialisation output - Setup the dummy SQLite database 3) Make the code modular and extensible; perhaps using a Python module per CLI module. 4) Allow the use of saved connections as well as arbitrary connection strings. I think 2 is the hardest issue. It may be that we add a flag to the main application code that tells it it's running under the CLI, so that it can avoid doing things it doesn't need to in those cases. One example is the module loading; we might want to skip that entirely under the CLI, and have the CLI module that's being called load just the modules it needs. > > On Wed, 31 Mar 2021 at 03:08, Dave Page <[email protected]> wrote: > >> Hi >> >> On Tue, Mar 30, 2021 at 3:36 PM Steve Chavez <[email protected]> wrote: >> >>> Hey all, >>> >>> In case anyone is interested, I've managed to enable a CLI mode for the >>> Schema Diff on this repo: >>> >>> https://github.com/steve-chavez/pgadmin4/blob/cli/web/cli.py >>> >>> It basically works by using a Flask test client that interacts with the >>> Schema Diff endpoints. >>> It's a single isolated file, I haven't patched any of the existing >>> modules. >>> >>> For a quickstart, there's also a docker image that can be used like: >>> >>> docker run supabase/pgadmin-schema-diff \ >>> 'postgres://postgres@host/diff_source' \ >>> 'postgres://postgres@host/diff_target' \ >>> > diff.sql >>> ## the stderr output shows the same messages as the Schema Diff GUI: >>> >>> Starting schema diff... >>> Comparision started......0% >>> Comparing FTS Dictionaries ...35% >>> Comparing Functions ...50% >>> Comparing Trigger Functions ...60% >>> Comparing Sequences ...70% >>> Comparing Tables ...80% >>> Comparing Views ...90% >>> Done. >>> >>> >> That's an interesting approach! Obviously the code is just a proof of >> concept at the moment (redirecting stdout is masking errors for example), >> but is this something you'd be interested in working on to become a more >> fully featured and production quality CLI? >> >> -- >> Dave Page >> Blog: http://pgsnake.blogspot.com >> Twitter: @pgsnake >> >> EDB: http://www.enterprisedb.com >> >> -- Dave Page Blog: http://pgsnake.blogspot.com Twitter: @pgsnake EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: CLI for Schema Diff 2021-03-10 18:42 CLI for Schema Diff Steve Chavez <[email protected]> 2021-03-29 23:41 ` Re: CLI for Schema Diff Steve Chavez <[email protected]> 2021-03-31 08:08 ` Re: CLI for Schema Diff Dave Page <[email protected]> 2021-03-31 23:37 ` Re: CLI for Schema Diff Steve Chavez <[email protected]> 2021-04-01 08:44 ` Re: CLI for Schema Diff Dave Page <[email protected]> @ 2021-04-19 05:47 ` Steve Chavez <[email protected]> 2021-04-19 08:22 ` Re: CLI for Schema Diff Dave Page <[email protected]> 0 siblings, 1 reply; 7+ messages in thread From: Steve Chavez @ 2021-04-19 05:47 UTC (permalink / raw) To: Dave Page <[email protected]>; +Cc: pgadmin-hackers Hi Dave, Thanks for the thoughtful reply. I got sidetracked but now I'm able to continue work on the CLI. > 1) Agree a standard for command line syntax. > 3) Make the code modular and extensible; perhaps using a Python module per CLI module. Cool, I agree with the standard. From my understanding, the "core" CLI module would be in charge of registering the servers, and thus it'll be needed by all the other CLI modules right? Regarding the grant wizard, I'm not familiar with the module so I can't be of much help in implementing a CLI mode for it. However, I can definitely help with both core and schema diff. Would you agree if I scope my contribution to only those 2 modules? > 2) Figure out how to avoid the hacks in your existing code to do things like: Absolutely. I've been meaning to speed up the CLI start-up time, my implementation currently loads many unneeded modules. I'll also avoid the hack to hide stdout. > 4) Allow the use of saved connections as well as arbitrary connection strings. Agree. I think saved connections will come with the "core" CLI module implementation(the register-server command). On Thu, 1 Apr 2021 at 03:44, Dave Page <[email protected]> wrote: > Hi > > On Thu, Apr 1, 2021 at 12:38 AM Steve Chavez <[email protected]> wrote: > >> > but is this something you'd be interested in working on to become a >> more fully featured and production quality CLI? >> >> Yes, absolutely! >> > > Cool! > > >> What would be the next step? I'm all for doing a CLI that covers pgAdmin >> quality standards. >> > > I think there are a few things we'd need to do: > > 1) Agree a standard for command line syntax. CLIs can easily become very > messy - I've spent time designing such standards in the past > (unfortunately, I can't just share that), but some thoughts might be: > > - have a consistent <command> <command options> <module> <action> > <action options> syntax. For example; > > pgacli --output-format=json schema-diff generate-diff --source=... > --target=... > pgacli core register-server --host=... --maintenance-db=... ... > pgacli grant-wizard grant-acl --select --type=tables > --objects=schema.tb_* > > (I've assumed a module name of 'core' for things directly related to > the heart of pgAdmin) > > - Actions are in the form "verb-noun" > - Every common option has both a short and a long form > - Uncommon options have a long form only > - Support various help options, showing appropriate info; > > pgacli -h > pgacli --help > pgacli core -h > pgacli schema-diff --help > > 2) Figure out how to avoid the hacks in your existing code to do things > like: > > - Hide initialisation output > - Setup the dummy SQLite database > > 3) Make the code modular and extensible; perhaps using a Python module per > CLI module. > > 4) Allow the use of saved connections as well as arbitrary connection > strings. > > I think 2 is the hardest issue. It may be that we add a flag to the main > application code that tells it it's running under the CLI, so that it can > avoid doing things it doesn't need to in those cases. One example is the > module loading; we might want to skip that entirely under the CLI, and have > the CLI module that's being called load just the modules it needs. > > >> >> On Wed, 31 Mar 2021 at 03:08, Dave Page <[email protected]> wrote: >> >>> Hi >>> >>> On Tue, Mar 30, 2021 at 3:36 PM Steve Chavez <[email protected]> wrote: >>> >>>> Hey all, >>>> >>>> In case anyone is interested, I've managed to enable a CLI mode for the >>>> Schema Diff on this repo: >>>> >>>> https://github.com/steve-chavez/pgadmin4/blob/cli/web/cli.py >>>> >>>> It basically works by using a Flask test client that interacts with the >>>> Schema Diff endpoints. >>>> It's a single isolated file, I haven't patched any of the existing >>>> modules. >>>> >>>> For a quickstart, there's also a docker image that can be used like: >>>> >>>> docker run supabase/pgadmin-schema-diff \ >>>> 'postgres://postgres@host/diff_source' \ >>>> 'postgres://postgres@host/diff_target' \ >>>> > diff.sql >>>> ## the stderr output shows the same messages as the Schema Diff GUI: >>>> >>>> Starting schema diff... >>>> Comparision started......0% >>>> Comparing FTS Dictionaries ...35% >>>> Comparing Functions ...50% >>>> Comparing Trigger Functions ...60% >>>> Comparing Sequences ...70% >>>> Comparing Tables ...80% >>>> Comparing Views ...90% >>>> Done. >>>> >>>> >>> That's an interesting approach! Obviously the code is just a proof of >>> concept at the moment (redirecting stdout is masking errors for example), >>> but is this something you'd be interested in working on to become a more >>> fully featured and production quality CLI? >>> >>> -- >>> Dave Page >>> Blog: http://pgsnake.blogspot.com >>> Twitter: @pgsnake >>> >>> EDB: http://www.enterprisedb.com >>> >>> > > -- > Dave Page > Blog: http://pgsnake.blogspot.com > Twitter: @pgsnake > > EDB: http://www.enterprisedb.com > > ^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: CLI for Schema Diff 2021-03-10 18:42 CLI for Schema Diff Steve Chavez <[email protected]> 2021-03-29 23:41 ` Re: CLI for Schema Diff Steve Chavez <[email protected]> 2021-03-31 08:08 ` Re: CLI for Schema Diff Dave Page <[email protected]> 2021-03-31 23:37 ` Re: CLI for Schema Diff Steve Chavez <[email protected]> 2021-04-01 08:44 ` Re: CLI for Schema Diff Dave Page <[email protected]> 2021-04-19 05:47 ` Re: CLI for Schema Diff Steve Chavez <[email protected]> @ 2021-04-19 08:22 ` Dave Page <[email protected]> 0 siblings, 0 replies; 7+ messages in thread From: Dave Page @ 2021-04-19 08:22 UTC (permalink / raw) To: Steve Chavez <[email protected]>; +Cc: pgadmin-hackers Hi On Mon, Apr 19, 2021 at 6:47 AM Steve Chavez <[email protected]> wrote: > Hi Dave, > > Thanks for the thoughtful reply. I got sidetracked but now I'm able to > continue work on the CLI. > > > 1) Agree a standard for command line syntax. > > 3) Make the code modular and extensible; perhaps using a Python module > per CLI module. > > Cool, I agree with the standard. From my understanding, the "core" CLI > module would be in charge of registering the servers, and thus it'll be > needed by all the other CLI modules right? > 'needed' in the sense that the CLI would be quite incomplete without at least the ability to add/delete servers and users I think. I don't think it would be absolutely required from a technical perspective. BTW; I'm open to better names than 'core'. 'pgadmin' might be one option. > Regarding the grant wizard, I'm not familiar with the module so I can't be > of much help in implementing a CLI mode for it. > > However, I can definitely help with both core and schema diff. Would you > agree if I scope my contribution to only those 2 modules? > Oh, definitely. I certainly wasn't suggesting you write everything! Your help on the CLI framework, and core/schema diff modules would certainly be appreciated. > > > 2) Figure out how to avoid the hacks in your existing code to do things > like: > > Absolutely. I've been meaning to speed up the CLI start-up time, my > implementation currently loads many unneeded modules. I'll also avoid the > hack to hide stdout. > > > 4) Allow the use of saved connections as well as arbitrary connection > strings. > > Agree. I think saved connections will come with the "core" CLI module > implementation(the register-server command). > Right, that's my thinking. Thanks! > > On Thu, 1 Apr 2021 at 03:44, Dave Page <[email protected]> wrote: > >> Hi >> >> On Thu, Apr 1, 2021 at 12:38 AM Steve Chavez <[email protected]> wrote: >> >>> > but is this something you'd be interested in working on to become a >>> more fully featured and production quality CLI? >>> >>> Yes, absolutely! >>> >> >> Cool! >> >> >>> What would be the next step? I'm all for doing a CLI that covers pgAdmin >>> quality standards. >>> >> >> I think there are a few things we'd need to do: >> >> 1) Agree a standard for command line syntax. CLIs can easily become very >> messy - I've spent time designing such standards in the past >> (unfortunately, I can't just share that), but some thoughts might be: >> >> - have a consistent <command> <command options> <module> <action> >> <action options> syntax. For example; >> >> pgacli --output-format=json schema-diff generate-diff >> --source=... --target=... >> pgacli core register-server --host=... --maintenance-db=... ... >> pgacli grant-wizard grant-acl --select --type=tables >> --objects=schema.tb_* >> >> (I've assumed a module name of 'core' for things directly related to >> the heart of pgAdmin) >> >> - Actions are in the form "verb-noun" >> - Every common option has both a short and a long form >> - Uncommon options have a long form only >> - Support various help options, showing appropriate info; >> >> pgacli -h >> pgacli --help >> pgacli core -h >> pgacli schema-diff --help >> >> 2) Figure out how to avoid the hacks in your existing code to do things >> like: >> >> - Hide initialisation output >> - Setup the dummy SQLite database >> >> 3) Make the code modular and extensible; perhaps using a Python module >> per CLI module. >> >> 4) Allow the use of saved connections as well as arbitrary connection >> strings. >> >> I think 2 is the hardest issue. It may be that we add a flag to the main >> application code that tells it it's running under the CLI, so that it can >> avoid doing things it doesn't need to in those cases. One example is the >> module loading; we might want to skip that entirely under the CLI, and have >> the CLI module that's being called load just the modules it needs. >> >> >>> >>> On Wed, 31 Mar 2021 at 03:08, Dave Page <[email protected]> wrote: >>> >>>> Hi >>>> >>>> On Tue, Mar 30, 2021 at 3:36 PM Steve Chavez <[email protected]> wrote: >>>> >>>>> Hey all, >>>>> >>>>> In case anyone is interested, I've managed to enable a CLI mode for >>>>> the Schema Diff on this repo: >>>>> >>>>> https://github.com/steve-chavez/pgadmin4/blob/cli/web/cli.py >>>>> >>>>> It basically works by using a Flask test client that interacts with >>>>> the Schema Diff endpoints. >>>>> It's a single isolated file, I haven't patched any of the existing >>>>> modules. >>>>> >>>>> For a quickstart, there's also a docker image that can be used like: >>>>> >>>>> docker run supabase/pgadmin-schema-diff \ >>>>> 'postgres://postgres@host/diff_source' \ >>>>> 'postgres://postgres@host/diff_target' \ >>>>> > diff.sql >>>>> ## the stderr output shows the same messages as the Schema Diff GUI: >>>>> >>>>> Starting schema diff... >>>>> Comparision started......0% >>>>> Comparing FTS Dictionaries ...35% >>>>> Comparing Functions ...50% >>>>> Comparing Trigger Functions ...60% >>>>> Comparing Sequences ...70% >>>>> Comparing Tables ...80% >>>>> Comparing Views ...90% >>>>> Done. >>>>> >>>>> >>>> That's an interesting approach! Obviously the code is just a proof of >>>> concept at the moment (redirecting stdout is masking errors for example), >>>> but is this something you'd be interested in working on to become a more >>>> fully featured and production quality CLI? >>>> >>>> -- >>>> Dave Page >>>> Blog: http://pgsnake.blogspot.com >>>> Twitter: @pgsnake >>>> >>>> EDB: http://www.enterprisedb.com >>>> >>>> >> >> -- >> Dave Page >> Blog: http://pgsnake.blogspot.com >> Twitter: @pgsnake >> >> EDB: http://www.enterprisedb.com >> >> -- Dave Page Blog: https://pgsnake.blogspot.com Twitter: @pgsnake EDB: https://www.enterprisedb.com ^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2021-04-19 08:22 UTC | newest] Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2021-03-10 18:42 CLI for Schema Diff Steve Chavez <[email protected]> 2021-03-29 23:41 ` Steve Chavez <[email protected]> 2021-03-31 08:08 ` Dave Page <[email protected]> 2021-03-31 23:37 ` Steve Chavez <[email protected]> 2021-04-01 08:44 ` Dave Page <[email protected]> 2021-04-19 05:47 ` Steve Chavez <[email protected]> 2021-04-19 08:22 ` Dave Page <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox