public inbox for [email protected]
help / color / mirror / Atom feedDatabase migrations
7+ messages / 4 participants
[nested] [flat]
* Database migrations
@ 2021-03-13 21:40 Simon Connah <[email protected]>
2021-03-13 21:58 ` Re: Database migrations Tom Lane <[email protected]>
2021-03-13 22:04 ` Re: Database migrations Alejandro Román <[email protected]>
2021-03-13 23:19 ` Re: Database migrations Steve Baldwin <[email protected]>
0 siblings, 3 replies; 7+ messages in thread
From: Simon Connah @ 2021-03-13 21:40 UTC (permalink / raw)
To: [email protected] <[email protected]>
Hi,
I'm sorry if this is a rather stupid question but I'm fairly new to databases and was wondering if anyone could offer some advice?
I have an app I am developing using Node.js and PostgreSQL 13.2. I have the basic tables I want to start with but I know that as development progresses change is going to be required. Since this is an open-source project people are going to want to upgrade from one version to the next and that might require changes to the database structure. What is the best way to handle this?
I was thinking about just having a bunch of SQL files with a version number and date and applying them to the database from oldest to newest (with a table storing information for files that have already been processed). Does this sound like a reasonable solution?
If not could someone point me in the right direction, please?
Thank you for your help.
Simon.
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Database migrations
2021-03-13 21:40 Database migrations Simon Connah <[email protected]>
@ 2021-03-13 21:58 ` Tom Lane <[email protected]>
2021-03-14 09:40 ` Re: Database migrations Simon Connah <[email protected]>
2 siblings, 1 reply; 7+ messages in thread
From: Tom Lane @ 2021-03-13 21:58 UTC (permalink / raw)
To: Simon Connah <[email protected]>; +Cc: [email protected] <[email protected]>
Simon Connah <[email protected]> writes:
> I'm sorry if this is a rather stupid question but I'm fairly new to databases and was wondering if anyone could offer some advice?
> I have an app I am developing using Node.js and PostgreSQL 13.2. I have the basic tables I want to start with but I know that as development progresses change is going to be required. Since this is an open-source project people are going to want to upgrade from one version to the next and that might require changes to the database structure. What is the best way to handle this?
> I was thinking about just having a bunch of SQL files with a version number and date and applying them to the database from oldest to newest (with a table storing information for files that have already been processed). Does this sound like a reasonable solution?
> If not could someone point me in the right direction, please?
If I understand your requirements properly, this is already largely
implemented in PG's "extension" infrastructure. The system can keep
track of which version of an extension is installed, and apply the
correct delta script on request for an upgrade (or downgrade).
See here:
https://www.postgresql.org/docs/current/extend-extensions.html
The documentation mostly talks about extensions that have some
underlying C code (in a .so library file). But it's perfectly possible
to have an extension that consists only of SQL definitions.
regards, tom lane
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Database migrations
2021-03-13 21:40 Database migrations Simon Connah <[email protected]>
2021-03-13 21:58 ` Re: Database migrations Tom Lane <[email protected]>
@ 2021-03-14 09:40 ` Simon Connah <[email protected]>
0 siblings, 0 replies; 7+ messages in thread
From: Simon Connah @ 2021-03-14 09:40 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: [email protected] <[email protected]>
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Saturday, March 13th, 2021 at 21:58, Tom Lane <[email protected]> wrote:
> Simon Connah [email protected] writes:
>
> > I'm sorry if this is a rather stupid question but I'm fairly new to databases and was wondering if anyone could offer some advice?
>
> > I have an app I am developing using Node.js and PostgreSQL 13.2. I have the basic tables I want to start with but I know that as development progresses change is going to be required. Since this is an open-source project people are going to want to upgrade from one version to the next and that might require changes to the database structure. What is the best way to handle this?
>
> > I was thinking about just having a bunch of SQL files with a version number and date and applying them to the database from oldest to newest (with a table storing information for files that have already been processed). Does this sound like a reasonable solution?
>
> > If not could someone point me in the right direction, please?
>
> If I understand your requirements properly, this is already largely
>
> implemented in PG's "extension" infrastructure. The system can keep
>
> track of which version of an extension is installed, and apply the
>
> correct delta script on request for an upgrade (or downgrade).
>
> See here:
>
> https://www.postgresql.org/docs/current/extend-extensions.html
>
> The documentation mostly talks about extensions that have some
>
> underlying C code (in a .so library file). But it's perfectly possible
>
> to have an extension that consists only of SQL definitions.
>
> regards, tom lane
Hi Tom,
Thank you for the suggestion. I had no idea PostgreSQL had a feature for things like that.
I'll do some reading on the subject but it certainly seems interesting.
Simon.
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Database migrations
2021-03-13 21:40 Database migrations Simon Connah <[email protected]>
@ 2021-03-13 22:04 ` Alejandro Román <[email protected]>
2021-03-14 09:47 ` Re: Database migrations Simon Connah <[email protected]>
2 siblings, 1 reply; 7+ messages in thread
From: Alejandro Román @ 2021-03-13 22:04 UTC (permalink / raw)
To: Simon Connah <[email protected]>; +Cc: [email protected] <[email protected]>
Hello,
> I was thinking about just having a bunch of SQL files with a version number and date and applying them to the database from oldest to newest (with a table storing information for files that have already been processed). Does this sound like a reasonable solution?
It does. Actually, there is a known JS library that does exactly that: Knex.js (1). It allows you to define your SQL in plain JS (e.g. `knex.schema.createTable(‘users’, …)`). It will then keep track of which migrations (JS files) have been ran and which need to be run. It stores this metadata in a table called “migrations” (you can configure the exact name).
(1) http://knexjs.org/#Migrations
Best wishes,
Alejandro
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Database migrations
2021-03-13 21:40 Database migrations Simon Connah <[email protected]>
2021-03-13 22:04 ` Re: Database migrations Alejandro Román <[email protected]>
@ 2021-03-14 09:47 ` Simon Connah <[email protected]>
0 siblings, 0 replies; 7+ messages in thread
From: Simon Connah @ 2021-03-14 09:47 UTC (permalink / raw)
To: Alejandro Román <[email protected]>; +Cc: [email protected] <[email protected]>
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Saturday, March 13th, 2021 at 22:04, Alejandro Román <[email protected]> wrote:
> Hello,
>
> > I was thinking about just having a bunch of SQL files with a version number and date and applying them to the database from oldest to newest (with a table storing information for files that have already been processed). Does this sound like a reasonable solution?
>
> It does. Actually, there is a known JS library that does exactly that: Knex.js (1). It allows you to define your SQL in plain JS (e.g. `knex.schema.createTable(‘users’, …)`). It will then keep track of which migrations (JS files) have been ran and which need to be run. It stores this metadata in a table called “migrations” (you can configure the exact name).
>
> (1) http://knexjs.org/#Migrations
>
> Best wishes,
>
> Alejandro
Hi Alejandro,
Thank you for the information. I'll read through the documentation for that as well. Having everything in JavaScript would be quite helpful in the long run I think but I do like the idea of extensions as well. I'll do some testing and see which option works out best.
Simon.
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Database migrations
2021-03-13 21:40 Database migrations Simon Connah <[email protected]>
@ 2021-03-13 23:19 ` Steve Baldwin <[email protected]>
2 siblings, 0 replies; 7+ messages in thread
From: Steve Baldwin @ 2021-03-13 23:19 UTC (permalink / raw)
To: Simon Connah <[email protected]>; +Cc: [email protected] <[email protected]>
Hi Simon,
Another option (and the one we use) is a tool called flyway -
https://flywaydb.org/. There is a free community edition, and it is pretty
comprehensive. There are docker images you can use during ci.
Flyway supports versioned (run once) migrations, repeatable (run when
changed) migrations, and a bunch more.
Other than being a user of the community edition, I have no association
with flyway.
Cheers,
Steve
On Sun, Mar 14, 2021 at 8:40 AM Simon Connah <[email protected]>
wrote:
> Hi,
>
> I'm sorry if this is a rather stupid question but I'm fairly new to
> databases and was wondering if anyone could offer some advice?
>
> I have an app I am developing using Node.js and PostgreSQL 13.2. I have
> the basic tables I want to start with but I know that as development
> progresses change is going to be required. Since this is an open-source
> project people are going to want to upgrade from one version to the next
> and that might require changes to the database structure. What is the best
> way to handle this?
>
> I was thinking about just having a bunch of SQL files with a version
> number and date and applying them to the database from oldest to newest
> (with a table storing information for files that have already been
> processed). Does this sound like a reasonable solution?
>
> If not could someone point me in the right direction, please?
>
> Thank you for your help.
>
> Simon.
>
>
>
>
>
^ permalink raw reply [nested|flat] 7+ messages in thread
* Database migrations
@ 2024-06-28 10:26 Simon Connah <[email protected]>
0 siblings, 0 replies; 7+ messages in thread
From: Simon Connah @ 2024-06-28 10:26 UTC (permalink / raw)
To: pgsql-novice
Hi,
I'm building a web app using the Python web framework Flask and I'm curious how I should handle database changes. At the moment I can just delete everything and load the SQL files but when I launch into production obviously I won't be able to do that. Currently all my database stuff is stored in a SQL file. One file for each table and includes stored procedures and function definitions.
I know there are libraries like Alembic but that seems to be for SQLAlchemy and not raw SQL and Flyway seems to require an expensive license to allow migration rollback so I'm wondering if anyone can offer some advice?
If you need more information then please let me know.
Simon.
Attachments:
[application/pgp-signature] signature.asc (249B, ../../fbpkeeZLFOGdtB9bbncwzbn-sAu-VEIoln2KStvcs7BgYCA3th1QZNffk0fFgaSEHAqSUPK8-Jl3pkqUQjvjcXQ9CRnJ4I2hrZWM9ykEvwY=@protonmail.com/2-signature.asc)
download
^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2024-06-28 10:26 UTC | newest]
Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-03-13 21:40 Database migrations Simon Connah <[email protected]>
2021-03-13 21:58 ` Tom Lane <[email protected]>
2021-03-14 09:40 ` Simon Connah <[email protected]>
2021-03-13 22:04 ` Alejandro Román <[email protected]>
2021-03-14 09:47 ` Simon Connah <[email protected]>
2021-03-13 23:19 ` Steve Baldwin <[email protected]>
2024-06-28 10:26 Database migrations Simon Connah <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox