public inbox for [email protected]help / color / mirror / Atom feed
Database migrations 7+ messages / 4 participants [nested] [flat]
* Database migrations @ 2021-03-13 21:40 Simon Connah <[email protected]> 0 siblings, 3 replies; 7+ messages in thread From: Simon Connah @ 2021-03-13 21:40 UTC (permalink / raw) To: [email protected] <[email protected]> Hi, I'm sorry if this is a rather stupid question but I'm fairly new to databases and was wondering if anyone could offer some advice? I have an app I am developing using Node.js and PostgreSQL 13.2. I have the basic tables I want to start with but I know that as development progresses change is going to be required. Since this is an open-source project people are going to want to upgrade from one version to the next and that might require changes to the database structure. What is the best way to handle this? I was thinking about just having a bunch of SQL files with a version number and date and applying them to the database from oldest to newest (with a table storing information for files that have already been processed). Does this sound like a reasonable solution? If not could someone point me in the right direction, please? Thank you for your help. Simon. ^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Database migrations @ 2021-03-13 21:58 Tom Lane <[email protected]> parent: Simon Connah <[email protected]> 2 siblings, 1 reply; 7+ messages in thread From: Tom Lane @ 2021-03-13 21:58 UTC (permalink / raw) To: Simon Connah <[email protected]>; +Cc: [email protected] <[email protected]> Simon Connah <[email protected]> writes: > I'm sorry if this is a rather stupid question but I'm fairly new to databases and was wondering if anyone could offer some advice? > I have an app I am developing using Node.js and PostgreSQL 13.2. I have the basic tables I want to start with but I know that as development progresses change is going to be required. Since this is an open-source project people are going to want to upgrade from one version to the next and that might require changes to the database structure. What is the best way to handle this? > I was thinking about just having a bunch of SQL files with a version number and date and applying them to the database from oldest to newest (with a table storing information for files that have already been processed). Does this sound like a reasonable solution? > If not could someone point me in the right direction, please? If I understand your requirements properly, this is already largely implemented in PG's "extension" infrastructure. The system can keep track of which version of an extension is installed, and apply the correct delta script on request for an upgrade (or downgrade). See here: https://www.postgresql.org/docs/current/extend-extensions.html The documentation mostly talks about extensions that have some underlying C code (in a .so library file). But it's perfectly possible to have an extension that consists only of SQL definitions. regards, tom lane ^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Database migrations @ 2021-03-13 22:04 Alejandro Román <[email protected]> parent: Simon Connah <[email protected]> 2 siblings, 1 reply; 7+ messages in thread From: Alejandro Román @ 2021-03-13 22:04 UTC (permalink / raw) To: Simon Connah <[email protected]>; +Cc: [email protected] <[email protected]> Hello, > I was thinking about just having a bunch of SQL files with a version number and date and applying them to the database from oldest to newest (with a table storing information for files that have already been processed). Does this sound like a reasonable solution? It does. Actually, there is a known JS library that does exactly that: Knex.js (1). It allows you to define your SQL in plain JS (e.g. `knex.schema.createTable(‘users’, …)`). It will then keep track of which migrations (JS files) have been ran and which need to be run. It stores this metadata in a table called “migrations” (you can configure the exact name). (1) http://knexjs.org/#Migrations Best wishes, Alejandro ^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Database migrations @ 2021-03-13 23:19 Steve Baldwin <[email protected]> parent: Simon Connah <[email protected]> 2 siblings, 0 replies; 7+ messages in thread From: Steve Baldwin @ 2021-03-13 23:19 UTC (permalink / raw) To: Simon Connah <[email protected]>; +Cc: [email protected] <[email protected]> Hi Simon, Another option (and the one we use) is a tool called flyway - https://flywaydb.org/. There is a free community edition, and it is pretty comprehensive. There are docker images you can use during ci. Flyway supports versioned (run once) migrations, repeatable (run when changed) migrations, and a bunch more. Other than being a user of the community edition, I have no association with flyway. Cheers, Steve On Sun, Mar 14, 2021 at 8:40 AM Simon Connah <[email protected]> wrote: > Hi, > > I'm sorry if this is a rather stupid question but I'm fairly new to > databases and was wondering if anyone could offer some advice? > > I have an app I am developing using Node.js and PostgreSQL 13.2. I have > the basic tables I want to start with but I know that as development > progresses change is going to be required. Since this is an open-source > project people are going to want to upgrade from one version to the next > and that might require changes to the database structure. What is the best > way to handle this? > > I was thinking about just having a bunch of SQL files with a version > number and date and applying them to the database from oldest to newest > (with a table storing information for files that have already been > processed). Does this sound like a reasonable solution? > > If not could someone point me in the right direction, please? > > Thank you for your help. > > Simon. > > > > > ^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Database migrations @ 2021-03-14 09:40 Simon Connah <[email protected]> parent: Tom Lane <[email protected]> 0 siblings, 0 replies; 7+ messages in thread From: Simon Connah @ 2021-03-14 09:40 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: [email protected] <[email protected]> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Saturday, March 13th, 2021 at 21:58, Tom Lane <[email protected]> wrote: > Simon Connah [email protected] writes: > > > I'm sorry if this is a rather stupid question but I'm fairly new to databases and was wondering if anyone could offer some advice? > > > I have an app I am developing using Node.js and PostgreSQL 13.2. I have the basic tables I want to start with but I know that as development progresses change is going to be required. Since this is an open-source project people are going to want to upgrade from one version to the next and that might require changes to the database structure. What is the best way to handle this? > > > I was thinking about just having a bunch of SQL files with a version number and date and applying them to the database from oldest to newest (with a table storing information for files that have already been processed). Does this sound like a reasonable solution? > > > If not could someone point me in the right direction, please? > > If I understand your requirements properly, this is already largely > > implemented in PG's "extension" infrastructure. The system can keep > > track of which version of an extension is installed, and apply the > > correct delta script on request for an upgrade (or downgrade). > > See here: > > https://www.postgresql.org/docs/current/extend-extensions.html > > The documentation mostly talks about extensions that have some > > underlying C code (in a .so library file). But it's perfectly possible > > to have an extension that consists only of SQL definitions. > > regards, tom lane Hi Tom, Thank you for the suggestion. I had no idea PostgreSQL had a feature for things like that. I'll do some reading on the subject but it certainly seems interesting. Simon. ^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Database migrations @ 2021-03-14 09:47 Simon Connah <[email protected]> parent: Alejandro Román <[email protected]> 0 siblings, 0 replies; 7+ messages in thread From: Simon Connah @ 2021-03-14 09:47 UTC (permalink / raw) To: Alejandro Román <[email protected]>; +Cc: [email protected] <[email protected]> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Saturday, March 13th, 2021 at 22:04, Alejandro Román <[email protected]> wrote: > Hello, > > > I was thinking about just having a bunch of SQL files with a version number and date and applying them to the database from oldest to newest (with a table storing information for files that have already been processed). Does this sound like a reasonable solution? > > It does. Actually, there is a known JS library that does exactly that: Knex.js (1). It allows you to define your SQL in plain JS (e.g. `knex.schema.createTable(‘users’, …)`). It will then keep track of which migrations (JS files) have been ran and which need to be run. It stores this metadata in a table called “migrations” (you can configure the exact name). > > (1) http://knexjs.org/#Migrations > > Best wishes, > > Alejandro Hi Alejandro, Thank you for the information. I'll read through the documentation for that as well. Having everything in JavaScript would be quite helpful in the long run I think but I do like the idea of extensions as well. I'll do some testing and see which option works out best. Simon. ^ permalink raw reply [nested|flat] 7+ messages in thread
* Database migrations @ 2024-06-28 10:26 Simon Connah <[email protected]> 0 siblings, 0 replies; 7+ messages in thread From: Simon Connah @ 2024-06-28 10:26 UTC (permalink / raw) To: pgsql-novice Hi, I'm building a web app using the Python web framework Flask and I'm curious how I should handle database changes. At the moment I can just delete everything and load the SQL files but when I launch into production obviously I won't be able to do that. Currently all my database stuff is stored in a SQL file. One file for each table and includes stored procedures and function definitions. I know there are libraries like Alembic but that seems to be for SQLAlchemy and not raw SQL and Flyway seems to require an expensive license to allow migration rollback so I'm wondering if anyone can offer some advice? If you need more information then please let me know. Simon. Attachments: [application/pgp-signature] signature.asc (249B, ../../fbpkeeZLFOGdtB9bbncwzbn-sAu-VEIoln2KStvcs7BgYCA3th1QZNffk0fFgaSEHAqSUPK8-Jl3pkqUQjvjcXQ9CRnJ4I2hrZWM9ykEvwY=@protonmail.com/2-signature.asc) download ^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2024-06-28 10:26 UTC | newest] Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2021-03-13 21:40 Database migrations Simon Connah <[email protected]> 2021-03-13 21:58 ` Tom Lane <[email protected]> 2021-03-14 09:40 ` Simon Connah <[email protected]> 2021-03-13 22:04 ` Alejandro Román <[email protected]> 2021-03-14 09:47 ` Simon Connah <[email protected]> 2021-03-13 23:19 ` Steve Baldwin <[email protected]> 2024-06-28 10:26 Database migrations Simon Connah <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox