Subject: Database migrations

From: Simon Connah <[email-masked]>
Date: Sat, 13 Mar 2021 21:40:27 +0000

Hi,

I'm sorry if this is a rather stupid question but I'm fairly new to databases and was wondering if anyone could offer some advice?

I have an app I am developing using Node.js and PostgreSQL 13.2. I have the basic tables I want to start with but I know that as development progresses change is going to be required. Since this is an open-source project people are going to want to upgrade from one version to the next and that might require changes to the database structure. What is the best way to handle this?

I was thinking about just having a bunch of SQL files with a version number and date and applying them to the database from oldest to newest (with a table storing information for files that have already been processed). Does this sound like a reasonable solution?

If not could someone point me in the right direction, please?

Thank you for your help.

Simon.

---

From: Tom Lane <[email-masked]>
Date: Sat, 13 Mar 2021 16:58:35 -0500

> If not could someone point me in the right direction, please?

If I understand your requirements properly, this is already largely
implemented in PG's "extension" infrastructure.  The system can keep
track of which version of an extension is installed, and apply the
correct delta script on request for an upgrade (or downgrade).
See here:
https://www.postgresql.org/docs/current/extend-extensions.html

The documentation mostly talks about extensions that have some
underlying C code (in a .so library file).  But it's perfectly possible
to have an extension that consists only of SQL definitions.

			regards, tom lane

---

From: Alejandro Román <[email-masked]>
Date: Sat, 13 Mar 2021 17:04:55 -0500

Hello,

> I was thinking about just having a bunch of SQL files with a version number and date and applying them to the database from oldest to newest (with a table storing information for files that have al…

It does. Actually, there is a known JS library that does exactly that: Knex.js (1). It allows you to define your SQL in plain JS (e.g. `knex.schema.createTable(‘users’, …)`). It will then keep track of which migrations (JS files) have been ran and which need to be run. It stores this metadata in a table called “migrations” (you can configure the exact name).

(1) http://knexjs.org/#Migrations

Best wishes,

Alejandro

---

From: Steve Baldwin <[email-masked]>
Date: Sun, 14 Mar 2021 10:19:35 +1100

Hi Simon,

Another option (and the one we use) is a tool called flyway -
https://flywaydb.org/. There is a free community edition, and it is pretty
comprehensive. There are docker images you can use during ci.

Flyway supports versioned (run once) migrations, repeatable (run when
changed) migrations, and a bunch more.

Other than being a user of the community edition, I have no association
with flyway.

Cheers,

Steve

On Sun, Mar 14, 2021 at 8:40 AM Simon Connah <[email-masked]>
wrote:

> Hi,
>
> I'm sorry if this is a rather stupid question but I'm fairly new to
> databases and was wondering if anyone could offer some advice?
>
> I have an app I am developing using Node.js and PostgreSQL 13.2. I have
> the basic tables I want to start with but I know that as development
> progresses change is going to be required. Since this is an open-source
> project people are going to want to upgrade from one version to the next
> and that might require changes to the database structure. What is the best
> way to handle this?
>
> I was thinking about just having a bunch of SQL files with a version
> number and date and applying them to the database from oldest to newest
> (with a table storing information for files that have already been
> processed). Does this sound like a reasonable solution?
>

---

From: Simon Connah <[email-masked]>
Date: Sun, 14 Mar 2021 09:40:05 +0000

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Saturday, March 13th, 2021 at 21:58, Tom Lane <[email-masked]> wrote:

> Simon Connah [email-masked] writes:
>

Hi Tom,

Thank you for the suggestion. I had no idea PostgreSQL had a feature for things like that.

I'll do some reading on the subject but it certainly seems interesting.

Simon.

---

From: Simon Connah <[email-masked]>
Date: Sun, 14 Mar 2021 09:47:53 +0000

> (1) http://knexjs.org/#Migrations

Hi Alejandro,

Thank you for the information. I'll read through the documentation for that as well. Having everything in JavaScript would be quite helpful in the long run I think but I do like the idea of extensions as well. I'll do some testing and see which option works out best.

Simon.