Subject: Best Practices for Managing Schema Changes Dynamically with libpq

From: Sasmit Utkarsh <[email-masked]>
Date: Tue, 3 Dec 2024 23:13:46 +0530

Dear PostgreSQL Community Team,

I am working on a project that uses libpq along with C language to interact
with PostgreSQL, and we face challenges with managing schema changes
dynamically in production while avoiding downtime. Specifically, we need
guidance on handling table structure changes/additions without tightly
coupling these changes to application updates.

*Current Approach:*
Schema changes are deployed first, followed by application updates to align
with the new structure.

*Challenges:*
Ensuring application stability during the transitional phase when the
schema and code are not fully in sync.
Handling table structure changes (e.g., adding new columns) dynamically
without requiring immediate code changes.

*Questions:*
Are there recommended best practices for managing such schema changes with
libpq?
How can we efficiently handle table additions/updates while keeping the
application and database in sync dynamically?

I would appreciate any guidance, patterns, or examples that can help us
implement a robust solution.

Thank you for your time and support!

Regards,
Sasmit Utkarsh
+91-7674022625

---

From: Ron Johnson <[email-masked]>
Date: Tue, 3 Dec 2024 13:10:05 -0500

On Tue, Dec 3, 2024 at 12:44 PM Sasmit Utkarsh <[email-masked]>
wrote:
[snip]

> application and database in sync dynamically?

Enumerate all relevant column names in SELECT and INSERT statements.  That
way, the application still works when you add columns or alter(*) data
types.  Dropping columns will still break your app.

*Altering to from a numeric type to a text type might still kill your app,
if it can't convert the string into your app's int or float type..

-- 
Death to <Redacted>, and butter sauce.
Don't boil me, I'm still alive.
<Redacted> lobster!

---

From: Adrian Klaver <[email-masked]>
Date: Tue, 3 Dec 2024 12:50:54 -0800

On 12/3/24 09:43, Sasmit Utkarsh wrote:
> I am working on a project that uses libpq along with C language to 
> interact with PostgreSQL, and we face challenges with managing schema 
> changes dynamically in production while avoiding downtime. Specifically, 
> we need guidance on handling table structure changes/additions without 
> tightly coupling these changes to application updates.
> 
> *Current Approach:*
> Schema changes are deployed first, followed by application updates to 
> align with the new structure.
> 
> *Challenges:*
> Are there recommended best practices for managing such schema changes 
> with libpq?

I use Sqitch(https://sqitch.org/). You have to squint but it is libpq, 
of a sort, as it uses psql to do its changes.

> application and database in sync dynamically?

There is way too many variations that enter into the above to give a 
complete concrete answer in anything less then a short book.

My general rule for this is to create a map of the process in outline 
form. Personally I still think better on paper and I pull out a legal 
pad and pencil and write out a work flow that goes from where I am to 
where I want to be. This starts with the 10000 foot view that I then 
drill down in to get the specific actions. I use a pencil as the drill 
down process often uncovers flaws in the 10000 foot view. This by the 
way was a method my 7th grade math teacher taught the class back way 
back when.

-- 
Adrian Klaver
[email-masked]

---

From: Erik Wienhold <[email-masked]>
Date: Fri, 6 Dec 2024 03:06:03 +0100

> without requiring immediate code changes.

What you're looking for is the "Expand and Contract" pattern[1][2].  The
transitional phase between expand and contract has to support both old
and new code until the old code is migrated as well.  How you keep the
schema compatible with the old code for some time depends on the kind of
schema changes.  Some use cases from the top of my head:

1) expand:     add unconstrained columns
   transition: adapt code to use new columns
   contract:   add constraints

2) expand:     rename tables/columns
   transition: add (updatable) views that expose the old names until the
               code is adapted to the new names
   contract:   drop views

3) expand:     add columns with constraints
   transition: backfill new columns with triggers
   contract:   drop triggers

[1] https://www.tim-wellhausen.de/papers/ExpandAndContract/ExpandAndContract.html
[2] https://martinfowler.com/articles/evodb.html#everything_refactoring

-- 
Erik