Subject: Postgresql long transaction support

From: Kirmo Uusitalo <[email-masked]>
Date: Wed, 14 Dec 2022 13:54:54 +0200

Hi,

I'm looking for a solution for version managed (or long transaction) data
in PostgreSQL (and Oracle).

This means database objects having different properties in different
versions of the data set. Versions could be organized in tree-like
hierarchy (each versios can have subversions). Objects can even be deleted
in some of the versions of the data.

I am looking for something which would require minimal change in the
application logic, like

    create table my_obj(id serial primary key,prop_1 text);

    insert into my_obj (prop_1) values ('object 1');

    insert into my_obj (prop_1) values ('object 2 (to be deleted)');

select * from my_obj;

1   object 1
2   object 2 (to be deleted)

create version 'xx'; //this functionality i'm looking for
switch to version 'xx'; //this functionality i'm looking for

select * from my_obj; //same data as in top version as we haven't
changed anything yet

1   object 1
2   object 2 (to be deleted)

update my_obj set prop_1='updated in version xx' where id = 1;delete
from my_obj where id=2;

select * from my_obj;

1   'updated in version xx'

switch to version 'top'; //let's go back to top version

select * from my_obj; //same data as before as we haven't changed posted
our changes made in 'xx' yet to top version

1   object 1
2   object 2 (to be deleted)

insert into my_obj (prop_1) values ('object 3 (inserted after creation of xx)');

switch to version 'xx'; //this functionality i'm looking for

merge; //brings changes made in parent 'top' version

1   updated in version xx
3   object 3 (inserted after creation of xx)

post; //sends all changes upwards

This functionality exists in GE Smallworld spatial database (see
https://www.ge.com/content/dam/gepower-new/global/en_US/downloads/gas-new-site/resources/reference/ger-4231-smallworld-4-managing-change-spatial-data.pdf)
and the use case is quite common (plan some changes in data which will
happen in real life in far future, much further away than when current
database session ends). Also the conflict management should exist (same
object being changed in multiple versions).

Does Postgresql have anything like this?

---

From: Samed YILDIRIM <[email-masked]>
Date: Wed, 14 Dec 2022 14:02:02 +0200

Hi Kirmo,

This is a pretty interesting feature. I'm not sure if I've seen something
like that before. I recommend you to check the pg_bitemporal[1] project. It
may be useful for you.

[1]: https://github.com/hettie-d/pg_bitemporal

Best regards.
Samed YILDIRIM

On Wed, 14 Dec 2022 at 13:55, Kirmo Uusitalo <[email-masked]>
wrote:

> select * from my_obj; //same data as in top version as we haven't changed anything yet
>
> 1   object 1
> update my_obj set prop_1='updated in version xx' where id = 1;delete from my_obj where id=2;
>
> select * from my_obj;
>

---

From: Kirmo Uusitalo <[email-masked]>
Date: Wed, 14 Dec 2022 14:45:47 +0200

Hi Samed,

thanks. The bitemporal tables are somewhat similar except that there is not
a date range to distinguish between versions of objects. One could use this
long transaction feature for comparing the results of different plans of
implementing something - and these could be affecting many objects from the
parent version (or it's parent and so forth).

Similarly as with bitemporal tables the FK constraints are problematic. If
an object is to be deleted in parent version, any created child object of
this in child version cannot be posted to parent version later. That's why
I believe it is best first merge the changes done in parent version to
current version before you are allowed to post.

To solve this in application layer for an existing application is quite
complex and this is why I am looking for a more generic solution within the
database.

---

From: Steve Midgley <[email-masked]>
Date: Wed, 14 Dec 2022 11:59:44 -0800

On Wed, Dec 14, 2022 at 4:46 AM Kirmo Uusitalo <[email-masked]>
wrote:

> Hi Samed,
>
> thanks. The bitemporal tables are somewhat similar except that there is
> not a date range to distinguish between versions of objects. One could use
> this long transaction feature for comparing the results of different plans
> of implementing something - and these could be affecting many objects from
> the parent version (or it's parent and so forth).
>
This seems pretty complex to address at any layer! But it would be a useful
feature for some situations, for sure. It seems like the approach taken by
Ruby/Rails and similar frameworks for "db migrations" might have some
useful design patterns for you. Each transaction has a link to its future
and past transaction. If the system wants to traverse from "migration 6 to
migration 3" and the DB is currently at migration 6 (stored a version
number in the db itself), it runs the exit function to downgrade to
migration 5, then runs the logic for migration 5, then runs the downgrade
to 4, etc.

It can be slow, as implemented, but it allows you to attach logic to every
step forward and backward in time that ensures you can handle even
structure changes in the tables, etc (as well as changing lookup values in
tables, etc).

Might be worth considering as a design model?
Steve

---

From: Tom Lane <[email-masked]>
Date: Wed, 14 Dec 2022 16:13:32 -0500

> Might be worth considering as a design model?

If you end up building things that way, PG's "extensions" could be a
useful way to package it.  What Steve is calling a "migration" corresponds
to an extension version, and you can provide scripts to go in either
direction between versions.

			regards, tom lane