Subject: PostgreSQL historical database

From: Erik Serrano <[email-masked]>
Date: Tue, 5 Nov 2024 12:31:11 -0300

Dear,

Along with greetings, I would like to ask if there is any product, way
(architecture), system that allows me to create a historical database from
a main transactional database in PostgreSQL.

I thank you in advance for any contributions that help me to approach this
new project.

Thank you very much, Guys,
Regards

*Erik R. Serrano Saavedra*
*      Data Base Administrator*

---

From: Samed YILDIRIM <[email-masked]>
Date: Tue, 5 Nov 2024 17:37:34 +0200

Hello Erik,

It is not very clear for me what you are looking for. But, pg_bitemporal
may be answer for you. I recommend to you to check the repository below. If
this is not what you want, can you elaborate a little more?

https://github.com/hettie-d/pg_bitemporal

Best regards.
Samed YILDIRIM

---

From: Ron Johnson <[email-masked]>
Date: Tue, 5 Nov 2024 12:01:55 -0500

> a main transactional database in PostgreSQL.

Are you asking about how to "detach" a *table* from one database, and then
attach it to another database?

-- 
Death to <Redacted>, and butter sauce.
Don't boil me, I'm still alive.
<Redacted> crustacean!

---

From: Thiemo Kellner <[email-masked]>
Date: Tue, 5 Nov 2024 18:11:39 +0100 (GMT+01:00)

Hi

I am not sure what the purpose of yours is. To the best of my knowledge, transactions have no validity duration and are immutable such that historicisation use useless.

Maybe it would help to know your goals in more detail.

Kind regards

Thiemo

---

From: Erik Serrano <[email-masked]>
Date: Tue, 5 Nov 2024 14:30:35 -0300

Dear Sirs,

I'll tell you a little about what I need. Normally, during the day, records
are made or recorded in the main database, which at the end of the day are
consolidated (accounting closings) and are recorded in the database. In
order not to make the main database grow without measure (which will only
maintain the range between 3 months to 1 year). For this reason, this data
must be transferred to another database so that it lasts over time and can
be consulted by other areas. (This action is done humanly every day of the
year at the end of the day)
Therefore, the project seeks to be able to carry out this extraction of the
consolidated data to another database, but automatically.

I was thinking of doing this with some triggers or with jobs that allow me
to carry out these actions. I also thought of creating a replication of
only the consolidated tables to the new historical database server, but I
have not yet defined the method.

That's why I need to know if there is a tool that allows me to create this
database.

I hope this clarifies a little the scope of the new historical database.

Thank you very much in advance
Regards

El mar, 5 nov 2024 a las 12:37, Samed YILDIRIM (<[email-masked]>)
escribió:

>> I thank you in advance for any contributions that help me to approach
>> this new project.
>>

---

From: Ron Johnson <[email-masked]>
Date: Tue, 5 Nov 2024 13:10:30 -0500

On Tue, Nov 5, 2024 at 12:30 PM Erik Serrano <[email-masked]> wrote:

>
> Dear Sirs,
>
> I'll tell you a little about what I need. Normally, during the day,
> records are made or recorded in the main database, which at the end of the
> day are consolidated (accounting closings) and are recorded in the
> database. In order not to make the main database grow without measure
> (which will only maintain the range between 3 months to 1 year). For this
> reason, this data must be transferred to another database so that it lasts
> over time and can be consulted by other areas. (This action is done humanly
> every day of the year at the end of the day)
> Therefore, the project seeks to be able to carry out this extraction of
> the consolidated data to another database, but automatically.
>

psql in a bash script, run in a cron job.  I wrote something quite similar
that runs every month to:
1. copy old records into CSV files,
2. move them to S3 buckets,
3. delete the rows,
4. VACUUM and ANALYZE the tables.

If there are FK relationships, you'll have to determine the proper order in
which to delete them, but that's relatively easy and just has to be done
once.

(The only difference is that your Step 2 would be "copy them into the new
database".)

Depending on the number of rows per year, table partitioning might also
simplify things.  Or it could add complexity and slow things down; you'll
have to test that yourself.

---

From: Keith Fiske <[email-masked]>
Date: Tue, 5 Nov 2024 13:15:12 -0500

> *      Data Base Administrator*

I would first recommend looking into partitioning for managing data
retention like this. As Ron says, you'll want to look into the performance
implications of this, but it allows for the most efficient method of
removing old data from PostgreSQL and is typically worth the overhead
costs. Otherwise you're dealing with potentially expensive deletion
operations and managing bloat vs just detaching/dropping a table.

---

From: <[email-masked]>
Date: Fri, 15 Nov 2024 22:58:31 +0100

Hi,

For your purpose, you should study using partitioning and postgres_fdw.

Regards

Michel SALAIS

De : Erik Serrano <[email-masked]> 
Envoyé : mardi 5 novembre 2024 18:31
À : Samed YILDIRIM <[email-masked]>
Cc : Pgsql-admin <[email-masked]>; pgsql-admin <[email-masked]>
Objet : Re: PostgreSQL historical database

Dear Sirs,

I'll tell you a little about what I need. Normally, during the day, records are made or recorded in the main database, which at the end of the day are consolidated (accounting closings) and are recorded in the database. In order not to make the main database grow without measure (which will only maintain the range between 3 months to 1 year). For this reason, this data must be transferred to another database so that it lasts over time and can be consulted by other areas. (This action is done humanly every day of the year at the end of the day)
Therefore, the project seeks to be able to carry out this extraction of the consolidated data to another database, but automatically.

I was thinking of doing this with some triggers or with jobs that allow me to carry out these actions. I also thought of creating a replication of only the consolidated tables to the new historical database server, but I have not yet defined the method.

That's why I need to know if there is a tool that allows me to create this database.

 

Erik R. Serrano Saavedra

      Data Base Administrator

El mar, 5 nov 2024 a las 12:37, Samed YILDIRIM (<[email-masked] <mailto:[email-masked]> >) escribió:

Hello Erik,

It is not very clear for me what you are looking for. But, pg_bitemporal may be answer for you. I recommend to you to check the repository below. If this is not what you want, can you elaborate a little more?

Best regards.
Samed YILDIRIM

On Tue, 5 Nov 2024, 17:31 Erik Serrano, <[email-masked] <mailto:[email-masked]> > wrote:

Dear,

Along with greetings, I would like to ask if there is any product, way (architecture), system that allows me to create a historical database from a main transactional database in PostgreSQL.

I thank you in advance for any contributions that help me to approach this new project.

 

Erik R. Serrano Saavedra

      Data Base Administrator

---

From: Ron Johnson <[email-masked]>
Date: Fri, 15 Nov 2024 17:51:26 -0500

On Tue, Nov 5, 2024 at 1:15 PM Keith Fiske <[email-masked]>
wrote:
[snip]

> operations and managing bloat vs just detaching/dropping a table.

Am I the only person to have ever gotten good DELETE performance from
Postgresql?

<Redacted> lobster!

---

From: Rui DeSousa <[email-masked]>
Date: Fri, 15 Nov 2024 17:51:58 -0500

> On Nov 5, 2024, at 1:15 PM, Keith Fiske <[email-masked]> wrote:
> 
> I would first recommend looking into partitioning for managing data retention like this.

I don’t think you need partitioning.  The is so many ways to slice this problem.  

You need some form of logical replication.  I’ve used both of these methods in the past to do exactly the issue at hand.  These are not the only solutions.

1, Queue tables via triggers and then asynchronous replicate those transaction to the archive database in real time. 
2. Messages queues; Modern application make use of massage queues; Just attach the archive database to the same message queue.

I don’t see a need for partitioning.  I managed a Muti-terabyte content delivery platform where the data was only licensed between 30 to 90 days.  Records where being purged more or less just as fast as new records were being created.  In essence the database was completely reloaded with new data every 30-90 days depending on the data stream.  We did no use any form of partitioned tables and had no performance issues.

---

From: Erik Serrano <[email-masked]>
Date: Sat, 16 Nov 2024 10:41:29 -0300

Muchas gracias, Michel .  Saludos

El vie, 15 nov 2024, 19:52, Rui DeSousa <[email-masked]> escribió:

> retention like this.
> You need some form of logical replication.  I’ve used both of these
> methods in the past to do exactly the issue at hand.  These are not the
> only solutions.
>
> 1, Queue tables via triggers and then asynchronous replicate those
> transaction to the archive database in real time.
> 2. Messages queues; Modern application make use of massage queues; Just
> attach the archive database to the same message queue.
>
> I don’t see a need for partitioning.  I managed a Muti-terabyte content
> delivery platform where the data was only licensed between 30 to 90 days.
> Records where being purged more or less just as fast as new records were
> being created.  In essence the database was completely reloaded with new
> data every 30-90 days depending on the data stream.  We did no use any form
> of partitioned tables and had no performance issues.
>
>