Subject: update faster way

From: yudhi s <[email-masked]>
Date: Sat, 14 Sep 2024 08:43:34 +0530

Hello,
We have to update a column value(from numbers like '123' to codes like
'abc' by looking into a reference table data) in a partitioned table with
billions of rows in it, with each partition having 100's millions rows. As
we tested for ~30million rows it's taking ~20minutes to update. So if we go
by this calculation, it's going to take days for updating all the values.
So my question is

1) If there is any inbuilt way of running the update query in parallel
(e.g. using parallel hints etc) to make it run faster?
2) should we run each individual partition in a separate session (e.g. five
partitions will have the updates done at same time from 5 different
sessions)? And will it have any locking effect or we can just start the
sessions and let them run without impacting our live transactions?

UPDATE tab_part1
SET column1 = reftab.code
FROM reference_tab reftab
WHERE tab_part1.column1 = subquery.column1;

Regards
Yudhi

---

From: Igor Korot <[email-masked]>
Date: Fri, 13 Sep 2024 22:29:12 -0500

Hii,

On Fri, Sep 13, 2024 at 10:22 PM yudhi s <[email-masked]> wrote:
>
> Hello,
> We have to update a column value(from numbers like '123' to codes like 'abc' by looking into a reference table data) in a partitioned table with billions of rows in it, with each partition having 100's millions rows. As we tested for ~30million rows it's taking ~20minutes to update. So if we go by this calculation, it's going to take days for updating all the values. So my question is
>
> 1) If there is any inbuilt way of running the update query in parallel (e.g. using parallel hints etc) to make it run faster?
> 2) should we run each individual partition in a separate session (e.g. five partitions will have the updates done at same time from 5 different sessions)? And will it have any locking effect or we can just start the sessions and let them run without impacting our live transactions?

Do you have any indexes?
If not - you should, if yes - what are they?

Thank you.

---

From: Muhammad Usman Khan <[email-masked]>
Date: Tue, 17 Sep 2024 10:36:50 +0500

Hi,

You can solve this problem using Citus in PostgreSQL, which is specifically
designed for parallelism

SELECT create_distributed_table('tab_part1', 'partition_key');
SELECT create_distributed_table('reference_tab', 'reference_key');

UPDATE tab_part1
SET column1 = reftab.code
FROM reference_tab reftab
WHERE tab_part1.column1 = reftab.column1;

On Sat, 14 Sept 2024 at 08:22, yudhi s <[email-masked]> wrote:

> 2) should we run each individual partition in a separate session (e.g.
> five partitions will have the updates done at same time from 5 different

---

From: Alvaro Herrera <[email-masked]>
Date: Tue, 17 Sep 2024 10:28:45 +0200

On 2024-Sep-14, yudhi s wrote:

> billions of rows in it, with each partition having 100's millions rows.

Another option is to not update anything, and instead create a view on
top of the partitioned table (joined to the reference table) that
returns the reference value instead of the original number value from
the column; when the application wants to receive those reference
values, it queries the view instead of the partitioned table directly.

-- 
Álvaro Herrera               48°01'N 7°57'E  —  https://www.EnterpriseDB.com/
"I think my standards have lowered enough that now I think 'good design'
is when the page doesn't irritate the living f*ck out of me." (JWZ)