pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedFix for consume_xids advancing XIDs incorrectly
19+ messages / 7 participants
[nested] [flat]
Commits from this thread:
2026-11-18 1b4abc23f3 03a2086a49 a47c0b82e6 344322da51 xid_wraparound: Fix XID consumption accounting
* Fix for consume_xids advancing XIDs incorrectly
@ 2024-10-15 01:51 Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>
0 siblings, 2 replies; 19+ messages in thread
From: Yushi Ogiwara @ 2024-10-15 01:51 UTC (permalink / raw)
To: pgsql-hackers
Hi,
I found that the consume_xids function incorrectly advances XIDs as
shown:
postgres=# select txid_current();
txid_current
--------------
746
(1 row)
postgres=# select consume_xids('100');
consume_xids
--------------
847
(1 row)
In the example, the consume_xids function consumes 100 XIDs when XID =
746, so the desired outcome from consume_xids should be 746 + 100 = 846,
which differs from the actual outcome, 847.
Behavior inside a transaction block:
postgres=# select txid_current();
txid_current
--------------
1410
(1 row)
postgres=# begin;
BEGIN
postgres=*# select consume_xids('100');
consume_xids
--------------
1511
(1 row)
postgres=*# select consume_xids('100');
consume_xids
--------------
1521
(1 row)
Here, the first call inside the transaction block consumes 100+1 XIDs
(incorrect), while the second call consumes exactly 100 XIDs (as
expected)
Summary:
The function performs incorrectly when:
- Outside of a transaction block
- The first call inside a transaction block
But works correctly when:
- After the second call inside a transaction block
The issue arises because consume_xids does not correctly count the
consumed XIDs when it calls the GetTopTransactionId function, which
allocates a new XID when none has been assigned.
With the attached patch, the consume_xids works as expected, both inside
and outside of transaction blocks as shown below:
postgres=# select txid_current();
txid_current
--------------
1546
(1 row)
postgres=# select consume_xids('100');
consume_xids
--------------
1646
(1 row)
postgres=# begin;
BEGIN
postgres=*# select consume_xids('100');
consume_xids
--------------
1746
(1 row)
postgres=*# select consume_xids('100');
consume_xids
--------------
1846
(1 row)
Regards,
Yushi
Attachments:
skip_id_correctly.diff (text/x-diff, 909B) download
From 59bc3f300b8447a407c8f90ebb542d1c0ef0e932 Mon Sep 17 00:00:00 2001
From: Yushi Ogiwara <yushiogiwara@keio.jp>
Date: Fri, 11 Oct 2024 14:45:26 +0900
Subject: [PATCH] fix xid
---
src/test/modules/xid_wraparound/xid_wraparound.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/src/test/modules/xid_wraparound/xid_wraparound.c b/src/test/modules/xid_wraparound/xid_wraparound.c
index dce81c0c6d..c5f3b717cf 100644
--- a/src/test/modules/xid_wraparound/xid_wraparound.c
+++ b/src/test/modules/xid_wraparound/xid_wraparound.c
@@ -89,7 +89,11 @@ consume_xids_common(FullTransactionId untilxid, uint64 nxids)
* the cache overflows, but beyond that, we don't keep track of the
* consumed XIDs.
*/
- (void) GetTopTransactionId();
+ if(!FullTransactionIdIsValid(GetTopFullTransactionIdIfAny()))
+ {
+ (void) GetTopTransactionId();
+ consumed++;
+ }
for (;;)
{
--
2.40.1
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Fix for consume_xids advancing XIDs incorrectly
@ 2024-10-15 22:26 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>
1 sibling, 1 reply; 19+ messages in thread
From: Masahiko Sawada @ 2024-10-15 22:26 UTC (permalink / raw)
To: Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>; +Cc: pgsql-hackers
Hi,
Thank you for the report.
On Mon, Oct 14, 2024 at 6:51 PM Yushi Ogiwara
<btogiwarayuushi@oss.nttdata.com> wrote:
>
> Hi,
>
> I found that the consume_xids function incorrectly advances XIDs as
> shown:
>
> postgres=# select txid_current();
> txid_current
> --------------
> 746
> (1 row)
>
> postgres=# select consume_xids('100');
> consume_xids
> --------------
> 847
> (1 row)
>
> In the example, the consume_xids function consumes 100 XIDs when XID =
> 746, so the desired outcome from consume_xids should be 746 + 100 = 846,
> which differs from the actual outcome, 847.
>
> Behavior inside a transaction block:
>
> postgres=# select txid_current();
> txid_current
> --------------
> 1410
> (1 row)
>
> postgres=# begin;
> BEGIN
> postgres=*# select consume_xids('100');
> consume_xids
> --------------
> 1511
> (1 row)
> postgres=*# select consume_xids('100');
> consume_xids
> --------------
> 1521
> (1 row)
>
> Here, the first call inside the transaction block consumes 100+1 XIDs
> (incorrect), while the second call consumes exactly 100 XIDs (as
> expected)
>
> Summary:
>
> The function performs incorrectly when:
> - Outside of a transaction block
> - The first call inside a transaction block
> But works correctly when:
> - After the second call inside a transaction block
>
> The issue arises because consume_xids does not correctly count the
> consumed XIDs when it calls the GetTopTransactionId function, which
> allocates a new XID when none has been assigned.
I agree with your analysis.
I have one comment on the patch:
- (void) GetTopTransactionId();
+ if(!FullTransactionIdIsValid(GetTopFullTransactionIdIfAny()))
+ {
+ (void) GetTopTransactionId();
+ consumed++;
+ }
If we count this case as consumed too, I think we need to set the
returned value of GetTopTranasctionId() to lastxid. Because otherwise,
the return value when passing 1 to the function would be the latest
XID at the time but not the last XID consumed by the function. Passing
1 to this function is very unrealistic case, though.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Fix for consume_xids advancing XIDs incorrectly
@ 2024-10-16 05:06 Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 19+ messages in thread
From: Yushi Ogiwara @ 2024-10-16 05:06 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: pgsql-hackers
Hi,
Thank you for your comment.
Regarding the first patch, I believe it works correctly when
consume_xids(1) is called. This is because the lastxid variable in the
consume_xids_common function is initialized as lastxid =
ReadNextFullTransactionId(), where the ReadNextFullTransactionId
function returns the (current XID) + 1.
Separately, I found that consume_xids(0) does not behave as expected.
Below is an example:
postgres=# select txid_current();
txid_current
--------------
45496
(1 row)
postgres=# select consume_xids(0);
consume_xids
--------------
45497
(1 row)
postgres=# select consume_xids(0);
consume_xids
--------------
45497
(1 row)
In the example, the argument to consume_xids is 0, meaning it should not
consume any XIDs. However, the first invocation of consume_xids(0) looks
like unexpectedly consuming 1 XID though it's not consuming actually.
This happens because consume_xids(0) returns the value from
ReadNextFullTransactionId.
I have updated the patch (skip.diff, attached to this e-mail) to address
this issue. Now, when consume_xids(0) is called, it returns
ReadNextFullTransactionId().value - 1, ensuring no XID is consumed as
shown below:
postgres=# select txid_current();
txid_current
--------------
45498
(1 row)
postgres=# select consume_xids(0);
consume_xids
--------------
45498
(1 row)
Regards,
Yushi
On 2024-10-16 07:26, Masahiko Sawada wrote:
> Hi,
>
> Thank you for the report.
>
> On Mon, Oct 14, 2024 at 6:51 PM Yushi Ogiwara
> <btogiwarayuushi@oss.nttdata.com> wrote:
>>
>> Hi,
>>
>> I found that the consume_xids function incorrectly advances XIDs as
>> shown:
>>
>> postgres=# select txid_current();
>> txid_current
>> --------------
>> 746
>> (1 row)
>>
>> postgres=# select consume_xids('100');
>> consume_xids
>> --------------
>> 847
>> (1 row)
>>
>> In the example, the consume_xids function consumes 100 XIDs when XID =
>> 746, so the desired outcome from consume_xids should be 746 + 100 =
>> 846,
>> which differs from the actual outcome, 847.
>>
>> Behavior inside a transaction block:
>>
>> postgres=# select txid_current();
>> txid_current
>> --------------
>> 1410
>> (1 row)
>>
>> postgres=# begin;
>> BEGIN
>> postgres=*# select consume_xids('100');
>> consume_xids
>> --------------
>> 1511
>> (1 row)
>> postgres=*# select consume_xids('100');
>> consume_xids
>> --------------
>> 1521
>> (1 row)
>>
>> Here, the first call inside the transaction block consumes 100+1 XIDs
>> (incorrect), while the second call consumes exactly 100 XIDs (as
>> expected)
>>
>> Summary:
>>
>> The function performs incorrectly when:
>> - Outside of a transaction block
>> - The first call inside a transaction block
>> But works correctly when:
>> - After the second call inside a transaction block
>>
>> The issue arises because consume_xids does not correctly count the
>> consumed XIDs when it calls the GetTopTransactionId function, which
>> allocates a new XID when none has been assigned.
>
> I agree with your analysis.
>
> I have one comment on the patch:
>
> - (void) GetTopTransactionId();
> + if(!FullTransactionIdIsValid(GetTopFullTransactionIdIfAny()))
> + {
> + (void) GetTopTransactionId();
> + consumed++;
> + }
>
> If we count this case as consumed too, I think we need to set the
> returned value of GetTopTranasctionId() to lastxid. Because otherwise,
> the return value when passing 1 to the function would be the latest
> XID at the time but not the last XID consumed by the function. Passing
> 1 to this function is very unrealistic case, though.
>
> Regards,
Attachments:
skip.diff (text/x-diff, 946B) download
diff --git a/src/test/modules/xid_wraparound/xid_wraparound.c b/src/test/modules/xid_wraparound/xid_wraparound.c
index dce81c0c6d..4b9788d2d9 100644
--- a/src/test/modules/xid_wraparound/xid_wraparound.c
+++ b/src/test/modules/xid_wraparound/xid_wraparound.c
@@ -38,7 +38,7 @@ consume_xids(PG_FUNCTION_ARGS)
elog(ERROR, "invalid nxids argument: %lld", (long long) nxids);
if (nxids == 0)
- lastxid = ReadNextFullTransactionId();
+ lastxid = FullTransactionIdFromU64(ReadNextFullTransactionId().value - 1);
else
lastxid = consume_xids_common(InvalidFullTransactionId, (uint64) nxids);
@@ -89,7 +89,11 @@ consume_xids_common(FullTransactionId untilxid, uint64 nxids)
* the cache overflows, but beyond that, we don't keep track of the
* consumed XIDs.
*/
- (void) GetTopTransactionId();
+ if(!FullTransactionIdIsValid(GetTopFullTransactionIdIfAny()))
+ {
+ (void) GetTopTransactionId();
+ consumed++;
+ }
for (;;)
{
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Fix for consume_xids advancing XIDs incorrectly
@ 2024-10-16 17:58 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>
0 siblings, 1 reply; 19+ messages in thread
From: Masahiko Sawada @ 2024-10-16 17:58 UTC (permalink / raw)
To: Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>; +Cc: pgsql-hackers
On Tue, Oct 15, 2024 at 10:06 PM Yushi Ogiwara
<btogiwarayuushi@oss.nttdata.com> wrote:
>
> Hi,
>
> Thank you for your comment.
>
> Regarding the first patch, I believe it works correctly when
> consume_xids(1) is called. This is because the lastxid variable in the
> consume_xids_common function is initialized as lastxid =
> ReadNextFullTransactionId(), where the ReadNextFullTransactionId
> function returns the (current XID) + 1.
But it's possible that concurrent transactions consume XIDs in meanwhile, no?
>
> Separately, I found that consume_xids(0) does not behave as expected.
> Below is an example:
>
> postgres=# select txid_current();
> txid_current
> --------------
> 45496
> (1 row)
>
> postgres=# select consume_xids(0);
> consume_xids
> --------------
> 45497
> (1 row)
>
> postgres=# select consume_xids(0);
> consume_xids
> --------------
> 45497
> (1 row)
>
> In the example, the argument to consume_xids is 0, meaning it should not
> consume any XIDs. However, the first invocation of consume_xids(0) looks
> like unexpectedly consuming 1 XID though it's not consuming actually.
> This happens because consume_xids(0) returns the value from
> ReadNextFullTransactionId.
>
> I have updated the patch (skip.diff, attached to this e-mail) to address
> this issue. Now, when consume_xids(0) is called, it returns
> ReadNextFullTransactionId().value - 1, ensuring no XID is consumed as
> shown below:
>
> postgres=# select txid_current();
> txid_current
> --------------
> 45498
> (1 row)
>
> postgres=# select consume_xids(0);
> consume_xids
> --------------
> 45498
> (1 row)
Hmm, I think if we expect this function to return the last XID that
the function actually consumed, calling consume_xids with 0 should
raise an error instead. Even if it returns
ReadNextFullTransactionId().value - 1 as you proposed, other
concurrent transactions might consume XIDs between txid_current() and
consume_xids(0), resulting in consume_xids() appearing to have
consumed XIDs.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Fix for consume_xids advancing XIDs incorrectly
@ 2024-10-18 05:57 Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 19+ messages in thread
From: Yushi Ogiwara @ 2024-10-18 05:57 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: pgsql-hackers
Hi,
Thank you for your suggestion. Let me organize the discussion as
follows:
When I created the patch, I did not account for situations where other
transactions concurrently consume XIDs. If we establish the rule that
"consume_xids should not return an XID that is consumed by any other
transaction," I believe it will simplify the discussion.
Case: consume_xids(1)
Let's consider a scenario where transaction 1 (t1) and transaction 2
(t2) run concurrently. t1 calls txid_current() and consume_xids(1),
while t2 calls txid_current().
The schedule above illustates when t1 and t2 run concurrently. The "next
XID" column shows how the TransamVariables->nextXid global variable
changes over time. The /**/ blocks shows the value of each variable and
outcome of a called function.
In my previous patch, the schedule above could occur, violating the rule
because consume_xids does not update lastxid when GetTopTransactionId is
called.
If your suggestion is applied, consume_xids would no longer violate the
rule, as it would update lastxid when generating a new XID.
Case: consume_xids(0)
Now, consider the case where transaction 1 consumes 0 XIDs.
In this example, both t1 and t2 consume XID = 20, which violates the
rule.
Using ReadNextFullTransactionId() - 1 also does not resolve the issue,
as demonstrated above. This is because consume_xids(0) does not allocate
any XIDs.
In conclusion, I agree that:
* Update lastxid with GetTopTransactionId().
* consume_xids with 0 should raise an error.
I have attached a new patch that incorporates your suggestions.
Regards,
Yushi
On 2024-10-17 02:58, Masahiko Sawada wrote:
> On Tue, Oct 15, 2024 at 10:06 PM Yushi Ogiwara
> <btogiwarayuushi@oss.nttdata.com> wrote:
>
>> Hi,
>>
>> Thank you for your comment.
>>
>> Regarding the first patch, I believe it works correctly when
>> consume_xids(1) is called. This is because the lastxid variable in
>> the
>> consume_xids_common function is initialized as lastxid =
>> ReadNextFullTransactionId(), where the ReadNextFullTransactionId
>> function returns the (current XID) + 1.
>
> But it's possible that concurrent transactions consume XIDs in
> meanwhile, no?
>
>> Separately, I found that consume_xids(0) does not behave as expected.
>> Below is an example:
>>
>> postgres=# select txid_current();
>> txid_current
>> --------------
>> 45496
>> (1 row)
>>
>> postgres=# select consume_xids(0);
>> consume_xids
>> --------------
>> 45497
>> (1 row)
>>
>> postgres=# select consume_xids(0);
>> consume_xids
>> --------------
>> 45497
>> (1 row)
>>
>> In the example, the argument to consume_xids is 0, meaning it should
>> not
>> consume any XIDs. However, the first invocation of consume_xids(0)
>> looks
>> like unexpectedly consuming 1 XID though it's not consuming actually.
>> This happens because consume_xids(0) returns the value from
>> ReadNextFullTransactionId.
>>
>> I have updated the patch (skip.diff, attached to this e-mail) to
>> address
>> this issue. Now, when consume_xids(0) is called, it returns
>> ReadNextFullTransactionId().value - 1, ensuring no XID is consumed as
>> shown below:
>>
>> postgres=# select txid_current();
>> txid_current
>> --------------
>> 45498
>> (1 row)
>>
>> postgres=# select consume_xids(0);
>> consume_xids
>> --------------
>> 45498
>> (1 row)
>
> Hmm, I think if we expect this function to return the last XID that
> the function actually consumed, calling consume_xids with 0 should
> raise an error instead. Even if it returns
> ReadNextFullTransactionId().value - 1 as you proposed, other
> concurrent transactions might consume XIDs between txid_current() and
> consume_xids(0), resulting in consume_xids() appearing to have
> consumed XIDs.
>
> Regards,
Attachments:
6f127201.png (image/png, 300.0K) download
view image
165e99ed.png (image/png, 148.5K) download
view image
39875719.png (image/png, 120.4K) download
view image
a99f3a2c.png (image/png, 117.0K) download
view image
consume_xid_fix.diff (text/x-diff, 975B) download
diff --git a/src/test/modules/xid_wraparound/xid_wraparound.c b/src/test/modules/xid_wraparound/xid_wraparound.c
index dce81c0c6d..70a748c41d 100644
--- a/src/test/modules/xid_wraparound/xid_wraparound.c
+++ b/src/test/modules/xid_wraparound/xid_wraparound.c
@@ -34,11 +34,8 @@ consume_xids(PG_FUNCTION_ARGS)
int64 nxids = PG_GETARG_INT64(0);
FullTransactionId lastxid;
- if (nxids < 0)
+ if (nxids <= 0)
elog(ERROR, "invalid nxids argument: %lld", (long long) nxids);
-
- if (nxids == 0)
- lastxid = ReadNextFullTransactionId();
else
lastxid = consume_xids_common(InvalidFullTransactionId, (uint64) nxids);
@@ -89,7 +86,11 @@ consume_xids_common(FullTransactionId untilxid, uint64 nxids)
* the cache overflows, but beyond that, we don't keep track of the
* consumed XIDs.
*/
- (void) GetTopTransactionId();
+ if(!FullTransactionIdIsValid(GetTopFullTransactionIdIfAny()))
+ {
+ lastxid = GetTopTransactionId();
+ consumed++;
+ }
for (;;)
{
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Fix for consume_xids advancing XIDs incorrectly
@ 2024-10-23 06:47 Fujii Masao <masao.fujii@oss.nttdata.com>
parent: Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>
0 siblings, 1 reply; 19+ messages in thread
From: Fujii Masao @ 2024-10-23 06:47 UTC (permalink / raw)
To: Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>; Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: pgsql-hackers
On 2024/10/18 14:57, Yushi Ogiwara wrote:
> In conclusion, I agree that:
>
> * Update lastxid with GetTopTransactionId().
> * consume_xids with 0 should raise an error.
>
> I have attached a new patch that incorporates your suggestions.
One concern in this discussion is that the return value of this function isn't
entirely clear. To address this, how about adding a comment at the beginning of
consume_xids() like: "Returns the last XID assigned by consume_xids()"?
* the cache overflows, but beyond that, we don't keep track of the
* consumed XIDs.
*/
- (void) GetTopTransactionId();
+ if(!FullTransactionIdIsValid(GetTopFullTransactionIdIfAny()))
+ {
+ lastxid = GetTopTransactionId();
+ consumed++;
+ }
How about appending the following to the end of the first paragraph in
the above source comments?
If no top-level XID is assigned, a new one is obtained,
and the consumed XID counter is incremented.
if (xids_left > 2000 &&
consumed - last_reported_at < REPORT_INTERVAL &&
MyProc->subxidStatus.overflowed)
{
int64 consumed_by_shortcut = consume_xids_shortcut();
if (consumed_by_shortcut > 0)
{
consumed += consumed_by_shortcut;
continue;
}
}
By the way, this isn't directly related to the proposed patch, but while reading
the xid_wraparound code, I noticed that consume_xids_common() could potentially
return an unexpected XID if consume_xids_shortcut() returns a value greater
than 2000. Based on the current logic, consume_xids_common() should always return
a value less than 2000, so the issue I'm concerned about shouldn't occur. Still,
would it be worth adding an assertion to ensure that consume_xids_common() never
returns a value greater than 2000?
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Fix for consume_xids advancing XIDs incorrectly
@ 2024-10-23 20:23 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Fujii Masao <masao.fujii@oss.nttdata.com>
0 siblings, 1 reply; 19+ messages in thread
From: Masahiko Sawada @ 2024-10-23 20:23 UTC (permalink / raw)
To: Fujii Masao <masao.fujii@oss.nttdata.com>; +Cc: Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>; pgsql-hackers
On Tue, Oct 22, 2024 at 11:47 PM Fujii Masao
<masao.fujii@oss.nttdata.com> wrote:
>
>
>
> On 2024/10/18 14:57, Yushi Ogiwara wrote:
> > In conclusion, I agree that:
> >
> > * Update lastxid with GetTopTransactionId().
> > * consume_xids with 0 should raise an error.
> >
> > I have attached a new patch that incorporates your suggestions.
>
> One concern in this discussion is that the return value of this function isn't
> entirely clear. To address this, how about adding a comment at the beginning of
> consume_xids() like: "Returns the last XID assigned by consume_xids()"?
Agreed. That value is what I expected this function to return.
>
>
> * the cache overflows, but beyond that, we don't keep track of the
> * consumed XIDs.
> */
> - (void) GetTopTransactionId();
> + if(!FullTransactionIdIsValid(GetTopFullTransactionIdIfAny()))
> + {
> + lastxid = GetTopTransactionId();
> + consumed++;
> + }
>
> How about appending the following to the end of the first paragraph in
> the above source comments?
>
> If no top-level XID is assigned, a new one is obtained,
> and the consumed XID counter is incremented.
Agreed.
>
>
>
> if (xids_left > 2000 &&
> consumed - last_reported_at < REPORT_INTERVAL &&
> MyProc->subxidStatus.overflowed)
> {
> int64 consumed_by_shortcut = consume_xids_shortcut();
>
> if (consumed_by_shortcut > 0)
> {
> consumed += consumed_by_shortcut;
> continue;
> }
> }
>
> By the way, this isn't directly related to the proposed patch, but while reading
> the xid_wraparound code, I noticed that consume_xids_common() could potentially
> return an unexpected XID if consume_xids_shortcut() returns a value greater
> than 2000. Based on the current logic, consume_xids_common() should always return
> a value less than 2000, so the issue I'm concerned about shouldn't occur.
Good point. Yes, the function doesn't return a value greater than 2000
as long as we do "distance = Min(distance, COMMIT_TS_XACTS_PER_PAGE -
rem);". But it's true with <= 16KB block sizes.
> Still,
> would it be worth adding an assertion to ensure that consume_xids_common() never
> returns a value greater than 2000?
While adding an assertion makes sense to me, another idea is to set
last_xid even in the shortcut path. That way, it works even with 32KB
block size.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Fix for consume_xids advancing XIDs incorrectly
@ 2024-10-24 14:55 Fujii Masao <masao.fujii@oss.nttdata.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 19+ messages in thread
From: Fujii Masao @ 2024-10-24 14:55 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>; pgsql-hackers
On 2024/10/24 5:23, Masahiko Sawada wrote:
>> if (xids_left > 2000 &&
>> consumed - last_reported_at < REPORT_INTERVAL &&
>> MyProc->subxidStatus.overflowed)
>> {
>> int64 consumed_by_shortcut = consume_xids_shortcut();
>>
>> if (consumed_by_shortcut > 0)
>> {
>> consumed += consumed_by_shortcut;
>> continue;
>> }
>> }
>>
>> By the way, this isn't directly related to the proposed patch, but while reading
>> the xid_wraparound code, I noticed that consume_xids_common() could potentially
>> return an unexpected XID if consume_xids_shortcut() returns a value greater
>> than 2000. Based on the current logic, consume_xids_common() should always return
>> a value less than 2000, so the issue I'm concerned about shouldn't occur.
>
> Good point. Yes, the function doesn't return a value greater than 2000
> as long as we do "distance = Min(distance, COMMIT_TS_XACTS_PER_PAGE -
> rem);". But it's true with <= 16KB block sizes.
>
>> Still,
>> would it be worth adding an assertion to ensure that consume_xids_common() never
>> returns a value greater than 2000?
>
> While adding an assertion makes sense to me, another idea is to set
> last_xid even in the shortcut path. That way, it works even with 32KB
> block size.
Agreed on making xid_wraparound compatible with a 32k block size. As you pointed out,
with 32k blocks, XidSkip() can return values greater than 2000. So, the first step is
to adjust the following if-condition by increasing "2000" to a higher value.
Since XidSkip() can return up to 3276 (based on COMMIT_TS_XACTS_PER_PAGE (BLCKSZ / 10) with 32k blocks),
we could, for instance, update "2000" to "4000."
if (xids_left > 2000 &&
consumed - last_reported_at < REPORT_INTERVAL &&
MyProc->subxidStatus.overflowed)
To ensure lastxid is set even in the shortcut case, what about modifying XidSkip()
so it returns "distance - 1" instead of "distance"? This change would make
consume_xids_common() run at least one more loop cycle,
triggering GetNewTransactionId() and setting lastxid correctly.
consumed = XidSkip(nextXid);
if (consumed > 0)
TransamVariables->nextXid.value += (uint64) consumed;
BTW, the code snippet above in consume_xids_shortcut() could potentially set
the next XID to a value below FirstNormalTransactionId? If yes, we should account for
FirstNormalFullTransactionId when increasing the next XID, similar to
how FullTransactionIdAdvance() handles it.
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Fix for consume_xids advancing XIDs incorrectly
@ 2024-10-25 21:31 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Fujii Masao <masao.fujii@oss.nttdata.com>
0 siblings, 1 reply; 19+ messages in thread
From: Masahiko Sawada @ 2024-10-25 21:31 UTC (permalink / raw)
To: Fujii Masao <masao.fujii@oss.nttdata.com>; +Cc: Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>; pgsql-hackers
On Thu, Oct 24, 2024 at 7:55 AM Fujii Masao <masao.fujii@oss.nttdata.com> wrote:
>
>
>
> On 2024/10/24 5:23, Masahiko Sawada wrote:
> >> if (xids_left > 2000 &&
> >> consumed - last_reported_at < REPORT_INTERVAL &&
> >> MyProc->subxidStatus.overflowed)
> >> {
> >> int64 consumed_by_shortcut = consume_xids_shortcut();
> >>
> >> if (consumed_by_shortcut > 0)
> >> {
> >> consumed += consumed_by_shortcut;
> >> continue;
> >> }
> >> }
> >>
> >> By the way, this isn't directly related to the proposed patch, but while reading
> >> the xid_wraparound code, I noticed that consume_xids_common() could potentially
> >> return an unexpected XID if consume_xids_shortcut() returns a value greater
> >> than 2000. Based on the current logic, consume_xids_common() should always return
> >> a value less than 2000, so the issue I'm concerned about shouldn't occur.
> >
> > Good point. Yes, the function doesn't return a value greater than 2000
> > as long as we do "distance = Min(distance, COMMIT_TS_XACTS_PER_PAGE -
> > rem);". But it's true with <= 16KB block sizes.
> >
> >> Still,
> >> would it be worth adding an assertion to ensure that consume_xids_common() never
> >> returns a value greater than 2000?
> >
> > While adding an assertion makes sense to me, another idea is to set
> > last_xid even in the shortcut path. That way, it works even with 32KB
> > block size.
>
> Agreed on making xid_wraparound compatible with a 32k block size. As you pointed out,
> with 32k blocks, XidSkip() can return values greater than 2000. So, the first step is
> to adjust the following if-condition by increasing "2000" to a higher value.
> Since XidSkip() can return up to 3276 (based on COMMIT_TS_XACTS_PER_PAGE (BLCKSZ / 10) with 32k blocks),
> we could, for instance, update "2000" to "4000."
>
> if (xids_left > 2000 &&
> consumed - last_reported_at < REPORT_INTERVAL &&
> MyProc->subxidStatus.overflowed)
>
>
> To ensure lastxid is set even in the shortcut case, what about modifying XidSkip()
> so it returns "distance - 1" instead of "distance"? This change would make
> consume_xids_common() run at least one more loop cycle,
> triggering GetNewTransactionId() and setting lastxid correctly.
Increasing "2000" to "4000" makes sense to me. I think that with this
change we don't necessarily need to change XidSkip() to return
"distance - 1'. What do you think?
>
>
> consumed = XidSkip(nextXid);
> if (consumed > 0)
> TransamVariables->nextXid.value += (uint64) consumed;
>
> BTW, the code snippet above in consume_xids_shortcut() could potentially set
> the next XID to a value below FirstNormalTransactionId? If yes, we should account for
> FirstNormalFullTransactionId when increasing the next XID, similar to
> how FullTransactionIdAdvance() handles it.
Good catch. I agree with you. We need to do something similar to what
FullTransactionIdAdvance() does so that it does not appear as a
special 32-bit XID.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Fix for consume_xids advancing XIDs incorrectly
@ 2024-10-29 09:00 Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 19+ messages in thread
From: Yushi Ogiwara @ 2024-10-29 09:00 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Fujii Masao <masao.fujii@oss.nttdata.com>; pgsql-hackers
>>
>>
>> consumed = XidSkip(nextXid);
>> if (consumed > 0)
>> TransamVariables->nextXid.value += (uint64) consumed;
>>
>> BTW, the code snippet above in consume_xids_shortcut() could
>> potentially set
>> the next XID to a value below FirstNormalTransactionId? If yes, we
>> should account for
>> FirstNormalFullTransactionId when increasing the next XID, similar to
>> how FullTransactionIdAdvance() handles it.
>
> Good catch. I agree with you. We need to do something similar to what
> FullTransactionIdAdvance() does so that it does not appear as a
> special 32-bit XID.
>
> Regards,
I think this is not the case since XidSkip returns min(UINT32_MAX - 5 -
low, *), which prevents the wrap-around of nextXid.
Regards,
Yushi Ogiwara
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Fix for consume_xids advancing XIDs incorrectly
@ 2024-10-29 18:09 Fujii Masao <masao.fujii@oss.nttdata.com>
parent: Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>
0 siblings, 1 reply; 19+ messages in thread
From: Fujii Masao @ 2024-10-29 18:09 UTC (permalink / raw)
To: Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>; Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: pgsql-hackers
On 2024/10/29 18:00, Yushi Ogiwara wrote:
>>>
>>>
>>> consumed = XidSkip(nextXid);
>>> if (consumed > 0)
>>> TransamVariables->nextXid.value += (uint64) consumed;
>>>
>>> BTW, the code snippet above in consume_xids_shortcut() could potentially set
>>> the next XID to a value below FirstNormalTransactionId? If yes, we should account for
>>> FirstNormalFullTransactionId when increasing the next XID, similar to
>>> how FullTransactionIdAdvance() handles it.
>>
>> Good catch. I agree with you. We need to do something similar to what
>> FullTransactionIdAdvance() does so that it does not appear as a
>> special 32-bit XID.
>>
>> Regards,
>
> I think this is not the case since XidSkip returns min(UINT32_MAX - 5 - low, *), which prevents the wrap-around of nextXid.
Yes, you're right. Thanks for the correction!
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Fix for consume_xids advancing XIDs incorrectly
@ 2024-10-30 07:00 Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>
parent: Fujii Masao <masao.fujii@oss.nttdata.com>
0 siblings, 2 replies; 19+ messages in thread
From: Yushi Ogiwara @ 2024-10-30 07:00 UTC (permalink / raw)
To: Fujii Masao <masao.fujii@oss.nttdata.com>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; pgsql-hackers
I made a new patch (skip_xid_correctly.diff) that incorporates the
points we discussed:
1. Fix the issue that consume_xids consumes nxids+1 XIDs.
2. Update lastxid when calling GetTopFullTransactionId() to support
nxids==1 case.
3. Forbid consume_xids when nxids==0.
4. Add comments explaining the return values of consume_xids and
consume_xids_until, and the rationale for incrementing consumed++ when
GetTopFullTransactionId() is called.
Also, I made another patch (support_blksz_32k.diff) that supports the
block size == 32K case.
Best,
Yushi Ogiwara
Attachments: download all (2 files, .tar.gz)
skip_xid_correctly.diff (text/x-diff, 1.6K) download
diff --git a/src/test/modules/xid_wraparound/xid_wraparound.c b/src/test/modules/xid_wraparound/xid_wraparound.c
index dce81c0c6d..63f3305ee9 100644
--- a/src/test/modules/xid_wraparound/xid_wraparound.c
+++ b/src/test/modules/xid_wraparound/xid_wraparound.c
@@ -26,6 +26,7 @@ static FullTransactionId consume_xids_common(FullTransactionId untilxid, uint64
/*
* Consume the specified number of XIDs.
+ * Returns the last XID assigned by this function.
*/
PG_FUNCTION_INFO_V1(consume_xids);
Datum
@@ -34,11 +35,8 @@ consume_xids(PG_FUNCTION_ARGS)
int64 nxids = PG_GETARG_INT64(0);
FullTransactionId lastxid;
- if (nxids < 0)
+ if (nxids <= 0)
elog(ERROR, "invalid nxids argument: %lld", (long long) nxids);
-
- if (nxids == 0)
- lastxid = ReadNextFullTransactionId();
else
lastxid = consume_xids_common(InvalidFullTransactionId, (uint64) nxids);
@@ -47,6 +45,7 @@ consume_xids(PG_FUNCTION_ARGS)
/*
* Consume XIDs, up to the given XID.
+ * Returns the last XID assigned by this function.
*/
PG_FUNCTION_INFO_V1(consume_xids_until);
Datum
@@ -88,8 +87,15 @@ consume_xids_common(FullTransactionId untilxid, uint64 nxids)
* GetNewTransactionId registers them in the subxid cache in PGPROC, until
* the cache overflows, but beyond that, we don't keep track of the
* consumed XIDs.
+ *
+ * If no top-level XID is assigned, a new one is obtained,
+ * and the consumed XID counter is incremented.
*/
- (void) GetTopTransactionId();
+ if(!FullTransactionIdIsValid(GetTopFullTransactionIdIfAny()))
+ {
+ lastxid = GetTopFullTransactionId();
+ consumed++;
+ }
for (;;)
{
support_blksz_32k.diff (text/x-diff, 1.6K) download
diff --git a/src/test/modules/xid_wraparound/xid_wraparound.c b/src/test/modules/xid_wraparound/xid_wraparound.c
index dce81c0c6d..95832fbecc 100644
--- a/src/test/modules/xid_wraparound/xid_wraparound.c
+++ b/src/test/modules/xid_wraparound/xid_wraparound.c
@@ -64,6 +64,14 @@ consume_xids_until(PG_FUNCTION_ARGS)
PG_RETURN_FULLTRANSACTIONID(lastxid);
}
+/*
+ * These constants copied from .c files, because they're private.
+ */
+#define COMMIT_TS_XACTS_PER_PAGE (BLCKSZ / 10)
+#define SUBTRANS_XACTS_PER_PAGE (BLCKSZ / sizeof(TransactionId))
+#define CLOG_XACTS_PER_BYTE 4
+#define CLOG_XACTS_PER_PAGE (BLCKSZ * CLOG_XACTS_PER_BYTE)
+
/*
* Common functionality between the two public functions.
*/
@@ -115,7 +123,7 @@ consume_xids_common(FullTransactionId untilxid, uint64 nxids)
* If we still have plenty of XIDs to consume, try to take a shortcut
* and bump up the nextXid counter directly.
*/
- if (xids_left > 2000 &&
+ if (xids_left > COMMIT_TS_XACTS_PER_PAGE &&
consumed - last_reported_at < REPORT_INTERVAL &&
MyProc->subxidStatus.overflowed)
{
@@ -153,14 +161,6 @@ consume_xids_common(FullTransactionId untilxid, uint64 nxids)
return lastxid;
}
-/*
- * These constants copied from .c files, because they're private.
- */
-#define COMMIT_TS_XACTS_PER_PAGE (BLCKSZ / 10)
-#define SUBTRANS_XACTS_PER_PAGE (BLCKSZ / sizeof(TransactionId))
-#define CLOG_XACTS_PER_BYTE 4
-#define CLOG_XACTS_PER_PAGE (BLCKSZ * CLOG_XACTS_PER_BYTE)
-
/*
* All the interesting action in GetNewTransactionId happens when we extend
* the SLRUs, or at the uint32 wraparound. If the nextXid counter is not close
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Fix for consume_xids advancing XIDs incorrectly
@ 2024-10-31 18:53 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>
1 sibling, 1 reply; 19+ messages in thread
From: Masahiko Sawada @ 2024-10-31 18:53 UTC (permalink / raw)
To: Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>; +Cc: Fujii Masao <masao.fujii@oss.nttdata.com>; pgsql-hackers
On Wed, Oct 30, 2024 at 12:00 AM Yushi Ogiwara
<btogiwarayuushi@oss.nttdata.com> wrote:
>
> I made a new patch (skip_xid_correctly.diff) that incorporates the
> points we discussed:
>
> 1. Fix the issue that consume_xids consumes nxids+1 XIDs.
> 2. Update lastxid when calling GetTopFullTransactionId() to support
> nxids==1 case.
> 3. Forbid consume_xids when nxids==0.
> 4. Add comments explaining the return values of consume_xids and
> consume_xids_until, and the rationale for incrementing consumed++ when
> GetTopFullTransactionId() is called.
>
> Also, I made another patch (support_blksz_32k.diff) that supports the
> block size == 32K case.
>
Thank you for making patches! Here are review comments.
skip_xid_correctly.diff:
- if (nxids < 0)
+ if (nxids <= 0)
elog(ERROR, "invalid nxids argument: %lld", (long long) nxids);
-
- if (nxids == 0)
- lastxid = ReadNextFullTransactionId();
else
lastxid = consume_xids_common(InvalidFullTransactionId, (uint64) nxids);
After calling elog(ERROR) we don't return here, so the "else" is not
necessary. That is, we can rewrite it to:
if (nxids <= 0)
elog(ERROR, "invalid nxids argument: %lld", (long long) nxids);
lastxid = consume_xids_common(InvalidFullTransactionId, (uint64) nxids);
---
+ *
+ * If no top-level XID is assigned, a new one is obtained,
+ * and the consumed XID counter is incremented.
I'm not sure this comment is appropriate since what we do here is
obvious and the comment doesn't explain why we do that. IMO we don't
need to update these comments.
support_blksz_32k.diff:
- if (xids_left > 2000 &&
+ if (xids_left > COMMIT_TS_XACTS_PER_PAGE &&
I think it's better to just replace 2000 with 4000 and explain why
4000 is sufficient. Probably we can define '#define
XID_SHORTCUT_THRESHOLD 4000" and put an assertion to XidSkip() or
consume_xids_common() to check if the number of consumed XIDs doesn't
exceed XID_SHORTCUT_THRESHOLD.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Fix for consume_xids advancing XIDs incorrectly
@ 2024-12-03 09:28 Kirill Reshke <reshkekirill@gmail.com>
parent: Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>
1 sibling, 0 replies; 19+ messages in thread
From: Kirill Reshke @ 2024-12-03 09:28 UTC (permalink / raw)
To: Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>; +Cc: Fujii Masao <masao.fujii@oss.nttdata.com>; Masahiko Sawada <sawada.mshk@gmail.com>; pgsql-hackers
On Wed, 30 Oct 2024 at 12:01, Yushi Ogiwara
<btogiwarayuushi@oss.nttdata.com> wrote:
>
> I made a new patch (skip_xid_correctly.diff) that incorporates the
> points we discussed:
>
> 1. Fix the issue that consume_xids consumes nxids+1 XIDs.
> 2. Update lastxid when calling GetTopFullTransactionId() to support
> nxids==1 case.
> 3. Forbid consume_xids when nxids==0.
> 4. Add comments explaining the return values of consume_xids and
> consume_xids_until, and the rationale for incrementing consumed++ when
> GetTopFullTransactionId() is called.
>
> Also, I made another patch (support_blksz_32k.diff) that supports the
> block size == 32K case.
>
> Best,
> Yushi Ogiwara
>
Hi!
There is review comments that need to be addressed in [1]
Patch status now is waiting on author
[1] https://www.postgresql.org/message-id/CAD21AoCthHcSQ5zeeivNpiz7HMi_FPG-dtwDDNYUx2oKG36bCQ%40mail.gma...
--
Best regards,
Kirill Reshke
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Fix for consume_xids advancing XIDs incorrectly
@ 2026-01-08 01:41 Michael Paquier <michael@paquier.xyz>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 0 replies; 19+ messages in thread
From: Michael Paquier @ 2026-01-08 01:41 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>; Fujii Masao <masao.fujii@oss.nttdata.com>; pgsql-hackers
On Thu, Oct 31, 2024 at 11:53:34AM -0700, Masahiko Sawada wrote:
> I think it's better to just replace 2000 with 4000 and explain why
> 4000 is sufficient. Probably we can define '#define
> XID_SHORTCUT_THRESHOLD 4000" and put an assertion to XidSkip() or
> consume_xids_common() to check if the number of consumed XIDs doesn't
> exceed XID_SHORTCUT_THRESHOLD.
The patch has not been updated in some time, for what I am tracking is
still a bug. Sawada-san, Ogiwara-san does not seem to be around,
could it be possible for you to step in and fix the problems of this
thread? It's been more than one year since the last update on this
thread.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Fix for consume_xids advancing XIDs incorrectly
@ 2026-11-10 22:52 João Rito <joaovictor.rito@gmail.com>
parent: Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>
1 sibling, 1 reply; 19+ messages in thread
From: João Rito @ 2026-11-10 22:52 UTC (permalink / raw)
To: Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>; +Cc: pgsql-hackers; sawada.mshk@gmail.com; masao.fujii@oss.nttdata.com; michael@paquier.xyz
Hi,
Following Michael's request in January, I picked this one up since the
thread has been idle for a while. I rebased the work onto current master
and addressed the review comments Sawada-san posted on 2024-10-31. Thank
you to Yushi Ogiwara for the original patch, and to Sawada-san and
Fujii-san for the review; I kept Yushi as the first author in the commit
message.
The attached v2 is a single patch (it merges skip_xid_correctly.diff and
support_blksz_32k.diff). Compared with the previous versions:
1. The XID assigned by GetTopFullTransactionId() is counted in "consumed"
and stored in "lastxid", so consume_xids(n) now consumes exactly n XIDs
and consume_xids(1) returns the XID it actually assigned.
2. consume_xids(0) raises an error, and the unnecessary "else" after the
elog(ERROR) is gone, as Sawada-san suggested.
3. I dropped the extra comment about the consumed counter, as suggested.
I kept the "Returns the last XID assigned by this function" comment on
consume_xids() and consume_xids_until(), as Fujii-san asked.
4. The hard-coded 2000 is replaced by XID_SHORTCUT_THRESHOLD (4000), with a
comment explaining why that is enough (XidSkip() returns at most
COMMIT_TS_XACTS_PER_PAGE, which is 3276 with 32kB blocks), and an
Assert() in XidSkip() checks that. The *_XACTS_PER_PAGE constants moved
above their first use.
Testing: I built with --enable-cassert --enable-debug --enable-tap-tests
(Debian bookworm, no compiler warnings). Using txid_current() followed by
consume_xids(n) in autocommit mode, unpatched master returned one more than
expected for every n I tried (1, 2, 100, 3999, 4000, 4001, 5000, 10000,
100000), for example txid_current() = 672 and consume_xids(100) = 773. With
the patch, the returned value is exactly txid_current() + n in all cases.
Inside a transaction block, the first and second consume_xids(5000) calls
now each advance by exactly 5000 (127777 -> 132777 -> 137777).
consume_xids(0) now fails with "invalid nxids argument: 0", and
consume_xids_until() still returns its target. I did not test with a 32kB
block size. "PG_TEST_EXTRA=xid_wraparound make -C
src/test/modules/xid_wraparound check" passes (4 files, 23 tests). The
patch applies on commit 952bab8b74 and was tested at that commit plus the patch
(the unpatched numbers above came from an older master where the module and
varsup.c are identical).
Comments are welcome.
Regards,
João Rito
Em ter., 10 de nov. de 2026 às 19:45, Yushi Ogiwara
<btogiwarayuushi@oss.nttdata.com> escreveu:
>
> Hi,
>
> I found that the consume_xids function incorrectly advances XIDs as
> shown:
>
> postgres=# select txid_current();
> txid_current
> --------------
> 746
> (1 row)
>
> postgres=# select consume_xids('100');
> consume_xids
> --------------
> 847
> (1 row)
>
> In the example, the consume_xids function consumes 100 XIDs when XID =
> 746, so the desired outcome from consume_xids should be 746 + 100 = 846,
> which differs from the actual outcome, 847.
>
> Behavior inside a transaction block:
>
> postgres=# select txid_current();
> txid_current
> --------------
> 1410
> (1 row)
>
> postgres=# begin;
> BEGIN
> postgres=*# select consume_xids('100');
> consume_xids
> --------------
> 1511
> (1 row)
> postgres=*# select consume_xids('100');
> consume_xids
> --------------
> 1521
> (1 row)
>
> Here, the first call inside the transaction block consumes 100+1 XIDs
> (incorrect), while the second call consumes exactly 100 XIDs (as
> expected)
>
> Summary:
>
> The function performs incorrectly when:
> - Outside of a transaction block
> - The first call inside a transaction block
> But works correctly when:
> - After the second call inside a transaction block
>
> The issue arises because consume_xids does not correctly count the
> consumed XIDs when it calls the GetTopTransactionId function, which
> allocates a new XID when none has been assigned.
>
> With the attached patch, the consume_xids works as expected, both inside
> and outside of transaction blocks as shown below:
>
> postgres=# select txid_current();
> txid_current
> --------------
> 1546
> (1 row)
>
> postgres=# select consume_xids('100');
> consume_xids
> --------------
> 1646
> (1 row)
>
> postgres=# begin;
> BEGIN
> postgres=*# select consume_xids('100');
> consume_xids
> --------------
> 1746
> (1 row)
>
> postgres=*# select consume_xids('100');
> consume_xids
> --------------
> 1846
> (1 row)
>
> Regards,
> Yushi
Attachments:
v2-0001-xid_wraparound-Fix-consume_xids-consuming-one-XID.patch (application/octet-stream, 5.0K) download
From 37f3cf2906308b00b31367cbaef90b7fc23759e6 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jo=C3=A3o=20Rito?= <ritojoao@users.noreply.github.com>
Date: Tue, 10 Nov 2026 17:31:18 -0300
Subject: [PATCH v2] xid_wraparound: Fix consume_xids() consuming one XID too
many
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
consume_xids() and consume_xids_until() called GetTopTransactionId()
without counting the XID that this may assign, so when the transaction
had no XID yet, they consumed one more XID than requested and returned
a XID that was not the last one they assigned.
Count the XID obtained here and remember it in lastxid, so that the
result is correct even for consume_xids(1). Since consume_xids(0)
cannot return an XID assigned by the function, make it raise an error
instead of returning the next XID.
Also, replace the hard-coded shortcut threshold of 2000 with
XID_SHORTCUT_THRESHOLD, set to 4000 so that the shortcut cannot skip
more XIDs than requested even with 32kB blocks, where XidSkip() can
return up to 3276. Add an assertion to XidSkip() to verify this.
Document what the two functions return.
Author: Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>
Author: João Rito <ritojoao@users.noreply.github.com>
Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@oss.nttdata.com>
Discussion: https://postgr.es/m/2a62f138c9dcb6fcf5889b076d123aa2@oss.nttdata.com
---
.../modules/xid_wraparound/xid_wraparound.c | 47 +++++++++++++------
1 file changed, 32 insertions(+), 15 deletions(-)
diff --git a/src/test/modules/xid_wraparound/xid_wraparound.c b/src/test/modules/xid_wraparound/xid_wraparound.c
index ca25d7e020..3de3ff472e 100644
--- a/src/test/modules/xid_wraparound/xid_wraparound.c
+++ b/src/test/modules/xid_wraparound/xid_wraparound.c
@@ -26,6 +26,8 @@ static FullTransactionId consume_xids_common(FullTransactionId untilxid, uint64
/*
* Consume the specified number of XIDs.
+ *
+ * Returns the last XID assigned by this function.
*/
PG_FUNCTION_INFO_V1(consume_xids);
Datum
@@ -34,19 +36,18 @@ consume_xids(PG_FUNCTION_ARGS)
int64 nxids = PG_GETARG_INT64(0);
FullTransactionId lastxid;
- if (nxids < 0)
+ if (nxids <= 0)
elog(ERROR, "invalid nxids argument: %" PRId64, nxids);
- if (nxids == 0)
- lastxid = ReadNextFullTransactionId();
- else
- lastxid = consume_xids_common(InvalidFullTransactionId, (uint64) nxids);
+ lastxid = consume_xids_common(InvalidFullTransactionId, (uint64) nxids);
PG_RETURN_FULLTRANSACTIONID(lastxid);
}
/*
* Consume XIDs, up to the given XID.
+ *
+ * Returns the last XID assigned by this function.
*/
PG_FUNCTION_INFO_V1(consume_xids_until);
Datum
@@ -64,6 +65,23 @@ consume_xids_until(PG_FUNCTION_ARGS)
PG_RETURN_FULLTRANSACTIONID(lastxid);
}
+/*
+ * These constants copied from .c files, because they're private.
+ */
+#define COMMIT_TS_XACTS_PER_PAGE (BLCKSZ / 10)
+#define SUBTRANS_XACTS_PER_PAGE (BLCKSZ / sizeof(TransactionId))
+#define CLOG_XACTS_PER_BYTE 4
+#define CLOG_XACTS_PER_PAGE (BLCKSZ * CLOG_XACTS_PER_BYTE)
+
+/*
+ * Take the shortcut in consume_xids_common() only if we have more than this
+ * many XIDs left to consume. XidSkip() never skips more than
+ * COMMIT_TS_XACTS_PER_PAGE XIDs, which is 3276 with the largest supported
+ * block size (32kB), so this is enough to make sure that the shortcut never
+ * consumes more XIDs than requested.
+ */
+#define XID_SHORTCUT_THRESHOLD 4000
+
/*
* Common functionality between the two public functions.
*/
@@ -89,7 +107,11 @@ consume_xids_common(FullTransactionId untilxid, uint64 nxids)
* the cache overflows, but beyond that, we don't keep track of the
* consumed XIDs.
*/
- (void) GetTopTransactionId();
+ if (!FullTransactionIdIsValid(GetTopFullTransactionIdIfAny()))
+ {
+ lastxid = GetTopFullTransactionId();
+ consumed++;
+ }
for (;;)
{
@@ -115,7 +137,7 @@ consume_xids_common(FullTransactionId untilxid, uint64 nxids)
* If we still have plenty of XIDs to consume, try to take a shortcut
* and bump up the nextXid counter directly.
*/
- if (xids_left > 2000 &&
+ if (xids_left > XID_SHORTCUT_THRESHOLD &&
consumed - last_reported_at < REPORT_INTERVAL &&
MyProc->subxidStatus.overflowed)
{
@@ -153,14 +175,6 @@ consume_xids_common(FullTransactionId untilxid, uint64 nxids)
return lastxid;
}
-/*
- * These constants copied from .c files, because they're private.
- */
-#define COMMIT_TS_XACTS_PER_PAGE (BLCKSZ / 10)
-#define SUBTRANS_XACTS_PER_PAGE (BLCKSZ / sizeof(TransactionId))
-#define CLOG_XACTS_PER_BYTE 4
-#define CLOG_XACTS_PER_PAGE (BLCKSZ * CLOG_XACTS_PER_BYTE)
-
/*
* All the interesting action in GetNewTransactionId happens when we extend
* the SLRUs, or at the uint32 wraparound. If the nextXid counter is not close
@@ -193,6 +207,9 @@ XidSkip(FullTransactionId fullxid)
return 0;
distance = Min(distance, CLOG_XACTS_PER_PAGE - rem);
+ /* The shortcut threshold in consume_xids_common() relies on this */
+ Assert(distance <= XID_SHORTCUT_THRESHOLD);
+
return distance;
}
--
2.53.0.windows.1
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Fix for consume_xids advancing XIDs incorrectly
@ 2026-11-11 08:52 Fujii Masao <masao.fujii@gmail.com>
parent: João Rito <joaovictor.rito@gmail.com>
0 siblings, 1 reply; 19+ messages in thread
From: Fujii Masao @ 2026-11-11 08:52 UTC (permalink / raw)
To: João Rito <joaovictor.rito@gmail.com>; +Cc: pgsql-hackers; sawada.mshk@gmail.com; michael@paquier.xyz
On Wed, Nov 11, 2026 at 7:53 AM João Rito <joaovictor.rito@gmail.com> wrote:
> Following Michael's request in January, I picked this one up since the
> thread has been idle for a while. I rebased the work onto current master
> and addressed the review comments Sawada-san posted on 2024-10-31.
Thanks for updating the patch! LGTM, except for one comment.
> I kept the "Returns the last XID assigned by this function" comment on
> consume_xids() and consume_xids_until(), as Fujii-san asked.
If the transaction already has a top-level XID and the next XID is
at least the target, consume_xids_until() returns that next XID
without assigning it. So isn't it better to update the comment for
consume_xids_until(), for example, as follows?
/*
* Advance XID allocation so that the next XID is at least the target.
* This may consume XIDs beyond the target.
*
* Ensures that the current transaction has a top-level XID.
* Returns the last XID assigned by this call, or the next XID observed
* if no XID was assigned.
*/
Regards,
--
Fujii Masao
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Fix for consume_xids advancing XIDs incorrectly
@ 2026-11-11 22:12 João Rito <joaovictor.rito@gmail.com>
parent: Fujii Masao <masao.fujii@gmail.com>
0 siblings, 1 reply; 19+ messages in thread
From: João Rito @ 2026-11-11 22:12 UTC (permalink / raw)
To: Fujii Masao <masao.fujii@gmail.com>; +Cc: pgsql-hackers; sawada.mshk@gmail.com; michael@paquier.xyz
Hi Fujii-san,
Thank you very much for the review and for the suggested comment.
You are right: if the transaction already has a top-level XID and the
next XID is already at least the target, consume_xids_until() returns
the next XID without assigning it, so the old comment was not
accurate.
The attached v3 updates the comment of consume_xids_until() as you
suggested. This is the only change from v2:
/*
* Advance XID allocation so that the next XID is at least the target.
* This may consume XIDs beyond the target.
*
* Ensures that the current transaction has a top-level XID.
* Returns the last XID assigned by this call, or the next XID observed
* if no XID was assigned.
*/
The code is unchanged. I applied v3 to current master (8d87367b39e),
built it with assertions, and compared it with the module from master,
each call in a new transaction:
- consume_xids(1), next XID 666: v3 returns 666. With the master
module (next XID 682) it returns 683, so two XIDs were consumed.
- consume_xids(10), next XID 668: v3 returns 677 (10 XIDs). Master
(next XID 685) returns 695 (11 XIDs).
- consume_xids(0): v3 raises "invalid nxids argument: 0"; master returns an XID.
- The case from your mail, consume_xids_until(pg_current_xact_id()) in
a transaction that already has XID 678: it returns 679, the next XID.
Master behaves the same, which matches the new comment.
I did not run the xid_wraparound TAP tests (PG_TEST_EXTRA) for this version.
Thanks,
João
Em qua., 11 de nov. de 2026 às 05:52, Fujii Masao
<masao.fujii@gmail.com> escreveu:
>
> On Wed, Nov 11, 2026 at 7:53 AM João Rito <joaovictor.rito@gmail.com> wrote:
> > Following Michael's request in January, I picked this one up since the
> > thread has been idle for a while. I rebased the work onto current master
> > and addressed the review comments Sawada-san posted on 2024-10-31.
>
> Thanks for updating the patch! LGTM, except for one comment.
>
> > I kept the "Returns the last XID assigned by this function" comment on
> > consume_xids() and consume_xids_until(), as Fujii-san asked.
>
> If the transaction already has a top-level XID and the next XID is
> at least the target, consume_xids_until() returns that next XID
> without assigning it. So isn't it better to update the comment for
> consume_xids_until(), for example, as follows?
>
> /*
> * Advance XID allocation so that the next XID is at least the target.
> * This may consume XIDs beyond the target.
> *
> * Ensures that the current transaction has a top-level XID.
> * Returns the last XID assigned by this call, or the next XID observed
> * if no XID was assigned.
> */
>
> Regards,
>
> --
> Fujii Masao
Attachments:
v3-0001-xid_wraparound-Fix-consume_xids-consuming-one-XID.patch (application/x-patch, 5.2K) download
From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jo=C3=A3o=20Rito?= <ritojoao@users.noreply.github.com>
Date: Tue, 10 Nov 2026 17:31:18 -0300
Subject: [PATCH v3] xid_wraparound: Fix consume_xids() consuming one XID too
many
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
consume_xids() and consume_xids_until() called GetTopTransactionId()
without counting the XID that this may assign, so when the transaction
had no XID yet, they consumed one more XID than requested and returned
a XID that was not the last one they assigned.
Count the XID obtained here and remember it in lastxid, so that the
result is correct even for consume_xids(1). Since consume_xids(0)
cannot return an XID assigned by the function, make it raise an error
instead of returning the next XID.
Also, replace the hard-coded shortcut threshold of 2000 with
XID_SHORTCUT_THRESHOLD, set to 4000 so that the shortcut cannot skip
more XIDs than requested even with 32kB blocks, where XidSkip() can
return up to 3276. Add an assertion to XidSkip() to verify this.
Document what the two functions return.
Author: Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>
Author: João Rito <ritojoao@users.noreply.github.com>
Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@oss.nttdata.com>
Discussion: https://postgr.es/m/2a62f138c9dcb6fcf5889b076d123aa2@oss.nttdata.com
---
.../modules/xid_wraparound/xid_wraparound.c | 52 +++++++++++++------
1 file changed, 36 insertions(+), 16 deletions(-)
diff --git a/src/test/modules/xid_wraparound/xid_wraparound.c b/src/test/modules/xid_wraparound/xid_wraparound.c
index ca25d7e..8a1cba7 100644
--- a/src/test/modules/xid_wraparound/xid_wraparound.c
+++ b/src/test/modules/xid_wraparound/xid_wraparound.c
@@ -26,6 +26,8 @@ static FullTransactionId consume_xids_common(FullTransactionId untilxid, uint64
/*
* Consume the specified number of XIDs.
+ *
+ * Returns the last XID assigned by this function.
*/
PG_FUNCTION_INFO_V1(consume_xids);
Datum
@@ -34,19 +36,21 @@ consume_xids(PG_FUNCTION_ARGS)
int64 nxids = PG_GETARG_INT64(0);
FullTransactionId lastxid;
- if (nxids < 0)
+ if (nxids <= 0)
elog(ERROR, "invalid nxids argument: %" PRId64, nxids);
- if (nxids == 0)
- lastxid = ReadNextFullTransactionId();
- else
- lastxid = consume_xids_common(InvalidFullTransactionId, (uint64) nxids);
+ lastxid = consume_xids_common(InvalidFullTransactionId, (uint64) nxids);
PG_RETURN_FULLTRANSACTIONID(lastxid);
}
/*
- * Consume XIDs, up to the given XID.
+ * Advance XID allocation so that the next XID is at least the target.
+ * This may consume XIDs beyond the target.
+ *
+ * Ensures that the current transaction has a top-level XID.
+ * Returns the last XID assigned by this call, or the next XID observed
+ * if no XID was assigned.
*/
PG_FUNCTION_INFO_V1(consume_xids_until);
Datum
@@ -64,6 +68,23 @@ consume_xids_until(PG_FUNCTION_ARGS)
PG_RETURN_FULLTRANSACTIONID(lastxid);
}
+/*
+ * These constants copied from .c files, because they're private.
+ */
+#define COMMIT_TS_XACTS_PER_PAGE (BLCKSZ / 10)
+#define SUBTRANS_XACTS_PER_PAGE (BLCKSZ / sizeof(TransactionId))
+#define CLOG_XACTS_PER_BYTE 4
+#define CLOG_XACTS_PER_PAGE (BLCKSZ * CLOG_XACTS_PER_BYTE)
+
+/*
+ * Take the shortcut in consume_xids_common() only if we have more than this
+ * many XIDs left to consume. XidSkip() never skips more than
+ * COMMIT_TS_XACTS_PER_PAGE XIDs, which is 3276 with the largest supported
+ * block size (32kB), so this is enough to make sure that the shortcut never
+ * consumes more XIDs than requested.
+ */
+#define XID_SHORTCUT_THRESHOLD 4000
+
/*
* Common functionality between the two public functions.
*/
@@ -89,7 +110,11 @@ consume_xids_common(FullTransactionId untilxid, uint64 nxids)
* the cache overflows, but beyond that, we don't keep track of the
* consumed XIDs.
*/
- (void) GetTopTransactionId();
+ if (!FullTransactionIdIsValid(GetTopFullTransactionIdIfAny()))
+ {
+ lastxid = GetTopFullTransactionId();
+ consumed++;
+ }
for (;;)
{
@@ -115,7 +140,7 @@ consume_xids_common(FullTransactionId untilxid, uint64 nxids)
* If we still have plenty of XIDs to consume, try to take a shortcut
* and bump up the nextXid counter directly.
*/
- if (xids_left > 2000 &&
+ if (xids_left > XID_SHORTCUT_THRESHOLD &&
consumed - last_reported_at < REPORT_INTERVAL &&
MyProc->subxidStatus.overflowed)
{
@@ -153,14 +178,6 @@ consume_xids_common(FullTransactionId untilxid, uint64 nxids)
return lastxid;
}
-/*
- * These constants copied from .c files, because they're private.
- */
-#define COMMIT_TS_XACTS_PER_PAGE (BLCKSZ / 10)
-#define SUBTRANS_XACTS_PER_PAGE (BLCKSZ / sizeof(TransactionId))
-#define CLOG_XACTS_PER_BYTE 4
-#define CLOG_XACTS_PER_PAGE (BLCKSZ * CLOG_XACTS_PER_BYTE)
-
/*
* All the interesting action in GetNewTransactionId happens when we extend
* the SLRUs, or at the uint32 wraparound. If the nextXid counter is not close
@@ -193,6 +210,9 @@ XidSkip(FullTransactionId fullxid)
return 0;
distance = Min(distance, CLOG_XACTS_PER_PAGE - rem);
+ /* The shortcut threshold in consume_xids_common() relies on this */
+ Assert(distance <= XID_SHORTCUT_THRESHOLD);
+
return distance;
}
--
2.53.0.windows.1
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Fix for consume_xids advancing XIDs incorrectly
@ 2026-11-18 05:11 Fujii Masao <masao.fujii@gmail.com>
parent: João Rito <joaovictor.rito@gmail.com>
0 siblings, 0 replies; 19+ messages in thread
From: Fujii Masao @ 2026-11-18 05:11 UTC (permalink / raw)
To: João Rito <joaovictor.rito@gmail.com>; +Cc: pgsql-hackers; sawada.mshk@gmail.com; michael@paquier.xyz
On Thu, Nov 12, 2026 at 7:12 AM João Rito <joaovictor.rito@gmail.com> wrote:
> The attached v3 updates the comment of consume_xids_until() as you
> suggested. This is the only change from v2:
Thanks for updating the patch! I've pushed it.
Regards,
--
Fujii Masao
^ permalink raw reply [nested|flat] 19+ messages in thread
end of thread, other threads:[~2026-11-18 05:11 UTC | newest]
Thread overview: 19+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-10-15 01:51 Fix for consume_xids advancing XIDs incorrectly Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>
2024-10-15 22:26 ` Masahiko Sawada <sawada.mshk@gmail.com>
2024-10-16 05:06 ` Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>
2024-10-16 17:58 ` Masahiko Sawada <sawada.mshk@gmail.com>
2024-10-18 05:57 ` Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>
2024-10-23 06:47 ` Fujii Masao <masao.fujii@oss.nttdata.com>
2024-10-23 20:23 ` Masahiko Sawada <sawada.mshk@gmail.com>
2024-10-24 14:55 ` Fujii Masao <masao.fujii@oss.nttdata.com>
2024-10-25 21:31 ` Masahiko Sawada <sawada.mshk@gmail.com>
2024-10-29 09:00 ` Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>
2024-10-29 18:09 ` Fujii Masao <masao.fujii@oss.nttdata.com>
2024-10-30 07:00 ` Yushi Ogiwara <btogiwarayuushi@oss.nttdata.com>
2024-10-31 18:53 ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-01-08 01:41 ` Michael Paquier <michael@paquier.xyz>
2024-12-03 09:28 ` Kirill Reshke <reshkekirill@gmail.com>
2026-11-10 22:52 ` João Rito <joaovictor.rito@gmail.com>
2026-11-11 08:52 ` Fujii Masao <masao.fujii@gmail.com>
2026-11-11 22:12 ` João Rito <joaovictor.rito@gmail.com>
2026-11-18 05:11 ` Fujii Masao <masao.fujii@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox