pg.ddx.io  pgsql-bugs@postgresql.org mailing list archive  
help / color / mirror / Atom feed
Don't try fetching future segment of a TLI.
13+ messages / 6 participants
[nested] [flat]

Commits from this thread:
  2020-04-07 4bd0ad9e44 683e0ef553 dcd1569fb0 660f1fc40d 70b5979742 e3f5b148ed  Prevent archive recovery from scanning non-existent WAL files.
* Don't try fetching future segment of a TLI.
@ 2020-01-29 03:02 Kyotaro Horiguchi <horikyota.ntt@gmail.com>
  2020-02-01 05:31 ` Re: Don't try fetching future segment of a TLI. David Steele <david@pgmasters.net>
  0 siblings, 1 reply; 13+ messages in thread

From: Kyotaro Horiguchi @ 2020-01-29 03:02 UTC (permalink / raw)
  To: pgsql-hackers@lists.postgresql.org; +Cc: psuderevsky@gmail.com; pgsql-bugs@lists.postgresql.org

Hello, I added (moved to) -hackers.

At Tue, 28 Jan 2020 19:13:32 +0300, Pavel Suderevsky <psuderevsky@gmail.com> wrote in 
> But for me it still seems that PostgreSQL has enough information to check
> that no WALs exist for the new timeline to omit searching all the
> possibly-existing WALs.
> 
> It can just look through the first received new-timeline's WAL and ensure
> timeline switch occured in this WAL. Finally, it can check archive for the
> only one possibly-existing previous WAL.

Right. The timeline history file tells where a timeline ends.

> Regading influence: issue is not about the large amount of WALs to apply
> but in searching for the non-existing WALs on the remote storage, each such
> search can take 5-10 seconds while obtaining existing WAL takes
> milliseconds.

Wow. I didn't know of a file system that takes that much seconds to
trying non-existent files. Although I still think this is not a bug,
but avoiding that actually leads to a big win on such systems.

After a thought, I think it's safe and effectively doable to let
XLogFileReadAnyTLI() refrain from trying WAL segments of too-high
TLIs.  Some garbage archive files out of the range of a timeline might
be seen, for example, after reusing archive directory without clearing
files.  However, fetching such garbages just to fail doesn't
contribute durability or reliablity at all, I think.

The attached does that. 

Any thoughts?

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

Attachments:

  v1-0001-Don-t-try-fetching-out-of-timeline-segments.patch (text/x-patch, 0B) download

^ permalink  raw  reply  [nested|flat] 13+ messages in thread

* Re: Don't try fetching future segment of a TLI.
  2020-01-29 03:02 Don't try fetching future segment of a TLI. Kyotaro Horiguchi <horikyota.ntt@gmail.com>
@ 2020-02-01 05:31 ` David Steele <david@pgmasters.net>
  2020-03-19 13:22   ` Re: Don't try fetching future segment of a TLI. Pavel Suderevsky <psuderevsky@gmail.com>
  0 siblings, 1 reply; 13+ messages in thread

From: David Steele @ 2020-02-01 05:31 UTC (permalink / raw)
  To: Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-hackers@lists.postgresql.org; +Cc: psuderevsky@gmail.com; pgsql-bugs@lists.postgresql.org

On 1/28/20 8:02 PM, Kyotaro Horiguchi wrote:
 > At Tue, 28 Jan 2020 19:13:32 +0300, Pavel Suderevsky
 >> Regading influence: issue is not about the large amount of WALs to apply
 >> but in searching for the non-existing WALs on the remote storage, 
each such
 >> search can take 5-10 seconds while obtaining existing WAL takes
 >> milliseconds.
 >
 > Wow. I didn't know of a file system that takes that much seconds to
 > trying non-existent files. Although I still think this is not a bug,
 > but avoiding that actually leads to a big win on such systems.

I have not tested this case but I can imagine it would be slow in 
practice.  It's axiomatic that is hard to prove a negative.  With 
multi-region replication it might well take some time to be sure that 
the file *really* doesn't exist and hasn't just been lost in a single 
region.

 > After a thought, I think it's safe and effectively doable to let
 > XLogFileReadAnyTLI() refrain from trying WAL segments of too-high
 > TLIs.  Some garbage archive files out of the range of a timeline might
 > be seen, for example, after reusing archive directory without clearing
 > files.  However, fetching such garbages just to fail doesn't
 > contribute durability or reliablity at all, I think.

The patch seems sane, the trick will be testing it.

Pavel, do you have an environment where you can ensure this is a 
performance benefit?

Regards,
-- 
-David
david@pgmasters.net





^ permalink  raw  reply  [nested|flat] 13+ messages in thread

* Re: Don't try fetching future segment of a TLI.
  2020-01-29 03:02 Don't try fetching future segment of a TLI. Kyotaro Horiguchi <horikyota.ntt@gmail.com>
  2020-02-01 05:31 ` Re: Don't try fetching future segment of a TLI. David Steele <david@pgmasters.net>
@ 2020-03-19 13:22   ` Pavel Suderevsky <psuderevsky@gmail.com>
  2020-04-06 17:43     ` Re: Don't try fetching future segment of a TLI. Fujii Masao <masao.fujii@oss.nttdata.com>
  0 siblings, 1 reply; 13+ messages in thread

From: Pavel Suderevsky @ 2020-03-19 13:22 UTC (permalink / raw)
  To: David Steele <david@pgmasters.net>; +Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-hackers@lists.postgresql.org; pgsql-bugs@lists.postgresql.org

Hi,

I've tested patch provided by Kyotaro and do confirm it fixes the issue.
Any chance it will be merged to one of the next minor releases?

Thank you very much!

сб, 1 февр. 2020 г. в 08:31, David Steele <david@pgmasters.net>:

> On 1/28/20 8:02 PM, Kyotaro Horiguchi wrote:
>  > At Tue, 28 Jan 2020 19:13:32 +0300, Pavel Suderevsky
>  >> Regading influence: issue is not about the large amount of WALs to
> apply
>  >> but in searching for the non-existing WALs on the remote storage,
> each such
>  >> search can take 5-10 seconds while obtaining existing WAL takes
>  >> milliseconds.
>  >
>  > Wow. I didn't know of a file system that takes that much seconds to
>  > trying non-existent files. Although I still think this is not a bug,
>  > but avoiding that actually leads to a big win on such systems.
>
> I have not tested this case but I can imagine it would be slow in
> practice.  It's axiomatic that is hard to prove a negative.  With
> multi-region replication it might well take some time to be sure that
> the file *really* doesn't exist and hasn't just been lost in a single
> region.
>
>  > After a thought, I think it's safe and effectively doable to let
>  > XLogFileReadAnyTLI() refrain from trying WAL segments of too-high
>  > TLIs.  Some garbage archive files out of the range of a timeline might
>  > be seen, for example, after reusing archive directory without clearing
>  > files.  However, fetching such garbages just to fail doesn't
>  > contribute durability or reliablity at all, I think.
>
> The patch seems sane, the trick will be testing it.
>
> Pavel, do you have an environment where you can ensure this is a
> performance benefit?
>
> Regards,
> --
> -David
> david@pgmasters.net
>

^ permalink  raw  reply  [nested|flat] 13+ messages in thread

* Re: Don't try fetching future segment of a TLI.
  2020-01-29 03:02 Don't try fetching future segment of a TLI. Kyotaro Horiguchi <horikyota.ntt@gmail.com>
  2020-02-01 05:31 ` Re: Don't try fetching future segment of a TLI. David Steele <david@pgmasters.net>
  2020-03-19 13:22   ` Re: Don't try fetching future segment of a TLI. Pavel Suderevsky <psuderevsky@gmail.com>
@ 2020-04-06 17:43     ` Fujii Masao <masao.fujii@oss.nttdata.com>
  2020-04-06 19:04       ` Re: Don't try fetching future segment of a TLI. David Steele <david@pgmasters.net>
  2020-04-07 01:29       ` Re: Don't try fetching future segment of a TLI. Kyotaro Horiguchi <horikyota.ntt@gmail.com>
  0 siblings, 2 replies; 13+ messages in thread

From: Fujii Masao @ 2020-04-06 17:43 UTC (permalink / raw)
  To: Pavel Suderevsky <psuderevsky@gmail.com>; David Steele <david@pgmasters.net>; +Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-hackers@lists.postgresql.org; pgsql-bugs@lists.postgresql.org



On 2020/03/19 22:22, Pavel Suderevsky wrote:
> Hi,
> 
> I've tested patch provided by Kyotaro and do confirm it fixes the issue.

The patch looks good to me. Attached is the updated version of the patch.
I updated only comments.

Barring any objection, I will commit this patch.

> Any chance it will be merged to one of the next minor releases?

This doesn't seem a bug, so I'm thinking to merge this to next *major*
version release, i.e., v13.

Regards,

-- 
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION

Attachments:

v2-0001-Don-t-try-fetching-out-of-timeline-segments.patch (text/plain, 1.5K) download diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index abf954ba39..d35094a0cb 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -3776,11 +3776,36 @@ XLogFileReadAnyTLI(XLogSegNo segno, int emode, XLogSource source) foreach(cell, tles) { - TimeLineID tli = ((TimeLineHistoryEntry *) lfirst(cell))->tli; + TimeLineHistoryEntry *hent = (TimeLineHistoryEntry *) lfirst(cell); + TimeLineID tli = hent->tli; if (tli < curFileTLI) break; /* don't bother looking at too-old TLIs */ + /* + * Skip scanning the timeline ID that the logfile segment to read + * doesn't belong to + */ + if (hent->begin != InvalidXLogRecPtr) + { + XLogSegNo beginseg = 0; + + XLByteToSeg(hent->begin, beginseg, wal_segment_size); + + /* + * The logfile segment that doesn't belong to the timeline is + * older or newer than the segment that the timeline started or + * eneded at, respectively. It's sufficient to check only the + * starting segment of the timeline here. Since the timelines are + * scanned in descending order in this loop, any segments newer + * than the ending segment should belong to newer timeline and + * have already been read before. So it's not necessary to check + * the ending segment of the timeline here. + */ + if (segno < beginseg) + continue; + } + if (source == XLOG_FROM_ANY || source == XLOG_FROM_ARCHIVE) { fd = XLogFileRead(segno, emode, tli,
^ permalink raw reply [nested|flat] 13+ messages in thread

* Re: Don't try fetching future segment of a TLI.
  2020-01-29 03:02 Don't try fetching future segment of a TLI. Kyotaro Horiguchi <horikyota.ntt@gmail.com>
  2020-02-01 05:31 ` Re: Don't try fetching future segment of a TLI. David Steele <david@pgmasters.net>
  2020-03-19 13:22   ` Re: Don't try fetching future segment of a TLI. Pavel Suderevsky <psuderevsky@gmail.com>
  2020-04-06 17:43     ` Re: Don't try fetching future segment of a TLI. Fujii Masao <masao.fujii@oss.nttdata.com>
@ 2020-04-06 19:04       ` David Steele <david@pgmasters.net>
  2020-04-07 03:15         ` Re: Don't try fetching future segment of a TLI. Fujii Masao <masao.fujii@oss.nttdata.com>
  1 sibling, 1 reply; 13+ messages in thread

From: David Steele @ 2020-04-06 19:04 UTC (permalink / raw)
  To: Fujii Masao <masao.fujii@oss.nttdata.com>; Pavel Suderevsky <psuderevsky@gmail.com>; +Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-hackers@lists.postgresql.org; pgsql-bugs@lists.postgresql.org

On 4/6/20 1:43 PM, Fujii Masao wrote:
> 
> 
> On 2020/03/19 22:22, Pavel Suderevsky wrote:
>> Hi,
>>
>> I've tested patch provided by Kyotaro and do confirm it fixes the issue.
> 
> The patch looks good to me. Attached is the updated version of the patch.
> I updated only comments.
> 
> Barring any objection, I will commit this patch.

The patch looks good to me.

>> Any chance it will be merged to one of the next minor releases?
> 
> This doesn't seem a bug, so I'm thinking to merge this to next *major*
> version release, i.e., v13.

Not a bug, perhaps, but I think we do consider back-patching performance 
problems. The rise in S3 usage has just exposed how poorly this 
performed code in high-latency environments.

Regards,
-- 
-David
david@pgmasters.net





^ permalink  raw  reply  [nested|flat] 13+ messages in thread

* Re: Don't try fetching future segment of a TLI.
  2020-01-29 03:02 Don't try fetching future segment of a TLI. Kyotaro Horiguchi <horikyota.ntt@gmail.com>
  2020-02-01 05:31 ` Re: Don't try fetching future segment of a TLI. David Steele <david@pgmasters.net>
  2020-03-19 13:22   ` Re: Don't try fetching future segment of a TLI. Pavel Suderevsky <psuderevsky@gmail.com>
  2020-04-06 17:43     ` Re: Don't try fetching future segment of a TLI. Fujii Masao <masao.fujii@oss.nttdata.com>
  2020-04-06 19:04       ` Re: Don't try fetching future segment of a TLI. David Steele <david@pgmasters.net>
@ 2020-04-07 03:15         ` Fujii Masao <masao.fujii@oss.nttdata.com>
  2020-04-07 07:48           ` Re: Don't try fetching future segment of a TLI. Kyotaro Horiguchi <horikyota.ntt@gmail.com>
  2020-04-07 08:17           ` Re: Don't try fetching future segment of a TLI. Michael Paquier <michael@paquier.xyz>
  0 siblings, 2 replies; 13+ messages in thread

From: Fujii Masao @ 2020-04-07 03:15 UTC (permalink / raw)
  To: David Steele <david@pgmasters.net>; Pavel Suderevsky <psuderevsky@gmail.com>; +Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-hackers@lists.postgresql.org; pgsql-bugs@lists.postgresql.org



On 2020/04/07 4:04, David Steele wrote:
> On 4/6/20 1:43 PM, Fujii Masao wrote:
>>
>>
>> On 2020/03/19 22:22, Pavel Suderevsky wrote:
>>> Hi,
>>>
>>> I've tested patch provided by Kyotaro and do confirm it fixes the issue.
>>
>> The patch looks good to me. Attached is the updated version of the patch.
>> I updated only comments.
>>
>> Barring any objection, I will commit this patch.
> 
> The patch looks good to me.
> 
>>> Any chance it will be merged to one of the next minor releases?
>>
>> This doesn't seem a bug, so I'm thinking to merge this to next *major*
>> version release, i.e., v13.
> 
> Not a bug, perhaps, but I think we do consider back-patching performance problems. The rise in S3 usage has just exposed how poorly this performed code in high-latency environments.

I understood the situation and am fine to back-patch that. But I'm not sure
if it's fair to do that. Maybe we need to hear more opinions about this?
OTOH, feature freeze for v13 is today, so what about committing the patch
in v13 at first, and then doing the back-patch after hearing opinions and
receiving many +1?

Regards,

-- 
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION





^ permalink  raw  reply  [nested|flat] 13+ messages in thread

* Re: Don't try fetching future segment of a TLI.
  2020-01-29 03:02 Don't try fetching future segment of a TLI. Kyotaro Horiguchi <horikyota.ntt@gmail.com>
  2020-02-01 05:31 ` Re: Don't try fetching future segment of a TLI. David Steele <david@pgmasters.net>
  2020-03-19 13:22   ` Re: Don't try fetching future segment of a TLI. Pavel Suderevsky <psuderevsky@gmail.com>
  2020-04-06 17:43     ` Re: Don't try fetching future segment of a TLI. Fujii Masao <masao.fujii@oss.nttdata.com>
  2020-04-06 19:04       ` Re: Don't try fetching future segment of a TLI. David Steele <david@pgmasters.net>
  2020-04-07 03:15         ` Re: Don't try fetching future segment of a TLI. Fujii Masao <masao.fujii@oss.nttdata.com>
@ 2020-04-07 07:48           ` Kyotaro Horiguchi <horikyota.ntt@gmail.com>
  2020-04-07 11:21             ` Re: Don't try fetching future segment of a TLI. David Steele <david@pgmasters.net>
  1 sibling, 1 reply; 13+ messages in thread

From: Kyotaro Horiguchi @ 2020-04-07 07:48 UTC (permalink / raw)
  To: masao.fujii@oss.nttdata.com; +Cc: david@pgmasters.net; psuderevsky@gmail.com; pgsql-hackers@lists.postgresql.org; pgsql-bugs@lists.postgresql.org

At Tue, 7 Apr 2020 12:15:00 +0900, Fujii Masao <masao.fujii@oss.nttdata.com> wrote in 
> 
> 
> On 2020/04/07 4:04, David Steele wrote:
> > On 4/6/20 1:43 PM, Fujii Masao wrote:
> >>
> >>
> >> On 2020/03/19 22:22, Pavel Suderevsky wrote:
> >>> Hi,
> >>>
> >>> I've tested patch provided by Kyotaro and do confirm it fixes the
> >>> issue.
> >>
> >> The patch looks good to me. Attached is the updated version of the
> >> patch.
> >> I updated only comments.
> >>
> >> Barring any objection, I will commit this patch.
> > The patch looks good to me.
> > 
> >>> Any chance it will be merged to one of the next minor releases?
> >>
> >> This doesn't seem a bug, so I'm thinking to merge this to next *major*
> >> version release, i.e., v13.
> > Not a bug, perhaps, but I think we do consider back-patching
> > performance problems. The rise in S3 usage has just exposed how poorly
> > this performed code in high-latency environments.
> 
> I understood the situation and am fine to back-patch that. But I'm not
> sure
> if it's fair to do that. Maybe we need to hear more opinions about
> this?
> OTOH, feature freeze for v13 is today, so what about committing the
> patch
> in v13 at first, and then doing the back-patch after hearing opinions
> and
> receiving many +1?

+1 for commit only v13 today, then back-patch if people wants and/or
accepts.

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center





^ permalink  raw  reply  [nested|flat] 13+ messages in thread

* Re: Don't try fetching future segment of a TLI.
  2020-01-29 03:02 Don't try fetching future segment of a TLI. Kyotaro Horiguchi <horikyota.ntt@gmail.com>
  2020-02-01 05:31 ` Re: Don't try fetching future segment of a TLI. David Steele <david@pgmasters.net>
  2020-03-19 13:22   ` Re: Don't try fetching future segment of a TLI. Pavel Suderevsky <psuderevsky@gmail.com>
  2020-04-06 17:43     ` Re: Don't try fetching future segment of a TLI. Fujii Masao <masao.fujii@oss.nttdata.com>
  2020-04-06 19:04       ` Re: Don't try fetching future segment of a TLI. David Steele <david@pgmasters.net>
  2020-04-07 03:15         ` Re: Don't try fetching future segment of a TLI. Fujii Masao <masao.fujii@oss.nttdata.com>
  2020-04-07 07:48           ` Re: Don't try fetching future segment of a TLI. Kyotaro Horiguchi <horikyota.ntt@gmail.com>
@ 2020-04-07 11:21             ` David Steele <david@pgmasters.net>
  2020-04-07 16:49               ` Re: Don't try fetching future segment of a TLI. Fujii Masao <masao.fujii@oss.nttdata.com>
  0 siblings, 1 reply; 13+ messages in thread

From: David Steele @ 2020-04-07 11:21 UTC (permalink / raw)
  To: Kyotaro Horiguchi <horikyota.ntt@gmail.com>; masao.fujii@oss.nttdata.com; +Cc: psuderevsky@gmail.com; pgsql-hackers@lists.postgresql.org; pgsql-bugs@lists.postgresql.org


On 4/7/20 3:48 AM, Kyotaro Horiguchi wrote:
> At Tue, 7 Apr 2020 12:15:00 +0900, Fujii Masao <masao.fujii@oss.nttdata.com> wrote in
>>>> This doesn't seem a bug, so I'm thinking to merge this to next *major*
>>>> version release, i.e., v13.
>>> Not a bug, perhaps, but I think we do consider back-patching
>>> performance problems. The rise in S3 usage has just exposed how poorly
>>> this performed code in high-latency environments.
>>
>> I understood the situation and am fine to back-patch that. But I'm not
>> sure
>> if it's fair to do that. Maybe we need to hear more opinions about
>> this?
>> OTOH, feature freeze for v13 is today, so what about committing the
>> patch
>> in v13 at first, and then doing the back-patch after hearing opinions
>> and
>> receiving many +1?
> 
> +1 for commit only v13 today, then back-patch if people wants and/or
> accepts.

Definitely +1 for a commit today to v13. I certainly was not trying to 
hold that up.

-- 
-David
david@pgmasters.net





^ permalink  raw  reply  [nested|flat] 13+ messages in thread

* Re: Don't try fetching future segment of a TLI.
  2020-01-29 03:02 Don't try fetching future segment of a TLI. Kyotaro Horiguchi <horikyota.ntt@gmail.com>
  2020-02-01 05:31 ` Re: Don't try fetching future segment of a TLI. David Steele <david@pgmasters.net>
  2020-03-19 13:22   ` Re: Don't try fetching future segment of a TLI. Pavel Suderevsky <psuderevsky@gmail.com>
  2020-04-06 17:43     ` Re: Don't try fetching future segment of a TLI. Fujii Masao <masao.fujii@oss.nttdata.com>
  2020-04-06 19:04       ` Re: Don't try fetching future segment of a TLI. David Steele <david@pgmasters.net>
  2020-04-07 03:15         ` Re: Don't try fetching future segment of a TLI. Fujii Masao <masao.fujii@oss.nttdata.com>
  2020-04-07 07:48           ` Re: Don't try fetching future segment of a TLI. Kyotaro Horiguchi <horikyota.ntt@gmail.com>
  2020-04-07 11:21             ` Re: Don't try fetching future segment of a TLI. David Steele <david@pgmasters.net>
@ 2020-04-07 16:49               ` Fujii Masao <masao.fujii@oss.nttdata.com>
  0 siblings, 0 replies; 13+ messages in thread

From: Fujii Masao @ 2020-04-07 16:49 UTC (permalink / raw)
  To: David Steele <david@pgmasters.net>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; +Cc: psuderevsky@gmail.com; pgsql-hackers@lists.postgresql.org; pgsql-bugs@lists.postgresql.org



On 2020/04/07 20:21, David Steele wrote:
> 
> On 4/7/20 3:48 AM, Kyotaro Horiguchi wrote:
>> At Tue, 7 Apr 2020 12:15:00 +0900, Fujii Masao <masao.fujii@oss.nttdata.com> wrote in
>>>>> This doesn't seem a bug, so I'm thinking to merge this to next *major*
>>>>> version release, i.e., v13.
>>>> Not a bug, perhaps, but I think we do consider back-patching
>>>> performance problems. The rise in S3 usage has just exposed how poorly
>>>> this performed code in high-latency environments.
>>>
>>> I understood the situation and am fine to back-patch that. But I'm not
>>> sure
>>> if it's fair to do that. Maybe we need to hear more opinions about
>>> this?
>>> OTOH, feature freeze for v13 is today, so what about committing the
>>> patch
>>> in v13 at first, and then doing the back-patch after hearing opinions
>>> and
>>> receiving many +1?
>>
>> +1 for commit only v13 today, then back-patch if people wants and/or
>> accepts.
> 
> Definitely +1 for a commit today to v13. I certainly was not trying to hold that up
Pushed the patch to v13, at first!

Regards,

-- 
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION





^ permalink  raw  reply  [nested|flat] 13+ messages in thread

* Re: Don't try fetching future segment of a TLI.
  2020-01-29 03:02 Don't try fetching future segment of a TLI. Kyotaro Horiguchi <horikyota.ntt@gmail.com>
  2020-02-01 05:31 ` Re: Don't try fetching future segment of a TLI. David Steele <david@pgmasters.net>
  2020-03-19 13:22   ` Re: Don't try fetching future segment of a TLI. Pavel Suderevsky <psuderevsky@gmail.com>
  2020-04-06 17:43     ` Re: Don't try fetching future segment of a TLI. Fujii Masao <masao.fujii@oss.nttdata.com>
  2020-04-06 19:04       ` Re: Don't try fetching future segment of a TLI. David Steele <david@pgmasters.net>
  2020-04-07 03:15         ` Re: Don't try fetching future segment of a TLI. Fujii Masao <masao.fujii@oss.nttdata.com>
@ 2020-04-07 08:17           ` Michael Paquier <michael@paquier.xyz>
  2020-04-07 09:36             ` Re: Don't try fetching future segment of a TLI. Grigory Smolkin <g.smolkin@postgrespro.ru>
  1 sibling, 1 reply; 13+ messages in thread

From: Michael Paquier @ 2020-04-07 08:17 UTC (permalink / raw)
  To: Fujii Masao <masao.fujii@oss.nttdata.com>; +Cc: David Steele <david@pgmasters.net>; Pavel Suderevsky <psuderevsky@gmail.com>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-hackers@lists.postgresql.org; pgsql-bugs@lists.postgresql.org

On Tue, Apr 07, 2020 at 12:15:00PM +0900, Fujii Masao wrote:
> I understood the situation and am fine to back-patch that. But I'm not sure
> if it's fair to do that. Maybe we need to hear more opinions about this?
> OTOH, feature freeze for v13 is today, so what about committing the patch
> in v13 at first, and then doing the back-patch after hearing opinions and
> receiving many +1?

I have not looked at the patch so I cannot say much about it, but it
is annoying to fetch segments you are not going to need anyway if you
target recovery with a timeline older than the segments fetched and
this has a cost when you pay for the bandwidth of your environment
with only one archive location.  So a backpatch sounds like a good
thing to do even if recovery is not broken per-se, only slower.

Designing a TAP test for that is tricky, but you could look at the
logs of the backend to make sure that only the wanted segments are
fetched with a central archived solution and multiple timelines
involved.  And costly it is.
--
Michael

Attachments:

  signature.asc (application/pgp-signature, 832B) download

^ permalink  raw  reply  [nested|flat] 13+ messages in thread

* Re: Don't try fetching future segment of a TLI.
  2020-01-29 03:02 Don't try fetching future segment of a TLI. Kyotaro Horiguchi <horikyota.ntt@gmail.com>
  2020-02-01 05:31 ` Re: Don't try fetching future segment of a TLI. David Steele <david@pgmasters.net>
  2020-03-19 13:22   ` Re: Don't try fetching future segment of a TLI. Pavel Suderevsky <psuderevsky@gmail.com>
  2020-04-06 17:43     ` Re: Don't try fetching future segment of a TLI. Fujii Masao <masao.fujii@oss.nttdata.com>
  2020-04-06 19:04       ` Re: Don't try fetching future segment of a TLI. David Steele <david@pgmasters.net>
  2020-04-07 03:15         ` Re: Don't try fetching future segment of a TLI. Fujii Masao <masao.fujii@oss.nttdata.com>
  2020-04-07 08:17           ` Re: Don't try fetching future segment of a TLI. Michael Paquier <michael@paquier.xyz>
@ 2020-04-07 09:36             ` Grigory Smolkin <g.smolkin@postgrespro.ru>
  0 siblings, 0 replies; 13+ messages in thread

From: Grigory Smolkin @ 2020-04-07 09:36 UTC (permalink / raw)
  To: pgsql-bugs@lists.postgresql.org

I`ve bumped into this issue recently:
https://www.postgresql.org/message-id/dd6690b0-ec03-6b3c-6fac-c963f91f87a7%40postgrespro.ru


On 4/6/20 8:43 PM, Fujii Masao wrote:

> The patch looks good to me. Attached is the updated version of the patch.
> I updated only comments.
>
> Barring any objection, I will commit this patch. 

I`ve been running tests on your patch. So far so good.

On Tue, Apr 07, 2020 at 12:15:00PM +0900, Fujii Masao wrote:

> I understood the situation and am fine to back-patch that. But I'm not sure
> if it's fair to do that. Maybe we need to hear more opinions about this?
> OTOH, feature freeze for v13 is today, so what about committing the patch
> in v13 at first, and then doing the back-patch after hearing opinions and
> receiving many +1?

+1 to back-patching it.

-- 
Grigory Smolkin
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

^ permalink  raw  reply  [nested|flat] 13+ messages in thread

* Re: Don't try fetching future segment of a TLI.
  2020-01-29 03:02 Don't try fetching future segment of a TLI. Kyotaro Horiguchi <horikyota.ntt@gmail.com>
  2020-02-01 05:31 ` Re: Don't try fetching future segment of a TLI. David Steele <david@pgmasters.net>
  2020-03-19 13:22   ` Re: Don't try fetching future segment of a TLI. Pavel Suderevsky <psuderevsky@gmail.com>
  2020-04-06 17:43     ` Re: Don't try fetching future segment of a TLI. Fujii Masao <masao.fujii@oss.nttdata.com>
@ 2020-04-07 01:29       ` Kyotaro Horiguchi <horikyota.ntt@gmail.com>
  2020-04-07 03:22         ` Re: Don't try fetching future segment of a TLI. Fujii Masao <masao.fujii@oss.nttdata.com>
  1 sibling, 1 reply; 13+ messages in thread

From: Kyotaro Horiguchi @ 2020-04-07 01:29 UTC (permalink / raw)
  To: masao.fujii@oss.nttdata.com; +Cc: psuderevsky@gmail.com; david@pgmasters.net; pgsql-hackers@lists.postgresql.org; pgsql-bugs@lists.postgresql.org

Thank you for picking this up.

At Tue, 7 Apr 2020 02:43:02 +0900, Fujii Masao <masao.fujii@oss.nttdata.com> wrote in 
> On 2020/03/19 22:22, Pavel Suderevsky wrote:
> > Hi,
> > I've tested patch provided by Kyotaro and do confirm it fixes the
> > issue.
> 
> The patch looks good to me. Attached is the updated version of the
> patch.
> I updated only comments.

+			 * The logfile segment that doesn't belong to the timeline is
+			 * older or newer than the segment that the timeline started or
+			 * eneded at, respectively. It's sufficient to check only the

s/eneded/ended/ ?

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center





^ permalink  raw  reply  [nested|flat] 13+ messages in thread

* Re: Don't try fetching future segment of a TLI.
  2020-01-29 03:02 Don't try fetching future segment of a TLI. Kyotaro Horiguchi <horikyota.ntt@gmail.com>
  2020-02-01 05:31 ` Re: Don't try fetching future segment of a TLI. David Steele <david@pgmasters.net>
  2020-03-19 13:22   ` Re: Don't try fetching future segment of a TLI. Pavel Suderevsky <psuderevsky@gmail.com>
  2020-04-06 17:43     ` Re: Don't try fetching future segment of a TLI. Fujii Masao <masao.fujii@oss.nttdata.com>
  2020-04-07 01:29       ` Re: Don't try fetching future segment of a TLI. Kyotaro Horiguchi <horikyota.ntt@gmail.com>
@ 2020-04-07 03:22         ` Fujii Masao <masao.fujii@oss.nttdata.com>
  0 siblings, 0 replies; 13+ messages in thread

From: Fujii Masao @ 2020-04-07 03:22 UTC (permalink / raw)
  To: Kyotaro Horiguchi <horikyota.ntt@gmail.com>; +Cc: psuderevsky@gmail.com; david@pgmasters.net; pgsql-hackers@lists.postgresql.org; pgsql-bugs@lists.postgresql.org



On 2020/04/07 10:29, Kyotaro Horiguchi wrote:
> Thank you for picking this up.
> 
> At Tue, 7 Apr 2020 02:43:02 +0900, Fujii Masao <masao.fujii@oss.nttdata.com> wrote in
>> On 2020/03/19 22:22, Pavel Suderevsky wrote:
>>> Hi,
>>> I've tested patch provided by Kyotaro and do confirm it fixes the
>>> issue.
>>
>> The patch looks good to me. Attached is the updated version of the
>> patch.
>> I updated only comments.
> 
> +			 * The logfile segment that doesn't belong to the timeline is
> +			 * older or newer than the segment that the timeline started or
> +			 * eneded at, respectively. It's sufficient to check only the
> 
> s/eneded/ended/ ?

Yes! Thanks!

Regards,

-- 
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION

Attachments:

v3-0001-Don-t-try-fetching-out-of-timeline-segments.patch (text/plain, 1.5K) download diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index abf954ba39..ec55d68d27 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -3776,11 +3776,36 @@ XLogFileReadAnyTLI(XLogSegNo segno, int emode, XLogSource source) foreach(cell, tles) { - TimeLineID tli = ((TimeLineHistoryEntry *) lfirst(cell))->tli; + TimeLineHistoryEntry *hent = (TimeLineHistoryEntry *) lfirst(cell); + TimeLineID tli = hent->tli; if (tli < curFileTLI) break; /* don't bother looking at too-old TLIs */ + /* + * Skip scanning the timeline ID that the logfile segment to read + * doesn't belong to + */ + if (hent->begin != InvalidXLogRecPtr) + { + XLogSegNo beginseg = 0; + + XLByteToSeg(hent->begin, beginseg, wal_segment_size); + + /* + * The logfile segment that doesn't belong to the timeline is + * older or newer than the segment that the timeline started or + * ended at, respectively. It's sufficient to check only the + * starting segment of the timeline here. Since the timelines are + * scanned in descending order in this loop, any segments newer + * than the ending segment should belong to newer timeline and + * have already been read before. So it's not necessary to check + * the ending segment of the timeline here. + */ + if (segno < beginseg) + continue; + } + if (source == XLOG_FROM_ANY || source == XLOG_FROM_ARCHIVE) { fd = XLogFileRead(segno, emode, tli,
^ permalink raw reply [nested|flat] 13+ messages in thread


end of thread, other threads:[~2020-04-07 16:49 UTC | newest]

Thread overview: 13+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2020-01-29 03:02 Don't try fetching future segment of a TLI. Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2020-02-01 05:31 ` David Steele <david@pgmasters.net>
2020-03-19 13:22   ` Pavel Suderevsky <psuderevsky@gmail.com>
2020-04-06 17:43     ` Fujii Masao <masao.fujii@oss.nttdata.com>
2020-04-06 19:04       ` David Steele <david@pgmasters.net>
2020-04-07 03:15         ` Fujii Masao <masao.fujii@oss.nttdata.com>
2020-04-07 07:48           ` Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2020-04-07 11:21             ` David Steele <david@pgmasters.net>
2020-04-07 16:49               ` Fujii Masao <masao.fujii@oss.nttdata.com>
2020-04-07 08:17           ` Michael Paquier <michael@paquier.xyz>
2020-04-07 09:36             ` Grigory Smolkin <g.smolkin@postgrespro.ru>
2020-04-07 01:29       ` Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2020-04-07 03:22         ` Fujii Masao <masao.fujii@oss.nttdata.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox