pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Antonin Houska <ah@cybertec.at>
To: pgsql-hackers@postgresql.org
Subject: sendFileWithContent() does not advance the source pointer
Date: Thu, 08 Dec 2022 20:44:05 +0100
Message-ID: <15764.1670528645@antos> (raw)

When checking something else in the base backup code, I've noticed that
sendFileWithContent() does not advance the 'content' pointer. The sink buffer
is large enough (32kB) so that the first iteration usually processes the whole
file (only special files are processed by this function), and thus that the
problem is hidden.

However it's possible to hit the issue: if there are too many tablespaces,
pg_basebackup generates corrupted tablespace_map. Instead of writing all the
tablespace paths it writes only some and then starts to write the contents
from the beginning again.

The attached script generates scripts to create many tablespaces as well as
the underlying directories. Fix is attached here as well.

-- 
Antonin Houska
Web: https://www.cybertec-postgresql.com

Attachments:

many.sh (text/x-shellscript, 431B) download #!/bin/bash TBSPDIR=/mnt/ramdisk/tbspcs TBSPCOUNT=2048 SCRIPT_SH=create.sh SCRIPT_SQL=create.sql echo "#!/bin/bash" > ${SCRIPT_SH} echo "mkdir -p $TBSPDIR" >> ${SCRIPT_SH} echo "" > ${SCRIPT_SQL} i=0 while [ $i -lt $TBSPCOUNT ]; do printf "mkdir %s/%.4d\n" $TBSPDIR $i >> ${SCRIPT_SH} printf "CREATE TABLESPACE tbsp_%.4d LOCATION '%s/%.4d';\n" $i $TBSPDIR $i >> ${SCRIPT_SQL} i=$((i+1)) done chmod u+x ${SCRIPT_SH}
advance_content_pointer.diff (text/x-diff, 461B) download diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index 74fb529380..7aa5f6e44d 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -1073,6 +1073,7 @@ sendFileWithContent(bbsink *sink, const char *filename, const char *content, memcpy(sink->bbs_buffer, content, nbytes); bbsink_archive_contents(sink, nbytes); bytes_done += nbytes; + content += nbytes; } _tarWritePadding(sink, len);

view thread (3+ messages)  latest in thread

Message-ID: <15764.1670528645@antos>
Permalink:  ../15764.1670528645@antos/
Also on:    postgresql.org/message-id/15764.1670528645@antos

 ·  · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-hackers@postgresql.org
  Cc: ah@cybertec.at
  Subject: Re: sendFileWithContent() does not advance the source pointer
  In-Reply-To: <15764.1670528645@antos>
  References: <15764.1670528645@antos>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox