pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedsendFileWithContent() does not advance the source pointer
3+ messages / 3 participants
[nested] [flat]
Commits from this thread:
2022-12-12 45f5c81ad2 8b5ba2f3f4 Fix failure to advance content pointer in sendFileWithContent.
* sendFileWithContent() does not advance the source pointer
@ 2022-12-08 19:44 Antonin Houska <ah@cybertec.at>
2022-12-08 21:53 ` Re: sendFileWithContent() does not advance the source pointer Andres Freund <andres@anarazel.de>
2022-12-12 15:40 ` Re: sendFileWithContent() does not advance the source pointer Robert Haas <robertmhaas@gmail.com>
0 siblings, 2 replies; 3+ messages in thread
From: Antonin Houska @ 2022-12-08 19:44 UTC (permalink / raw)
To: pgsql-hackers
When checking something else in the base backup code, I've noticed that
sendFileWithContent() does not advance the 'content' pointer. The sink buffer
is large enough (32kB) so that the first iteration usually processes the whole
file (only special files are processed by this function), and thus that the
problem is hidden.
However it's possible to hit the issue: if there are too many tablespaces,
pg_basebackup generates corrupted tablespace_map. Instead of writing all the
tablespace paths it writes only some and then starts to write the contents
from the beginning again.
The attached script generates scripts to create many tablespaces as well as
the underlying directories. Fix is attached here as well.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
Attachments:
many.sh (text/x-shellscript, 431B) download
#!/bin/bash
TBSPDIR=/mnt/ramdisk/tbspcs
TBSPCOUNT=2048
SCRIPT_SH=create.sh
SCRIPT_SQL=create.sql
echo "#!/bin/bash" > ${SCRIPT_SH}
echo "mkdir -p $TBSPDIR" >> ${SCRIPT_SH}
echo "" > ${SCRIPT_SQL}
i=0
while [ $i -lt $TBSPCOUNT ];
do
printf "mkdir %s/%.4d\n" $TBSPDIR $i >> ${SCRIPT_SH}
printf "CREATE TABLESPACE tbsp_%.4d LOCATION '%s/%.4d';\n" $i $TBSPDIR $i >> ${SCRIPT_SQL}
i=$((i+1))
done
chmod u+x ${SCRIPT_SH}
advance_content_pointer.diff (text/x-diff, 461B) download
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index 74fb529380..7aa5f6e44d 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -1073,6 +1073,7 @@ sendFileWithContent(bbsink *sink, const char *filename, const char *content,
memcpy(sink->bbs_buffer, content, nbytes);
bbsink_archive_contents(sink, nbytes);
bytes_done += nbytes;
+ content += nbytes;
}
_tarWritePadding(sink, len);
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: sendFileWithContent() does not advance the source pointer
2022-12-08 19:44 sendFileWithContent() does not advance the source pointer Antonin Houska <ah@cybertec.at>
@ 2022-12-08 21:53 ` Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 3+ messages in thread
From: Andres Freund @ 2022-12-08 21:53 UTC (permalink / raw)
To: Antonin Houska <ah@cybertec.at>; Robert Haas <robertmhaas@gmail.com>; +Cc: pgsql-hackers
Hi,
On 2022-12-08 20:44:05 +0100, Antonin Houska wrote:
> When checking something else in the base backup code, I've noticed that
> sendFileWithContent() does not advance the 'content' pointer.
Oof. Luckily it looks like that is a relatively recent issue, introduced in
bef47ff85df, which is only in 15.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: sendFileWithContent() does not advance the source pointer
2022-12-08 19:44 sendFileWithContent() does not advance the source pointer Antonin Houska <ah@cybertec.at>
@ 2022-12-12 15:40 ` Robert Haas <robertmhaas@gmail.com>
1 sibling, 0 replies; 3+ messages in thread
From: Robert Haas @ 2022-12-12 15:40 UTC (permalink / raw)
To: Antonin Houska <ah@cybertec.at>; +Cc: pgsql-hackers
On Thu, Dec 8, 2022 at 2:43 PM Antonin Houska <ah@cybertec.at> wrote:
> When checking something else in the base backup code, I've noticed that
> sendFileWithContent() does not advance the 'content' pointer. The sink buffer
> is large enough (32kB) so that the first iteration usually processes the whole
> file (only special files are processed by this function), and thus that the
> problem is hidden.
>
> However it's possible to hit the issue: if there are too many tablespaces,
> pg_basebackup generates corrupted tablespace_map. Instead of writing all the
> tablespace paths it writes only some and then starts to write the contents
> from the beginning again.
Thanks for the report, analysis, and fix. I have committed your patch
and back-patched to v15.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2022-12-12 15:40 UTC | newest]
Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-12-08 19:44 sendFileWithContent() does not advance the source pointer Antonin Houska <ah@cybertec.at>
2022-12-08 21:53 ` Andres Freund <andres@anarazel.de>
2022-12-12 15:40 ` Robert Haas <robertmhaas@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox