Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1p3MnS-0003Qh-1W for pgsql-hackers@arkaria.postgresql.org; Thu, 08 Dec 2022 19:43:06 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1p3MnP-0000D7-9J for pgsql-hackers@arkaria.postgresql.org; Thu, 08 Dec 2022 19:43:03 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1p3MnO-0000CQ-SV for pgsql-hackers@lists.postgresql.org; Thu, 08 Dec 2022 19:43:02 +0000 Received: from mail-ej1-x634.google.com ([2a00:1450:4864:20::634]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1p3MnH-0004jQ-Ou for pgsql-hackers@postgresql.org; Thu, 08 Dec 2022 19:43:02 +0000 Received: by mail-ej1-x634.google.com with SMTP id gh17so6471211ejb.6 for ; Thu, 08 Dec 2022 11:42:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cybertec-at.20210112.gappssmtp.com; s=20210112; h=message-id:date:mime-version:subject:to:from:from:to:cc:subject :date:message-id:reply-to; bh=44U10FhNPWSHilpq3cyhKaTrqi5MlGjnb+MNU1EuavI=; b=PbUMoVRd005G6SrQ1lSzN+gaF1pqpEk8eCmWgWUIw9Y8R8kReu9w67BrnID3wBmwy9 lHbiMhqQk2UQs/QklQuJOm+4PbH5r8shx+ATtSQ7gF87LIDo1NvsKtMWl44QDN9bm2Ro Oz7BwilU+m74J295+LwrmZRjCm0dgHv65IhkqVBYdX5MBHCzcAXzmggvxr4iodyo/iuH iOfDa1WmT15VL3t8sMDG6zg+kXmHTVOjUfqsTLT0gAdWRYb/cPB2C+laf25sV6h8zqfZ 8f3Q3M10tk4CNoVVugDoy8vw25O9Dr1P2WK9JLTV4xvRRV4TtiKGUgM+olyCqIM7+7Hp Ej0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=message-id:date:mime-version:subject:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=44U10FhNPWSHilpq3cyhKaTrqi5MlGjnb+MNU1EuavI=; b=VGHLi+RnTysYDn8zvZEDesnJMRUutxVbUXl7kANBMED6p2BkH66oMNeGGnuaqTSwFN OS4y2x/6N9/WsI9CPVxQsu/aSjHRyy2BLHPh0x4D8aFyeoktcqow5gGCFkRx+bhqFrk+ AyS97rOMhr6hWTBdzp8LApAdUTeRMSCyXrxPOVwN6pxRlSE141xRpurmFlzFA0rArYee PZrFXMhcWQPws+w8WPdEbM6nIz9Jyc/ngnzwPw7oZMybyhanrqvO6iMwGyeMr4qr/j+r 8/TiJu/BF4Vze28PJXCSXWmCxjHBjbceAKjhxgSmlJPzh/I9NWxLr1JAYZWjDLndr/fo 3rBA== X-Gm-Message-State: ANoB5pn2xAvGB+mCq2dQ4pbNmsDf40GpzZ/Ci8FjY/kop0W4kba540M0 kixWqZCiIGaBvCcsOi+mwqMG9P4sVG52st0x X-Google-Smtp-Source: AA0mqf7Q56KX9sr2V1Ho/jw1UBe5I55oNWkItzk3FXKGGWfr1KzrAKgdMRoPDKlcFCq1jpj3eRdVdQ== X-Received: by 2002:a17:906:9488:b0:7c0:f908:eafe with SMTP id t8-20020a170906948800b007c0f908eafemr13543427ejx.547.1670528574406; Thu, 08 Dec 2022 11:42:54 -0800 (PST) Received: from antos ([88.103.134.108]) by smtp.gmail.com with ESMTPSA id h5-20020aa7c945000000b0044dbecdcd29sm3687027edt.12.2022.12.08.11.42.53 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Dec 2022 11:42:53 -0800 (PST) From: Antonin Houska To: pgsql-hackers@postgresql.org Subject: sendFileWithContent() does not advance the source pointer X-Mailer: MH-E 8.6+git; nmh 1.7+dev; GNU Emacs 28.2.50 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Date: Thu, 08 Dec 2022 20:44:05 +0100 Message-ID: <15764.1670528645@antos> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --=-=-= Content-Type: text/plain When checking something else in the base backup code, I've noticed that sendFileWithContent() does not advance the 'content' pointer. The sink buffer is large enough (32kB) so that the first iteration usually processes the whole file (only special files are processed by this function), and thus that the problem is hidden. However it's possible to hit the issue: if there are too many tablespaces, pg_basebackup generates corrupted tablespace_map. Instead of writing all the tablespace paths it writes only some and then starts to write the contents from the beginning again. The attached script generates scripts to create many tablespaces as well as the underlying directories. Fix is attached here as well. -- Antonin Houska Web: https://www.cybertec-postgresql.com --=-=-= Content-Type: text/x-shellscript Content-Disposition: inline; filename=many.sh #!/bin/bash TBSPDIR=/mnt/ramdisk/tbspcs TBSPCOUNT=2048 SCRIPT_SH=create.sh SCRIPT_SQL=create.sql echo "#!/bin/bash" > ${SCRIPT_SH} echo "mkdir -p $TBSPDIR" >> ${SCRIPT_SH} echo "" > ${SCRIPT_SQL} i=0 while [ $i -lt $TBSPCOUNT ]; do printf "mkdir %s/%.4d\n" $TBSPDIR $i >> ${SCRIPT_SH} printf "CREATE TABLESPACE tbsp_%.4d LOCATION '%s/%.4d';\n" $i $TBSPDIR $i >> ${SCRIPT_SQL} i=$((i+1)) done chmod u+x ${SCRIPT_SH} --=-=-= Content-Type: text/x-diff Content-Disposition: inline; filename=advance_content_pointer.diff diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index 74fb529380..7aa5f6e44d 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -1073,6 +1073,7 @@ sendFileWithContent(bbsink *sink, const char *filename, const char *content, memcpy(sink->bbs_buffer, content, nbytes); bbsink_archive_contents(sink, nbytes); bytes_done += nbytes; + content += nbytes; } _tarWritePadding(sink, len); --=-=-=--