Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1usmEg-00AhIt-3L for pgsql-general@arkaria.postgresql.org; Sun, 31 Aug 2025 17:53:03 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1usmEf-002OHB-Ff for pgsql-general@arkaria.postgresql.org; Sun, 31 Aug 2025 17:53:02 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1usmEe-002OH2-Uw for pgsql-general@lists.postgresql.org; Sun, 31 Aug 2025 17:53:01 +0000 Received: from mout.gmx.net ([212.227.15.15]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1usmEc-002iEy-2A for pgsql-general@lists.postgresql.org; Sun, 31 Aug 2025 17:53:00 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmx.net; s=s31663417; t=1756662777; x=1757267577; i=jimis@gmx.net; bh=fZ9k+slKr8MRW57VD5wNWudyvEHWci2KcyAns95t25k=; h=X-UI-Sender-Class:Date:From:To:CC:Subject:In-Reply-To:References: Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:cc: content-transfer-encoding:content-type:date:from:message-id: mime-version:reply-to:subject:to; b=oEawcMl3V0T4/KRBX8MMO+U4fnNLKjzc5Qd/67Om2tty/9i+Jc3FkrTZOM5YZe6F oRreLDbbzfA4u0Hon8hMmznvYkJtqq1ARCcsH6Odpl9wPh0aAhe3EVCvC0GwcWV2I qUjwiIPeTVthfwFd2EYTUFMB222SL1I6mvoULNqDrp9fiLefNjCsInLrlYbkfXCWs enkxnr3saSnL823NSg3wGZFLB30di7+oWyrOspMhngUDubZi4b5Gw4K2v9CUaS+Ui 8kx30rU1j6rDKKarx9j6hbAior3/brC8Xnjl7ZSJH0/eEDngz0BNnyYtpHRkOm7kX iHhLfWvKpeLUZL8skg== X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a Received: from ehlo.thunderbird.net ([84.215.109.181]) by mail.gmx.net (mrgmx004 [212.227.17.190]) with ESMTPSA (Nemesis) id 1M6DWs-1uqaMy3Gau-008k27; Sun, 31 Aug 2025 19:52:56 +0200 Date: Sun, 31 Aug 2025 19:52:55 +0200 From: Dimitrios Apostolou To: Adrian Klaver , Dimitrios Apostolou CC: pgsql-general@lists.postgresql.org Subject: Re: In-order pg_dump (or in-order COPY TO) User-Agent: K-9 Mail for Android In-Reply-To: References: <3541781s-75o7-26pp-46pp-qs54o4406192@tzk.arg> <4ss66r31-558o-qq24-332q-no351p7n5osr@tzk.arg> <0dc17a73-2372-4613-a50e-610ae7d02b93@aklaver.com> Message-ID: <84F8AAF7-8FC6-463C-827F-5D09E24EA911@gmx.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Provags-ID: V03:K1:WGpRV27iGqC72UYF9jYQwF8QNV1TtB3Jo/lDW+OtutkRdEWEiq/ dxuhwQJ3TrTeDcFiCQ06B5S92BPz1n4aQvP4IlMQBpISvy8t/x/TNnFSz17t+mctHkwcFTY 6TdWj9R+/G+iGae1PHsIfdN8Y/WpFXMSW/VC8ntomRCZXuxx8IhlvQOb+TjE+U0yIKt4ioS xwLW/RrIvOHxPntAmMtRA== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:ySF9qB1nitA=;LmCQu74XeFwXvb1mQ0S1vNd/1jO 53g2TOHFPKKmy6Z/prY+FqxorSMX+uIv5IkFCeJ+l2UzqdTBCTaQ/9tjDuJv5x5rxRv/dW6In Qk1x+KfQEz1MUuxNGkLkq2iAXL7+OjjtB1mr5g4MV1Y6TZXAuKc0M2dNx33rLRKpDwwEiZhi6 AFBCuJFQli/GGccWl7i/AtYQiNTkPD+4ScDwXvM+3fWGo/9sDJD8wbsi+rnCKC+hHSKJrebvj mTDamcycztGKDK70m0rO3se8/JJT7pp9vfux4oGYtxVtweUeHlt/3fbo0s/42QWBIfDmWYLrc 1r0FEqmeZqdniCPEzD1xXI4oYMc4LprIqga8wLoOZH7tRro/OqD+NtB3i92EazHBKlTNRTG+z 1KJl0K9pi5Iyzt8Q+vz+mhZRZVqMDCr8nmKRPEs03a2yxS+N7dM1xy5mU8bx/xVocp9gm5AZC sA5h9Da4BshApkB9Tlyl5axyISJCI2Eh1BLPTmavFYmbtmIEqXs8Mm9l0irRSx5JRPrp6iPDl IGZiN041V1ly75FqQUfbtFBoWPXdCwLkgVSTDph4ZCE+Prc6x5GlyGn/HQ9z0IiG1UgW5yY+E 1Xo9dH/afajjII/avgexHdWWQcAY/Zl6cm8toOjwlT8/19lDDCuRNlsVtl6ZlKs7rNWwjGhcr 2l6Z5rdbDlDNy26cwNYE7f+bNYsJvLusP5m6eKbQnjlkm1JFLO9v6GAefuvuPB7JZFKZupj6C uvHXyx4DrQRqAnq3cd8R5utU3liJ/8kbZBtPlE0PNnH5Vmw1MqNHMrUahb24Obn0J4BvhZQX8 7QTObERUtRlwdx//a5npgFXfYXIYwrOBHsoCop2BdQZRVO/FWdI5hOFjLrIN/45pan0/c1MSG yYWvWSt7uN8n00R1s2jWljXW8+o0pHJhrhFX6lxr1O2Y0OH40GvkF9+WPu0YA6kdsG1knU+eZ CpX9c/Dtsx6uJ/EZmYU05GBvvxdIZ3MwVjIQBFjIFxrhKLgj1VYVzCZbK410SwC6Gke0WtRdo Ds0ffAn3bat6n/X3eB0xJCjILfs80V81Uygpnny2SaTccN9QhVPxnEnaRKr35yVGy6gfYqpIm 10GtUXDYX9G3DCVnIgttqxCT2MGe/ruDUneENLwB1noF78sGPKJ1WBbGM0Bnmt/B7BRlR1S/h uFogYIvWmx7bZQnZLiholEMp0xob6a0wzfFCvqu8Nng1JjR+OUT/AP8/ZgWpvs3qfe2UT3Kwy 2i1iuJipnIDgDa+Fzx1WbjlsiuvRtHKLZ0Irs22zIXYi7LdXosKJZP0WqlGSNNaeSOh1QUQKc ifvZ9Th9sdKzzP71FlE8Mpb47CMujFhzCQxbBgIUpTwoYjlkYwM35+C6IiAFrmpR3QHI9x4Dq 4UtwSkpIjdLJgruMD+HCVGR29sVFPMTGuNYBUEBnz9QIuPLMilSLNiwuFtluZyVHMq3SL0vPH 94rwCR4bH5BlOsm5tmsczbwowrtM2+b1fVEux6mZ95lNPoZFU4ySyPu7y4LCZ4NuszJznedxs Ae0jSukY7MgZbZAOMxxeR+5NS0WRmYDJxTzuyvsbnPlTtppsg6cVriOxJk0ZIMx2CB0pNomJc UIM7t+K2cS7kVm6O+hcINtry1TN+f6L9iV7efL8d/PEgFZ+8dE/w+u5nieAMxpuykI2Zo2099 XyLHwtwo6/baEYEyFcZL95Ucc6j6J2FqFhKSM8KAb3Sc+qGEIy4Zd0JDYHjbuzjqS6ked0rl4 KwE4w2a+hBTBia6+cLl7WM/RrfANuwu4A6xW8prdCKoLJCcBUDiBiunLmMKge7iOq5we+5wiF xPxE7FkVMTm7zeBHu8InqEsFjTGiBwa9v+Jmu0371r5dWIC2mD981HcEOVnoeamnVi9mwxv4/ Md8Sy+ejPvl3KgsiqShN+KjHLzVGrzhvNcfnAyn9NCzKbV6eHktqUJrzfrDkz70FEvPmRVgSu /mvd6zKC9NmherLq6FLXnMrJENsV+Ch8uIMxtxwcTKAIO+dtu+5FvnPAmavh0TXgd7rOThR38 DRSpAfhhi+jeeZpifef7iWavnpKQuyUUKRXMIYviGAeroy+CzTRBuN6WM5sv2XNLEfXMsW+L3 0DELmoXa48wCex07qTsXLo3n4d9tL1u9Qd8SthHcFern0etRaSR1H94ZDHUb8aZYRi1S8d0fZ gny9ICS8C83y2beQk9LjhgI5jA5bNf77oHSRMtraUICdF7sD2t2RizV1sMr88IWk1RNA03sLR J6YDRqyPUa+0f58oN2nRKZvPy6z3tO2l7jOYonAivrhggcSxldoaIhipKee/iRZsRXCMoMcVf wtAtM1e7tufvY7bnyYTMpSfvC+3wjqmNgQHA7mo6E6yD+kiGY7kYcduVqURJWj68v9pdWvTw1 RYjF9IHTSvWegUPiiiDi7Lil26oKYYnWak5gWylCI2QOpjITQgMUtHjrb3w1kKqiNgQCT91Dv jpfNzwaaFfeRb5CScYxnCLZU5/7ScEWvhbbI6W3nN7nJGVPVmiRlujbWuMG/RK9fEiu0gyXGr CbEFBEZRXOxIUKw6uZ1Nu/mxI6ECPWTsARxS+QpOlfd+ubKeeK6SaM7TvxA9IS8eEyUF6HvWh hdxCEBJ5fzq8wN7EGV1/W2j4nifmImefjbHcRynWig7slgBZH5qfPn3sQdxZ8lbFCgWMN1qNV WMhyNwkJ32Qwx83o2fVsJc/kexf05Y7lzmSVG1EbPvCwOXMar2pWL61BCHg/E1VgRUPNz9Vej jb2+lBFz1C5AIAEo9El06MaIKeu2JNv1PNXk8rky296aSvjdX1tFUgdEGM3tRoioLB1b7n724 1MM9wdRluANW7MXxaua7y5mkH6t1UVw9dLY6LJa17nNutfzKs+cRNFMKuKsbuX8buc0o4N61S cBP2BltLs4euzimO9thZkEk7Aw1tu9LWcn7jBlIV/cda8G5azXcxW1fpQ+lvWkmVVkTdGgaiW dMQhExxFkkWq08+vYE1n87cSiLi4NolsYdeWxMNW5pw0BQ66kBNtE8XIXOi7o8UCSt1whc6x9 XHReD26aiMbNUtyXP8X0Qsci4wDrMatmWvm+Df4N9qkVUCHB4oREvtdoEnbITL9isPptUIcHK gIRu4nTfoRP3ws+1yugZ7AcxuIaYGGPBd47Ij2luit0YXJI85+Q63J5vcGkx2VyvU9SETpM2L NlYKVUuzdsVDS6p1cNNL+8a91qC2d9xSyeB2bAs6lYM0oOKU4s0ozbTOf2BKRgWmUs88MrD8S 3GzQS7U2wv4fpBVh3yF3Z6Rh5o814TPQPWfL+SpbTii+rR9j39Bg9TOiiuh2rviNn0H8KB9MF xqE+kWLXVncV2HqUCam3cv8a7OEoaiWXue3Wl2U9EYSDLic7s5Oy8PsOvqn5DVMazGjcLGqIJ ybcrUQbNq2WS+knUqX/hxN9bFxu02hpGfFZOMqZLWUF8v0v/TJdIFY058qyirf5EvlyTXrBrF fruq4pYnviji/tiMEFCttgb7JC3Av9cai6REfZ8wrc3ptcYvflFWGI/O6GQjeeh9WCIeyP6R+ 7/qP44bsunedjYTuwWDc3PorENW6F5/xuGO7xTPoxtnV1uO7s6/fSD5ZRCiA2N8bczP2OpdCs o+XGazmXWmy5geIcS+6OJVUqn97UEhgAIsGK2KN/r2wa15MJ5/Xuj6nmulXdY8tib/vL6sg2F Qw5H6HFPZaHiQrKhVuywilg2gXFURcsotjluS33DY8IIWO8q4dz8azZCg12ByitHFzR4V+OUx gIQHx8lDdwLAq/FxLh0VVo4iF89ETx55Me1Vj4iaucd6Qq3tFYjSDLIgAJXANmZszfrJI6G8u VWynG1kS5BXYxWBB7vQLbxyg3wuyQSf6W/nos0CM57DVnyM70fH8KOF7hxFt0Tqg3dIYsm9rf 2jJPx0b3MKMdX1QVqHfw4hYmfgXOdu68zLC3SM2z7tGcN8xPN+O8UUKNtoRqljyUAaP4sv/Rk yXFQuB8Mla+F4wcWenj/RRP+cjm3XnvLtaqQkp2YSDiYdAUuP/YBfY07i+47B4YtyM210N9Db I96WBNHQbeRxxGd7yOPYvP4t2EyTDOpN7hh8vmdxItRCbBcVswJOraPqRCelnMX14gdAtY4S2 GNEjaDsec8n5XduOSrHAivei/+2hP7+PPZ5aOoyZkqTEPMKwPo+q/EEythuFVvUcY44dNyVu5 +YwVsxckDMVrlTZa1GnTj3dacINn5wYXQK6F4L1Wm2TcC61hzOx9kKaR57Wocyswm80dq/t0T bIFtCfaZ5Ru0hy9yFObCtFZUflA9jB88se3Ta6tsy9r3X/gdxawkGU3UwTNs9tCpCIK0Ix1X5 mv5+uD+mzdcifDyOH4Fyq0aD2GxOVpWIIFmMkZ7WTEvmvNQCc4yvkDwBmAcpKGrp1FEhFAYNL bwWtb8AiyuSv4JxvpzS2DS4b7JUaK1tTe+PdOxacQWDEsoebg7SUFgFgOBt+4ePDkVJGXAlM+ GzstBfSOQwGoaiB2Htqay93NtP7EmdS/ighgcI2djDVWKzgWGaxz5Sf7UEWP50z4NIxrQVIzh uaA5c0CTK9faopWnvXb4Q1bC/4BV6HHPYBVehu70RIitQyvCik5hKp6VU7vRaL11zi2x76658 t7moYFVoqNscjYNwAd7dHOwX212rXWsUOqEx1SSWMzwIkH9R+18d3J1JLZqDmHBcPV9CKkp10 TKquzfwZ2qypnIsuUcEZSfB+2mwSyRsVlsVgU39vf/usv0NnffIcHEX1E8Inw6M4Qb086eQ= List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk When I first said "dump file without TOC" I actually meant "without offsets= in the TOC"=2E The fact that you get a TOC printed does not prove that the dump file incl= udes a TOC with offsets=2E All pg_dump -Fc commands that write to stdout, produce a file without offs= ets in the TOC=2E It has nothing to do with borg=2E ToC offsets must be fil= led in right before streaming each table, but this is impossible when the w= hole TOC has already been written to stdout in the beginning=2E Dimitris On 31 August 2025 17:41:34 CEST, Adrian Klaver wrote: >On 8/30/25 18:21, Dimitrios Apostolou wrote: >> Sorry I was not remembering the details=2E Probably there is a TOC in y= our dump file, but it does not contain any positions for the data=2E The pg= _restore command has to scan the whole file in advance, and fill in the TOC= offsets in memory=2E >>=20 >> This scanning happens in a very inefficient way, with many seek calls a= nd small block reads=2E Try strace to see them=2E This initial phase can ta= ke hours in a huge dump file, before even starting any actual restoration= =2E > >It may be that my coffee is not strong enough, but I don't understand wha= t you are trying to say=2E > >Are you using, from previous post, the following?: > >"Two patches for speeding up scanning an archive without TOC, like the on= e I'm having (because it is piped into borg, instead of written to file)=2E= These were activated, but shouldn't matter=2E They only build the TOC in p= g_restore's memory=2E " > >The part I don't see is how you get a dump file without a TOC? > >When I do the pg_dump and pipe it to Borg the resulting file has a TOC=2E > >Can you show the rest of the | borg create =2E=2E=2E command? > >>=20 >> Thank you for testing=2E >> Dimitris >>=20 >> On 30 August 2025 20:19:13 CEST, Adrian Klaver wrote: > > >