Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iBKm9-0005cA-3n for pgsql-hackers@arkaria.postgresql.org; Fri, 20 Sep 2019 15:24:49 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1iBKm7-0000sY-U3 for pgsql-hackers@arkaria.postgresql.org; Fri, 20 Sep 2019 15:24:47 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iBKiv-0005hP-Rr for pgsql-hackers@lists.postgresql.org; Fri, 20 Sep 2019 15:21:29 +0000 Received: from mail.thelabyrinth.net ([45.56.70.56]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iBKis-0006xT-HX for pgsql-hackers@postgresql.org; Fri, 20 Sep 2019 15:21:29 +0000 Received: from Davids-iMac.local (c-73-39-102-148.hsd1.md.comcast.net [73.39.102.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: dsteele) by mail.thelabyrinth.net (Postfix) with ESMTPSA id F15F354683; Fri, 20 Sep 2019 15:21:24 +0000 (UTC) Subject: Re: backup manifests To: Robert Haas Cc: "pgsql-hackers@postgresql.org" References: <4b6244ea-4094-9055-080b-7afb274bf270@pgmasters.net> From: David Steele Openpgp: preference=signencrypt Autocrypt: addr=david@pgmasters.net; prefer-encrypt=mutual; keydata= mQINBFSheh0BEADGNb3c1kOaX5OGUDhqHTwJ3JW7XAllfWLdknpRg+/fihgLPPRnQHZhjq28 nmMLqM+C8TjezT2BgVACeu8qqASnuzN/E5+bAl9hVcgCP4eV/lEIWwVxmhcHqSdeWqWZSBRB tf07fLBnz22+3LwVsESN+fanWxnoqqvE0+hyGnCEDhOBkihgbYHuq4vw1+xgoGN1uDk4NShV El0ZWO90P4e5mOWA2Zsex+8YhGJlfRwMpU69okFCvjpWJKE+ufeJdlrxEpSSKW7SQ0kG1VF7 CvDKpF70TCWkb4Bagb6c7lwZrMjxuBnlIvOlV3RRY67PAaVIFRRvQJeJABOPHp8HnH8cwici 7gwwfWBrZZeAdPqs6VNjANqI9uaDasRnD3yHP6CcHIAeCl5hYnzNTkIqBRlnmD5y8s0VRDLr 6JR3FxscOsvwXPJ+He1RpRlxQDlBboaRFUz8endeQHhGUeHyekQcSYUaz4PX3OJBG0Xi7nr8 EOP6zOeP61z3dL8NuO3QjgPiygvS0qwMEPvOYo52nEGYn08nvFVajI8yxOXFnIwnmpgr5gdq nL9ga72tvSyJQAUx+YB7r1bKjRn/Mlt/fPIS5PXONofiO1K8uX+pzvIX6dnm7ihpFf1XC+uG 5+DKBnU7yS7jCRL52GxWHbuC3mT4tvzqH2b13TLng1Y7Izi+hwARAQABtCJEYXZpZCBTdGVl bGUgPGRhdmlkQHBnbWFzdGVycy5uZXQ+iQI5BBMBCAAjAhsDAh4BAheABQsJCAcDBRUKCQgL BBYCAwEFAlShuEECGQEACgkQn830JKJznPPsLhAAtKkIu0mFfxcVqApg4PNjZKYY4TWeN+I+ go/VKX8QWMadr33t8Sxcgyn48bjBYzgWQXhF4rMcFxDxI3ly5bdE8QKZkhFJJT7N2fBoo9Zm ZOsOvPd+5hz07VZpgNSiTXMAphj8+GSTLbYufm/OUnfpS1eovPtPdYPQpNC3RVHA0gtisz26 NgMLg6QN0iQuITs9N+wiaGtyVIornl4XAY2dM/Hvfr1aDFkukNNr52sc/mGqocG/dKC7dONA 3860y2MLszQcbAE45iEkmwpGkvxyc3i3tvdD3VNlVI1MKJUBnxFPUEI0lVq5WpApFSIU31+j aNunKIcfVxRnM8FTJBsw3RnFRpSsEtiLaZRm5sOZHa9+dy/DWDb+8O1I+momCo88+Jksd6pZ 2Tz9msOiJiTnFsCZgfbbmayH6w8ZyfHuO0DuxScM1UK+6CLC2VVjQFuH6WHqJpaENg5INswS qpUc05iC4i+6dZOe44cviX6LpHuP8vWM0E63CMun0l42yHWy+k6nLvx/fj7t++tF2JIJ92OG n/2MkM5lIKknRg27UjYAspD6f3NSZUMIe6tGyWPTtLOu8zOL5CeRyCHMWwUVDKJSBzlK2mV8 e0EbmtUJFsoNWXWKPKUIxX7JXPN3v523HmOhtyz5SnkGtwEqCZiv/JccYmh+S2IokAuscLNP EJq5Ag0EVKF6HQEQALPpUsP67xfcj6NUMyySXSytUzMIVi5XNTyazuNcVfFZzT9QL8wFQkgy i/HOD3tmvdGxz6H8V+v+cszp88gCd8TwClss7u+iPsLlAQQWC4q0WiX4OhwFv2H004iuft8i odjiWDkYEgIFxlPNxQfHFmchkyODIJXQmnzbgsXwGZtJzAxEVh6AgYJqE/Ev/+kTJhJ7703n oIb+PXvafoB5axnU4nf7DuFa9X+q2/ZfzyAXUCKvzDvxb24iUgWFzlWBx0CnI21Kh6aS6hR6 o00D2OBiU+yCpW9eRTi3SuDuslmtCUaZz5V4SHo37ochEYfZ2LYOPsBO3PUMx8T7MDYrmv/x 4W5rdjugz4NXWROStNNLpS74LW20rdrEKMDltph7U2zfjjS663K+CUxoXn/ha2WnzBocoY5o KwdViLln4s0ivOTa4clHJJUsYiAxLShAe2AP+5kJim24tAabbEOsrRTn9RpT7J8jVT/G0YDE glujEt0U04Akkj5NO4G7vJP/AyX5zKZetkp/HFiFBZMkUuXD/KgDc5EM2fpQtaxGGY2fcQqY VHUbbxstxyl5d349M9rok7O90hw9BkP0Q8UEpgmlZ6ftwx+ZfZmBEdQ7HcMHnSPSNJkXt/Vi ozlcrSnkVXzCXvH9DHrIpgenp8btsJ2XyjSdrEu4jAqYeirW+3qRABEBAAGJAh8EGAEIAAkF AlSheh0CGwwACgkQn830JKJznPNmERAApYaQK6i9F6hM2RUD/PqQC45WmzOXK9bDSS9hdk4p Nc5jSW2+FkcMFe1XFG2NyGf1v5KBPxuDTQp35PIwRsL59xNmPICu0ae6zmCPUpxbMIEzb/VU wXqeuNp3cRVll06goDt5w+2LlAtBrxuDjiWxby9LH+vsIUMLWYu+M50pdtOcXOsO1+zyfs3w U6xRZ3PaFDKO1cB68okDFG/cFHnYfY5aETV7LgVBHEOQLwL01U01QC/iqY/aNO4iUH/+PBc0 R6mYlP7IqDQOQVTJ0dXP365thl0Q7c9e2JQSS9wuSv2EEiR//nnZREtLyrQNLwsXE8UFBDe7 uA8Lm2ldmQYpHHKAcorGVTHYJaaus2LjH8tZ1i4RGg4hKMJSHil/5C+Ghl+PmcACVeGaxB0c mJ8+0jyWDkLCDMAXJJyvXRWIONTAxVt5kKg1/iMAq1IQYNd66agnmgxdSvKLPUo0JJ5fhQJp 2bqDDTrINTZg+Rub2HeoOMxYJsMypK+6aY4iVlm2Z7A9SgyQDjGAXzD9+XaezNFeXxfPbjo6 46qA9SycdYhiwaTcTQakiLkZWRIkSjGQwNCK9NktE1fYnXfR0+WaWFduQ4BfM5uta1bt8esB /mLEjVYS9RRAWZ/untJrR9JYGTC4Mu6tcLGZD7385yX3TNM1iaIUGxlKkHHCZCnILyq5Ag0E VKGipwEQALe8Af9Kgql5rC6NPrOeuOuuqwXMNIwKsTLG8PmmoHKEHKukYuX/YfnGWFU1Ir6t y0qXe2XZs8AnLNI753b/7Z7ftMuMWzWqzfU8Pb+8/NrOQqfO05++gtsczY/tflVW2YWJWDZ4 lhytY4frlOSRuMOI4TjM3NwgkVri4sM4U+OdM8tdkW9IdajLCP0J+wFO+U2dU9b3yCss0GNY c0dlbmI6sOmpV70ldAb8i0BqfkGpq3xDcnzf4EQDdF/mBLZwBe0v6dzlxHh2HqL9yztjGXnk NtRFYGzKw+ZBuOaSj+OPG7iaBSgeNxDYjww2KTzdSE3DbfGqwrxn/ejD+ZueNcqxsoaT3oNr ADeluXmN63C1CcdFmJjAiCCjt3diLgmVL3MyBV+LcSwEz+2B/tlRHV1FWvy8VX1Nj2CwCO9u RuJ9HrF05TXjjNC2o6fFME/TkEzsklzvLxP/Bs/wKDvZQKGlzshk8EsmRhnJjzekOg4gOyVM LvjbGiBPMh309i0nVgExz0LIvI5Wp8G1YXHiuf6i52pTgeiGfiV6D12b69ABqInOfvROSvfe F36OLmoRicJcd8wEwkNPoSHHn/Zxf0fMHe7ZV7EtNSeqB+179SxDqeIEAOhpye6/q57LqvpZ qm0ADa5O3J6IMpId+ov0el553e7HxfDmaWJpm+NemLWvABEBAAGJBD4EGAEIAAkFAlShoqcC GwICKQkQn830JKJznPPBXSAEGQEIAAYFAlShoqcACgkQgDa4glDlIUCXfQ//aozIciE9Je5m kUlKECjXut8vvpO9fbAbxY5VIEWANV2PpIw1LO/8WVstCl38vBZtOxCcZqccLge12Ow72WPF 5G+v+8OJRwHNjrdI/WvFeMUfqlWB2ok+OOlm9t4U0lTK7YX4K6PvjXiMh1Kpvx/JsxrnGJJx X+Ky4icNWdiyaVBs+NDROasZXmy+7S8yeR9ECPxu/DPziC+aPW5lJ3tGqFEPADpmeD8G5K7U QyDJ4GRgD+Agez8r60qyhiwFyUXfcn0bOsIx/og4gae8GnadPl1+93+PI0L8Xb+oYcJnvVGE zJZJIMsp1Zjhi3PnJEulvDNOxkY7Og2dEwkNSHrru3Z5odl8s629VaYyT1b0Q/GnDAnKtZc3 tKZVKCvvAf9UmbuBfbuP084dBFeMeFWl3ET9UWpGhKm22KXg8TrMKoM62bUSjW5gIxC8ywLT mu7ivcKbxj0rXFB5AdYUff9QsCZFdDZG3fUymOQmYgNhwXAAIwKDKD8CINc2CIVc0qMhh3Hy WyyA8JZKw/iK+652WHoKEMmxVunSvvlDIMDf/bYN/HxoUEBcCSWsISKW6xGu1hPKfYK2t6/9 CVjluTOPeFiBt1ELXsOlpRPe4f0M2Cc/AEFukE1ZoJWlUhtU4zuOpvDtGST9D9imYsgMaMa8 TtEGFWGDmOfws4yU4tTTUNqFRRAArysPyXEhNNbLncfR5lPGPfZORPqLmIdkNMcDAwmzvC80 g1WH46wPzIcUUTc7DnP0pZNWGmI2NAnAW/c4Ii3aiXQXZDNVRfdZKL5rEY6dtgxdHl4hc6KN KH17wsX4T3dglMqP7VikYT23sNQJVE1oIGjfVR1sOe+DleI5Awa50DgGyLZS9vLfZz/9F28J 2YM1DExVPxbj75bwfOl65hh8B5BCqqrYp5n/JqV2x5+OSPgCHmd7rJtTbTl89MVKc0uR79ZZ hxUBU6m5v2UGHakSlgdbh+qmr6rdFkqRxRFhdepD7u9qvPKHdVIPp23dqkx9r7gIe+RjfkT6 aq12wQhqgGYaae5bqlg0ls1CpfZxsxJmwmO/DJ2W1hdrPhs/7ml9rv552FBQuQ7i9rddeSju lkFsXVWOsbLf45pKAWTwqZihb3nSIoBv53EH0V+1SPzDbgQk2CaQ+DR6k7eujgsI/srPpMtl ehqEQ7kzaKFACBgXBOxkheDR/kJzVJxx0a1rIFnvJGL/h5lgo8onxP5scCpotNx/bJjLkBVk XQvKDUIu9YPy0yENT58C91WWX/Nac3B5W5ZkmWHh6Lh+5eOiRSLV9MDau6ziULNNpuIA90iY 1onr0xRNO2DMyn0NzZzL+Y59WlMyATUcQdTgtmlmTcaxWeRiBRlUBLvHvgHLV94= Message-ID: <2cec49b2-eeab-574f-2b5e-1f437b2cbec5@pgmasters.net> Date: Fri, 20 Sep 2019 11:21:24 -0400 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk On 9/20/19 10:59 AM, Robert Haas wrote: > On Fri, Sep 20, 2019 at 9:46 AM Robert Haas wrote: >> - appendStringInfo et. al. I don't think it would be that hard to move >> this to src/common, but I'm also not sure it really solves the >> problem, because StringInfo has a 1GB limit, and there's no rule at >> all that a backup manifest has got to be less than 1GB. > > Hmm. That's actually going to be a problem on the server side, no > matter what we do on the client side. We have to send the manifest > after we send everything else, so that we know what we sent. But if we > sent a lot of files, the manifest might be really huge. I had been > thinking that we would generate the manifest on the server and send it > to the client after everything else, but maybe this is an argument for > generating the manifest on the client side and writing it > incrementally. That would require the client to peek at the contents > of every tar file it receives all the time, which it currently doesn't > need to do, but it does peek inside them a little bit, so maybe it's > OK. > > Another alternative would be to have the server spill the manifest in > progress to a temp file and then stream it from there to the client. This seems reasonable to me. We keep an in-memory representation which is just an array of structs and is fairly compact -- 1 million files uses ~150MB of memory. We just format and stream this to storage when saving. Saving is easier than loading, of course. -- -David david@pgmasters.net