Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ciVdI-0002bs-P2 for pgsql-hackers@arkaria.postgresql.org; Tue, 28 Feb 2017 00:27:12 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.84_2) (envelope-from ) id 1ciVdH-0004Ft-TQ for pgsql-hackers@arkaria.postgresql.org; Tue, 28 Feb 2017 00:27:11 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1ciVbl-0008I6-9E for pgsql-hackers@postgresql.org; Tue, 28 Feb 2017 00:25:37 +0000 Received: from mail.thelabyrinth.net ([45.56.70.56]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1ciVbi-00060h-S3 for pgsql-hackers@postgresql.org; Tue, 28 Feb 2017 00:25:36 +0000 Received: from Davids-iMac.local (unknown [98.204.177.190]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: dsteele) by mail.thelabyrinth.net (Postfix) with ESMTPSA id 9758B5574C for ; Tue, 28 Feb 2017 00:25:33 +0000 (UTC) To: Pg Hackers From: David Steele Subject: PATCH: Make pg_stop_backup() archive wait optional Message-ID: <758e3fd1-45b4-5e28-75cd-e9e7f93a4c02@pgmasters.net> Date: Mon, 27 Feb 2017 19:25:32 -0500 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:45.0) Gecko/20100101 Thunderbird/45.7.1 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="------------4D60BB99EB5563CA9C3317F3" X-Pg-Spam-Score: -1.9 (-) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-hackers Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org This is a multi-part message in MIME format. --------------4D60BB99EB5563CA9C3317F3 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Currently pg_stop_backup() will wait for all WAL to be archived before returning. If there is a problem with the archive command or a large backlog it may not return for a very long time (if ever). Backup software is faced with the choice of cancelling the query and hoping the stop backup record was written or waiting indefinitely. The purpose of this patch is to make waiting for archive optional, with the default being the current behavior, i.e., to wait for all WAL to be archived. This functionality is already used internally by pg_basebackup, so the only real change is to expose it through the pg_stop_backup() function. I wasn't sure where, if anywhere, to put tests (the underlying functionality is tested in the pg_basebackup TAP tests). test/recovery and bin/pg_basebackup did not seem like appropriate places and the pg_regress tests are not capable enough. It seems like a new test/backup suite would be a good idea but I wanted to get some feedback before proceeding in that direction. I also marked the pg_stop_* functions as parallel restricted, the same as pg_start_backup(). Previously they were parallel safe which I don't believe is accurate for the non-exclusive version at the very least, since it is tied to a particular backend. The patch applies cleanly on 30df93f but will be broken (and easily fixed) by any cat version bump in later commits. -- -David david@pgmasters.net --------------4D60BB99EB5563CA9C3317F3 Content-Type: text/plain; charset=UTF-8; x-mac-type="0"; x-mac-creator="0"; name="pgstopbackup-wait-v1.patch" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="pgstopbackup-wait-v1.patch" ZGlmZiAtLWdpdCBhL2RvYy9zcmMvc2dtbC9mdW5jLnNnbWwgYi9kb2Mvc3Jj L3NnbWwvZnVuYy5zZ21sCmluZGV4IDljNTNlNDIuLjY4MGEwZGMgMTAwNjQ0 Ci0tLSBhL2RvYy9zcmMvc2dtbC9mdW5jLnNnbWwKKysrIGIvZG9jL3NyYy9z Z21sL2Z1bmMuc2dtbApAQCAtMTgwNDQsNyArMTgwNDQsNyBAQCBTRUxFQ1Qg c2V0X2NvbmZpZygnbG9nX3N0YXRlbWVudF9zdGF0cycsICdvZmYnLCBmYWxz ZSk7CiAgICAgICA8L3Jvdz4KICAgICAgIDxyb3c+CiAgICAgICAgPGVudHJ5 PgotICAgICAgICA8bGl0ZXJhbD48ZnVuY3Rpb24+cGdfc3RvcF9iYWNrdXAo PHBhcmFtZXRlcj5leGNsdXNpdmU8Lz4gPHR5cGU+Ym9vbGVhbjwvPik8L2Z1 bmN0aW9uPjwvbGl0ZXJhbD4KKyAgICAgICAgPGxpdGVyYWw+PGZ1bmN0aW9u PnBnX3N0b3BfYmFja3VwKDxwYXJhbWV0ZXI+ZXhjbHVzaXZlPC8+IDx0eXBl PmJvb2xlYW48Lz4gPG9wdGlvbmFsPiwgPHBhcmFtZXRlcj53YWl0X2Zvcl9h cmNoaXZlPC8+IDx0eXBlPmJvb2xlYW48Lz4gPC9vcHRpb25hbD4pPC9mdW5j dGlvbj48L2xpdGVyYWw+CiAgICAgICAgIDwvZW50cnk+CiAgICAgICAgPGVu dHJ5Pjx0eXBlPnNldG9mIHJlY29yZDwvdHlwZT48L2VudHJ5PgogICAgICAg IDxlbnRyeT5GaW5pc2ggcGVyZm9ybWluZyBleGNsdXNpdmUgb3Igbm9uLWV4 Y2x1c2l2ZSBvbi1saW5lIGJhY2t1cCAocmVzdHJpY3RlZCB0byBzdXBlcnVz ZXJzIGJ5IGRlZmF1bHQsIGJ1dCBvdGhlciB1c2VycyBjYW4gYmUgZ3JhbnRl ZCBFWEVDVVRFIHRvIHJ1biB0aGUgZnVuY3Rpb24pPC9lbnRyeT4KQEAgLTE4 MTI4LDcgKzE4MTI4LDEzIEBAIHBvc3RncmVzPSMgc2VsZWN0IHBnX3N0YXJ0 X2JhY2t1cCgnbGFiZWxfZ29lc19oZXJlJyk7CiAgICAgPGZ1bmN0aW9uPnBn X3N0YXJ0X2JhY2t1cDwvPi4gSW4gYSBub24tZXhjbHVzaXZlIGJhY2t1cCwg dGhlIGNvbnRlbnRzIG9mCiAgICAgdGhlIDxmaWxlbmFtZT5iYWNrdXBfbGFi ZWw8Lz4gYW5kIDxmaWxlbmFtZT50YWJsZXNwYWNlX21hcDwvPiBhcmUgcmV0 dXJuZWQKICAgICBpbiB0aGUgcmVzdWx0IG9mIHRoZSBmdW5jdGlvbiwgYW5k IHNob3VsZCBiZSB3cml0dGVuIHRvIGZpbGVzIGluIHRoZQotICAgIGJhY2t1 cCAoYW5kIG5vdCBpbiB0aGUgZGF0YSBkaXJlY3RvcnkpLgorICAgIGJhY2t1 cCAoYW5kIG5vdCBpbiB0aGUgZGF0YSBkaXJlY3RvcnkpLiAgVGhlcmUgaXMg YW4gb3B0aW9uYWwgc2Vjb25kCisgICAgcGFyYW1ldGVyIG9mIHR5cGUgYm9v bGVhbi4gIElmIGZhbHNlLCB0aGUgPGZ1bmN0aW9uPnBnX3N0b3BfYmFja3Vw PC8+CisgICAgd2lsbCByZXR1cm4gaW1tZWRpYXRlbHkgYWZ0ZXIgdGhlIGJh Y2t1cCBpcyBjb21wbGV0ZWQgd2l0aG91dCB3YWl0aW5nIGZvcgorICAgIFdB TCB0byBiZSBhcmNoaXZlZC4gIFRoaXMgYmVoYXZpb3IgaXMgb25seSB1c2Vm dWwgZm9yIGJhY2t1cAorICAgIHNvZnR3YXJlIHdoaWNoIGluZGVwZW5kZW50 bHkgbW9uaXRvcnMgV0FMIGFyY2hpdmluZywgb3RoZXJ3aXNlIFdBTAorICAg IHJlcXVpcmVkIHRvIG1ha2UgdGhlIGJhY2t1cCBjb25zaXN0ZW50IG1pZ2h0 IGJlIG1pc3NpbmcgYW5kIG1ha2UgdGhlIGJhY2t1cAorICAgIHVzZWxlc3Mu CiAgICA8L3BhcmE+CiAKICAgIDxwYXJhPgpkaWZmIC0tZ2l0IGEvc3JjL2Jh Y2tlbmQvYWNjZXNzL3RyYW5zYW0veGxvZ2Z1bmNzLmMgYi9zcmMvYmFja2Vu ZC9hY2Nlc3MvdHJhbnNhbS94bG9nZnVuY3MuYwppbmRleCAyN2MwYzU2Li5k OGZkYWNmIDEwMDY0NAotLS0gYS9zcmMvYmFja2VuZC9hY2Nlc3MvdHJhbnNh bS94bG9nZnVuY3MuYworKysgYi9zcmMvYmFja2VuZC9hY2Nlc3MvdHJhbnNh bS94bG9nZnVuY3MuYwpAQCAtMTkwLDYgKzE5MCw3IEBAIHBnX3N0b3BfYmFj a3VwX3YyKFBHX0ZVTkNUSU9OX0FSR1MpCiAJYm9vbAkJbnVsbHNbM107CiAK IAlib29sCQlleGNsdXNpdmUgPSBQR19HRVRBUkdfQk9PTCgwKTsKKwlib29s CQl3YWl0Zm9yYXJjaGl2ZSA9IFBHX0dFVEFSR19CT09MKDEpOwogCVhMb2dS ZWNQdHIJc3RvcHBvaW50OwogCiAJLyogY2hlY2sgdG8gc2VlIGlmIGNhbGxl ciBzdXBwb3J0cyB1cyByZXR1cm5pbmcgYSB0dXBsZXN0b3JlICovCkBAIC0y MzIsNyArMjMzLDcgQEAgcGdfc3RvcF9iYWNrdXBfdjIoUEdfRlVOQ1RJT05f QVJHUykKIAkJICogU3RvcCB0aGUgZXhjbHVzaXZlIGJhY2t1cCwgYW5kIHNp bmNlIHdlJ3JlIGluIGFuIGV4Y2x1c2l2ZSBiYWNrdXAKIAkJICogcmV0dXJu IE5VTEwgZm9yIGJvdGggYmFja3VwX2xhYmVsIGFuZCB0YWJsZXNwYWNlX21h cC4KIAkJICovCi0JCXN0b3Bwb2ludCA9IGRvX3BnX3N0b3BfYmFja3VwKE5V TEwsIHRydWUsIE5VTEwpOworCQlzdG9wcG9pbnQgPSBkb19wZ19zdG9wX2Jh Y2t1cChOVUxMLCB3YWl0Zm9yYXJjaGl2ZSwgTlVMTCk7CiAJCWV4Y2x1c2l2 ZV9iYWNrdXBfcnVubmluZyA9IGZhbHNlOwogCiAJCW51bGxzWzFdID0gdHJ1 ZTsKQEAgLTI1MCw3ICsyNTEsNyBAQCBwZ19zdG9wX2JhY2t1cF92MihQR19G VU5DVElPTl9BUkdTKQogCQkgKiBTdG9wIHRoZSBub24tZXhjbHVzaXZlIGJh Y2t1cC4gUmV0dXJuIGEgY29weSBvZiB0aGUgYmFja3VwIGxhYmVsCiAJCSAq IGFuZCB0YWJsZXNwYWNlIG1hcCBzbyB0aGV5IGNhbiBiZSB3cml0dGVuIHRv IGRpc2sgYnkgdGhlIGNhbGxlci4KIAkJICovCi0JCXN0b3Bwb2ludCA9IGRv X3BnX3N0b3BfYmFja3VwKGxhYmVsX2ZpbGUtPmRhdGEsIHRydWUsIE5VTEwp OworCQlzdG9wcG9pbnQgPSBkb19wZ19zdG9wX2JhY2t1cChsYWJlbF9maWxl LT5kYXRhLCB3YWl0Zm9yYXJjaGl2ZSwgTlVMTCk7CiAJCW5vbmV4Y2x1c2l2 ZV9iYWNrdXBfcnVubmluZyA9IGZhbHNlOwogCQljYW5jZWxfYmVmb3JlX3No bWVtX2V4aXQobm9uZXhjbHVzaXZlX2Jhc2VfYmFja3VwX2NsZWFudXAsIChE YXR1bSkgMCk7CiAKZGlmZiAtLWdpdCBhL3NyYy9iYWNrZW5kL2NhdGFsb2cv c3lzdGVtX3ZpZXdzLnNxbCBiL3NyYy9iYWNrZW5kL2NhdGFsb2cvc3lzdGVt X3ZpZXdzLnNxbAppbmRleCAzOGJlOWNmLi5jMmNhMmI4IDEwMDY0NAotLS0g YS9zcmMvYmFja2VuZC9jYXRhbG9nL3N5c3RlbV92aWV3cy5zcWwKKysrIGIv c3JjL2JhY2tlbmQvY2F0YWxvZy9zeXN0ZW1fdmlld3Muc3FsCkBAIC05ODQs NiArOTg0LDEyIEBAIENSRUFURSBPUiBSRVBMQUNFIEZVTkNUSU9OCiAgIFJF VFVSTlMgcGdfbHNuIFNUUklDVCBWT0xBVElMRSBMQU5HVUFHRSBpbnRlcm5h bCBBUyAncGdfc3RhcnRfYmFja3VwJwogICBQQVJBTExFTCBSRVNUUklDVEVE OwogCitDUkVBVEUgT1IgUkVQTEFDRSBGVU5DVElPTiBwZ19zdG9wX2JhY2t1 cCAoCisgICAgICAgIGV4Y2x1c2l2ZSBib29sZWFuLCB3YWl0X2Zvcl9hcmNo aXZlIGJvb2xlYW4gREVGQVVMVCB0cnVlLAorICAgICAgICBPVVQgbHNuIHBn X2xzbiwgT1VUIGxhYmVsZmlsZSB0ZXh0LCBPVVQgc3BjbWFwZmlsZSB0ZXh0 KQorICBSRVRVUk5TIFNFVE9GIHJlY29yZCBTVFJJQ1QgVk9MQVRJTEUgTEFO R1VBR0UgaW50ZXJuYWwgYXMgJ3BnX3N0b3BfYmFja3VwJworICBQQVJBTExF TCBSRVNUUklDVEVEOworCiAtLSBsZWdhY3kgZGVmaW5pdGlvbiBmb3IgY29t cGF0aWJpbGl0eSB3aXRoIDkuMwogQ1JFQVRFIE9SIFJFUExBQ0UgRlVOQ1RJ T04KICAganNvbl9wb3B1bGF0ZV9yZWNvcmQoYmFzZSBhbnllbGVtZW50LCBm cm9tX2pzb24ganNvbiwgdXNlX2pzb25fYXNfdGV4dCBib29sZWFuIERFRkFV TFQgZmFsc2UpCkBAIC0xMDg0LDcgKzEwOTAsNyBAQCBBUyAnanNvbmJfaW5z ZXJ0JzsKIC0tIGF2YWlsYWJsZSB0byBzdXBlcnVzZXIgLyBjbHVzdGVyIG93 bmVyLCBpZiB0aGV5IGNob29zZS4KIFJFVk9LRSBFWEVDVVRFIE9OIEZVTkNU SU9OIHBnX3N0YXJ0X2JhY2t1cCh0ZXh0LCBib29sZWFuLCBib29sZWFuKSBG Uk9NIHB1YmxpYzsKIFJFVk9LRSBFWEVDVVRFIE9OIEZVTkNUSU9OIHBnX3N0 b3BfYmFja3VwKCkgRlJPTSBwdWJsaWM7Ci1SRVZPS0UgRVhFQ1VURSBPTiBG VU5DVElPTiBwZ19zdG9wX2JhY2t1cChib29sZWFuKSBGUk9NIHB1YmxpYzsK K1JFVk9LRSBFWEVDVVRFIE9OIEZVTkNUSU9OIHBnX3N0b3BfYmFja3VwKGJv b2xlYW4sIGJvb2xlYW4pIEZST00gcHVibGljOwogUkVWT0tFIEVYRUNVVEUg T04gRlVOQ1RJT04gcGdfY3JlYXRlX3Jlc3RvcmVfcG9pbnQodGV4dCkgRlJP TSBwdWJsaWM7CiBSRVZPS0UgRVhFQ1VURSBPTiBGVU5DVElPTiBwZ19zd2l0 Y2hfd2FsKCkgRlJPTSBwdWJsaWM7CiBSRVZPS0UgRVhFQ1VURSBPTiBGVU5D VElPTiBwZ193YWxfcmVwbGF5X3BhdXNlKCkgRlJPTSBwdWJsaWM7CmRpZmYg LS1naXQgYS9zcmMvaW5jbHVkZS9jYXRhbG9nL2NhdHZlcnNpb24uaCBiL3Ny Yy9pbmNsdWRlL2NhdGFsb2cvY2F0dmVyc2lvbi5oCmluZGV4IDllOGMxYzIu LmRmNDdlZWEgMTAwNjQ0Ci0tLSBhL3NyYy9pbmNsdWRlL2NhdGFsb2cvY2F0 dmVyc2lvbi5oCisrKyBiL3NyYy9pbmNsdWRlL2NhdGFsb2cvY2F0dmVyc2lv bi5oCkBAIC01Myw2ICs1Myw2IEBACiAgKi8KIAogLyoJCQkJCQkJeXl5eW1t ZGROICovCi0jZGVmaW5lIENBVEFMT0dfVkVSU0lPTl9OTwkyMDE3MDIyMzEK KyNkZWZpbmUgQ0FUQUxPR19WRVJTSU9OX05PCTIwMTcwMjI3MQogCiAjZW5k aWYKZGlmZiAtLWdpdCBhL3NyYy9pbmNsdWRlL2NhdGFsb2cvcGdfcHJvYy5o IGIvc3JjL2luY2x1ZGUvY2F0YWxvZy9wZ19wcm9jLmgKaW5kZXggYTRjYzg2 ZC4uNjg3MzVjMyAxMDA2NDQKLS0tIGEvc3JjL2luY2x1ZGUvY2F0YWxvZy9w Z19wcm9jLmgKKysrIGIvc3JjL2luY2x1ZGUvY2F0YWxvZy9wZ19wcm9jLmgK QEAgLTMxNDEsOSArMzE0MSw5IEBAIERBVEEoaW5zZXJ0IE9JRCA9IDIwOTYg KCBwZ190ZXJtaW5hdGVfYmFja2VuZAkJUEdOU1AgUEdVSUQgMTIgMSAwIDAg MCBmIGYgZiBmIHQKIERFU0NSKCJ0ZXJtaW5hdGUgYSBzZXJ2ZXIgcHJvY2Vz cyIpOwogREFUQShpbnNlcnQgT0lEID0gMjE3MiAoIHBnX3N0YXJ0X2JhY2t1 cAkJUEdOU1AgUEdVSUQgMTIgMSAwIDAgMCBmIGYgZiBmIHQgZiB2IHIgMyAw IDMyMjAgIjI1IDE2IDE2IiBfbnVsbF8gX251bGxfIF9udWxsXyBfbnVsbF8g X251bGxfIHBnX3N0YXJ0X2JhY2t1cCBfbnVsbF8gX251bGxfIF9udWxsXyAp KTsKIERFU0NSKCJwcmVwYXJlIGZvciB0YWtpbmcgYW4gb25saW5lIGJhY2t1 cCIpOwotREFUQShpbnNlcnQgT0lEID0gMjE3MyAoIHBnX3N0b3BfYmFja3Vw CQkJUEdOU1AgUEdVSUQgMTIgMSAwIDAgMCBmIGYgZiBmIHQgZiB2IHMgMCAw IDMyMjAgIiIgX251bGxfIF9udWxsXyBfbnVsbF8gX251bGxfIF9udWxsXyBw Z19zdG9wX2JhY2t1cCBfbnVsbF8gX251bGxfIF9udWxsXyApKTsKK0RBVEEo aW5zZXJ0IE9JRCA9IDIxNzMgKCBwZ19zdG9wX2JhY2t1cAkJCVBHTlNQIFBH VUlEIDEyIDEgMCAwIDAgZiBmIGYgZiB0IGYgdiByIDAgMCAzMjIwICIiIF9u dWxsXyBfbnVsbF8gX251bGxfIF9udWxsXyBfbnVsbF8gcGdfc3RvcF9iYWNr dXAgX251bGxfIF9udWxsXyBfbnVsbF8gKSk7CiBERVNDUigiZmluaXNoIHRh a2luZyBhbiBvbmxpbmUgYmFja3VwIik7Ci1EQVRBKGluc2VydCBPSUQgPSAy NzM5ICggcGdfc3RvcF9iYWNrdXAJCQlQR05TUCBQR1VJRCAxMiAxIDEgMCAw IGYgZiBmIGYgdCB0IHYgcyAxIDAgMjI0OSAiMTYiICJ7MTYsMzIyMCwyNSwy NX0iICJ7aSxvLG8sb30iICJ7ZXhjbHVzaXZlLGxzbixsYWJlbGZpbGUsc3Bj bWFwZmlsZX0iIF9udWxsXyBfbnVsbF8gcGdfc3RvcF9iYWNrdXBfdjIgX251 bGxfIF9udWxsXyBfbnVsbF8gKSk7CitEQVRBKGluc2VydCBPSUQgPSAyNzM5 ICggcGdfc3RvcF9iYWNrdXAJCQlQR05TUCBQR1VJRCAxMiAxIDEgMCAwIGYg ZiBmIGYgdCB0IHYgciAyIDAgMjI0OSAiMTYgMTYiICJ7MTYsMTYsMzIyMCwy NSwyNX0iICJ7aSxpLG8sbyxvfSIgIntleGNsdXNpdmUsd2FpdF9mb3JfYXJj aGl2ZSxsc24sbGFiZWxmaWxlLHNwY21hcGZpbGV9IiBfbnVsbF8gX251bGxf IHBnX3N0b3BfYmFja3VwX3YyIF9udWxsXyBfbnVsbF8gX251bGxfICkpOwog REVTQ1IoImZpbmlzaCB0YWtpbmcgYW4gb25saW5lIGJhY2t1cCIpOwogREFU QShpbnNlcnQgT0lEID0gMzgxMyAoIHBnX2lzX2luX2JhY2t1cAkJUEdOU1Ag UEdVSUQgMTIgMSAwIDAgMCBmIGYgZiBmIHQgZiB2IHMgMCAwIDE2ICIiIF9u dWxsXyBfbnVsbF8gX251bGxfIF9udWxsXyBfbnVsbF8gcGdfaXNfaW5fYmFj a3VwIF9udWxsXyBfbnVsbF8gX251bGxfICkpOwogREVTQ1IoInRydWUgaWYg c2VydmVyIGlzIGluIG9ubGluZSBiYWNrdXAiKTsK --------------4D60BB99EB5563CA9C3317F3 Content-Type: text/plain Content-Disposition: inline Content-Transfer-Encoding: 8bit MIME-Version: 1.0 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers --------------4D60BB99EB5563CA9C3317F3--