Received: from localhost (maia-1.hub.org [200.46.204.191]) by postgresql.org (Postfix) with ESMTP id 4C27F9FA456; Mon, 7 May 2007 09:47:16 -0300 (ADT) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.191]) (amavisd-maia, port 10024) with ESMTP id 48195-02; Mon, 7 May 2007 09:47:12 -0300 (ADT) X-Greylist: from auto-whitelisted by SQLgrey-1.7.4 X-Greylist: from auto-whitelisted by SQLgrey-1.7.4 Received: from serv1.oss.ntt.co.jp (ns.oss.ntt.co.jp [222.151.198.98]) by postgresql.org (Postfix) with ESMTP id 9607B9FA431; Mon, 7 May 2007 09:47:11 -0300 (ADT) Received: from [172.17.1.90] (unknown [172.17.1.90]) by serv1.oss.ntt.co.jp (Postfix) with ESMTP id 7F9FF104001; Mon, 7 May 2007 21:47:09 +0900 (JST) Message-ID: <463F1F32.5070707@oss.ntt.co.jp> Date: Mon, 07 May 2007 21:44:34 +0900 From: Koichi Suzuki User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) MIME-Version: 1.0 To: Tom Lane CC: Josh Berkus , Zeugswetter Andreas ADI SD , pgsql-hackers@postgresql.org, pgsql-patches@postgresql.org Subject: Re: [PATCHES] Full page writes improvement, code update References: <460886E8.6040907@oss.ntt.co.jp> <200704240831.42873.josh@agliodbs.com> <200704251127.13624.josh@agliodbs.com> <6292.1177538104@sss.pgh.pa.us> In-Reply-To: <6292.1177538104@sss.pgh.pa.us> Content-Type: multipart/mixed; boundary="------------080508060806030601030809" X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200705/324 X-Sequence-Number: 102926 This is a multi-part message in MIME format. --------------080508060806030601030809 Content-Type: text/plain; charset=ISO-2022-JP Content-Transfer-Encoding: 7bit Hi, As replied to "Patch queue triage" by Tom, here's simplified patch to mark WAL record as "compressable", with no increase in WAL itself. Compression/decompression commands will be posted separately to PG Foundary for further review. ------------------------------- As suggested by Tom, I agree that WAL should not include "both" full page write and incremental (logical) log. I began to examine WAL record format to see if incremental log can be made from full page writes. It will be okay even before 8.4, if simplified patch to the core is accepted. I will post simplified patch to the core as follows: 1. Mark the flag to indicate that the WAL record is compressable from full page writes to incremental log. This flag will be set if a) It is not written during the hot backup, and b) Archive command is active, and c) WAL contains full page writes, and d) full_page_writes=on. No logical log will be written to WAL in this case, and 2. During recovery, xl_tot_len check will be skipped for compressed WAL records. Please note that new GUC is not needed in this patch. With this patch, compress/decompress can be developped outside the core. I'd be very grateful if this patch can be considered again. Best Regards; -- ------------- Koichi Suzuki --------------080508060806030601030809 Content-Type: text/plain; name="lesslog_core.patch_flagonly" Content-Transfer-Encoding: base64 Content-Disposition: inline; filename="lesslog_core.patch_flagonly" ZGlmZiAtY3IgcGdzcWxfb3JnL3NyYy9iYWNrZW5kL2FjY2Vzcy90cmFuc2FtL3hsb2cuYyBw Z3NxbC9zcmMvYmFja2VuZC9hY2Nlc3MvdHJhbnNhbS94bG9nLmMKKioqIHBnc3FsX29yZy9z cmMvYmFja2VuZC9hY2Nlc3MvdHJhbnNhbS94bG9nLmMJMjAwNy0wNS0wMiAxNTo1NjozOC4w MDAwMDAwMDAgKzA5MDAKLS0tIHBnc3FsL3NyYy9iYWNrZW5kL2FjY2Vzcy90cmFuc2FtL3hs b2cuYwkyMDA3LTA1LTA3IDE2OjMwOjM4LjAwMDAwMDAwMCArMDkwMAoqKioqKioqKioqKioq KioKKioqIDgzNyw4NDIgKioqKgotLS0gODM3LDg1NCAtLS0tCiAgCQlyZXR1cm4gUmVjUHRy OwogIAl9CiAgCisgCS8qCisgCSAqIElmIG9ubGluZSBiYWNrdXAgaXMgbm90IGluIHByb2dy ZXNzIGFuZCBXQUwgYXJjaGl2aW5nIGlzIGFjdGl2ZSwgbWFyaworIAkgKiBiYWNrdXAgYmxv Y2tzIHJlbW92YWJsZSBpZiBhbnkuCisgCSAqIFRoaXMgbWFyayB3aWxsIGJlIHJlZmVyZW5j ZWQgZHVyaW5nIGFyY2hpdmluZyB0byByZW1vdmUgbmVlZGxlc3MgYmFja3VwCisgCSAqIGJs b2NrcyBpbiB0aGUgcmVjb3JkIGFuZCBjb21wcmVzcyBXQUwgc2VnbWVudCBmaWxlcy4KKyAJ ICovCisgCWlmIChYTG9nQXJjaGl2aW5nQWN0aXZlKCkgJiYgZnVsbFBhZ2VXcml0ZXMgJiYK KyAJCQkoaW5mbyAmIFhMUl9CS1BfQkxPQ0tfTUFTSykgJiYgIUluc2VydC0+Zm9yY2VQYWdl V3JpdGVzKQorIAl7CisgCQlpbmZvIHw9IFhMUl9CS1BfUkVNT1ZBQkxFOworIAl9CisgCiAg CS8qIEluc2VydCByZWNvcmQgaGVhZGVyICovCiAgCiAgCXJlY29yZCA9IChYTG9nUmVjb3Jk ICopIEluc2VydC0+Y3VycnBvczsKKioqKioqKioqKioqKioqCioqKiAyNzM4LDI3NTAgKioq KgogIAkJYmxrICs9IGJsZW47CiAgCX0KICAKISAJLyogQ2hlY2sgdGhhdCB4bF90b3RfbGVu IGFncmVlcyB3aXRoIG91ciBjYWxjdWxhdGlvbiAqLwohIAlpZiAoYmxrICE9IChjaGFyICop IHJlY29yZCArIHJlY29yZC0+eGxfdG90X2xlbikKICAJewohIAkJZXJlcG9ydChlbW9kZSwK ISAJCQkJKGVycm1zZygiaW5jb3JyZWN0IHRvdGFsIGxlbmd0aCBpbiByZWNvcmQgYXQgJVgv JVgiLAohIAkJCQkJCXJlY3B0ci54bG9naWQsIHJlY3B0ci54cmVjb2ZmKSkpOwohIAkJcmV0 dXJuIGZhbHNlOwogIAl9CiAgCiAgCS8qIEZpbmFsbHkgaW5jbHVkZSB0aGUgcmVjb3JkIGhl YWRlciAqLwotLS0gMjc1MCwyNzc4IC0tLS0KICAJCWJsayArPSBibGVuOwogIAl9CiAgCiEg CS8qCiEgCSAqIElmIHBoeXNpY2FsIGxvZyBoYXMgbm90IGJlZW4gcmVtb3ZlZCwgY2hlY2sg dGhlIGxlbmd0aCB0byBzZWUKISAJICogdGhlIGZvbGxvd2luZy4KISAJICogICAtIE5vIHBo eXNpY2FsIGxvZyBleGlzdGVkIG9yaWdpbmFsbHksCiEgCSAqICAgLSBXQUwgcmVjb3JkIHdh cyBub3QgcmVtb3ZhYmxlIGJlY2F1c2UgaXQgaXMgZ2VuZXJhdGVkIGR1cmluZwohIAkgKiAg ICAgdGhlIG9ubGluZSBiYWNrdXAsCiEgCSAqICAgLSBDYW5ub3QgYmUgcmVtb3ZlZCBiZWNh dXNlIHRoZSBwaHlzaWNhbCBsb2cgc3Bhbm5lZCBpbgohIAkgKiAgICAgdHdvIHNlZ21lbnRz LgohIAkgKiBUaGUgcmVhc29uIHdoeSB3ZSBza2lwIHRoZSBsZW5ndGggY2hlY2sgb24gdGhl IHBoeXNpY2FsIGxvZyByZW1vdmFsIGlzCiEgCSAqIHRoYXQgdGhlIGZsYWcgWExSX1NFVF9C S0JfQkxPQ0soMC4uMikgaXMgcmVzZXQgdG8gemVybyBhbmQgaXQgcHJldmVudHMKISAJICog dGhlIGFib3ZlIGxvb3AgdG8gcHJvY2VlZCBibGsgdG8gdGhlIGVuZCBvZiB0aGUgcmVjb3Jk LgohIAkgKi8KISAJaWYgKCEocmVjb3JkLT54bF9pbmZvICYgWExSX0JLUF9SRU1PVkFCTEUp IHx8CiEgCQlyZWNvcmQtPnhsX2luZm8gJiBYTFJfQktQX0JMT0NLX01BU0spCiAgCXsKISAJ CS8qIENoZWNrIHRoYXQgeGxfdG90X2xlbiBhZ3JlZXMgd2l0aCBvdXIgY2FsY3VsYXRpb24g Ki8KISAJCWlmIChibGsgIT0gKGNoYXIgKikgcmVjb3JkICsgcmVjb3JkLT54bF90b3RfbGVu KQohIAkJewohIAkJCWVyZXBvcnQoZW1vZGUsCiEgCQkJCQkoZXJybXNnKCJpbmNvcnJlY3Qg dG90YWwgbGVuZ3RoIGluIHJlY29yZCBhdCAlWC8lWCIsCiEgCQkJCQkJCXJlY3B0ci54bG9n aWQsIHJlY3B0ci54cmVjb2ZmKSkpOwohIAkJCXJldHVybiBmYWxzZTsKISAJCX0KICAJfQog IAogIAkvKiBGaW5hbGx5IGluY2x1ZGUgdGhlIHJlY29yZCBoZWFkZXIgKi8KcGdzcWwvc3Jj L2JhY2tlbmQvYWNjZXNzL3RyYW5zYW3jgaDjgZHjgavnmbropos6IHhsb2cuYy5vcmlnCmRp ZmYgLWNyIHBnc3FsX29yZy9zcmMvaW5jbHVkZS9hY2Nlc3MveGxvZy5oIHBnc3FsL3NyYy9p bmNsdWRlL2FjY2Vzcy94bG9nLmgKKioqIHBnc3FsX29yZy9zcmMvaW5jbHVkZS9hY2Nlc3Mv eGxvZy5oCTIwMDctMDEtMDYgMDc6MTk6NTEuMDAwMDAwMDAwICswOTAwCi0tLSBwZ3NxbC9z cmMvaW5jbHVkZS9hY2Nlc3MveGxvZy5oCTIwMDctMDUtMDcgMTY6MzA6MzguMDAwMDAwMDAw ICswOTAwCioqKioqKioqKioqKioqKgoqKiogNjYsNzMgKioqKgogIC8qCiAgICogSWYgd2Ug YmFja2VkIHVwIGFueSBkaXNrIGJsb2NrcyB3aXRoIHRoZSBYTE9HIHJlY29yZCwgd2UgdXNl IGZsYWcgYml0cyBpbgogICAqIHhsX2luZm8gdG8gc2lnbmFsIGl0LiAgV2Ugc3VwcG9ydCBi YWNrdXAgb2YgdXAgdG8gMyBkaXNrIGJsb2NrcyBwZXIgWExPRwohICAqIHJlY29yZC4JKENv dWxkIHN1cHBvcnQgNCBpZiB3ZSBjYXJlZCB0byBkZWRpY2F0ZSBhbGwgdGhlIHhsX2luZm8g Yml0cyBmb3IKISAgKiB0aGlzIHB1cnBvc2U7IGN1cnJlbnRseSBiaXQgMCBvZiB4bF9pbmZv IGlzIHVudXNlZCBhbmQgYXZhaWxhYmxlLikKICAgKi8KICAjZGVmaW5lIFhMUl9CS1BfQkxP Q0tfTUFTSwkJMHgwRQkvKiBhbGwgaW5mbyBiaXRzIHVzZWQgZm9yIGJrcCBibG9ja3MgKi8K ICAjZGVmaW5lIFhMUl9NQVhfQktQX0JMT0NLUwkJMwotLS0gNjYsNzQgLS0tLQogIC8qCiAg ICogSWYgd2UgYmFja2VkIHVwIGFueSBkaXNrIGJsb2NrcyB3aXRoIHRoZSBYTE9HIHJlY29y ZCwgd2UgdXNlIGZsYWcgYml0cyBpbgogICAqIHhsX2luZm8gdG8gc2lnbmFsIGl0LiAgV2Ug c3VwcG9ydCBiYWNrdXAgb2YgdXAgdG8gMyBkaXNrIGJsb2NrcyBwZXIgWExPRwohICAqIHJl Y29yZC4KISAgKiBCaXQgMCBvZiB4bF9pbmZvIGlzIHVzZWQgdG8gcmVwcmVzZW50IHRoYXQg YmFja3VwIGJsb2NrcyBhcmUgbm90IG5lY2Vzc2FyeQohICAqIGluIGFyY2hpdmUtbG9nLgog ICAqLwogICNkZWZpbmUgWExSX0JLUF9CTE9DS19NQVNLCQkweDBFCS8qIGFsbCBpbmZvIGJp dHMgdXNlZCBmb3IgYmtwIGJsb2NrcyAqLwogICNkZWZpbmUgWExSX01BWF9CS1BfQkxPQ0tT CQkzCioqKioqKioqKioqKioqKgoqKiogNzUsODAgKioqKgotLS0gNzYsODIgLS0tLQogICNk ZWZpbmUgWExSX0JLUF9CTE9DS18xCQkJWExSX1NFVF9CS1BfQkxPQ0soMCkJLyogMHgwOCAq LwogICNkZWZpbmUgWExSX0JLUF9CTE9DS18yCQkJWExSX1NFVF9CS1BfQkxPQ0soMSkJLyog MHgwNCAqLwogICNkZWZpbmUgWExSX0JLUF9CTE9DS18zCQkJWExSX1NFVF9CS1BfQkxPQ0so MikJLyogMHgwMiAqLworICNkZWZpbmUgWExSX0JLUF9SRU1PVkFCTEUJCVhMUl9TRVRfQktQ X0JMT0NLKDMpCS8qIDB4MDEgKi8KICAKICAvKgogICAqIFNvbWV0aW1lcyB3ZSBsb2cgcmVj b3JkcyB3aGljaCBhcmUgb3V0IG9mIHRyYW5zYWN0aW9uIGNvbnRyb2wuCg== --------------080508060806030601030809--