X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id ABFFB329E3D for ; Mon, 13 Sep 2004 07:11:13 +0100 (BST) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 55564-04 for ; Mon, 13 Sep 2004 06:11:08 +0000 (GMT) Received: from mail0.rawbw.com (mail0.rawbw.com [198.144.192.41]) by svr1.postgresql.org (Postfix) with ESMTP id 0A8FB329E45 for ; Mon, 13 Sep 2004 07:11:07 +0100 (BST) Received: (from www@localhost) by mail0.rawbw.com (8.11.6p2/8.11.6) id i8D6B6g81218 for pgsql-performance@postgresql.org; Sun, 12 Sep 2004 23:11:06 -0700 (PDT) Received: from 198.144.203.171 ([198.144.203.171]) by webmail.rawbw.com (IMP) with HTTP for ; Sun, 12 Sep 2004 23:11:06 -0700 Message-ID: <1095055866.414539fadb90d@webmail.rawbw.com> Date: Sun, 12 Sep 2004 23:11:06 -0700 From: mudfoot@rawbw.com To: pgsql-performance@postgresql.org Subject: Determine optimal fdatasync/fsync, O_SYNC/O_DSYNC options MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="-MOQ1095055866e1dd2dbc1ed81c0f7e60b7cb17d6f7b3" User-Agent: Internet Messaging Program (IMP) 3.2.1 X-Originating-IP: 198.144.203.171 X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.3 tagged_above=0.0 required=5.0 tests=LINES_OF_YELLING, NO_REAL_NAME X-Spam-Level: X-Archive-Number: 200409/80 X-Sequence-Number: 8208 This message is in MIME format. ---MOQ1095055866e1dd2dbc1ed81c0f7e60b7cb17d6f7b3 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Hi, I'd like to help with the topic in the Subject: line. It seems to be a TODO item. I've reviewed some threads discussing the matter, so I hope I've acquired enough history concerning it. I've taken an initial swipe at figuring out how to optimize sync'ing methods. It's based largely on recommendations I've read on previous threads about fsync/O_SYNC and so on. After reviewing, if anybody has recommendations on how to proceed then I'd love to hear them. Attached is a little program that basically does a bunch of sequential writes to a file. All of the sync'ing methods supported by PostgreSQL WAL can be used. Results are printed in microseconds. Size and quanity of writes are configurable. The documentation is in the code (how to configure, build, run, etc.). I realize that this program doesn't reflect all of the possible activities of a production database system, but I hope it's a step in the right direction for this task. I've used it to see differences in behavior between the various sync'ing methods on various platforms. Here's what I've found running the benchmark on some systems to which I have access. The differences in behavior between platforms is quite vast. Summary first... PostgreSQL should be run on an old Apple MacIntosh attached to its own Hitachi disk array with 2GB cache or so. Use any sync method except for fsync(). Anyway, there is *a lot* of variance in file synching behavior across different hardware and O/S platforms. It's probably not safe to conclude much. That said, here are some findings so far based on tests I've run: 1. under no circumstances do fsync() or fdatasync() seem to perform better than opening files with O_SYNC or O_DSYNC 2. where there are differences, opening files with O_SYNC or O_DSYNC tends to be quite faster. 3. fsync() seems to be the slowest where there are differences. And O_DSYNC seems to be the fastest where results differ. 4. the safest thing to assert at this point is that Solaris systems ought to use the O_DSYNC method for WAL. ----------- Test system(s) Athlon Linux: AMD Athlon XP2000, 512MB RAM, single (54 or 7200?) RPM 20GB IDE disk, reiserfs filesystem (3 something I think) SuSE Linux kernel 2.4.21-99 Mac Linux: I don't know the specific model. 400MHz G3, 512MB, single IDE disk, ext2 filesystem Debian GNU/Linux 2.4.16-powerpc HP Intel Linux: Prolient HPDL380G3, 2 x 3GHz Xeon, 2GB RAM, SmartArray 5i 64MB cache, 2 x 15,000RPM 36GB U320 SCSI drives mirrored. I'm not sure if writes are cached or not. There's no battery backup. ext3 filesystem. Redhat Enterprise Linux 3.0 kernel based on 2.4.21 Dell Intel OpenBSD: Poweredge ?, single 1GHz PIII, 128MB RAM, single 7200RPM 80GB IDE disk, ffs filesystem OpenBSD 3.2 GENERIC kernel SUN Ultra2: Ultra2, 2 x 296MHz UltraSPARC II, 2GB RAM, 2 x 10,000RPM 18GB U160 SCSI drives mirrored with Solstice DiskSuite. UFS filesystem. Solaris 8. SUN E4500 + HDS Thunder 9570v E4500, 8 x 400MHz UltraSPARC II, 3GB RAM, HDS Thunder 9570v, 2GB mirrored battery-backed cache, RAID5 with a bunch of 146GB 10,000RPM FC drives. LUN is on single 2GB FC fabric connection. Veritas filesystem (VxFS) Solaris 8. Test methodology: All test runs were done with CHUNKSIZE 8 * 1024, CHUNKS 2 * 1024, FILESIZE_MULTIPLIER 2, and SLEEP 5. So a total of 16MB was sequentially written for each benchmark. Results are in microseconds. PLATFORM: Athlon Linux buffered: 48220 fsync: 74854397 fdatasync: 75061357 open_sync: 73869239 open_datasync: 74748145 Notes: System mostly idle. Even during tests, top showed about 95% idle. Something's not right on this box. All sync methods similarly horrible on this system. PLATFORM: Mac Linux buffered: 58912 fsync: 1539079 fdatasync: 769058 open_sync: 767094 open_datasync: 763074 Notes: system mostly idle. fsync seems worst. Otherwise, they seem pretty equivalent. This is the fastest system tested. PLATFORM: HP Intel Linux buffered: 33026 fsync: 29330067 fdatasync: 28673880 open_sync: 8783417 open_datasync: 8747971 Notes: system idle. O_SYNC and O_DSYNC methods seem to be a lot better on this platform than fsync & fdatasync. PLATFORM: Dell Intel OpenBSD buffered: 511890 fsync: 1769190 fdatasync: -------- open_sync: 1748764 open_datasync: 1747433 Notes: system idle. I couldn't locate fdatasync() on this box, so I couldn't test it. All sync methods seem equivalent and are very fast -- though still trail the old Mac. PLATFORM: SUN Ultra2 buffered: 1814824 fsync: 73954800 fdatasync: 52594532 open_sync: 34405585 open_datasync: 13883758 Notes: system mostly idle, with occasional spikes from 1-10% utilization. It looks like substantial difference between each sync method, with O_DSYNC the best and fsync() the worst. There is substantial difference between the open* and f* methods. PLATFORM: SUN E4500 + HDS Thunder 9570v buffered: 233947 fsync: 57802065 fdatasync: 56631013 open_sync: 2362207 open_datasync: 1976057 Notes: host about 30% idle, but the array tested on was completely idle. Something looks seriously not right about fsync and fdatasync -- write cache seems to have no effect on them. As for write cache, that probably explains the 2 seconds or so for the open_sync and open_datasync methods. -------------- Thanks for reading...I look forward to feedback, and hope to be helpful in this effort! Mark ---MOQ1095055866e1dd2dbc1ed81c0f7e60b7cb17d6f7b3 Content-Type: text/x-csrc; name="syncbench.c" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="syncbench.c" I2luY2x1ZGUgPHN0ZGlvLmg+CiNpbmNsdWRlIDxzdGRsaWIuaD4KI2luY2x1 ZGUgPHVuaXN0ZC5oPgojaW5jbHVkZSA8c3lzL3R5cGVzLmg+CiNpbmNsdWRl IDxzeXMvc3RhdC5oPgojaW5jbHVkZSA8ZmNudGwuaD4KI2luY2x1ZGUgPGVy cm5vLmg+CiNpbmNsdWRlIDxzdHJpbmcuaD4KI2luY2x1ZGUgPHN5cy90aW1l Lmg+CgovKgoqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioKCkxpY2Vuc2U6CnN5bmNiZW5jaDogYSBw cm9ncmFtIHRvIGhlbHAgZGV0ZXJtaW5lIG9wdGltYWwgaS9vIHN5bmNocm9u aXphdGlvbgptZWNoYW5pc21zCndyaXR0ZW4gYnkgTWFyayBUcmF2aXMKCkNv cHlyaWdodCAoYykgMjAwNCBNYXJrIFRyYXZpcwoKUGVybWlzc2lvbiB0byB1 c2UsIGNvcHksIG1vZGlmeSwgYW5kIGRpc3RyaWJ1dGUgdGhpcyBzb2Z0d2Fy ZSBhbmQKaXRzIGRvY3VtZW50YXRpb24gZm9yIGFueSBwdXJwb3NlIGFuZCB3 aXRob3V0IGZlZSBpcyBoZXJlYnkKZ3JhbnRlZCwgcHJvdmlkZWQgdGhhdCB0 aGUgYWJvdmUgY29weXJpZ2h0IG5vdGljZSBhcHBlYXIgaW4gYWxsCmNvcGll cyBhbmQgdGhhdCBib3RoIHRoYXQgY29weXJpZ2h0IG5vdGljZSBhbmQgdGhp cyBwZXJtaXNzaW9uCm5vdGljZSBhcHBlYXIgaW4gc3VwcG9ydGluZyBkb2N1 bWVudGF0aW9uLCBhbmQgdGhhdCB0aGUgbmFtZSBvZiB0aGUKYXV0aG9yIG5v dCBiZSB1c2VkIGluIGFkdmVydGlzaW5nIG9yIHB1YmxpY2l0eSBwZXJ0YWlu aW5nIHRvCmRpc3RyaWJ1dGlvbiBvZiB0aGUgc29mdHdhcmUgd2l0aG91dCBz cGVjaWZpYywgd3JpdHRlbiBwcmlvcgpwZXJtaXNzaW9uLiBUaGUgYXV0aG9y IG1ha2VzIG5vIHJlcHJlc2VudGF0aW9ucyBhYm91dCB0aGUKc3VpdGFiaWxp dHkgb2YgdGhpcyBzb2Z0d2FyZSBmb3IgYW55IHB1cnBvc2UuICBJdCBpcyBw cm92aWRlZCAiYXMKaXMiIHdpdGhvdXQgZXhwcmVzcyBvciBpbXBsaWVkIHdh cnJhbnR5LgoKR09BTDoKClRvIGhlbHAgIkRldGVybWluZSBvcHRpbWFsIGZk YXRhc3luYy9mc3luYywgT19TWU5DL09fRFNZTkMgb3B0aW9ucyIKZnJvbSB0 aGUgUG9zdGdyZVNRTCBUT0RPIGxpc3QgKGh0dHA6Ly9kZXZlbG9wZXIucG9z dGdyZXNxbC5vcmcvdG9kby5waHApCgpXSEFUIFRISVMgRE9FUzoKCldyaXRl cyBzZXF1ZW50aWFsIGNodW5rcyBvZiBkYXRhIHRvIGEgZmlsZSBhbmQgc3lu Y2hyb25pemVzIHdpdGggdGhlCkkvTyBkZXZpY2UgaW4gYSB2YXJpZXR5IG9m IHdheXMuICBHaXZlcyByZXN1bHRzIGluIG1pY3Jvc2Vjb25kcy4KCkVESVQg QkVGT1JFIENPTVBJTElORzoKClRoZXJlIGFyZSA2IHRoaW5ncyB3aGljaCBt dXN0IGJlIGRlZmluZWQgaW4gb3JkZXIgZm9yIHRoaXMgdGhpbmcgdG8KY29t cGlsZS4KCkZJTEVfU1lOQyBhbmQgT1BFTl9TWU5DIGRlZmluZSBob3cgZmls ZSBjb250ZW50cyBhcmUgdG8gYmUgc3luYydkLCBhbmQKZWFjaCBwbGF0Zm9y bSBoYXMgdmFyaW91cyBvcHRpb25zLiAgRklMRV9TWU5DIGRlZmluZXMgdGhl IGZ1bmN0aW9uCmNhbGwgbWFkZSB0byBzeW5jIGFmdGVyIGVhY2ggd3JpdGUo KS4gIE1vc3QgcGxhdGZvcm1zIHNob3VsZCBzdXBwb3J0IGZzeW5jKDIpCmF0 IGEgbWluaW11bS4gIGZkYXRhc3luYygyKSBzaG91bGQgYmUgZmFzdGVyIG9u IHBsYXRmb3JtcyB3aGljaCBzdXBwb3J0Cml0LgoKT1BFTl9TWU5DIGRlZmlu ZXMgdGhlIGZsYWcgdG8gYmUgc2V0IGluIG9wZW4oMikgZm9yCnN5bmNocm9u aXppbmcgYWxsIHdyaXRlcyB3aXRob3V0IGV4cGxpY2l0bHkgY2FsbGluZyBm c3luYygyKSBvciBpdHMKa2luZC4gIE1vc3QgcGxhdGZvcm1zIHNob3VsZCBz dXBwb3J0IE9fU1lOQyBhdCBhIG1pbmltdW0uICBPX0RTWU5DIHNob3VsZCBi ZQpmYXN0ZXIgb24gcGxhdGZvcm1zIHdoaWNoIHN1cHBvcnQgaXQuICBBbmQg c29tZSBlbnZpcm9ubWVudHMgaGF2ZQpuZWl0aGVyLiAgSSBjb3VsZCBvbmx5 IGZpbmQgT19GU1lOQyBvbiBGcmVlQlNEIDQuMTAuICBBbnl3YXksIHJlYWRp bmcgbWFuCnBhZ2VzIGZvciBvcGVuKDIpLCBmc3luYygyKSwgYW5kIGZkYXRh c3luYygyKSBpcyBwcm9iYWJseSBhIGdvb2QgaWRlYQpiZWZvcmUgc2V0dGlu ZyBGSUxFX1NZTkMgYW5kIE9QRU5fU1lOQy4KCioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKgoqLwoKLyoK I2RlZmluZSBGSUxFX1NZTkMoWCkgICAgZmRhdGFzeW5jKFgpCiNkZWZpbmUg T1BFTl9TWU5DICAgICAgIE9fRFNZTkMKKi8KCi8qCioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioK ClRoZSBvdGhlciA0IG1hY3JvcyBhcmUgQ0hVTktTSVpFLCBDSFVOS1MsIEZJ TEVTSVpFX01VTFRJUExJRVIsIGFuZApTTEVFUC4gIENIVU5LU0laRSBpcyB0 aGUgc2l6ZSBvZiBlYWNoIGNodW5rIG9mIGRhdGEgd3JpdHRlbiBpbiB0aGUg dGVzdC4KQ0hVTktTIGlzIHRoZSBudW1iZXIgb2YgdGltZXMgdGhleSBhcmUg d3JpdHRlbi4gIEFsbCB3cml0ZXMgYXJlCnNlcXVlbnRpYWwuICBCZWZvcmUg dGhlIHRlc3QgaXMgcnVuLCBhIGZpbGUgaXMgY3JlYXRlZCwgZmlsbGVkIHdp dGgKIkEiIGNoYXJhY3RlcnMsIGZzeW5jKDIpJ2QsIGFuZCB0aGVuIGNsb3Nl ZC4gIFRoZW4gd2Ugc2xlZXAgZm9yIFNMRUVQCnNlY29uZHMgYmVmb3JlIHBy b2NlZWRpbmcgd2l0aCB0aGUgYWN0dWFsIGJlbmNobWFyayBydW4uICBUaGUg c2xlZXAKaXMgdG8gbGV0IHRoZSBJL08gZGV2aWNlIHF1aWVzY2UgaWYgaXQg d2FudHMgdG8uCgpUaGUgc2l6ZSBvZiB0aGUgZmlsZSBjcmVhdGVkIGlzIEZJ TEVTSVpFX01VTFRJUExFUiB0aW1lcyBDSFVOS1NJWkUKdGltZXMgQ0hVTktT LiAgU28gaWYgRklMRVNJWkVfTVVMVElQTElFUiBleGNlZWRzIDEgdGhlbiB0 aGUgZmlsZSB3aWxsCmJlIGJpZ2dlciB0aGFuIHRoZSBhbW91bnQgb2YgZGF0 YSB3cml0dGVuIHRvIGl0LiAgSGF2aW5nIHRoZSBmaWxlIHNpemUKZXF1YWwg dG8gb3IgZXhjZWVkaW5nIHRoZSBhbW91bnQgb2YgZGF0YSB3cml0dGVuIHNo b3VsZCBoZWxwIHRvIHNpbXVsYXRlCnJlYWwtd29ybGQgV0FMIGJlaGF2aW9y LiAgRXh0ZW5kaW5nIHRoZSBzaXplIG9mIGEgZmlsZSByZXF1aXJlcyBleHRy YQp3b3JrIGZvciB0aGUgZmlsZXN5c3RlbSB0byBwZXJmb3JtLiAgVG8gbGVh cm4gdGhlIGltcGFjdCBvZiB0aGF0LCBnbwphaGVhZCBhbmQgc2V0IEZJTEVT SVpFX01VTFRJUExJRVIgdG8gbGVzcyB0aGFuIDEgaWYgeW91IHdhbnQuCgoq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqCiovCgovKgojZGVmaW5lIENIVU5LU0laRQk4ICogMTAyNAoj ZGVmaW5lIENIVU5LUwkJMiAqIDEwMjQKI2RlZmluZSBGSUxFU0laRV9NVUxU SVBMSUVSCTIKI2RlZmluZSBTTEVFUAkJNQoqLwoKLyogCioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioK Ck5vdGhpbmcgZWxzZSBzaG91bGQgbmVlZCB0byBiZSBtb2RpZmllZCBmcm9t IHRoaXMgcG9pbnQsIGJ1dCBwbGVhc2UKa2VlcCByZWFkaW5nIGZvciBidWls ZGluZyBhbmQgcnVubmluZyBpbnN0cnVjdGlvbnMuCgpCVUlMRElORzoKSXQg c2hvdWxkIGJlIHNpbXBsZSBhcyBsb25nIGFzIHRoZSByaWdodCB0b29scyBh cmUgaW4geW91ciAkUEFUSCAoY2MsCmV0Yy4pCgpMaW51eCBhbmQgKkJTRCBz ZWVtIHRvIGJlIGhhcHB5IHdpdGggdGhpczoKY2Mgc3luY2JlbmNoLmMgLW8g c3luY2JlbmNoCgpTb2xhcmlzIDggcmVxdWlyZXMgLWxydCBpbiBvcmRlciB0 byBzdXBwb3J0IGZkYXRhc3luYzoKY2Mgc3luY2JlbmNoLmMgLWxydCAtbyBz eW5jYmVuY2gKCk90aGVyIE8vUydzIHlvdSdsbCBoYXZlIHRvIGZpZ3VyZSBv dXQgb24geW91ciBvd24uCgpSVU5OSU5HOgpJdCB0YWtlcyB0d28gYXJndW1l bnRzLiAgVGhlIGZpcnN0IGlzIHRoZSBuYW1lIG9mIHRoZSBmaWxlIHRvIHVz ZSBmb3IKdGhlIHRlc3QuICBJZiB0aGUgZmlsZSBkb2Vzbid0IGV4aXN0IGFs cmVhZHksIGl0IHdpbGwgYmUgY3JlYXRlZC4gIElmCml0IGRvZXMgZXhpc3Qg YWxyZWFkeSwgdGhlbiBpdCB3aWxsIGJlIHRydW5jYXRlZCBiZWZvcmUgd3Jp dGluZy4KClRoZSBzZWNvbmQgYXJndW1lbnQgaXMgdGhlIG1vZGUgaW4gd2hp Y2ggZmlsZSBkYXRhIGlzIHN5bmNocm9uaXplZCB0bwpkaXNrLiAgSXQgY2Fu IGJlIG9uZSBvZiBidWZmZXJlZCwgZmlsZXN5bmMgYW5kIG9wZW5zeW5jLiAg YnVmZmVyZWQKbWVhbnMgbm8gc3luY2luZyB0YWtlcyBwbGFjZS4gIFVubGVz cyB5b3VyIGZpbGVzeXN0ZW0gaXMgbW91bnRlZApzeW5jaHJvbm91c2x5IHRo aXMgc2hvdWxkIGJlIHRoZSBmYXN0ZXN0IG9wdGlvbiBieSBmYXIuICBmaWxl c3luYwpleGVjdXRlcyB0aGUgZnVuY3Rpb24gZGVmaW5lZCBieSBGSUxFX1NZ TkMgYWJvdmUgYWZ0ZXIgZWFjaCBjaHVuayBvZgpkYXRhIGlzIHdyaXR0ZW4u ICBvcGVuc3luYyBvcGVuKCkncyB0aGUgZmlsZSB3aXRoIHRoZSBmbGFnIGRl ZmluZWQgaW4KT1BFTl9TWU5DLgoKUG9zdGdyZVNRTCBzdXBwb3J0cyA1IG1l dGhvZHMgb2Ygc3luYydpbmcgaWYgeW91IGNvdW50ICJmc3luYz1mYWxzZSIu ICBUaGVzZQptZXRob2RzIGFyZSBpbiBwb3N0Z3Jlc3FsLmNvbmYgdW5kZXIg dGhlICJXUklURSBBSEVBRCBMT0ciIHNlY3Rpb24uCgpIZXJlJ3MgaG93IHRv IHVzZSB0aGlzIHRvb2wgdG8gdHJ5IGFuZCBzaW11bGF0ZSB0aG9zZSBtZXRo b2RzOgoKUG9zdGdyZVNRTCBNZXRob2Q6CVN5bmNiZW5jaApmc3luYz1mYWxz ZQkJcnVuIHdpdGggMm5kIGFyZ3VtZW50PWJ1ZmZlcmVkCmZzeW5jCQkJI2Rl ZmluZSBGSUxFX1NZTkMoWCkJZnN5bmMoWCkKCQkJMm5kIGFyZ3VtZW50PWZp bGVzeW5jCmZkYXRhc3luYwkJI2RlZmluZSBGSUxFX1NZTkMoWCkJZmRhdGFz eW5jKFgpCgkJCTJuZCBhcmd1bWVudD1maWxlc3luYwpvcGVuX3N5bmMJCSNk ZWZpbmUgT1BFTl9TWU5DCU9fU1lOQwoJCQkybmQgYXJndW1lbnQ9b3BlbnN5 bmMKb3Blbl9kYXRhc3luYwkJI2RlZmluZSBPUEVOX1NZTkMJT19EU1lOQwoJ CQkybmQgYXJndW1lbnQ9b3BlbnN5bmMKCk9idmlvdXNseSwgaWYgdGhlIHBs YXRmb3JtIGRvZXNuJ3Qgc3VwcG9ydCB0aG9zZSB0aGluZ3MgZm9yIHdoYXRl dmVyCnJlYXNvbiB0aGVuIHRoZSBwcm9ncmFtIHdvbid0IGJ1aWxkLgoKUkVT VUxUUzoKClRoZSBudW1iZXIgb2YgbWljcm9zZWNvbmRzIGJldHdlZW4gd3Jp dGVzIHN0YXJ0aW5nIGFuZCBlbmRpbmcgaXMKZGlzcGxheWVkLiAgVGltZSB0 YWtlbiB0byBwcmUtcG9wdWxhdGUsIG9wZW4oKSwgb3IgY2xvc2UoKSB0aGUK ZGF0YWZpbGUgaXMgbm90IGluY2x1ZGVkIGluIHRoaXMgY2FsY3VsYXRpb24u ICBPYnZpb3VzbHksIHRoZSByZXN1bHRzCndpbGwgdmFyeSBiYXNlZCBvbiB0 aGUgYW1vdW50IG9mIGRhdGEgd3JpdHRlbiwgc3luY2hyb25pemF0aW9uCm9w dGlvbnMgdXNlZCwgaGFyZHdhcmUsIE8vUywgZmlsZXN5c3RlbSwgZXRjLiAg SXQncyBhIGdvb2QgaWRlYSB0byB0cnkKdG8gcnVuIHRoaXMgb24gc3lzdGVt cyB3aGljaCBhcmUgYXMgaWRsZSBhcyBwb3NzaWJsZSAtLSBlc3BlY2lhbGx5 IHRoZQpJL08gZGV2aWNlKHMpIGJlaW5nIHRlc3RlZC4KCioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioK Ki8KCmludCBtYWluKGludCBhcmdjLCBjaGFyICphcmd2W10pCnsKCiAgaW50 IGZkLCBuLCBibV9vcGVuZmxhZywgYm1fZG9fZmlsZXN5bmM7CiAgY2hhciAq YnVmOyAKICBzdHJ1Y3QgdGltZXZhbCB0dl9iZWZvcmUsIHR2X2FmdGVyOwog IHN0cnVjdCB0aW1lem9uZSB0el9nYXJiYWdlOwoKICBpZiAoIGFyZ2MgIT0g MyApCiAgewogIHByaW50ZigidXNhZ2U6ICVzIDxmaWxlbmFtZT4gPGJ1ZmZl cmVkfGZpbGVzeW5jfG9wZW5zeW5jPlxuIiwgYXJndlswXSApOwogIGV4aXQo MSk7CiAgfQoKICBibV9vcGVuZmxhZz0wOwogIGJtX2RvX2ZpbGVzeW5jPTA7 CgogIGlmICggIXN0cm5jbXAoIGFyZ3ZbMl0sICJidWZmZXJlZCIsIHN0cmxl bigiYXN5bmMiKSApICkKICB7CiAgICBwcmludGYoInRlc3QgdXNlcyAlc1xu IiwgYXJndlsyXSk7CiAgfSBlbHNlIGlmICggIXN0cm5jbXAoIGFyZ3ZbMl0s ICJmaWxlc3luYyIsIHN0cmxlbigiZmlsZXN5bmMiKSApICkKICB7CiAgICBw cmludGYoInRlc3QgdXNlcyAlc1xuIiwgYXJndlsyXSk7CiAgICBibV9kb19m aWxlc3luYz0xOwogIH0gZWxzZSBpZiAoICFzdHJuY21wKCBhcmd2WzJdLCAi b3BlbnN5bmMiLCBzdHJsZW4oIm9wZW5zeW5jIikgKSApCiAgewogICAgcHJp bnRmKCJ0ZXN0IHVzZXMgJXNcbiIsIGFyZ3ZbMl0pOwogICAgYm1fb3BlbmZs YWc9T1BFTl9TWU5DOwogIH0gZWxzZQogIHsKICAgIHB1dHMoIlNlY29uZCBh cmd1bWVudCBtdXN0IGJlIG9uZSBvZiBhc3luYyBmaWxlc3luYyBvcGVuc3lu YyIpOwogICAgZXhpdCgxKTsKICB9CgogIHByaW50ZigiU3RhcnRpbmcgdGVz dCB3aXRoIEZJTEVTSVpFOiAlaSwgQ0hVTktTSVpFOiAlaSwgQ0hVTktTOiAl aVxuIiwgXAogICAgRklMRVNJWkVfTVVMVElQTElFUiAqIENIVU5LU0laRSAq IENIVU5LUywgQ0hVTktTSVpFLCBDSFVOS1MgKTsKICBmZCA9IG9wZW4oIGFy Z3ZbMV0sIE9fV1JPTkxZIHwgT19DUkVBVCB8IE9fVFJVTkMsIDA2NjYgKTsK ICBpZiAoIGZkID09IC0xICkKICB7CiAgICBwZXJyb3IoIkNhbid0IGNyZWF0 ZSBkYXRhIGZpbGUiKTsKICAgIGV4aXQoMSk7CiAgfQogIGJ1ZiA9IG1hbGxv YyhDSFVOS1NJWkUpOwogIGlmICggYnVmID09IE5VTEwgKQogIHsKICAgIHB1 dHMoIm1hbGxvYyBjaG9rZWQgZm9yIHNvbWUgcmVhc29uLiAgQnllISIpOwog ICAgZXhpdCgxKTsKICB9CiAgLyoKICAgICAqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKgog ICAgIE1ha2Ugc3VyZSB0aGF0IHRoZSB3aG9sZSBmaWxlIGlzIG5vdCBtYWRl IHVwIG9mIE5VTExzLgogICAgIEkgc2VlbSB0byByZWNhbGwgYSBjaGFyYWN0 ZXJpc3RpYyBvZiAqTklYIGZpbGVzeXN0ZW1zIHRoYXQgbGlrZXMKICAgICB0 byBub3QgcG9wdWxhdGUgTlVMTC1maWxsZWQgZmlsZXMgd2l0aCBhY3R1YWwg YmxvY2tzIGZ1bGwgb2YgTlVMTHMuCiAgICAgVGhpcyBtYXkgY2F1c2UgYWN0 dWFsIHBvcHVsYXRpb24gb2YgdGhlIGZpbGUgdG8gY2F1c2UgYW4gdXBkYXRl CiAgICAgb2YgbWV0YWRhdGEsIHdoaWNoIG1pZ2h0IGNhdXNlIHBlcmZvcm1h bmNlIGlzc3Vlcy4gIEFueXdheSwgCiAgICAgcHJlLXBvcHVsYXRlIHRoZSBm aWxlIHdpdGggIkEiIHRvIGhlZGdlIGFnYWluc3QgdGhhdC4KICAgICBJdCBt YXkgYmUgZGVzaXJhYmxlIHRvIHByZS1wb3B1bGF0ZSBqdXN0IGxpa2UgV0FM IGlzIHByZS1wb3B1bGF0ZWQuCiAgICAgT3IgbWF5YmUgdGhpcyBkb2Vzbid0 IG1hdHRlci4KICAgICAqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioKICAqLwogIG1lbXNl dChidWYsIDY1LCBDSFVOS1NJWkUpOwogIAogIGZvciAoIG49MDsgbjxDSFVO S1MqRklMRVNJWkVfTVVMVElQTElFUjsgbisrKQogIHsKICAgIGlmICggd3Jp dGUoZmQsIGJ1ZiwgQ0hVTktTSVpFKSA9PSAtMSApCiAgICB7CiAgICAgIHBl cnJvcigiQ2FuJ3Qgd3JpdGUgdG8gZGF0YSBmaWxlIik7CiAgICAgIGV4aXQo MSk7CiAgICB9CiAgfQoKICBmcmVlKGJ1Zik7CgogIGlmICggZnN5bmMoZmQp ICkKICB7CiAgICBwZXJyb3IoIkNhbid0IGZzeW5jIGRhdGEgZmlsZSIpOwog ICAgZXhpdCgxKTsKICB9CgogIGlmICggY2xvc2UoZmQpICkKICB7CiAgICBw ZXJyb3IoIkNhbid0IGNsb3NlIGRhdGEgZmlsZSIpOwogICAgZXhpdCgxKTsK ICB9CgogIC8qIHNsZWVwIGZvciBncmlucy4gIHNvIG1heWJlIHRoZSBkZXZp Y2Ugc2xvd3MgZG93biAqLwogIHNsZWVwKFNMRUVQKTsKCi8qIE9LIGl0J3Mg Y3JlYXRlZC4gIE5vdyBvcGVuIGl0IGFnYWluIHRoZW4gc3RhcnQgdGhlIHNo b3cgKi8KCiAgZmQgPSBvcGVuKCBhcmd2WzFdLCBPX1dST05MWSB8IGJtX29w ZW5mbGFnLCAwNjY2ICk7CiAgaWYgKCBmZCA9PSAtMSApCiAgewogICAgcGVy cm9yKCJDYW4ndCBvcGVuIGRhdGEgZmlsZSBmb3Igd3JpdGluZyIpOwogICAg ZXhpdCgxKTsKICB9CiAgYnVmPW1hbGxvYyhDSFVOS1NJWkUpOwogIGlmICgg YnVmID09IE5VTEwgKQogIHsKICAgIHB1dHMoIm1hbGxvYyBjaG9rZWQgZm9y IHNvbWUgcmVhc29uLiIpOwogICAgZXhpdCgxKTsKICB9CiAgbWVtc2V0KGJ1 ZiwgNjYsIENIVU5LU0laRSk7CgogIGdldHRpbWVvZmRheSggJnR2X2JlZm9y ZSwgJnR6X2dhcmJhZ2UgKTsKICBmb3IgKCBuPTA7IG48Q0hVTktTOyBuKysp CiAgewogICAgaWYgKCB3cml0ZShmZCwgYnVmLCBDSFVOS1NJWkUpID09IC0x ICkKICAgIHsKICAgICAgcGVycm9yKCJDYW4ndCB3cml0ZSB0byBleGlzdGlu ZyBkYXRhIGZpbGUiKTsKICAgICAgZXhpdCgxKTsKICAgIH0KICAgIGlmICgg Ym1fZG9fZmlsZXN5bmMgKQogICAgewogICAgICBpZiAoIEZJTEVfU1lOQyhm ZCkgKQogICAgICB7CiAgICAgICAgcGVycm9yKCJDYW4ndCBzeW5jIGRhdGEg ZmlsZSIpOwogICAgICAgIGV4aXQoMSk7CiAgICAgIH0KICAgIH0KICB9IAog IGdldHRpbWVvZmRheSggJnR2X2FmdGVyLCAmdHpfZ2FyYmFnZSApOwoKICBm cmVlKGJ1Zik7CiAgcHJpbnRmKCJFbGFwc2VkIHVzZWM6ICVpXG4iLCAoICgo dHZfYWZ0ZXIudHZfc2VjICogMTAwMDAwMCkgKwogICAgdHZfYWZ0ZXIudHZf dXNlYykgLSAoKHR2X2JlZm9yZS50dl9zZWMgKiAxMDAwMDAwKSArIHR2X2Jl Zm9yZS50dl91c2VjKSApICk7Cgp9Cgo= ---MOQ1095055866e1dd2dbc1ed81c0f7e60b7cb17d6f7b3--