Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iygU5-00082C-Gv for pgsql-hackers@arkaria.postgresql.org; Mon, 03 Feb 2020 18:30:10 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1iygU3-0003Pu-Cj for pgsql-hackers@arkaria.postgresql.org; Mon, 03 Feb 2020 18:30:07 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iygU2-0003Pn-N1 for pgsql-hackers@lists.postgresql.org; Mon, 03 Feb 2020 18:30:07 +0000 Received: from mail-pf1-x442.google.com ([2607:f8b0:4864:20::442]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1iygTz-0002TG-I4 for pgsql-hackers@postgresql.org; Mon, 03 Feb 2020 18:30:05 +0000 Received: by mail-pf1-x442.google.com with SMTP id i6so8020976pfc.1 for ; Mon, 03 Feb 2020 10:30:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=j-davis-com.20150623.gappssmtp.com; s=20150623; h=message-id:subject:from:to:cc:date:in-reply-to:references :mime-version; bh=iyG8cSJbgMyrRrQ8Xmx3lheCQbaplPVrUgIjh6plIOU=; b=cmiGDPntt3u/bASW7XEZXGIbx/CDmyagb0tQfZpC6HbO7Olk2EqBPHyzHb+kpjgkiA ICxDkGLSi5C67EhKJ4mvVVHfocbbApys2jPormmRtJkZDzAs/70PxQSfKbqdgSJT68Sq FNaytj+HYLGsx798Wf24eoul4upROVnu2mRwcujzE1r+MaHZoyTMXQKDSb2Z5mpNzoVr j4HnuLtEwtO6PDS5YnAcGBSJ+8jBoKvKHlqEwn08pzG5KUVbnl9xsglXJO5LwQl1JoUR bCXTUfOBPb0aTKtFbqAkkYePM9mg7LcTg5c2+zofHqx39xCH2p6PL+6cFayxvHW513ud FUXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to :references:mime-version; bh=iyG8cSJbgMyrRrQ8Xmx3lheCQbaplPVrUgIjh6plIOU=; b=iP94F6+SqVIczBYr1pgIKdcvVQp8EJJL5aG5iezVIvohrTKJeZz6OmmcAgM3mqjvIb gdnQSYF3Q9pkQDd/yvh9dG2H1/K7WBO0sM733tkZy0s/SkeOqjieL+mu1UiTh+CxwQD4 udG16H8FDae8frRY5LPSixhxXxWfbt94cfD0FKeAuhSTLJ0Atuc+EERiSTgO+YlcYhdX BoFhlC+N3ktnIIAzrXN8qrZMva+XgKtXCcsuUUrBRaK2gDQTannPpMguHcQQqIErGEDq 4t+RLUjfR/cyTIsGGeFz+7Smkq6rgfICpLPgT41o0PYOjPXeJYPVF6M253PU+tFiwLYj 4SkQ== X-Gm-Message-State: APjAAAX0y+rbDcqqF+jkI/5FxxchMC7qaanOahvbCTVbuDBLEHiNoB4z RKeR41fZJ9oD/kjQ94nel5lFbw== X-Google-Smtp-Source: APXvYqwYyIluuzBZYPR/4Kk+Ct4rnmJInl3nOZRWE+IgB4kj28SJ76TbrC6CCg761US/B0wEF0YBQQ== X-Received: by 2002:a63:5fcf:: with SMTP id t198mr11897611pgb.383.1580754602029; Mon, 03 Feb 2020 10:30:02 -0800 (PST) Received: from jdavis ([2600:1010:b015:a9b8:6458:80b:898f:69bc]) by smtp.gmail.com with ESMTPSA id g21sm22084385pfb.126.2020.02.03.10.29.58 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 03 Feb 2020 10:30:00 -0800 (PST) Message-ID: <97c46a59c27f3c38e486ca170fcbc618d97ab049.camel@j-davis.com> Subject: Re: Memory-Bounded Hash Aggregation From: Jeff Davis To: Peter Geoghegan Cc: Heikki Linnakangas , Tomas Vondra , Taylor Vesely , Adam Lee , Melanie Plageman , PostgreSQL Hackers Date: Mon, 03 Feb 2020 10:29:55 -0800 In-Reply-To: <6e2b4a4475a690fa515829950aeed935696b0df3.camel@j-davis.com> References: <507ac540ec7c20136364b5272acbcd4574aa76ef.camel@j-davis.com> <20190703001753.s5e5cme3ucjgeu6c@development> <20190711155520.zqipi23o3yybx7ym@development> <9be86fc1adc315f69b8af4b379087ab451008d8a.camel@j-davis.com> <20190802064405.GA94036@mars.local> <617fea8258ba3df7bc8b21cd7f3afc219f2383d9.camel@j-davis.com> <20190802152123.ichmacgx3y3x7dc3@development> <20191214173225.3kzsbncr6brpxdll@development> <775d754c-9a61-5363-6998-f87d15f1b592@iki.fi> <6e2b4a4475a690fa515829950aeed935696b0df3.camel@j-davis.com> Content-Type: multipart/mixed; boundary="=-7o3W/GwhD0EDcfojZIpm" X-Mailer: Evolution 3.28.5-0ubuntu0.18.04.1 Mime-Version: 1.0 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --=-7o3W/GwhD0EDcfojZIpm Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit On Wed, 2020-01-29 at 14:48 -0800, Jeff Davis wrote: > 2) Use a different structure more capable of handling a large > fraction > of free space. A compressed bitmap might make sense, but that seems > like overkill to waste effort tracking a lot of space that is > unlikely > to ever be used. I ended up converting the freelist to a min heap. Attached is a patch which makes three changes to better support HashAgg: 1. Use a minheap for the freelist. The original design used an array that had to be sorted between a read (which frees a block) and a write (which needs to sort the array to consume the lowest block number). The comments said: * sorted. This is an efficient way to handle it because we expect cycles * of releasing many blocks followed by re-using many blocks, due to * the larger read buffer. But I didn't find a case where that actually wins over a simple minheap. With that in mind, a minheap seems closer to what one might expect for that purpose, and more robust when the assumptions don't hold up as well. If someone knows of a case where the re-sorting behavior is important, please let me know. Changing to a minheap effectively solves the problem for HashAgg, though in theory the memory consumption of the freelist itself could become significant (though it's only 0.1% of the free space being tracked). 2. Lazily-allocate the read buffer. The write buffer was always lazily- allocated, so this patch creates better symmetry. More importantly, it means freshly-rewound tapes don't have any buffer allocated, so it greatly expands the number of tapes that can be managed efficiently as long as only a limited number are active at once. 3. Allow expanding the number of tapes for an existing tape set. This is useful for HashAgg, which doesn't know how many tapes will be needed in advance. Regards, Jeff Davis --=-7o3W/GwhD0EDcfojZIpm Content-Disposition: attachment; filename="logtape.patch" Content-Type: text/x-patch; name="logtape.patch"; charset="UTF-8" Content-Transfer-Encoding: base64 ZGlmZiAtLWdpdCBhL3NyYy9iYWNrZW5kL3V0aWxzL3NvcnQvbG9ndGFwZS5jIGIvc3JjL2JhY2tl bmQvdXRpbHMvc29ydC9sb2d0YXBlLmMKaW5kZXggNDJjZmIxZjlmOTguLjIwYjI3YjM1NThiIDEw MDY0NAotLS0gYS9zcmMvYmFja2VuZC91dGlscy9zb3J0L2xvZ3RhcGUuYworKysgYi9zcmMvYmFj a2VuZC91dGlscy9zb3J0L2xvZ3RhcGUuYwpAQCAtNDksMTIgKzQ5LDggQEAKICAqIHdoZW4gcmVh ZGluZywgYW5kIHJlYWQgbXVsdGlwbGUgYmxvY2tzIGZyb20gdGhlIHNhbWUgdGFwZSBpbiBvbmUg Z28sCiAgKiB3aGVuZXZlciB0aGUgYnVmZmVyIGJlY29tZXMgZW1wdHkuCiAgKgotICogVG8gc3Vw cG9ydCB0aGUgYWJvdmUgcG9saWN5IG9mIHdyaXRpbmcgdG8gdGhlIGxvd2VzdCBmcmVlIGJsb2Nr LAotICogbHRzR2V0RnJlZUJsb2NrIHNvcnRzIHRoZSBsaXN0IG9mIGZyZWUgYmxvY2sgbnVtYmVy cyBpbnRvIGRlY3JlYXNpbmcKLSAqIG9yZGVyIGVhY2ggdGltZSBpdCBpcyBhc2tlZCBmb3IgYSBi bG9jayBhbmQgdGhlIGxpc3QgaXNuJ3QgY3VycmVudGx5Ci0gKiBzb3J0ZWQuICBUaGlzIGlzIGFu IGVmZmljaWVudCB3YXkgdG8gaGFuZGxlIGl0IGJlY2F1c2Ugd2UgZXhwZWN0IGN5Y2xlcwotICog b2YgcmVsZWFzaW5nIG1hbnkgYmxvY2tzIGZvbGxvd2VkIGJ5IHJlLXVzaW5nIG1hbnkgYmxvY2tz LCBkdWUgdG8KLSAqIHRoZSBsYXJnZXIgcmVhZCBidWZmZXIuCisgKiBUbyBzdXBwb3J0IHRoZSBh Ym92ZSBwb2xpY3kgb2Ygd3JpdGluZyB0byB0aGUgbG93ZXN0IGZyZWUgYmxvY2ssIHRoZQorICog ZnJlZWxpc3QgaXMgYSBtaW4gaGVhcC4KICAqCiAgKiBTaW5jZSBhbGwgdGhlIGJvb2trZWVwaW5n IGFuZCBidWZmZXIgbWVtb3J5IGlzIGFsbG9jYXRlZCB3aXRoIHBhbGxvYygpLAogICogYW5kIHRo ZSB1bmRlcmx5aW5nIGZpbGUocykgYXJlIG1hZGUgd2l0aCBPcGVuVGVtcG9yYXJ5RmlsZSwgYWxs IHJlc291cmNlcwpAQCAtMTcwLDcgKzE2Niw3IEBAIHN0cnVjdCBMb2dpY2FsVGFwZVNldAogCS8q CiAJICogRmlsZSBzaXplIHRyYWNraW5nLiAgbkJsb2Nrc1dyaXR0ZW4gaXMgdGhlIHNpemUgb2Yg dGhlIHVuZGVybHlpbmcgZmlsZSwKIAkgKiBpbiBCTENLU1ogYmxvY2tzLiAgbkJsb2Nrc0FsbG9j YXRlZCBpcyB0aGUgbnVtYmVyIG9mIGJsb2NrcyBhbGxvY2F0ZWQKLQkgKiBieSBsdHNHZXRGcmVl QmxvY2soKSwgYW5kIGl0IGlzIGFsd2F5cyBncmVhdGVyIHRoYW4gb3IgZXF1YWwgdG8KKwkgKiBi eSBsdHNSZWxlYXNlQmxvY2soKSwgYW5kIGl0IGlzIGFsd2F5cyBncmVhdGVyIHRoYW4gb3IgZXF1 YWwgdG8KIAkgKiBuQmxvY2tzV3JpdHRlbi4gIEJsb2NrcyBiZXR3ZWVuIG5CbG9ja3NBbGxvY2F0 ZWQgYW5kIG5CbG9ja3NXcml0dGVuIGFyZQogCSAqIGJsb2NrcyB0aGF0IGhhdmUgYmVlbiBhbGxv Y2F0ZWQgZm9yIGEgdGFwZSwgYnV0IGhhdmUgbm90IGJlZW4gd3JpdHRlbgogCSAqIHRvIHRoZSB1 bmRlcmx5aW5nIGZpbGUgeWV0LiAgbkhvbGVCbG9ja3MgdHJhY2tzIHRoZSB0b3RhbCBudW1iZXIg b2YKQEAgLTE4OCwxNSArMTg0LDkgQEAgc3RydWN0IExvZ2ljYWxUYXBlU2V0CiAJICogSWYgZm9y Z2V0RnJlZVNwYWNlIGlzIHRydWUgdGhlbiBhbnkgZnJlZWQgYmxvY2tzIGFyZSBzaW1wbHkgZm9y Z290dGVuCiAJICogcmF0aGVyIHRoYW4gYmVpbmcgcmVtZW1iZXJlZCBpbiBmcmVlQmxvY2tzW10u ICBTZWUgbm90ZXMgZm9yCiAJICogTG9naWNhbFRhcGVTZXRGb3JnZXRGcmVlU3BhY2UoKS4KLQkg KgotCSAqIElmIGJsb2Nrc1NvcnRlZCBpcyB0cnVlIHRoZW4gdGhlIGJsb2NrIG51bWJlcnMgaW4g ZnJlZUJsb2NrcyBhcmUgaW4KLQkgKiAqZGVjcmVhc2luZyogb3JkZXIsIHNvIHRoYXQgcmVtb3Zp bmcgdGhlIGxhc3QgZW50cnkgZ2l2ZXMgdXMgdGhlIGxvd2VzdAotCSAqIGZyZWUgYmxvY2suICBX ZSByZS1zb3J0IHRoZSBibG9ja3Mgd2hlbmV2ZXIgYSBibG9jayBpcyBkZW1hbmRlZDsgdGhpcwot CSAqIHNob3VsZCBiZSByZWFzb25hYmx5IGVmZmljaWVudCBnaXZlbiB0aGUgZXhwZWN0ZWQgdXNh Z2UgcGF0dGVybi4KIAkgKi8KIAlib29sCQlmb3JnZXRGcmVlU3BhY2U7CS8qIGFyZSB3ZSByZW1l bWJlcmluZyBmcmVlIGJsb2Nrcz8gKi8KLQlib29sCQlibG9ja3NTb3J0ZWQ7CS8qIGlzIGZyZWVC bG9ja3NbXSBjdXJyZW50bHkgaW4gb3JkZXI/ICovCi0JbG9uZwkgICAqZnJlZUJsb2NrczsJCS8q IHJlc2l6YWJsZSBhcnJheSAqLworCWxvbmcJICAgKmZyZWVCbG9ja3M7CQkvKiByZXNpemFibGUg YXJyYXkgaG9sZGluZyBtaW5oZWFwICovCiAJaW50CQkJbkZyZWVCbG9ja3M7CS8qICMgb2YgY3Vy cmVudGx5IGZyZWUgYmxvY2tzICovCiAJaW50CQkJZnJlZUJsb2Nrc0xlbjsJLyogY3VycmVudCBh bGxvY2F0ZWQgbGVuZ3RoIG9mIGZyZWVCbG9ja3NbXSAqLwogCkBAIC0yMTEsNiArMjAxLDcgQEAg c3RhdGljIGxvbmcgbHRzR2V0RnJlZUJsb2NrKExvZ2ljYWxUYXBlU2V0ICpsdHMpOwogc3RhdGlj IHZvaWQgbHRzUmVsZWFzZUJsb2NrKExvZ2ljYWxUYXBlU2V0ICpsdHMsIGxvbmcgYmxvY2tudW0p Owogc3RhdGljIHZvaWQgbHRzQ29uY2F0V29ya2VyVGFwZXMoTG9naWNhbFRhcGVTZXQgKmx0cywg VGFwZVNoYXJlICpzaGFyZWQsCiAJCQkJCQkJCSBTaGFyZWRGaWxlU2V0ICpmaWxlc2V0KTsKK3N0 YXRpYyB2b2lkIGx0c0luaXRUYXBlKExvZ2ljYWxUYXBlICpsdCk7CiAKIAogLyoKQEAgLTMyMSw0 NiArMzEyLDg4IEBAIGx0c1JlYWRGaWxsQnVmZmVyKExvZ2ljYWxUYXBlU2V0ICpsdHMsIExvZ2lj YWxUYXBlICpsdCkKIAlyZXR1cm4gKGx0LT5uYnl0ZXMgPiAwKTsKIH0KIAotLyoKLSAqIHFzb3J0 IGNvbXBhcmF0b3IgZm9yIHNvcnRpbmcgZnJlZUJsb2Nrc1tdIGludG8gZGVjcmVhc2luZyBvcmRl ci4KLSAqLwotc3RhdGljIGludAotZnJlZUJsb2Nrc19jbXAoY29uc3Qgdm9pZCAqYSwgY29uc3Qg dm9pZCAqYikKK3N0YXRpYyBpbmxpbmUgdm9pZAorc3dhcF9ub2Rlcyhsb25nICpoZWFwLCBpbnQg YSwgaW50IGIpCiB7Ci0JbG9uZwkJYWJsayA9ICooKGNvbnN0IGxvbmcgKikgYSk7Ci0JbG9uZwkJ YmJsayA9ICooKGNvbnN0IGxvbmcgKikgYik7Ci0KLQkvKiBjYW4ndCBqdXN0IHN1YnRyYWN0IGJl Y2F1c2UgbG9uZyBtaWdodCBiZSB3aWRlciB0aGFuIGludCAqLwotCWlmIChhYmxrIDwgYmJsaykK LQkJcmV0dXJuIDE7Ci0JaWYgKGFibGsgPiBiYmxrKQotCQlyZXR1cm4gLTE7Ci0JcmV0dXJuIDA7 CisJbG9uZwkJc3dhcDsKKworCXN3YXAgPSBoZWFwW2FdOworCWhlYXBbYV0gPSBoZWFwW2JdOwor CWhlYXBbYl0gPSBzd2FwOworfQorCitzdGF0aWMgaW5saW5lIGludAorbGVmdF9vZmZzZXQoaW50 IGkpCit7CisJcmV0dXJuIDIgKiBpICsgMTsKK30KKworc3RhdGljIGlubGluZSBpbnQKK3JpZ2h0 X29mZnNldChpbnQgaSkKK3sKKwlyZXR1cm4gMiAqIGkgKyAyOworfQorCitzdGF0aWMgaW5saW5l IGludAorcGFyZW50X29mZnNldChpbnQgaSkKK3sKKwlyZXR1cm4gKGkgLSAxKSAvIDI7CiB9CiAK IC8qCi0gKiBTZWxlY3QgYSBjdXJyZW50bHkgdW51c2VkIGJsb2NrIGZvciB3cml0aW5nIHRvLgor ICogU2VsZWN0IHRoZSBsb3dlc3QgY3VycmVudGx5IHVudXNlZCBibG9jayBieSB0YWtpbmcgdGhl IGZpcnN0IGVsZW1lbnQgZnJvbQorICogdGhlIGZyZWVsaXN0IG1pbiBoZWFwLgogICovCiBzdGF0 aWMgbG9uZwogbHRzR2V0RnJlZUJsb2NrKExvZ2ljYWxUYXBlU2V0ICpsdHMpCiB7Ci0JLyoKLQkg KiBJZiB0aGVyZSBhcmUgbXVsdGlwbGUgZnJlZSBibG9ja3MsIHdlIHNlbGVjdCB0aGUgb25lIGFw cGVhcmluZyBsYXN0IGluCi0JICogZnJlZUJsb2Nrc1tdIChhZnRlciBzb3J0aW5nIHRoZSBhcnJh eSBpZiBuZWVkZWQpLiAgSWYgdGhlcmUgYXJlIG5vbmUsCi0JICogYXNzaWduIHRoZSBuZXh0IGJs b2NrIGF0IHRoZSBlbmQgb2YgdGhlIGZpbGUuCi0JICovCi0JaWYgKGx0cy0+bkZyZWVCbG9ja3Mg PiAwKQorCWxvbmcJKmhlYXAgPSBsdHMtPmZyZWVCbG9ja3M7CisJbG9uZwkgYmxvY2tudW07CisJ aW50CQkgaGVhcHNpemU7CisJaW50CQkgcG9zOworCisJLyogZnJlZWxpc3QgZW1wdHk7IGFsbG9j YXRlIGEgbmV3IGJsb2NrICovCisJaWYgKGx0cy0+bkZyZWVCbG9ja3MgPT0gMCkKKwkJcmV0dXJu IGx0cy0+bkJsb2Nrc0FsbG9jYXRlZCsrOworCisJaWYgKGx0cy0+bkZyZWVCbG9ja3MgPT0gMSkK IAl7Ci0JCWlmICghbHRzLT5ibG9ja3NTb3J0ZWQpCi0JCXsKLQkJCXFzb3J0KCh2b2lkICopIGx0 cy0+ZnJlZUJsb2NrcywgbHRzLT5uRnJlZUJsb2NrcywKLQkJCQkgIHNpemVvZihsb25nKSwgZnJl ZUJsb2Nrc19jbXApOwotCQkJbHRzLT5ibG9ja3NTb3J0ZWQgPSB0cnVlOwotCQl9Ci0JCXJldHVy biBsdHMtPmZyZWVCbG9ja3NbLS1sdHMtPm5GcmVlQmxvY2tzXTsKKwkJbHRzLT5uRnJlZUJsb2Nr cy0tOworCQlyZXR1cm4gbHRzLT5mcmVlQmxvY2tzWzBdOwogCX0KLQllbHNlCi0JCXJldHVybiBs dHMtPm5CbG9ja3NBbGxvY2F0ZWQrKzsKKworCS8qIHRha2UgdG9wIG9mIG1pbmhlYXAgKi8KKwli bG9ja251bSA9IGhlYXBbMF07CisKKwkvKiByZXBsYWNlIHdpdGggZW5kIG9mIG1pbmhlYXAgYXJy YXkgKi8KKwloZWFwWzBdID0gaGVhcFstLWx0cy0+bkZyZWVCbG9ja3NdOworCisJLyogc2lmdCBk b3duICovCisJcG9zID0gMDsKKwloZWFwc2l6ZSA9IGx0cy0+bkZyZWVCbG9ja3M7CisJd2hpbGUg KHRydWUpCisJeworCQlpbnQgbGVmdCAgPSBsZWZ0X29mZnNldChwb3MpOworCQlpbnQgcmlnaHQg PSByaWdodF9vZmZzZXQocG9zKTsKKwkJaW50IG1pbl9jaGlsZDsKKworCQlpZiAobGVmdCA8IGhl YXBzaXplICYmIHJpZ2h0IDwgaGVhcHNpemUpCisJCQltaW5fY2hpbGQgPSAoaGVhcFtsZWZ0XSA8 IGhlYXBbcmlnaHRdKSA/IGxlZnQgOiByaWdodDsKKwkJZWxzZSBpZiAobGVmdCA8IGhlYXBzaXpl KQorCQkJbWluX2NoaWxkID0gbGVmdDsKKwkJZWxzZSBpZiAocmlnaHQgPCBoZWFwc2l6ZSkKKwkJ CW1pbl9jaGlsZCA9IHJpZ2h0OworCQllbHNlCisJCQlicmVhazsKKworCQlpZiAoaGVhcFttaW5f Y2hpbGRdID49IGhlYXBbcG9zXSkKKwkJCWJyZWFrOworCisJCXN3YXBfbm9kZXMoaGVhcCwgbWlu X2NoaWxkLCBwb3MpOworCQlwb3MgPSBtaW5fY2hpbGQ7CisJfQorCisJcmV0dXJuIGJsb2NrbnVt OwogfQogCiAvKgpAQCAtMzY5LDcgKzQwMiw4IEBAIGx0c0dldEZyZWVCbG9jayhMb2dpY2FsVGFw ZVNldCAqbHRzKQogc3RhdGljIHZvaWQKIGx0c1JlbGVhc2VCbG9jayhMb2dpY2FsVGFwZVNldCAq bHRzLCBsb25nIGJsb2NrbnVtKQogewotCWludAkJCW5keDsKKwlsb25nCSpoZWFwOworCWludAkJ IHBvczsKIAogCS8qCiAJICogRG8gbm90aGluZyBpZiB3ZSdyZSBubyBsb25nZXIgaW50ZXJlc3Rl ZCBpbiByZW1lbWJlcmluZyBmcmVlIHNwYWNlLgpAQCAtMzg3LDE0ICs0MjEsMjMgQEAgbHRzUmVs ZWFzZUJsb2NrKExvZ2ljYWxUYXBlU2V0ICpsdHMsIGxvbmcgYmxvY2tudW0pCiAJCQkJCQkJCQkJ CWx0cy0+ZnJlZUJsb2Nrc0xlbiAqIHNpemVvZihsb25nKSk7CiAJfQogCi0JLyoKLQkgKiBBZGQg YmxvY2tudW0gdG8gYXJyYXksIGFuZCBtYXJrIHRoZSBhcnJheSB1bnNvcnRlZCBpZiBpdCdzIG5v IGxvbmdlciBpbgotCSAqIGRlY3JlYXNpbmcgb3JkZXIuCi0JICovCi0JbmR4ID0gbHRzLT5uRnJl ZUJsb2NrcysrOwotCWx0cy0+ZnJlZUJsb2Nrc1tuZHhdID0gYmxvY2tudW07Ci0JaWYgKG5keCA+ IDAgJiYgbHRzLT5mcmVlQmxvY2tzW25keCAtIDFdIDwgYmxvY2tudW0pCi0JCWx0cy0+YmxvY2tz U29ydGVkID0gZmFsc2U7CisJaGVhcCA9IGx0cy0+ZnJlZUJsb2NrczsKKwlwb3MgPSBsdHMtPm5G cmVlQmxvY2tzOworCisJLyogcGxhY2UgZW50cnkgYXQgZW5kIG9mIG1pbmhlYXAgYXJyYXkgKi8K KwloZWFwW3Bvc10gPSBibG9ja251bTsKKwlsdHMtPm5GcmVlQmxvY2tzKys7CisKKwkvKiBzaWZ0 IHVwICovCisJd2hpbGUgKHBvcyAhPSAwKQorCXsKKwkJaW50IHBhcmVudCA9IHBhcmVudF9vZmZz ZXQocG9zKTsKKwkJaWYgKGhlYXBbcGFyZW50XSA8IGhlYXBbcG9zXSkKKwkJCWJyZWFrOworCisJ CXN3YXBfbm9kZXMoaGVhcCwgcGFyZW50LCBwb3MpOworCQlwb3MgPSBwYXJlbnQ7CisJfQogfQog CiAvKgpAQCAtNDg2LDYgKzUyOSwzMCBAQCBsdHNDb25jYXRXb3JrZXJUYXBlcyhMb2dpY2FsVGFw ZVNldCAqbHRzLCBUYXBlU2hhcmUgKnNoYXJlZCwKIAlsdHMtPm5Ib2xlQmxvY2tzID0gbHRzLT5u QmxvY2tzQWxsb2NhdGVkIC0gbnBoeXNpY2FsYmxvY2tzOwogfQogCisvKgorICogSW5pdGlhbGl6 ZSBwZXItdGFwZSBzdHJ1Y3QuICBOb3RlIHdlIGFsbG9jYXRlIHRoZSBJL08gYnVmZmVyIGFuZCB0 aGUgZmlyc3QKKyAqIGJsb2NrIGZvciBhIHRhcGUgb25seSB3aGVuIGl0IGlzIGZpcnN0IGFjdHVh bGx5IHdyaXR0ZW4gdG8uICBUaGlzIGF2b2lkcworICogd2FzdGluZyBtZW1vcnkgc3BhY2Ugd2hl biB0dXBsZXNvcnQuYyBvdmVyZXN0aW1hdGVzIHRoZSBudW1iZXIgb2YgdGFwZXMKKyAqIG5lZWRl ZC4KKyAqLworc3RhdGljIHZvaWQKK2x0c0luaXRUYXBlKExvZ2ljYWxUYXBlICpsdCkKK3sKKwls dC0+d3JpdGluZyAgICAgICAgICAgPSB0cnVlOworCWx0LT5mcm96ZW4gICAgICAgICAgICA9IGZh bHNlOworCWx0LT5kaXJ0eSAgICAgICAgICAgICA9IGZhbHNlOworCWx0LT5maXJzdEJsb2NrTnVt YmVyICA9IC0xTDsKKwlsdC0+Y3VyQmxvY2tOdW1iZXIgICAgPSAtMUw7CisJbHQtPm5leHRCbG9j a051bWJlciAgID0gLTFMOworCWx0LT5vZmZzZXRCbG9ja051bWJlciA9IDBMOworCWx0LT5idWZm ZXIgICAgICAgICAgICA9IE5VTEw7CisJbHQtPmJ1ZmZlcl9zaXplICAgICAgID0gMDsKKwkvKiBw YWxsb2MoKSBsYXJnZXIgdGhhbiBNYXhBbGxvY1NpemUgd291bGQgZmFpbCAqLworCWx0LT5tYXhf c2l6ZSAgICAgICAgICA9IE1heEFsbG9jU2l6ZTsKKwlsdC0+cG9zICAgICAgICAgICAgICAgPSAw OworCWx0LT5uYnl0ZXMgICAgICAgICAgICA9IDA7Cit9CisKIC8qCiAgKiBDcmVhdGUgYSBzZXQg b2YgbG9naWNhbCB0YXBlcyBpbiBhIHRlbXBvcmFyeSB1bmRlcmx5aW5nIGZpbGUuCiAgKgpAQCAt NTExLDcgKzU3OCw2IEBAIExvZ2ljYWxUYXBlU2V0Q3JlYXRlKGludCBudGFwZXMsIFRhcGVTaGFy ZSAqc2hhcmVkLCBTaGFyZWRGaWxlU2V0ICpmaWxlc2V0LAogCQkJCQkgaW50IHdvcmtlcikKIHsK IAlMb2dpY2FsVGFwZVNldCAqbHRzOwotCUxvZ2ljYWxUYXBlICpsdDsKIAlpbnQJCQlpOwogCiAJ LyoKQEAgLTUyNCwzNSArNTkwLDEzIEBAIExvZ2ljYWxUYXBlU2V0Q3JlYXRlKGludCBudGFwZXMs IFRhcGVTaGFyZSAqc2hhcmVkLCBTaGFyZWRGaWxlU2V0ICpmaWxlc2V0LAogCWx0cy0+bkJsb2Nr c1dyaXR0ZW4gPSAwTDsKIAlsdHMtPm5Ib2xlQmxvY2tzID0gMEw7CiAJbHRzLT5mb3JnZXRGcmVl U3BhY2UgPSBmYWxzZTsKLQlsdHMtPmJsb2Nrc1NvcnRlZCA9IHRydWU7CS8qIGEgemVyby1sZW5n dGggYXJyYXkgaXMgc29ydGVkIC4uLiAqLwogCWx0cy0+ZnJlZUJsb2Nrc0xlbiA9IDMyOwkvKiBy ZWFzb25hYmxlIGluaXRpYWwgZ3Vlc3MgKi8KIAlsdHMtPmZyZWVCbG9ja3MgPSAobG9uZyAqKSBw YWxsb2MobHRzLT5mcmVlQmxvY2tzTGVuICogc2l6ZW9mKGxvbmcpKTsKIAlsdHMtPm5GcmVlQmxv Y2tzID0gMDsKIAlsdHMtPm5UYXBlcyA9IG50YXBlczsKIAotCS8qCi0JICogSW5pdGlhbGl6ZSBw ZXItdGFwZSBzdHJ1Y3RzLiAgTm90ZSB3ZSBhbGxvY2F0ZSB0aGUgSS9PIGJ1ZmZlciBhbmQgdGhl Ci0JICogZmlyc3QgYmxvY2sgZm9yIGEgdGFwZSBvbmx5IHdoZW4gaXQgaXMgZmlyc3QgYWN0dWFs bHkgd3JpdHRlbiB0by4gIFRoaXMKLQkgKiBhdm9pZHMgd2FzdGluZyBtZW1vcnkgc3BhY2Ugd2hl biB0dXBsZXNvcnQuYyBvdmVyZXN0aW1hdGVzIHRoZSBudW1iZXIKLQkgKiBvZiB0YXBlcyBuZWVk ZWQuCi0JICovCiAJZm9yIChpID0gMDsgaSA8IG50YXBlczsgaSsrKQotCXsKLQkJbHQgPSAmbHRz LT50YXBlc1tpXTsKLQkJbHQtPndyaXRpbmcgPSB0cnVlOwotCQlsdC0+ZnJvemVuID0gZmFsc2U7 Ci0JCWx0LT5kaXJ0eSA9IGZhbHNlOwotCQlsdC0+Zmlyc3RCbG9ja051bWJlciA9IC0xTDsKLQkJ bHQtPmN1ckJsb2NrTnVtYmVyID0gLTFMOwotCQlsdC0+bmV4dEJsb2NrTnVtYmVyID0gLTFMOwot CQlsdC0+b2Zmc2V0QmxvY2tOdW1iZXIgPSAwTDsKLQkJbHQtPmJ1ZmZlciA9IE5VTEw7Ci0JCWx0 LT5idWZmZXJfc2l6ZSA9IDA7Ci0JCS8qIHBhbGxvYygpIGxhcmdlciB0aGFuIE1heEFsbG9jU2l6 ZSB3b3VsZCBmYWlsICovCi0JCWx0LT5tYXhfc2l6ZSA9IE1heEFsbG9jU2l6ZTsKLQkJbHQtPnBv cyA9IDA7Ci0JCWx0LT5uYnl0ZXMgPSAwOwotCX0KKwkJbHRzSW5pdFRhcGUoJmx0cy0+dGFwZXNb aV0pOwogCiAJLyoKIAkgKiBDcmVhdGUgdGVtcCBCdWZGaWxlIHN0b3JhZ2UgYXMgcmVxdWlyZWQu CkBAIC03NzMsMTUgKzgxNywxMiBAQCBMb2dpY2FsVGFwZVJld2luZEZvclJlYWQoTG9naWNhbFRh cGVTZXQgKmx0cywgaW50IHRhcGVudW0sIHNpemVfdCBidWZmZXJfc2l6ZSkKIAlsdC0+YnVmZmVy X3NpemUgPSAwOwogCWlmIChsdC0+Zmlyc3RCbG9ja051bWJlciAhPSAtMUwpCiAJewotCQlsdC0+ YnVmZmVyID0gcGFsbG9jKGJ1ZmZlcl9zaXplKTsKKwkJLyoKKwkJICogVGhlIGJ1ZmZlciBpcyBs YXppbHkgYWxsb2NhdGVkIGluIExvZ2ljYWxUYXBlUmVhZCgpLCBidXQgd2Ugc2V0IHRoZQorCQkg KiBzaXplIGhlcmUuCisJCSAqLwogCQlsdC0+YnVmZmVyX3NpemUgPSBidWZmZXJfc2l6ZTsKIAl9 Ci0KLQkvKiBSZWFkIHRoZSBmaXJzdCBibG9jaywgb3IgcmVzZXQgaWYgdGFwZSBpcyBlbXB0eSAq LwotCWx0LT5uZXh0QmxvY2tOdW1iZXIgPSBsdC0+Zmlyc3RCbG9ja051bWJlcjsKLQlsdC0+cG9z ID0gMDsKLQlsdC0+bmJ5dGVzID0gMDsKLQlsdHNSZWFkRmlsbEJ1ZmZlcihsdHMsIGx0KTsKIH0K IAogLyoKQEAgLTgzMCw2ICs4NzEsMjIgQEAgTG9naWNhbFRhcGVSZWFkKExvZ2ljYWxUYXBlU2V0 ICpsdHMsIGludCB0YXBlbnVtLAogCWx0ID0gJmx0cy0+dGFwZXNbdGFwZW51bV07CiAJQXNzZXJ0 KCFsdC0+d3JpdGluZyk7CiAKKwlpZiAobHQtPmJ1ZmZlciA9PSBOVUxMKQorCXsKKwkJLyogbGF6 aWx5IGFsbG9jYXRlIGJ1ZmZlciAqLworCQlpZiAobHQtPmZpcnN0QmxvY2tOdW1iZXIgIT0gLTFM KQorCQl7CisJCQlBc3NlcnQobHQtPmJ1ZmZlcl9zaXplID4gMCk7CisJCQlsdC0+YnVmZmVyID0g cGFsbG9jKGx0LT5idWZmZXJfc2l6ZSk7CisJCX0KKworCQkvKiBSZWFkIHRoZSBmaXJzdCBibG9j aywgb3IgcmVzZXQgaWYgdGFwZSBpcyBlbXB0eSAqLworCQlsdC0+bmV4dEJsb2NrTnVtYmVyID0g bHQtPmZpcnN0QmxvY2tOdW1iZXI7CisJCWx0LT5wb3MgPSAwOworCQlsdC0+bmJ5dGVzID0gMDsK KwkJbHRzUmVhZEZpbGxCdWZmZXIobHRzLCBsdCk7CisJfQorCiAJd2hpbGUgKHNpemUgPiAwKQog CXsKIAkJaWYgKGx0LT5wb3MgPj0gbHQtPm5ieXRlcykKQEAgLTk0Myw2ICsxMDAwLDI4IEBAIExv Z2ljYWxUYXBlRnJlZXplKExvZ2ljYWxUYXBlU2V0ICpsdHMsIGludCB0YXBlbnVtLCBUYXBlU2hh cmUgKnNoYXJlKQogCX0KIH0KIAorLyoKKyAqIEFkZCBhZGRpdGlvbmFsIHRhcGVzIHRvIHRoaXMg dGFwZSBzZXQuCisgKi8KK0xvZ2ljYWxUYXBlU2V0ICoKK0xvZ2ljYWxUYXBlU2V0RXh0ZW5kKExv Z2ljYWxUYXBlU2V0ICpsdHMsIGludCBuQWRkaXRpb25hbCkKK3sKKwlpbnQgICAgIGk7CisJaW50 CQluVGFwZXNPcmlnID0gbHRzLT5uVGFwZXM7CisJU2l6ZQluZXdTaXplOworCisJbHRzLT5uVGFw ZXMgKz0gbkFkZGl0aW9uYWw7CisJbmV3U2l6ZSA9IG9mZnNldG9mKExvZ2ljYWxUYXBlU2V0LCB0 YXBlcykgKworCQlsdHMtPm5UYXBlcyAqIHNpemVvZihMb2dpY2FsVGFwZSk7CisKKwlsdHMgPSAo TG9naWNhbFRhcGVTZXQgKikgcmVwYWxsb2MobHRzLCBuZXdTaXplKTsKKworCWZvciAoaSA9IG5U YXBlc09yaWc7IGkgPCBsdHMtPm5UYXBlczsgaSsrKQorCQlsdHNJbml0VGFwZSgmbHRzLT50YXBl c1tpXSk7CisKKwlyZXR1cm4gbHRzOworfQorCiAvKgogICogQmFja3NwYWNlIHRoZSB0YXBlIGEg Z2l2ZW4gbnVtYmVyIG9mIGJ5dGVzLiAgKFdlIGFsc28gc3VwcG9ydCBhIG1vcmUKICAqIGdlbmVy YWwgc2VlayBpbnRlcmZhY2UsIHNlZSBiZWxvdy4pCmRpZmYgLS1naXQgYS9zcmMvaW5jbHVkZS91 dGlscy9sb2d0YXBlLmggYi9zcmMvaW5jbHVkZS91dGlscy9sb2d0YXBlLmgKaW5kZXggNjk1ZDJj MDBlZTQuLjNlYmU1MjIzOWY4IDEwMDY0NAotLS0gYS9zcmMvaW5jbHVkZS91dGlscy9sb2d0YXBl LmgKKysrIGIvc3JjL2luY2x1ZGUvdXRpbHMvbG9ndGFwZS5oCkBAIC02Nyw2ICs2Nyw4IEBAIGV4 dGVybiB2b2lkIExvZ2ljYWxUYXBlUmV3aW5kRm9yUmVhZChMb2dpY2FsVGFwZVNldCAqbHRzLCBp bnQgdGFwZW51bSwKIGV4dGVybiB2b2lkIExvZ2ljYWxUYXBlUmV3aW5kRm9yV3JpdGUoTG9naWNh bFRhcGVTZXQgKmx0cywgaW50IHRhcGVudW0pOwogZXh0ZXJuIHZvaWQgTG9naWNhbFRhcGVGcmVl emUoTG9naWNhbFRhcGVTZXQgKmx0cywgaW50IHRhcGVudW0sCiAJCQkJCQkJICBUYXBlU2hhcmUg KnNoYXJlKTsKK2V4dGVybiBMb2dpY2FsVGFwZVNldCAqTG9naWNhbFRhcGVTZXRFeHRlbmQoTG9n aWNhbFRhcGVTZXQgKmx0cywKKwkJCQkJCQkJCQkJaW50IG5BZGRpdGlvbmFsKTsKIGV4dGVybiBz aXplX3QgTG9naWNhbFRhcGVCYWNrc3BhY2UoTG9naWNhbFRhcGVTZXQgKmx0cywgaW50IHRhcGVu dW0sCiAJCQkJCQkJCSAgIHNpemVfdCBzaXplKTsKIGV4dGVybiB2b2lkIExvZ2ljYWxUYXBlU2Vl ayhMb2dpY2FsVGFwZVNldCAqbHRzLCBpbnQgdGFwZW51bSwK --=-7o3W/GwhD0EDcfojZIpm--