Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kfePe-0007oD-HS for pgsql-hackers@arkaria.postgresql.org; Thu, 19 Nov 2020 07:31:26 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1kfePd-0003S6-GG for pgsql-hackers@arkaria.postgresql.org; Thu, 19 Nov 2020 07:31:25 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kfePd-0003QS-0A for pgsql-hackers@lists.postgresql.org; Thu, 19 Nov 2020 07:31:25 +0000 Received: from oss.nttdata.com ([49.212.34.109]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kfePR-0001I2-2y for pgsql-hackers@postgresql.org; Thu, 19 Nov 2020 07:31:23 +0000 Received: from oss.nttdata.com (localhost [127.0.0.1]) by oss.nttdata.com (Postfix) with ESMTP id 5CEDC6151A; Thu, 19 Nov 2020 16:31:09 +0900 (JST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.102.3 at oss.nttdata.com MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=_a38ec8be5c94edba0f02bfe32f890fa8" Date: Thu, 19 Nov 2020 16:31:09 +0900 From: Masahiro Ikeda To: Fujii Masao Cc: pgsql-hackers Subject: Re: Add statistics to pg_stat_wal view for wal related parameter tuning In-Reply-To: <79620b5c-ecf1-3d4f-80b5-7e311b7a0662@oss.nttdata.com> References: <35ef960128b90bfae3b3fdf60a3a860f@oss.nttdata.com> <11e81f0b7d90dd69077d3d964a9a4da9@oss.nttdata.com> <79620b5c-ecf1-3d4f-80b5-7e311b7a0662@oss.nttdata.com> Message-ID: <89de5056627ebdd144967959bd789bd6@oss.nttdata.com> X-Sender: ikedamsh@oss.nttdata.com User-Agent: Roundcube Webmail/1.1.9 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --=_a38ec8be5c94edba0f02bfe32f890fa8 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8; format=flowed On 2020-11-17 11:46, Fujii Masao wrote: > On 2020/11/16 16:35, Masahiro Ikeda wrote: >> On 2020-11-12 14:58, Fujii Masao wrote: >>> On 2020/11/06 10:25, Masahiro Ikeda wrote: >>>> On 2020-10-30 11:50, Fujii Masao wrote: >>>>> On 2020/10/29 17:03, Masahiro Ikeda wrote: >>>>>> Hi, >>>>>> >>>>>> Thanks for your comments and advice. I updated the patch. >>>>>> >>>>>> On 2020-10-21 18:03, Kyotaro Horiguchi wrote: >>>>>>> At Tue, 20 Oct 2020 16:11:29 +0900, Masahiro Ikeda >>>>>>> wrote in >>>>>>>> On 2020-10-20 12:46, Amit Kapila wrote: >>>>>>>> > I see that we also need to add extra code to capture these stats (some >>>>>>>> > of which is in performance-critical path especially in >>>>>>>> > XLogInsertRecord) which again makes me a bit uncomfortable. It might >>>>>>>> > be that it is all fine as it is very important to collect these stats >>>>>>>> > at cluster-level in spite that the same information can be gathered at >>>>>>>> > statement-level to help customers but I don't see a very strong case >>>>>>>> > for that in your proposal. >>>>>>> >>>>>>> We should avoid that duplication as possible even if the both >>>>>>> number >>>>>>> are important. >>>>>>> >>>>>>>> Also about performance, I thought there are few impacts because >>>>>>>> it >>>>>>>> increments stats in memory. If I can implement to reuse >>>>>>>> pgWalUsage's >>>>>>>> value which already collects these stats, there is no impact in >>>>>>>> XLogInsertRecord. >>>>>>>> For example, how about pg_stat_wal() calculates the accumulated >>>>>>>> value of wal_records, wal_fpi, and wal_bytes to use pgWalUsage's >>>>>>>> value? >>>>>>> >>>>>>> I don't think that works, but it would work that >>>>>>> pgstat_send_wal() >>>>>>> takes the difference of that values between two successive calls. >>>>>>> >>>>>>> WalUsage prevWalUsage; >>>>>>> ... >>>>>>> pgstat_send_wal() >>>>>>> { >>>>>>> .. >>>>>>>    /* fill in some values using pgWalUsage */ >>>>>>>    WalStats.m_wal_bytes   = pgWalUsage.wal_bytes   - >>>>>>> prevWalUsage.wal_bytes; >>>>>>>    WalStats.m_wal_records = pgWalUsage.wal_records - >>>>>>> prevWalUsage.wal_records; >>>>>>>    WalStats.m_wal_wal_fpi = pgWalUsage.wal_fpi     - >>>>>>> prevWalUsage.wal_fpi; >>>>>>> ... >>>>>>>    pgstat_send(&WalStats, sizeof(WalStats)); >>>>>>> >>>>>>>    /* remember the current numbers */ >>>>>>>    prevWalUsage = pgWalUsage; >>>>>> >>>>>> Thanks for Horiguchi-san's advice, I changed to reuse pgWalUsage >>>>>> which is already defined and eliminates the extra overhead. >>>>> >>>>> +    /* fill in some values using pgWalUsage */ >>>>> +    WalStats.m_wal_bytes = pgWalUsage.wal_bytes - >>>>> prevWalUsage.wal_bytes; >>>>> +    WalStats.m_wal_records = pgWalUsage.wal_records - >>>>> prevWalUsage.wal_records; >>>>> +    WalStats.m_wal_fpi = pgWalUsage.wal_fpi - >>>>> prevWalUsage.wal_fpi; >>>>> >>>>> It's better to use WalUsageAccumDiff() here? >>>> >>>> Yes, thanks. I fixed it. >>>> >>>>> prevWalUsage needs to be initialized with pgWalUsage? >>>>> >>>>> +                if (AmWalWriterProcess()){ >>>>> +                    WalStats.m_wal_write_walwriter++; >>>>> +                } >>>>> +                else >>>>> +                { >>>>> +                    WalStats.m_wal_write_backend++; >>>>> +                } >>>>> >>>>> I think that it's better not to separate m_wal_write_xxx into two >>>>> for >>>>> walwriter and other processes. Instead, we can use one >>>>> m_wal_write_xxx >>>>> counter and make pgstat_send_wal() send also the process type to >>>>> the stats collector. Then the stats collector can accumulate the >>>>> counters >>>>> per process type if necessary. If we adopt this approach, we can >>>>> easily >>>>> extend pg_stat_wal so that any fields can be reported per process >>>>> type. >>>> >>>> I'll remove the above source code because these counters are not >>>> useful. >>>> >>>> >>>> On 2020-10-30 12:00, Fujii Masao wrote: >>>>> On 2020/10/20 11:31, Masahiro Ikeda wrote: >>>>>> Hi, >>>>>> >>>>>> I think we need to add some statistics to pg_stat_wal view. >>>>>> >>>>>> Although there are some parameter related WAL, >>>>>> there are few statistics for tuning them. >>>>>> >>>>>> I think it's better to provide the following statistics. >>>>>> Please let me know your comments. >>>>>> >>>>>> ``` >>>>>> postgres=# SELECT * from pg_stat_wal; >>>>>> -[ RECORD 1 ]-------+------------------------------ >>>>>> wal_records         | 2000224 >>>>>> wal_fpi             | 47 >>>>>> wal_bytes           | 248216337 >>>>>> wal_buffers_full    | 20954 >>>>>> wal_init_file       | 8 >>>>>> wal_write_backend   | 20960 >>>>>> wal_write_walwriter | 46 >>>>>> wal_write_time      | 51 >>>>>> wal_sync_backend    | 7 >>>>>> wal_sync_walwriter  | 8 >>>>>> wal_sync_time       | 0 >>>>>> stats_reset         | 2020-10-20 11:04:51.307771+09 >>>>>> ``` >>>>>> >>>>>> 1. Basic statistics of WAL activity >>>>>> >>>>>> - wal_records: Total number of WAL records generated >>>>>> - wal_fpi: Total number of WAL full page images generated >>>>>> - wal_bytes: Total amount of WAL bytes generated >>>>>> >>>>>> To understand DB's performance, first, we will check the >>>>>> performance >>>>>> trends for the entire database instance. >>>>>> For example, if the number of wal_fpi becomes higher, users may >>>>>> tune >>>>>> "wal_compression", "checkpoint_timeout" and so on. >>>>>> >>>>>> Although users can check the above statistics via EXPLAIN, >>>>>> auto_explain, >>>>>> autovacuum and pg_stat_statements now, >>>>>> if users want to see the performance trends  for the entire >>>>>> database, >>>>>> they must recalculate the statistics. >>>>>> >>>>>> I think it is useful to add the sum of the basic statistics. >>>>>> >>>>>> >>>>>> 2.  WAL segment file creation >>>>>> >>>>>> - wal_init_file: Total number of WAL segment files created. >>>>>> >>>>>> To create a new WAL file may have an impact on the performance of >>>>>> a write-heavy workload generating lots of WAL. If this number is >>>>>> reported high, >>>>>> to reduce the number of this initialization, we can tune >>>>>> WAL-related parameters >>>>>> so that more "recycled" WAL files can be held. >>>>>> >>>>>> >>>>>> >>>>>> 3. Number of when WAL is flushed >>>>>> >>>>>> - wal_write_backend : Total number of WAL data written to the disk >>>>>> by backends >>>>>> - wal_write_walwriter : Total number of WAL data written to the >>>>>> disk by walwriter >>>>>> - wal_sync_backend : Total number of WAL data synced to the disk >>>>>> by backends >>>>>> - wal_sync_walwriter : Total number of WAL data synced to the disk >>>>>> by walwrite >>>>>> >>>>>> I think it's useful for tuning "synchronous_commit" and >>>>>> "commit_delay" for query executions. >>>>>> If the number of WAL is flushed is high, users can know >>>>>> "synchronous_commit" is useful for the workload. >>>>> >>>>> I just wonder how useful these counters are. Even without these >>>>> counters, >>>>> we already know synchronous_commit=off is likely to cause the >>>>> better >>>>> performance (but has the risk of data loss). So ISTM that these >>>>> counters are >>>>> not so useful when tuning synchronous_commit. >>>> >>>> Thanks, my understanding was wrong. >>>> I agreed that your comments. >>>> >>>> I merged the statistics of *_backend and *_walwriter. >>>> I think the sum of them is useful to calculate the average per >>>> write/sync time. >>>> For example, per write time is equals wal_write_time / wal_write. >>> >>> Understood. >>> >>> Thanks for updating the patch! >> >> Thanks for your comments. >> >>> patching file src/include/catalog/pg_proc.dat >>> Hunk #1 FAILED at 5491. >>> 1 out of 1 hunk FAILED -- saving rejects to file >>> src/include/catalog/pg_proc.dat.rej >>> >>> I got this failure when applying the patch. Could you update the >>> patch? >> >> Thanks, I updated the patch. >> >>> -       Number of times WAL data was written to the disk because WAL >>> buffers got full >>> +       Total number of times WAL data written to the disk because >>> WAL >>> buffers got full >>> >>> Isn't "was" necessary between "data" and "written"? >> >> Yes, I fixed it. >> >>> +      >> role="column_definition"> >>> +       wal_bytes bigint >>> >>> Shouldn't the type of wal_bytes be numeric because the total number >>> of >>> WAL bytes can exceed the range of bigint? I think that the type of >>> pg_stat_statements.wal_bytes is also numeric for the same reason. >> >> Thanks, I fixed it. >> >> Since I cast the type of wal_bytes from PgStat_Counter to uint64, >> I changed the type of PgStat_MsgWal and PgStat_WalStats too. >> >>> +      >> role="column_definition"> >>> +       wal_write_time bigint >>> >>> Shouldn't the type of wal_xxx_time be double precision, >>> like pg_stat_database.blk_write_time? >> >> Thanks, I changed it. >> >>> Even when fsync is set to off or wal_sync_method is set to open_sync, >>> wal_sync is incremented. Isn't this behavior confusing? > > What do you think about this comment? Sorry, I'll change to increment wal_sync and wal_sync_time only if a specific fsync method is called. > I found that we discussed track-WAL-IO-timing feature at the past > discussion > about the similar feature [1]. But the feature was droppped from the > proposal > patch because there was the performance concern. So probably we need to > revisit the past discussion and benchmark the performance. Thought? > > If track-WAL-IO-timing feature may cause performance regression, > it might be an idea to extract wal_records, wal_fpi and wal_bytes parts > from the patch and commit it at first. > > [1] > https://postgr.es/m/CAJrrPGc6APFUGYNcPe4qcNxpL8gXKYv1KST+vwJcFtCSCEySnA@mail.gmail.com Thanks, I'll check the thread. I agree to add basic statistics at first and I attached the patch. >>> >>> >>> +       Total amount of time that has been spent in the portion of >>> +       WAL data was written to disk by backend and walwriter, in >>> milliseconds >>> +       (if is enabled, >>> otherwise zero) >>> >>> With the patch, track_io_timing controls both IO for data files and >>> WAL files. But we may want to track only either of them. So it's >>> better >>> to extend track_io_timing so that we can specify the tracking target >>> in the parameter? For example, we can make track_io_timing accept >>> data, wal and all. Or we should introduce new GUC for WAL, e.g., >>> track_wal_io_timing? Thought? >> >> OK, I introduced the new GUC "track_wal_io_timing". >> >>> I'm afraid that "by backend and walwriter" part can make us thinkg >>> incorrectly that WAL writes by other processes like autovacuum >>> are not tracked. >> >> Sorry, I removed "by backend and walwriter". > > Thanks for updating the patch! > > +WalUsage prevWalUsage; > > ISTM that we can declare this as static variable because > it's used only in pgstat.c. Thanks, I fixed it. > + memset(&walusage, 0, sizeof(WalUsage)); > + WalUsageAccumDiff(&walusage, &pgWalUsage, &prevWalUsage); > > This memset seems unnecessary. I couldn't understand why this memset is unnecessary. Since WalUsageAccumDiff not only calculates the difference but also adds the value, I thought walusage needs to be initialized. > /* We assume this initializes to zeroes */ > static const PgStat_MsgWal all_zeroes; > > This declaration of the variable should be placed around > the top of pgstat_send_wal(). Sorry, I fixed it. Regards, -- Masahiro Ikeda NTT DATA CORPORATION --=_a38ec8be5c94edba0f02bfe32f890fa8 Content-Transfer-Encoding: base64 Content-Type: text/x-diff; name=0001_add_basic_statistics_to_pg_stat_wal_view Content-Disposition: attachment; filename=0001_add_basic_statistics_to_pg_stat_wal_view; size=8957 ZGlmZiAtLWdpdCBhL2RvYy9zcmMvc2dtbC9tb25pdG9yaW5nLnNnbWwgYi9kb2Mvc3JjL3NnbWwv bW9uaXRvcmluZy5zZ21sCmluZGV4IDk4ZTE5OTU0NTMuLjZlNjYxZDZmYWUgMTAwNjQ0Ci0tLSBh L2RvYy9zcmMvc2dtbC9tb25pdG9yaW5nLnNnbWwKKysrIGIvZG9jL3NyYy9zZ21sL21vbml0b3Jp bmcuc2dtbApAQCAtMzQ0Nyw2ICszNDQ3LDMzIEBAIFNFTEVDVCBwaWQsIHdhaXRfZXZlbnRfdHlw ZSwgd2FpdF9ldmVudCBGUk9NIHBnX3N0YXRfYWN0aXZpdHkgV0hFUkUgd2FpdF9ldmVudCBpCiAg ICAgPC90aGVhZD4KIAogICAgIDx0Ym9keT4KKyAgICAgPHJvdz4KKyAgICAgIDxlbnRyeSByb2xl PSJjYXRhbG9nX3RhYmxlX2VudHJ5Ij48cGFyYSByb2xlPSJjb2x1bW5fZGVmaW5pdGlvbiI+Cisg ICAgICAgPHN0cnVjdGZpZWxkPndhbF9yZWNvcmRzPC9zdHJ1Y3RmaWVsZD4gPHR5cGU+YmlnaW50 PC90eXBlPgorICAgICAgPC9wYXJhPgorICAgICAgPHBhcmE+CisgICAgICAgVG90YWwgbnVtYmVy IG9mIFdBTCByZWNvcmRzIGdlbmVyYXRlZAorICAgICAgPC9wYXJhPjwvZW50cnk+CisgICAgIDwv cm93PgorCisgICAgIDxyb3c+CisgICAgICA8ZW50cnkgcm9sZT0iY2F0YWxvZ190YWJsZV9lbnRy eSI+PHBhcmEgcm9sZT0iY29sdW1uX2RlZmluaXRpb24iPgorICAgICAgIDxzdHJ1Y3RmaWVsZD53 YWxfZnBpPC9zdHJ1Y3RmaWVsZD4gPHR5cGU+YmlnaW50PC90eXBlPgorICAgICAgPC9wYXJhPgor ICAgICAgPHBhcmE+CisgICAgICAgVG90YWwgbnVtYmVyIG9mIFdBTCBmdWxsIHBhZ2UgaW1hZ2Vz IGdlbmVyYXRlZAorICAgICAgPC9wYXJhPjwvZW50cnk+CisgICAgIDwvcm93PgorCisgICAgIDxy b3c+CisgICAgICA8ZW50cnkgcm9sZT0iY2F0YWxvZ190YWJsZV9lbnRyeSI+PHBhcmEgcm9sZT0i Y29sdW1uX2RlZmluaXRpb24iPgorICAgICAgIDxzdHJ1Y3RmaWVsZD53YWxfYnl0ZXM8L3N0cnVj dGZpZWxkPiA8dHlwZT5udW1lcmljPC90eXBlPgorICAgICAgPC9wYXJhPgorICAgICAgPHBhcmE+ CisgICAgICAgVG90YWwgYW1vdW50IG9mIFdBTCBieXRlcyBnZW5lcmF0ZWQKKyAgICAgIDwvcGFy YT48L2VudHJ5PgorICAgICA8L3Jvdz4KKwogICAgICA8cm93PgogICAgICAgPGVudHJ5IHJvbGU9 ImNhdGFsb2dfdGFibGVfZW50cnkiPjxwYXJhIHJvbGU9ImNvbHVtbl9kZWZpbml0aW9uIj4KICAg ICAgICA8c3RydWN0ZmllbGQ+d2FsX2J1ZmZlcnNfZnVsbDwvc3RydWN0ZmllbGQ+IDx0eXBlPmJp Z2ludDwvdHlwZT4KZGlmZiAtLWdpdCBhL3NyYy9iYWNrZW5kL2NhdGFsb2cvc3lzdGVtX3ZpZXdz LnNxbCBiL3NyYy9iYWNrZW5kL2NhdGFsb2cvc3lzdGVtX3ZpZXdzLnNxbAppbmRleCAyZTRhYTFj NGI2Li5iMTQwYzIxMGJjIDEwMDY0NAotLS0gYS9zcmMvYmFja2VuZC9jYXRhbG9nL3N5c3RlbV92 aWV3cy5zcWwKKysrIGIvc3JjL2JhY2tlbmQvY2F0YWxvZy9zeXN0ZW1fdmlld3Muc3FsCkBAIC05 OTMsNiArOTkzLDkgQEAgQ1JFQVRFIFZJRVcgcGdfc3RhdF9iZ3dyaXRlciBBUwogCiBDUkVBVEUg VklFVyBwZ19zdGF0X3dhbCBBUwogICAgIFNFTEVDVAorICAgICAgICB3LndhbF9yZWNvcmRzLAor ICAgICAgICB3LndhbF9mcGksCisgICAgICAgIHcud2FsX2J5dGVzLAogICAgICAgICB3LndhbF9i dWZmZXJzX2Z1bGwsCiAgICAgICAgIHcuc3RhdHNfcmVzZXQKICAgICBGUk9NIHBnX3N0YXRfZ2V0 X3dhbCgpIHc7CmRpZmYgLS1naXQgYS9zcmMvYmFja2VuZC9wb3N0bWFzdGVyL3Bnc3RhdC5jIGIv c3JjL2JhY2tlbmQvcG9zdG1hc3Rlci9wZ3N0YXQuYwppbmRleCBlNzZlNjI3YzZiLi43ODFmMmE1 MjQ4IDEwMDY0NAotLS0gYS9zcmMvYmFja2VuZC9wb3N0bWFzdGVyL3Bnc3RhdC5jCisrKyBiL3Ny Yy9iYWNrZW5kL3Bvc3RtYXN0ZXIvcGdzdGF0LmMKQEAgLTQxLDYgKzQxLDcgQEAKICNpbmNsdWRl ICJjYXRhbG9nL3BnX2RhdGFiYXNlLmgiCiAjaW5jbHVkZSAiY2F0YWxvZy9wZ19wcm9jLmgiCiAj aW5jbHVkZSAiY29tbW9uL2lwLmgiCisjaW5jbHVkZSAiZXhlY3V0b3IvaW5zdHJ1bWVudC5oIgog I2luY2x1ZGUgImxpYnBxL2xpYnBxLmgiCiAjaW5jbHVkZSAibGlicHEvcHFzaWduYWwuaCIKICNp bmNsdWRlICJtYi9wZ193Y2hhci5oIgpAQCAtMTQyLDYgKzE0Myw3IEBAIGNoYXIJICAgKnBnc3Rh dF9zdGF0X3RtcG5hbWUgPSBOVUxMOwogICovCiBQZ1N0YXRfTXNnQmdXcml0ZXIgQmdXcml0ZXJT dGF0czsKIFBnU3RhdF9Nc2dXYWwgV2FsU3RhdHM7CitzdGF0aWMgV2FsVXNhZ2UgcHJldldhbFVz YWdlOwogCiAvKgogICogTGlzdCBvZiBTTFJVIG5hbWVzIHRoYXQgd2Uga2VlcCBzdGF0cyBmb3Iu ICBUaGVyZSBpcyBubyBjZW50cmFsIHJlZ2lzdHJ5IG9mCkBAIC0zMDQ4LDYgKzMwNTAsOSBAQCBw Z3N0YXRfaW5pdGlhbGl6ZSh2b2lkKQogCQlNeUJFRW50cnkgPSAmQmFja2VuZFN0YXR1c0FycmF5 W01heEJhY2tlbmRzICsgTXlBdXhQcm9jVHlwZV07CiAJfQogCisJLyogSW5pdGlhbGl6ZSBXQUwg dXNhZ2Ugc3RhdHVzIGluZm9ybWF0aW9uICovCisJcHJldldhbFVzYWdlID0gcGdXYWxVc2FnZTsK KwogCS8qIFNldCB1cCBhIHByb2Nlc3MtZXhpdCBob29rIHRvIGNsZWFuIHVwICovCiAJb25fc2ht ZW1fZXhpdChwZ3N0YXRfYmVzaHV0ZG93bl9ob29rLCAwKTsKIH0KQEAgLTQ1NzcsNiArNDU4Miwx NSBAQCBwZ3N0YXRfc2VuZF93YWwodm9pZCkKIAkvKiBXZSBhc3N1bWUgdGhpcyBpbml0aWFsaXpl cyB0byB6ZXJvZXMgKi8KIAlzdGF0aWMgY29uc3QgUGdTdGF0X01zZ1dhbCBhbGxfemVyb2VzOwog CisJLyogZmlsbCBpbiBzb21lIHZhbHVlcyB1c2luZyBwZ1dhbFVzYWdlICovCisJV2FsVXNhZ2Ug d2FsdXNhZ2U7CisJbWVtc2V0KCZ3YWx1c2FnZSwgMCwgc2l6ZW9mKFdhbFVzYWdlKSk7CisJV2Fs VXNhZ2VBY2N1bURpZmYoJndhbHVzYWdlLCAmcGdXYWxVc2FnZSwgJnByZXZXYWxVc2FnZSk7CisK KwlXYWxTdGF0cy5tX3dhbF9ieXRlcyA9IHdhbHVzYWdlLndhbF9ieXRlczsKKwlXYWxTdGF0cy5t X3dhbF9yZWNvcmRzID0gd2FsdXNhZ2Uud2FsX3JlY29yZHM7CisJV2FsU3RhdHMubV93YWxfZnBp ID0gd2FsdXNhZ2Uud2FsX2ZwaTsKKwogCS8qCiAJICogVGhpcyBmdW5jdGlvbiBjYW4gYmUgY2Fs bGVkIGV2ZW4gaWYgbm90aGluZyBhdCBhbGwgaGFzIGhhcHBlbmVkLiBJbgogCSAqIHRoaXMgY2Fz ZSwgYXZvaWQgc2VuZGluZyBhIGNvbXBsZXRlbHkgZW1wdHkgbWVzc2FnZSB0byB0aGUgc3RhdHMK QEAgLTQ1OTEsNiArNDYwNSw5IEBAIHBnc3RhdF9zZW5kX3dhbCh2b2lkKQogCXBnc3RhdF9zZXRo ZWFkZXIoJldhbFN0YXRzLm1faGRyLCBQR1NUQVRfTVRZUEVfV0FMKTsKIAlwZ3N0YXRfc2VuZCgm V2FsU3RhdHMsIHNpemVvZihXYWxTdGF0cykpOwogCisJLyogcmVtZW1iZXIgdGhlIGN1cnJlbnQg bnVtYmVycyAqLworCXByZXZXYWxVc2FnZSA9IHBnV2FsVXNhZ2U7CisKIAkvKgogCSAqIENsZWFy IG91dCB0aGUgc3RhdGlzdGljcyBidWZmZXIsIHNvIGl0IGNhbiBiZSByZS11c2VkLgogCSAqLwpA QCAtNjc1OSw2ICs2Nzc2LDkgQEAgcGdzdGF0X3JlY3ZfYmd3cml0ZXIoUGdTdGF0X01zZ0JnV3Jp dGVyICptc2csIGludCBsZW4pCiBzdGF0aWMgdm9pZAogcGdzdGF0X3JlY3Zfd2FsKFBnU3RhdF9N c2dXYWwgKm1zZywgaW50IGxlbikKIHsKKwl3YWxTdGF0cy53YWxfcmVjb3JkcyArPSBtc2ctPm1f d2FsX3JlY29yZHM7CisJd2FsU3RhdHMud2FsX2ZwaSArPSBtc2ctPm1fd2FsX2ZwaTsKKwl3YWxT dGF0cy53YWxfYnl0ZXMgKz0gbXNnLT5tX3dhbF9ieXRlczsKIAl3YWxTdGF0cy53YWxfYnVmZmVy c19mdWxsICs9IG1zZy0+bV93YWxfYnVmZmVyc19mdWxsOwogfQogCmRpZmYgLS1naXQgYS9zcmMv YmFja2VuZC9wb3N0bWFzdGVyL3dhbHdyaXRlci5jIGIvc3JjL2JhY2tlbmQvcG9zdG1hc3Rlci93 YWx3cml0ZXIuYwppbmRleCBhNTI4MzJmZTkwLi5jZTlmNGI3Y2Y3IDEwMDY0NAotLS0gYS9zcmMv YmFja2VuZC9wb3N0bWFzdGVyL3dhbHdyaXRlci5jCisrKyBiL3NyYy9iYWNrZW5kL3Bvc3RtYXN0 ZXIvd2Fsd3JpdGVyLmMKQEAgLTI1Myw2ICsyNTMsOSBAQCBXYWxXcml0ZXJNYWluKHZvaWQpCiAJ CWVsc2UgaWYgKGxlZnRfdGlsbF9oaWJlcm5hdGUgPiAwKQogCQkJbGVmdF90aWxsX2hpYmVybmF0 ZS0tOwogCisJCS8qIFNlbmQgV0FMIHN0YXRpc3RpY3MgKi8KKwkJcGdzdGF0X3NlbmRfd2FsKCk7 CisKIAkJLyoKIAkJICogU2xlZXAgdW50aWwgd2UgYXJlIHNpZ25hbGVkIG9yIFdhbFdyaXRlckRl bGF5IGhhcyBlbGFwc2VkLiAgSWYgd2UKIAkJICogaGF2ZW4ndCBkb25lIGFueXRoaW5nIHVzZWZ1 bCBmb3IgcXVpdGUgc29tZSB0aW1lLCBsZW5ndGhlbiB0aGUKZGlmZiAtLWdpdCBhL3NyYy9iYWNr ZW5kL3V0aWxzL2FkdC9wZ3N0YXRmdW5jcy5jIGIvc3JjL2JhY2tlbmQvdXRpbHMvYWR0L3Bnc3Rh dGZ1bmNzLmMKaW5kZXggYTIxMGZjOTNiNC4uNmFmZTFiNmY1NiAxMDA2NDQKLS0tIGEvc3JjL2Jh Y2tlbmQvdXRpbHMvYWR0L3Bnc3RhdGZ1bmNzLmMKKysrIGIvc3JjL2JhY2tlbmQvdXRpbHMvYWR0 L3Bnc3RhdGZ1bmNzLmMKQEAgLTE3MDMsMTAgKzE3MDMsMTEgQEAgcGdfc3RhdF9nZXRfYnVmX2Fs bG9jKFBHX0ZVTkNUSU9OX0FSR1MpCiBEYXR1bQogcGdfc3RhdF9nZXRfd2FsKFBHX0ZVTkNUSU9O X0FSR1MpCiB7Ci0jZGVmaW5lIFBHX1NUQVRfR0VUX1dBTF9DT0xTCTIKKyNkZWZpbmUgUEdfU1RB VF9HRVRfV0FMX0NPTFMJNQogCVR1cGxlRGVzYwl0dXBkZXNjOwogCURhdHVtCQl2YWx1ZXNbUEdf U1RBVF9HRVRfV0FMX0NPTFNdOwogCWJvb2wJCW51bGxzW1BHX1NUQVRfR0VUX1dBTF9DT0xTXTsK KwljaGFyCQlidWZbMjU2XTsKIAlQZ1N0YXRfV2FsU3RhdHMgKndhbF9zdGF0czsKIAogCS8qIElu aXRpYWxpc2UgdmFsdWVzIGFuZCBOVUxMIGZsYWdzIGFycmF5cyAqLwpAQCAtMTcxNSw5ICsxNzE2 LDE1IEBAIHBnX3N0YXRfZ2V0X3dhbChQR19GVU5DVElPTl9BUkdTKQogCiAJLyogSW5pdGlhbGlz ZSBhdHRyaWJ1dGVzIGluZm9ybWF0aW9uIGluIHRoZSB0dXBsZSBkZXNjcmlwdG9yICovCiAJdHVw ZGVzYyA9IENyZWF0ZVRlbXBsYXRlVHVwbGVEZXNjKFBHX1NUQVRfR0VUX1dBTF9DT0xTKTsKLQlU dXBsZURlc2NJbml0RW50cnkodHVwZGVzYywgKEF0dHJOdW1iZXIpIDEsICJ3YWxfYnVmZmVyc19m dWxsIiwKKwlUdXBsZURlc2NJbml0RW50cnkodHVwZGVzYywgKEF0dHJOdW1iZXIpIDEsICJ3YWxf cmVjb3JkcyIsCiAJCQkJCSAgIElOVDhPSUQsIC0xLCAwKTsKLQlUdXBsZURlc2NJbml0RW50cnko dHVwZGVzYywgKEF0dHJOdW1iZXIpIDIsICJzdGF0c19yZXNldCIsCisJVHVwbGVEZXNjSW5pdEVu dHJ5KHR1cGRlc2MsIChBdHRyTnVtYmVyKSAyLCAid2FsX2ZwaSIsCisJCQkJCSAgIElOVDhPSUQs IC0xLCAwKTsKKwlUdXBsZURlc2NJbml0RW50cnkodHVwZGVzYywgKEF0dHJOdW1iZXIpIDMsICJ3 YWxfYnl0ZXMiLAorCQkJCQkgICBOVU1FUklDT0lELCAtMSwgMCk7CisJVHVwbGVEZXNjSW5pdEVu dHJ5KHR1cGRlc2MsIChBdHRyTnVtYmVyKSA0LCAid2FsX2J1ZmZlcnNfZnVsbCIsCisJCQkJCSAg IElOVDhPSUQsIC0xLCAwKTsKKwlUdXBsZURlc2NJbml0RW50cnkodHVwZGVzYywgKEF0dHJOdW1i ZXIpIDUsICJzdGF0c19yZXNldCIsCiAJCQkJCSAgIFRJTUVTVEFNUFRaT0lELCAtMSwgMCk7CiAK IAlCbGVzc1R1cGxlRGVzYyh0dXBkZXNjKTsKQEAgLTE3MjYsOCArMTczMywxOCBAQCBwZ19zdGF0 X2dldF93YWwoUEdfRlVOQ1RJT05fQVJHUykKIAl3YWxfc3RhdHMgPSBwZ3N0YXRfZmV0Y2hfc3Rh dF93YWwoKTsKIAogCS8qIEZpbGwgdmFsdWVzIGFuZCBOVUxMcyAqLwotCXZhbHVlc1swXSA9IElu dDY0R2V0RGF0dW0od2FsX3N0YXRzLT53YWxfYnVmZmVyc19mdWxsKTsKLQl2YWx1ZXNbMV0gPSBU aW1lc3RhbXBUekdldERhdHVtKHdhbF9zdGF0cy0+c3RhdF9yZXNldF90aW1lc3RhbXApOworCXZh bHVlc1swXSA9IEludDY0R2V0RGF0dW0od2FsX3N0YXRzLT53YWxfcmVjb3Jkcyk7CisJdmFsdWVz WzFdID0gSW50NjRHZXREYXR1bSh3YWxfc3RhdHMtPndhbF9mcGkpOworCisJLyogQ29udmVydCB0 byBudW1lcmljLiAqLworCXNucHJpbnRmKGJ1Ziwgc2l6ZW9mIGJ1ZiwgVUlOVDY0X0ZPUk1BVCwg d2FsX3N0YXRzLT53YWxfYnl0ZXMpOworCXZhbHVlc1syXSA9IERpcmVjdEZ1bmN0aW9uQ2FsbDMo bnVtZXJpY19pbiwKKwkJCQkJCQkJCUNTdHJpbmdHZXREYXR1bShidWYpLAorCQkJCQkJCQkJT2Jq ZWN0SWRHZXREYXR1bSgwKSwKKwkJCQkJCQkJCUludDMyR2V0RGF0dW0oLTEpKTsKKworCXZhbHVl c1szXSA9IEludDY0R2V0RGF0dW0od2FsX3N0YXRzLT53YWxfYnVmZmVyc19mdWxsKTsKKwl2YWx1 ZXNbNF0gPSBUaW1lc3RhbXBUekdldERhdHVtKHdhbF9zdGF0cy0+c3RhdF9yZXNldF90aW1lc3Rh bXApOwogCiAJLyogUmV0dXJucyB0aGUgcmVjb3JkIGFzIERhdHVtICovCiAJUEdfUkVUVVJOX0RB VFVNKEhlYXBUdXBsZUdldERhdHVtKGhlYXBfZm9ybV90dXBsZSh0dXBkZXNjLCB2YWx1ZXMsIG51 bGxzKSkpOwpkaWZmIC0tZ2l0IGEvc3JjL2luY2x1ZGUvY2F0YWxvZy9wZ19wcm9jLmRhdCBiL3Ny Yy9pbmNsdWRlL2NhdGFsb2cvcGdfcHJvYy5kYXQKaW5kZXggYzAxZGE0YmYwMS4uNjdhYjg2MTI2 MCAxMDA2NDQKLS0tIGEvc3JjL2luY2x1ZGUvY2F0YWxvZy9wZ19wcm9jLmRhdAorKysgYi9zcmMv aW5jbHVkZS9jYXRhbG9nL3BnX3Byb2MuZGF0CkBAIC01NTAzLDggKzU1MDMsOSBAQAogeyBvaWQg PT4gJzExMzYnLCBkZXNjciA9PiAnc3RhdGlzdGljczogaW5mb3JtYXRpb24gYWJvdXQgV0FMIGFj dGl2aXR5JywKICAgcHJvbmFtZSA9PiAncGdfc3RhdF9nZXRfd2FsJywgcHJvaXNzdHJpY3QgPT4g J2YnLCBwcm92b2xhdGlsZSA9PiAncycsCiAgIHByb3BhcmFsbGVsID0+ICdyJywgcHJvcmV0dHlw ZSA9PiAncmVjb3JkJywgcHJvYXJndHlwZXMgPT4gJycsCi0gIHByb2FsbGFyZ3R5cGVzID0+ICd7 aW50OCx0aW1lc3RhbXB0en0nLCBwcm9hcmdtb2RlcyA9PiAne28sb30nLAotICBwcm9hcmduYW1l cyA9PiAne3dhbF9idWZmZXJzX2Z1bGwsc3RhdHNfcmVzZXR9JywKKyAgIHByb2FsbGFyZ3R5cGVz ID0+ICd7aW50OCxpbnQ4LG51bWVyaWMsaW50OCx0aW1lc3RhbXB0en0nLAorICAgcHJvYXJnbW9k ZXMgPT4gJ3tvLG8sbyxvLG99JywKKyAgIHByb2FyZ25hbWVzID0+ICd7d2FsX3JlY29yZHMsd2Fs X2ZwaSx3YWxfYnl0ZXMsd2FsX2J1ZmZlcnNfZnVsbCxzdGF0c19yZXNldH0nLAogICBwcm9zcmMg PT4gJ3BnX3N0YXRfZ2V0X3dhbCcgfSwKIAogeyBvaWQgPT4gJzIzMDYnLCBkZXNjciA9PiAnc3Rh dGlzdGljczogaW5mb3JtYXRpb24gYWJvdXQgU0xSVSBjYWNoZXMnLApkaWZmIC0tZ2l0IGEvc3Jj L2luY2x1ZGUvcGdzdGF0LmggYi9zcmMvaW5jbHVkZS9wZ3N0YXQuaAppbmRleCAyNTdlNTE1YmZl Li5jZWUzZDI4YTJkIDEwMDY0NAotLS0gYS9zcmMvaW5jbHVkZS9wZ3N0YXQuaAorKysgYi9zcmMv aW5jbHVkZS9wZ3N0YXQuaApAQCAtNDU5LDYgKzQ1OSw5IEBAIHR5cGVkZWYgc3RydWN0IFBnU3Rh dF9Nc2dCZ1dyaXRlcgogdHlwZWRlZiBzdHJ1Y3QgUGdTdGF0X01zZ1dhbAogewogCVBnU3RhdF9N c2dIZHIgbV9oZHI7CisJUGdTdGF0X0NvdW50ZXIgbV93YWxfcmVjb3JkczsKKwlQZ1N0YXRfQ291 bnRlciBtX3dhbF9mcGk7CisJdWludDY0IG1fd2FsX2J5dGVzOwogCVBnU3RhdF9Db3VudGVyIG1f d2FsX2J1ZmZlcnNfZnVsbDsKIH0gUGdTdGF0X01zZ1dhbDsKIApAQCAtNzk4LDYgKzgwMSw5IEBA IHR5cGVkZWYgc3RydWN0IFBnU3RhdF9HbG9iYWxTdGF0cwogICovCiB0eXBlZGVmIHN0cnVjdCBQ Z1N0YXRfV2FsU3RhdHMKIHsKKwlQZ1N0YXRfQ291bnRlciB3YWxfcmVjb3JkczsKKwlQZ1N0YXRf Q291bnRlciB3YWxfZnBpOworCXVpbnQ2NCB3YWxfYnl0ZXM7CiAJUGdTdGF0X0NvdW50ZXIgd2Fs X2J1ZmZlcnNfZnVsbDsKIAlUaW1lc3RhbXBUeiBzdGF0X3Jlc2V0X3RpbWVzdGFtcDsKIH0gUGdT dGF0X1dhbFN0YXRzOwpkaWZmIC0tZ2l0IGEvc3JjL3Rlc3QvcmVncmVzcy9leHBlY3RlZC9ydWxl cy5vdXQgYi9zcmMvdGVzdC9yZWdyZXNzL2V4cGVjdGVkL3J1bGVzLm91dAppbmRleCAwOTdmZjVk MTExLi42MjkzYWI1N2JjIDEwMDY0NAotLS0gYS9zcmMvdGVzdC9yZWdyZXNzL2V4cGVjdGVkL3J1 bGVzLm91dAorKysgYi9zcmMvdGVzdC9yZWdyZXNzL2V4cGVjdGVkL3J1bGVzLm91dApAQCAtMjEz OCw5ICsyMTM4LDEyIEBAIHBnX3N0YXRfdXNlcl90YWJsZXN8IFNFTEVDVCBwZ19zdGF0X2FsbF90 YWJsZXMucmVsaWQsCiAgICAgcGdfc3RhdF9hbGxfdGFibGVzLmF1dG9hbmFseXplX2NvdW50CiAg ICBGUk9NIHBnX3N0YXRfYWxsX3RhYmxlcwogICBXSEVSRSAoKHBnX3N0YXRfYWxsX3RhYmxlcy5z Y2hlbWFuYW1lIDw+IEFMTCAoQVJSQVlbJ3BnX2NhdGFsb2cnOjpuYW1lLCAnaW5mb3JtYXRpb25f c2NoZW1hJzo6bmFtZV0pKSBBTkQgKHBnX3N0YXRfYWxsX3RhYmxlcy5zY2hlbWFuYW1lICF+ICde cGdfdG9hc3QnOjp0ZXh0KSk7Ci1wZ19zdGF0X3dhbHwgU0VMRUNUIHcud2FsX2J1ZmZlcnNfZnVs bCwKK3BnX3N0YXRfd2FsfCBTRUxFQ1Qgdy53YWxfcmVjb3JkcywKKyAgICB3LndhbF9mcGksCisg ICAgdy53YWxfYnl0ZXMsCisgICAgdy53YWxfYnVmZmVyc19mdWxsLAogICAgIHcuc3RhdHNfcmVz ZXQKLSAgIEZST00gcGdfc3RhdF9nZXRfd2FsKCkgdyh3YWxfYnVmZmVyc19mdWxsLCBzdGF0c19y ZXNldCk7CisgICBGUk9NIHBnX3N0YXRfZ2V0X3dhbCgpIHcod2FsX3JlY29yZHMsIHdhbF9mcGks IHdhbF9ieXRlcywgd2FsX2J1ZmZlcnNfZnVsbCwgc3RhdHNfcmVzZXQpOwogcGdfc3RhdF93YWxf cmVjZWl2ZXJ8IFNFTEVDVCBzLnBpZCwKICAgICBzLnN0YXR1cywKICAgICBzLnJlY2VpdmVfc3Rh cnRfbHNuLAo= --=_a38ec8be5c94edba0f02bfe32f890fa8--