Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1tEGFE-006NEB-HV for pgsql-hackers@arkaria.postgresql.org; Thu, 21 Nov 2024 23:05:53 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1tEGFC-002hyO-3T for pgsql-hackers@arkaria.postgresql.org; Thu, 21 Nov 2024 23:05:50 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1tEGFA-002hyG-OG for pgsql-hackers@lists.postgresql.org; Thu, 21 Nov 2024 23:05:49 +0000 Received: from relay4-d.mail.gandi.net ([2001:4b98:dc4:8::224]) by magus.postgresql.org with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1tEGF8-003BEd-1Z for pgsql-hackers@lists.postgresql.org; Thu, 21 Nov 2024 23:05:47 +0000 Received: by mail.gandi.net (Postfix) with ESMTPSA id 13933E0002 for ; Thu, 21 Nov 2024 23:05:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vondra.me; s=gm1; t=1732230342; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Ivnfp5GA6wi+OJYwCqJZAPI6DHHyF+D3UfWmeiOqs7I=; b=JSi16H3Lxg0925dFwThXl+MJPhULw3dbnnU/cgHczLm4n3oIQP3Cc98tyTQcRh713mkXPP W+eDS5dZezPSKZ8X1WZX+fRShkHufgz4A++OzPeSeCgUFwKYyUfUnxjQBnWxMsWqfCWnUq eBbICF9neyf3uMwkLkMl1r4DU/6gpbhkpILeuTC8srbN1D+hcNtS5Kwcr28J915m4HOOxq PZ6XSvDUGJ0iiRn1yo4Azmr+MA3Fye0dZwDI4CjWB+1u3fHnvOREzBvJbdW+8tyammtbBi 45EwZ24A0FNQC3WRZDEnyD+joXLBMssrrR3RUwVeQdvcQNf5I5ATEHTRcojRZw== Content-Type: multipart/mixed; boundary="------------X5LK3I0KWYHP7scWJn47T3Jp" Message-ID: Date: Fri, 22 Nov 2024 00:05:40 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Slot's restart_lsn may point to removed WAL segment after hard restart unexpectedly From: Tomas Vondra To: pgsql-hackers@lists.postgresql.org References: <1d12d2-67235980-35-19a406a0@63439497> <683f3c82-d38e-436d-88fd-27722af62005@vondra.me> <8679fea7-94ce-4a52-8e48-1a8cd0857fcb@vondra.me> Content-Language: en-US In-Reply-To: X-GND-Sasl: tomas@vondra.me List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk This is a multi-part message in MIME format. --------------X5LK3I0KWYHP7scWJn47T3Jp Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit On 11/21/24 14:59, Tomas Vondra wrote: > > ... > > But then there's the SQL API - pg_logical_slot_get_changes(). And it > turns out it ends up syncing the slot to disk pretty often, because for > RUNNING_XACTS we call LogicalDecodingProcessRecord() + standby_decode(), > which ends up calling SaveSlotToDisk(). And at the end we call > LogicalConfirmReceivedLocation() for good measure, which saves the slot > too, just to be sure. > > FWIW I suspect this still is not perfectly safe, because we may still > crash / restart before the updated data.restart_lsn makes it to disk, > but after it was already used to remove old WAL, although that's > probably harder to hit. With streaming the subscriber will still send us > the new startpoint, so that should not fail I think. But with the SQL > API we probably can get into the "segment already removed" issues. > > I haven't tried reproducing this yet, I guess it should be possible > using the injection points. Not sure when I get to this, though. > I kept pulling on this loose thread, and the deeper I look the more I'm concvinced ReplicationSlotsComputeRequiredLSN() is fundamentally unsafe. I may be missing something, of course, in which case I'd be grateful if someone could correct me. I believe the main problem is that ReplicationSlotsComputeRequiredLSN() operates on data that may not be on-disk yet. It just iterates over slots in shared memory, looks at the data.restart_lsn, and rolls with that. So some of the data may be lost after a crash or "immediate" restart, which for restart_lsn means it can move backwards by some unknown amount. Unfortunately, some of the callers use the value as if it was durable, and do irreversible actions based on it. This whole thread is about checkpointer using the value to discard WAL supposedly not required by any slot, only to find out we're missing WAL. That seems like a rather fundamental problem, and the only reason why we don't see this causing trouble more often is that (a) abrupt restarts are not very common, (b) most slots are likely not lagging very much, and thus not in danger of actually losing WAL, and (c) the streaming replication tracks startpoint, which masks the issue. But with the SQL API it's quite simple to cause issues with the right timing, as I'll show in a bit. There's an interesting difference in how different places update the slot. For example LogicalConfirmReceivedLocation() does this: 1) update slot->data.restart_lsn 2) mark slot dirty: ReplicationSlotMarkDirty() 3) save slot to disk: ReplicationSlotSave() 4) recalculate required LSN: ReplicationSlotsComputeRequiredLSN() while pg_replication_slot_advance() does only this: 1) update slot->data.restart_lsn 2) mark slot dirty: ReplicationSlotMarkDirty() 3) recalculate required LSN: ReplicationSlotsComputeRequiredLSN() That is, it doesn't save the slot to disk. It just updates the LSN and them proceeds to recalculate the "required LSN" for all slots. That makes is very easy to hit the issue, as demonstrated by Vitaly. However, it doesn't mean LogicalConfirmReceivedLocation() is safe. It would be safe without concurrency, but it can happen that the logical decoding does (1) and maybe (2), but before the slot gets persisted, some other session gets to call ReplicationSlotsComputeRequiredLSN(). It might be logical decoding on another slot, or advance of a physical slot. I haven't checked what else can trigger that. So ultimately logical slots have exactly the same issue. Attached are two patches, demonstrating the issue. 0001 adds injection points into two places - before (2) in LogicalConfirmReceivedLocation, and before removal of old WAL in a checkpoint. 0002 then adds a simple TAP test triggering the issue in pg_logical_slot_get_changes(), leading to: ERROR: requested WAL segment pg_wal/000000010000000000000001 has already been removed The same issue could be demonstrated on a physical slot - it would actually be simpler, I think. I've been unable to cause issues for streaming replication (both physical and logical), because the subscriber sends startpoint which adjusts the restart_lsn to a "good" value. But I'm not sure if that's reliable in all cases, or if the replication could break too. It's entirely possible this behavior is common knowledge, but it was a surprise for me. Even if the streaming replication is safe, it does seem to make using the SQL functions less reliable (not that it doesn't have other challenges, e.g. with Ctrl-C). But maybe it could be made safer? I don't have a great idea how to improve this. It seems wrong for ReplicationSlotsComputeRequiredLSN() to calculate the LSN using values from dirty slots, so maybe it should simply retry if any slot is dirty? Or retry on that one slot? But various places update the restart_lsn before marking the slot as dirty, so right now this won't work. regards -- Tomas Vondra --------------X5LK3I0KWYHP7scWJn47T3Jp Content-Type: text/x-patch; charset=UTF-8; name="0002-TAP-test.patch" Content-Disposition: attachment; filename="0002-TAP-test.patch" Content-Transfer-Encoding: base64 RnJvbSA3OWEwNDU3MjhiMDkyMzcyMzRmMjMxMzBhMmE3MTBlMWJkZGU3ODcwIE1vbiBTZXAg MTcgMDA6MDA6MDAgMjAwMQpGcm9tOiBUb21hcyBWb25kcmEgPHRvbWFzQHZvbmRyYS5tZT4K RGF0ZTogVGh1LCAyMSBOb3YgMjAyNCAyMzowNzoyMiArMDEwMApTdWJqZWN0OiBbUEFUQ0gg Mi8yXSBUQVAgdGVzdAoKLS0tCiBzcmMvdGVzdC9tb2R1bGVzL3Rlc3RfcmVxdWlyZWRfbHNu L01ha2VmaWxlICAgfCAgMTggKysrCiAuLi4vbW9kdWxlcy90ZXN0X3JlcXVpcmVkX2xzbi9t ZXNvbi5idWlsZCAgICAgfCAgMTUgKysrCiAuLi4vdGVzdF9yZXF1aXJlZF9sc24vdC8wMDFf bG9naWNhbF9zbG90LnBsICAgfCAxMjYgKysrKysrKysrKysrKysrKysrCiAzIGZpbGVzIGNo YW5nZWQsIDE1OSBpbnNlcnRpb25zKCspCiBjcmVhdGUgbW9kZSAxMDA2NDQgc3JjL3Rlc3Qv bW9kdWxlcy90ZXN0X3JlcXVpcmVkX2xzbi9NYWtlZmlsZQogY3JlYXRlIG1vZGUgMTAwNjQ0 IHNyYy90ZXN0L21vZHVsZXMvdGVzdF9yZXF1aXJlZF9sc24vbWVzb24uYnVpbGQKIGNyZWF0 ZSBtb2RlIDEwMDY0NCBzcmMvdGVzdC9tb2R1bGVzL3Rlc3RfcmVxdWlyZWRfbHNuL3QvMDAx X2xvZ2ljYWxfc2xvdC5wbAoKZGlmZiAtLWdpdCBhL3NyYy90ZXN0L21vZHVsZXMvdGVzdF9y ZXF1aXJlZF9sc24vTWFrZWZpbGUgYi9zcmMvdGVzdC9tb2R1bGVzL3Rlc3RfcmVxdWlyZWRf bHNuL01ha2VmaWxlCm5ldyBmaWxlIG1vZGUgMTAwNjQ0CmluZGV4IDAwMDAwMDAwMDAwLi4z ZWIyYjAyZDM4ZgotLS0gL2Rldi9udWxsCisrKyBiL3NyYy90ZXN0L21vZHVsZXMvdGVzdF9y ZXF1aXJlZF9sc24vTWFrZWZpbGUKQEAgLTAsMCArMSwxOCBAQAorIyBzcmMvdGVzdC9tb2R1 bGVzL3Rlc3RfcmVxdWlyZWRfbHNuL01ha2VmaWxlCisKK0VYVFJBX0lOU1RBTEw9c3JjL3Rl c3QvbW9kdWxlcy9pbmplY3Rpb25fcG9pbnRzIFwKKwljb250cmliL3Rlc3RfZGVjb2RpbmcK KworZXhwb3J0IGVuYWJsZV9pbmplY3Rpb25fcG9pbnRzCitUQVBfVEVTVFMgPSAxCisKK2lm ZGVmIFVTRV9QR1hTCitQR19DT05GSUcgPSBwZ19jb25maWcKK1BHWFMgOj0gJChzaGVsbCAk KFBHX0NPTkZJRykgLS1wZ3hzKQoraW5jbHVkZSAkKFBHWFMpCitlbHNlCitzdWJkaXIgPSBz cmMvdGVzdC9tb2R1bGVzL3Rlc3RfcmVxdWlyZWRfbHNuCit0b3BfYnVpbGRkaXIgPSAuLi8u Li8uLi8uLgoraW5jbHVkZSAkKHRvcF9idWlsZGRpcikvc3JjL01ha2VmaWxlLmdsb2JhbAor aW5jbHVkZSAkKHRvcF9zcmNkaXIpL2NvbnRyaWIvY29udHJpYi1nbG9iYWwubWsKK2VuZGlm CmRpZmYgLS1naXQgYS9zcmMvdGVzdC9tb2R1bGVzL3Rlc3RfcmVxdWlyZWRfbHNuL21lc29u LmJ1aWxkIGIvc3JjL3Rlc3QvbW9kdWxlcy90ZXN0X3JlcXVpcmVkX2xzbi9tZXNvbi5idWls ZApuZXcgZmlsZSBtb2RlIDEwMDY0NAppbmRleCAwMDAwMDAwMDAwMC4uOTllZjNhNjBhNGUK LS0tIC9kZXYvbnVsbAorKysgYi9zcmMvdGVzdC9tb2R1bGVzL3Rlc3RfcmVxdWlyZWRfbHNu L21lc29uLmJ1aWxkCkBAIC0wLDAgKzEsMTUgQEAKKyMgQ29weXJpZ2h0IChjKSAyMDIyLTIw MjQsIFBvc3RncmVTUUwgR2xvYmFsIERldmVsb3BtZW50IEdyb3VwCisKK3Rlc3RzICs9IHsK KyAgJ25hbWUnOiAndGVzdF9yZXF1aXJlZF9sc24nLAorICAnc2QnOiBtZXNvbi5jdXJyZW50 X3NvdXJjZV9kaXIoKSwKKyAgJ2JkJzogbWVzb24uY3VycmVudF9idWlsZF9kaXIoKSwKKyAg J3RhcCc6IHsKKyAgICAnZW52JzogeworICAgICAgICdlbmFibGVfaW5qZWN0aW9uX3BvaW50 cyc6IGdldF9vcHRpb24oJ2luamVjdGlvbl9wb2ludHMnKSA/ICd5ZXMnIDogJ25vJywKKyAg ICB9LAorICAgICd0ZXN0cyc6IFsKKyAgICAgICd0LzAwMV9sb2dpY2FsX3JlcGxpY2F0aW9u LnBsJworICAgIF0sCisgIH0sCit9CmRpZmYgLS1naXQgYS9zcmMvdGVzdC9tb2R1bGVzL3Rl c3RfcmVxdWlyZWRfbHNuL3QvMDAxX2xvZ2ljYWxfc2xvdC5wbCBiL3NyYy90ZXN0L21vZHVs ZXMvdGVzdF9yZXF1aXJlZF9sc24vdC8wMDFfbG9naWNhbF9zbG90LnBsCm5ldyBmaWxlIG1v ZGUgMTAwNjQ0CmluZGV4IDAwMDAwMDAwMDAwLi40MTI2MWY0YWE2YgotLS0gL2Rldi9udWxs CisrKyBiL3NyYy90ZXN0L21vZHVsZXMvdGVzdF9yZXF1aXJlZF9sc24vdC8wMDFfbG9naWNh bF9zbG90LnBsCkBAIC0wLDAgKzEsMTI2IEBACisjIENvcHlyaWdodCAoYykgMjAyNCwgUG9z dGdyZVNRTCBHbG9iYWwgRGV2ZWxvcG1lbnQgR3JvdXAKKworIyBUaGlzIHRlc3QgdmVyaWZp ZXMgZWRnZSBjYXNlIG9mIHJlYWRpbmcgYSBtdWx0aXhhY3Q6CisjIHdoZW4gd2UgaGF2ZSBt dWx0aXhhY3QgdGhhdCBpcyBmb2xsb3dlZCBieSBleGFjdGx5IG9uZSBhbm90aGVyIG11bHRp eGFjdCwKKyMgYW5kIGFub3RoZXIgbXVsdGl4YWN0IGhhdmUgbm8gb2Zmc2V0IHlldCwgd2Ug bXVzdCB3YWl0IHVudGlsIHRoaXMgb2Zmc2V0CisjIGJlY29tZXMgb2JzZXJ2YWJsZS4gUHJl dmlvdXNseSB3ZSB1c2VkIHRvIHdhaXQgZm9yIDFtcyBpbiBhIGxvb3AgaW4gdGhpcworIyBj YXNlLCBidXQgbm93IHdlIHVzZSBDViBmb3IgdGhpcy4gVGhpcyB0ZXN0IGlzIGV4ZXJjaXNp bmcgc3VjaCBhIHNsZWVwLgorCit1c2Ugc3RyaWN0OwordXNlIHdhcm5pbmdzIEZBVEFMID0+ ICdhbGwnOworCit1c2UgUG9zdGdyZVNRTDo6VGVzdDo6Q2x1c3RlcjsKK3VzZSBQb3N0Z3Jl U1FMOjpUZXN0OjpVdGlsczsKKwordXNlIFRlc3Q6Ok1vcmU7CisKK2lmICgkRU5We2VuYWJs ZV9pbmplY3Rpb25fcG9pbnRzfSBuZSAneWVzJykKK3sKKwlwbGFuIHNraXBfYWxsID0+ICdJ bmplY3Rpb24gcG9pbnRzIG5vdCBzdXBwb3J0ZWQgYnkgdGhpcyBidWlsZCc7Cit9CisKK215 ICgkbm9kZSwgJHJlc3VsdCk7CisKKyRub2RlID0gUG9zdGdyZVNRTDo6VGVzdDo6Q2x1c3Rl ci0+bmV3KCdtaWtlJyk7Ciskbm9kZS0+aW5pdDsKKyRub2RlLT5hcHBlbmRfY29uZigncG9z dGdyZXNxbC5jb25mJywKKwkic2hhcmVkX3ByZWxvYWRfbGlicmFyaWVzID0gJ2luamVjdGlv bl9wb2ludHMnIik7Ciskbm9kZS0+YXBwZW5kX2NvbmYoJ3Bvc3RncmVzcWwuY29uZicsCisJ IndhbF9sZXZlbCA9ICdsb2dpY2FsJyIpOworJG5vZGUtPnN0YXJ0OworJG5vZGUtPnNhZmVf cHNxbCgncG9zdGdyZXMnLCBxKENSRUFURSBFWFRFTlNJT04gaW5qZWN0aW9uX3BvaW50cykp OworCisjIGNyZWF0ZSBhIHNpbXBsZSB0YWJsZSB0byBnZW5lcmF0ZSBkYXRhIGludG8KKyRu b2RlLT5zYWZlX3BzcWwoJ3Bvc3RncmVzJywKKwlxe2NyZWF0ZSB0YWJsZSB0IChpZCBzZXJp YWwgcHJpbWFyeSBrZXksIGIgdGV4dCl9KTsKKworIyBjcmVhdGUgdGhlIHR3byBzbG90cyB3 ZSdsbCBuZWVkCiskbm9kZS0+c2FmZV9wc3FsKCdwb3N0Z3JlcycsCisJcXtzZWxlY3QgcGdf Y3JlYXRlX2xvZ2ljYWxfcmVwbGljYXRpb25fc2xvdCgnc2xvdF9sb2dpY2FsJywgJ3Rlc3Rf ZGVjb2RpbmcnKX0pOworJG5vZGUtPnNhZmVfcHNxbCgncG9zdGdyZXMnLAorCXF7c2VsZWN0 IHBnX2NyZWF0ZV9waHlzaWNhbF9yZXBsaWNhdGlvbl9zbG90KCdzbG90X3BoeXNpY2FsJywg dHJ1ZSl9KTsKKworIyBhZHZhbmNlIGJvdGggdG8gY3VycmVudCBwb3NpdGlvbiwganVzdCB0 byBoYXZlIGV2ZXJ5dGhpbmcgInZhbGlkIgorJG5vZGUtPnNhZmVfcHNxbCgncG9zdGdyZXMn LAorCXF7c2VsZWN0IGNvdW50KCopIGZyb20gcGdfbG9naWNhbF9zbG90X2dldF9jaGFuZ2Vz KCdzbG90X2xvZ2ljYWwnLCBudWxsLCBudWxsKX0pOworJG5vZGUtPnNhZmVfcHNxbCgncG9z dGdyZXMnLAorCXF7c2VsZWN0IHBnX3JlcGxpY2F0aW9uX3Nsb3RfYWR2YW5jZSgnc2xvdF9w aHlzaWNhbCcsIHBnX2N1cnJlbnRfd2FsX2xzbigpKX0pOworCisjIHJ1biBjaGVja3BvaW50 LCB0byBmbHVzaCBjdXJyZW50IHN0YXRlIHRvIGRpc2sgYW5kIHNldCBhIGJhc2VsaW5lCisk bm9kZS0+c2FmZV9wc3FsKCdwb3N0Z3JlcycsIHF7Y2hlY2twb2ludH0pOworCisjIGdlbmVy YXRlIHRyYW5zYWN0aW9ucyB0byBnZXQgUlVOTklOR19YQUNUUworbXkgJHhhY3RzID0gJG5v ZGUtPmJhY2tncm91bmRfcHNxbCgncG9zdGdyZXMnKTsKKyR4YWN0cy0+cXVlcnlfdW50aWwo cXIvcnVuX3hhY3RzLywKK3EoXGVjaG8gcnVuX3hhY3RzCitTRUxFQ1QgMSBcd2F0Y2ggMC4x CitccQorKSk7CisKKyMgaW5zZXJ0IDJNIHJvd3MsIHRoYXQncyBhYm91dCAyNjBNQiAofjIw IHNlZ21lbnRzKSB3b3J0aCBvZiBXQUwKKyRub2RlLT5zYWZlX3BzcWwoJ3Bvc3RncmVzJywK Kwlxe2luc2VydCBpbnRvIHQgKGIpIHNlbGVjdCBtZDUoaTo6dGV4dCkgZnJvbSBnZW5lcmF0 ZV9zZXJpZXMoMSwxMDAwMDAwKSBzKGkpfSk7CisKKyMgcnVuIGFub3RoZXIgY2hlY2twb2lu dCwgdG8gc2V0IGEgbmV3IHJlc3RvcmUgTFNOCiskbm9kZS0+c2FmZV9wc3FsKCdwb3N0Z3Jl cycsIHF7Y2hlY2twb2ludH0pOworCisjIGFub3RoZXIgMk0gcm93cywgdGhhdCdzIGFib3V0 IDI2ME1CICh+MjAgc2VnbWVudHMpIHdvcnRoIG9mIFdBTAorJG5vZGUtPnNhZmVfcHNxbCgn cG9zdGdyZXMnLAorCXF7aW5zZXJ0IGludG8gdCAoYikgc2VsZWN0IG1kNShpOjp0ZXh0KSBm cm9tIGdlbmVyYXRlX3NlcmllcygxLDEwMDAwMDApIHMoaSl9KTsKKworIyBydW4gYW5vdGhl ciBjaGVja3BvaW50LCB0aGlzIHRpbWUgaW4gdGhlIGJhY2tncm91bmQsIGFuZCBtYWtlIGl0 IHdhaXQKKyMgb24gdGhlIGluamVjdGlvbiBwb2ludCksIHNvIHRoYXQgdGhlIGNoZWNrcG9p bnQgc3RvcHMgcmlnaHQgYmVmb3JlCisjIHJlbW92aW5nIG9sZCBXQUwgc2VnbWVudHMKK3By aW50KCdzdGFydGluZyBjaGVja3BvaW50XG4nKTsKKworbXkgJGNoZWNrcG9pbnQgPSAkbm9k ZS0+YmFja2dyb3VuZF9wc3FsKCdwb3N0Z3JlcycpOworJGNoZWNrcG9pbnQtPnF1ZXJ5X3Nh ZmUocShzZWxlY3QgaW5qZWN0aW9uX3BvaW50c19hdHRhY2goJ2NoZWNrcG9pbnQtYmVmb3Jl LW9sZC13YWwtcmVtb3ZhbCcsJ3dhaXQnKSkpOworJGNoZWNrcG9pbnQtPnF1ZXJ5X3VudGls KHFyL3N0YXJ0aW5nX2NoZWNrcG9pbnQvLAorcShcZWNobyBzdGFydGluZ19jaGVja3BvaW50 CitjaGVja3BvaW50OworXHEKKykpOworCitwcmludCgnd2FpdGluZyBmb3IgaW5qZWN0aW9u X3BvaW50XG4nKTsKKyMgd2FpdCB1bnRpbCB0aGUgY2hlY2twb2ludCBzdG9wcyByaWdodCBi ZWZvcmUgcmVtb3ZpbmcgV0FMIHNlZ21lbnRzCiskbm9kZS0+d2FpdF9mb3JfZXZlbnQoJ2No ZWNrcG9pbnRlcicsICdjaGVja3BvaW50LWJlZm9yZS1vbGQtd2FsLXJlbW92YWwnKTsKKwor CisjIHRyeSB0byBhZHZhbmNlIHRoZSBsb2dpY2FsIHNsb3QsIGJ1dCBtYWtlIGl0IHN0b3Ag d2hlbiBpdCBtb3ZlcyB0byB0aGUKKyMgbmV4dCBXQUwgc2VnbWVudCAoaGFzIHRvIGhhcHBl biBpbiB0aGUgYmFja2dyb3VuZCB0b28pCitteSAkbG9naWNhbCA9ICRub2RlLT5iYWNrZ3Jv dW5kX3BzcWwoJ3Bvc3RncmVzJyk7CiskbG9naWNhbC0+cXVlcnlfc2FmZShxe3NlbGVjdCBp bmplY3Rpb25fcG9pbnRzX2F0dGFjaCgnbG9naWNhbC1yZXBsaWNhdGlvbi1zbG90LWFkdmFu Y2Utc2VnbWVudCcsJ3dhaXQnKTt9KTsKKyRsb2dpY2FsLT5xdWVyeV91bnRpbChxci9nZXRf Y2hhbmdlcy8sCitxKAorXGVjaG8gZ2V0X2NoYW5nZXMKK3NlbGVjdCBjb3VudCgqKSBmcm9t IHBnX2xvZ2ljYWxfc2xvdF9nZXRfY2hhbmdlcygnc2xvdF9sb2dpY2FsJywgbnVsbCwgbnVs bCkgXHdhdGNoIDEKK1xxCispKTsKKworCisjIHdhaXQgdW50aWwgdGhlIGNoZWNrcG9pbnQg c3RvcHMgcmlnaHQgYmVmb3JlIHJlbW92aW5nIFdBTCBzZWdtZW50cworJG5vZGUtPndhaXRf Zm9yX2V2ZW50KCdjbGllbnQgYmFja2VuZCcsICdsb2dpY2FsLXJlcGxpY2F0aW9uLXNsb3Qt YWR2YW5jZS1zZWdtZW50Jyk7CisKKworIyBPSywgd2UncmUgaW4gdGhlIHJpZ2h0IHNpdHVh dGlvbiwgIHRpbWUgdG8gYWR2YW5jZSB0aGUgcGh5c2ljYWwgc2xvdCwKKyMgd2hpY2ggcmVj YWxjdWxhdGVzIHRoZSByZXF1aXJlZCBMU04sIGFuZCB0aGVuIHVuYmxvY2sgdGhlIGNoZWNr cG9pbnQsCisjIHdoaWNoIHJlbW92ZXMgdGhlIFdBTCBzdGlsbCBuZWVkZWQgYnkgdGhlIGxv Z2ljYWwgc2xvdAorJG5vZGUtPnNhZmVfcHNxbCgncG9zdGdyZXMnLAorCXF7c2VsZWN0IHBn X3JlcGxpY2F0aW9uX3Nsb3RfYWR2YW5jZSgnc2xvdF9waHlzaWNhbCcsIHBnX2N1cnJlbnRf d2FsX2xzbigpKX0pOworCiskbm9kZS0+c2FmZV9wc3FsKCdwb3N0Z3JlcycsCisJcXtzZWxl Y3QgaW5qZWN0aW9uX3BvaW50c193YWtldXAoJ2NoZWNrcG9pbnQtYmVmb3JlLW9sZC13YWwt cmVtb3ZhbCcpfSk7CisKKyMgYWJydXB0bHkgc3RvcCB0aGUgc2VydmVyICgxIHNlY29uZCBz aG91bGQgYmUgZW5vdWdoIGZvciB0aGUgY2hlY2twb2ludAorIyB0byBmaW5pc2gsIHdvdWxk IGJlIGJldHRlciApCiskbm9kZS0+c3RvcCgnaW1tZWRpYXRlJyk7CisKKyRub2RlLT5zdGFy dDsKKworJG5vZGUtPnNhZmVfcHNxbCgncG9zdGdyZXMnLCBxe3NlbGVjdCBjb3VudCgqKSBm cm9tIHBnX2xvZ2ljYWxfc2xvdF9nZXRfY2hhbmdlcygnc2xvdF9sb2dpY2FsJywgbnVsbCwg bnVsbCk7fSk7CisKKyRub2RlLT5zdG9wOworCisjIElmIHdlIHJlYWNoZWQgdGhpcyBwb2lu dCAtIGV2ZXJ5dGhpbmcgaXMgT0suCitvaygxKTsKK2RvbmVfdGVzdGluZygpOwotLSAKMi40 Ny4wCgo= --------------X5LK3I0KWYHP7scWJn47T3Jp Content-Type: text/x-patch; charset=UTF-8; name="0001-injection-points.patch" Content-Disposition: attachment; filename="0001-injection-points.patch" Content-Transfer-Encoding: base64 RnJvbSBlZWY1ZjAyYTVjMjJjY2M1MjBjMjA2MjNkNzBlYWYwOTNhMDM5ZjA5IE1vbiBTZXAg MTcgMDA6MDA6MDAgMjAwMQpGcm9tOiBUb21hcyBWb25kcmEgPHRvbWFzQHZvbmRyYS5tZT4K RGF0ZTogVGh1LCAyMSBOb3YgMjAyNCAyMDozNzowMCArMDEwMApTdWJqZWN0OiBbUEFUQ0gg MS8yXSBpbmplY3Rpb24gcG9pbnRzCgotLS0KIHNyYy9iYWNrZW5kL2FjY2Vzcy90cmFuc2Ft L3hsb2cuYyAgICAgICAgIHwgIDQgKysrKwogc3JjL2JhY2tlbmQvcmVwbGljYXRpb24vbG9n aWNhbC9sb2dpY2FsLmMgfCAxOCArKysrKysrKysrKysrKysrKysKIDIgZmlsZXMgY2hhbmdl ZCwgMjIgaW5zZXJ0aW9ucygrKQoKZGlmZiAtLWdpdCBhL3NyYy9iYWNrZW5kL2FjY2Vzcy90 cmFuc2FtL3hsb2cuYyBiL3NyYy9iYWNrZW5kL2FjY2Vzcy90cmFuc2FtL3hsb2cuYwppbmRl eCA2ZjU4NDEyYmNhYi4uOGY5NjI5ODY2YzMgMTAwNjQ0Ci0tLSBhL3NyYy9iYWNrZW5kL2Fj Y2Vzcy90cmFuc2FtL3hsb2cuYworKysgYi9zcmMvYmFja2VuZC9hY2Nlc3MvdHJhbnNhbS94 bG9nLmMKQEAgLTczMTAsNiArNzMxMCwxMCBAQCBDcmVhdGVDaGVja1BvaW50KGludCBmbGFn cykKIAlpZiAoUHJpb3JSZWRvUHRyICE9IEludmFsaWRYTG9nUmVjUHRyKQogCQlVcGRhdGVD aGVja1BvaW50RGlzdGFuY2VFc3RpbWF0ZShSZWRvUmVjUHRyIC0gUHJpb3JSZWRvUHRyKTsK IAorI2lmZGVmIFVTRV9JTkpFQ1RJT05fUE9JTlRTCisJSU5KRUNUSU9OX1BPSU5UKCJjaGVj a3BvaW50LWJlZm9yZS1vbGQtd2FsLXJlbW92YWwiKTsKKyNlbmRpZgorCiAJLyoKIAkgKiBE ZWxldGUgb2xkIGxvZyBmaWxlcywgdGhvc2Ugbm8gbG9uZ2VyIG5lZWRlZCBmb3IgbGFzdCBj aGVja3BvaW50IHRvCiAJICogcHJldmVudCB0aGUgZGlzayBob2xkaW5nIHRoZSB4bG9nIGZy b20gZ3Jvd2luZyBmdWxsLgpkaWZmIC0tZ2l0IGEvc3JjL2JhY2tlbmQvcmVwbGljYXRpb24v bG9naWNhbC9sb2dpY2FsLmMgYi9zcmMvYmFja2VuZC9yZXBsaWNhdGlvbi9sb2dpY2FsL2xv Z2ljYWwuYwppbmRleCBlOTQxYmI0OTFkOC4uNTY5YzE5MjVlY2MgMTAwNjQ0Ci0tLSBhL3Ny Yy9iYWNrZW5kL3JlcGxpY2F0aW9uL2xvZ2ljYWwvbG9naWNhbC5jCisrKyBiL3NyYy9iYWNr ZW5kL3JlcGxpY2F0aW9uL2xvZ2ljYWwvbG9naWNhbC5jCkBAIC0zMCw2ICszMCw3IEBACiAK ICNpbmNsdWRlICJhY2Nlc3MveGFjdC5oIgogI2luY2x1ZGUgImFjY2Vzcy94bG9ndXRpbHMu aCIKKyNpbmNsdWRlICJhY2Nlc3MveGxvZ19pbnRlcm5hbC5oIgogI2luY2x1ZGUgImZtZ3Iu aCIKICNpbmNsdWRlICJtaXNjYWRtaW4uaCIKICNpbmNsdWRlICJwZ3N0YXQuaCIKQEAgLTQx LDYgKzQyLDcgQEAKICNpbmNsdWRlICJzdG9yYWdlL3Byb2MuaCIKICNpbmNsdWRlICJzdG9y YWdlL3Byb2NhcnJheS5oIgogI2luY2x1ZGUgInV0aWxzL2J1aWx0aW5zLmgiCisjaW5jbHVk ZSAidXRpbHMvaW5qZWN0aW9uX3BvaW50LmgiCiAjaW5jbHVkZSAidXRpbHMvaW52YWwuaCIK ICNpbmNsdWRlICJ1dGlscy9tZW11dGlscy5oIgogCkBAIC0xODQ0LDkgKzE4NDYsMTMgQEAg TG9naWNhbENvbmZpcm1SZWNlaXZlZExvY2F0aW9uKFhMb2dSZWNQdHIgbHNuKQogCXsKIAkJ Ym9vbAkJdXBkYXRlZF94bWluID0gZmFsc2U7CiAJCWJvb2wJCXVwZGF0ZWRfcmVzdGFydCA9 IGZhbHNlOworCQlYTG9nUmVjUHRyCXJlc3RhcnRfbHNuOwogCiAJCVNwaW5Mb2NrQWNxdWly ZSgmTXlSZXBsaWNhdGlvblNsb3QtPm11dGV4KTsKIAorCQkvKiByZW1lbWJlciB0aGUgb2xk IHJlc3RhcnQgbHNuICovCisJCXJlc3RhcnRfbHNuID0gTXlSZXBsaWNhdGlvblNsb3QtPmRh dGEucmVzdGFydF9sc247CisKIAkJTXlSZXBsaWNhdGlvblNsb3QtPmRhdGEuY29uZmlybWVk X2ZsdXNoID0gbHNuOwogCiAJCS8qIGlmIHdlJ3JlIHBhc3QgdGhlIGxvY2F0aW9uIHJlcXVp cmVkIGZvciBidW1waW5nIHhtaW4sIGRvIHNvICovCkBAIC0xODg4LDYgKzE4OTQsMTggQEAg TG9naWNhbENvbmZpcm1SZWNlaXZlZExvY2F0aW9uKFhMb2dSZWNQdHIgbHNuKQogCQkvKiBm aXJzdCB3cml0ZSBuZXcgeG1pbiB0byBkaXNrLCBzbyB3ZSBrbm93IHdoYXQncyB1cCBhZnRl ciBhIGNyYXNoICovCiAJCWlmICh1cGRhdGVkX3htaW4gfHwgdXBkYXRlZF9yZXN0YXJ0KQog CQl7CisjaWZkZWYgVVNFX0lOSkVDVElPTl9QT0lOVFMKKwkJCVhMb2dTZWdObwlzZWcxLAor CQkJCQkJc2VnMjsKKworCQkJWExCeXRlVG9TZWcocmVzdGFydF9sc24sIHNlZzEsIHdhbF9z ZWdtZW50X3NpemUpOworCQkJWExCeXRlVG9TZWcoTXlSZXBsaWNhdGlvblNsb3QtPmRhdGEu cmVzdGFydF9sc24sIHNlZzIsIHdhbF9zZWdtZW50X3NpemUpOworCisJCQkvKiB0cmlnZ2Vy IGluamVjdGlvbiBwb2ludCwgYnV0IG9ubHkgaWYgc2VnbWVudCBjaGFuZ2VzICovCisJCQlp ZiAoc2VnMSAhPSBzZWcyKQorCQkJCUlOSkVDVElPTl9QT0lOVCgibG9naWNhbC1yZXBsaWNh dGlvbi1zbG90LWFkdmFuY2Utc2VnbWVudCIpOworI2VuZGlmCisKIAkJCVJlcGxpY2F0aW9u U2xvdE1hcmtEaXJ0eSgpOwogCQkJUmVwbGljYXRpb25TbG90U2F2ZSgpOwogCQkJZWxvZyhE RUJVRzEsICJ1cGRhdGVkIHhtaW46ICV1IHJlc3RhcnQ6ICV1IiwgdXBkYXRlZF94bWluLCB1 cGRhdGVkX3Jlc3RhcnQpOwotLSAKMi40Ny4wCgo= --------------X5LK3I0KWYHP7scWJn47T3Jp--