From: "Tharakan, Robins" <tharar@amazon.com>
To: "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: RE: pg_upgrade failing for 200+ million Large Objects
Thread-Topic: pg_upgrade failing for 200+ million Large Objects
Thread-Index: AdcQGHPbpt3tdBnIQ6apvAiQCfVXxwDEYO8Q
Date: Sun, 7 Mar 2021 08:43:28 +0000
Message-ID: <2543eda3e609455e9cf283646a8b2bc8@EX13D05UWC001.ant.amazon.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: multipart/signed; protocol="application/x-pkcs7-signature";
	micalg=SHA1; boundary="----=_NextPart_000_00DA_01D7138A.2325BF90"
MIME-Version: 1.0
Archived-At: 
 <https://www.postgresql.org/message-id/2543eda3e609455e9cf283646a8b2bc8%40EX13D05UWC001.ant.amazon.com>
Precedence: bulk

------=_NextPart_000_00DA_01D7138A.2325BF90
Content-Type: multipart/mixed;
	boundary="----=_NextPart_001_00DB_01D7138A.2325BF90"


------=_NextPart_001_00DB_01D7138A.2325BF90
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit

Hi all,

Attached is a proof-of-concept patch that allows Postgres to perform
pg_upgrade if the instance has Millions of objects.

It would be great if someone could take a look and see if this patch is in
the right direction. There are some pending tasks (such as documentation /
pg_resetxlog vs pg_resetwal related changes) but for now, the patch helps
remove a stalemate where if a Postgres instance has a large number
(accurately speaking 146+ Million) of Large Objects, pg_upgrade fails. This
is easily reproducible and besides deleting Large Objects before upgrade,
there is no other (apparent) way for pg_upgrade to complete.

The patch (attached):
- Applies cleanly on REL9_6_STABLE -
c7a4fc3dd001646d5938687ad59ab84545d5d043
- 'make check' passes
- Allows the user to provide a constant via pg_upgrade command-line, that
overrides the 2 billion constant in pg_resetxlog [1] thereby increasing the
(window of) Transaction IDs available for pg_upgrade to complete. 


Sample argument for pg_upgrade:
$ /opt/postgres/96/bin/pg_upgrade --max-limit-xid 1000000000 --old-bindir
...


With this patch, pg_upgrade is now able to upgrade a v9.5 cluster with 500
million Large Objects successfully to v9.6 - some stats below:

Source Postgres - v9.5.24
Target Version - v9.6.21
Large Object Count: 500 Million Large Objects
Machine - r5.4xlarge (16vCPU / 128GB RAM + 1TB swap)
Memory used during pg_upgrade - ~350GB
Time taken - 25+ hrs. (tested twice) - (All LOs processed sequentially ->
Scope for optimization)

Although counter-intuitive, for this testing purpose all Large Objects were
small (essentially the idea was to test the count) and created by using
something like this:

seq 1 50000 | xargs -n 1 -i -P 10 /opt/postgres/95/bin/psql -c "select
lo_from_bytea(0, '\xffffff00') from generate_series(1,10000);" > /dev/null

I am not married to the patch (especially the argument name) but ideally I'd
prefer a way to get this upgrade going without a patch. For now, I am unable
to find any other way to upgrade a v9.5 Postgres database in this scenario,
facing End-of-Life.

Reference:
1) 2 Billion constant -
https://github.com/postgres/postgres/blob/ca3b37487be333a1d241dab1bbdd17a211
a88f43/src/bin/pg_resetwal/pg_resetwal.c#L444

Thanks,
Robins Tharakan

> -----Original Message-----
> From: Tharakan, Robins
> Sent: Wednesday, 3 March 2021 10:36 PM
> To: pgsql-hackers@postgresql.org
> Subject: pg_upgrade failing for 200+ million Large Objects
> 
> Hi,
> 
> While reviewing a failed upgrade from Postgres v9.5 (to v9.6) I saw that
> the
> instance had ~200 million (in-use) Large Objects. I was able to reproduce
> this on a test instance which too fails with a similar error.
> 
> 
> pg_restore: executing BLOB 4980622
> pg_restore: WARNING: database with OID 0 must be vacuumed within 1000001
> transactions
> HINT: To avoid a database shutdown, execute a database-wide VACUUM in
> that
> database.
> You might also need to commit or roll back old prepared transactions.
> pg_restore: executing BLOB 4980623
> pg_restore: [archiver (db)] Error while PROCESSING TOC:
> pg_restore: [archiver (db)] Error from TOC entry 2565; 2613 4980623 BLOB
> 4980623 postgres
> pg_restore: [archiver (db)] could not execute query: ERROR: database is
> not
> accepting commands to avoid wraparound data loss in database with OID 0
> HINT: Stop the postmaster and vacuum that database in single-user mode.
> You might also need to commit or roll back old prepared transactions.
> Command was: SELECT pg_catalog.lo_create('4980623');
> 
> 
> 
> To remove the obvious possibilities, these Large Objects that are still
> in-use (so vacuumlo wouldn't help), giving more system resources doesn't
> help, moving Large Objects around to another database doesn't help (since
> this is cluster-wide restriction), the source instance is nowhere close
> to
> wraparound and lastly recent-most minor versions don't help either (I
> tried
> compiling 9_6_STABLE + upgrade database with 150 million LO and still
> encountered the same issue).
> 
> Do let me know if I am missing something obvious but it appears that this
> is
> happening owing to 2 things coming together:
> 
> * Each Large Object is migrated in its own transaction during pg_upgrade
> * pg_resetxlog appears to be narrowing the window (available for
> pg_upgrade)
> to ~146 Million XIDs (2^31 - 1 million XID wraparound margin - 2 billion
> which is a hard-coded constant - see [1] - in what appears to be an
> attempt
> to force an Autovacuum Wraparound session soon after upgrade completes).
> 
> Ideally such an XID based restriction, is limiting for an instance that's
> actively using a lot of Large Objects. Besides forcing AutoVacuum
> Wraparound
> logic to kick in soon after, I am unclear what much else it aims to do.
> What
> it does seem to be doing is to block Major Version upgrades if the
> pre-upgrade instance has >146 Million Large Objects (half that, if the LO
> additionally requires ALTER LARGE OBJECT OWNER TO for each of those
> objects
> during pg_restore)
> 
> For long-term these ideas came to mind, although am unsure which are
> low-hanging fruits and which outright impossible - For e.g. clubbing
> multiple objects in a transaction [2] / Force AutoVacuum post upgrade
> (and
> thus remove this limitation altogether) or see if "pg_resetxlog -x" (from
> within pg_upgrade) could help in some way to work-around this limitation.
> 
> Is there a short-term recommendation for this scenario?
> 
> I can understand a high number of small-sized objects is not a great way
> to
> use pg_largeobject (since Large Objects was intended to be for, well,
> 'large
> objects') but this magic number of Large Objects is now a stalemate at
> this
> point (with respect to v9.5 EOL).
> 
> 
> Reference:
> 1) pg_resetxlog -
> https://github.com/postgres/postgres/blob/ca3b37487be333a1d241dab1bbdd17a
> 211
> a88f43/src/bin/pg_resetwal/pg_resetwal.c#L444
> 2)
> https://www.postgresql.org/message-id/ed7d86a1-b907-4f53-9f6e-
> 63482d2f2bac%4
> 0manitou-mail.org
> 
> -
> Thanks
> Robins Tharakan

------=_NextPart_001_00DB_01D7138A.2325BF90
Content-Type: application/octet-stream;
	name="pgupgrade_lo_v2.patch"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
	filename="pgupgrade_lo_v2.patch"

diff --git a/src/bin/pg_resetxlog/pg_resetxlog.c =
b/src/bin/pg_resetxlog/pg_resetxlog.c
index 3e79482ca2..f2e9824cb5 100644
--- a/src/bin/pg_resetxlog/pg_resetxlog.c
+++ b/src/bin/pg_resetxlog/pg_resetxlog.c
@@ -67,6 +67,7 @@ static TransactionId set_xid =3D 0;
 static TransactionId set_oldest_commit_ts_xid =3D 0;
 static TransactionId set_newest_commit_ts_xid =3D 0;
 static Oid	set_oid =3D 0;
+static Oid	max_limit_xid =3D 2000000000;
 static MultiXactId set_mxid =3D 0;
 static MultiXactOffset set_mxoff =3D (MultiXactOffset) -1;
 static uint32 minXlogTli =3D 0;
@@ -116,7 +117,7 @@ main(int argc, char *argv[])
 	}
=20
=20
-	while ((c =3D getopt(argc, argv, "c:D:e:fl:m:no:O:x:")) !=3D -1)
+	while ((c =3D getopt(argc, argv, "c:D:e:fl:L:m:no:O:x:")) !=3D -1)
 	{
 		switch (c)
 		{
@@ -210,6 +211,21 @@ main(int argc, char *argv[])
 				}
 				break;
=20
+			case 'L':
+				max_limit_xid =3D strtoul(optarg, &endptr, 0);
+				if (endptr =3D=3D optarg || *endptr !=3D '\0')
+				{
+					fprintf(stderr, _("%s: invalid argument for option %s\n"), =
progname, "-L");
+					fprintf(stderr, _("Try \"%s --help\" for more information.\n"), =
progname);
+					exit(1);
+				}
+				if (max_limit_xid <=3D 500000000)
+				{
+					fprintf(stderr, _("%s: Max Limit XID (-L) must not be less than =
500 Million\n"), progname);
+					exit(1);
+				}
+				break;
+
 			case 'm':
 				set_mxid =3D strtoul(optarg, &endptr, 0);
 				if (endptr =3D=3D optarg || *endptr !=3D ',')
@@ -381,7 +397,7 @@ main(int argc, char *argv[])
 		 * reasonably safe.  The magic constant here corresponds to the
 		 * maximum allowed value of autovacuum_freeze_max_age.
 		 */
-		ControlFile.checkPointCopy.oldestXid =3D set_xid - 2000000000;
+		ControlFile.checkPointCopy.oldestXid =3D set_xid - max_limit_xid;
 		if (ControlFile.checkPointCopy.oldestXid < FirstNormalTransactionId)
 			ControlFile.checkPointCopy.oldestXid +=3D FirstNormalTransactionId;
 		ControlFile.checkPointCopy.oldestXidDB =3D InvalidOid;
@@ -1239,6 +1255,7 @@ usage(void)
 	printf(_("  -e XIDEPOCH      set next transaction ID epoch\n"));
 	printf(_("  -f               force update to be done\n"));
 	printf(_("  -l XLOGFILE      force minimum WAL starting location for =
new transaction log\n"));
+	printf(_("  -L MAXLIMITXID   force max XID starting location for new =
transaction log\n"));
 	printf(_("  -m MXID,MXID     set next and oldest multitransaction =
ID\n"));
 	printf(_("  -n               no update, just show what would be done =
(for testing)\n"));
 	printf(_("  -o OID           set next OID\n"));
diff --git a/src/bin/pg_upgrade/option.c b/src/bin/pg_upgrade/option.c
index 7ab284a51b..3a861739f7 100644
--- a/src/bin/pg_upgrade/option.c
+++ b/src/bin/pg_upgrade/option.c
@@ -53,6 +53,7 @@ parseCommandLine(int argc, char *argv[])
 		{"link", no_argument, NULL, 'k'},
 		{"retain", no_argument, NULL, 'r'},
 		{"jobs", required_argument, NULL, 'j'},
+		{"max-limit-xid", required_argument, NULL, 'L'},
 		{"verbose", no_argument, NULL, 'v'},
 		{NULL, 0, NULL, 0}
 	};
@@ -64,6 +65,7 @@ parseCommandLine(int argc, char *argv[])
 	time_t		run_time =3D time(NULL);
=20
 	user_opts.transfer_mode =3D TRANSFER_MODE_COPY;
+	user_opts.maxlimitxid =3D 2000000000;
=20
 	os_info.progname =3D get_progname(argv[0]);
=20
@@ -101,7 +103,7 @@ parseCommandLine(int argc, char *argv[])
 	if ((log_opts.internal =3D fopen_priv(INTERNAL_LOG_FILE, "a")) =3D=3D =
NULL)
 		pg_fatal("cannot write to log file %s\n", INTERNAL_LOG_FILE);
=20
-	while ((option =3D getopt_long(argc, argv, "d:D:b:B:cj:ko:O:p:P:rU:v",
+	while ((option =3D getopt_long(argc, argv, =
"d:D:b:B:cj:ko:L:O:p:P:rU:v",
 								 long_options, &optindex)) !=3D -1)
 	{
 		switch (option)
@@ -132,6 +134,10 @@ parseCommandLine(int argc, char *argv[])
 				user_opts.jobs =3D atoi(optarg);
 				break;
=20
+			case 'L':
+				user_opts.maxlimitxid =3D atoi(optarg);
+				break;
+
 			case 'k':
 				user_opts.transfer_mode =3D TRANSFER_MODE_LINK;
 				break;
@@ -287,6 +293,7 @@ Options:\n\
   -D, --new-datadir=3DDATADIR     new cluster data directory\n\
   -j, --jobs=3DNUM                number of simultaneous processes or =
threads to use\n\
   -k, --link                    link instead of copying files to new =
cluster\n\
+  -L, --max-limit-xid=3DNUM       max-limit XIDs to consider\n\
   -o, --old-options=3DOPTIONS     old cluster options to pass to the =
server\n\
   -O, --new-options=3DOPTIONS     new cluster options to pass to the =
server\n\
   -p, --old-port=3DPORT           old cluster port number (default =
%d)\n\
diff --git a/src/bin/pg_upgrade/pg_upgrade.c =
b/src/bin/pg_upgrade/pg_upgrade.c
index 02078c0357..2d0f3a7e04 100644
--- a/src/bin/pg_upgrade/pg_upgrade.c
+++ b/src/bin/pg_upgrade/pg_upgrade.c
@@ -411,8 +411,10 @@ copy_clog_xlog_xid(void)
 	/* set the next transaction id and epoch of the new cluster */
 	prep_status("Setting next transaction ID and epoch for new cluster");
 	exec_prog(UTILITY_LOG_FILE, NULL, true,
-			  "\"%s/pg_resetxlog\" -f -x %u \"%s\"",
-			  new_cluster.bindir, old_cluster.controldata.chkpnt_nxtxid,
+			  "\"%s/pg_resetxlog\" -L %u -f -x %u \"%s\"",
+			  new_cluster.bindir,
+			  user_opts.maxlimitxid,
+			  old_cluster.controldata.chkpnt_nxtxid,
 			  new_cluster.pgdata);
 	exec_prog(UTILITY_LOG_FILE, NULL, true,
 			  "\"%s/pg_resetxlog\" -f -e %u \"%s\"",
diff --git a/src/bin/pg_upgrade/pg_upgrade.h =
b/src/bin/pg_upgrade/pg_upgrade.h
index 9fbdacc53e..50fe73ae09 100644
--- a/src/bin/pg_upgrade/pg_upgrade.h
+++ b/src/bin/pg_upgrade/pg_upgrade.h
@@ -298,6 +298,7 @@ typedef struct
 								 * changes */
 	transferMode transfer_mode; /* copy files or link them? */
 	int			jobs;
+	int			maxlimitxid;
 } UserOpts;
=20
=20

------=_NextPart_001_00DB_01D7138A.2325BF90--

------=_NextPart_000_00DA_01D7138A.2325BF90
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIQATCCBA8w
ggL3oAMCAQICEFeHFPCN4dmrSiV65deu/iEwDQYJKoZIhvcNAQEFBQAwOTE3MDUGA1UEAxMuQW1h
em9uLmNvbSBJbnRlcm5hbCBSb290IENlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0wNzA4MzAxODAy
MjVaFw0yNzA4MzAxODEwNTlaMDkxNzA1BgNVBAMTLkFtYXpvbi5jb20gSW50ZXJuYWwgUm9vdCBD
ZXJ0aWZpY2F0ZSBBdXRob3JpdHkwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDugYdG
nDajFgUbfr6ek3APnSiy6nHUWUEbvxKPFc4eiWaWcGBge4JTiamxVmZu3yFh86klFGYBxixwJ32z
oD56NfWlubhv2feMQFxxKKOiK3dZSIDLbGqCMvILDW/hYHLGL6/rFDFPYbSbs7mJLxFBmWdyCFvf
pDFEMDcPVOFNw4EEQL7TgmPn5loWvtgkSAue5EIgpEcKwy86ylpvzq/Oj/GEWqD8sXAUnhWLgSm6
r1jsAKlk1tmaKsaWBjMC4fiSg8Zq1JI/CQ+FckZ5nHkiCBztzWEYp1m7uBQBBcZ/+l2qdz93u/7f
DxmyICIE4enGr5xTWSv9MDNwQQd9YLh9AgMBAAGjggERMIIBDTALBgNVHQ8EBAMCAYYwDwYDVR0T
AQH/BAUwAwEB/zAdBgNVHQ4EFgQUf42xTkyimArciye/YgVpPCUSs8IwEAYJKwYBBAGCNxUBBAMC
AQAwgbsGA1UdIASBszCBsDCBrQYMKwYBBAGla4FIAQEBMIGcMHIGCCsGAQUFBwICMGYeZABBAG0A
YQB6AG8AbgAuAGMAbwBtACAASQBuAHQAZQByAG4AYQBsACAAQwBlAHIAdABpAGYAaQBjAGEAdABl
ACAAUAByAGEAYwB0AGkAYwBlACAAUwB0AGEAdABlAG0AZQBuAHQwJgYIKwYBBQUHAgEWGmh0dHA6
Ly9wa2kuYW1hem9uLmNvbS9jcHMvMA0GCSqGSIb3DQEBBQUAA4IBAQB0AW6ePZaQ83mcE9Etduc1
aSp4nvLUoJ2NAI1v4UDB3A0iBggNo9XfEsfin/tJoXkWuHxtB5ucZNAW3ZletXQfW3DAbWVr5kAZ
Tv4h/u/9OqAVZCOuxYMUZqfwJiPybm4xj9dnll6F9mF7Ur5I7D+PX+MmuJNsEza3MqcJaxcefrI5
0XTn9uCMgxo6/x56LqWD46AxrYBe4YjF8FQ9VBRz4S1cS0KI7mA40i5dyOc2nmnET6e+iIQPGHzQ
iTuereCRhGybLkKi3yChe4Uw6KqQ4qSVVAYf1nJjrDYk3RUHHF95JcWCHyTh4safTHcTETNWx8F9
MWWlF96gZ4B8/ORlMIIFjjCCBHagAwIBAgIKYRwhegAAAAAAJjANBgkqhkiG9w0BAQsFADA5MTcw
NQYDVQQDEy5BbWF6b24uY29tIEludGVybmFsIFJvb3QgQ2VydGlmaWNhdGUgQXV0aG9yaXR5MB4X
DTE4MTEwNjIzMzExOFoXDTIzMTEwNjIzNDExOFowZDETMBEGCgmSJomT8ixkARkWA2NvbTEWMBQG
CgmSJomT8ixkARkWBmFtYXpvbjETMBEGCgmSJomT8ixkARkWA2FudDEgMB4GA1UEAxMXQW1hem9u
LmNvbSBDSUEgQ0EgRzQgMDEwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDBNKNhXeYZ
RggQpGc194zGplBxDWOWYsMY7jc4EkhplrCkkPWqcfCuJh0SLJpcbZwin70hnu3DXwmt6N5nZu7c
Yd1h40YL/p6zULMebqn0p9AwUMK+8qV514J79VjioN/GE7h5G8ZkyjUjmqeBtTczJUifwAJ72n6b
wSFLLZS28Nar4blskiGdX6ZaB+N3nNp4DN+RzCqfFc15Ta9iVIj50ChknahX8Ia4Hjw3azNKWJl1
JGTsZIETr9N/m6ta1LMjuAMpJCqwCx8Ndbkml+cjyul6T5MMHNMBYCeABm7P0HWcSoinZbK7qNJO
TZDfP6NWFm/6VgSr/U/WqMJKZrN9Gb3KEr3kVBDMG07Al7YM6nwzLG60P3oPodOUW5MSU+zpN3Fa
vqWsopJ5uBBk26YFpn1+8CJnuwwT4VjQtRF6SQk04fNpowHxOtgrVlIr7Wuf4bD3eMU6A8mbSZyp
9SBHxR1n8GLO8o3/IQhC75AawgBTm04X+K9eQdoaLdk5rNxxMHJ+y15fwZpab5WfpBNN71Qi/fWO
/hoimZ1ytSMHSMmAnf4Q/jQC7GLXTIRikjR4f+KLSPrWu17h4Cqv2ehbHlOS8tJG866cViWlm9pc
JNfz1QOpgoPn1fT6UpoVh40YsigC/qJktW3HmyCWOzXU0u7H61BpgxJBZS7AMvBSqQIDAQABo4IB
azCCAWcwEAYJKwYBBAGCNxUBBAMCAQAwHQYDVR0OBBYEFOAp5m6qUdZa9admS/aG9DXnqw+rMBkG
CSsGAQQBgjcUAgQMHgoAUwB1AGIAQwBBMAsGA1UdDwQEAwIBhjASBgNVHRMBAf8ECDAGAQH/AgEA
MB8GA1UdIwQYMBaAFH+NsU5MopgK3Isnv2IFaTwlErPCMGUGA1UdHwReMFwwWqBYoFaGVGh0dHA6
Ly9wa2kuYW1hem9uLmNvbS9jcmwvQW1hem9uLmNvbSUyMEludGVybmFsJTIwUm9vdCUyMENlcnRp
ZmljYXRlJTIwQXV0aG9yaXR5LmNybDBwBggrBgEFBQcBAQRkMGIwYAYIKwYBBQUHMAKGVGh0dHA6
Ly9wa2kuYW1hem9uLmNvbS9jcnQvQW1hem9uLmNvbSUyMEludGVybmFsJTIwUm9vdCUyMENlcnRp
ZmljYXRlJTIwQXV0aG9yaXR5LmNydDANBgkqhkiG9w0BAQsFAAOCAQEAFY6kjPmVGLYIPhHlDX+f
HshBGUEoGXAPbtNLg+WV7vvpu5dT3vgNsAFb9tjfa1j46D0ZppFMtxpM6GHGP89yqB4r/6+R3dMg
kwd5uinR22ExYISRwS4Aw8f6KwS4xK1SMnVGNWj2PhYrUyLRqPmRK/FviIgbCR55tPEZrDtsvaVW
GMlfpLFa3WIozj8Zkd7Q0Wn7H3+SRWwcSnf88bIbGyllr2Hr6QzC2tJlaD2VxcZdPqFxRkA7as3X
RQ2u/GnavSdbs/oTh5UO7mEMNASHfyl2pAgpGyLDy2ab8F5FjihnNCxHaKP9AXVrkm6ncqsYAHdG
ipZ3WqWrIosMV6m+VjCCBlgwggRAoAMCAQICE2gAyK8og25ErglxdPcAAADIrygwDQYJKoZIhvcN
AQELBQAwZDETMBEGCgmSJomT8ixkARkWA2NvbTEWMBQGCgmSJomT8ixkARkWBmFtYXpvbjETMBEG
CgmSJomT8ixkARkWA2FudDEgMB4GA1UEAxMXQW1hem9uLmNvbSBDSUEgQ0EgRzQgMDEwHhcNMjAx
MTEzMDczNDQ1WhcNMjExMTEzMDczNDQ1WjCBtzETMBEGCgmSJomT8ixkARkWA2NvbTEWMBQGCgmS
JomT8ixkARkWBmFtYXpvbjETMBEGCgmSJomT8ixkARkWA2FudDEMMAoGA1UECxMDT1JHMRIwEAYD
VQQLEwlBdXN0cmFsaWExDjAMBgNVBAsTBVNZRDEwMQ4wDAYDVQQLEwVVc2VyczEPMA0GA1UEAxMG
dGhhcmFyMSAwHgYJKoZIhvcNAQkBFhF0aGFyYXJAYW1hem9uLmNvbTCCASIwDQYJKoZIhvcNAQEB
BQADggEPADCCAQoCggEBAPXXLe5ApZHjG7dzdJANAxxaqxU2yp0rcBOtQEWdjIDVxTkI6AiSyEMe
0yaQMy0oaWAzfennPaRwzduDwIw6B91hsFqP3bE0QDcc6GUgRfW8WfAN1nfsCJZbBVZWBp5kV/B/
kTHoKwlgWW58EnpBdwg5C+WmcO1vgUGGZ7aAa5yAuhpN6gLqJR7atBxC4VpX+cgnpu7Q47E97acq
OIYfuYcpZy+zuCKNZ0Vi6bnnHw8Opc+xFeM0EF9YHwoA+1P4FMGvYvRWZ9UDnakYTWdpRRB+BPXU
eKdziAEinKFjtU9pa42k6/GpyBWtdaA2OIuhXLnNVWZKfaJkS6SGvufLdrUCAwEAAaOCAa0wggGp
MDwGCSsGAQQBgjcVBwQvMC0GJSsGAQQBgjcVCITFkUGFu5oogsGRL4TQqFfGmyMIh6brIoeJ+xoC
AWQCAQswEwYDVR0lBAwwCgYIKwYBBQUHAwQwCwYDVR0PBAQDAgXgMBsGCSsGAQQBgjcVCgQOMAww
CgYIKwYBBQUHAwQwHQYDVR0OBBYEFE14CkFPHcko33oMylI28+V9rXpyMB8GA1UdIwQYMBaAFOAp
5m6qUdZa9admS/aG9DXnqw+rME4GA1UdHwRHMEUwQ6BBoD+GPWh0dHA6Ly9wa2kuYW1hem9uLmNv
bS9jcmwvQW1hem9uLmNvbSUyMENJQSUyMENBJTIwRzQlMjAwMS5jcmwwWQYIKwYBBQUHAQEETTBL
MEkGCCsGAQUFBzAChj1odHRwOi8vcGtpLmFtYXpvbi5jb20vY3J0L0FtYXpvbi5jb20lMjBDSUEl
MjBDQSUyMEc0JTIwMDEuY3J0MD8GA1UdEQQ4MDagIQYKKwYBBAGCNxQCA6ATDBF0aGFyYXJAYW1h
em9uLmNvbYERdGhhcmFyQGFtYXpvbi5jb20wDQYJKoZIhvcNAQELBQADggIBAL1rzJpNq8ppMxZo
7viCegsnANvePrebNGn4QrkAk/xtgOQ8do1WypOCsGMbdSc8EKAEjlaufXXGf4KUkAQ5XLZoRLX+
DlrL1vZB38PILiaxtK0HnyN0kWQYA7GfKfBuUYr3AMjCsGbfTvoFcGQcTHOQZryXOW3xugDKedGU
LDp/cVn1jxlUwDmcNhXu9Yd+XjHNbEAeDhMw/QY6I1o/DFYXweazPICE8MkT4yEHoOotDxBlpbCA
gIYQSreJ0yYT8wtS2INBl3HCLoMMatUDmP2HeYHsLT/PKEqEou68eAwTrc44k5Bjvy1sHMOD4OAo
C92vb33Y8Sp/uvXT93tvBExLkIILBN+Py95Kb+6WEg5Ko741XmAv+z6DJ54MbAcJC/dT05e6Yt22
dBYVHW1rRWcrfJbmuS18dcuXhRizYBK3QuVXWU46bJgBft3VaPAqbxMX9E7jfD7Yh3Q9S1sPL9ad
gmaiDwnLRpXoCwiuZ57LLWPn2BoAphx7IexWIDNXac36j7hLw9PyY8tWsVz6VPowquWPfO4IoOmu
VXKaKUvlmp+ugCNqXRHpJkL+hbTsyxzeCvBMDVlz01aE3PDZIfc3wd5cmrXJT4XMS0nSX563BS9C
xA820RSUn1hFuFTg0zzovcksHMfuPxAWfjoKxi+bieOhbmXr+dlxh9EmY7jRMYIDtTCCA7ECAQEw
ezBkMRMwEQYKCZImiZPyLGQBGRYDY29tMRYwFAYKCZImiZPyLGQBGRYGYW1hem9uMRMwEQYKCZIm
iZPyLGQBGRYDYW50MSAwHgYDVQQDExdBbWF6b24uY29tIENJQSBDQSBHNCAwMQITaADIryiDbkSu
CXF09wAAAMivKDAJBgUrDgMCGgUAoIICDzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqG
SIb3DQEJBTEPFw0yMTAzMDcwODQzMjRaMCMGCSqGSIb3DQEJBDEWBBTyQnYSsoXiPbomQBZuWlRB
v0f0XTCBigYJKwYBBAGCNxAEMX0wezBkMRMwEQYKCZImiZPyLGQBGRYDY29tMRYwFAYKCZImiZPy
LGQBGRYGYW1hem9uMRMwEQYKCZImiZPyLGQBGRYDYW50MSAwHgYDVQQDExdBbWF6b24uY29tIENJ
QSBDQSBHNCAwMQITaADIryiDbkSuCXF09wAAAMivKDCBjAYLKoZIhvcNAQkQAgsxfaB7MGQxEzAR
BgoJkiaJk/IsZAEZFgNjb20xFjAUBgoJkiaJk/IsZAEZFgZhbWF6b24xEzARBgoJkiaJk/IsZAEZ
FgNhbnQxIDAeBgNVBAMTF0FtYXpvbi5jb20gQ0lBIENBIEc0IDAxAhNoAMivKINuRK4JcXT3AAAA
yK8oMIGTBgkqhkiG9w0BCQ8xgYUwgYIwCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBFjAKBggqhkiG
9w0DBzALBglghkgBZQMEAQIwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFAMAcGBSsOAwIa
MAsGCWCGSAFlAwQCAzALBglghkgBZQMEAgIwCwYJYIZIAWUDBAIBMA0GCSqGSIb3DQEBAQUABIIB
AFP5qhQupQYvzTYcm/at9RqFxYcqVPuaAkScpgBkQ5eYqTM4AMHOUD+ixqKBD3Hk5UayKNzsP6gu
73MCbz7jXEUhm22C1qG1doLVY4eXIL2jOZGvxtyEbNi3pTdivcmKDVoUqsU1+x4DYrDpATHWIGyr
ZiSzTbB8E98vEF5IegHS3oEpwe4wVgR4gKIaZCd1TzKYHOzOg05eHoP3+8AomOMdWyjlyDzf73hg
aZt5GvhyOngiXa/tixGHTwX9ujKeRss9FCu7CSDr2+NbdyHqUA9WhRV7iv63kvVb2jx8Xt4JMNZl
0MZaxd1Da/htKfH+DYQMliR4uNH25HL0pN7Wik8AAAAAAAA=

------=_NextPart_000_00DA_01D7138A.2325BF90--