Received: from localhost (maia-4.hub.org [200.46.204.183]) by postgresql.org (Postfix) with ESMTP id 953999FB1BD for ; Thu, 8 Feb 2007 17:55:31 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.183]) (amavisd-new, port 10024) with ESMTP id 47428-09 for ; Thu, 8 Feb 2007 17:55:28 -0400 (AST) X-Greylist: from auto-whitelisted by SQLgrey-1.7.4 Received: from beast.liquidweb.com (beast.liquidweb.com [64.91.224.2]) by postgresql.org (Postfix) with ESMTP id 1FD809FA5D8 for ; Thu, 8 Feb 2007 17:55:28 -0400 (AST) Received: from c-24-11-214-165.hsd1.mi.comcast.net ([24.11.214.165]:63437 helo=[10.2.1.2]) by beast.liquidweb.com with esmtpsa (TLSv1:RC4-MD5:128) (Exim 4.52) id 1HFHEb-00089Q-2I; Thu, 08 Feb 2007 16:55:17 -0500 Message-ID: <45CB9C4D.30902@intellicon.biz> Date: Thu, 08 Feb 2007 16:55:25 -0500 From: Benjamin Minshall Organization: Intellicon, Inc. User-Agent: Thunderbird 1.5.0.9 (Windows/20061207) MIME-Version: 1.0 To: Tom Lane CC: pgsql-performance@postgresql.org Subject: Re: stats collector process high CPU utilization References: <45CB5441.1040707@intellicon.biz> <21013.1170955076@sss.pgh.pa.us> <45CB5D06.7090004@intellicon.biz> <7994.1170960391@sss.pgh.pa.us> <45CB9330.3000603@intellicon.biz> <19935.1170969862@sss.pgh.pa.us> In-Reply-To: <19935.1170969862@sss.pgh.pa.us> Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=sha1; boundary="------------ms010007090105010406090404" X-ClamAntiVirus-Scanner: This mail is clean X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - beast.liquidweb.com X-AntiAbuse: Original Domain - postgresql.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - intellicon.biz X-Source: X-Source-Args: X-Source-Dir: X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=2.026 tagged_above=0 required=5 tests=AWL, BAYES_50, BIZ_TLD X-Spam-Level: ** X-Archive-Number: 200702/93 X-Sequence-Number: 23147 This is a cryptographically signed message in MIME format. --------------ms010007090105010406090404 Content-Type: multipart/mixed; boundary="------------040108000502050604010209" This is a multi-part message in MIME format. --------------040108000502050604010209 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Tom Lane wrote: > Benjamin Minshall writes: >> Tom Lane wrote: >>> Can you gather some info about what it's doing? >>> strace'ing the stats collector might prove interesting, also if you have >>> built it with --enable-debug then oprofile results would be helpful. > >> I will gather some strace info later today when I have a chance to >> shutdown the server. > > I don't see why you'd need to shut anything down. Just run > strace -p stats-process-ID > for a few seconds or minutes (enough to gather maybe a few thousand > lines of output). > Seems the problem may be related to a huge global/pgstat.stat file. Under 8.1.5 it was about 1 MB; now it's 90 MB in 8.2.3. I ran strace for 60 seconds: % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 95.71 1.119004 48652 23 rename 4.29 0.050128 0 508599 write 0.00 0.000019 0 249 22 poll 0.00 0.000000 0 23 open 0.00 0.000000 0 23 close 0.00 0.000000 0 34 getppid 0.00 0.000000 0 23 munmap 0.00 0.000000 0 23 setitimer 0.00 0.000000 0 23 22 sigreturn 0.00 0.000000 0 23 mmap2 0.00 0.000000 0 23 fstat64 0.00 0.000000 0 216 recv ------ ----------- ----------- --------- --------- ---------------- 100.00 1.169151 509282 44 total I attached an excerpt of the full strace with the many thousands of write calls filtered. -- Benjamin Minshall Senior Developer -- Intellicon, Inc. http://www.intellicon.biz --------------040108000502050604010209 Content-Type: text/plain; name="strace.excerpt" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="strace.excerpt" . . . write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0T\274\355"..., 4096) = 4096 write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0/O\227\27\230"..., 4096) = 4096 write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096 write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 3363) = 3363 close(3) = 0 munmap(0xacdfc000, 4096) = 0 rename("global/pgstat.tmp", "global/pgstat.stat") = 0 poll([{fd=5, events=POLLIN|POLLERR, revents=POLLIN}], 1, 2000) = 1 recv(5, "\1\0\0\0\320\3\0\0:\204\30\0\16\0\0\0\1\0\0\0\0\0\0\0\301"..., 1000, 0) = 976 setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0, 500}}, NULL) = 0 poll([{fd=5, events=POLLIN|POLLERR, revents=POLLIN}], 1, 2000) = 1 recv(5, "\1\0\0\0\320\3\0\0:\204\30\0\16\0\0\0\0\0\0\0\0\0\0\0\322"..., 1000, 0) = 976 poll([{fd=5, events=POLLIN|POLLERR, revents=POLLIN}], 1, 2000) = 1 recv(5, "\1\0\0\0H\3\0\0:\204\30\0\f\0\0\0\1\0\0\0\0\0\0\0\301\204"..., 1000, 0) = 840 poll([{fd=5, events=POLLIN|POLLERR, revents=POLLIN}], 1, 2000) = 1 recv(5, "\1\0\0\0\320\3\0\0:\204\30\0\16\0\0\0\1\0\0\0\0\0\0\0\301"..., 1000, 0) = 976 poll([{fd=5, events=POLLIN|POLLERR, revents=POLLIN}], 1, 2000) = 1 recv(5, "\1\0\0\0\320\3\0\0:\204\30\0\16\0\0\0\0\0\0\0\0\0\0\000"..., 1000, 0) = 976 poll([{fd=5, events=POLLIN|POLLERR, revents=POLLIN}], 1, 2000) = 1 recv(5, "\1\0\0\0\240\0\0\0:\204\30\0\2\0\0\0\0\0\0\0\0\0\0\0\303"..., 1000, 0) = 160 poll([{fd=5, events=POLLIN|POLLERR}], 1, 2000) = -1 EINTR (Interrupted system call) --- SIGALRM (Alarm clock) @ 0 (0) --- sigreturn() = ? (mask now []) getppid() = 22447 open("global/pgstat.tmp", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 3 fstat64(3, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xacdfc000 write(3, "\226\274\245\1D\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096 write(3, "Z\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096 write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096 . . . --------------040108000502050604010209-- --------------ms010007090105010406090404 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIQODCC BIowggNyoAMCAQICECf06hH0eobEbp27bqkXBwcwDQYJKoZIhvcNAQEFBQAwbzELMAkGA1UE BhMCU0UxFDASBgNVBAoTC0FkZFRydXN0IEFCMSYwJAYDVQQLEx1BZGRUcnVzdCBFeHRlcm5h bCBUVFAgTmV0d29yazEiMCAGA1UEAxMZQWRkVHJ1c3QgRXh0ZXJuYWwgQ0EgUm9vdDAeFw0w NTA2MDcwODA5MTBaFw0yMDA1MzAxMDQ4MzhaMIGuMQswCQYDVQQGEwJVUzELMAkGA1UECBMC VVQxFzAVBgNVBAcTDlNhbHQgTGFrZSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5l dHdvcmsxITAfBgNVBAsTGGh0dHA6Ly93d3cudXNlcnRydXN0LmNvbTE2MDQGA1UEAxMtVVRO LVVTRVJGaXJzdC1DbGllbnQgQXV0aGVudGljYXRpb24gYW5kIEVtYWlsMIIBIjANBgkqhkiG 9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsjmFpPJ9q0E7YkY3rs3BYHW8OWX5ShpHornMSMxqmNVN NRm5pELlzkniii8efNIxB8dOtINknS4p1aJkxIW9hVE1eaROaJB7HHqkkqgX8pgV8pPMyaQy lbsMTzC9mKALi+VuG6JG+ni8om+rWV6lL8/K2m2qL+usobNqqrcuZzWLeeEeaYji5kbNoKXq vgvOdjp6Dpvq/NonWz1zHyLmSGHGTPNpsaguG7bUMSAsvIKKjqQOpdeJQ/wWWq8dcdcRWdq6 hw2v+vPhwvCkxWeM1tZUOt4KpLoDd7NlyP0e03RiqhjKaJMeoYV+9Udly/hNVyh00jT/MLbu 9mIwFIws6wIDAQABo4HhMIHeMB8GA1UdIwQYMBaAFK29mHo0tCb3+sQmVO8DveAky1QaMB0G A1UdDgQWBBSJgmd9xJ0mcABLtFBIfN49rgRufTAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/ BAUwAwEB/zB7BgNVHR8EdDByMDigNqA0hjJodHRwOi8vY3JsLmNvbW9kb2NhLmNvbS9BZGRU cnVzdEV4dGVybmFsQ0FSb290LmNybDA2oDSgMoYwaHR0cDovL2NybC5jb21vZG8ubmV0L0Fk ZFRydXN0RXh0ZXJuYWxDQVJvb3QuY3JsMA0GCSqGSIb3DQEBBQUAA4IBAQAZ2IkRbyispgCi 54fBm5AD236hEv0e8+LwAamUVEJrmgnEoG3XkJIEA2Z5Q3H8+G+v23ZF4jcaPd3kWQR4rBz0 g0bzes9bhHIt5UbBuhgRKfPLSXmHPLptBZ2kbWhPrXIUNqi5sf2/z3/wpGqUNVCPz4FtVbHd WTBK322gnGQfSXzvNrv042n0+DmPWq1LhTq3Du3Tzw1EovsEv+QvcI4l+1pUBrPQxLxtjftz Mizpm4QkLdZ/kXpoAlAfDj9N6cz1u2fo3BwuO/xOzf4CjuOoEwqlJkRl6RDyTVKnrtw+ymsy XEFs/vVdoOr/0fqbhlhtPZZH5f4ulQTCAMyOofK7MIIF0TCCBLmgAwIBAgIRAO/VhC9jut1L VBHVMdY1Si4wDQYJKoZIhvcNAQEFBQAwga4xCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJVVDEX MBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNVBAoTFVRoZSBVU0VSVFJVU1QgTmV0d29y azEhMB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3QuY29tMTYwNAYDVQQDEy1VVE4tVVNF UkZpcnN0LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1haWwwHhcNMDcwMTAzMDAwMDAw WhcNMDgwMTAzMjM1OTU5WjCB5DE1MDMGA1UECxMsQ29tb2RvIFRydXN0IE5ldHdvcmsgLSBQ RVJTT05BIE5PVCBWQUxJREFURUQxRjBEBgNVBAsTPVRlcm1zIGFuZCBDb25kaXRpb25zIG9m IHVzZTogaHR0cDovL3d3dy5jb21vZG8ubmV0L3JlcG9zaXRvcnkxHzAdBgNVBAsTFihjKTIw MDMgQ29tb2RvIExpbWl0ZWQxGjAYBgNVBAMTEUJlbmphbWluIE1pbnNoYWxsMSYwJAYJKoZI hvcNAQkBFhdtaW5zaGFsbEBpbnRlbGxpY29uLmJpejCBnzANBgkqhkiG9w0BAQEFAAOBjQAw gYkCgYEA4TykjizBp56X8r4qtMHghhYPSqjmfPc2khSEGsVmHcGZ4PvF3PcEzZAH7379blkA MofRC64IssMJnXhh6B4LZwPC6GxOK3TtsU0W76w1KBumVGWOGdz7qKbTCorRMlv2eZe4wQ/L DEdH3COLU14q0AaJPoKsSBp9s4QkRqGPP8kCAwEAAaOCAjQwggIwMB8GA1UdIwQYMBaAFImC Z33EnSZwAEu0UEh83j2uBG59MB0GA1UdDgQWBBTSE99IHnuw6o43lN4Ok9ErH4IjvDAOBgNV HQ8BAf8EBAMCBaAwDAYDVR0TAQH/BAIwADAgBgNVHSUEGTAXBggrBgEFBQcDBAYLKwYBBAGy MQEDBQIwEQYJYIZIAYb4QgEBBAQDAgUgMEYGA1UdIAQ/MD0wOwYMKwYBBAGyMQECAQEBMCsw KQYIKwYBBQUHAgEWHWh0dHBzOi8vc2VjdXJlLmNvbW9kby5uZXQvQ1BTMIGlBgNVHR8EgZ0w gZowTKBKoEiGRmh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL1VUTi1VU0VSRmlyc3QtQ2xpZW50 QXV0aGVudGljYXRpb25hbmRFbWFpbC5jcmwwSqBIoEaGRGh0dHA6Ly9jcmwuY29tb2RvLm5l dC9VVE4tVVNFUkZpcnN0LUNsaWVudEF1dGhlbnRpY2F0aW9uYW5kRW1haWwuY3JsMIGGBggr BgEFBQcBAQR6MHgwOwYIKwYBBQUHMAKGL2h0dHA6Ly9jcnQuY29tb2RvY2EuY29tL1VUTkFk ZFRydXN0Q2xpZW50Q0EuY3J0MDkGCCsGAQUFBzAChi1odHRwOi8vY3J0LmNvbW9kby5uZXQv VVROQWRkVHJ1c3RDbGllbnRDQS5jcnQwIgYDVR0RBBswGYEXbWluc2hhbGxAaW50ZWxsaWNv bi5iaXowDQYJKoZIhvcNAQEFBQADggEBAFpPQSgEbe4qqAluhCVH5VVbS1IAAKEudK/pZB2A 4/wJD7W12XAtSzt4snOYEAecIU9YHrTAPqG592X30+7Vb3EsItTe5YQMUsHIYPM6vV1KYHRs gHTVi0qRnuQDrUsg6QjgSS/N1MaIbwBKXXBegsDQko27YuhTMe5nMr0Fk6MZn2HXIlVES4nu cpovUbNkipMeHNAh8zRC/04/Cs5n5eKarpM+xgedcOgNlsJoKrqzD8n6DRMm417xugLJvSPt ZO1ZpmGvqP4FHV2hu8/T2Xz6vklPHQfkY9ltnbmuyzhG7MfdVN2VVzZxVwpilMmdfpfO0dR/ +TiWRw3M5gl7rtcwggXRMIIEuaADAgECAhEA79WEL2O63UtUEdUx1jVKLjANBgkqhkiG9w0B AQUFADCBrjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5TYWx0IExha2Ug Q2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYDVQQLExhodHRwOi8v d3d3LnVzZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1VU0VSRmlyc3QtQ2xpZW50IEF1dGhl bnRpY2F0aW9uIGFuZCBFbWFpbDAeFw0wNzAxMDMwMDAwMDBaFw0wODAxMDMyMzU5NTlaMIHk MTUwMwYDVQQLEyxDb21vZG8gVHJ1c3QgTmV0d29yayAtIFBFUlNPTkEgTk9UIFZBTElEQVRF RDFGMEQGA1UECxM9VGVybXMgYW5kIENvbmRpdGlvbnMgb2YgdXNlOiBodHRwOi8vd3d3LmNv bW9kby5uZXQvcmVwb3NpdG9yeTEfMB0GA1UECxMWKGMpMjAwMyBDb21vZG8gTGltaXRlZDEa MBgGA1UEAxMRQmVuamFtaW4gTWluc2hhbGwxJjAkBgkqhkiG9w0BCQEWF21pbnNoYWxsQGlu dGVsbGljb24uYml6MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDhPKSOLMGnnpfyviq0 weCGFg9KqOZ89zaSFIQaxWYdwZng+8Xc9wTNkAfvfv1uWQAyh9ELrgiywwmdeGHoHgtnA8Lo bE4rdO2xTRbvrDUoG6ZUZY4Z3PuoptMKitEyW/Z5l7jBD8sMR0fcI4tTXirQBok+gqxIGn2z hCRGoY8/yQIDAQABo4ICNDCCAjAwHwYDVR0jBBgwFoAUiYJnfcSdJnAAS7RQSHzePa4Ebn0w HQYDVR0OBBYEFNIT30gee7DqjjeU3g6T0SsfgiO8MA4GA1UdDwEB/wQEAwIFoDAMBgNVHRMB Af8EAjAAMCAGA1UdJQQZMBcGCCsGAQUFBwMEBgsrBgEEAbIxAQMFAjARBglghkgBhvhCAQEE BAMCBSAwRgYDVR0gBD8wPTA7BgwrBgEEAbIxAQIBAQEwKzApBggrBgEFBQcCARYdaHR0cHM6 Ly9zZWN1cmUuY29tb2RvLm5ldC9DUFMwgaUGA1UdHwSBnTCBmjBMoEqgSIZGaHR0cDovL2Ny bC5jb21vZG9jYS5jb20vVVROLVVTRVJGaXJzdC1DbGllbnRBdXRoZW50aWNhdGlvbmFuZEVt YWlsLmNybDBKoEigRoZEaHR0cDovL2NybC5jb21vZG8ubmV0L1VUTi1VU0VSRmlyc3QtQ2xp ZW50QXV0aGVudGljYXRpb25hbmRFbWFpbC5jcmwwgYYGCCsGAQUFBwEBBHoweDA7BggrBgEF BQcwAoYvaHR0cDovL2NydC5jb21vZG9jYS5jb20vVVROQWRkVHJ1c3RDbGllbnRDQS5jcnQw OQYIKwYBBQUHMAKGLWh0dHA6Ly9jcnQuY29tb2RvLm5ldC9VVE5BZGRUcnVzdENsaWVudENB LmNydDAiBgNVHREEGzAZgRdtaW5zaGFsbEBpbnRlbGxpY29uLmJpejANBgkqhkiG9w0BAQUF AAOCAQEAWk9BKARt7iqoCW6EJUflVVtLUgAAoS50r+lkHYDj/AkPtbXZcC1LO3iyc5gQB5wh T1getMA+obn3ZffT7tVvcSwi1N7lhAxSwchg8zq9XUpgdGyAdNWLSpGe5AOtSyDpCOBJL83U xohvAEpdcF6CwNCSjbti6FMx7mcyvQWToxmfYdciVURLie5ymi9Rs2SKkx4c0CHzNEL/Tj8K zmfl4pqukz7GB51w6A2WwmgqurMPyfoNEybjXvG6Asm9I+1k7VmmYa+o/gUdXaG7z9PZfPq+ SU8dB+Rj2W2dua7LOEbsx91U3ZVXNnFXCmKUyZ1+l87R1H/5OJZHDczmCXuu1zGCA9IwggPO AgEBMIHEMIGuMQswCQYDVQQGEwJVUzELMAkGA1UECBMCVVQxFzAVBgNVBAcTDlNhbHQgTGFr ZSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxITAfBgNVBAsTGGh0dHA6 Ly93d3cudXNlcnRydXN0LmNvbTE2MDQGA1UEAxMtVVROLVVTRVJGaXJzdC1DbGllbnQgQXV0 aGVudGljYXRpb24gYW5kIEVtYWlsAhEA79WEL2O63UtUEdUx1jVKLjAJBgUrDgMCGgUAoIIC YzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0wNzAyMDgyMTU1 MjVaMCMGCSqGSIb3DQEJBDEWBBSnuw4N3TasMXHFo6ViYM3co7ZnNzBSBgkqhkiG9w0BCQ8x RTBDMAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMC BzANBggqhkiG9w0DAgIBKDCB1QYJKwYBBAGCNxAEMYHHMIHEMIGuMQswCQYDVQQGEwJVUzEL MAkGA1UECBMCVVQxFzAVBgNVBAcTDlNhbHQgTGFrZSBDaXR5MR4wHAYDVQQKExVUaGUgVVNF UlRSVVNUIE5ldHdvcmsxITAfBgNVBAsTGGh0dHA6Ly93d3cudXNlcnRydXN0LmNvbTE2MDQG A1UEAxMtVVROLVVTRVJGaXJzdC1DbGllbnQgQXV0aGVudGljYXRpb24gYW5kIEVtYWlsAhEA 79WEL2O63UtUEdUx1jVKLjCB1wYLKoZIhvcNAQkQAgsxgceggcQwga4xCzAJBgNVBAYTAlVT MQswCQYDVQQIEwJVVDEXMBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNVBAoTFVRoZSBV U0VSVFJVU1QgTmV0d29yazEhMB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3QuY29tMTYw NAYDVQQDEy1VVE4tVVNFUkZpcnN0LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1haWwC EQDv1YQvY7rdS1QR1THWNUouMA0GCSqGSIb3DQEBAQUABIGAb7rwUsdwofduF5JsPv908EPe iFR1gIFmREdlRZYkmFNlPl/1Ekmi2lgw2pa2cGWcNCKvIJGvofsKVfNps3CL+YHbz6a7CSES LEdPq6CqI7e5gE4nPEcJONzNBjeZIzwe4CHh0n+EI+YN5DH4/3NYf5zjfMXzzsznQZnxnVj0 YTsAAAAAAAA= --------------ms010007090105010406090404--