Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1ZCQc4-0005Cw-FN for pgsql-performance@arkaria.postgresql.org; Tue, 07 Jul 2015 11:00:32 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.84) (envelope-from ) id 1ZCQc3-0000Sr-VS for pgsql-performance@arkaria.postgresql.org; Tue, 07 Jul 2015 11:00:32 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84) (envelope-from ) id 1ZCQYg-0005NS-LI for pgsql-performance@postgresql.org; Tue, 07 Jul 2015 10:57:02 +0000 Received: from smtp-o-1.desy.de ([2001:638:700:1038::1:9a]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84) (envelope-from ) id 1ZCQYd-00089h-69 for pgsql-performance@postgresql.org; Tue, 07 Jul 2015 10:57:01 +0000 X-Clacks-Overhead: GNU Terry Pratchett Received: from smtp-map-1.desy.de (smtp-map-1.desy.de [131.169.56.66]) by smtp-o-1.desy.de (DESY-O-1) with ESMTP id 41087280A16 for ; Tue, 7 Jul 2015 12:56:55 +0200 (CEST) Received: from ZITSWEEP2.win.desy.de (zitsweep2.win.desy.de [131.169.97.96]) by smtp-map-1.desy.de (DESY_MAP_1) with ESMTP id 2665813E9F for ; Tue, 7 Jul 2015 12:56:54 +0200 (MEST) Received: from smtp-intra-3.desy.de (lb-40-26.desy.de) by ZITSWEEP2.win.desy.de (Clearswift SMTPRS 5.5.0) with ESMTP id ; Tue, 7 Jul 2015 12:56:54 +0200 Received: from z-mta-3.desy.de (z-mta-3.desy.de [131.169.55.137]) by smtp-intra-3.desy.de (DESY-INTRA-3) with ESMTP id 221C3114A; Tue, 7 Jul 2015 12:56:54 +0200 (MEST) Received: from z-mta-3.desy.de (localhost [127.0.0.1]) by z-mta-3.desy.de (Postfix) with ESMTPS id 1B940401A6; Tue, 7 Jul 2015 12:56:54 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by z-mta-3.desy.de (Postfix) with ESMTP id 0C9F0401A5; Tue, 7 Jul 2015 12:56:54 +0200 (CEST) Received: from z-mta-3.desy.de ([127.0.0.1]) by localhost (z-mta-3.desy.de [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id P2wN9wuVNuEQ; Tue, 7 Jul 2015 12:56:53 +0200 (CEST) Received: from z-mbx-2.desy.de (z-mbx-2.desy.de [131.169.55.140]) by z-mta-3.desy.de (Postfix) with ESMTP id E9A6440004; Tue, 7 Jul 2015 12:56:53 +0200 (CEST) Date: Tue, 7 Jul 2015 12:56:53 +0200 (CEST) From: "Mkrtchyan, Tigran" To: "Graeme B. Bell" Cc: Steve Crawford , "Wes Vaske (wvaske)" , pgsql-performance Message-ID: <1052663055.3807152.1436266613895.JavaMail.zimbra@desy.de> In-Reply-To: <985C5CEF-2E55-46AD-A29F-90D48D057A3F@skogoglandskap.no> References: <31701C89A44B714FB8D242F806E7286C5AED32FA@NTXBOIMBX01.micron.com> <559AB332.50304@pinpointresearch.com> <52A08B87-2A8A-43F2-92CE-2514F5AD141B@skogoglandskap.no> <1253229687.3798367.1436264898828.JavaMail.zimbra@desy.de> <985C5CEF-2E55-46AD-A29F-90D48D057A3F@skogoglandskap.no> Subject: Re: New server: SSD/RAID recommendations? MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_3807150_141659235.1436266613894" X-Mailer: Zimbra 8.6.0_GA_1169 (ZimbraWebClient - FF38 (Linux)/8.6.0_GA_1169) Thread-Topic: [PERFORM] New server: SSD/RAID recommendations? Thread-Index: AQHQuAz+Ylwsxg8sFk+LO4PXBB+2uZ3Pq+MAG0CLk6X/JgAoALuRZPaN X-Pg-Spam-Score: -2.7 (--) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-performance Precedence: bulk Sender: pgsql-performance-owner@postgresql.org ------=_Part_3807150_141659235.1436266613894 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit ----- Original Message ----- > From: "Graeme B. Bell" > To: "Mkrtchyan, Tigran" > Cc: "Graeme B. Bell" , "Steve Crawford" , "Wes Vaske (wvaske)" > , "pgsql-performance" > Sent: Tuesday, July 7, 2015 12:38:10 PM > Subject: Re: [PERFORM] New server: SSD/RAID recommendations? > I am unsure about the performance side but, ZFS is generally very attractive to > me. > > Key advantages: > > 1) Checksumming and automatic fixing-of-broken-things on every file (not just > postgres pages, but your scripts, O/S, program files). > 2) Built-in lightweight compression (doesn't help with TOAST tables, in fact > may slow them down, but helpful for other things). This may actually be a net > negative for pg so maybe turn it off. > 3) ZRAID mirroring or ZRAID5/6. If you have trouble persuading someone that it's > safe to replace a RAID array with a single drive... you can use a couple of > NVMe SSDs with ZFS mirror or zraid, and get the same availability you'd get > from a RAID controller. Slightly better, arguably, since they claim to have > fixed the raid write-hole problem. > 4) filesystem snapshotting > > Despite the costs of checksumming etc., I suspect ZRAID running on a fast CPU > with multiple NVMe drives will outperform quite a lot of the alternatives, with > great data integrity guarantees. We are planing to have a test setup as well. For now I have single NVMe SSD on my test system: # lspci | grep NVM 85:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller 171X (rev 03) # mount | grep nvm /dev/nvme0n1p1 on /var/lib/pgsql/9.5 type ext4 (rw,noatime,nodiratime,data=ordered) and quite happy with it. We have write heavy workload on it to see when it will break. Postgres Performs very well. About x2.5 faster than with regular disks with a single client and almost linear with multiple clients (picture attached. On Y number of high level op/s our application does, X number of clients). The setup is used last 3 months. Looks promising but for production we need to to have disk size twice as big as on the test system. Until today, I was planning to use a RAID10 with a HW controller... Related to ZFS. We use ZFSonlinux and behaviour is not as good as with solaris. Let's re-phrase it: performance is unpredictable. We run READZ2 with 30x3TB disks. Tigran. > > Haven't built one yet. Hope to, later this year. Steve, I would love to know > more about how you're getting on with your NVMe disk in postgres! > > Graeme. > > On 07 Jul 2015, at 12:28, Mkrtchyan, Tigran wrote: > >> Thanks for the Info. >> >> So if RAID controllers are not an option, what one should use to build >> big databases? LVM with xfs? BtrFs? Zfs? >> >> Tigran. >> >> ----- Original Message ----- >>> From: "Graeme B. Bell" >>> To: "Steve Crawford" >>> Cc: "Wes Vaske (wvaske)" , "pgsql-performance" >>> >>> Sent: Tuesday, July 7, 2015 12:22:00 PM >>> Subject: Re: [PERFORM] New server: SSD/RAID recommendations? >> >>> Completely agree with Steve. >>> >>> 1. Intel NVMe looks like the best bet if you have modern enough hardware for >>> NVMe. Otherwise e.g. S3700 mentioned elsewhere. >>> >>> 2. RAID controllers. >>> >>> We have e.g. 10-12 of these here and e.g. 25-30 SSDs, among various machines. >>> This might give people idea about where the risk lies in the path from disk to >>> CPU. >>> >>> We've had 2 RAID card failures in the last 12 months that nuked the array with >>> days of downtime, and 2 problems with batteries suddenly becoming useless or >>> suddenly reporting wildly varying temperatures/overheating. There may have been >>> other RAID problems I don't know about. >>> >>> Our IT dept were replacing Seagate HDDs last year at a rate of 2-3 per week (I >>> guess they have 100-200 disks?). We also have about 25-30 Hitachi/HGST HDDs. >>> >>> So by my estimates: >>> 30% annual problem rate with RAID controllers >>> 30-50% failure rate with Seagate HDDs (backblaze saw similar results) >>> 0% failure rate with HGST HDDs. >>> 0% failure in our SSDs. (to be fair, our one samsung SSD apparently has a bug >>> in TRIM under linux, which I'll need to investigate to see if we have been >>> affected by). >>> >>> also, RAID controllers aren't free - not just the money but also the management >>> of them (ever tried writing a complex install script that interacts work with >>> MegaCLI? It can be done but it's not much fun.). Just take a look at the >>> MegaCLI manual and ask yourself... is this even worth it (if you have a good >>> MTBF on an enterprise SSD). >>> >>> RAID was meant to be about ensuring availability of data. I have trouble >>> believing that these days.... >>> >>> Graeme Bell >>> >>> >>> On 06 Jul 2015, at 18:56, Steve Crawford wrote: >>> >>>> >>>> 2. We don't typically have redundant electronic components in our servers. Sure, >>>> we have dual power supplies and dual NICs (though generally to handle external >>>> failures) and ECC-RAM but no hot-backup CPU or redundant RAM banks and...no >>>> backup RAID card. Intel Enterprise SSD already have power-fail protection so I >>>> don't need a RAID card to give me BBU. Given the MTBF of good enterprise SSD >>>> I'm left to wonder if placing a RAID card in front merely adds a new point of >>>> failure and scheduled-downtime-inducing hands-on maintenance (I'm looking at >>>> you, RAID backup battery). >>> >>> >>> >>> -- >>> Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) >>> To make changes to your subscription: >>> http://www.postgresql.org/mailpref/pgsql-performance > > > > -- > Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-performance ------=_Part_3807150_141659235.1436266613894 Content-Type: image/png; name="pg-with-ssd.png" Content-Disposition: attachment; filename="pg-with-ssd.png" Content-Transfer-Encoding: base64 iVBORw0KGgoAAAANSUhEUgAAAoAAAAHgCAMAAAACDyzWAAABNVBMVEX///8A AACgoKD/AAAAwAAAgP/AAP8A7u7AQADIyABBaeH/wCAAgEDAgP8wYICLAABA gAD/gP9//9SlKir//wBA4NAAAAAaGhozMzNNTU1mZmZ/f3+ZmZmzs7PAwMDM zMzl5eX////wMjKQ7pCt2ObwVfDg///u3YL/tsGv7u7/1wAA/wAAZAAA/38i iyIui1cAAP8AAIsZGXAAAIAAAM2HzusA////AP8AztH/FJP/f1DwgID/RQD6 gHLplnrw5oy9t2u4hgv19dyggCD/pQDugu6UANPdoN2QUEBVay+AFACAFBSA QBSAQICAYMCAYP+AgAD/gED/oED/oGD/oHD/wMD//4D//8DNt57w//Cgts3B /8HNwLB8/0Cg/yC+vr7f399fX18fHx+fn58/Pz+/v7+ZZgDWVIPzAAAACXBI WXMAAA7EAAAOxAGVKw4bAAARzUlEQVR4nO2dCZKjOhAFTUTfwpfDLHP/I4xZ zGZhg1VAVSkz4vdoxm6hj7MlJNGP2w0AAAAAAAAAAAAAAAAAAAAAAAAOJ2uZ l/JHlj3yeQngELJAqbjn+b2YlwAOISDg/dF8fZTTEsAxBASs6uZrXU1LAMeQ FVnRXeUNpaL9a15MSwDH8Ozknld59bTUd4XZtARwIGU1LSEgnEyeTUvfh+AM 4IW8gN8nIYJdoveqVDZKXVXzIfj7Moy29iuuSmWjFFVVlXk/9RhL3xei9bRf fVUqG6WoqvrxWnwZS9+34vS0X31VKhultCrlBwWVIKBGyioruhncUGomjEV1 z0Nv70aful9eqJpr7uEU7zjX6biAgJ+5N4v6j2xWav7L63sR2FgvizK/1VW/ 8JqXzXsQUN1BDdGtn95npf6c1d2/TMn7f7q/3lQXCKjwoIYI3eP2KnVLW9OX Jstcb5tQYyG73Z/jdPfWZ2+ZdSN8X2fR/j3rXxvf1ZWy23OIr/KI/6EPIKBC qvK99Dpnw77ScBInfeKHHjB7PK8f69ao59f8OVAPBjav5HXVDeDPd9TtS2Mp e7TvHw8jCQIqJH/ONsp8Xgp0acvCx2vArBnHu12prsssH29vaV67D+8aSt23 Dl2vLAiokvr+KLpB71XaJuD6LDhra2v7z+UWfdUv1Wav18Z3taVs8X5REFAt j2pamg3Bs538tyF4Vsq6d0/vTMrmrzTT7G4RZ6x5XlpW3vFvL8H/SwRUy/wW jz2TkNto5fKScejl5j1aXlaPyWu3WYkeMDU6x5pPfCz15ywfvBj1XC7DtFXc uz+XvrbXeYtrwK6WbPLabVbiGjA12su4urjPSrfFQvR4EucL0R150Wya5Pel r80fy1lwdx/JMAvOu6nHWGIWnBrN1lo3lxhLb1tx2fz9w1VcT958ZzHeCTK9 kCvn64Dtt7/6z3GNcCixDgiXcujHhYDwDQSES0FA8AsCwqUgIFwKAsKlICD0 LLePX6uOn34ZQPCoZ4KAGpkJeBv3Xd72YI456pkgoEaWAt76XwCYlg886pkg oEYCArZ3ILzdh3PMUc8EATUSEnByH85B92MhYKq83R86DawKyXjMxxZb6Xhj biiQ4/hoDviR9/uT16RTLuBQCkUSHR9OBD8Ruj9+7xD8t5dgS8QEDIWynRDP Br8Q/PUMm5OQ4ftDsZQnBFTCD3z59aBRwHyyDDP+MoAo0QJ+ysYfS7IHhSjC /hldiP6YjT+9DVzyoBDByq9Hmt6KW8vGR0B9rOl3FSIu5NnOIVgqHR32oso/ 0ZR8JiEWUOVfi9gQzDKMflYv/y4k0oUv2fgsRGtCoX7RLnzJxmcrThEq/eNm hFTQOPw2IGAaKNUPARNBrX8ImAJah98GBPSPYv0QMAFU+4eA3tE8/DYgoG+U 64eAzlHvHwJ6Rvvw24CAfjGgHwL6xUL3d0NAt9jQDwG9YsU/BHSJkeG3AQEd Ykc/BPSIJf8Q0B2Ght8GBHSGLf0Q0BvW/ENAVxgbfhsQ0BH29ENARxjs/m4I 6Aab+iGgE6zqh4AusKufiAt50VbyOS9f+qAwYFk/EReqMpvXRDjRidjWT8KF ++M2F5B4tvOwrp+AC/XrSRIf8/KFDwod5vUTcGEIKP+Yly98UGhx4F+0C4/7 q5KPefmyB4UWD/5FB1RW80rW8vJFDwotLvyLdaEbZMdK1vLyFwclJT8eB/4J eDB5wmfLWl7+4rviDgo3F/61iLgwH4JZhjkBL/7JCfglL1/+oCljf/lvQEzA L3n58gdNGD/6cTOCRTz5h4D2cOUfAprDl38IaA1n/iGgMbz5h4C2cOcfAprC n38IaAhHy88jCGgGj/ohoB18+oeAVnDqHwLawOXlXwsCWsCtfghoAsf+IaAB PPuHgOrxe/nXgoDK8a0fAmrHu38IqBv3/iGgavz7h4CKcT796EBAtaSgHwLq JQ3/EFArifiHgEpJxT8E1Eky/gmm5IcCOYjm+I10/BNMyQ9FEhFO9AtJLL+8 EEvJD4WyEc/2CynpJ5iSH4qlJKDyB9LyTy4lPxTMS0r+fhLzTy4lPxRNTkj5 XpK6/GsRS8lHQAGS008wJX/XEExKfpDU/BNNyWcSEkt6w2+DWEY0yzCRJKmf 5GMaWIiOIlH/BAVkKy6CNIffBm5G0ECy+iGgChL2DwGvJ93htwEBryZp/RDw chL3DwGvJe3htwEBryR5/RDwUvAPAS+E4bcBAa8C/VoQ8CLwrwMBrwH/ehDw Crj8G0DAC0C/EQQ8H/ybgICng39TEPBs8G8GAp4M/s1BwHPBvwUIeCr4twQB zwT/3kDA82D5OQACngb6hUDAs8C/IAh4EvgXBgHPAf9WiM0HfBRZUbXpQ9kQ tkU0xxv4t0akC1WZ3/K6esWkdhBOtAT/VhFxIZ9EpBLP9g7+rSMjYDGtiYDK Bfj3AQkX+iH4eTXYXu+Rkj8H/z4R7UIz82i7uGd397zeqwkpX4B/HxFwIS+L +6tcVgg4B/8+I+JCPQyyzXSElPwpCLiOnAdDLY2ATEIm4N8XhHvAZghmGWYE /74RuxBd5801YN0vSbeTEBaiB/DvKxJbcY96KOXNP7IV14N/3+FmhOPAvw0g 4GHg3xYQ8CjwbxMIeBQIuAkEPAj82wYCHgP+bQQBDwH/toKAR4B/m0FAefgF 9B0goDTotwsEFAb99oGAotD97QUBBUG//SCgHOj3AwgoBd3fTyCgDOj3Iwgo Avr9CgJKgH8/g4AC4N/vIGA8+BcBAkaDfzEgYCz4FwUCRoJ/cSBgHPgXCQJG gX+xIGAM+BeNXEp+KJDDeTQH/sUjl5IfiiTyHU6EfwKIpeSHQtl8x7PhnwRi KfmhWErXAZVm/fubI1jVL1WKpeSHgnk9p+Sr8m/LR7/2nhWJNiHQdLGU/FA0 ueOQchX+vYuwYoWQLEcglpKfloDX+7fu1OIVxfI1iGVE7xqCrafkn+Dfp4Hu u1P9O7TLJ5mSn9Ik5DD/wtbN/mm7U7rleyHWAya0DCPv37aLeqnrflWIpeSn sxAt6Z/cbNIqYin5yWzFyfiHeT3cjLAPgV+/xLwpCLiLaP1wbwEC7iHSP+x7 BwF3EOUf9gVBwO1E+Id9ayDgZn72D/s+gIBb+c0/ZrxfQMCN/OAf8m0AAbex 1z/k2wgCbmKff8i3HQTcwi7/sG8PCLiBPf6h3z4Q8Ds7/EO/vSDgV7b7h377 QcBvbPYP/X4BAb+w1T/0+w0E/MIWAVn0+x0E/Mw3/7i7NBIE/Mgn/3BPAgT8 xKp/uCcFAn5gzT/kkwMB1wn7R98nCgKuEvQP+4RBwDUC/tH5yYOAK7z7h31H gIBhlv7R+R0EAgaZ+4d9xyGQDdM9paELe2urs58NM/UP+w4l0oWsqm91Uc5r Mp+ONfqHfUcT6ULzgJA+AnWoyXw+4Ms/7DsBEReyWU3WE1J7/7DvFCRcKLse sEkKzG/mH9PQ+Yd+JyGRkt8GpDbd3fN6rzafkt8IiH6nEe9C3g20HWVlXcB/ 6Hcq0S70/d/rb5nxxzT8Q7/zkPBg7l8roOVJyD/0O5lIFxb+tUOw4WUY9Dud 2Mc0lGMp7ychZhei0e8CYndCOvLXAxvy5h9tbsX9/V3/ALgE4WaEjmfvh39X gIC3fs8N/y4BAftLP/y7htQFfN1wgH8XkbaAw7wX/64iZQHHZRf8u4yEBRxX /fDvOtIVEP9UkKyA+KeDVAWc7Loh4JUkKiD+aSFNAfFPDUkKiH96SFFA/FNE 2gLi3+UkKCD+aSI9AfFPFckJiH+6SE1A/FNGYgLinzbSEhD/1JGUgPinj5QE xD+FpCPgH/5pJBkB2X/TSSoCTlM3EFARcin5oUAONdEc+KcVuZT8UCSRknCi P/xTi1hKfiiUTUk82yz0Cv90IZaSH4ql1BFQiX+aEUvJDwXzqkjJxz/ViKXk h6LJFYSU/+GfbsRS8nUKOM88xT99iKXk7xqCz0rJxz/ViKbka5yE4J9+xFLy FS7D4J8BxFLy9S1ELzLvEVAlYin56rbi8M8Ebm9GwD8beBUQ/4zgVED8s4JP AfHPDAgIl+JSQPyzg0cB8c8QDgXEP0v4ExD/TOFOQPyzhTcBlw89R0DlOBMQ /6zhS0D8M4drAfFPP64ExD97eBIQ/wziSED8swgCwqX4ERD/TOJGQPyziVMB 8c8KXgTEP6M4ERD/rOJDQPwzi0MB8c8SLgSkA7RLrAv14xWwlQ1hW2dHc+Cf YWJdKO75Wwzl2eFEDMCGEXBhKeDZ8Wz4Z5kDBDw5oBL/TCMoYJEV7fXeuSn5 +GcbOQGf3d3zeq8+O6SchyDZRk7AlrI6WUD8M46wgM2U+MyUfAZgywh5sBTw zEkIHaB1DhiCT1yGwT/ziAlYlXk/CTlxIZrHoJsn1oXstQPXPLq6ODklH//s Y/pmhFFA/LOKZQHpAB1gWED884ALAfHPLnYFxD8XOBAQ/yxjVkD884FVARmA nWBeQPyzjVEB8c8LxgXEP+vYFBD/3ICAcCkmBcQ/P1gUEP8cYVhA/POAQQHx zxMICJdiT0D8c4VVAfHPCeYExD9fWBOQAdgZNgXEPzcYExD/vGFRQPxzhC0B 8c8dcin5oUAO6WgOBHSHXEp+KJJIOJwI//whlo4VCmWTjmdrBMQ/X8jFswVi KYUDKvHPIWIChoJ5ZVPy8c8jYgKGosllQ8oR0CN2BMQ/l1wzBP+Sjv6Hf94Q Tck/ehKCfz4xswzzh38ukXtMw7EL0XSAThFLyT94K44O0ClGbkbAP6/YEBD/ 3IKAcCkmBMQ/v9gQEP/cYkFA/HOMCQGPaQVowICAdICesSDgQa0ADegXEP9c o15A/PONdgH/IaBvtAuIf85RLiD+eUe3gAzA7lEtIP75BwHhUjQLiH8JoFhA /EsBvQL+YwqcAooFxL8UUCsg/qWBVgEZgBNBqYD4lwoICJeiU0D8SwY5AYeM jvhoDvxLB0EBh1JsONE/7oJJhwMEjI5nowNMiAMEjA2oxL+UEBSwyIr2ei8y JZ8BOCnkBHx2d8/rvTo6pJwOMCmEl2HKKlZAOsC0EBaweXBcVEo+/iWEUEr+ lEbAmEkI/qXGAUNwzDIMF4CpISZgVeb9JCRiIZoOMDnEBKwfr2WY37fi8C89 VN2MwACcHpoExL8EUSQgA3CK6BEQ/5JEk4D4lyBqBMS/NNEiIANwougREP+S RImA+JcqOgTEv2RRISAXgOmiRED8SxUNAuJfwmgQEP8SBgHhUjQIiH8Jo0FA SBgEhEtBQLgUBIRLQUC4FASES0FAuBQEhEtBQLgUBIRLOdiFcDYHAsKLg10I pxMhILw41oWVfDYEhBfHurCSUCl4UO9VqWyU0qoCrGT06my/yqpUNkppVeu1 Lw+is/0qq1LZKKVVrdeOgApq8l9VgLUhGODFoQKuxeQDnMJaTD7AOazE5AOc w0pMPgAAAAAAAAAAADhHcG1QZDexfryqiG7ZWFVky5pnj1alSKMmVcWervaJ qF1dEq3qqzphS3iO4O6ISLOf7cmGUlzLxqoiW5ZV9a0uSolGTaqKPV3tM6Hr 6i7QqklVZ98fL7k/LNX2rh6RlskI2Hww3V1E0Y0aqxI6Xc3PmNCH2P64ni2g 5B0ysgKKtExGwLESodOVDV+iaWQWalX7c3G2gCv3CP5ENjylPbKe9qtIy14C CrSsLKQa1VUlc7racVOmVf0QLPQhbkWyi3j+ID6vROroerLJ18iW9d8t0LK8 /XaRRnVVCTSqmS5UMq16VSX2IW4+8OwPAcr40VxewJaoluXdMCfRqL6qjsjT lZfFXerHoq1KpFV7kByCW/J4l+WH4JaYlvWdlkSj8ln3En266kLsQ6yHCgQ+ xK2I36YvJqDkJKQlomWDNPGNmvsncLoyuQ9xaMuJAorfpi82BEsuw7T83rJR muhGLfyLP11NtyX0IY494IlDsOBCdLuaKTcJEWlZ35mWcS2rxo82fsl3LEWe rqrOmwu3WqJVQ1ViH+Jm5Lbi2v2c6KrGXw+MbtlQVWzL+opygUaNVUWfrq6C VhWRrbi2KpkPEQAAAAAAAAAAAAAAAAAAAAAAAL7xH5zeSIiF+vMJAAAAAElF TkSuQmCC ------=_Part_3807150_141659235.1436266613894 Content-Type: text/plain Content-Disposition: inline Content-Transfer-Encoding: 8bit MIME-Version: 1.0 -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance ------=_Part_3807150_141659235.1436266613894--