Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1ZCR8o-0006pf-TF for pgsql-performance@arkaria.postgresql.org; Tue, 07 Jul 2015 11:34:23 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.84) (envelope-from ) id 1ZCR8o-00065b-1p for pgsql-performance@arkaria.postgresql.org; Tue, 07 Jul 2015 11:34:22 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84) (envelope-from ) id 1ZCR3H-0008V4-F6 for pgsql-performance@postgresql.org; Tue, 07 Jul 2015 11:28:39 +0000 Received: from wsip-70-169-168-7.pn.at.cox.net ([70.169.168.7] helo=fs.denninger.net) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84) (envelope-from ) id 1ZCR3D-0008BY-Oq for pgsql-performance@postgresql.org; Tue, 07 Jul 2015 11:28:38 +0000 Received: from [192.168.1.40] (localhost [127.0.0.1]) by fs.denninger.net (8.14.9/8.14.8) with ESMTP id t67BSVuF050521 for ; Tue, 7 Jul 2015 06:28:31 -0500 (CDT) (envelope-from karl@denninger.net) Received: from [192.168.1.40] [192.168.1.40] (Via SSLv3 AES128-SHA) ; by Spamblock-sys (LOCAL/AUTH) Tue Jul 7 06:28:31 2015 Message-ID: <559BB7D8.6090701@denninger.net> Date: Tue, 07 Jul 2015 06:28:24 -0500 From: Karl Denninger User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: pgsql-performance@postgresql.org Subject: Re: New server: SSD/RAID recommendations? References: <31701C89A44B714FB8D242F806E7286C5AED32FA@NTXBOIMBX01.micron.com> <559AB332.50304@pinpointresearch.com> <52A08B87-2A8A-43F2-92CE-2514F5AD141B@skogoglandskap.no> <1253229687.3798367.1436264898828.JavaMail.zimbra@desy.de> <985C5CEF-2E55-46AD-A29F-90D48D057A3F@skogoglandskap.no> <1052663055.3807152.1436266613895.JavaMail.zimbra@desy.de> In-Reply-To: <1052663055.3807152.1436266613895.JavaMail.zimbra@desy.de> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms090201090609000303080401" X-Pg-Spam-Score: -0.9 (/) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-performance Precedence: bulk Sender: pgsql-performance-owner@postgresql.org This is a cryptographically signed message in MIME format. --------------ms090201090609000303080401 Content-Type: multipart/alternative; boundary="------------060000010500000107060706" This is a multi-part message in MIME format. --------------060000010500000107060706 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 7/7/2015 05:56, Mkrtchyan, Tigran wrote: > > ----- Original Message ----- >> From: "Graeme B. Bell" >> To: "Mkrtchyan, Tigran" >> Cc: "Graeme B. Bell" , "Steve Crawford" , "Wes Vaske (wvaske)" >> , "pgsql-performance" >> Sent: Tuesday, July 7, 2015 12:38:10 PM >> Subject: Re: [PERFORM] New server: SSD/RAID recommendations? >> I am unsure about the performance side but, ZFS is generally very attr= active to >> me. >> >> Key advantages: >> >> 1) Checksumming and automatic fixing-of-broken-things on every file (n= ot just >> postgres pages, but your scripts, O/S, program files). >> 2) Built-in lightweight compression (doesn't help with TOAST tables, = in fact >> may slow them down, but helpful for other things). This may actually b= e a net >> negative for pg so maybe turn it off. >> 3) ZRAID mirroring or ZRAID5/6. If you have trouble persuading someone= that it's >> safe to replace a RAID array with a single drive... you can use a coup= le of >> NVMe SSDs with ZFS mirror or zraid, and get the same availability you= 'd get >> from a RAID controller. Slightly better, arguably, since they claim to= have >> fixed the raid write-hole problem. >> 4) filesystem snapshotting >> >> Despite the costs of checksumming etc., I suspect ZRAID running on a f= ast CPU >> with multiple NVMe drives will outperform quite a lot of the alternati= ves, with >> great data integrity guarantees. Lz4 compression and standard 128kb block size has shown to be materially faster here than using 8kb blocks and no compression, both with rotating disks and SSDs. This is workload dependent in my experience but in the applications we put Postgres to there is a very material improvement in throughput using compression and the larger blocksize, which is counter-intuitive and also opposite the "conventional wisdom." For best throughput we use mirrored vdev sets. --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------060000010500000107060706 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 7/7/2015 05:56, Mkrtchyan, Tigran wrote:

----- Original Message -----

From: "Graeme B. Bell" <graeme.bell@nibio.no>=

To: "Mkrtchyan, Tigran" <tigran.mkrtchyan@desy.de>
Cc: "Graeme B. Bell" <graeme.bell@nibio.no>, "Steve Crawford" <scrawford@pinpointresearch.com>, "Wes Vaske (wvaske)"
<=
wvaske@micron.com>, "pgsql-performance" <pgsql-perfo=
rmance@postgresql.org>
Sent: Tuesday, July 7, 2015 12:38:10 PM
Subject: Re: [PERFORM] New server: SSD/RAID recommendations?


      

        
I am unsure about the performance side but, ZFS is=
 generally very attractive to
me.

Key advantages:

1) Checksumming and automatic fixing-of-broken-things on every file (not =
just
postgres pages, but your scripts, O/S, program files).
2) Built-in  lightweight compression (doesn't help with TOAST tables, in =
fact
may slow them down, but helpful for other things). This may actually be a=
 net
negative for pg so maybe turn it off.
3) ZRAID mirroring or ZRAID5/6. If you have trouble persuading someone th=
at it's
safe to replace a RAID array with a single drive... you can use a couple =
of
NVMe SSDs with ZFS mirror or zraid, and  get the same availability you'd =
get
from a RAID controller. Slightly better, arguably, since they claim to ha=
ve
fixed the raid write-hole problem.
4) filesystem snapshotting

Despite the costs of checksumming etc., I suspect ZRAID running on a fast=
 CPU
with multiple NVMe drives will outperform quite a lot of the alternatives=
, with
great data integrity guarantees.


      

    

    Lz4 compression and standard 128kb block size has shown to be
    materially faster here than using 8kb blocks and no compression,
    both with rotating disks and SSDs.

    

    This is workload dependent in my experience but in the applications
    we put Postgres to there is a very material improvement in
    throughput using compression and the larger blocksize, which is
    counter-intuitive and also opposite the "conventional wisdom."

    

    For best throughput we use mirrored vdev sets.

    

    
-- 

      Karl Denninger

      karl@denninger.net

      The Market Ticker

      [S/MIME encrypted email preferred]
    

  


--------------060000010500000107060706--

--------------ms090201090609000303080401
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIGXzCC
BlswggRDoAMCAQICASkwDQYJKoZIhvcNAQELBQAwgZAxCzAJBgNVBAYTAlVTMRAwDgYDVQQI
EwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM
TEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG9w0BCQEWE0N1ZGEg
U3lzdGVtcyBMTEMgQ0EwHhcNMTUwNDIxMDIyMTU5WhcNMjAwNDE5MDIyMTU5WjBaMQswCQYD
VQQGEwJVUzEQMA4GA1UECBMHRmxvcmlkYTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExMQzEe
MBwGA1UEAxMVS2FybCBEZW5uaW5nZXIgKE9DU1ApMIICIjANBgkqhkiG9w0BAQEFAAOCAg8A
MIICCgKCAgEAuYRY+EB2mGtZ3grlVO8TmnEvduVFA/IYXcCmNSOC1q+pTVjylsjcHKBcOPb9
TP1KLxdWP+Q1soSORGHlKw2/HcVzShDW5WPIKrvML+Ry0XvIvNBu9adTiCsA9nci4Cnf98XE
hVpenER0qbJkBUOGT1rP4iAcfjet0lEgzPEnm+pAxv6fYSNp1WqIY9u0b1pkQiaWrt8hgNOc
rJOiLbc8CeQ/DBP6rUiQjYNO9/aPNauEtHkNNfR9RgLSfGUdZuOCmJqnIla1HsrZhA5p69Bv
/e832BKiNPaH5wF6btAiPpTr2sRhwQO8/IIxcRX1Vxd1yZbjYtJGw+9lwEcWRYAmoxkzKLPi
S6Zo/6z5wgNpeK1H+zOioMoZIczgI8BlX1iHxqy/FAvm4PHPnC8s+BLnJLwr+jvMNHm82QwL
J9hC5Ho8AnFU6TkCuq+P2V8/clJVqnBuvTUKhYMGSm4mUp+lAgR4L+lwIEqSeWVsxirIcE7Z
OKkvI7k5x3WeE3+c6w74L6PfWVAd84xFlo9DKRdU9YbkFuFZPu21fi/LmE5brImB5P+jdqnK
eWnVwRq+RBFLy4kehCzMXooitAwgP8l/JJa9VDiSyd/PAHaVGiat2vCdDh4b8cFL7SV6jPA4
k0MgGUA/6Et7wDmhZmCigggr9K6VQCx8jpKB3x1NlNNiaWECAwEAAaOB9DCB8TA3BggrBgEF
BQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNV
HRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIFoDALBgNVHQ8EBAMCBeAwLAYJYIZIAYb4QgENBB8W
HU9wZW5TU0wgR2VuZXJhdGVkIENlcnRpZmljYXRlMB0GA1UdDgQWBBTFHJQt6cloXBdG1Pv1
o2YgH+7lWTAfBgNVHSMEGDAWgBQkcZudhX383d29sMqSlAOh+tNtNTAdBgNVHREEFjAUgRJr
YXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcNAQELBQADggIBAE9/dxi2YqjCYYhiybp4GKcm
7tBVa/GLW+qcHPcoT4dqmqghlLz8+iUH+HCJjRQATVGyMEnvISOKFVHC6aZIG+Sg7J8bfS4+
fjKDi9smRH2VPPx3bV8+yFYRNroMGHaPHZB/Xctmmvc+PZ9O2W7rExgrODtxIOB3Zs6wkYf+
ty+9r1KmTHlV+rRHI6timH1uiyFE3cPi1taAEBxf0851cJV8k40PGF8G48ewnq8SY9sCf5cv
liXbpdgU+I4ND5BuTjg63WS32zuhLd1VSuH3ZC/QbcncMX5W3oLXmcQP5/5uTiBJy74kdPtG
MSZ9rXwZPwNxP/8PXMSR7ViaFvjUkf4bJlyENFa2PGxLk4EUzOuO7t3brjMlQW1fuInfG+ko
3tVxko20Hp0tKGPe/9cOxBVBZeZH/VgpZn3cLculGzZjmdh2fqAQ6kv9Z9AVOG1+dq0c1zt8
2zm+Oi1pikGXkfz5UJq60psY6zbX25BuEZkthO/qiS4pxjxb7gQkS0rTEHTy+qv0l3QVL0wa
NAT74Zaj7l5DEW3qdQQ0dtVieyvptg9CxkfQJE3JyBMb0zBj9Qhc5/hbTfhSlHzZMEbUuIyx
h9vxqFAmGzfB1/WfOKkiNHChkpPW8ZeH9yPeDBKvrgZ96dREHFoVkDk7Vpw5lSM+tFOfdyLg
xxhb/RZVUDeUMYIE4zCCBN8CAQEwgZYwgZAxCzAJBgNVBAYTAlVTMRAwDgYDVQQIEwdGbG9y
aWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBMTEMxHDAa
BgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG9w0BCQEWE0N1ZGEgU3lzdGVt
cyBMTEMgQ0ECASkwCQYFKw4DAhoFAKCCAiEwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAc
BgkqhkiG9w0BCQUxDxcNMTUwNzA3MTEyODI0WjAjBgkqhkiG9w0BCQQxFgQUVmwVRX+Izwhj
qJ0H3bDWkRkAWEowbAYJKoZIhvcNAQkPMV8wXTALBglghkgBZQMEASowCwYJYIZIAWUDBAEC
MAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzAN
BggqhkiG9w0DAgIBKDCBpwYJKwYBBAGCNxAEMYGZMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4G
A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl
bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND
dWRhIFN5c3RlbXMgTExDIENBAgEpMIGpBgsqhkiG9w0BCRACCzGBmaCBljCBkDELMAkGA1UE
BhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQ
Q3VkYSBTeXN0ZW1zIExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqG
SIb3DQEJARYTQ3VkYSBTeXN0ZW1zIExMQyBDQQIBKTANBgkqhkiG9w0BAQEFAASCAgCbh9Od
MpjYXrzSJfO6MN0A98tu2XPtszZDhA7JTjVzsPxD92KVR95/r1hKiqh+96P6cFRSCXrr8H/b
IQElLLcohS/uDKQj8RyDOI5CZHzD/CR0BAN+nN3pN34k0rEttmfDEpgI4fki9+HCXGGYgKMw
gSuLHDeK8Ks53t0zVB0me9BFFrYJDz2AwhzfOX1V4fnX+FA6VPcMKCVl91ng/QhUGll/O2U8
u4Jv+mdTZ4KLvXORBqWN58FQRfjKsAC1Ify0Pk/L2oO6ZrrSenLPeKWMkGCJ6q09Jj6WB4mc
zyMkxhHBw2DvSLd8F1J/QndSLPbiUQBwWg05Jf+vkBgaMocuZZqWLQSIDWvRUgqluO7+CkZE
1YlE2oOBIcRJA3BtDG4NQmJ5Z3T5skFfd1egraLx7pAsNX2TuN79exDT1VZx1xDv7bzcJTVx
pTh/iM+ETxr42JtxDyM0lIB1FRpdh1A7DH/esQu/Ui9GgPZEGYvauye+2Z9HhcW8n48lHpHS
d/I7yGEBp1+8Y0fgZxC2N+QYU3L7RtTyRbDC+/3gUP9UqYxMZ7ySXv8CkpcaoRn97/kOrExR
bjhoaY0VOI7wqexpD+Lem+AdvjS8OPupC2VuFyVNhdkdQRan1uzoci4vyJUcshf7HWNzlf5z
/3FSoD+Uv7yKw4qKnjLamp9LYeoKawAAAAAAAA==
--------------ms090201090609000303080401--