public inbox for [email protected]help / color / mirror / Atom feed
Testing in AWS, EBS 15+ messages / 4 participants [nested] [flat]
* Testing in AWS, EBS @ 2016-05-25 22:34 Tory M Blue <[email protected]> 0 siblings, 2 replies; 15+ messages in thread From: Tory M Blue @ 2016-05-25 22:34 UTC (permalink / raw) To: pgsql-performance We are starting some testing in AWS, with EC2, EBS backed setups. What I found interesting today, was a single EBS 1TB volume, gave me something like 108MB/s throughput, however a RAID10 (4 250GB EBS volumes), gave me something like 31MB/s (test after test after test). I'm wondering what you folks are using inside of Amazon (not interested in RDS at the moment). Thanks Tory -- Sent via pgsql-performance mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance ^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Testing in AWS, EBS @ 2016-05-25 23:02 Rayson Ho <[email protected]> parent: Tory M Blue <[email protected]> 1 sibling, 1 reply; 15+ messages in thread From: Rayson Ho @ 2016-05-25 23:02 UTC (permalink / raw) To: Tory M Blue <[email protected]>; +Cc: pgsql-performance There are many factors that can affect EBS performance. For example, the type of EBS volume, the instance type, whether EBS-optimized is turned on or not, etc. Without the details, then there is no apples to apples comparsion... Rayson ================================================== Open Grid Scheduler - The Official Open Source Grid Engine http://gridscheduler.sourceforge.net/ http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html On Wed, May 25, 2016 at 6:34 PM, Tory M Blue <[email protected]> wrote: > > We are starting some testing in AWS, with EC2, EBS backed setups. > > What I found interesting today, was a single EBS 1TB volume, gave me > something like 108MB/s throughput, however a RAID10 (4 250GB EBS > volumes), gave me something like 31MB/s (test after test after test). > > I'm wondering what you folks are using inside of Amazon (not > interested in RDS at the moment). > > Thanks > Tory > > > -- > Sent via pgsql-performance mailing list ([email protected]) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-performance ^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Testing in AWS, EBS @ 2016-05-25 23:56 Yves Dorfsman <[email protected]> parent: Rayson Ho <[email protected]> 0 siblings, 1 reply; 15+ messages in thread From: Yves Dorfsman @ 2016-05-25 23:56 UTC (permalink / raw) To: pgsql-performance Indeed, old-style disk EBS vs new-style SSd EBS. Be aware that EBS traffic is considered as part of the total "network" traffic, and each type of instance has different limits on maximum network throughput. Those difference are very significant, do tests on the same volume between two different type of instances, both with enough cpu and memory for the I/O to be the bottleneck, you will be surprised! On 2016-05-25 17:02, Rayson Ho wrote: > There are many factors that can affect EBS performance. For example, the type > of EBS volume, the instance type, whether EBS-optimized is turned on or not, etc. > > Without the details, then there is no apples to apples comparsion... > > Rayson > > ================================================== > Open Grid Scheduler - The Official Open Source Grid Engine > http://gridscheduler.sourceforge.net/ > http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html > > > > On Wed, May 25, 2016 at 6:34 PM, Tory M Blue <[email protected] > <mailto:[email protected]>> wrote: >> >> We are starting some testing in AWS, with EC2, EBS backed setups. >> >> What I found interesting today, was a single EBS 1TB volume, gave me >> something like 108MB/s throughput, however a RAID10 (4 250GB EBS >> volumes), gave me something like 31MB/s (test after test after test). >> >> I'm wondering what you folks are using inside of Amazon (not >> interested in RDS at the moment). >> >> Thanks >> Tory >> >> >> -- >> Sent via pgsql-performance mailing list ([email protected] > <mailto:[email protected]>) >> To make changes to your subscription: >> http://www.postgresql.org/mailpref/pgsql-performance -- http://yves.zioup.com gpg: 4096R/32B0F416 -- Sent via pgsql-performance mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance ^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Testing in AWS, EBS @ 2016-05-26 01:08 Rayson Ho <[email protected]> parent: Yves Dorfsman <[email protected]> 0 siblings, 1 reply; 15+ messages in thread From: Rayson Ho @ 2016-05-26 01:08 UTC (permalink / raw) To: Yves Dorfsman <[email protected]>; +Cc: pgsql-performance Actually, when "EBS-Optimized" is on, then the instance gets dedicated bandwidth to EBS. Rayson ================================================== Open Grid Scheduler - The Official Open Source Grid Engine http://gridscheduler.sourceforge.net/ http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html On Wed, May 25, 2016 at 7:56 PM, Yves Dorfsman <[email protected]> wrote: > Indeed, old-style disk EBS vs new-style SSd EBS. > > Be aware that EBS traffic is considered as part of the total "network" > traffic, and each type of instance has different limits on maximum network > throughput. Those difference are very significant, do tests on the same > volume > between two different type of instances, both with enough cpu and memory > for > the I/O to be the bottleneck, you will be surprised! > > > On 2016-05-25 17:02, Rayson Ho wrote: > > There are many factors that can affect EBS performance. For example, the > type > > of EBS volume, the instance type, whether EBS-optimized is turned on or > not, etc. > > > > Without the details, then there is no apples to apples comparsion... > > > > Rayson > > > > ================================================== > > Open Grid Scheduler - The Official Open Source Grid Engine > > http://gridscheduler.sourceforge.net/ > > http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html > > > > > > > > On Wed, May 25, 2016 at 6:34 PM, Tory M Blue <[email protected] > > <mailto:[email protected]>> wrote: > >> > >> We are starting some testing in AWS, with EC2, EBS backed setups. > >> > >> What I found interesting today, was a single EBS 1TB volume, gave me > >> something like 108MB/s throughput, however a RAID10 (4 250GB EBS > >> volumes), gave me something like 31MB/s (test after test after test). > >> > >> I'm wondering what you folks are using inside of Amazon (not > >> interested in RDS at the moment). > >> > >> Thanks > >> Tory > >> > >> > >> -- > >> Sent via pgsql-performance mailing list ( > [email protected] > > <mailto:[email protected]>) > >> To make changes to your subscription: > >> http://www.postgresql.org/mailpref/pgsql-performance > > > -- > http://yves.zioup.com > gpg: 4096R/32B0F416 > > > > -- > Sent via pgsql-performance mailing list ([email protected]) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-performance > ^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Testing in AWS, EBS @ 2016-05-26 08:41 Artem Tomyuk <[email protected]> parent: Tory M Blue <[email protected]> 1 sibling, 0 replies; 15+ messages in thread From: Artem Tomyuk @ 2016-05-26 08:41 UTC (permalink / raw) To: Tory M Blue <[email protected]>; +Cc: pgsql-performance Hi. AWS EBS its a really painful story.... How was created volumes for RAID? From snapshots? If you want to get the best performance from EBS it needs to pre-warmed. Here is the tutorial how to achieve that: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html Also you should read this one if you want to get really great for performance: http://hatim.eu/2014/05/24/leveraging-ssd-ephemeral-disks-in-ec2-part-1/ Good luck! 2016-05-26 1:34 GMT+03:00 Tory M Blue <[email protected]>: > We are starting some testing in AWS, with EC2, EBS backed setups. > > What I found interesting today, was a single EBS 1TB volume, gave me > something like 108MB/s throughput, however a RAID10 (4 250GB EBS > volumes), gave me something like 31MB/s (test after test after test). > > I'm wondering what you folks are using inside of Amazon (not > interested in RDS at the moment). > > Thanks > Tory > > > -- > Sent via pgsql-performance mailing list ([email protected]) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-performance > ^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Testing in AWS, EBS @ 2016-05-26 12:53 Yves Dorfsman <[email protected]> parent: Rayson Ho <[email protected]> 0 siblings, 1 reply; 15+ messages in thread From: Yves Dorfsman @ 2016-05-26 12:53 UTC (permalink / raw) To: pgsql-performance On 2016-05-25 19:08, Rayson Ho wrote: > Actually, when "EBS-Optimized" is on, then the instance gets dedicated > bandwidth to EBS. Hadn't realised that, thanks. Is the EBS bandwidth then somewhat limited depending on the type of instance too? -- http://yves.zioup.com gpg: 4096R/32B0F416 -- Sent via pgsql-performance mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance ^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Testing in AWS, EBS @ 2016-05-26 13:00 Artem Tomyuk <[email protected]> parent: Yves Dorfsman <[email protected]> 0 siblings, 1 reply; 15+ messages in thread From: Artem Tomyuk @ 2016-05-26 13:00 UTC (permalink / raw) To: Yves Dorfsman <[email protected]>; +Cc: pgsql-performance Yes, the smaller instance you choose - the slower ebs will be. EBS lives separately from EC2, they are communicating via network. So small instance = low network bandwidth = poorer disk performance. But still strong recommendation to pre-warm your ebs in any case, especially if they created from snapshot. 2016-05-26 15:53 GMT+03:00 Yves Dorfsman <[email protected]>: > On 2016-05-25 19:08, Rayson Ho wrote: > > Actually, when "EBS-Optimized" is on, then the instance gets dedicated > > bandwidth to EBS. > > Hadn't realised that, thanks. > Is the EBS bandwidth then somewhat limited depending on the type of > instance too? > > -- > http://yves.zioup.com > gpg: 4096R/32B0F416 > > > > -- > Sent via pgsql-performance mailing list ([email protected]) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-performance > ^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Testing in AWS, EBS @ 2016-05-26 13:50 Rayson Ho <[email protected]> parent: Artem Tomyuk <[email protected]> 0 siblings, 1 reply; 15+ messages in thread From: Rayson Ho @ 2016-05-26 13:50 UTC (permalink / raw) To: Artem Tomyuk <[email protected]>; +Cc: Yves Dorfsman <[email protected]>; pgsql-performance On Thu, May 26, 2016 at 9:00 AM, Artem Tomyuk <[email protected]> wrote: > > But still strong recommendation to pre-warm your ebs in any case, especially if they created from snapshot. That used to be true. However, at AWS re:Invent 2015, Amazon engineers said that EBS pre-warming is not needed anymore. Rayson ================================================== Open Grid Scheduler - The Official Open Source Grid Engine http://gridscheduler.sourceforge.net/ http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html > 2016-05-26 15:53 GMT+03:00 Yves Dorfsman <[email protected]>: >> >> On 2016-05-25 19:08, Rayson Ho wrote: >> > Actually, when "EBS-Optimized" is on, then the instance gets dedicated >> > bandwidth to EBS. >> >> Hadn't realised that, thanks. >> Is the EBS bandwidth then somewhat limited depending on the type of instance too? >> >> -- >> http://yves.zioup.com >> gpg: 4096R/32B0F416 >> >> >> >> -- >> Sent via pgsql-performance mailing list ([email protected] ) >> To make changes to your subscription: >> http://www.postgresql.org/mailpref/pgsql-performance > > ^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Testing in AWS, EBS @ 2016-05-26 14:00 Artem Tomyuk <[email protected]> parent: Rayson Ho <[email protected]> 0 siblings, 1 reply; 15+ messages in thread From: Artem Tomyuk @ 2016-05-26 14:00 UTC (permalink / raw) To: Rayson Ho <[email protected]>; +Cc: Yves Dorfsman <[email protected]>; pgsql-performance 2016-05-26 16:50 GMT+03:00 Rayson Ho <[email protected]>: > Amazon engineers said that EBS pre-warming is not needed anymore. but still if you will skip this step you wont get much performance on ebs created from snapshot. ^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Testing in AWS, EBS @ 2016-05-26 14:47 Rayson Ho <[email protected]> parent: Artem Tomyuk <[email protected]> 0 siblings, 1 reply; 15+ messages in thread From: Rayson Ho @ 2016-05-26 14:47 UTC (permalink / raw) To: Artem Tomyuk <[email protected]>; +Cc: Yves Dorfsman <[email protected]>; pgsql-performance On Thu, May 26, 2016 at 10:00 AM, Artem Tomyuk <[email protected]> wrote: > > 2016-05-26 16:50 GMT+03:00 Rayson Ho <[email protected]>: > >> Amazon engineers said that EBS pre-warming is not needed anymore. > > > but still if you will skip this step you wont get much performance on ebs > created from snapshot. > IIRC, that's not what Amazon engineers said. Is that from your personal experience, and if so, when did you do the test?? Rayson ================================================== Open Grid Scheduler - The Official Open Source Grid Engine http://gridscheduler.sourceforge.net/ http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html ^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Testing in AWS, EBS @ 2016-05-26 14:52 Artem Tomyuk <[email protected]> parent: Rayson Ho <[email protected]> 0 siblings, 1 reply; 15+ messages in thread From: Artem Tomyuk @ 2016-05-26 14:52 UTC (permalink / raw) To: Rayson Ho <[email protected]>; +Cc: Yves Dorfsman <[email protected]>; pgsql-performance Please look at the official doc. "New EBS volumes receive their maximum performance the moment that they are available and do not require initialization (formerly known as pre-warming). However, storage blocks on volumes that were restored from snapshots must be initialized (pulled down from Amazon S3 and written to the volume) before you can access the block" Quotation from: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html 2016-05-26 17:47 GMT+03:00 Rayson Ho <[email protected]>: > On Thu, May 26, 2016 at 10:00 AM, Artem Tomyuk <[email protected]> > wrote: > >> >> 2016-05-26 16:50 GMT+03:00 Rayson Ho <[email protected]>: >> >>> Amazon engineers said that EBS pre-warming is not needed anymore. >> >> >> but still if you will skip this step you wont get much performance on ebs >> created from snapshot. >> > > > IIRC, that's not what Amazon engineers said. Is that from your personal > experience, and if so, when did you do the test?? > > Rayson > > ================================================== > Open Grid Scheduler - The Official Open Source Grid Engine > http://gridscheduler.sourceforge.net/ > http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html > > > > ^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Testing in AWS, EBS @ 2016-05-26 14:54 Rayson Ho <[email protected]> parent: Artem Tomyuk <[email protected]> 0 siblings, 1 reply; 15+ messages in thread From: Rayson Ho @ 2016-05-26 14:54 UTC (permalink / raw) To: Artem Tomyuk <[email protected]>; +Cc: Yves Dorfsman <[email protected]>; pgsql-performance Thanks Artem. So no EBS pre-warming does not apply to EBS volumes created from snapshots. Rayson ================================================== Open Grid Scheduler - The Official Open Source Grid Engine http://gridscheduler.sourceforge.net/ http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html On Thu, May 26, 2016 at 10:52 AM, Artem Tomyuk <[email protected]> wrote: > Please look at the official doc. > > "New EBS volumes receive their maximum performance the moment that they are > available and do not require initialization (formerly known as pre-warming). > However, storage blocks on volumes that were restored from snapshots must be > initialized (pulled down from Amazon S3 and written to the volume) before > you can access the block" > > Quotation from: > http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html > > 2016-05-26 17:47 GMT+03:00 Rayson Ho <[email protected]>: >> >> On Thu, May 26, 2016 at 10:00 AM, Artem Tomyuk <[email protected]> >> wrote: >>> >>> >>> 2016-05-26 16:50 GMT+03:00 Rayson Ho <[email protected]>: >>>> >>>> Amazon engineers said that EBS pre-warming is not needed anymore. >>> >>> >>> but still if you will skip this step you wont get much performance on ebs >>> created from snapshot. >> >> >> >> IIRC, that's not what Amazon engineers said. Is that from your personal >> experience, and if so, when did you do the test?? >> >> Rayson >> >> ================================================== >> Open Grid Scheduler - The Official Open Source Grid Engine >> http://gridscheduler.sourceforge.net/ >> http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html >> >> >> > -- Sent via pgsql-performance mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance ^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Testing in AWS, EBS @ 2016-05-26 15:03 Artem Tomyuk <[email protected]> parent: Rayson Ho <[email protected]> 0 siblings, 1 reply; 15+ messages in thread From: Artem Tomyuk @ 2016-05-26 15:03 UTC (permalink / raw) To: Rayson Ho <[email protected]>; +Cc: Yves Dorfsman <[email protected]>; pgsql-performance Why no? Or you missed something? It should be done on every EBS restored from snapshot. Is that from your personal experience, and if so, when did you do the test?? Yes we are using this practice, because as a part of our production load we are using auto scale groups to create new instances, wheech are created from AMI, wheech stands on snapshots, so... 2016-05-26 17:54 GMT+03:00 Rayson Ho <[email protected]>: > Thanks Artem. > > So no EBS pre-warming does not apply to EBS volumes created from snapshots. > > Rayson > > ================================================== > Open Grid Scheduler - The Official Open Source Grid Engine > http://gridscheduler.sourceforge.net/ > http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html > > > On Thu, May 26, 2016 at 10:52 AM, Artem Tomyuk <[email protected]> > wrote: > > Please look at the official doc. > > > > "New EBS volumes receive their maximum performance the moment that they > are > > available and do not require initialization (formerly known as > pre-warming). > > However, storage blocks on volumes that were restored from snapshots > must be > > initialized (pulled down from Amazon S3 and written to the volume) before > > you can access the block" > > > > Quotation from: > > http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html > > > > 2016-05-26 17:47 GMT+03:00 Rayson Ho <[email protected]>: > >> > >> On Thu, May 26, 2016 at 10:00 AM, Artem Tomyuk <[email protected]> > >> wrote: > >>> > >>> > >>> 2016-05-26 16:50 GMT+03:00 Rayson Ho <[email protected]>: > >>>> > >>>> Amazon engineers said that EBS pre-warming is not needed anymore. > >>> > >>> > >>> but still if you will skip this step you wont get much performance on > ebs > >>> created from snapshot. > >> > >> > >> > >> IIRC, that's not what Amazon engineers said. Is that from your personal > >> experience, and if so, when did you do the test?? > >> > >> Rayson > >> > >> ================================================== > >> Open Grid Scheduler - The Official Open Source Grid Engine > >> http://gridscheduler.sourceforge.net/ > >> http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html > >> > >> > >> > > > ^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Testing in AWS, EBS @ 2016-05-26 15:41 Yves Dorfsman <[email protected]> parent: Artem Tomyuk <[email protected]> 0 siblings, 1 reply; 15+ messages in thread From: Yves Dorfsman @ 2016-05-26 15:41 UTC (permalink / raw) To: pgsql-performance On 2016-05-26 09:03, Artem Tomyuk wrote: > Why no? Or you missed something? I think Rayson is correct, but the double negative makes it hard to read: "So no EBS pre-warming does not apply to EBS volumes created from snapshots." Which I interpret as: So, "no EBS pre-warming", does not apply to EBS volumes created from snapshots. Which is correct, you sitll have to warm your EBS when created from sanpshots (to get the data from S3 to the filesystem). -- http://yves.zioup.com gpg: 4096R/32B0F416 -- Sent via pgsql-performance mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance ^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Testing in AWS, EBS @ 2016-05-26 21:26 Rayson Ho <[email protected]> parent: Yves Dorfsman <[email protected]> 0 siblings, 0 replies; 15+ messages in thread From: Rayson Ho @ 2016-05-26 21:26 UTC (permalink / raw) To: Yves Dorfsman <[email protected]>; +Cc: pgsql-performance Thanks Yves for the clarification! It used to be very important to pre-warm EBS before running benchmarks in order to get consistent results. Then at re:Invent 2015, the AWS engineers said that it is not needed anymore, which IMO is a lot less work for us to do benchmarking in AWS, because pre-warming a multi-TB EBS vol is very time consuming, and the I/Os were not free. Rayson ================================================== Open Grid Scheduler - The Official Open Source Grid Engine http://gridscheduler.sourceforge.net/ http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html On Thu, May 26, 2016 at 11:41 AM, Yves Dorfsman <[email protected]> wrote: > On 2016-05-26 09:03, Artem Tomyuk wrote: >> Why no? Or you missed something? > > I think Rayson is correct, but the double negative makes it hard to read: > > "So no EBS pre-warming does not apply to EBS volumes created from snapshots." > > Which I interpret as: > So, "no EBS pre-warming", does not apply to EBS volumes created from snapshots. > > Which is correct, you sitll have to warm your EBS when created from sanpshots (to get the data from S3 to the filesystem). > > > -- > http://yves.zioup.com > gpg: 4096R/32B0F416 > > > > -- > Sent via pgsql-performance mailing list ([email protected]) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-performance -- Sent via pgsql-performance mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance ^ permalink raw reply [nested|flat] 15+ messages in thread
end of thread, other threads:[~2016-05-26 21:26 UTC | newest] Thread overview: 15+ messages (download: mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2016-05-25 22:34 Testing in AWS, EBS Tory M Blue <[email protected]> 2016-05-25 23:02 ` Rayson Ho <[email protected]> 2016-05-25 23:56 ` Yves Dorfsman <[email protected]> 2016-05-26 01:08 ` Rayson Ho <[email protected]> 2016-05-26 12:53 ` Yves Dorfsman <[email protected]> 2016-05-26 13:00 ` Artem Tomyuk <[email protected]> 2016-05-26 13:50 ` Rayson Ho <[email protected]> 2016-05-26 14:00 ` Artem Tomyuk <[email protected]> 2016-05-26 14:47 ` Rayson Ho <[email protected]> 2016-05-26 14:52 ` Artem Tomyuk <[email protected]> 2016-05-26 14:54 ` Rayson Ho <[email protected]> 2016-05-26 15:03 ` Artem Tomyuk <[email protected]> 2016-05-26 15:41 ` Yves Dorfsman <[email protected]> 2016-05-26 21:26 ` Rayson Ho <[email protected]> 2016-05-26 08:41 ` Artem Tomyuk <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox