public inbox for [email protected]
help / color / mirror / Atom feedTesting in AWS, EBS
15+ messages / 4 participants
[nested] [flat]
* Testing in AWS, EBS
@ 2016-05-25 22:34 Tory M Blue <[email protected]>
2016-05-25 23:02 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-26 08:41 ` Re: Testing in AWS, EBS Artem Tomyuk <[email protected]>
0 siblings, 2 replies; 15+ messages in thread
From: Tory M Blue @ 2016-05-25 22:34 UTC (permalink / raw)
To: pgsql-performance
We are starting some testing in AWS, with EC2, EBS backed setups.
What I found interesting today, was a single EBS 1TB volume, gave me
something like 108MB/s throughput, however a RAID10 (4 250GB EBS
volumes), gave me something like 31MB/s (test after test after test).
I'm wondering what you folks are using inside of Amazon (not
interested in RDS at the moment).
Thanks
Tory
--
Sent via pgsql-performance mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Testing in AWS, EBS
2016-05-25 22:34 Testing in AWS, EBS Tory M Blue <[email protected]>
@ 2016-05-25 23:02 ` Rayson Ho <[email protected]>
2016-05-25 23:56 ` Re: Testing in AWS, EBS Yves Dorfsman <[email protected]>
1 sibling, 1 reply; 15+ messages in thread
From: Rayson Ho @ 2016-05-25 23:02 UTC (permalink / raw)
To: Tory M Blue <[email protected]>; +Cc: pgsql-performance
There are many factors that can affect EBS performance. For example, the
type of EBS volume, the instance type, whether EBS-optimized is turned on
or not, etc.
Without the details, then there is no apples to apples comparsion...
Rayson
==================================================
Open Grid Scheduler - The Official Open Source Grid Engine
http://gridscheduler.sourceforge.net/
http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html
On Wed, May 25, 2016 at 6:34 PM, Tory M Blue <[email protected]> wrote:
>
> We are starting some testing in AWS, with EC2, EBS backed setups.
>
> What I found interesting today, was a single EBS 1TB volume, gave me
> something like 108MB/s throughput, however a RAID10 (4 250GB EBS
> volumes), gave me something like 31MB/s (test after test after test).
>
> I'm wondering what you folks are using inside of Amazon (not
> interested in RDS at the moment).
>
> Thanks
> Tory
>
>
> --
> Sent via pgsql-performance mailing list ([email protected])
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Testing in AWS, EBS
2016-05-25 22:34 Testing in AWS, EBS Tory M Blue <[email protected]>
2016-05-25 23:02 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
@ 2016-05-25 23:56 ` Yves Dorfsman <[email protected]>
2016-05-26 01:08 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
0 siblings, 1 reply; 15+ messages in thread
From: Yves Dorfsman @ 2016-05-25 23:56 UTC (permalink / raw)
To: pgsql-performance
Indeed, old-style disk EBS vs new-style SSd EBS.
Be aware that EBS traffic is considered as part of the total "network"
traffic, and each type of instance has different limits on maximum network
throughput. Those difference are very significant, do tests on the same volume
between two different type of instances, both with enough cpu and memory for
the I/O to be the bottleneck, you will be surprised!
On 2016-05-25 17:02, Rayson Ho wrote:
> There are many factors that can affect EBS performance. For example, the type
> of EBS volume, the instance type, whether EBS-optimized is turned on or not, etc.
>
> Without the details, then there is no apples to apples comparsion...
>
> Rayson
>
> ==================================================
> Open Grid Scheduler - The Official Open Source Grid Engine
> http://gridscheduler.sourceforge.net/
> http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html
>
>
>
> On Wed, May 25, 2016 at 6:34 PM, Tory M Blue <[email protected]
> <mailto:[email protected]>> wrote:
>>
>> We are starting some testing in AWS, with EC2, EBS backed setups.
>>
>> What I found interesting today, was a single EBS 1TB volume, gave me
>> something like 108MB/s throughput, however a RAID10 (4 250GB EBS
>> volumes), gave me something like 31MB/s (test after test after test).
>>
>> I'm wondering what you folks are using inside of Amazon (not
>> interested in RDS at the moment).
>>
>> Thanks
>> Tory
>>
>>
>> --
>> Sent via pgsql-performance mailing list ([email protected]
> <mailto:[email protected]>)
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-performance
--
http://yves.zioup.com
gpg: 4096R/32B0F416
--
Sent via pgsql-performance mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Testing in AWS, EBS
2016-05-25 22:34 Testing in AWS, EBS Tory M Blue <[email protected]>
2016-05-25 23:02 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-25 23:56 ` Re: Testing in AWS, EBS Yves Dorfsman <[email protected]>
@ 2016-05-26 01:08 ` Rayson Ho <[email protected]>
2016-05-26 12:53 ` Re: Testing in AWS, EBS Yves Dorfsman <[email protected]>
0 siblings, 1 reply; 15+ messages in thread
From: Rayson Ho @ 2016-05-26 01:08 UTC (permalink / raw)
To: Yves Dorfsman <[email protected]>; +Cc: pgsql-performance
Actually, when "EBS-Optimized" is on, then the instance gets dedicated
bandwidth to EBS.
Rayson
==================================================
Open Grid Scheduler - The Official Open Source Grid Engine
http://gridscheduler.sourceforge.net/
http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html
On Wed, May 25, 2016 at 7:56 PM, Yves Dorfsman <[email protected]> wrote:
> Indeed, old-style disk EBS vs new-style SSd EBS.
>
> Be aware that EBS traffic is considered as part of the total "network"
> traffic, and each type of instance has different limits on maximum network
> throughput. Those difference are very significant, do tests on the same
> volume
> between two different type of instances, both with enough cpu and memory
> for
> the I/O to be the bottleneck, you will be surprised!
>
>
> On 2016-05-25 17:02, Rayson Ho wrote:
> > There are many factors that can affect EBS performance. For example, the
> type
> > of EBS volume, the instance type, whether EBS-optimized is turned on or
> not, etc.
> >
> > Without the details, then there is no apples to apples comparsion...
> >
> > Rayson
> >
> > ==================================================
> > Open Grid Scheduler - The Official Open Source Grid Engine
> > http://gridscheduler.sourceforge.net/
> > http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html
> >
> >
> >
> > On Wed, May 25, 2016 at 6:34 PM, Tory M Blue <[email protected]
> > <mailto:[email protected]>> wrote:
> >>
> >> We are starting some testing in AWS, with EC2, EBS backed setups.
> >>
> >> What I found interesting today, was a single EBS 1TB volume, gave me
> >> something like 108MB/s throughput, however a RAID10 (4 250GB EBS
> >> volumes), gave me something like 31MB/s (test after test after test).
> >>
> >> I'm wondering what you folks are using inside of Amazon (not
> >> interested in RDS at the moment).
> >>
> >> Thanks
> >> Tory
> >>
> >>
> >> --
> >> Sent via pgsql-performance mailing list (
> [email protected]
> > <mailto:[email protected]>)
> >> To make changes to your subscription:
> >> http://www.postgresql.org/mailpref/pgsql-performance
>
>
> --
> http://yves.zioup.com
> gpg: 4096R/32B0F416
>
>
>
> --
> Sent via pgsql-performance mailing list ([email protected])
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
>
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Testing in AWS, EBS
2016-05-25 22:34 Testing in AWS, EBS Tory M Blue <[email protected]>
2016-05-25 23:02 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-25 23:56 ` Re: Testing in AWS, EBS Yves Dorfsman <[email protected]>
2016-05-26 01:08 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
@ 2016-05-26 12:53 ` Yves Dorfsman <[email protected]>
2016-05-26 13:00 ` Re: Testing in AWS, EBS Artem Tomyuk <[email protected]>
0 siblings, 1 reply; 15+ messages in thread
From: Yves Dorfsman @ 2016-05-26 12:53 UTC (permalink / raw)
To: pgsql-performance
On 2016-05-25 19:08, Rayson Ho wrote:
> Actually, when "EBS-Optimized" is on, then the instance gets dedicated
> bandwidth to EBS.
Hadn't realised that, thanks.
Is the EBS bandwidth then somewhat limited depending on the type of instance too?
--
http://yves.zioup.com
gpg: 4096R/32B0F416
--
Sent via pgsql-performance mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Testing in AWS, EBS
2016-05-25 22:34 Testing in AWS, EBS Tory M Blue <[email protected]>
2016-05-25 23:02 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-25 23:56 ` Re: Testing in AWS, EBS Yves Dorfsman <[email protected]>
2016-05-26 01:08 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-26 12:53 ` Re: Testing in AWS, EBS Yves Dorfsman <[email protected]>
@ 2016-05-26 13:00 ` Artem Tomyuk <[email protected]>
2016-05-26 13:50 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
0 siblings, 1 reply; 15+ messages in thread
From: Artem Tomyuk @ 2016-05-26 13:00 UTC (permalink / raw)
To: Yves Dorfsman <[email protected]>; +Cc: pgsql-performance
Yes, the smaller instance you choose - the slower ebs will be.
EBS lives separately from EC2, they are communicating via network. So small
instance = low network bandwidth = poorer disk performance.
But still strong recommendation to pre-warm your ebs in any case,
especially if they created from snapshot.
2016-05-26 15:53 GMT+03:00 Yves Dorfsman <[email protected]>:
> On 2016-05-25 19:08, Rayson Ho wrote:
> > Actually, when "EBS-Optimized" is on, then the instance gets dedicated
> > bandwidth to EBS.
>
> Hadn't realised that, thanks.
> Is the EBS bandwidth then somewhat limited depending on the type of
> instance too?
>
> --
> http://yves.zioup.com
> gpg: 4096R/32B0F416
>
>
>
> --
> Sent via pgsql-performance mailing list ([email protected])
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
>
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Testing in AWS, EBS
2016-05-25 22:34 Testing in AWS, EBS Tory M Blue <[email protected]>
2016-05-25 23:02 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-25 23:56 ` Re: Testing in AWS, EBS Yves Dorfsman <[email protected]>
2016-05-26 01:08 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-26 12:53 ` Re: Testing in AWS, EBS Yves Dorfsman <[email protected]>
2016-05-26 13:00 ` Re: Testing in AWS, EBS Artem Tomyuk <[email protected]>
@ 2016-05-26 13:50 ` Rayson Ho <[email protected]>
2016-05-26 14:00 ` Re: Testing in AWS, EBS Artem Tomyuk <[email protected]>
0 siblings, 1 reply; 15+ messages in thread
From: Rayson Ho @ 2016-05-26 13:50 UTC (permalink / raw)
To: Artem Tomyuk <[email protected]>; +Cc: Yves Dorfsman <[email protected]>; pgsql-performance
On Thu, May 26, 2016 at 9:00 AM, Artem Tomyuk <[email protected]> wrote:
>
> But still strong recommendation to pre-warm your ebs in any case,
especially if they created from snapshot.
That used to be true. However, at AWS re:Invent 2015, Amazon engineers said
that EBS pre-warming is not needed anymore.
Rayson
==================================================
Open Grid Scheduler - The Official Open Source Grid Engine
http://gridscheduler.sourceforge.net/
http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html
> 2016-05-26 15:53 GMT+03:00 Yves Dorfsman <[email protected]>:
>>
>> On 2016-05-25 19:08, Rayson Ho wrote:
>> > Actually, when "EBS-Optimized" is on, then the instance gets dedicated
>> > bandwidth to EBS.
>>
>> Hadn't realised that, thanks.
>> Is the EBS bandwidth then somewhat limited depending on the type of
instance too?
>>
>> --
>> http://yves.zioup.com
>> gpg: 4096R/32B0F416
>>
>>
>>
>> --
>> Sent via pgsql-performance mailing list ([email protected]
)
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-performance
>
>
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Testing in AWS, EBS
2016-05-25 22:34 Testing in AWS, EBS Tory M Blue <[email protected]>
2016-05-25 23:02 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-25 23:56 ` Re: Testing in AWS, EBS Yves Dorfsman <[email protected]>
2016-05-26 01:08 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-26 12:53 ` Re: Testing in AWS, EBS Yves Dorfsman <[email protected]>
2016-05-26 13:00 ` Re: Testing in AWS, EBS Artem Tomyuk <[email protected]>
2016-05-26 13:50 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
@ 2016-05-26 14:00 ` Artem Tomyuk <[email protected]>
2016-05-26 14:47 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
0 siblings, 1 reply; 15+ messages in thread
From: Artem Tomyuk @ 2016-05-26 14:00 UTC (permalink / raw)
To: Rayson Ho <[email protected]>; +Cc: Yves Dorfsman <[email protected]>; pgsql-performance
2016-05-26 16:50 GMT+03:00 Rayson Ho <[email protected]>:
> Amazon engineers said that EBS pre-warming is not needed anymore.
but still if you will skip this step you wont get much performance on ebs
created from snapshot.
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Testing in AWS, EBS
2016-05-25 22:34 Testing in AWS, EBS Tory M Blue <[email protected]>
2016-05-25 23:02 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-25 23:56 ` Re: Testing in AWS, EBS Yves Dorfsman <[email protected]>
2016-05-26 01:08 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-26 12:53 ` Re: Testing in AWS, EBS Yves Dorfsman <[email protected]>
2016-05-26 13:00 ` Re: Testing in AWS, EBS Artem Tomyuk <[email protected]>
2016-05-26 13:50 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-26 14:00 ` Re: Testing in AWS, EBS Artem Tomyuk <[email protected]>
@ 2016-05-26 14:47 ` Rayson Ho <[email protected]>
2016-05-26 14:52 ` Re: Testing in AWS, EBS Artem Tomyuk <[email protected]>
0 siblings, 1 reply; 15+ messages in thread
From: Rayson Ho @ 2016-05-26 14:47 UTC (permalink / raw)
To: Artem Tomyuk <[email protected]>; +Cc: Yves Dorfsman <[email protected]>; pgsql-performance
On Thu, May 26, 2016 at 10:00 AM, Artem Tomyuk <[email protected]> wrote:
>
> 2016-05-26 16:50 GMT+03:00 Rayson Ho <[email protected]>:
>
>> Amazon engineers said that EBS pre-warming is not needed anymore.
>
>
> but still if you will skip this step you wont get much performance on ebs
> created from snapshot.
>
IIRC, that's not what Amazon engineers said. Is that from your personal
experience, and if so, when did you do the test??
Rayson
==================================================
Open Grid Scheduler - The Official Open Source Grid Engine
http://gridscheduler.sourceforge.net/
http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Testing in AWS, EBS
2016-05-25 22:34 Testing in AWS, EBS Tory M Blue <[email protected]>
2016-05-25 23:02 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-25 23:56 ` Re: Testing in AWS, EBS Yves Dorfsman <[email protected]>
2016-05-26 01:08 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-26 12:53 ` Re: Testing in AWS, EBS Yves Dorfsman <[email protected]>
2016-05-26 13:00 ` Re: Testing in AWS, EBS Artem Tomyuk <[email protected]>
2016-05-26 13:50 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-26 14:00 ` Re: Testing in AWS, EBS Artem Tomyuk <[email protected]>
2016-05-26 14:47 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
@ 2016-05-26 14:52 ` Artem Tomyuk <[email protected]>
2016-05-26 14:54 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
0 siblings, 1 reply; 15+ messages in thread
From: Artem Tomyuk @ 2016-05-26 14:52 UTC (permalink / raw)
To: Rayson Ho <[email protected]>; +Cc: Yves Dorfsman <[email protected]>; pgsql-performance
Please look at the official doc.
"New EBS volumes receive their maximum performance the moment that they are
available and do not require initialization (formerly known as
pre-warming). However, storage blocks on volumes that were restored from
snapshots must be initialized (pulled down from Amazon S3 and written to
the volume) before you can access the block"
Quotation from:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html
2016-05-26 17:47 GMT+03:00 Rayson Ho <[email protected]>:
> On Thu, May 26, 2016 at 10:00 AM, Artem Tomyuk <[email protected]>
> wrote:
>
>>
>> 2016-05-26 16:50 GMT+03:00 Rayson Ho <[email protected]>:
>>
>>> Amazon engineers said that EBS pre-warming is not needed anymore.
>>
>>
>> but still if you will skip this step you wont get much performance on ebs
>> created from snapshot.
>>
>
>
> IIRC, that's not what Amazon engineers said. Is that from your personal
> experience, and if so, when did you do the test??
>
> Rayson
>
> ==================================================
> Open Grid Scheduler - The Official Open Source Grid Engine
> http://gridscheduler.sourceforge.net/
> http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html
>
>
>
>
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Testing in AWS, EBS
2016-05-25 22:34 Testing in AWS, EBS Tory M Blue <[email protected]>
2016-05-25 23:02 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-25 23:56 ` Re: Testing in AWS, EBS Yves Dorfsman <[email protected]>
2016-05-26 01:08 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-26 12:53 ` Re: Testing in AWS, EBS Yves Dorfsman <[email protected]>
2016-05-26 13:00 ` Re: Testing in AWS, EBS Artem Tomyuk <[email protected]>
2016-05-26 13:50 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-26 14:00 ` Re: Testing in AWS, EBS Artem Tomyuk <[email protected]>
2016-05-26 14:47 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-26 14:52 ` Re: Testing in AWS, EBS Artem Tomyuk <[email protected]>
@ 2016-05-26 14:54 ` Rayson Ho <[email protected]>
2016-05-26 15:03 ` Re: Testing in AWS, EBS Artem Tomyuk <[email protected]>
0 siblings, 1 reply; 15+ messages in thread
From: Rayson Ho @ 2016-05-26 14:54 UTC (permalink / raw)
To: Artem Tomyuk <[email protected]>; +Cc: Yves Dorfsman <[email protected]>; pgsql-performance
Thanks Artem.
So no EBS pre-warming does not apply to EBS volumes created from snapshots.
Rayson
==================================================
Open Grid Scheduler - The Official Open Source Grid Engine
http://gridscheduler.sourceforge.net/
http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html
On Thu, May 26, 2016 at 10:52 AM, Artem Tomyuk <[email protected]> wrote:
> Please look at the official doc.
>
> "New EBS volumes receive their maximum performance the moment that they are
> available and do not require initialization (formerly known as pre-warming).
> However, storage blocks on volumes that were restored from snapshots must be
> initialized (pulled down from Amazon S3 and written to the volume) before
> you can access the block"
>
> Quotation from:
> http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html
>
> 2016-05-26 17:47 GMT+03:00 Rayson Ho <[email protected]>:
>>
>> On Thu, May 26, 2016 at 10:00 AM, Artem Tomyuk <[email protected]>
>> wrote:
>>>
>>>
>>> 2016-05-26 16:50 GMT+03:00 Rayson Ho <[email protected]>:
>>>>
>>>> Amazon engineers said that EBS pre-warming is not needed anymore.
>>>
>>>
>>> but still if you will skip this step you wont get much performance on ebs
>>> created from snapshot.
>>
>>
>>
>> IIRC, that's not what Amazon engineers said. Is that from your personal
>> experience, and if so, when did you do the test??
>>
>> Rayson
>>
>> ==================================================
>> Open Grid Scheduler - The Official Open Source Grid Engine
>> http://gridscheduler.sourceforge.net/
>> http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html
>>
>>
>>
>
--
Sent via pgsql-performance mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Testing in AWS, EBS
2016-05-25 22:34 Testing in AWS, EBS Tory M Blue <[email protected]>
2016-05-25 23:02 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-25 23:56 ` Re: Testing in AWS, EBS Yves Dorfsman <[email protected]>
2016-05-26 01:08 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-26 12:53 ` Re: Testing in AWS, EBS Yves Dorfsman <[email protected]>
2016-05-26 13:00 ` Re: Testing in AWS, EBS Artem Tomyuk <[email protected]>
2016-05-26 13:50 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-26 14:00 ` Re: Testing in AWS, EBS Artem Tomyuk <[email protected]>
2016-05-26 14:47 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-26 14:52 ` Re: Testing in AWS, EBS Artem Tomyuk <[email protected]>
2016-05-26 14:54 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
@ 2016-05-26 15:03 ` Artem Tomyuk <[email protected]>
2016-05-26 15:41 ` Re: Testing in AWS, EBS Yves Dorfsman <[email protected]>
0 siblings, 1 reply; 15+ messages in thread
From: Artem Tomyuk @ 2016-05-26 15:03 UTC (permalink / raw)
To: Rayson Ho <[email protected]>; +Cc: Yves Dorfsman <[email protected]>; pgsql-performance
Why no? Or you missed something?
It should be done on every EBS restored from snapshot.
Is that from your personal experience, and if so, when did you do the test??
Yes we are using this practice, because as a part of our production load we
are using auto scale groups to create new instances, wheech are created
from AMI, wheech stands on snapshots, so...
2016-05-26 17:54 GMT+03:00 Rayson Ho <[email protected]>:
> Thanks Artem.
>
> So no EBS pre-warming does not apply to EBS volumes created from snapshots.
>
> Rayson
>
> ==================================================
> Open Grid Scheduler - The Official Open Source Grid Engine
> http://gridscheduler.sourceforge.net/
> http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html
>
>
> On Thu, May 26, 2016 at 10:52 AM, Artem Tomyuk <[email protected]>
> wrote:
> > Please look at the official doc.
> >
> > "New EBS volumes receive their maximum performance the moment that they
> are
> > available and do not require initialization (formerly known as
> pre-warming).
> > However, storage blocks on volumes that were restored from snapshots
> must be
> > initialized (pulled down from Amazon S3 and written to the volume) before
> > you can access the block"
> >
> > Quotation from:
> > http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html
> >
> > 2016-05-26 17:47 GMT+03:00 Rayson Ho <[email protected]>:
> >>
> >> On Thu, May 26, 2016 at 10:00 AM, Artem Tomyuk <[email protected]>
> >> wrote:
> >>>
> >>>
> >>> 2016-05-26 16:50 GMT+03:00 Rayson Ho <[email protected]>:
> >>>>
> >>>> Amazon engineers said that EBS pre-warming is not needed anymore.
> >>>
> >>>
> >>> but still if you will skip this step you wont get much performance on
> ebs
> >>> created from snapshot.
> >>
> >>
> >>
> >> IIRC, that's not what Amazon engineers said. Is that from your personal
> >> experience, and if so, when did you do the test??
> >>
> >> Rayson
> >>
> >> ==================================================
> >> Open Grid Scheduler - The Official Open Source Grid Engine
> >> http://gridscheduler.sourceforge.net/
> >> http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html
> >>
> >>
> >>
> >
>
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Testing in AWS, EBS
2016-05-25 22:34 Testing in AWS, EBS Tory M Blue <[email protected]>
2016-05-25 23:02 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-25 23:56 ` Re: Testing in AWS, EBS Yves Dorfsman <[email protected]>
2016-05-26 01:08 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-26 12:53 ` Re: Testing in AWS, EBS Yves Dorfsman <[email protected]>
2016-05-26 13:00 ` Re: Testing in AWS, EBS Artem Tomyuk <[email protected]>
2016-05-26 13:50 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-26 14:00 ` Re: Testing in AWS, EBS Artem Tomyuk <[email protected]>
2016-05-26 14:47 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-26 14:52 ` Re: Testing in AWS, EBS Artem Tomyuk <[email protected]>
2016-05-26 14:54 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-26 15:03 ` Re: Testing in AWS, EBS Artem Tomyuk <[email protected]>
@ 2016-05-26 15:41 ` Yves Dorfsman <[email protected]>
2016-05-26 21:26 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
0 siblings, 1 reply; 15+ messages in thread
From: Yves Dorfsman @ 2016-05-26 15:41 UTC (permalink / raw)
To: pgsql-performance
On 2016-05-26 09:03, Artem Tomyuk wrote:
> Why no? Or you missed something?
I think Rayson is correct, but the double negative makes it hard to read:
"So no EBS pre-warming does not apply to EBS volumes created from snapshots."
Which I interpret as:
So, "no EBS pre-warming", does not apply to EBS volumes created from snapshots.
Which is correct, you sitll have to warm your EBS when created from sanpshots (to get the data from S3 to the filesystem).
--
http://yves.zioup.com
gpg: 4096R/32B0F416
--
Sent via pgsql-performance mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Testing in AWS, EBS
2016-05-25 22:34 Testing in AWS, EBS Tory M Blue <[email protected]>
2016-05-25 23:02 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-25 23:56 ` Re: Testing in AWS, EBS Yves Dorfsman <[email protected]>
2016-05-26 01:08 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-26 12:53 ` Re: Testing in AWS, EBS Yves Dorfsman <[email protected]>
2016-05-26 13:00 ` Re: Testing in AWS, EBS Artem Tomyuk <[email protected]>
2016-05-26 13:50 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-26 14:00 ` Re: Testing in AWS, EBS Artem Tomyuk <[email protected]>
2016-05-26 14:47 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-26 14:52 ` Re: Testing in AWS, EBS Artem Tomyuk <[email protected]>
2016-05-26 14:54 ` Re: Testing in AWS, EBS Rayson Ho <[email protected]>
2016-05-26 15:03 ` Re: Testing in AWS, EBS Artem Tomyuk <[email protected]>
2016-05-26 15:41 ` Re: Testing in AWS, EBS Yves Dorfsman <[email protected]>
@ 2016-05-26 21:26 ` Rayson Ho <[email protected]>
0 siblings, 0 replies; 15+ messages in thread
From: Rayson Ho @ 2016-05-26 21:26 UTC (permalink / raw)
To: Yves Dorfsman <[email protected]>; +Cc: pgsql-performance
Thanks Yves for the clarification!
It used to be very important to pre-warm EBS before running benchmarks
in order to get consistent results.
Then at re:Invent 2015, the AWS engineers said that it is not needed
anymore, which IMO is a lot less work for us to do benchmarking in
AWS, because pre-warming a multi-TB EBS vol is very time consuming,
and the I/Os were not free.
Rayson
==================================================
Open Grid Scheduler - The Official Open Source Grid Engine
http://gridscheduler.sourceforge.net/
http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html
On Thu, May 26, 2016 at 11:41 AM, Yves Dorfsman <[email protected]> wrote:
> On 2016-05-26 09:03, Artem Tomyuk wrote:
>> Why no? Or you missed something?
>
> I think Rayson is correct, but the double negative makes it hard to read:
>
> "So no EBS pre-warming does not apply to EBS volumes created from snapshots."
>
> Which I interpret as:
> So, "no EBS pre-warming", does not apply to EBS volumes created from snapshots.
>
> Which is correct, you sitll have to warm your EBS when created from sanpshots (to get the data from S3 to the filesystem).
>
>
> --
> http://yves.zioup.com
> gpg: 4096R/32B0F416
>
>
>
> --
> Sent via pgsql-performance mailing list ([email protected])
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
--
Sent via pgsql-performance mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Testing in AWS, EBS
2016-05-25 22:34 Testing in AWS, EBS Tory M Blue <[email protected]>
@ 2016-05-26 08:41 ` Artem Tomyuk <[email protected]>
1 sibling, 0 replies; 15+ messages in thread
From: Artem Tomyuk @ 2016-05-26 08:41 UTC (permalink / raw)
To: Tory M Blue <[email protected]>; +Cc: pgsql-performance
Hi.
AWS EBS its a really painful story....
How was created volumes for RAID? From snapshots?
If you want to get the best performance from EBS it needs to pre-warmed.
Here is the tutorial how to achieve that:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html
Also you should read this one if you want to get really great for
performance:
http://hatim.eu/2014/05/24/leveraging-ssd-ephemeral-disks-in-ec2-part-1/
Good luck!
2016-05-26 1:34 GMT+03:00 Tory M Blue <[email protected]>:
> We are starting some testing in AWS, with EC2, EBS backed setups.
>
> What I found interesting today, was a single EBS 1TB volume, gave me
> something like 108MB/s throughput, however a RAID10 (4 250GB EBS
> volumes), gave me something like 31MB/s (test after test after test).
>
> I'm wondering what you folks are using inside of Amazon (not
> interested in RDS at the moment).
>
> Thanks
> Tory
>
>
> --
> Sent via pgsql-performance mailing list ([email protected])
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
>
^ permalink raw reply [nested|flat] 15+ messages in thread
end of thread, other threads:[~2016-05-26 21:26 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2016-05-25 22:34 Testing in AWS, EBS Tory M Blue <[email protected]>
2016-05-25 23:02 ` Rayson Ho <[email protected]>
2016-05-25 23:56 ` Yves Dorfsman <[email protected]>
2016-05-26 01:08 ` Rayson Ho <[email protected]>
2016-05-26 12:53 ` Yves Dorfsman <[email protected]>
2016-05-26 13:00 ` Artem Tomyuk <[email protected]>
2016-05-26 13:50 ` Rayson Ho <[email protected]>
2016-05-26 14:00 ` Artem Tomyuk <[email protected]>
2016-05-26 14:47 ` Rayson Ho <[email protected]>
2016-05-26 14:52 ` Artem Tomyuk <[email protected]>
2016-05-26 14:54 ` Rayson Ho <[email protected]>
2016-05-26 15:03 ` Artem Tomyuk <[email protected]>
2016-05-26 15:41 ` Yves Dorfsman <[email protected]>
2016-05-26 21:26 ` Rayson Ho <[email protected]>
2016-05-26 08:41 ` Artem Tomyuk <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox