public inbox for [email protected]
help / color / mirror / Atom feedVery poor read performance, query independent
52+ messages / 12 participants
[nested] [flat]
* Very poor read performance, query independent
@ 2017-07-10 14:03 Charles Nadeau <[email protected]>
2017-07-10 15:25 ` Re: Very poor read performance, query independent Rick Otten <[email protected]>
2017-07-10 15:35 ` Re: Very poor read performance, query independent Andreas Kretschmer <[email protected]>
2017-07-10 19:24 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-11 23:15 ` Re: Very poor read performance, query independent Merlin Moncure <[email protected]>
2017-07-25 09:36 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
0 siblings, 5 replies; 52+ messages in thread
From: Charles Nadeau @ 2017-07-10 14:03 UTC (permalink / raw)
To: pgsql-performance
I’m running PostgreSQL 9.6.3 on Ubuntu 16.10 (kernel 4.4.0-85-generic).
Hardware is:
*2x Intel Xeon E5550
*72GB RAM
*Hardware RAID10 (4 x 146GB SAS 10k) P410i controller with 1GB FBWC (80%
read/20% write) for Postgresql data only:
Logical Drive: 3
Size: 273.4 GB
Fault Tolerance: 1+0
Heads: 255
Sectors Per Track: 32
Cylinders: 65535
Strip Size: 128 KB
Full Stripe Size: 256 KB
Status: OK
Caching: Enabled
Unique Identifier: 600508B1001037383941424344450A00
Disk Name: /dev/sdc
Mount Points: /mnt/data 273.4 GB
OS Status: LOCKED
Logical Drive Label: A00A194750123456789ABCDE516F
Mirror Group 0:
physicaldrive 2I:1:5 (port 2I:box 1:bay 5, SAS, 146 GB, OK)
physicaldrive 2I:1:6 (port 2I:box 1:bay 6, SAS, 146 GB, OK)
Mirror Group 1:
physicaldrive 2I:1:7 (port 2I:box 1:bay 7, SAS, 146 GB, OK)
physicaldrive 2I:1:8 (port 2I:box 1:bay 8, SAS, 146 GB, OK)
Drive Type: Data
Formatted with ext4 with: sudo mkfs.ext4 -E stride=32,stripe_width=64 -v
/dev/sdc1.
Mounted in /etc/fstab with this line:
"UUID=99fef4ae-51dc-4365-9210-0b153b1cbbd0 /mnt/data ext4
rw,nodiratime,user_xattr,noatime,nobarrier,errors=remount-ro 0 1"
Postgresql is the only application running on this server.
Postgresql is used as a mini data warehouse to generate reports and do
statistical analysis. It is used by at most 2 users and fresh data is added
every 10 days. The database has 16 tables: one is 224GB big and the rest
are between 16kB and 470MB big.
My configuration is:
name | current_setting | source
---------------------------------+------------------------------------------------+----------------------
application_name | psql | client
autovacuum_vacuum_scale_factor | 0 | configuration file
autovacuum_vacuum_threshold | 2000 | configuration file
checkpoint_completion_target | 0.9 | configuration file
checkpoint_timeout | 30min | configuration file
client_encoding | UTF8 | client
client_min_messages | log | configuration file
cluster_name | 9.6/main | configuration file
cpu_index_tuple_cost | 0.001 | configuration file
cpu_operator_cost | 0.0005 | configuration file
cpu_tuple_cost | 0.003 | configuration file
DateStyle | ISO, YMD | configuration file
default_statistics_target | 100 | configuration file
default_text_search_config | pg_catalog.english | configuration file
dynamic_shared_memory_type | posix | configuration file
effective_cache_size | 22GB | configuration file
effective_io_concurrency | 4 | configuration file
external_pid_file | /var/run/postgresql/9.6-main.pid | configuration file
lc_messages | C | configuration file
lc_monetary | en_CA.UTF-8 | configuration file
lc_numeric | en_CA.UTF-8 | configuration file
lc_time | en_CA.UTF-8 | configuration file
listen_addresses | * | configuration file
lock_timeout | 100s | configuration file
log_autovacuum_min_duration | 0 | configuration file
log_checkpoints | on | configuration file
log_connections | on | configuration file
log_destination | csvlog | configuration file
log_directory | /mnt/bigzilla/data/toburn/hp/postgresql/pg_log |
configuration file
log_disconnections | on | configuration file
log_error_verbosity | default | configuration file
log_file_mode | 0600 | configuration file
log_filename | postgresql-%Y-%m-%d_%H%M%S.log | configuration file
log_line_prefix | user=%u,db=%d,app=%aclient=%h | configuration file
log_lock_waits | on | configuration file
log_min_duration_statement | 0 | configuration file
log_min_error_statement | debug1 | configuration file
log_min_messages | debug1 | configuration file
log_rotation_size | 1GB | configuration file
log_temp_files | 0 | configuration file
log_timezone | localtime | configuration file
logging_collector | on | configuration file
maintenance_work_mem | 3GB | configuration file
max_connections | 10 | configuration file
max_locks_per_transaction | 256 | configuration file
max_parallel_workers_per_gather | 14 | configuration file
max_stack_depth | 2MB | environment variable
max_wal_size | 4GB | configuration file
max_worker_processes | 14 | configuration file
min_wal_size | 2GB | configuration file
parallel_setup_cost | 1000 | configuration file
parallel_tuple_cost | 0.012 | configuration file
port | 5432 | configuration file
random_page_cost | 22 | configuration file
seq_page_cost | 1 | configuration file
shared_buffers | 34GB | configuration file
shared_preload_libraries | pg_stat_statements | configuration file
ssl | on | configuration file
ssl_cert_file | /etc/ssl/certs/ssl-cert-snakeoil.pem | configuration file
ssl_key_file | /etc/ssl/private/ssl-cert-snakeoil.key | configuration file
statement_timeout | 1000000s | configuration file
stats_temp_directory | /var/run/postgresql/9.6-main.pg_stat_tmp |
configuration file
superuser_reserved_connections | 1 | configuration file
syslog_facility | local1 | configuration file
syslog_ident | postgres | configuration file
syslog_sequence_numbers | on | configuration file
temp_file_limit | 80GB | configuration file
TimeZone | localtime | configuration file
track_activities | on | configuration file
track_counts | on | configuration file
track_functions | all | configuration file
unix_socket_directories | /var/run/postgresql | configuration file
vacuum_cost_delay | 1ms | configuration file
vacuum_cost_limit | 5000 | configuration file
vacuum_cost_page_dirty | 200 | configuration file
vacuum_cost_page_hit | 10 | configuration file
vacuum_cost_page_miss | 100 | configuration file
wal_buffers | 16MB | configuration file
wal_compression | on | configuration file
wal_sync_method | fdatasync | configuration file
work_mem | 1468006kB | configuration file
The part of /etc/sysctl.conf I modified is:
vm.swappiness = 1
vm.dirty_background_bytes = 134217728
vm.dirty_bytes = 1073741824
vm.overcommit_ratio = 100
vm.zone_reclaim_mode = 0
kernel.numa_balancing = 0
kernel.sched_autogroup_enabled = 0
kernel.sched_migration_cost_ns = 5000000
The problem I have is very poor read. When I benchmark my array with fio I
get random reads of about 200MB/s and 1100IOPS and sequential reads of
about 286MB/s and 21000IPS. But when I watch my queries using pg_activity,
I get at best 4MB/s. Also using dstat I can see that iowait time is at
about 25%. This problem is not query-dependent.
I backed up the database, I reformated the array making sure it is well
aligned then restored the database and got the same result.
Where should I target my troubleshooting at this stage? I reformatted my
drive, I tuned my postgresql.conf and OS as much as I could. The hardware
doesn’t seem to have any issues, I am really puzzled.
Thanks!
Charles
--
Charles Nadeau Ph.D.
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
@ 2017-07-10 15:25 ` Rick Otten <[email protected]>
2017-07-11 10:39 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 13:38 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
4 siblings, 2 replies; 52+ messages in thread
From: Rick Otten @ 2017-07-10 15:25 UTC (permalink / raw)
To: Charles Nadeau <[email protected]>; +Cc: pgsql-performance
Although probably not the root cause, at the least I would set up hugepages
(
https://www.postgresql.org/docs/9.6/static/kernel-resources.html#LINUX-HUGE-PAGES
), and bump effective_io_concurrency up quite a bit as well (256 ?).
On Mon, Jul 10, 2017 at 10:03 AM, Charles Nadeau <[email protected]>
wrote:
> I’m running PostgreSQL 9.6.3 on Ubuntu 16.10 (kernel 4.4.0-85-generic).
> Hardware is:
>
> *2x Intel Xeon E5550
>
> *72GB RAM
>
> *Hardware RAID10 (4 x 146GB SAS 10k) P410i controller with 1GB FBWC (80%
> read/20% write) for Postgresql data only:
>
> Logical Drive: 3
>
> Size: 273.4 GB
>
> Fault Tolerance: 1+0
>
> Heads: 255
>
> Sectors Per Track: 32
>
> Cylinders: 65535
>
> Strip Size: 128 KB
>
> Full Stripe Size: 256 KB
>
> Status: OK
>
> Caching: Enabled
>
> Unique Identifier: 600508B1001037383941424344450A00
>
> Disk Name: /dev/sdc
>
> Mount Points: /mnt/data 273.4 GB
>
> OS Status: LOCKED
>
> Logical Drive Label: A00A194750123456789ABCDE516F
>
> Mirror Group 0:
>
> physicaldrive 2I:1:5 (port 2I:box 1:bay 5, SAS, 146 GB, OK)
>
> physicaldrive 2I:1:6 (port 2I:box 1:bay 6, SAS, 146 GB, OK)
>
> Mirror Group 1:
>
> physicaldrive 2I:1:7 (port 2I:box 1:bay 7, SAS, 146 GB, OK)
>
> physicaldrive 2I:1:8 (port 2I:box 1:bay 8, SAS, 146 GB, OK)
>
> Drive Type: Data
>
> Formatted with ext4 with: sudo mkfs.ext4 -E stride=32,stripe_width=64 -v
> /dev/sdc1.
>
> Mounted in /etc/fstab with this line: "UUID=99fef4ae-51dc-4365-9210-0b153b1cbbd0
> /mnt/data ext4 rw,nodiratime,user_xattr,noatime,nobarrier,errors=remount-ro
> 0 1"
>
> Postgresql is the only application running on this server.
>
>
> Postgresql is used as a mini data warehouse to generate reports and do
> statistical analysis. It is used by at most 2 users and fresh data is added
> every 10 days. The database has 16 tables: one is 224GB big and the rest
> are between 16kB and 470MB big.
>
>
> My configuration is:
>
>
> name | current_setting | source
>
> ---------------------------------+--------------------------
> ----------------------+----------------------
>
> application_name | psql | client
>
> autovacuum_vacuum_scale_factor | 0 | configuration file
>
> autovacuum_vacuum_threshold | 2000 | configuration file
>
> checkpoint_completion_target | 0.9 | configuration file
>
> checkpoint_timeout | 30min | configuration file
>
> client_encoding | UTF8 | client
>
> client_min_messages | log | configuration file
>
> cluster_name | 9.6/main | configuration file
>
> cpu_index_tuple_cost | 0.001 | configuration file
>
> cpu_operator_cost | 0.0005 | configuration file
>
> cpu_tuple_cost | 0.003 | configuration file
>
> DateStyle | ISO, YMD | configuration file
>
> default_statistics_target | 100 | configuration file
>
> default_text_search_config | pg_catalog.english | configuration file
>
> dynamic_shared_memory_type | posix | configuration file
>
> effective_cache_size | 22GB | configuration file
>
> effective_io_concurrency | 4 | configuration file
>
> external_pid_file | /var/run/postgresql/9.6-main.pid | configuration file
>
> lc_messages | C | configuration file
>
> lc_monetary | en_CA.UTF-8 | configuration file
>
> lc_numeric | en_CA.UTF-8 | configuration file
>
> lc_time | en_CA.UTF-8 | configuration file
>
> listen_addresses | * | configuration file
>
> lock_timeout | 100s | configuration file
>
> log_autovacuum_min_duration | 0 | configuration file
>
> log_checkpoints | on | configuration file
>
> log_connections | on | configuration file
>
> log_destination | csvlog | configuration file
>
> log_directory | /mnt/bigzilla/data/toburn/hp/postgresql/pg_log |
> configuration file
>
> log_disconnections | on | configuration file
>
> log_error_verbosity | default | configuration file
>
> log_file_mode | 0600 | configuration file
>
> log_filename | postgresql-%Y-%m-%d_%H%M%S.log | configuration file
>
> log_line_prefix | user=%u,db=%d,app=%aclient=%h | configuration file
>
> log_lock_waits | on | configuration file
>
> log_min_duration_statement | 0 | configuration file
>
> log_min_error_statement | debug1 | configuration file
>
> log_min_messages | debug1 | configuration file
>
> log_rotation_size | 1GB | configuration file
>
> log_temp_files | 0 | configuration file
>
> log_timezone | localtime | configuration file
>
> logging_collector | on | configuration file
>
> maintenance_work_mem | 3GB | configuration file
>
> max_connections | 10 | configuration file
>
> max_locks_per_transaction | 256 | configuration file
>
> max_parallel_workers_per_gather | 14 | configuration file
>
> max_stack_depth | 2MB | environment variable
>
> max_wal_size | 4GB | configuration file
>
> max_worker_processes | 14 | configuration file
>
> min_wal_size | 2GB | configuration file
>
> parallel_setup_cost | 1000 | configuration file
>
> parallel_tuple_cost | 0.012 | configuration file
>
> port | 5432 | configuration file
>
> random_page_cost | 22 | configuration file
>
> seq_page_cost | 1 | configuration file
>
> shared_buffers | 34GB | configuration file
>
> shared_preload_libraries | pg_stat_statements | configuration file
>
> ssl | on | configuration file
>
> ssl_cert_file | /etc/ssl/certs/ssl-cert-snakeoil.pem | configuration file
>
> ssl_key_file | /etc/ssl/private/ssl-cert-snakeoil.key | configuration file
>
> statement_timeout | 1000000s | configuration file
>
> stats_temp_directory | /var/run/postgresql/9.6-main.pg_stat_tmp |
> configuration file
>
> superuser_reserved_connections | 1 | configuration file
>
> syslog_facility | local1 | configuration file
>
> syslog_ident | postgres | configuration file
>
> syslog_sequence_numbers | on | configuration file
>
> temp_file_limit | 80GB | configuration file
>
> TimeZone | localtime | configuration file
>
> track_activities | on | configuration file
>
> track_counts | on | configuration file
>
> track_functions | all | configuration file
>
> unix_socket_directories | /var/run/postgresql | configuration file
>
> vacuum_cost_delay | 1ms | configuration file
>
> vacuum_cost_limit | 5000 | configuration file
>
> vacuum_cost_page_dirty | 200 | configuration file
>
> vacuum_cost_page_hit | 10 | configuration file
>
> vacuum_cost_page_miss | 100 | configuration file
>
> wal_buffers | 16MB | configuration file
>
> wal_compression | on | configuration file
>
> wal_sync_method | fdatasync | configuration file
>
> work_mem | 1468006kB | configuration file
>
>
> The part of /etc/sysctl.conf I modified is:
>
> vm.swappiness = 1
>
> vm.dirty_background_bytes = 134217728
>
> vm.dirty_bytes = 1073741824
>
> vm.overcommit_ratio = 100
>
> vm.zone_reclaim_mode = 0
>
> kernel.numa_balancing = 0
>
> kernel.sched_autogroup_enabled = 0
>
> kernel.sched_migration_cost_ns = 5000000
>
>
> The problem I have is very poor read. When I benchmark my array with fio I
> get random reads of about 200MB/s and 1100IOPS and sequential reads of
> about 286MB/s and 21000IPS. But when I watch my queries using pg_activity,
> I get at best 4MB/s. Also using dstat I can see that iowait time is at
> about 25%. This problem is not query-dependent.
>
> I backed up the database, I reformated the array making sure it is well
> aligned then restored the database and got the same result.
>
> Where should I target my troubleshooting at this stage? I reformatted my
> drive, I tuned my postgresql.conf and OS as much as I could. The hardware
> doesn’t seem to have any issues, I am really puzzled.
>
> Thanks!
>
>
> Charles
>
> --
> Charles Nadeau Ph.D.
>
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 15:25 ` Re: Very poor read performance, query independent Rick Otten <[email protected]>
@ 2017-07-11 10:39 ` Charles Nadeau <[email protected]>
1 sibling, 0 replies; 52+ messages in thread
From: Charles Nadeau @ 2017-07-11 10:39 UTC (permalink / raw)
To: Rick Otten <[email protected]>; +Cc: pgsql-performance
Rick,
I applied the change you recommended but it didn't speed up the reads.
One thing I forgot to mention earlier is the speed of the backup made with
the COPY operations seems almost normal: I have read speed of up to 85MB/s.
Thanks for your help!
Charles
On Mon, Jul 10, 2017 at 5:25 PM, Rick Otten <[email protected]>
wrote:
> Although probably not the root cause, at the least I would set up
> hugepages ( https://www.postgresql.org/docs/9.6/static/kernel-
> resources.html#LINUX-HUGE-PAGES ), and bump effective_io_concurrency up
> quite a bit as well (256 ?).
>
>
> On Mon, Jul 10, 2017 at 10:03 AM, Charles Nadeau <[email protected]
> > wrote:
>
>> I’m running PostgreSQL 9.6.3 on Ubuntu 16.10 (kernel 4.4.0-85-generic).
>> Hardware is:
>>
>> *2x Intel Xeon E5550
>>
>> *72GB RAM
>>
>> *Hardware RAID10 (4 x 146GB SAS 10k) P410i controller with 1GB FBWC (80%
>> read/20% write) for Postgresql data only:
>>
>> Logical Drive: 3
>>
>> Size: 273.4 GB
>>
>> Fault Tolerance: 1+0
>>
>> Heads: 255
>>
>> Sectors Per Track: 32
>>
>> Cylinders: 65535
>>
>> Strip Size: 128 KB
>>
>> Full Stripe Size: 256 KB
>>
>> Status: OK
>>
>> Caching: Enabled
>>
>> Unique Identifier: 600508B1001037383941424344450A00
>>
>> Disk Name: /dev/sdc
>>
>> Mount Points: /mnt/data 273.4 GB
>>
>> OS Status: LOCKED
>>
>> Logical Drive Label: A00A194750123456789ABCDE516F
>>
>> Mirror Group 0:
>>
>> physicaldrive 2I:1:5 (port 2I:box 1:bay 5, SAS, 146 GB, OK)
>>
>> physicaldrive 2I:1:6 (port 2I:box 1:bay 6, SAS, 146 GB, OK)
>>
>> Mirror Group 1:
>>
>> physicaldrive 2I:1:7 (port 2I:box 1:bay 7, SAS, 146 GB, OK)
>>
>> physicaldrive 2I:1:8 (port 2I:box 1:bay 8, SAS, 146 GB, OK)
>>
>> Drive Type: Data
>>
>> Formatted with ext4 with: sudo mkfs.ext4 -E stride=32,stripe_width=64 -v
>> /dev/sdc1.
>>
>> Mounted in /etc/fstab with this line: "UUID=99fef4ae-51dc-4365-9210-0b153b1cbbd0
>> /mnt/data ext4 rw,nodiratime,user_xattr,noatime,nobarrier,errors=remount-ro
>> 0 1"
>>
>> Postgresql is the only application running on this server.
>>
>>
>> Postgresql is used as a mini data warehouse to generate reports and do
>> statistical analysis. It is used by at most 2 users and fresh data is added
>> every 10 days. The database has 16 tables: one is 224GB big and the rest
>> are between 16kB and 470MB big.
>>
>>
>> My configuration is:
>>
>>
>> name | current_setting | source
>>
>> ---------------------------------+--------------------------
>> ----------------------+----------------------
>>
>> application_name | psql | client
>>
>> autovacuum_vacuum_scale_factor | 0 | configuration file
>>
>> autovacuum_vacuum_threshold | 2000 | configuration file
>>
>> checkpoint_completion_target | 0.9 | configuration file
>>
>> checkpoint_timeout | 30min | configuration file
>>
>> client_encoding | UTF8 | client
>>
>> client_min_messages | log | configuration file
>>
>> cluster_name | 9.6/main | configuration file
>>
>> cpu_index_tuple_cost | 0.001 | configuration file
>>
>> cpu_operator_cost | 0.0005 | configuration file
>>
>> cpu_tuple_cost | 0.003 | configuration file
>>
>> DateStyle | ISO, YMD | configuration file
>>
>> default_statistics_target | 100 | configuration file
>>
>> default_text_search_config | pg_catalog.english | configuration file
>>
>> dynamic_shared_memory_type | posix | configuration file
>>
>> effective_cache_size | 22GB | configuration file
>>
>> effective_io_concurrency | 4 | configuration file
>>
>> external_pid_file | /var/run/postgresql/9.6-main.pid | configuration file
>>
>> lc_messages | C | configuration file
>>
>> lc_monetary | en_CA.UTF-8 | configuration file
>>
>> lc_numeric | en_CA.UTF-8 | configuration file
>>
>> lc_time | en_CA.UTF-8 | configuration file
>>
>> listen_addresses | * | configuration file
>>
>> lock_timeout | 100s | configuration file
>>
>> log_autovacuum_min_duration | 0 | configuration file
>>
>> log_checkpoints | on | configuration file
>>
>> log_connections | on | configuration file
>>
>> log_destination | csvlog | configuration file
>>
>> log_directory | /mnt/bigzilla/data/toburn/hp/postgresql/pg_log |
>> configuration file
>>
>> log_disconnections | on | configuration file
>>
>> log_error_verbosity | default | configuration file
>>
>> log_file_mode | 0600 | configuration file
>>
>> log_filename | postgresql-%Y-%m-%d_%H%M%S.log | configuration file
>>
>> log_line_prefix | user=%u,db=%d,app=%aclient=%h | configuration file
>>
>> log_lock_waits | on | configuration file
>>
>> log_min_duration_statement | 0 | configuration file
>>
>> log_min_error_statement | debug1 | configuration file
>>
>> log_min_messages | debug1 | configuration file
>>
>> log_rotation_size | 1GB | configuration file
>>
>> log_temp_files | 0 | configuration file
>>
>> log_timezone | localtime | configuration file
>>
>> logging_collector | on | configuration file
>>
>> maintenance_work_mem | 3GB | configuration file
>>
>> max_connections | 10 | configuration file
>>
>> max_locks_per_transaction | 256 | configuration file
>>
>> max_parallel_workers_per_gather | 14 | configuration file
>>
>> max_stack_depth | 2MB | environment variable
>>
>> max_wal_size | 4GB | configuration file
>>
>> max_worker_processes | 14 | configuration file
>>
>> min_wal_size | 2GB | configuration file
>>
>> parallel_setup_cost | 1000 | configuration file
>>
>> parallel_tuple_cost | 0.012 | configuration file
>>
>> port | 5432 | configuration file
>>
>> random_page_cost | 22 | configuration file
>>
>> seq_page_cost | 1 | configuration file
>>
>> shared_buffers | 34GB | configuration file
>>
>> shared_preload_libraries | pg_stat_statements | configuration file
>>
>> ssl | on | configuration file
>>
>> ssl_cert_file | /etc/ssl/certs/ssl-cert-snakeoil.pem | configuration file
>>
>> ssl_key_file | /etc/ssl/private/ssl-cert-snakeoil.key | configuration
>> file
>>
>> statement_timeout | 1000000s | configuration file
>>
>> stats_temp_directory | /var/run/postgresql/9.6-main.pg_stat_tmp |
>> configuration file
>>
>> superuser_reserved_connections | 1 | configuration file
>>
>> syslog_facility | local1 | configuration file
>>
>> syslog_ident | postgres | configuration file
>>
>> syslog_sequence_numbers | on | configuration file
>>
>> temp_file_limit | 80GB | configuration file
>>
>> TimeZone | localtime | configuration file
>>
>> track_activities | on | configuration file
>>
>> track_counts | on | configuration file
>>
>> track_functions | all | configuration file
>>
>> unix_socket_directories | /var/run/postgresql | configuration file
>>
>> vacuum_cost_delay | 1ms | configuration file
>>
>> vacuum_cost_limit | 5000 | configuration file
>>
>> vacuum_cost_page_dirty | 200 | configuration file
>>
>> vacuum_cost_page_hit | 10 | configuration file
>>
>> vacuum_cost_page_miss | 100 | configuration file
>>
>> wal_buffers | 16MB | configuration file
>>
>> wal_compression | on | configuration file
>>
>> wal_sync_method | fdatasync | configuration file
>>
>> work_mem | 1468006kB | configuration file
>>
>>
>> The part of /etc/sysctl.conf I modified is:
>>
>> vm.swappiness = 1
>>
>> vm.dirty_background_bytes = 134217728
>>
>> vm.dirty_bytes = 1073741824
>>
>> vm.overcommit_ratio = 100
>>
>> vm.zone_reclaim_mode = 0
>>
>> kernel.numa_balancing = 0
>>
>> kernel.sched_autogroup_enabled = 0
>>
>> kernel.sched_migration_cost_ns = 5000000
>>
>>
>> The problem I have is very poor read. When I benchmark my array with fio
>> I get random reads of about 200MB/s and 1100IOPS and sequential reads of
>> about 286MB/s and 21000IPS. But when I watch my queries using pg_activity,
>> I get at best 4MB/s. Also using dstat I can see that iowait time is at
>> about 25%. This problem is not query-dependent.
>>
>> I backed up the database, I reformated the array making sure it is well
>> aligned then restored the database and got the same result.
>>
>> Where should I target my troubleshooting at this stage? I reformatted my
>> drive, I tuned my postgresql.conf and OS as much as I could. The hardware
>> doesn’t seem to have any issues, I am really puzzled.
>>
>> Thanks!
>>
>>
>> Charles
>>
>> --
>> Charles Nadeau Ph.D.
>>
>
>
--
Charles Nadeau Ph.D.
http://charlesnadeau.blogspot.com/
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 15:25 ` Re: Very poor read performance, query independent Rick Otten <[email protected]>
@ 2017-07-12 13:38 ` Charles Nadeau <[email protected]>
2017-07-12 14:10 ` Re: Very poor read performance, query independent Rick Otten <[email protected]>
1 sibling, 1 reply; 52+ messages in thread
From: Charles Nadeau @ 2017-07-12 13:38 UTC (permalink / raw)
To: Rick Otten <[email protected]>; +Cc: pgsql-performance
Rick,
Should the number of page should always be correlated to the VmPeak of the
postmaster or could it be set to reflect shared_buffer or another setting?
Thanks!
Charles
On Mon, Jul 10, 2017 at 5:25 PM, Rick Otten <[email protected]>
wrote:
> Although probably not the root cause, at the least I would set up
> hugepages ( https://www.postgresql.org/docs/9.6/static/kernel-
> resources.html#LINUX-HUGE-PAGES ), and bump effective_io_concurrency up
> quite a bit as well (256 ?).
>
>
> On Mon, Jul 10, 2017 at 10:03 AM, Charles Nadeau <[email protected]
> > wrote:
>
>> I’m running PostgreSQL 9.6.3 on Ubuntu 16.10 (kernel 4.4.0-85-generic).
>> Hardware is:
>>
>> *2x Intel Xeon E5550
>>
>> *72GB RAM
>>
>> *Hardware RAID10 (4 x 146GB SAS 10k) P410i controller with 1GB FBWC (80%
>> read/20% write) for Postgresql data only:
>>
>> Logical Drive: 3
>>
>> Size: 273.4 GB
>>
>> Fault Tolerance: 1+0
>>
>> Heads: 255
>>
>> Sectors Per Track: 32
>>
>> Cylinders: 65535
>>
>> Strip Size: 128 KB
>>
>> Full Stripe Size: 256 KB
>>
>> Status: OK
>>
>> Caching: Enabled
>>
>> Unique Identifier: 600508B1001037383941424344450A00
>>
>> Disk Name: /dev/sdc
>>
>> Mount Points: /mnt/data 273.4 GB
>>
>> OS Status: LOCKED
>>
>> Logical Drive Label: A00A194750123456789ABCDE516F
>>
>> Mirror Group 0:
>>
>> physicaldrive 2I:1:5 (port 2I:box 1:bay 5, SAS, 146 GB, OK)
>>
>> physicaldrive 2I:1:6 (port 2I:box 1:bay 6, SAS, 146 GB, OK)
>>
>> Mirror Group 1:
>>
>> physicaldrive 2I:1:7 (port 2I:box 1:bay 7, SAS, 146 GB, OK)
>>
>> physicaldrive 2I:1:8 (port 2I:box 1:bay 8, SAS, 146 GB, OK)
>>
>> Drive Type: Data
>>
>> Formatted with ext4 with: sudo mkfs.ext4 -E stride=32,stripe_width=64 -v
>> /dev/sdc1.
>>
>> Mounted in /etc/fstab with this line: "UUID=99fef4ae-51dc-4365-9210-0b153b1cbbd0
>> /mnt/data ext4 rw,nodiratime,user_xattr,noatime,nobarrier,errors=remount-ro
>> 0 1"
>>
>> Postgresql is the only application running on this server.
>>
>>
>> Postgresql is used as a mini data warehouse to generate reports and do
>> statistical analysis. It is used by at most 2 users and fresh data is added
>> every 10 days. The database has 16 tables: one is 224GB big and the rest
>> are between 16kB and 470MB big.
>>
>>
>> My configuration is:
>>
>>
>> name | current_setting | source
>>
>> ---------------------------------+--------------------------
>> ----------------------+----------------------
>>
>> application_name | psql | client
>>
>> autovacuum_vacuum_scale_factor | 0 | configuration file
>>
>> autovacuum_vacuum_threshold | 2000 | configuration file
>>
>> checkpoint_completion_target | 0.9 | configuration file
>>
>> checkpoint_timeout | 30min | configuration file
>>
>> client_encoding | UTF8 | client
>>
>> client_min_messages | log | configuration file
>>
>> cluster_name | 9.6/main | configuration file
>>
>> cpu_index_tuple_cost | 0.001 | configuration file
>>
>> cpu_operator_cost | 0.0005 | configuration file
>>
>> cpu_tuple_cost | 0.003 | configuration file
>>
>> DateStyle | ISO, YMD | configuration file
>>
>> default_statistics_target | 100 | configuration file
>>
>> default_text_search_config | pg_catalog.english | configuration file
>>
>> dynamic_shared_memory_type | posix | configuration file
>>
>> effective_cache_size | 22GB | configuration file
>>
>> effective_io_concurrency | 4 | configuration file
>>
>> external_pid_file | /var/run/postgresql/9.6-main.pid | configuration file
>>
>> lc_messages | C | configuration file
>>
>> lc_monetary | en_CA.UTF-8 | configuration file
>>
>> lc_numeric | en_CA.UTF-8 | configuration file
>>
>> lc_time | en_CA.UTF-8 | configuration file
>>
>> listen_addresses | * | configuration file
>>
>> lock_timeout | 100s | configuration file
>>
>> log_autovacuum_min_duration | 0 | configuration file
>>
>> log_checkpoints | on | configuration file
>>
>> log_connections | on | configuration file
>>
>> log_destination | csvlog | configuration file
>>
>> log_directory | /mnt/bigzilla/data/toburn/hp/postgresql/pg_log |
>> configuration file
>>
>> log_disconnections | on | configuration file
>>
>> log_error_verbosity | default | configuration file
>>
>> log_file_mode | 0600 | configuration file
>>
>> log_filename | postgresql-%Y-%m-%d_%H%M%S.log | configuration file
>>
>> log_line_prefix | user=%u,db=%d,app=%aclient=%h | configuration file
>>
>> log_lock_waits | on | configuration file
>>
>> log_min_duration_statement | 0 | configuration file
>>
>> log_min_error_statement | debug1 | configuration file
>>
>> log_min_messages | debug1 | configuration file
>>
>> log_rotation_size | 1GB | configuration file
>>
>> log_temp_files | 0 | configuration file
>>
>> log_timezone | localtime | configuration file
>>
>> logging_collector | on | configuration file
>>
>> maintenance_work_mem | 3GB | configuration file
>>
>> max_connections | 10 | configuration file
>>
>> max_locks_per_transaction | 256 | configuration file
>>
>> max_parallel_workers_per_gather | 14 | configuration file
>>
>> max_stack_depth | 2MB | environment variable
>>
>> max_wal_size | 4GB | configuration file
>>
>> max_worker_processes | 14 | configuration file
>>
>> min_wal_size | 2GB | configuration file
>>
>> parallel_setup_cost | 1000 | configuration file
>>
>> parallel_tuple_cost | 0.012 | configuration file
>>
>> port | 5432 | configuration file
>>
>> random_page_cost | 22 | configuration file
>>
>> seq_page_cost | 1 | configuration file
>>
>> shared_buffers | 34GB | configuration file
>>
>> shared_preload_libraries | pg_stat_statements | configuration file
>>
>> ssl | on | configuration file
>>
>> ssl_cert_file | /etc/ssl/certs/ssl-cert-snakeoil.pem | configuration file
>>
>> ssl_key_file | /etc/ssl/private/ssl-cert-snakeoil.key | configuration
>> file
>>
>> statement_timeout | 1000000s | configuration file
>>
>> stats_temp_directory | /var/run/postgresql/9.6-main.pg_stat_tmp |
>> configuration file
>>
>> superuser_reserved_connections | 1 | configuration file
>>
>> syslog_facility | local1 | configuration file
>>
>> syslog_ident | postgres | configuration file
>>
>> syslog_sequence_numbers | on | configuration file
>>
>> temp_file_limit | 80GB | configuration file
>>
>> TimeZone | localtime | configuration file
>>
>> track_activities | on | configuration file
>>
>> track_counts | on | configuration file
>>
>> track_functions | all | configuration file
>>
>> unix_socket_directories | /var/run/postgresql | configuration file
>>
>> vacuum_cost_delay | 1ms | configuration file
>>
>> vacuum_cost_limit | 5000 | configuration file
>>
>> vacuum_cost_page_dirty | 200 | configuration file
>>
>> vacuum_cost_page_hit | 10 | configuration file
>>
>> vacuum_cost_page_miss | 100 | configuration file
>>
>> wal_buffers | 16MB | configuration file
>>
>> wal_compression | on | configuration file
>>
>> wal_sync_method | fdatasync | configuration file
>>
>> work_mem | 1468006kB | configuration file
>>
>>
>> The part of /etc/sysctl.conf I modified is:
>>
>> vm.swappiness = 1
>>
>> vm.dirty_background_bytes = 134217728
>>
>> vm.dirty_bytes = 1073741824
>>
>> vm.overcommit_ratio = 100
>>
>> vm.zone_reclaim_mode = 0
>>
>> kernel.numa_balancing = 0
>>
>> kernel.sched_autogroup_enabled = 0
>>
>> kernel.sched_migration_cost_ns = 5000000
>>
>>
>> The problem I have is very poor read. When I benchmark my array with fio
>> I get random reads of about 200MB/s and 1100IOPS and sequential reads of
>> about 286MB/s and 21000IPS. But when I watch my queries using pg_activity,
>> I get at best 4MB/s. Also using dstat I can see that iowait time is at
>> about 25%. This problem is not query-dependent.
>>
>> I backed up the database, I reformated the array making sure it is well
>> aligned then restored the database and got the same result.
>>
>> Where should I target my troubleshooting at this stage? I reformatted my
>> drive, I tuned my postgresql.conf and OS as much as I could. The hardware
>> doesn’t seem to have any issues, I am really puzzled.
>>
>> Thanks!
>>
>>
>> Charles
>>
>> --
>> Charles Nadeau Ph.D.
>>
>
>
--
Charles Nadeau Ph.D.
http://charlesnadeau.blogspot.com/
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 15:25 ` Re: Very poor read performance, query independent Rick Otten <[email protected]>
2017-07-12 13:38 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
@ 2017-07-12 14:10 ` Rick Otten <[email protected]>
0 siblings, 0 replies; 52+ messages in thread
From: Rick Otten @ 2017-07-12 14:10 UTC (permalink / raw)
To: Charles Nadeau <[email protected]>; +Cc: pgsql-performance
On Wed, Jul 12, 2017 at 9:38 AM, Charles Nadeau <[email protected]>
wrote:
> Rick,
>
> Should the number of page should always be correlated to the VmPeak of the
> postmaster or could it be set to reflect shared_buffer or another setting?
> Thanks!
>
>
The documentation implies that you may need to adjust its size when you
change shared_buffer settings.
I usually check it every now and then (I haven't build a formal monitor
yet.) to see if all of the huge pages are free/used and if it looks like
they are all getting consumed - consider bumping it higher. If there are
lots free, you are probably fine.
cat /proc/meminfo | grep -i "^huge"
--
Also regarding my note on effective_io_concurrency, which I'm not sure you
tried tweaking yet.
With file system and hardware caching between you and your spindles, your
best setting for effective_io_concurrency may be much higher than the
actual number of spindles. It is worth experimenting with. If you can,
try several values. You can use pg_bench to put consistent workloads on
your database for measurement purposes.
Charles
>
> On Mon, Jul 10, 2017 at 5:25 PM, Rick Otten <[email protected]>
> wrote:
>
>> Although probably not the root cause, at the least I would set up
>> hugepages ( https://www.postgresql.org/docs/9.6/static/kernel-resourc
>> es.html#LINUX-HUGE-PAGES ), and bump effective_io_concurrency up quite a
>> bit as well (256 ?).
>>
>>
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
@ 2017-07-10 15:35 ` Andreas Kretschmer <[email protected]>
2017-07-10 15:48 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
4 siblings, 1 reply; 52+ messages in thread
From: Andreas Kretschmer @ 2017-07-10 15:35 UTC (permalink / raw)
To: pgsql-performance
Am 10.07.2017 um 16:03 schrieb Charles Nadeau:
> random_page_cost | 22
why such a high value for random_page_cost?
Regards, Andreas
--
2ndQuadrant - The PostgreSQL Support Company.
www.2ndQuadrant.com
--
Sent via pgsql-performance mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 15:35 ` Re: Very poor read performance, query independent Andreas Kretschmer <[email protected]>
@ 2017-07-10 15:48 ` Charles Nadeau <[email protected]>
2017-07-10 18:35 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
0 siblings, 1 reply; 52+ messages in thread
From: Charles Nadeau @ 2017-07-10 15:48 UTC (permalink / raw)
To: Andreas Kretschmer <[email protected]>; +Cc: pgsql-performance
Andreas,
Because the ratio between the Sequential IOPS and Random IOPS is about 29.
Taking into account that part of the data is in RAM, I obtained an
"effective" ratio of about 22.
Thanks!
Charles
On Mon, Jul 10, 2017 at 5:35 PM, Andreas Kretschmer <[email protected]
> wrote:
>
>
> Am 10.07.2017 um 16:03 schrieb Charles Nadeau:
>
>> random_page_cost | 22
>>
>
>
> why such a high value for random_page_cost?
>
> Regards, Andreas
>
> --
> 2ndQuadrant - The PostgreSQL Support Company.
> www.2ndQuadrant.com
>
>
>
> --
> Sent via pgsql-performance mailing list ([email protected])
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
>
--
Charles Nadeau Ph.D.
http://charlesnadeau.blogspot.com/
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 15:35 ` Re: Very poor read performance, query independent Andreas Kretschmer <[email protected]>
2017-07-10 15:48 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
@ 2017-07-10 18:35 ` Igor Neyman <[email protected]>
2017-07-11 10:42 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-11 12:46 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
0 siblings, 2 replies; 52+ messages in thread
From: Igor Neyman @ 2017-07-10 18:35 UTC (permalink / raw)
To: Charles Nadeau <[email protected]>; Andreas Kretschmer <[email protected]>; +Cc: pgsql-performance
From: [email protected] [mailto:[email protected]] On Behalf Of Charles Nadeau
Sent: Monday, July 10, 2017 11:48 AM
To: Andreas Kretschmer <[email protected]>
Cc: [email protected]
Subject: Re: [PERFORM] Very poor read performance, query independent
Andreas,
Because the ratio between the Sequential IOPS and Random IOPS is about 29. Taking into account that part of the data is in RAM, I obtained an "effective" ratio of about 22.
Thanks!
Charles
On Mon, Jul 10, 2017 at 5:35 PM, Andreas Kretschmer <[email protected]<mailto:[email protected]>> wrote:
Am 10.07.2017 um 16:03 schrieb Charles Nadeau:
random_page_cost | 22
why such a high value for random_page_cost?
Regards, Andreas
--
2ndQuadrant - The PostgreSQL Support Company.
www.2ndQuadrant.com<http://www.2ndQuadrant.com;
--
Charles Nadeau Ph.D.
http://charlesnadeau.blogspot.com/
Considering RAM size of 72 GB and your database size of ~225GB, and also the fact that Postgres is the only app running on the server, probably 1/3 of your database resides in memory, so random_page_cost = 22 looks extremely high, probably it completely precludes index usage in your queries.
You should try this setting at least at its default value: random_page_cost =4, and probably go even lower.
Also, effective_cache_size is at least as big as your shared_buffers. Having 72GB RAM t effective_cache_size should be set around 64GB (again considering that Postgres is the only app running on the server).
Regards,
Igor Neyman
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 15:35 ` Re: Very poor read performance, query independent Andreas Kretschmer <[email protected]>
2017-07-10 15:48 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 18:35 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
@ 2017-07-11 10:42 ` Charles Nadeau <[email protected]>
2017-07-11 14:34 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
1 sibling, 1 reply; 52+ messages in thread
From: Charles Nadeau @ 2017-07-11 10:42 UTC (permalink / raw)
To: Igor Neyman <[email protected]>; +Cc: Andreas Kretschmer <[email protected]>; pgsql-performance
Igor,
I reduced the value of random_page_cost to 4 but the read speed remains low.
Regarding effective_cache_size and shared_buffer, do you mean they should
be both equal to 64GB?
Thanks for suggestions!
Charles
On Mon, Jul 10, 2017 at 8:35 PM, Igor Neyman <[email protected]> wrote:
>
>
> *From:* [email protected] [mailto:pgsql-performance-
> [email protected]] *On Behalf Of *Charles Nadeau
> *Sent:* Monday, July 10, 2017 11:48 AM
> *To:* Andreas Kretschmer <[email protected]>
> *Cc:* [email protected]
> *Subject:* Re: [PERFORM] Very poor read performance, query independent
>
>
>
> Andreas,
>
>
>
> Because the ratio between the Sequential IOPS and Random IOPS is about 29.
> Taking into account that part of the data is in RAM, I obtained an
> "effective" ratio of about 22.
>
> Thanks!
>
>
>
> Charles
>
>
>
> On Mon, Jul 10, 2017 at 5:35 PM, Andreas Kretschmer <
> [email protected]> wrote:
>
>
>
> Am 10.07.2017 um 16:03 schrieb Charles Nadeau:
>
> random_page_cost | 22
>
>
>
> why such a high value for random_page_cost?
>
> Regards, Andreas
>
> --
> 2ndQuadrant - The PostgreSQL Support Company.
> www.2ndQuadrant.com
>
>
> --
>
> Charles Nadeau Ph.D.
> http://charlesnadeau.blogspot.com/
>
>
>
>
>
> Considering RAM size of 72 GB and your database size of ~225GB, and also
> the fact that Postgres is the only app running on the server, probably 1/3
> of your database resides in memory, so random_page_cost = 22 looks
> extremely high, probably it completely precludes index usage in your
> queries.
>
>
>
> You should try this setting at least at its default value:
> random_page_cost =4, and probably go even lower.
>
> Also, effective_cache_size is at least as big as your shared_buffers.
> Having 72GB RAM t effective_cache_size should be set around 64GB (again
> considering that Postgres is the only app running on the server).
>
>
>
> Regards,
>
> Igor Neyman
>
>
>
>
>
>
>
>
>
--
Charles Nadeau Ph.D.
http://charlesnadeau.blogspot.com/
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 15:35 ` Re: Very poor read performance, query independent Andreas Kretschmer <[email protected]>
2017-07-10 15:48 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 18:35 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-11 10:42 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
@ 2017-07-11 14:34 ` Igor Neyman <[email protected]>
2017-07-11 15:16 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-11 15:25 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
0 siblings, 2 replies; 52+ messages in thread
From: Igor Neyman @ 2017-07-11 14:34 UTC (permalink / raw)
To: Charles Nadeau <[email protected]>; +Cc: pgsql-performance
From: Charles Nadeau [mailto:[email protected]]
Sent: Tuesday, July 11, 2017 6:43 AM
To: Igor Neyman <[email protected]>
Cc: Andreas Kretschmer <[email protected]>; [email protected]
Subject: Re: [PERFORM] Very poor read performance, query independent
Igor,
I reduced the value of random_page_cost to 4 but the read speed remains low.
Regarding effective_cache_size and shared_buffer, do you mean they should be both equal to 64GB?
Thanks for suggestions!
Charles
No, they should not be equal.
From the docs:
effective_cache_size (integer)
Sets the planner's assumption about the effective size of the disk cache that is available to a single query. This is factored into estimates of the cost of using an index; a higher value makes it more likely index scans will be used, a lower value makes it more likely sequential scans will be used. When setting this parameter you should consider both PostgreSQL's shared buffers and the portion of the kernel's disk cache that will be used for PostgreSQL data files. Also, take into account the expected number of concurrent queries on different tables, since they will have to share the available space. This parameter has no effect on the size of shared memory allocated by PostgreSQL, nor does it reserve kernel disk cache; it is used only for estimation purposes. The system also does not assume data remains in the disk cache between queries. The default is 4 gigabytes (4GB).
So, I’d set shared_buffers at 24GB and effective_cache_size at 64GB.
Regards,
Igor
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 15:35 ` Re: Very poor read performance, query independent Andreas Kretschmer <[email protected]>
2017-07-10 15:48 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 18:35 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-11 10:42 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-11 14:34 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
@ 2017-07-11 15:16 ` Igor Neyman <[email protected]>
2017-07-12 07:21 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
1 sibling, 1 reply; 52+ messages in thread
From: Igor Neyman @ 2017-07-11 15:16 UTC (permalink / raw)
To: Charles Nadeau <[email protected]>; +Cc: pgsql-performance
From: [email protected] [mailto:[email protected]] On Behalf Of Igor Neyman
Sent: Tuesday, July 11, 2017 10:34 AM
To: Charles Nadeau <[email protected]>
Cc: [email protected]
Subject: Re: [PERFORM] Very poor read performance, query independent
From: Charles Nadeau [mailto:[email protected]]
Sent: Tuesday, July 11, 2017 6:43 AM
To: Igor Neyman <[email protected]<mailto:[email protected]>>
Cc: Andreas Kretschmer <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>
Subject: Re: [PERFORM] Very poor read performance, query independent
Igor,
I reduced the value of random_page_cost to 4 but the read speed remains low.
Regarding effective_cache_size and shared_buffer, do you mean they should be both equal to 64GB?
Thanks for suggestions!
Charles
No, they should not be equal.
From the docs:
effective_cache_size (integer)
Sets the planner's assumption about the effective size of the disk cache that is available to a single query. This is factored into estimates of the cost of using an index; a higher value makes it more likely index scans will be used, a lower value makes it more likely sequential scans will be used. When setting this parameter you should consider both PostgreSQL's shared buffers and the portion of the kernel's disk cache that will be used for PostgreSQL data files. Also, take into account the expected number of concurrent queries on different tables, since they will have to share the available space. This parameter has no effect on the size of shared memory allocated by PostgreSQL, nor does it reserve kernel disk cache; it is used only for estimation purposes. The system also does not assume data remains in the disk cache between queries. The default is 4 gigabytes (4GB).
So, I’d set shared_buffers at 24GB and effective_cache_size at 64GB.
Regards,
Igor
Also, maybe it’s time to look at execution plans (explain analyze) of specific slow queries, instead of trying to solve the problem “in general”.
Igor
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 15:35 ` Re: Very poor read performance, query independent Andreas Kretschmer <[email protected]>
2017-07-10 15:48 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 18:35 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-11 10:42 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-11 14:34 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-11 15:16 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
@ 2017-07-12 07:21 ` Charles Nadeau <[email protected]>
2017-07-12 13:57 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
0 siblings, 1 reply; 52+ messages in thread
From: Charles Nadeau @ 2017-07-12 07:21 UTC (permalink / raw)
To: Igor Neyman <[email protected]>; +Cc: pgsql-performance
Igor,
I set shared_buffers to 24 GB and effective_cache_size to 64GB and I can
see that the queries are faster due to the fact that the index are used
more often. Knowing I have 72GB of RAM and the server is exclusively
dedicated to Postgresql, what could be the maximum value for
effective_cache?
Thanks!
Charles
On Tue, Jul 11, 2017 at 5:16 PM, Igor Neyman <[email protected]> wrote:
>
>
> *From:* [email protected] [mailto:pgsql-performance-
> [email protected]] *On Behalf Of *Igor Neyman
> *Sent:* Tuesday, July 11, 2017 10:34 AM
> *To:* Charles Nadeau <[email protected]>
> *Cc:* [email protected]
> *Subject:* Re: [PERFORM] Very poor read performance, query independent
>
>
>
> *From:* Charles Nadeau [mailto:[email protected]
> <[email protected]>]
> *Sent:* Tuesday, July 11, 2017 6:43 AM
> *To:* Igor Neyman <[email protected]>
> *Cc:* Andreas Kretschmer <[email protected]>;
> [email protected]
> *Subject:* Re: [PERFORM] Very poor read performance, query independent
>
>
>
> Igor,
>
>
>
> I reduced the value of random_page_cost to 4 but the read speed remains
> low.
>
> Regarding effective_cache_size and shared_buffer, do you mean they should
> be both equal to 64GB?
>
> Thanks for suggestions!
>
>
>
> Charles
>
>
>
> No, they should not be equal.
>
> From the docs:
>
>
>
> effective_cache_size (integer)
>
> Sets the planner's assumption about the effective size of the disk cache
> that is available to a single query. This is factored into estimates of the
> cost of using an index; a higher value makes it more likely index scans
> will be used, a lower value makes it more likely sequential scans will be
> used. When setting this parameter you should consider both PostgreSQL's
> shared buffers and the portion of the kernel's disk cache that will be used
> for PostgreSQL data files. Also, take into account the expected number of
> concurrent queries on different tables, since they will have to share the
> available space. This parameter has no effect on the size of shared memory
> allocated by PostgreSQL, nor does it reserve kernel disk cache; it is used
> only for estimation purposes. The system also does not assume data remains
> in the disk cache between queries. The default is 4 gigabytes (4GB).
>
> So, I’d set shared_buffers at 24GB and effective_cache_size at 64GB.
>
>
>
> Regards,
>
> Igor
>
>
>
> Also, maybe it’s time to look at execution plans (explain analyze) of
> specific slow queries, instead of trying to solve the problem “in general”.
>
>
>
> Igor
>
>
>
--
Charles Nadeau Ph.D.
http://charlesnadeau.blogspot.com/
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 15:35 ` Re: Very poor read performance, query independent Andreas Kretschmer <[email protected]>
2017-07-10 15:48 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 18:35 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-11 10:42 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-11 14:34 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-11 15:16 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-12 07:21 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
@ 2017-07-12 13:57 ` Igor Neyman <[email protected]>
0 siblings, 0 replies; 52+ messages in thread
From: Igor Neyman @ 2017-07-12 13:57 UTC (permalink / raw)
To: Charles Nadeau <[email protected]>; +Cc: pgsql-performance
From: Charles Nadeau [mailto:[email protected]]
Sent: Wednesday, July 12, 2017 3:21 AM
To: Igor Neyman <[email protected]>
Cc: [email protected]
Subject: Re: [PERFORM] Very poor read performance, query independent
Igor,
I set shared_buffers to 24 GB and effective_cache_size to 64GB and I can see that the queries are faster due to the fact that the index are used more often. Knowing I have 72GB of RAM and the server is exclusively dedicated to Postgresql, what could be the maximum value for effective_cache?
Thanks!
Charles
64GB for effective_cache_size should be good enough, adding couple more GB wouldn’t change much.
Igor
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 15:35 ` Re: Very poor read performance, query independent Andreas Kretschmer <[email protected]>
2017-07-10 15:48 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 18:35 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-11 10:42 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-11 14:34 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
@ 2017-07-11 15:25 ` Charles Nadeau <[email protected]>
2017-07-11 15:46 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
1 sibling, 1 reply; 52+ messages in thread
From: Charles Nadeau @ 2017-07-11 15:25 UTC (permalink / raw)
To: Igor Neyman <[email protected]>; +Cc: pgsql-performance
Igor,
The sum of effective_cache_size and shared_buffer will be higher than the
physical memory I have. Is it OK?
Thanks!
Charles
On Tue, Jul 11, 2017 at 4:34 PM, Igor Neyman <[email protected]> wrote:
>
>
> *From:* Charles Nadeau [mailto:[email protected]]
> *Sent:* Tuesday, July 11, 2017 6:43 AM
> *To:* Igor Neyman <[email protected]>
> *Cc:* Andreas Kretschmer <[email protected]>;
> [email protected]
> *Subject:* Re: [PERFORM] Very poor read performance, query independent
>
>
>
> Igor,
>
>
>
> I reduced the value of random_page_cost to 4 but the read speed remains
> low.
>
> Regarding effective_cache_size and shared_buffer, do you mean they should
> be both equal to 64GB?
>
> Thanks for suggestions!
>
>
>
> Charles
>
>
>
> No, they should not be equal.
>
> From the docs:
>
>
>
> effective_cache_size (integer)
>
> Sets the planner's assumption about the effective size of the disk cache
> that is available to a single query. This is factored into estimates of the
> cost of using an index; a higher value makes it more likely index scans
> will be used, a lower value makes it more likely sequential scans will be
> used. When setting this parameter you should consider both PostgreSQL's
> shared buffers and the portion of the kernel's disk cache that will be used
> for PostgreSQL data files. Also, take into account the expected number of
> concurrent queries on different tables, since they will have to share the
> available space. This parameter has no effect on the size of shared memory
> allocated by PostgreSQL, nor does it reserve kernel disk cache; it is used
> only for estimation purposes. The system also does not assume data remains
> in the disk cache between queries. The default is 4 gigabytes (4GB).
>
> So, I’d set shared_buffers at 24GB and effective_cache_size at 64GB.
>
>
>
> Regards,
>
> Igor
>
>
>
--
Charles Nadeau Ph.D.
http://charlesnadeau.blogspot.com/
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 15:35 ` Re: Very poor read performance, query independent Andreas Kretschmer <[email protected]>
2017-07-10 15:48 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 18:35 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-11 10:42 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-11 14:34 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-11 15:25 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
@ 2017-07-11 15:46 ` Igor Neyman <[email protected]>
0 siblings, 0 replies; 52+ messages in thread
From: Igor Neyman @ 2017-07-11 15:46 UTC (permalink / raw)
To: Charles Nadeau <[email protected]>; +Cc: pgsql-performance
From: Charles Nadeau [mailto:[email protected]]
Sent: Tuesday, July 11, 2017 11:25 AM
To: Igor Neyman <[email protected]>
Cc: [email protected]
Subject: Re: [PERFORM] Very poor read performance, query independent
Attention: This email was sent from someone outside of Perceptron. Always exercise caution when opening attachments or clicking links from unknown senders or when receiving unexpected emails.
Igor,
The sum of effective_cache_size and shared_buffer will be higher than the physical memory I have. Is it OK?
Thanks!
Charles
Yes, that’s normal.
shared_buffers is the maximum that Postgres allowed to allocate, while effective_cache_size is just a number that optimizer takes into account when creating execution plan.
Igor
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 15:35 ` Re: Very poor read performance, query independent Andreas Kretschmer <[email protected]>
2017-07-10 15:48 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 18:35 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
@ 2017-07-11 12:46 ` Charles Nadeau <[email protected]>
2017-07-12 02:11 ` Re: Very poor read performance, query independent Mark Kirkwood <[email protected]>
1 sibling, 1 reply; 52+ messages in thread
From: Charles Nadeau @ 2017-07-11 12:46 UTC (permalink / raw)
To: Igor Neyman <[email protected]>; +Cc: Andreas Kretschmer <[email protected]>; pgsql-performance
After reducing random_page_cost to 4 and testing more, I can report that
the aggregate read throughput for parallel sequential scan is about 90MB/s.
However the throughput for sequential scan is still around 4MB/s.
One more question: if a query uses more than one table, can more than one
table be read through a parallel sequential scan? I have many queries
joining tables and I noticed that there was never more than one table read
in parallel.
Thanks!
Charles
On Mon, Jul 10, 2017 at 8:35 PM, Igor Neyman <[email protected]> wrote:
>
>
> *From:* [email protected] [mailto:pgsql-performance-
> [email protected]] *On Behalf Of *Charles Nadeau
> *Sent:* Monday, July 10, 2017 11:48 AM
> *To:* Andreas Kretschmer <[email protected]>
> *Cc:* [email protected]
> *Subject:* Re: [PERFORM] Very poor read performance, query independent
>
>
>
> Andreas,
>
>
>
> Because the ratio between the Sequential IOPS and Random IOPS is about 29.
> Taking into account that part of the data is in RAM, I obtained an
> "effective" ratio of about 22.
>
> Thanks!
>
>
>
> Charles
>
>
>
> On Mon, Jul 10, 2017 at 5:35 PM, Andreas Kretschmer <
> [email protected]> wrote:
>
>
>
> Am 10.07.2017 um 16:03 schrieb Charles Nadeau:
>
> random_page_cost | 22
>
>
>
> why such a high value for random_page_cost?
>
> Regards, Andreas
>
> --
> 2ndQuadrant - The PostgreSQL Support Company.
> www.2ndQuadrant.com
>
>
> --
>
> Charles Nadeau Ph.D.
> http://charlesnadeau.blogspot.com/
>
>
>
>
>
> Considering RAM size of 72 GB and your database size of ~225GB, and also
> the fact that Postgres is the only app running on the server, probably 1/3
> of your database resides in memory, so random_page_cost = 22 looks
> extremely high, probably it completely precludes index usage in your
> queries.
>
>
>
> You should try this setting at least at its default value:
> random_page_cost =4, and probably go even lower.
>
> Also, effective_cache_size is at least as big as your shared_buffers.
> Having 72GB RAM t effective_cache_size should be set around 64GB (again
> considering that Postgres is the only app running on the server).
>
>
>
> Regards,
>
> Igor Neyman
>
>
>
>
>
>
>
>
>
--
Charles Nadeau Ph.D.
http://charlesnadeau.blogspot.com/
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 15:35 ` Re: Very poor read performance, query independent Andreas Kretschmer <[email protected]>
2017-07-10 15:48 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 18:35 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-11 12:46 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
@ 2017-07-12 02:11 ` Mark Kirkwood <[email protected]>
2017-07-14 14:34 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-14 23:09 ` Re: Very poor read performance, query independent Mark Kirkwood <[email protected]>
2017-07-15 17:53 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
0 siblings, 3 replies; 52+ messages in thread
From: Mark Kirkwood @ 2017-07-12 02:11 UTC (permalink / raw)
To: Charles Nadeau <[email protected]>; Igor Neyman <[email protected]>; +Cc: Andreas Kretschmer <[email protected]>; pgsql-performance
Hmm - how are you measuring that sequential scan speed of 4MB/s? I'd
recommend doing a very simple test e.g, here's one on my workstation -
13 GB single table on 1 SATA drive - cold cache after reboot, sequential
scan using Postgres 9.6.2:
bench=# EXPLAIN SELECT count(*) FROM pgbench_accounts;
QUERY PLAN
------------------------------------------------------------------------------------
Aggregate (cost=2889345.00..2889345.01 rows=1 width=8)
-> Seq Scan on pgbench_accounts (cost=0.00..2639345.00
rows=100000000 width=0)
(2 rows)
bench=# SELECT pg_relation_size('pgbench_accounts');
pg_relation_size
------------------
13429514240
(1 row)
bench=# SELECT count(*) FROM pgbench_accounts;
count
-----------
100000000
(1 row)
Time: 118884.277 ms
So doing the math seq read speed is about 110MB/s (i.e 13 GB in 120
sec). Sure enough, while I was running the query iostat showed:
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 926.00 0.00 114.89 0.00
254.10 1.90 2.03 2.03 0.00 1.08 100.00
So might be useful for us to see something like that from your system -
note you need to check you really have flushed the cache, and that no
other apps are using the db.
regards
Mark
On 12/07/17 00:46, Charles Nadeau wrote:
> After reducing random_page_cost to 4 and testing more, I can report
> that the aggregate read throughput for parallel sequential scan is
> about 90MB/s. However the throughput for sequential scan is still
> around 4MB/s.
>
--
Sent via pgsql-performance mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 15:35 ` Re: Very poor read performance, query independent Andreas Kretschmer <[email protected]>
2017-07-10 15:48 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 18:35 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-11 12:46 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 02:11 ` Re: Very poor read performance, query independent Mark Kirkwood <[email protected]>
@ 2017-07-14 14:34 ` Charles Nadeau <[email protected]>
2 siblings, 0 replies; 52+ messages in thread
From: Charles Nadeau @ 2017-07-14 14:34 UTC (permalink / raw)
To: ; +Cc: Igor Neyman <[email protected]>; Andreas Kretschmer <[email protected]>; pgsql-performance
Mark,
First I must say that I changed my disks configuration from 4 disks in RAID
10 to 5 disks in RAID 0 because I almost ran out of disk space during the
last ingest of data.
Here is the result test you asked. It was done with a cold cache:
flows=# \timing
Timing is on.
flows=# explain select count(*) from flows;
QUERY PLAN
------------------------------------------------------------
-----------------------------------
Finalize Aggregate (cost=17214914.09..17214914.09 rows=1 width=8)
-> Gather (cost=17214914.07..17214914.09 rows=1 width=8)
Workers Planned: 1
-> Partial Aggregate (cost=17213914.07..17213914.07 rows=1
width=8)
-> Parallel Seq Scan on flows (cost=0.00..17019464.49
rows=388899162 width=0)
(5 rows)
Time: 171.835 ms
flows=# select pg_relation_size('flows');
pg_relation_size
------------------
129865867264
(1 row)
Time: 57.157 ms
flows=# select count(*) from flows;
LOG: duration: 625546.522 ms statement: select count(*) from flows;
count
-----------
589831190
(1 row)
Time: 625546.662 ms
The throughput reported by Postgresql is almost 198MB/s, and the throughput
as mesured by dstat during the query execution was between 25 and 299MB/s.
It is much better than what I had before! The i/o wait was about 12% all
through the query. One thing I noticed is the discrepency between the read
throughput reported by pg_activity and the one reported by dstat:
pg_activity always report a value lower than dstat.
Besides the change of disks configuration, here is what contributed the
most to the improvment of the performance so far:
Using Hugepage
Increasing effective_io_concurrency to 256
Reducing random_page_cost from 22 to 4
Reducing min_parallel_relation_size to 512kB to have more workers when
doing sequential parallel scan of my biggest table
Thanks for recomending this test, I now know what the real throughput
should be!
Charles
On Wed, Jul 12, 2017 at 4:11 AM, Mark Kirkwood <
[email protected]> wrote:
> Hmm - how are you measuring that sequential scan speed of 4MB/s? I'd
> recommend doing a very simple test e.g, here's one on my workstation - 13
> GB single table on 1 SATA drive - cold cache after reboot, sequential scan
> using Postgres 9.6.2:
>
> bench=# EXPLAIN SELECT count(*) FROM pgbench_accounts;
> QUERY PLAN
> ------------------------------------------------------------
> ------------------------
> Aggregate (cost=2889345.00..2889345.01 rows=1 width=8)
> -> Seq Scan on pgbench_accounts (cost=0.00..2639345.00 rows=100000000
> width=0)
> (2 rows)
>
>
> bench=# SELECT pg_relation_size('pgbench_accounts');
> pg_relation_size
> ------------------
> 13429514240
> (1 row)
>
> bench=# SELECT count(*) FROM pgbench_accounts;
> count
> -----------
> 100000000
> (1 row)
>
> Time: 118884.277 ms
>
>
> So doing the math seq read speed is about 110MB/s (i.e 13 GB in 120 sec).
> Sure enough, while I was running the query iostat showed:
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
> avgqu-sz await r_await w_await svctm %util
> sda 0.00 0.00 926.00 0.00 114.89 0.00 254.10
> 1.90 2.03 2.03 0.00 1.08 100.00
>
>
> So might be useful for us to see something like that from your system -
> note you need to check you really have flushed the cache, and that no other
> apps are using the db.
>
> regards
>
> Mark
>
>
> On 12/07/17 00:46, Charles Nadeau wrote:
>
>> After reducing random_page_cost to 4 and testing more, I can report that
>> the aggregate read throughput for parallel sequential scan is about 90MB/s.
>> However the throughput for sequential scan is still around 4MB/s.
>>
>>
>
--
Charles Nadeau Ph.D.
http://charlesnadeau.blogspot.com/
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 15:35 ` Re: Very poor read performance, query independent Andreas Kretschmer <[email protected]>
2017-07-10 15:48 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 18:35 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-11 12:46 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 02:11 ` Re: Very poor read performance, query independent Mark Kirkwood <[email protected]>
@ 2017-07-14 23:09 ` Mark Kirkwood <[email protected]>
2017-07-14 23:57 ` Re: Very poor read performance, query independent Mark Kirkwood <[email protected]>
2 siblings, 1 reply; 52+ messages in thread
From: Mark Kirkwood @ 2017-07-14 23:09 UTC (permalink / raw)
To: pgsql-performance
Ah yes - that seems more sensible (but still slower than I would expect
for 5 disks RAID 0). You should be able to get something like 5 *
(single disk speed) i.e about 500MB/s.
Might be worth increasing device read ahead (more than you have
already). Some of these so-called 'smart' RAID cards need to be hit over
the head before they will perform. E.g: I believe you have it set to 128
- I'd try 4096 or even 16384 (In the past I've used those settings on
some extremely stupid cards that refused to max out their disks known
speeds).
Also worth investigating is RAID stripe size - for DW work it makes
sense for it to be reasonably big (256K to 1M), which again will help
speed is sequential scans.
Cheers
Mark
On 15/07/17 02:09, Charles Nadeau wrote:
> Mark,
>
> First I must say that I changed my disks configuration from 4 disks in
> RAID 10 to 5 disks in RAID 0 because I almost ran out of disk space
> during the last ingest of data.
> Here is the result test you asked. It was done with a cold cache:
>
> flows=# \timing
> Timing is on.
> flows=# explain select count(*) from flows;
> QUERY PLAN
> -----------------------------------------------------------------------------------------------
> Finalize Aggregate (cost=17214914.09..17214914.09 rows=1 width=8)
> -> Gather (cost=17214914.07..17214914.09 rows=1 width=8)
> Workers Planned: 1
> -> Partial Aggregate (cost=17213914.07..17213914.07
> rows=1 width=8)
> -> Parallel Seq Scan on flows
> (cost=0.00..17019464.49 rows=388899162 width=0)
> (5 rows)
>
> Time: 171.835 ms
> flows=# select pg_relation_size('flows');
> pg_relation_size
> ------------------
> 129865867264
> (1 row)
>
> Time: 57.157 ms
> flows=# select count(*) from flows;
> LOG: duration: 625546.522 ms statement: select count(*) from flows;
> count
> -----------
> 589831190
> (1 row)
>
> Time: 625546.662 ms
>
> The throughput reported by Postgresql is almost 198MB/s, and the
> throughput as mesured by dstat during the query execution was between
> 25 and 299MB/s. It is much better than what I had before! The i/o wait
> was about 12% all through the query. One thing I noticed is the
> discrepency between the read throughput reported by pg_activity and
> the one reported by dstat: pg_activity always report a value lower
> than dstat.
>
> Besides the change of disks configuration, here is what contributed
> the most to the improvment of the performance so far:
>
> Using Hugepage
> Increasing effective_io_concurrency to 256
> Reducing random_page_cost from 22 to 4
> Reducing min_parallel_relation_size to 512kB to have more workers
> when doing sequential parallel scan of my biggest table
>
>
> Thanks for recomending this test, I now know what the real throughput
> should be!
>
> Charles
>
> On Wed, Jul 12, 2017 at 4:11 AM, Mark Kirkwood
> <[email protected] <mailto:[email protected]>>
> wrote:
>
> Hmm - how are you measuring that sequential scan speed of 4MB/s?
> I'd recommend doing a very simple test e.g, here's one on my
> workstation - 13 GB single table on 1 SATA drive - cold cache
> after reboot, sequential scan using Postgres 9.6.2:
>
> bench=# EXPLAIN SELECT count(*) FROM pgbench_accounts;
> QUERY PLAN
> ------------------------------------------------------------------------------------
> Aggregate (cost=2889345.00..2889345.01 rows=1 width=8)
> -> Seq Scan on pgbench_accounts (cost=0.00..2639345.00
> rows=100000000 width=0)
> (2 rows)
>
>
> bench=# SELECT pg_relation_size('pgbench_accounts');
> pg_relation_size
> ------------------
> 13429514240
> (1 row)
>
> bench=# SELECT count(*) FROM pgbench_accounts;
> count
> -----------
> 100000000
> (1 row)
>
> Time: 118884.277 ms
>
>
> So doing the math seq read speed is about 110MB/s (i.e 13 GB in
> 120 sec). Sure enough, while I was running the query iostat showed:
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
> avgrq-sz avgqu-sz await r_await w_await svctm %util
> sda 0.00 0.00 926.00 0.00 114.89 0.00
> 254.10 1.90 2.03 2.03 0.00 1.08 100.00
>
>
> So might be useful for us to see something like that from your
> system - note you need to check you really have flushed the cache,
> and that no other apps are using the db.
>
> regards
>
> Mark
>
>
> On 12/07/17 00:46, Charles Nadeau wrote:
>
> After reducing random_page_cost to 4 and testing more, I can
> report that the aggregate read throughput for parallel
> sequential scan is about 90MB/s. However the throughput for
> sequential scan is still around 4MB/s.
>
>
>
>
>
> --
> Charles Nadeau Ph.D.
> http://charlesnadeau.blogspot.com/
--
Sent via pgsql-performance mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 15:35 ` Re: Very poor read performance, query independent Andreas Kretschmer <[email protected]>
2017-07-10 15:48 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 18:35 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-11 12:46 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 02:11 ` Re: Very poor read performance, query independent Mark Kirkwood <[email protected]>
2017-07-14 23:09 ` Re: Very poor read performance, query independent Mark Kirkwood <[email protected]>
@ 2017-07-14 23:57 ` Mark Kirkwood <[email protected]>
2017-07-15 16:12 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-20 12:50 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
0 siblings, 2 replies; 52+ messages in thread
From: Mark Kirkwood @ 2017-07-14 23:57 UTC (permalink / raw)
To: pgsql-performance
Thinking about this a bit more - if somewhat more blazing performance is
needed, then this could be achieved via losing the RAID card and
spinning disks altogether and buying 1 of the NVME or SATA solid state
products: e.g
- Samsung 960 Pro or Evo 2 TB (approx 1 or 2 GB/s seq scan speeds and
200K IOPS)
- Intel S3610 or similar 1.2 TB (500 MB/s seq scan and 30K IOPS)
The Samsung needs an M.2 port on the mobo (but most should have 'em -
and if not PCIe X4 adapter cards are quite cheap). The Intel is a bit
more expensive compared to the Samsung, and is slower but has a longer
lifetime. However for your workload the Sammy is probably fine.
regards
Mark
On 15/07/17 11:09, Mark Kirkwood wrote:
> Ah yes - that seems more sensible (but still slower than I would
> expect for 5 disks RAID 0).
--
Sent via pgsql-performance mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 15:35 ` Re: Very poor read performance, query independent Andreas Kretschmer <[email protected]>
2017-07-10 15:48 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 18:35 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-11 12:46 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 02:11 ` Re: Very poor read performance, query independent Mark Kirkwood <[email protected]>
2017-07-14 23:09 ` Re: Very poor read performance, query independent Mark Kirkwood <[email protected]>
2017-07-14 23:57 ` Re: Very poor read performance, query independent Mark Kirkwood <[email protected]>
@ 2017-07-15 16:12 ` Charles Nadeau <[email protected]>
2017-07-15 23:48 ` Re: Very poor read performance, query independent Mark Kirkwood <[email protected]>
1 sibling, 1 reply; 52+ messages in thread
From: Charles Nadeau @ 2017-07-15 16:12 UTC (permalink / raw)
To: Mark Kirkwood <[email protected]>; +Cc: pgsql-performance
Mark,
The server is a HP DL 380 G6. It doesn't really work with SATA drives. And
when you find one that is compatible, it is only used at 3Gb/s with a
maximum of 50000 IOPS (a well know caracteristic of the HP P410i SAS RAID
controller). I am looking at getting a Kingston Digital HyperX Predator
that I could use in one of the PCIe 2.0 x4 slot. However I am worried about
the "thermal runaway", i.e. when the server can't get a temperature reading
from a PCIe card, it spins the fans at full speed to protect the server
against high temperature. The machine being next to my desk I worry about
the deafening noise it will create.
Thanks!
Chales
On Sat, Jul 15, 2017 at 1:57 AM, Mark Kirkwood <
[email protected]> wrote:
> Thinking about this a bit more - if somewhat more blazing performance is
> needed, then this could be achieved via losing the RAID card and spinning
> disks altogether and buying 1 of the NVME or SATA solid state products: e.g
>
> - Samsung 960 Pro or Evo 2 TB (approx 1 or 2 GB/s seq scan speeds and 200K
> IOPS)
>
> - Intel S3610 or similar 1.2 TB (500 MB/s seq scan and 30K IOPS)
>
>
> The Samsung needs an M.2 port on the mobo (but most should have 'em - and
> if not PCIe X4 adapter cards are quite cheap). The Intel is a bit more
> expensive compared to the Samsung, and is slower but has a longer lifetime.
> However for your workload the Sammy is probably fine.
>
> regards
>
> Mark
>
> On 15/07/17 11:09, Mark Kirkwood wrote:
>
>> Ah yes - that seems more sensible (but still slower than I would expect
>> for 5 disks RAID 0).
>>
>
>
>
> --
> Sent via pgsql-performance mailing list ([email protected])
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
>
--
Charles Nadeau Ph.D.
http://charlesnadeau.blogspot.com/
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 15:35 ` Re: Very poor read performance, query independent Andreas Kretschmer <[email protected]>
2017-07-10 15:48 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 18:35 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-11 12:46 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 02:11 ` Re: Very poor read performance, query independent Mark Kirkwood <[email protected]>
2017-07-14 23:09 ` Re: Very poor read performance, query independent Mark Kirkwood <[email protected]>
2017-07-14 23:57 ` Re: Very poor read performance, query independent Mark Kirkwood <[email protected]>
2017-07-15 16:12 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
@ 2017-07-15 23:48 ` Mark Kirkwood <[email protected]>
0 siblings, 0 replies; 52+ messages in thread
From: Mark Kirkwood @ 2017-07-15 23:48 UTC (permalink / raw)
To: Charles Nadeau <[email protected]>; +Cc: pgsql-performance
Right, that is a bit of a show stopper for those SSD (the Intel needs
SATA 6Gb/s and the Sammy's need PCIe 3.0 to perform to their rated specs).
regards
Mark
On 16/07/17 04:12, Charles Nadeau wrote:
> Mark,
>
> The server is a . It doesn't really work with SATA drives. And when
> you find one that is compatible, it is only used at 3Gb/s with a
> maximum of 50000 IOPS (a well know caracteristic of the HP P410i SAS
> RAID controller). I am looking at getting a Kingston Digital HyperX
> Predator that I could use in one of the PCIe 2.0 x4 slot. However I am
> worried about the "thermal runaway", i.e. when the server can't get a
> temperature reading from a PCIe card, it spins the fans at full speed
> to protect the server against high temperature. The machine being next
> to my desk I worry about the deafening noise it will create.
> Thanks!
>
> Chales
>
> On Sat, Jul 15, 2017 at 1:57 AM, Mark Kirkwood
> <[email protected] <mailto:[email protected]>>
> wrote:
>
> Thinking about this a bit more - if somewhat more blazing
> performance is needed, then this could be achieved via losing the
> RAID card and spinning disks altogether and buying 1 of the NVME
> or SATA solid state products: e.g
>
> - Samsung 960 Pro or Evo 2 TB (approx 1 or 2 GB/s seq scan speeds
> and 200K IOPS)
>
> - Intel S3610 or similar 1.2 TB (500 MB/s seq scan and 30K IOPS)
>
>
> The Samsung needs an M.2 port on the mobo (but most should have
> 'em - and if not PCIe X4 adapter cards are quite cheap). The Intel
> is a bit more expensive compared to the Samsung, and is slower but
> has a longer lifetime. However for your workload the Sammy is
> probably fine.
>
> regards
>
> Mark
>
> On 15/07/17 11:09, Mark Kirkwood wrote:
>
> Ah yes - that seems more sensible (but still slower than I
> would expect for 5 disks RAID 0).
>
>
>
>
> --
> Sent via pgsql-performance mailing list
> ([email protected]
> <mailto:[email protected]>)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
> <http://www.postgresql.org/mailpref/pgsql-performance;
>
>
>
>
> --
> Charles Nadeau Ph.D.
> http://charlesnadeau.blogspot.com/
--
Sent via pgsql-performance mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 15:35 ` Re: Very poor read performance, query independent Andreas Kretschmer <[email protected]>
2017-07-10 15:48 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 18:35 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-11 12:46 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 02:11 ` Re: Very poor read performance, query independent Mark Kirkwood <[email protected]>
2017-07-14 23:09 ` Re: Very poor read performance, query independent Mark Kirkwood <[email protected]>
2017-07-14 23:57 ` Re: Very poor read performance, query independent Mark Kirkwood <[email protected]>
@ 2017-07-20 12:50 ` Charles Nadeau <[email protected]>
2017-08-19 06:51 ` Re: Very poor read performance, query independent Mark Kirkwood <[email protected]>
1 sibling, 1 reply; 52+ messages in thread
From: Charles Nadeau @ 2017-07-20 12:50 UTC (permalink / raw)
To: Mark Kirkwood <[email protected]>; +Cc: pgsql-performance
Mark,
I received yesterday a second server having 16 drives bays. Just for a
quick trial, I used 2 old 60GB SSD (a Kingston V300 and a ADATA SP900) to
build a RAID0. To my surprise, my very pecky RAID controller (HP P410i)
recognised them without a fuss (although as SATAII drives at 3Gb/s. A quick
fio benchmark gives me 22000 random 4k read IOPS, more than my 5 146GB 10k
SAS disks in RAID0). I moved my most frequently used index to this array
and will try to do some benchmarks.
Knowing that SSDs based on SandForce-2281 controller are recognised by my
server, I may buy a pair of bigger/newer ones to put my tables on.
Thanks!
Charles
On Sat, Jul 15, 2017 at 1:57 AM, Mark Kirkwood <
[email protected]> wrote:
> Thinking about this a bit more - if somewhat more blazing performance is
> needed, then this could be achieved via losing the RAID card and spinning
> disks altogether and buying 1 of the NVME or SATA solid state products: e.g
>
> - Samsung 960 Pro or Evo 2 TB (approx 1 or 2 GB/s seq scan speeds and 200K
> IOPS)
>
> - Intel S3610 or similar 1.2 TB (500 MB/s seq scan and 30K IOPS)
>
>
> The Samsung needs an M.2 port on the mobo (but most should have 'em - and
> if not PCIe X4 adapter cards are quite cheap). The Intel is a bit more
> expensive compared to the Samsung, and is slower but has a longer lifetime.
> However for your workload the Sammy is probably fine.
>
> regards
>
> Mark
>
> On 15/07/17 11:09, Mark Kirkwood wrote:
>
>> Ah yes - that seems more sensible (but still slower than I would expect
>> for 5 disks RAID 0).
>>
>
>
>
> --
> Sent via pgsql-performance mailing list ([email protected])
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
>
--
Charles Nadeau Ph.D.
http://charlesnadeau.blogspot.com/
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 15:35 ` Re: Very poor read performance, query independent Andreas Kretschmer <[email protected]>
2017-07-10 15:48 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 18:35 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-11 12:46 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 02:11 ` Re: Very poor read performance, query independent Mark Kirkwood <[email protected]>
2017-07-14 23:09 ` Re: Very poor read performance, query independent Mark Kirkwood <[email protected]>
2017-07-14 23:57 ` Re: Very poor read performance, query independent Mark Kirkwood <[email protected]>
2017-07-20 12:50 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
@ 2017-08-19 06:51 ` Mark Kirkwood <[email protected]>
0 siblings, 0 replies; 52+ messages in thread
From: Mark Kirkwood @ 2017-08-19 06:51 UTC (permalink / raw)
To: Charles Nadeau <[email protected]>; +Cc: pgsql-performance
Nice!
Pleased that the general idea worked well for you!
I'm also relieved that you did not follow my recommendation exactly -
I'm been trialling a Samsung 960 Evo (256GB) and Intel 600p (256GB) and
I've stumbled across the serious disadvantages of (consumer) M.2 drives
using TLC NAND - terrible sustained write performance! While these guys
can happily do ~ 2GB/s reads, their write performance is only 'burst
capable'. They have small SLC NAND 'write caches' that do ~1GB/s for a
*limited time* (10-20s) and after that you get ~ 200 MB/s! Ouch - my old
Crucial 550 can do 350 MB/s sustained writes (so two of them in RAID0
are doing 700 MB/s for hours).
Bigger capacity drives can do better - but overall I'm not that
impressed with the current trend of using TLC NAND.
regards
Mark
On 21/07/17 00:50, Charles Nadeau wrote:
> Mark,
>
> I received yesterday a second server having 16 drives bays. Just for a
> quick trial, I used 2 old 60GB SSD (a Kingston V300 and a ADATA SP900)
> to build a RAID0. To my surprise, my very pecky RAID controller (HP
> P410i) recognised them without a fuss (although as SATAII drives at
> 3Gb/s. A quick fio benchmark gives me 22000 random 4k read IOPS, more
> than my 5 146GB 10k SAS disks in RAID0). I moved my most frequently
> used index to this array and will try to do some benchmarks.
> Knowing that SSDs based on SandForce-2281 controller are recognised by
> my server, I may buy a pair of bigger/newer ones to put my tables on.
>
> Thanks!
>
> Charles
>
> On Sat, Jul 15, 2017 at 1:57 AM, Mark Kirkwood
> <[email protected] <mailto:[email protected]>>
> wrote:
>
> Thinking about this a bit more - if somewhat more blazing
> performance is needed, then this could be achieved via losing the
> RAID card and spinning disks altogether and buying 1 of the NVME
> or SATA solid state products: e.g
>
> - Samsung 960 Pro or Evo 2 TB (approx 1 or 2 GB/s seq scan speeds
> and 200K IOPS)
>
> - Intel S3610 or similar 1.2 TB (500 MB/s seq scan and 30K IOPS)
>
>
> The Samsung needs an M.2 port on the mobo (but most should have
> 'em - and if not PCIe X4 adapter cards are quite cheap). The Intel
> is a bit more expensive compared to the Samsung, and is slower but
> has a longer lifetime. However for your workload the Sammy is
> probably fine.
>
> regards
>
> Mark
>
> On 15/07/17 11:09, Mark Kirkwood wrote:
>
> Ah yes - that seems more sensible (but still slower than I
> would expect for 5 disks RAID 0).
>
>
>
>
> --
> Sent via pgsql-performance mailing list
> ([email protected]
> <mailto:[email protected]>)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
> <http://www.postgresql.org/mailpref/pgsql-performance;
>
>
>
>
> --
> Charles Nadeau Ph.D.
> http://charlesnadeau.blogspot.com/
--
Sent via pgsql-performance mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 15:35 ` Re: Very poor read performance, query independent Andreas Kretschmer <[email protected]>
2017-07-10 15:48 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 18:35 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-11 12:46 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 02:11 ` Re: Very poor read performance, query independent Mark Kirkwood <[email protected]>
@ 2017-07-15 17:53 ` Charles Nadeau <[email protected]>
2017-07-15 17:58 ` Re: Very poor read performance, query independent Scott Marlowe <[email protected]>
2 siblings, 1 reply; 52+ messages in thread
From: Charles Nadeau @ 2017-07-15 17:53 UTC (permalink / raw)
To: Mark Kirkwood <[email protected]>; pgsql-performance
Mark,
I increased the read ahead to 16384 and it doesn't improve performance. My
RAID 0 use a stripe size of 256k, the maximum size supported by the
controller.
Thanks!
Charles
On Sat, Jul 15, 2017 at 1:02 AM, Mark Kirkwood <
[email protected]> wrote:
> Ah yes - that seems more sensible (but still slower than I would expect
> for 5 disks RAID 0). You should be able to get something like 5 * (single
> disk speed) i.e about 500MB/s.
>
> Might be worth increasing device read ahead (more than you have already).
> Some of these so-called 'smart' RAID cards need to be hit over the head
> before they will perform. E.g: I believe you have it set to 128 - I'd try
> 4096 or even 16384 (In the past I've used those settings on some extremely
> stupid cards that refused to max out their disks known speeds).
>
> Also worth investigating is RAID stripe size - for DW work it makes sense
> for it to be reasonably big (256K to 1M), which again will help speed is
> sequential scans.
>
> Cheers
>
> Mark
>
> P.s I used to work for Greenplum, so this type of problem came up a lot
> :-) . The best cards were the LSI and Areca!
>
>
>
> On 15/07/17 02:09, Charles Nadeau wrote:
>
>> Mark,
>>
>> First I must say that I changed my disks configuration from 4 disks in
>> RAID 10 to 5 disks in RAID 0 because I almost ran out of disk space during
>> the last ingest of data.
>> Here is the result test you asked. It was done with a cold cache:
>>
>> flows=# \timing
>> Timing is on.
>> flows=# explain select count(*) from flows;
>> QUERY PLAN
>> ------------------------------------------------------------
>> -----------------------------------
>> Finalize Aggregate (cost=17214914.09..17214914.09 rows=1 width=8)
>> -> Gather (cost=17214914.07..17214914.09 rows=1 width=8)
>> Workers Planned: 1
>> -> Partial Aggregate (cost=17213914.07..17213914.07
>> rows=1 width=8)
>> -> Parallel Seq Scan on flows
>> (cost=0.00..17019464.49 rows=388899162 width=0)
>> (5 rows)
>>
>> Time: 171.835 ms
>> flows=# select pg_relation_size('flows');
>> pg_relation_size
>> ------------------
>> 129865867264
>> (1 row)
>>
>> Time: 57.157 ms
>> flows=# select count(*) from flows;
>> LOG: duration: 625546.522 ms statement: select count(*) from flows;
>> count
>> -----------
>> 589831190
>> (1 row)
>>
>> Time: 625546.662 ms
>>
>> The throughput reported by Postgresql is almost 198MB/s, and the
>> throughput as mesured by dstat during the query execution was between 25
>> and 299MB/s. It is much better than what I had before! The i/o wait was
>> about 12% all through the query. One thing I noticed is the discrepency
>> between the read throughput reported by pg_activity and the one reported by
>> dstat: pg_activity always report a value lower than dstat.
>>
>> Besides the change of disks configuration, here is what contributed the
>> most to the improvment of the performance so far:
>>
>> Using Hugepage
>> Increasing effective_io_concurrency to 256
>> Reducing random_page_cost from 22 to 4
>> Reducing min_parallel_relation_size to 512kB to have more workers
>> when doing sequential parallel scan of my biggest table
>>
>>
>> Thanks for recomending this test, I now know what the real throughput
>> should be!
>>
>> Charles
>>
>> On Wed, Jul 12, 2017 at 4:11 AM, Mark Kirkwood <
>> [email protected] <mailto:[email protected]>>
>> wrote:
>>
>> Hmm - how are you measuring that sequential scan speed of 4MB/s?
>> I'd recommend doing a very simple test e.g, here's one on my
>> workstation - 13 GB single table on 1 SATA drive - cold cache
>> after reboot, sequential scan using Postgres 9.6.2:
>>
>> bench=# EXPLAIN SELECT count(*) FROM pgbench_accounts;
>> QUERY PLAN
>> ------------------------------------------------------------
>> ------------------------
>> Aggregate (cost=2889345.00..2889345.01 rows=1 width=8)
>> -> Seq Scan on pgbench_accounts (cost=0.00..2639345.00
>> rows=100000000 width=0)
>> (2 rows)
>>
>>
>> bench=# SELECT pg_relation_size('pgbench_accounts');
>> pg_relation_size
>> ------------------
>> 13429514240
>> (1 row)
>>
>> bench=# SELECT count(*) FROM pgbench_accounts;
>> count
>> -----------
>> 100000000
>> (1 row)
>>
>> Time: 118884.277 ms
>>
>>
>> So doing the math seq read speed is about 110MB/s (i.e 13 GB in
>> 120 sec). Sure enough, while I was running the query iostat showed:
>>
>> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
>> avgrq-sz avgqu-sz await r_await w_await svctm %util
>> sda 0.00 0.00 926.00 0.00 114.89 0.00
>> 254.10 1.90 2.03 2.03 0.00 1.08 100.00
>>
>>
>> So might be useful for us to see something like that from your
>> system - note you need to check you really have flushed the cache,
>> and that no other apps are using the db.
>>
>> regards
>>
>> Mark
>>
>>
>> On 12/07/17 00:46, Charles Nadeau wrote:
>>
>> After reducing random_page_cost to 4 and testing more, I can
>> report that the aggregate read throughput for parallel
>> sequential scan is about 90MB/s. However the throughput for
>> sequential scan is still around 4MB/s.
>>
>>
>>
>>
>>
>> --
>> Charles Nadeau Ph.D.
>> http://charlesnadeau.blogspot.com/
>>
>
>
--
Charles Nadeau Ph.D.
http://charlesnadeau.blogspot.com/
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 15:35 ` Re: Very poor read performance, query independent Andreas Kretschmer <[email protected]>
2017-07-10 15:48 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 18:35 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-11 12:46 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 02:11 ` Re: Very poor read performance, query independent Mark Kirkwood <[email protected]>
2017-07-15 17:53 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
@ 2017-07-15 17:58 ` Scott Marlowe <[email protected]>
2017-07-16 09:22 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
0 siblings, 1 reply; 52+ messages in thread
From: Scott Marlowe @ 2017-07-15 17:58 UTC (permalink / raw)
To: Charles Nadeau <[email protected]>; +Cc: Mark Kirkwood <[email protected]>; pgsql-performance
On Sat, Jul 15, 2017 at 11:53 AM, Charles Nadeau
<[email protected]> wrote:
> Mark,
>
> I increased the read ahead to 16384 and it doesn't improve performance. My
> RAID 0 use a stripe size of 256k, the maximum size supported by the
> controller.
Are your queries still spilling to disk for sorts? If this is the
case, and they're just too big to fit in memory, then you need to move
your temp space, where sorts happen, onto another disk array that
isn't your poor overworked raid-10 array. Contention between sorts and
reads can kill performance quick, esp on spinning rust.
--
Sent via pgsql-performance mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 15:35 ` Re: Very poor read performance, query independent Andreas Kretschmer <[email protected]>
2017-07-10 15:48 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 18:35 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-11 12:46 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 02:11 ` Re: Very poor read performance, query independent Mark Kirkwood <[email protected]>
2017-07-15 17:53 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-15 17:58 ` Re: Very poor read performance, query independent Scott Marlowe <[email protected]>
@ 2017-07-16 09:22 ` Charles Nadeau <[email protected]>
0 siblings, 0 replies; 52+ messages in thread
From: Charles Nadeau @ 2017-07-16 09:22 UTC (permalink / raw)
To: Scott Marlowe <[email protected]>; +Cc: Mark Kirkwood <[email protected]>; pgsql-performance
Scott,
The temp tablespace is on a disk of his own.
Thanks!
Charles
On Sat, Jul 15, 2017 at 7:58 PM, Scott Marlowe <[email protected]>
wrote:
> On Sat, Jul 15, 2017 at 11:53 AM, Charles Nadeau
> <[email protected]> wrote:
> > Mark,
> >
> > I increased the read ahead to 16384 and it doesn't improve performance.
> My
> > RAID 0 use a stripe size of 256k, the maximum size supported by the
> > controller.
>
> Are your queries still spilling to disk for sorts? If this is the
> case, and they're just too big to fit in memory, then you need to move
> your temp space, where sorts happen, onto another disk array that
> isn't your poor overworked raid-10 array. Contention between sorts and
> reads can kill performance quick, esp on spinning rust.
>
--
Charles Nadeau Ph.D.
http://charlesnadeau.blogspot.com/
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
@ 2017-07-10 19:24 ` Jeff Janes <[email protected]>
2017-07-11 11:02 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
4 siblings, 1 reply; 52+ messages in thread
From: Jeff Janes @ 2017-07-10 19:24 UTC (permalink / raw)
To: Charles Nadeau <[email protected]>; +Cc: pgsql-performance
On Mon, Jul 10, 2017 at 7:03 AM, Charles Nadeau <[email protected]>
wrote:
>
> The problem I have is very poor read. When I benchmark my array with fio I
> get random reads of about 200MB/s and 1100IOPS and sequential reads of
> about 286MB/s and 21000IPS.
>
That doesn't seem right. Sequential is only 43% faster? What job file are
giving to fio?
What do you get if you do something simpler, like:
time cat ~/$PGDATA/base/16402/*|wc -c
replacing 16402 with whatever your biggest database is.
Cheers,
Jeff
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 19:24 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
@ 2017-07-11 11:02 ` Charles Nadeau <[email protected]>
2017-07-12 00:39 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
0 siblings, 1 reply; 52+ messages in thread
From: Charles Nadeau @ 2017-07-11 11:02 UTC (permalink / raw)
To: Jeff Janes <[email protected]>; +Cc: pgsql-performance
Jeff,
I used fio in a quick benchmarking script inspired by
https://smcleod.net/benchmarking-io/:
#!/bin/bash
#Random throughput
echo "Random throughput"
sync
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test
--filename=test --bs=4M --iodepth=256 --size=10G --readwrite=randread
--ramp_time=4
#Random IOPS
echo "Random IOPS"
sync
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test
--filename=test --bs=4k --iodepth=256 --size=4G --readwrite=randread
--ramp_time=4
#Sequential throughput
echo "Sequential throughput"
sync
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test
--filename=test --bs=4M --iodepth=256 --size=10G --readwrite=read
--ramp_time=4
#Sequential IOPS
echo "Sequential IOPS"
sync
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test
--filename=test --bs=4k --iodepth=256 --size=4G --readwrite=read
--ramp_time=4
Performing the test you suggested, I get 128.5MB/s. Monitoring the test, I
find that the throughput is constant from start to finish and that the
iowait is also constant at 5%:
charles@hpdl380g6:~$ sudo sh -c 'time cat /mnt/data/postgresql/base/16385/*
| wc -c'
[sudo] password for charles:
1.62user 179.94system 29:50.79elapsed 10%CPU (0avgtext+0avgdata
1920maxresident)k
448026264inputs+0outputs (0major+117minor)pagefaults 0swaps
241297594904
After making the changes to HugePage suggested by Rick Otten (above), I
found slightly better results (135.7MB/s):
charles@hpdl380g6:~$ sudo sh -c 'time cat /mnt/data/postgresql/base/16385/*
| wc -c'
[sudo] password for charles:
0.86user 130.84system 28:15.78elapsed 7%CPU (0avgtext+0avgdata
1820maxresident)k
471286792inputs+0outputs (1major+118minor)pagefaults 0swaps
241297594904
Could you suggest another way to benchmark random reads?
Thanks for your help!
Charles
On Mon, Jul 10, 2017 at 9:24 PM, Jeff Janes <[email protected]> wrote:
> On Mon, Jul 10, 2017 at 7:03 AM, Charles Nadeau <[email protected]>
> wrote:
>
>>
>> The problem I have is very poor read. When I benchmark my array with fio
>> I get random reads of about 200MB/s and 1100IOPS and sequential reads of
>> about 286MB/s and 21000IPS.
>>
>
>
> That doesn't seem right. Sequential is only 43% faster? What job file
> are giving to fio?
>
> What do you get if you do something simpler, like:
>
> time cat ~/$PGDATA/base/16402/*|wc -c
>
> replacing 16402 with whatever your biggest database is.
>
> Cheers,
>
> Jeff
>
--
Charles Nadeau Ph.D.
http://charlesnadeau.blogspot.com/
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 19:24 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-11 11:02 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
@ 2017-07-12 00:39 ` Jeff Janes <[email protected]>
2017-07-12 10:04 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
0 siblings, 1 reply; 52+ messages in thread
From: Jeff Janes @ 2017-07-12 00:39 UTC (permalink / raw)
To: Charles Nadeau <[email protected]>; +Cc: pgsql-performance
On Tue, Jul 11, 2017 at 4:02 AM, Charles Nadeau <[email protected]>
wrote:
> Jeff,
>
> I used fio in a quick benchmarking script inspired by https://smcleod.net/
> benchmarking-io/:
>
> #!/bin/bash
> #Random throughput
> echo "Random throughput"
> sync
> fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1
> --name=test --filename=test --bs=4M --iodepth=256 --size=10G
> --readwrite=randread --ramp_time=4
> #Random IOPS
> echo "Random IOPS"
> sync
> fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1
> --name=test --filename=test --bs=4k --iodepth=256 --size=4G
> --readwrite=randread --ramp_time=4
> #Sequential throughput
> echo "Sequential throughput"
> sync
> fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1
> --name=test --filename=test --bs=4M --iodepth=256 --size=10G
> --readwrite=read --ramp_time=4
> #Sequential IOPS
> echo "Sequential IOPS"
> sync
> fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1
> --name=test --filename=test --bs=4k --iodepth=256 --size=4G
> --readwrite=read --ramp_time=4
>
>
I don't think any of those are directly relevant to PostgreSQL, as it
doesn't use direct IO, doesn't use libaio, and is rarely going to get
anywhere near 256 iodepth. So the best they can do is put a theoretical
ceiling on the performance. Also, random IO with a 4MB stride doesn't make
any sense from a PostgreSQL perspective.
>
> Performing the test you suggested, I get 128.5MB/s. Monitoring the test, I
> find that the throughput is constant from start to finish and that the
> iowait is also constant at 5%:
>
I would have expected it to do better than that. Maybe you increase the
kernel readahead setting. I've found the default to be much too small.
But it doesn't make much difference to you, as you appear to be doing
random IO in your queries, not sequential.
> Could you suggest another way to benchmark random reads?
>
Your 1100 IOPS times 8kb block size gives about 8MB/s of throughput, which
is close to what you report. So I think I'd would instead focus on tuning
your actual queries. You say the problem is not query-dependent, but I
think that that just means all the queries you looked at are similar. If
you looked at a query that can't use indexes, like count(unindexed_column)
from biggest_table; you would find it doing much more IO than 4MB/s.
Can you pick the simplest query you actually care about, and post both an
"explain (analyze, timing off)" and an "explain (analyze, buffers)" for it?
(Preferably turning "track_io_timing" on first).
One other question I had, you said you had "2x Intel Xeon E5550", which
should be 8 CPU (or 16, if the hyperthreads
are reported as separate CPUs). But you also said: "Also using dstat I can
see that iowait time is at about 25%". Usually if there is only one thing
going on on the server, then IOWAIT won't be more than reciprocal of #CPU.
Is the server busy doing other stuff at the same time you are benchmarking
it?
Cheers,
Jeff
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 19:24 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-11 11:02 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 00:39 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
@ 2017-07-12 10:04 ` Charles Nadeau <[email protected]>
2017-07-12 14:31 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-12 22:27 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
0 siblings, 2 replies; 52+ messages in thread
From: Charles Nadeau @ 2017-07-12 10:04 UTC (permalink / raw)
To: Jeff Janes <[email protected]>; +Cc: pgsql-performance
Jeff,
Here are the 2 EXPLAINs for one of my simplest query:
flows=# SET track_io_timing = on;
LOG: duration: 24.101 ms statement: SET track_io_timing = on;
SET
flows=# explain (analyze, timing off) SELECT DISTINCT
flows-# srcaddr,
flows-# dstaddr,
flows-# dstport,
flows-# COUNT(*) AS conversation,
flows-# SUM(doctets) / 1024 / 1024 AS mbytes
flows-# FROM
flows-# flowscompact,
flows-# mynetworks
flows-# WHERE
flows-# mynetworks.ipaddr >>= flowscompact.srcaddr
flows-# AND dstaddr IN
flows-# (
flows(# SELECT
flows(# dstaddr
flows(# FROM
flows(# dstexterne
flows(# )
flows-# GROUP BY
flows-# srcaddr,
flows-# dstaddr,
flows-# dstport
flows-# ORDER BY
flows-# mbytes DESC LIMIT 50;
LOG: temporary file: path
"pg_tblspc/36238/PG_9.6_201608131/pgsql_tmp/pgsql_tmp14573.3", size
1073741824
LOG: temporary file: path
"pg_tblspc/36238/PG_9.6_201608131/pgsql_tmp/pgsql_tmp14573.4", size
1073741824
LOG: temporary file: path
"pg_tblspc/36238/PG_9.6_201608131/pgsql_tmp/pgsql_tmp14573.5", size
639696896
LOG: duration: 2632108.352 ms statement: explain (analyze, timing off)
SELECT DISTINCT
srcaddr,
dstaddr,
dstport,
COUNT(*) AS conversation,
SUM(doctets) / 1024 / 1024 AS mbytes
FROM
flowscompact,
mynetworks
WHERE
mynetworks.ipaddr >>= flowscompact.srcaddr
AND dstaddr IN
(
SELECT
dstaddr
FROM
dstexterne
)
GROUP BY
srcaddr,
dstaddr,
dstport
ORDER BY
mbytes DESC LIMIT 50;
QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=37762321.83..37762321.98 rows=50 width=52) (actual rows=50
loops=1)
-> Unique (cost=37762321.83..37769053.57 rows=2243913 width=52)
(actual rows=50 loops=1)
-> Sort (cost=37762321.83..37763443.79 rows=2243913 width=52)
(actual rows=50 loops=1)
Sort Key: (((sum(flows.doctets) / '1024'::numeric) /
'1024'::numeric)) DESC, flows.srcaddr, flows.dstaddr, flows.dstport,
(count(*))
Sort Method: quicksort Memory: 563150kB
-> GroupAggregate (cost=37698151.34..37714980.68
rows=2243913 width=52) (actual rows=4691734 loops=1)
Group Key: flows.srcaddr, flows.dstaddr, flows.dstport
-> Sort (cost=37698151.34..37699273.29 rows=2243913
width=20) (actual rows=81896988 loops=1)
Sort Key: flows.srcaddr, flows.dstaddr,
flows.dstport
Sort Method: external merge Disk: 2721856kB
-> Gather (cost=19463936.00..37650810.19
rows=2243913 width=20) (actual rows=81896988 loops=1)
Workers Planned: 9
Workers Launched: 9
-> Hash Semi Join
(cost=19462936.00..37622883.23 rows=249324 width=20) (actual rows=8189699
loops=10)
Hash Cond: (flows.dstaddr =
flows_1.dstaddr)
-> Nested Loop
(cost=0.03..18159012.30 rows=249324 width=20) (actual rows=45499045
loops=10)
-> Parallel Seq Scan on flows
(cost=0.00..16039759.79 rows=62330930 width=20) (actual rows=54155970
loops=10)
-> Index Only Scan using
mynetworks_ipaddr_idx on mynetworks (cost=0.03..0.03 rows=1 width=8)
(actual rows=1 loops=541559704)
Index Cond: (ipaddr >>=
(flows.srcaddr)::ip4r)
Heap Fetches: 48679396
-> Hash
(cost=19462896.74..19462896.74 rows=11210 width=4) (actual rows=3099798
loops=10)
Buckets: 4194304 (originally
16384) Batches: 1 (originally 1) Memory Usage: 141746kB
-> HashAggregate
(cost=19462829.48..19462863.11 rows=11210 width=4) (actual rows=3099798
loops=10)
Group Key:
flows_1.dstaddr
-> Nested Loop Anti
Join (cost=0.12..19182620.78 rows=560417390 width=4) (actual
rows=113420172 loops=10)
Join Filter:
(mynetworks_1.ipaddr >> (flows_1.dstaddr)::ip4r)
Rows Removed by
Join Filter: 453681377
-> Index Only
Scan using flows_srcaddr_dstaddr_idx on flows flows_1
(cost=0.12..9091067.70 rows=560978368 width=4) (actual rows=541559704
loops=10)
Heap
Fetches: 91
-> Materialize
(cost=0.00..1.02 rows=4 width=8) (actual rows=2 loops=5415597040)
-> Seq Scan
on mynetworks mynetworks_1 (cost=0.00..1.01 rows=4 width=8) (actual rows=4
loops=10)
Planning time: 62.066 ms
Execution time: 2631923.716 ms
(33 rows)
flows=# explain (analyze, buffers) SELECT DISTINCT
flows-# srcaddr,
flows-# dstaddr,
flows-# dstport,
flows-# COUNT(*) AS conversation,
flows-# SUM(doctets) / 1024 / 1024 AS mbytes
flows-# FROM
flows-# flowscompact,
flows-# mynetworks
flows-# WHERE
flows-# mynetworks.ipaddr >>= flowscompact.srcaddr
flows-# AND dstaddr IN
flows-# (
flows(# SELECT
flows(# dstaddr
flows(# FROM
flows(# dstexterne
flows(# )
flows-# GROUP BY
flows-# srcaddr,
flows-# dstaddr,
flows-# dstport
flows-# ORDER BY
flows-# mbytes DESC LIMIT 50;
LOG: temporary file: path
"pg_tblspc/36238/PG_9.6_201608131/pgsql_tmp/pgsql_tmp14573.6", size
1073741824
LOG: temporary file: path
"pg_tblspc/36238/PG_9.6_201608131/pgsql_tmp/pgsql_tmp14573.7", size
1073741824
LOG: temporary file: path
"pg_tblspc/36238/PG_9.6_201608131/pgsql_tmp/pgsql_tmp14573.8", size
639696896
LOG: duration: 2765020.327 ms statement: explain (analyze, buffers)
SELECT DISTINCT
srcaddr,
dstaddr,
dstport,
COUNT(*) AS conversation,
SUM(doctets) / 1024 / 1024 AS mbytes
FROM
flowscompact,
mynetworks
WHERE
mynetworks.ipaddr >>= flowscompact.srcaddr
AND dstaddr IN
(
SELECT
dstaddr
FROM
dstexterne
)
GROUP BY
srcaddr,
dstaddr,
dstport
ORDER BY
mbytes DESC LIMIT 50;
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=37762321.83..37762321.98 rows=50 width=52) (actual
time=2764548.863..2764548.891 rows=50 loops=1)
Buffers: shared hit=1116590560 read=15851133, temp read=340244
written=340244
I/O Timings: read=5323746.860
-> Unique (cost=37762321.83..37769053.57 rows=2243913 width=52)
(actual time=2764548.861..2764548.882 rows=50 loops=1)
Buffers: shared hit=1116590560 read=15851133, temp read=340244
written=340244
I/O Timings: read=5323746.860
-> Sort (cost=37762321.83..37763443.79 rows=2243913 width=52)
(actual time=2764548.859..2764548.872 rows=50 loops=1)
Sort Key: (((sum(flows.doctets) / '1024'::numeric) /
'1024'::numeric)) DESC, flows.srcaddr, flows.dstaddr, flows.dstport,
(count(*))
Sort Method: quicksort Memory: 563150kB
Buffers: shared hit=1116590560 read=15851133, temp
read=340244 written=340244
I/O Timings: read=5323746.860
-> GroupAggregate (cost=37698151.34..37714980.68
rows=2243913 width=52) (actual time=2696721.610..2752109.551 rows=4691734
loops=1)
Group Key: flows.srcaddr, flows.dstaddr, flows.dstport
Buffers: shared hit=1116590560 read=15851133, temp
read=340244 written=340244
I/O Timings: read=5323746.860
-> Sort (cost=37698151.34..37699273.29 rows=2243913
width=20) (actual time=2696711.428..2732781.705 rows=81896988 loops=1)
Sort Key: flows.srcaddr, flows.dstaddr,
flows.dstport
Sort Method: external merge Disk: 2721856kB
Buffers: shared hit=1116590560 read=15851133,
temp read=340244 written=340244
I/O Timings: read=5323746.860
-> Gather (cost=19463936.00..37650810.19
rows=2243913 width=20) (actual time=1777219.713..2590530.887 rows=81896988
loops=1)
Workers Planned: 9
Workers Launched: 9
Buffers: shared hit=1116590559
read=15851133
I/O Timings: read=5323746.860
-> Hash Semi Join
(cost=19462936.00..37622883.23 rows=249324 width=20) (actual
time=1847579.360..2602039.780 rows=8189699 loops=10)
Hash Cond: (flows.dstaddr =
flows_1.dstaddr)
Buffers: shared hit=1116588309
read=15851133
I/O Timings: read=5323746.860
-> Nested Loop
(cost=0.03..18159012.30 rows=249324 width=20) (actual
time=1.562..736556.583 rows=45499045 loops=10)
Buffers: shared hit=996551813
read=15851133
I/O Timings: read=5323746.860
-> Parallel Seq Scan on flows
(cost=0.00..16039759.79 rows=62330930 width=20) (actual
time=1.506..547485.066 rows=54155970 loops=10)
Buffers: shared hit=1634
read=15851133
I/O Timings:
read=5323746.860
-> Index Only Scan using
mynetworks_ipaddr_idx on mynetworks (cost=0.03..0.03 rows=1 width=8)
(actual time=0.002..0.002 rows=1 loops=541559704)
Index Cond: (ipaddr >>=
(flows.srcaddr)::ip4r)
Heap Fetches: 59971474
Buffers: shared
hit=996550152
-> Hash
(cost=19462896.74..19462896.74 rows=11210 width=4) (actual
time=1847228.894..1847228.894 rows=3099798 loops=10)
Buckets: 4194304 (originally
16384) Batches: 1 (originally 1) Memory Usage: 141746kB
Buffers: shared hit=120036496
-> HashAggregate
(cost=19462829.48..19462863.11 rows=11210 width=4) (actual
time=1230049.015..1845955.764 rows=3099798 loops=10)
Group Key:
flows_1.dstaddr
Buffers: shared
hit=120036496
-> Nested Loop Anti
Join (cost=0.12..19182620.78 rows=560417390 width=4) (actual
time=0.084..831832.333 rows=113420172 loops=10)
Join Filter:
(mynetworks_1.ipaddr >> (flows_1.dstaddr)::ip4r)
Rows Removed by
Join Filter: 453681377
Buffers: shared
hit=120036496
-> Index Only
Scan using flows_srcaddr_dstaddr_idx on flows flows_1
(cost=0.12..9091067.70 rows=560978368 width=4) (actual
time=0.027..113052.437 rows=541559704 loops=10)
Heap
Fetches: 91
Buffers:
shared hit=120036459
-> Materialize
(cost=0.00..1.02 rows=4 width=8) (actual time=0.000..0.000 rows=2
loops=5415597040)
Buffers:
shared hit=10
-> Seq Scan
on mynetworks mynetworks_1 (cost=0.00..1.01 rows=4 width=8) (actual
time=0.007..0.008 rows=4 loops=10)
Buffers: shared hit=10
Planning time: 6.689 ms
Execution time: 2764860.853 ms
(58 rows)
Regarding "Also using dstat I can see that iowait time is at about 25%", I
don't think the server was doing anything else. If it is important, I can
repeat the benchmarks.
Thanks!
Charles
On Wed, Jul 12, 2017 at 2:39 AM, Jeff Janes <[email protected]> wrote:
> On Tue, Jul 11, 2017 at 4:02 AM, Charles Nadeau <[email protected]>
> wrote:
>
>> Jeff,
>>
>> I used fio in a quick benchmarking script inspired by
>> https://smcleod.net/benchmarking-io/:
>>
>> #!/bin/bash
>> #Random throughput
>> echo "Random throughput"
>> sync
>> fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1
>> --name=test --filename=test --bs=4M --iodepth=256 --size=10G
>> --readwrite=randread --ramp_time=4
>> #Random IOPS
>> echo "Random IOPS"
>> sync
>> fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1
>> --name=test --filename=test --bs=4k --iodepth=256 --size=4G
>> --readwrite=randread --ramp_time=4
>> #Sequential throughput
>> echo "Sequential throughput"
>> sync
>> fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1
>> --name=test --filename=test --bs=4M --iodepth=256 --size=10G
>> --readwrite=read --ramp_time=4
>> #Sequential IOPS
>> echo "Sequential IOPS"
>> sync
>> fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1
>> --name=test --filename=test --bs=4k --iodepth=256 --size=4G
>> --readwrite=read --ramp_time=4
>>
>>
> I don't think any of those are directly relevant to PostgreSQL, as it
> doesn't use direct IO, doesn't use libaio, and is rarely going to get
> anywhere near 256 iodepth. So the best they can do is put a theoretical
> ceiling on the performance. Also, random IO with a 4MB stride doesn't make
> any sense from a PostgreSQL perspective.
>
>
>
>>
>> Performing the test you suggested, I get 128.5MB/s. Monitoring the test,
>> I find that the throughput is constant from start to finish and that the
>> iowait is also constant at 5%:
>>
>
> I would have expected it to do better than that. Maybe you increase the
> kernel readahead setting. I've found the default to be much too small.
> But it doesn't make much difference to you, as you appear to be doing
> random IO in your queries, not sequential.
>
>
>> Could you suggest another way to benchmark random reads?
>>
>
> Your 1100 IOPS times 8kb block size gives about 8MB/s of throughput, which
> is close to what you report. So I think I'd would instead focus on tuning
> your actual queries. You say the problem is not query-dependent, but I
> think that that just means all the queries you looked at are similar. If
> you looked at a query that can't use indexes, like count(unindexed_column)
> from biggest_table; you would find it doing much more IO than 4MB/s.
>
> Can you pick the simplest query you actually care about, and post both an
> "explain (analyze, timing off)" and an "explain (analyze, buffers)" for it?
> (Preferably turning "track_io_timing" on first).
>
> One other question I had, you said you had "2x Intel Xeon E5550", which
> should be 8 CPU (or 16, if the hyperthreads
> are reported as separate CPUs). But you also said: "Also using dstat I
> can see that iowait time is at about 25%". Usually if there is only one
> thing going on on the server, then IOWAIT won't be more than reciprocal of
> #CPU. Is the server busy doing other stuff at the same time you are
> benchmarking it?
>
> Cheers,
>
> Jeff
>
--
Charles Nadeau Ph.D.
http://charlesnadeau.blogspot.com/
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 19:24 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-11 11:02 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 00:39 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-12 10:04 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
@ 2017-07-12 14:31 ` Igor Neyman <[email protected]>
2017-07-12 16:39 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
1 sibling, 1 reply; 52+ messages in thread
From: Igor Neyman @ 2017-07-12 14:31 UTC (permalink / raw)
To: Charles Nadeau <[email protected]>; Jeff Janes <[email protected]>; +Cc: pgsql-performance
From: [email protected] [mailto:[email protected]] On Behalf Of Charles Nadeau
Sent: Wednesday, July 12, 2017 6:05 AM
To: Jeff Janes <[email protected]>
Cc: [email protected]
Subject: Re: [PERFORM] Very poor read performance, query independent
flows=# explain (analyze, buffers) SELECT DISTINCT
flows-# srcaddr,
flows-# dstaddr,
flows-# dstport,
flows-# COUNT(*) AS conversation,
flows-# SUM(doctets) / 1024 / 1024 AS mbytes
flows-# FROM
flows-# flowscompact,
flows-# mynetworks
flows-# WHERE
flows-# mynetworks.ipaddr >>= flowscompact.srcaddr
flows-# AND dstaddr IN
flows-# (
flows(# SELECT
flows(# dstaddr
flows(# FROM
flows(# dstexterne
flows(# )
flows-# GROUP BY
flows-# srcaddr,
flows-# dstaddr,
flows-# dstport
flows-# ORDER BY
flows-# mbytes DESC LIMIT 50;
LOG: temporary file: path "pg_tblspc/36238/PG_9.6_201608131/pgsql_tmp/pgsql_tmp14573.6", size 1073741824
LOG: temporary file: path "pg_tblspc/36238/PG_9.6_201608131/pgsql_tmp/pgsql_tmp14573.7", size 1073741824
LOG: temporary file: path "pg_tblspc/36238/PG_9.6_201608131/pgsql_tmp/pgsql_tmp14573.8", size 639696896
LOG: duration: 2765020.327 ms statement: explain (analyze, buffers) SELECT DISTINCT
srcaddr,
dstaddr,
dstport,
COUNT(*) AS conversation,
SUM(doctets) / 1024 / 1024 AS mbytes
FROM
flowscompact,
mynetworks
WHERE
mynetworks.ipaddr >>= flowscompact.srcaddr
AND dstaddr IN
(
SELECT
dstaddr
FROM
dstexterne
)
GROUP BY
srcaddr,
dstaddr,
dstport
ORDER BY
mbytes DESC LIMIT 50;
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=37762321.83..37762321.98 rows=50 width=52) (actual time=2764548.863..2764548.891 rows=50 loops=1)
Buffers: shared hit=1116590560 read=15851133, temp read=340244 written=340244
I/O Timings: read=5323746.860
-> Unique (cost=37762321.83..37769053.57 rows=2243913 width=52) (actual time=2764548.861..2764548.882 rows=50 loops=1)
Buffers: shared hit=1116590560 read=15851133, temp read=340244 written=340244
I/O Timings: read=5323746.860
-> Sort (cost=37762321.83..37763443.79 rows=2243913 width=52) (actual time=2764548.859..2764548.872 rows=50 loops=1)
Sort Key: (((sum(flows.doctets) / '1024'::numeric) / '1024'::numeric)) DESC, flows.srcaddr, flows.dstaddr, flows.dstport, (count(*))
Sort Method: quicksort Memory: 563150kB
Buffers: shared hit=1116590560 read=15851133, temp read=340244 written=340244
I/O Timings: read=5323746.860
-> GroupAggregate (cost=37698151.34..37714980.68 rows=2243913 width=52) (actual time=2696721.610..2752109.551 rows=4691734 loops=1)
Group Key: flows.srcaddr, flows.dstaddr, flows.dstport
Buffers: shared hit=1116590560 read=15851133, temp read=340244 written=340244
I/O Timings: read=5323746.860
-> Sort (cost=37698151.34..37699273.29 rows=2243913 width=20) (actual time=2696711.428..2732781.705 rows=81896988 loops=1)
Sort Key: flows.srcaddr, flows.dstaddr, flows.dstport
Sort Method: external merge Disk: 2721856kB
Buffers: shared hit=1116590560 read=15851133, temp read=340244 written=340244
I/O Timings: read=5323746.860
-> Gather (cost=19463936.00..37650810.19 rows=2243913 width=20) (actual time=1777219.713..2590530.887 rows=81896988 loops=1)
Workers Planned: 9
Workers Launched: 9
Buffers: shared hit=1116590559 read=15851133
I/O Timings: read=5323746.860
-> Hash Semi Join (cost=19462936.00..37622883.23 rows=249324 width=20) (actual time=1847579.360..2602039.780 rows=8189699 loops=10)
Hash Cond: (flows.dstaddr = flows_1.dstaddr)
Buffers: shared hit=1116588309 read=15851133
I/O Timings: read=5323746.860
-> Nested Loop (cost=0.03..18159012.30 rows=249324 width=20) (actual time=1.562..736556.583 rows=45499045 loops=10)
Buffers: shared hit=996551813 read=15851133
I/O Timings: read=5323746.860
-> Parallel Seq Scan on flows (cost=0.00..16039759.79 rows=62330930 width=20) (actual time=1.506..547485.066 rows=54155970 loops=10)
Buffers: shared hit=1634 read=15851133
I/O Timings: read=5323746.860
-> Index Only Scan using mynetworks_ipaddr_idx on mynetworks (cost=0.03..0.03 rows=1 width=8) (actual time=0.002..0.002 rows=1 loops=541559704)
Index Cond: (ipaddr >>= (flows.srcaddr)::ip4r)
Heap Fetches: 59971474
Buffers: shared hit=996550152
-> Hash (cost=19462896.74..19462896.74 rows=11210 width=4) (actual time=1847228.894..1847228.894 rows=3099798 loops=10)
Buckets: 4194304 (originally 16384) Batches: 1 (originally 1) Memory Usage: 141746kB
Buffers: shared hit=120036496
-> HashAggregate (cost=19462829.48..19462863.11 rows=11210 width=4) (actual time=1230049.015..1845955.764 rows=3099798 loops=10)
Group Key: flows_1.dstaddr
Buffers: shared hit=120036496
-> Nested Loop Anti Join (cost=0.12..19182620.78 rows=560417390 width=4) (actual time=0.084..831832.333 rows=113420172 loops=10)
Join Filter: (mynetworks_1.ipaddr >> (flows_1.dstaddr)::ip4r)
Rows Removed by Join Filter: 453681377
Buffers: shared hit=120036496
-> Index Only Scan using flows_srcaddr_dstaddr_idx on flows flows_1 (cost=0.12..9091067.70 rows=560978368 width=4) (actual time=0.027..113052.437 rows=541559704 loops=10)
Heap Fetches: 91
Buffers: shared hit=120036459
-> Materialize (cost=0.00..1.02 rows=4 width=8) (actual time=0.000..0.000 rows=2 loops=5415597040)
Buffers: shared hit=10
-> Seq Scan on mynetworks mynetworks_1 (cost=0.00..1.01 rows=4 width=8) (actual time=0.007..0.008 rows=4 loops=10)
Buffers: shared hit=10
Planning time: 6.689 ms
Execution time: 2764860.853 ms
(58 rows)
Regarding "Also using dstat I can see that iowait time is at about 25%", I don't think the server was doing anything else. If it is important, I can repeat the benchmarks.
Thanks!
Charles
Charles,
In your original posting I couldn’t find what value you set for temp_buffers.
Considering you have plenty of RAM, try setting temp_buffers=’6GB’ and then run ‘explain (analyze, buffers) select…’ in the same session. This should alleviate “disk sort’ problem.
Also, could you post the structure of flowscompact, mynetworks, and dstextern tables with all the indexes and number of rows. Actually, are they all – tables, or some of them – views?
Igor
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 19:24 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-11 11:02 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 00:39 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-12 10:04 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 14:31 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
@ 2017-07-12 16:39 ` Igor Neyman <[email protected]>
2017-07-14 15:34 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
0 siblings, 1 reply; 52+ messages in thread
From: Igor Neyman @ 2017-07-12 16:39 UTC (permalink / raw)
To: Igor Neyman <[email protected]>; Charles Nadeau <[email protected]>; Jeff Janes <[email protected]>; +Cc: pgsql-performance
From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Charles Nadeau
Sent: Wednesday, July 12, 2017 6:05 AM
To: Jeff Janes <[email protected]<mailto:[email protected]>>
Cc: [email protected]<mailto:[email protected]>
Subject: Re: [PERFORM] Very poor read performance, query independent
flows=# explain (analyze, buffers) SELECT DISTINCT
flows-# srcaddr,
flows-# dstaddr,
flows-# dstport,
flows-# COUNT(*) AS conversation,
flows-# SUM(doctets) / 1024 / 1024 AS mbytes
flows-# FROM
flows-# flowscompact,
flows-# mynetworks
flows-# WHERE
flows-# mynetworks.ipaddr >>= flowscompact.srcaddr
flows-# AND dstaddr IN
flows-# (
flows(# SELECT
flows(# dstaddr
flows(# FROM
flows(# dstexterne
flows(# )
flows-# GROUP BY
flows-# srcaddr,
flows-# dstaddr,
flows-# dstport
flows-# ORDER BY
flows-# mbytes DESC LIMIT 50;
LOG: temporary file: path "pg_tblspc/36238/PG_9.6_201608131/pgsql_tmp/pgsql_tmp14573.6", size 1073741824
LOG: temporary file: path "pg_tblspc/36238/PG_9.6_201608131/pgsql_tmp/pgsql_tmp14573.7", size 1073741824
LOG: temporary file: path "pg_tblspc/36238/PG_9.6_201608131/pgsql_tmp/pgsql_tmp14573.8", size 639696896
LOG: duration: 2765020.327 ms statement: explain (analyze, buffers) SELECT DISTINCT
srcaddr,
dstaddr,
dstport,
COUNT(*) AS conversation,
SUM(doctets) / 1024 / 1024 AS mbytes
FROM
flowscompact,
mynetworks
WHERE
mynetworks.ipaddr >>= flowscompact.srcaddr
AND dstaddr IN
(
SELECT
dstaddr
FROM
dstexterne
)
GROUP BY
srcaddr,
dstaddr,
dstport
ORDER BY
mbytes DESC LIMIT 50;
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=37762321.83..37762321.98 rows=50 width=52) (actual time=2764548.863..2764548.891 rows=50 loops=1)
Buffers: shared hit=1116590560 read=15851133, temp read=340244 written=340244
I/O Timings: read=5323746.860
-> Unique (cost=37762321.83..37769053.57 rows=2243913 width=52) (actual time=2764548.861..2764548.882 rows=50 loops=1)
Buffers: shared hit=1116590560 read=15851133, temp read=340244 written=340244
I/O Timings: read=5323746.860
-> Sort (cost=37762321.83..37763443.79 rows=2243913 width=52) (actual time=2764548.859..2764548.872 rows=50 loops=1)
Sort Key: (((sum(flows.doctets) / '1024'::numeric) / '1024'::numeric)) DESC, flows.srcaddr, flows.dstaddr, flows.dstport, (count(*))
Sort Method: quicksort Memory: 563150kB
Buffers: shared hit=1116590560 read=15851133, temp read=340244 written=340244
I/O Timings: read=5323746.860
-> GroupAggregate (cost=37698151.34..37714980.68 rows=2243913 width=52) (actual time=2696721.610..2752109.551 rows=4691734 loops=1)
Group Key: flows.srcaddr, flows.dstaddr, flows.dstport
Buffers: shared hit=1116590560 read=15851133, temp read=340244 written=340244
I/O Timings: read=5323746.860
-> Sort (cost=37698151.34..37699273.29 rows=2243913 width=20) (actual time=2696711.428..2732781.705 rows=81896988 loops=1)
Sort Key: flows.srcaddr, flows.dstaddr, flows.dstport
Sort Method: external merge Disk: 2721856kB
Buffers: shared hit=1116590560 read=15851133, temp read=340244 written=340244
I/O Timings: read=5323746.860
-> Gather (cost=19463936.00..37650810.19 rows=2243913 width=20) (actual time=1777219.713..2590530.887 rows=81896988 loops=1)
Workers Planned: 9
Workers Launched: 9
Buffers: shared hit=1116590559 read=15851133
I/O Timings: read=5323746.860
-> Hash Semi Join (cost=19462936.00..37622883.23 rows=249324 width=20) (actual time=1847579.360..2602039.780 rows=8189699 loops=10)
Hash Cond: (flows.dstaddr = flows_1.dstaddr)
Buffers: shared hit=1116588309 read=15851133
I/O Timings: read=5323746.860
-> Nested Loop (cost=0.03..18159012.30 rows=249324 width=20) (actual time=1.562..736556.583 rows=45499045 loops=10)
Buffers: shared hit=996551813 read=15851133
I/O Timings: read=5323746.860
-> Parallel Seq Scan on flows (cost=0.00..16039759.79 rows=62330930 width=20) (actual time=1.506..547485.066 rows=54155970 loops=10)
Buffers: shared hit=1634 read=15851133
I/O Timings: read=5323746.860
-> Index Only Scan using mynetworks_ipaddr_idx on mynetworks (cost=0.03..0.03 rows=1 width=8) (actual time=0.002..0.002 rows=1 loops=541559704)
Index Cond: (ipaddr >>= (flows.srcaddr)::ip4r)
Heap Fetches: 59971474
Buffers: shared hit=996550152
-> Hash (cost=19462896.74..19462896.74 rows=11210 width=4) (actual time=1847228.894..1847228.894 rows=3099798 loops=10)
Buckets: 4194304 (originally 16384) Batches: 1 (originally 1) Memory Usage: 141746kB
Buffers: shared hit=120036496
-> HashAggregate (cost=19462829.48..19462863.11 rows=11210 width=4) (actual time=1230049.015..1845955.764 rows=3099798 loops=10)
Group Key: flows_1.dstaddr
Buffers: shared hit=120036496
-> Nested Loop Anti Join (cost=0.12..19182620.78 rows=560417390 width=4) (actual time=0.084..831832.333 rows=113420172 loops=10)
Join Filter: (mynetworks_1.ipaddr >> (flows_1.dstaddr)::ip4r)
Rows Removed by Join Filter: 453681377
Buffers: shared hit=120036496
-> Index Only Scan using flows_srcaddr_dstaddr_idx on flows flows_1 (cost=0.12..9091067.70 rows=560978368 width=4) (actual time=0.027..113052.437 rows=541559704 loops=10)
Heap Fetches: 91
Buffers: shared hit=120036459
-> Materialize (cost=0.00..1.02 rows=4 width=8) (actual time=0.000..0.000 rows=2 loops=5415597040)
Buffers: shared hit=10
-> Seq Scan on mynetworks mynetworks_1 (cost=0.00..1.01 rows=4 width=8) (actual time=0.007..0.008 rows=4 loops=10)
Buffers: shared hit=10
Planning time: 6.689 ms
Execution time: 2764860.853 ms
(58 rows)
Regarding "Also using dstat I can see that iowait time is at about 25%", I don't think the server was doing anything else. If it is important, I can repeat the benchmarks.
Thanks!
Charles
Charles,
In your original posting I couldn’t find what value you set for temp_buffers.
Considering you have plenty of RAM, try setting temp_buffers=’6GB’ and then run ‘explain (analyze, buffers) select…’ in the same session. This should alleviate “disk sort’ problem.
Also, could you post the structure of flowscompact, mynetworks, and dstextern tables with all the indexes and number of rows. Actually, are they all – tables, or some of them – views?
Igor
Sorry, I misstated the parameter to change.
It is work_mem (not temp_buffers) you should try to increase to 6GB.
Igor
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 19:24 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-11 11:02 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 00:39 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-12 10:04 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 14:31 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-12 16:39 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
@ 2017-07-14 15:34 ` Charles Nadeau <[email protected]>
2017-07-14 19:13 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-17 20:56 ` Re: Very poor read performance, query independent Claudio Freire <[email protected]>
0 siblings, 2 replies; 52+ messages in thread
From: Charles Nadeau @ 2017-07-14 15:34 UTC (permalink / raw)
To: Igor Neyman <[email protected]>; +Cc: Jeff Janes <[email protected]>; pgsql-performance
Igor,
Initially temp_buffer was left to its default value (8MB). Watching the
content of the directory that stores the temporary files, I found that I
need at most 21GB of temporary files space. Should I set temp_buffer to
21GB?
Here is the explain you requested with work_mem set to 6GB:
flows=# set work_mem='6GB';
SET
flows=# explain (analyze, buffers) SELECT DISTINCT
srcaddr,
dstaddr,
dstport,
COUNT(*) AS conversation,
SUM(doctets) / 1024 / 1024 AS mbytes
FROM
flowscompact,
mynetworks
WHERE
mynetworks.ipaddr >>= flowscompact.srcaddr
AND dstaddr IN
(
SELECT
dstaddr
FROM
dstexterne
)
GROUP BY
srcaddr,
dstaddr,
dstport
ORDER BY
mbytes DESC LIMIT 50;
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=48135680.07..48135680.22 rows=50 width=52) (actual
time=2227678.196..2227678.223 rows=50 loops=1)
Buffers: shared hit=728798038 read=82974833, temp read=381154
written=381154
-> Unique (cost=48135680.07..48143613.62 rows=2644514 width=52)
(actual time=2227678.194..2227678.217 rows=50 loops=1)
Buffers: shared hit=728798038 read=82974833, temp read=381154
written=381154
-> Sort (cost=48135680.07..48137002.33 rows=2644514 width=52)
(actual time=2227678.192..2227678.202 rows=50 loops=1)
Sort Key: (((sum(flows.doctets) / '1024'::numeric) /
'1024'::numeric)) DESC, flows.srcaddr, flows.dstaddr, flows.dstport,
(count(*))
Sort Method: quicksort Memory: 654395kB
Buffers: shared hit=728798038 read=82974833, temp
read=381154 written=381154
-> GroupAggregate (cost=48059426.65..48079260.50
rows=2644514 width=52) (actual time=2167909.030..2211446.192 rows=5859671
loops=1)
Group Key: flows.srcaddr, flows.dstaddr, flows.dstport
Buffers: shared hit=728798038 read=82974833, temp
read=381154 written=381154
-> Sort (cost=48059426.65..48060748.90 rows=2644514
width=20) (actual time=2167896.815..2189107.205 rows=91745640 loops=1)
Sort Key: flows.srcaddr, flows.dstaddr,
flows.dstport
Sort Method: external merge Disk: 3049216kB
Buffers: shared hit=728798038 read=82974833,
temp read=381154 written=381154
-> Gather (cost=30060688.07..48003007.07
rows=2644514 width=20) (actual time=1268989.000..1991357.232 rows=91745640
loops=1)
Workers Planned: 12
Workers Launched: 12
Buffers: shared hit=728798037 read=82974833
-> Hash Semi Join
(cost=30059688.07..47951761.31 rows=220376 width=20) (actual
time=1268845.181..2007864.725 rows=7057357 loops=13)
Hash Cond: (flows.dstaddr =
flows_1.dstaddr)
Buffers: shared hit=728795193
read=82974833
-> Nested Loop
(cost=0.03..17891246.86 rows=220376 width=20) (actual
time=0.207..723790.283 rows=37910370 loops=13)
Buffers: shared hit=590692229
read=14991777
-> Parallel Seq Scan on flows
(cost=0.00..16018049.14 rows=55094048 width=20) (actual
time=0.152..566179.117 rows=45371630 loops=13)
Buffers: shared
hit=860990 read=14991777
-> Index Only Scan using
mynetworks_ipaddr_idx on mynetworks (cost=0.03..0.03 rows=1 width=8)
(actual time=0.002..0.002 rows=1 loops=589831190)
Index Cond: (ipaddr >>=
(flows.srcaddr)::ip4r)
Heap Fetches: 0
Buffers: shared
hit=589831203
-> Hash
(cost=30059641.47..30059641.47 rows=13305 width=4) (actual
time=1268811.101..1268811.101 rows=3803508 loops=13)
Buckets: 4194304 (originally
16384) Batches: 1 (originally 1) Memory Usage: 166486kB
Buffers: shared hit=138102964
read=67983056
-> HashAggregate
(cost=30059561.64..30059601.56 rows=13305 width=4) (actual
time=1265248.165..1267432.083 rows=3803508 loops=13)
Group Key:
flows_1.dstaddr
Buffers: shared
hit=138102964 read=67983056
-> Nested Loop Anti
Join (cost=0.00..29729327.92 rows=660467447 width=4) (actual
time=0.389..1201072.707 rows=125838232 loops=13)
Join Filter:
(mynetworks_1.ipaddr >> (flows_1.dstaddr)::ip4r)
Rows Removed by
Join Filter: 503353617
Buffers: shared
hit=138102964 read=67983056
-> Seq Scan on
flows flows_1 (cost=0.00..17836152.73 rows=661128576 width=4) (actual
time=0.322..343152.274 rows=589831190 loops=13)
Buffers:
shared hit=138102915 read=67983056
-> Materialize
(cost=0.00..1.02 rows=4 width=8) (actual time=0.000..0.000 rows=2
loops=7667805470)
Buffers:
shared hit=13
-> Seq Scan
on mynetworks mynetworks_1 (cost=0.00..1.01 rows=4 width=8) (actual
time=0.006..0.007 rows=4 loops=13)
Buffers: shared hit=13
Planning time: 0.941 ms
Execution time: 2228345.171 ms
(48 rows)
With a work_mem at 6GB, I noticed that for the first 20 minutes the query
was running, the i/o wait was much lower, hovering aroun 3% then it jumped
45% until almost the end of the query.
flowscompact and dstexterne are actually views. I use views to simplify
query writing and to "abstract" queries that are use often in other
queries. flowscompact is a view built on table flows (having about 590
million rows), it only keeps the most often used fields.
flows=# \d+ flowscompact;
View "public.flowscompact"
Column | Type | Modifiers | Storage | Description
-----------+--------------------------+-----------+---------+-------------
flow_id | bigint | | plain |
sysuptime | bigint | | plain |
exaddr | ip4 | | plain |
dpkts | integer | | plain |
doctets | bigint | | plain |
first | bigint | | plain |
last | bigint | | plain |
srcaddr | ip4 | | plain |
dstaddr | ip4 | | plain |
srcport | integer | | plain |
dstport | integer | | plain |
prot | smallint | | plain |
tos | smallint | | plain |
tcp_flags | smallint | | plain |
timestamp | timestamp with time zone | | plain |
View definition:
SELECT flowstimestamp.flow_id,
flowstimestamp.sysuptime,
flowstimestamp.exaddr,
flowstimestamp.dpkts,
flowstimestamp.doctets,
flowstimestamp.first,
flowstimestamp.last,
flowstimestamp.srcaddr,
flowstimestamp.dstaddr,
flowstimestamp.srcport,
flowstimestamp.dstport,
flowstimestamp.prot,
flowstimestamp.tos,
flowstimestamp.tcp_flags,
flowstimestamp."timestamp"
FROM flowstimestamp;
mynetworks is a table having one column and 4 rows; it contains a list of
our network networks:
flows=# select * from mynetworks;
ipaddr
----------------
192.168.0.0/24
10.112.12.0/30
10.112.12.4/30
10.112.12.8/30
(4 row)
flows=# \d+ mynetworks;
Table "public.mynetworks"
Column | Type | Modifiers | Storage | Stats target | Description
--------+------+-----------+---------+--------------+-------------
ipaddr | ip4r | | plain | |
Indexes:
"mynetworks_ipaddr_idx" gist (ipaddr)
dstexterne is a view listing all the destination IPv4 addresses not inside
our network; it has one column and 3.8 million rows.
flows=# \d+ dstexterne;
View "public.dstexterne"
Column | Type | Modifiers | Storage | Description
---------+------+-----------+---------+-------------
dstaddr | ip4 | | plain |
View definition:
SELECT DISTINCT flowscompact.dstaddr
FROM flowscompact
LEFT JOIN mynetworks ON mynetworks.ipaddr >> flowscompact.dstaddr::ip4r
WHERE mynetworks.ipaddr IS NULL;
Thanks!
Charles
On Wed, Jul 12, 2017 at 6:39 PM, Igor Neyman <[email protected]> wrote:
>
>
>
>
> *From:* [email protected] [mailto:pgsql-performance-
> [email protected] <[email protected]>] *On Behalf
> Of *Charles Nadeau
> *Sent:* Wednesday, July 12, 2017 6:05 AM
> *To:* Jeff Janes <[email protected]>
> *Cc:* [email protected]
> *Subject:* Re: [PERFORM] Very poor read performance, query independent
>
>
>
>
>
> flows=# explain (analyze, buffers) SELECT DISTINCT
>
> flows-# srcaddr,
>
> flows-# dstaddr,
>
> flows-# dstport,
>
> flows-# COUNT(*) AS conversation,
>
> flows-# SUM(doctets) / 1024 / 1024 AS mbytes
>
> flows-# FROM
>
> flows-# flowscompact,
>
> flows-# mynetworks
>
> flows-# WHERE
>
> flows-# mynetworks.ipaddr >>= flowscompact.srcaddr
>
> flows-# AND dstaddr IN
>
> flows-# (
>
> flows(# SELECT
>
> flows(# dstaddr
>
> flows(# FROM
>
> flows(# dstexterne
>
> flows(# )
>
> flows-# GROUP BY
>
> flows-# srcaddr,
>
> flows-# dstaddr,
>
> flows-# dstport
>
> flows-# ORDER BY
>
> flows-# mbytes DESC LIMIT 50;
>
> LOG: temporary file: path "pg_tblspc/36238/PG_9.6_
> 201608131/pgsql_tmp/pgsql_tmp14573.6", size 1073741824
>
> LOG: temporary file: path "pg_tblspc/36238/PG_9.6_
> 201608131/pgsql_tmp/pgsql_tmp14573.7", size 1073741824
>
> LOG: temporary file: path "pg_tblspc/36238/PG_9.6_
> 201608131/pgsql_tmp/pgsql_tmp14573.8", size 639696896
>
> LOG: duration: 2765020.327 ms statement: explain (analyze, buffers)
> SELECT DISTINCT
>
> srcaddr,
>
> dstaddr,
>
> dstport,
>
> COUNT(*) AS conversation,
>
> SUM(doctets) / 1024 / 1024 AS mbytes
>
> FROM
>
> flowscompact,
>
> mynetworks
>
> WHERE
>
> mynetworks.ipaddr >>= flowscompact.srcaddr
>
> AND dstaddr IN
>
> (
>
> SELECT
>
> dstaddr
>
> FROM
>
> dstexterne
>
> )
>
> GROUP BY
>
> srcaddr,
>
> dstaddr,
>
> dstport
>
> ORDER BY
>
> mbytes DESC LIMIT 50;
>
>
> QUERY PLAN
>
>
>
> ------------------------------------------------------------
> ------------------------------------------------------------
> ------------------------------------------------------------
> --------------------------------------------------
>
> Limit (cost=37762321.83..37762321.98 rows=50 width=52) (actual
> time=2764548.863..2764548.891 rows=50 loops=1)
>
> Buffers: shared hit=1116590560 read=15851133, temp read=340244
> written=340244
>
> I/O Timings: read=5323746.860
>
> -> Unique (cost=37762321.83..37769053.57 rows=2243913 width=52)
> (actual time=2764548.861..2764548.882 rows=50 loops=1)
>
> Buffers: shared hit=1116590560 read=15851133, temp read=340244
> written=340244
>
> I/O Timings: read=5323746.860
>
> -> Sort (cost=37762321.83..37763443.79 rows=2243913 width=52)
> (actual time=2764548.859..2764548.872 rows=50 loops=1)
>
> Sort Key: (((sum(flows.doctets) / '1024'::numeric) /
> '1024'::numeric)) DESC, flows.srcaddr, flows.dstaddr, flows.dstport,
> (count(*))
>
> Sort Method: quicksort Memory: 563150kB
>
> Buffers: shared hit=1116590560 read=15851133, temp
> read=340244 written=340244
>
> I/O Timings: read=5323746.860
>
> -> GroupAggregate (cost=37698151.34..37714980.68
> rows=2243913 width=52) (actual time=2696721.610..2752109.551 rows=4691734
> loops=1)
>
> Group Key: flows.srcaddr, flows.dstaddr, flows.dstport
>
> Buffers: shared hit=1116590560 read=15851133, temp
> read=340244 written=340244
>
> I/O Timings: read=5323746.860
>
> -> Sort (cost=37698151.34..37699273.29
> rows=2243913 width=20) (actual time=2696711.428..2732781.705 rows=81896988
> loops=1)
>
> Sort Key: flows.srcaddr, flows.dstaddr,
> flows.dstport
>
> Sort Method: external merge Disk: 2721856kB
>
> Buffers: shared hit=1116590560 read=15851133,
> temp read=340244 written=340244
>
> I/O Timings: read=5323746.860
>
> -> Gather (cost=19463936.00..37650810.19
> rows=2243913 width=20) (actual time=1777219.713..2590530.887 rows=81896988
> loops=1)
>
> Workers Planned: 9
>
> Workers Launched: 9
>
> Buffers: shared hit=1116590559
> read=15851133
>
> I/O Timings: read=5323746.860
>
> -> Hash Semi Join
> (cost=19462936.00..37622883.23 rows=249324 width=20) (actual
> time=1847579.360..2602039.780 rows=8189699 loops=10)
>
> Hash Cond: (flows.dstaddr =
> flows_1.dstaddr)
>
> Buffers: shared hit=1116588309
> read=15851133
>
> I/O Timings: read=5323746.860
>
> -> Nested Loop
> (cost=0.03..18159012.30 rows=249324 width=20) (actual
> time=1.562..736556.583 rows=45499045 loops=10)
>
> Buffers: shared hit=996551813
> read=15851133
>
> I/O Timings: read=5323746.860
>
> -> Parallel Seq Scan on
> flows (cost=0.00..16039759.79 rows=62330930 width=20) (actual
> time=1.506..547485.066 rows=54155970 loops=10)
>
> Buffers: shared
> hit=1634 read=15851133
>
> I/O Timings:
> read=5323746.860
>
> -> Index Only Scan using
> mynetworks_ipaddr_idx on mynetworks (cost=0.03..0.03 rows=1 width=8)
> (actual time=0.002..0.002 rows=1 loops=541559704)
>
> Index Cond: (ipaddr >>=
> (flows.srcaddr)::ip4r)
>
> Heap Fetches: 59971474
>
> Buffers: shared
> hit=996550152
>
> -> Hash
> (cost=19462896.74..19462896.74 rows=11210 width=4) (actual
> time=1847228.894..1847228.894 rows=3099798 loops=10)
>
> Buckets: 4194304 (originally
> 16384) Batches: 1 (originally 1) Memory Usage: 141746kB
>
> Buffers: shared hit=120036496
>
> -> HashAggregate
> (cost=19462829.48..19462863.11 rows=11210 width=4) (actual
> time=1230049.015..1845955.764 rows=3099798 loops=10)
>
> Group Key:
> flows_1.dstaddr
>
> Buffers: shared
> hit=120036496
>
> -> Nested Loop Anti
> Join (cost=0.12..19182620.78 rows=560417390 width=4) (actual
> time=0.084..831832.333 rows=113420172 loops=10)
>
> Join Filter:
> (mynetworks_1.ipaddr >> (flows_1.dstaddr)::ip4r)
>
> Rows Removed by
> Join Filter: 453681377
>
> Buffers: shared
> hit=120036496
>
> -> Index Only
> Scan using flows_srcaddr_dstaddr_idx on flows flows_1
> (cost=0.12..9091067.70 rows=560978368 width=4) (actual
> time=0.027..113052.437 rows=541559704 loops=10)
>
> Heap
> Fetches: 91
>
> Buffers:
> shared hit=120036459
>
> -> Materialize
> (cost=0.00..1.02 rows=4 width=8) (actual time=0.000..0.000 rows=2
> loops=5415597040)
>
> Buffers:
> shared hit=10
>
> -> Seq
> Scan on mynetworks mynetworks_1 (cost=0.00..1.01 rows=4 width=8) (actual
> time=0.007..0.008 rows=4 loops=10)
>
>
> Buffers: shared hit=10
>
> Planning time: 6.689 ms
>
> Execution time: 2764860.853 ms
>
> (58 rows)
>
>
>
> Regarding "Also using dstat I can see that iowait time is at about 25%", I
> don't think the server was doing anything else. If it is important, I can
> repeat the benchmarks.
>
> Thanks!
>
>
>
> Charles
>
>
>
> Charles,
>
>
>
> In your original posting I couldn’t find what value you set for
> temp_buffers.
>
> Considering you have plenty of RAM, try setting temp_buffers=’6GB’ and
> then run ‘explain (analyze, buffers) select…’ in the same session. This
> should alleviate “disk sort’ problem.
>
>
>
> Also, could you post the structure of flowscompact, mynetworks, and
> dstextern tables with all the indexes and number of rows. Actually, are
> they all – tables, or some of them – views?
>
>
>
> Igor
>
>
>
>
>
> Sorry, I misstated the parameter to change.
>
> It is work_mem (not temp_buffers) you should try to increase to 6GB.
>
>
>
> Igor
>
>
>
>
>
--
Charles Nadeau Ph.D.
http://charlesnadeau.blogspot.com/
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 19:24 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-11 11:02 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 00:39 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-12 10:04 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 14:31 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-12 16:39 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-14 15:34 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
@ 2017-07-14 19:13 ` Igor Neyman <[email protected]>
2017-07-14 20:18 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-16 09:20 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
1 sibling, 2 replies; 52+ messages in thread
From: Igor Neyman @ 2017-07-14 19:13 UTC (permalink / raw)
To: Charles Nadeau <[email protected]>; +Cc: Jeff Janes <[email protected]>; pgsql-performance
From: Charles Nadeau [mailto:[email protected]]
Sent: Friday, July 14, 2017 11:35 AM
To: Igor Neyman <[email protected]>
Cc: Jeff Janes <[email protected]>; [email protected]
Subject: Re: [PERFORM] Very poor read performance, query independent
Igor,
Initially temp_buffer was left to its default value (8MB). Watching the content of the directory that stores the temporary files, I found that I need at most 21GB of temporary files space. Should I set temp_buffer to 21GB?
Here is the explain you requested with work_mem set to 6GB:
flows=# set work_mem='6GB';
SET
flows=# explain (analyze, buffers) SELECT DISTINCT
srcaddr,
dstaddr,
dstport,
COUNT(*) AS conversation,
SUM(doctets) / 1024 / 1024 AS mbytes
FROM
flowscompact,
mynetworks
WHERE
mynetworks.ipaddr >>= flowscompact.srcaddr
AND dstaddr IN
(
SELECT
dstaddr
FROM
dstexterne
)
GROUP BY
srcaddr,
dstaddr,
dstport
ORDER BY
mbytes DESC LIMIT 50;
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=48135680.07..48135680.22 rows=50 width=52) (actual time=2227678.196..2227678.223 rows=50 loops=1)
Buffers: shared hit=728798038 read=82974833, temp read=381154 written=381154
-> Unique (cost=48135680.07..48143613.62 rows=2644514 width=52) (actual time=2227678.194..2227678.217 rows=50 loops=1)
Buffers: shared hit=728798038 read=82974833, temp read=381154 written=381154
-> Sort (cost=48135680.07..48137002.33 rows=2644514 width=52) (actual time=2227678.192..2227678.202 rows=50 loops=1)
Sort Key: (((sum(flows.doctets) / '1024'::numeric) / '1024'::numeric)) DESC, flows.srcaddr, flows.dstaddr, flows.dstport, (count(*))
Sort Method: quicksort Memory: 654395kB
Buffers: shared hit=728798038 read=82974833, temp read=381154 written=381154
-> GroupAggregate (cost=48059426.65..48079260.50 rows=2644514 width=52) (actual time=2167909.030..2211446.192 rows=5859671 loops=1)
Group Key: flows.srcaddr, flows.dstaddr, flows.dstport
Buffers: shared hit=728798038 read=82974833, temp read=381154 written=381154
-> Sort (cost=48059426.65..48060748.90 rows=2644514 width=20) (actual time=2167896.815..2189107.205 rows=91745640 loops=1)
Sort Key: flows.srcaddr, flows.dstaddr, flows.dstport
Sort Method: external merge Disk: 3049216kB
Buffers: shared hit=728798038 read=82974833, temp read=381154 written=381154
-> Gather (cost=30060688.07..48003007.07 rows=2644514 width=20) (actual time=1268989.000..1991357.232 rows=91745640 loops=1)
Workers Planned: 12
Workers Launched: 12
Buffers: shared hit=728798037 read=82974833
-> Hash Semi Join (cost=30059688.07..47951761.31 rows=220376 width=20) (actual time=1268845.181..2007864.725 rows=7057357 loops=13)
Hash Cond: (flows.dstaddr = flows_1.dstaddr)
Buffers: shared hit=728795193 read=82974833
-> Nested Loop (cost=0.03..17891246.86 rows=220376 width=20) (actual time=0.207..723790.283 rows=37910370 loops=13)
Buffers: shared hit=590692229 read=14991777
-> Parallel Seq Scan on flows (cost=0.00..16018049.14 rows=55094048 width=20) (actual time=0.152..566179.117 rows=45371630 loops=13)
Buffers: shared hit=860990 read=14991777
-> Index Only Scan using mynetworks_ipaddr_idx on mynetworks (cost=0.03..0.03 rows=1 width=8) (actual time=0.002..0.002 rows=1 loops=589831190)
Index Cond: (ipaddr >>= (flows.srcaddr)::ip4r)
Heap Fetches: 0
Buffers: shared hit=589831203
-> Hash (cost=30059641.47..30059641.47 rows=13305 width=4) (actual time=1268811.101..1268811.101 rows=3803508 loops=13)
Buckets: 4194304 (originally 16384) Batches: 1 (originally 1) Memory Usage: 166486kB
Buffers: shared hit=138102964 read=67983056
-> HashAggregate (cost=30059561.64..30059601.56 rows=13305 width=4) (actual time=1265248.165..1267432.083 rows=3803508 loops=13)
Group Key: flows_1.dstaddr
Buffers: shared hit=138102964 read=67983056
-> Nested Loop Anti Join (cost=0.00..29729327.92 rows=660467447 width=4) (actual time=0.389..1201072.707 rows=125838232 loops=13)
Join Filter: (mynetworks_1.ipaddr >> (flows_1.dstaddr)::ip4r)
Rows Removed by Join Filter: 503353617
Buffers: shared hit=138102964 read=67983056
-> Seq Scan on flows flows_1 (cost=0.00..17836152.73 rows=661128576 width=4) (actual time=0.322..343152.274 rows=589831190 loops=13)
Buffers: shared hit=138102915 read=67983056
-> Materialize (cost=0.00..1.02 rows=4 width=8) (actual time=0.000..0.000 rows=2 loops=7667805470)
Buffers: shared hit=13
-> Seq Scan on mynetworks mynetworks_1 (cost=0.00..1.01 rows=4 width=8) (actual time=0.006..0.007 rows=4 loops=13)
Buffers: shared hit=13
Planning time: 0.941 ms
Execution time: 2228345.171 ms
(48 rows)
With a work_mem at 6GB, I noticed that for the first 20 minutes the query was running, the i/o wait was much lower, hovering aroun 3% then it jumped 45% until almost the end of the query.
flowscompact and dstexterne are actually views. I use views to simplify query writing and to "abstract" queries that are use often in other queries. flowscompact is a view built on table flows (having about 590 million rows), it only keeps the most often used fields.
flows=# \d+ flowscompact;
View "public.flowscompact"
Column | Type | Modifiers | Storage | Description
-----------+--------------------------+-----------+---------+-------------
flow_id | bigint | | plain |
sysuptime | bigint | | plain |
exaddr | ip4 | | plain |
dpkts | integer | | plain |
doctets | bigint | | plain |
first | bigint | | plain |
last | bigint | | plain |
srcaddr | ip4 | | plain |
dstaddr | ip4 | | plain |
srcport | integer | | plain |
dstport | integer | | plain |
prot | smallint | | plain |
tos | smallint | | plain |
tcp_flags | smallint | | plain |
timestamp | timestamp with time zone | | plain |
View definition:
SELECT flowstimestamp.flow_id,
flowstimestamp.sysuptime,
flowstimestamp.exaddr,
flowstimestamp.dpkts,
flowstimestamp.doctets,
flowstimestamp.first,
flowstimestamp.last,
flowstimestamp.srcaddr,
flowstimestamp.dstaddr,
flowstimestamp.srcport,
flowstimestamp.dstport,
flowstimestamp.prot,
flowstimestamp.tos,
flowstimestamp.tcp_flags,
flowstimestamp."timestamp"
FROM flowstimestamp;
mynetworks is a table having one column and 4 rows; it contains a list of our network networks:
flows=# select * from mynetworks;
ipaddr
----------------
192.168.0.0/24<http://192.168.0.0/24;
10.112.12.0/30<http://10.112.12.0/30;
10.112.12.4/30<http://10.112.12.4/30;
10.112.12.8/30<http://10.112.12.8/30;
(4 row)
flows=# \d+ mynetworks;
Table "public.mynetworks"
Column | Type | Modifiers | Storage | Stats target | Description
--------+------+-----------+---------+--------------+-------------
ipaddr | ip4r | | plain | |
Indexes:
"mynetworks_ipaddr_idx" gist (ipaddr)
dstexterne is a view listing all the destination IPv4 addresses not inside our network; it has one column and 3.8 million rows.
flows=# \d+ dstexterne;
View "public.dstexterne"
Column | Type | Modifiers | Storage | Description
---------+------+-----------+---------+-------------
dstaddr | ip4 | | plain |
View definition:
SELECT DISTINCT flowscompact.dstaddr
FROM flowscompact
LEFT JOIN mynetworks ON mynetworks.ipaddr >> flowscompact.dstaddr::ip4r
WHERE mynetworks.ipaddr IS NULL;
Thanks!
Charles
Charles,
Don’t change temp_buffers.
Try to increase work_mem even more, say work_mem=’12GB’, because it’s still using disk for sorting (starting around 20th minute as you noticed).
See if this:
“Sort Method: external merge Disk: 3049216kB”
goes away.
Igor
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 19:24 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-11 11:02 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 00:39 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-12 10:04 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 14:31 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-12 16:39 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-14 15:34 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-14 19:13 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
@ 2017-07-14 20:18 ` Igor Neyman <[email protected]>
2017-07-17 11:22 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
1 sibling, 1 reply; 52+ messages in thread
From: Igor Neyman @ 2017-07-14 20:18 UTC (permalink / raw)
To: Charles Nadeau <[email protected]>; +Cc: Jeff Janes <[email protected]>; pgsql-performance
From: [email protected] [mailto:[email protected]] On Behalf Of Igor Neyman
Sent: Friday, July 14, 2017 3:13 PM
To: Charles Nadeau <[email protected]>
Cc: Jeff Janes <[email protected]>; [email protected]
Subject: Re: [PERFORM] Very poor read performance, query independent
From: Charles Nadeau [mailto:[email protected]]
Sent: Friday, July 14, 2017 11:35 AM
To: Igor Neyman <[email protected]<mailto:[email protected]>>
Cc: Jeff Janes <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>
Subject: Re: [PERFORM] Very poor read performance, query independent
Igor,
Initially temp_buffer was left to its default value (8MB). Watching the content of the directory that stores the temporary files, I found that I need at most 21GB of temporary files space. Should I set temp_buffer to 21GB?
Here is the explain you requested with work_mem set to 6GB:
flows=# set work_mem='6GB';
SET
flows=# explain (analyze, buffers) SELECT DISTINCT
srcaddr,
dstaddr,
dstport,
COUNT(*) AS conversation,
SUM(doctets) / 1024 / 1024 AS mbytes
FROM
flowscompact,
mynetworks
WHERE
mynetworks.ipaddr >>= flowscompact.srcaddr
AND dstaddr IN
(
SELECT
dstaddr
FROM
dstexterne
)
GROUP BY
srcaddr,
dstaddr,
dstport
ORDER BY
mbytes DESC LIMIT 50;
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=48135680.07..48135680.22 rows=50 width=52) (actual time=2227678.196..2227678.223 rows=50 loops=1)
Buffers: shared hit=728798038 read=82974833, temp read=381154 written=381154
-> Unique (cost=48135680.07..48143613.62 rows=2644514 width=52) (actual time=2227678.194..2227678.217 rows=50 loops=1)
Buffers: shared hit=728798038 read=82974833, temp read=381154 written=381154
-> Sort (cost=48135680.07..48137002.33 rows=2644514 width=52) (actual time=2227678.192..2227678.202 rows=50 loops=1)
Sort Key: (((sum(flows.doctets) / '1024'::numeric) / '1024'::numeric)) DESC, flows.srcaddr, flows.dstaddr, flows.dstport, (count(*))
Sort Method: quicksort Memory: 654395kB
Buffers: shared hit=728798038 read=82974833, temp read=381154 written=381154
-> GroupAggregate (cost=48059426.65..48079260.50 rows=2644514 width=52) (actual time=2167909.030..2211446.192 rows=5859671 loops=1)
Group Key: flows.srcaddr, flows.dstaddr, flows.dstport
Buffers: shared hit=728798038 read=82974833, temp read=381154 written=381154
-> Sort (cost=48059426.65..48060748.90 rows=2644514 width=20) (actual time=2167896.815..2189107.205 rows=91745640 loops=1)
Sort Key: flows.srcaddr, flows.dstaddr, flows.dstport
Sort Method: external merge Disk: 3049216kB
Buffers: shared hit=728798038 read=82974833, temp read=381154 written=381154
-> Gather (cost=30060688.07..48003007.07 rows=2644514 width=20) (actual time=1268989.000..1991357.232 rows=91745640 loops=1)
Workers Planned: 12
Workers Launched: 12
Buffers: shared hit=728798037 read=82974833
-> Hash Semi Join (cost=30059688.07..47951761.31 rows=220376 width=20) (actual time=1268845.181..2007864.725 rows=7057357 loops=13)
Hash Cond: (flows.dstaddr = flows_1.dstaddr)
Buffers: shared hit=728795193 read=82974833
-> Nested Loop (cost=0.03..17891246.86 rows=220376 width=20) (actual time=0.207..723790.283 rows=37910370 loops=13)
Buffers: shared hit=590692229 read=14991777
-> Parallel Seq Scan on flows (cost=0.00..16018049.14 rows=55094048 width=20) (actual time=0.152..566179.117 rows=45371630 loops=13)
Buffers: shared hit=860990 read=14991777
-> Index Only Scan using mynetworks_ipaddr_idx on mynetworks (cost=0.03..0.03 rows=1 width=8) (actual time=0.002..0.002 rows=1 loops=589831190)
Index Cond: (ipaddr >>= (flows.srcaddr)::ip4r)
Heap Fetches: 0
Buffers: shared hit=589831203
-> Hash (cost=30059641.47..30059641.47 rows=13305 width=4) (actual time=1268811.101..1268811.101 rows=3803508 loops=13)
Buckets: 4194304 (originally 16384) Batches: 1 (originally 1) Memory Usage: 166486kB
Buffers: shared hit=138102964 read=67983056
-> HashAggregate (cost=30059561.64..30059601.56 rows=13305 width=4) (actual time=1265248.165..1267432.083 rows=3803508 loops=13)
Group Key: flows_1.dstaddr
Buffers: shared hit=138102964 read=67983056
-> Nested Loop Anti Join (cost=0.00..29729327.92 rows=660467447 width=4) (actual time=0.389..1201072.707 rows=125838232 loops=13)
Join Filter: (mynetworks_1.ipaddr >> (flows_1.dstaddr)::ip4r)
Rows Removed by Join Filter: 503353617
Buffers: shared hit=138102964 read=67983056
-> Seq Scan on flows flows_1 (cost=0.00..17836152.73 rows=661128576 width=4) (actual time=0.322..343152.274 rows=589831190 loops=13)
Buffers: shared hit=138102915 read=67983056
-> Materialize (cost=0.00..1.02 rows=4 width=8) (actual time=0.000..0.000 rows=2 loops=7667805470)
Buffers: shared hit=13
-> Seq Scan on mynetworks mynetworks_1 (cost=0.00..1.01 rows=4 width=8) (actual time=0.006..0.007 rows=4 loops=13)
Buffers: shared hit=13
Planning time: 0.941 ms
Execution time: 2228345.171 ms
(48 rows)
With a work_mem at 6GB, I noticed that for the first 20 minutes the query was running, the i/o wait was much lower, hovering aroun 3% then it jumped 45% until almost the end of the query.
flowscompact and dstexterne are actually views. I use views to simplify query writing and to "abstract" queries that are use often in other queries. flowscompact is a view built on table flows (having about 590 million rows), it only keeps the most often used fields.
flows=# \d+ flowscompact;
View "public.flowscompact"
Column | Type | Modifiers | Storage | Description
-----------+--------------------------+-----------+---------+-------------
flow_id | bigint | | plain |
sysuptime | bigint | | plain |
exaddr | ip4 | | plain |
dpkts | integer | | plain |
doctets | bigint | | plain |
first | bigint | | plain |
last | bigint | | plain |
srcaddr | ip4 | | plain |
dstaddr | ip4 | | plain |
srcport | integer | | plain |
dstport | integer | | plain |
prot | smallint | | plain |
tos | smallint | | plain |
tcp_flags | smallint | | plain |
timestamp | timestamp with time zone | | plain |
View definition:
SELECT flowstimestamp.flow_id,
flowstimestamp.sysuptime,
flowstimestamp.exaddr,
flowstimestamp.dpkts,
flowstimestamp.doctets,
flowstimestamp.first,
flowstimestamp.last,
flowstimestamp.srcaddr,
flowstimestamp.dstaddr,
flowstimestamp.srcport,
flowstimestamp.dstport,
flowstimestamp.prot,
flowstimestamp.tos,
flowstimestamp.tcp_flags,
flowstimestamp."timestamp"
FROM flowstimestamp;
mynetworks is a table having one column and 4 rows; it contains a list of our network networks:
flows=# select * from mynetworks;
ipaddr
----------------
192.168.0.0/24<http://192.168.0.0/24;
10.112.12.0/30<http://10.112.12.0/30;
10.112.12.4/30<http://10.112.12.4/30;
10.112.12.8/30<http://10.112.12.8/30;
(4 row)
flows=# \d+ mynetworks;
Table "public.mynetworks"
Column | Type | Modifiers | Storage | Stats target | Description
--------+------+-----------+---------+--------------+-------------
ipaddr | ip4r | | plain | |
Indexes:
"mynetworks_ipaddr_idx" gist (ipaddr)
dstexterne is a view listing all the destination IPv4 addresses not inside our network; it has one column and 3.8 million rows.
flows=# \d+ dstexterne;
View "public.dstexterne"
Column | Type | Modifiers | Storage | Description
---------+------+-----------+---------+-------------
dstaddr | ip4 | | plain |
View definition:
SELECT DISTINCT flowscompact.dstaddr
FROM flowscompact
LEFT JOIN mynetworks ON mynetworks.ipaddr >> flowscompact.dstaddr::ip4r
WHERE mynetworks.ipaddr IS NULL;
Thanks!
Charles
Charles,
Also, let’s try to simplify your query and see if it performs better.
You are grouping by srcaddr, dstaddr, dstport, that makes DISTINCT not needed.
And after simplifying WHERE clause (let me know if the result is not what you want), the query looks like:
SELECT srcaddr, dstaddr, dstport,
COUNT(*) AS conversation,
SUM(doctets) / 1024 / 1024 AS mbytes
FROM flowscompact
WHERE srcaddr IN (SELECT ipaddr FROM mynetworks)
AND dstaddr NOT IN (SELECT ipaddr FROM mynetworks)
GROUP BY srcaddr, dstaddr, dstport
ORDER BY mbytes DESC
LIMIT 50;
Now, you didn’t provide the definition of flowstimestamp table.
If this table doesn’t have an index on (srcaddr, dstaddr, dstport) creating one should help (I think).
Igor
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 19:24 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-11 11:02 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 00:39 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-12 10:04 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 14:31 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-12 16:39 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-14 15:34 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-14 19:13 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-14 20:18 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
@ 2017-07-17 11:22 ` Charles Nadeau <[email protected]>
0 siblings, 0 replies; 52+ messages in thread
From: Charles Nadeau @ 2017-07-17 11:22 UTC (permalink / raw)
To: Igor Neyman <[email protected]>; +Cc: Jeff Janes <[email protected]>; pgsql-performance
Igor,
The 1st clause of the where statement won't select addresses the same way
as the one I wrote using the extension for IPv6 and IPv6 data types.
flowstimestamp is a view:
flows=# \d+ flowstimestamp
View "public.flowstimestamp"
Column | Type | Modifiers | Storage | Description
-------------+--------------------------+-----------+---------+-------------
flow_id | bigint | | plain |
unix_secs | bigint | | plain |
unix_nsecs | bigint | | plain |
sysuptime | bigint | | plain |
exaddr | ip4 | | plain |
dpkts | integer | | plain |
doctets | bigint | | plain |
first | bigint | | plain |
last | bigint | | plain |
engine_type | smallint | | plain |
engine_id | smallint | | plain |
srcaddr | ip4 | | plain |
dstaddr | ip4 | | plain |
nexthop | ip4 | | plain |
input | integer | | plain |
output | integer | | plain |
srcport | integer | | plain |
dstport | integer | | plain |
prot | smallint | | plain |
tos | smallint | | plain |
tcp_flags | smallint | | plain |
src_mask | smallint | | plain |
dst_mask | smallint | | plain |
src_as | integer | | plain |
dst_as | integer | | plain |
timestamp | timestamp with time zone | | plain |
View definition:
SELECT flows.flow_id,
flows.unix_secs,
flows.unix_nsecs,
flows.sysuptime,
flows.exaddr,
flows.dpkts,
flows.doctets,
flows.first,
flows.last,
flows.engine_type,
flows.engine_id,
flows.srcaddr,
flows.dstaddr,
flows.nexthop,
flows.input,
flows.output,
flows.srcport,
flows.dstport,
flows.prot,
flows.tos,
flows.tcp_flags,
flows.src_mask,
flows.dst_mask,
flows.src_as,
flows.dst_as,
to_timestamp((flows.unix_secs + flows.unix_nsecs / 1000000000)::double
precision) AS "timestamp"
FROM flows;
And it can use the indexes of flows:
Indexes:
"flows_pkey" PRIMARY KEY, btree (flow_id)
"flows_dstaddr_dstport" btree (dstaddr, dstport)
"flows_srcaddr_dstaddr_idx" btree (srcaddr, dstaddr)
"flows_srcaddr_srcport" btree (srcaddr, srcport)
"flows_srcport_dstport_idx" btree (srcport, dstport)
Thanks!
Charles
On Fri, Jul 14, 2017 at 10:18 PM, Igor Neyman <[email protected]>
wrote:
>
>
>
>
> *From:* [email protected] [mailto:pgsql-performance-
> [email protected]] *On Behalf Of *Igor Neyman
> *Sent:* Friday, July 14, 2017 3:13 PM
> *To:* Charles Nadeau <[email protected]>
>
> *Cc:* Jeff Janes <[email protected]>; [email protected]
> *Subject:* Re: [PERFORM] Very poor read performance, query independent
>
>
>
> *From:* Charles Nadeau [mailto:[email protected]
> <[email protected]>]
> *Sent:* Friday, July 14, 2017 11:35 AM
> *To:* Igor Neyman <[email protected]>
> *Cc:* Jeff Janes <[email protected]>; [email protected]
> *Subject:* Re: [PERFORM] Very poor read performance, query independent
>
>
>
> Igor,
>
>
>
> Initially temp_buffer was left to its default value (8MB). Watching the
> content of the directory that stores the temporary files, I found that I
> need at most 21GB of temporary files space. Should I set temp_buffer to
> 21GB?
>
> Here is the explain you requested with work_mem set to 6GB:
>
> flows=# set work_mem='6GB';
>
> SET
>
> flows=# explain (analyze, buffers) SELECT DISTINCT
>
> srcaddr,
>
> dstaddr,
>
> dstport,
>
> COUNT(*) AS conversation,
>
> SUM(doctets) / 1024 / 1024 AS mbytes
>
> FROM
>
> flowscompact,
>
> mynetworks
>
> WHERE
>
> mynetworks.ipaddr >>= flowscompact.srcaddr
>
> AND dstaddr IN
>
> (
>
> SELECT
>
> dstaddr
>
> FROM
>
> dstexterne
>
> )
>
> GROUP BY
>
> srcaddr,
>
> dstaddr,
>
> dstport
>
> ORDER BY
>
> mbytes DESC LIMIT 50;
>
>
> QUERY PLAN
>
>
> ------------------------------------------------------------
> ------------------------------------------------------------
> ------------------------------------------------------------------------
>
> Limit (cost=48135680.07..48135680.22 rows=50 width=52) (actual
> time=2227678.196..2227678.223 rows=50 loops=1)
>
> Buffers: shared hit=728798038 read=82974833, temp read=381154
> written=381154
>
> -> Unique (cost=48135680.07..48143613.62 rows=2644514 width=52)
> (actual time=2227678.194..2227678.217 rows=50 loops=1)
>
> Buffers: shared hit=728798038 read=82974833, temp read=381154
> written=381154
>
> -> Sort (cost=48135680.07..48137002.33 rows=2644514 width=52)
> (actual time=2227678.192..2227678.202 rows=50 loops=1)
>
> Sort Key: (((sum(flows.doctets) / '1024'::numeric) /
> '1024'::numeric)) DESC, flows.srcaddr, flows.dstaddr, flows.dstport,
> (count(*))
>
> Sort Method: quicksort Memory: 654395kB
>
> Buffers: shared hit=728798038 read=82974833, temp
> read=381154 written=381154
>
> -> GroupAggregate (cost=48059426.65..48079260.50
> rows=2644514 width=52) (actual time=2167909.030..2211446.192 rows=5859671
> loops=1)
>
> Group Key: flows.srcaddr, flows.dstaddr, flows.dstport
>
> Buffers: shared hit=728798038 read=82974833, temp
> read=381154 written=381154
>
> -> Sort (cost=48059426.65..48060748.90
> rows=2644514 width=20) (actual time=2167896.815..2189107.205 rows=91745640
> loops=1)
>
> Sort Key: flows.srcaddr, flows.dstaddr,
> flows.dstport
>
> Sort Method: external merge Disk: 3049216kB
>
> Buffers: shared hit=728798038 read=82974833,
> temp read=381154 written=381154
>
> -> Gather (cost=30060688.07..48003007.07
> rows=2644514 width=20) (actual time=1268989.000..1991357.232 rows=91745640
> loops=1)
>
> Workers Planned: 12
>
> Workers Launched: 12
>
> Buffers: shared hit=728798037
> read=82974833
>
> -> Hash Semi Join
> (cost=30059688.07..47951761.31 rows=220376 width=20) (actual
> time=1268845.181..2007864.725 rows=7057357 loops=13)
>
> Hash Cond: (flows.dstaddr =
> flows_1.dstaddr)
>
> Buffers: shared hit=728795193
> read=82974833
>
> -> Nested Loop
> (cost=0.03..17891246.86 rows=220376 width=20) (actual
> time=0.207..723790.283 rows=37910370 loops=13)
>
> Buffers: shared hit=590692229
> read=14991777
>
> -> Parallel Seq Scan on
> flows (cost=0.00..16018049.14 rows=55094048 width=20) (actual
> time=0.152..566179.117 rows=45371630 loops=13)
>
> Buffers: shared
> hit=860990 read=14991777
>
> -> Index Only Scan using
> mynetworks_ipaddr_idx on mynetworks (cost=0.03..0.03 rows=1 width=8)
> (actual time=0.002..0.002 rows=1 loops=589831190)
>
> Index Cond: (ipaddr >>=
> (flows.srcaddr)::ip4r)
>
> Heap Fetches: 0
>
> Buffers: shared
> hit=589831203
>
> -> Hash
> (cost=30059641.47..30059641.47 rows=13305 width=4) (actual
> time=1268811.101..1268811.101 rows=3803508 loops=13)
>
> Buckets: 4194304 (originally
> 16384) Batches: 1 (originally 1) Memory Usage: 166486kB
>
> Buffers: shared hit=138102964
> read=67983056
>
> -> HashAggregate
> (cost=30059561.64..30059601.56 rows=13305 width=4) (actual
> time=1265248.165..1267432.083 rows=3803508 loops=13)
>
> Group Key:
> flows_1.dstaddr
>
> Buffers: shared
> hit=138102964 read=67983056
>
> -> Nested Loop Anti
> Join (cost=0.00..29729327.92 rows=660467447 width=4) (actual
> time=0.389..1201072.707 rows=125838232 loops=13)
>
> Join Filter:
> (mynetworks_1.ipaddr >> (flows_1.dstaddr)::ip4r)
>
> Rows Removed by
> Join Filter: 503353617
>
> Buffers: shared
> hit=138102964 read=67983056
>
> -> Seq Scan on
> flows flows_1 (cost=0.00..17836152.73 rows=661128576 width=4) (actual
> time=0.322..343152.274 rows=589831190 loops=13)
>
> Buffers:
> shared hit=138102915 read=67983056
>
> -> Materialize
> (cost=0.00..1.02 rows=4 width=8) (actual time=0.000..0.000 rows=2
> loops=7667805470)
>
> Buffers:
> shared hit=13
>
> -> Seq
> Scan on mynetworks mynetworks_1 (cost=0.00..1.01 rows=4 width=8) (actual
> time=0.006..0.007 rows=4 loops=13)
>
>
> Buffers: shared hit=13
>
> Planning time: 0.941 ms
>
> Execution time: 2228345.171 ms
>
> (48 rows)
>
>
>
> With a work_mem at 6GB, I noticed that for the first 20 minutes the query
> was running, the i/o wait was much lower, hovering aroun 3% then it jumped
> 45% until almost the end of the query.
>
>
>
> flowscompact and dstexterne are actually views. I use views to simplify
> query writing and to "abstract" queries that are use often in other
> queries. flowscompact is a view built on table flows (having about 590
> million rows), it only keeps the most often used fields.
>
> flows=# \d+ flowscompact;
>
> View "public.flowscompact"
>
> Column | Type | Modifiers | Storage | Description
>
> -----------+--------------------------+-----------+---------+-------------
>
> flow_id | bigint | | plain |
>
> sysuptime | bigint | | plain |
>
> exaddr | ip4 | | plain |
>
> dpkts | integer | | plain |
>
> doctets | bigint | | plain |
>
> first | bigint | | plain |
>
> last | bigint | | plain |
>
> srcaddr | ip4 | | plain |
>
> dstaddr | ip4 | | plain |
>
> srcport | integer | | plain |
>
> dstport | integer | | plain |
>
> prot | smallint | | plain |
>
> tos | smallint | | plain |
>
> tcp_flags | smallint | | plain |
>
> timestamp | timestamp with time zone | | plain |
>
> View definition:
>
> SELECT flowstimestamp.flow_id,
>
> flowstimestamp.sysuptime,
>
> flowstimestamp.exaddr,
>
> flowstimestamp.dpkts,
>
> flowstimestamp.doctets,
>
> flowstimestamp.first,
>
> flowstimestamp.last,
>
> flowstimestamp.srcaddr,
>
> flowstimestamp.dstaddr,
>
> flowstimestamp.srcport,
>
> flowstimestamp.dstport,
>
> flowstimestamp.prot,
>
> flowstimestamp.tos,
>
> flowstimestamp.tcp_flags,
>
> flowstimestamp."timestamp"
>
> FROM flowstimestamp;
>
> mynetworks is a table having one column and 4 rows; it contains a list of
> our network networks:
>
> flows=# select * from mynetworks;
>
> ipaddr
>
> ----------------
>
> 192.168.0.0/24
>
> 10.112.12.0/30
>
> 10.112.12.4/30
>
> 10.112.12.8/30
>
> (4 row)
>
> flows=# \d+ mynetworks;
>
> Table "public.mynetworks"
>
> Column | Type | Modifiers | Storage | Stats target | Description
>
> --------+------+-----------+---------+--------------+-------------
>
> ipaddr | ip4r | | plain | |
>
> Indexes:
>
> "mynetworks_ipaddr_idx" gist (ipaddr)
>
> dstexterne is a view listing all the destination IPv4 addresses not inside
> our network; it has one column and 3.8 million rows.
>
> flows=# \d+ dstexterne;
>
> View "public.dstexterne"
>
> Column | Type | Modifiers | Storage | Description
>
> ---------+------+-----------+---------+-------------
>
> dstaddr | ip4 | | plain |
>
> View definition:
>
> SELECT DISTINCT flowscompact.dstaddr
>
> FROM flowscompact
>
> LEFT JOIN mynetworks ON mynetworks.ipaddr >>
> flowscompact.dstaddr::ip4r
>
> WHERE mynetworks.ipaddr IS NULL;
>
> Thanks!
>
>
>
> Charles
>
>
>
> Charles,
>
>
>
> Also, let’s try to simplify your query and see if it performs better.
>
> You are grouping by srcaddr, dstaddr, dstport, that makes DISTINCT not
> needed.
>
> And after simplifying WHERE clause (let me know if the result is not what
> you want), the query looks like:
>
>
>
> SELECT srcaddr, dstaddr, dstport,
>
> COUNT(*) AS conversation,
>
> SUM(doctets) / 1024 / 1024 AS mbytes
>
> FROM flowscompact
>
> WHERE srcaddr IN (SELECT ipaddr FROM mynetworks)
>
> AND dstaddr NOT IN (SELECT ipaddr FROM mynetworks)
>
> GROUP BY srcaddr, dstaddr, dstport
>
> ORDER BY mbytes DESC
>
> LIMIT 50;
>
>
>
> Now, you didn’t provide the definition of flowstimestamp table.
>
> If this table doesn’t have an index on (srcaddr, dstaddr, dstport)
> creating one should help (I think).
>
>
>
> Igor
>
>
>
>
>
>
>
--
Charles Nadeau Ph.D.
http://charlesnadeau.blogspot.com/
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 19:24 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-11 11:02 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 00:39 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-12 10:04 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 14:31 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-12 16:39 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-14 15:34 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-14 19:13 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
@ 2017-07-16 09:20 ` Charles Nadeau <[email protected]>
1 sibling, 0 replies; 52+ messages in thread
From: Charles Nadeau @ 2017-07-16 09:20 UTC (permalink / raw)
To: Igor Neyman <[email protected]>; +Cc: Jeff Janes <[email protected]>; pgsql-performance
Igor,
I set the work_mem to 12GB, restarted postrgresql, repeat the same "explain
(analyze, buffers)..." as above and the read throughput was very low, at
most 10MB/s. All the sorting operation are now done in memory.
I lowered the work_mem back to 6GB, restarted postrgresql, repeat the same
"explain (analyze, buffers)..." as above and the read throughput was very
low, at most 10MB/s. The 1st sorting operation is still done in memory, the
second one to disk. I think I need about 4GB to do all sort in memory.
One thing I remember from Friday's "explain (analyze, buffers)...". I set
temp_buffer and work_mem to 6GB as I read both your message one after the
other. So I decided to try again: I then set work_mem=6GB,
temp_buffers=6GB, restarted postrgresql, repeat the same "explain (analyze,
buffers)..." as above and the read throughput was very low again, at most
10MB/s. The 1st sorting operation is still done in memory, the second one
to disk.
For the last test, I brought back work_mem and temp_buffer to their
original value. The read throughput is still low.
In all cases, about 20 minutes after the query starts, it start writing to
disk furiously. Here are the peak values as reported by dstat:
work_mem=12GB, temp_buffers=8MB: peak of 393MB/s
work_mem=6GB, temp_buffers=8MB: peak of 579MB/s
work_mem=6GB, temp_buffers=6GB: peak of 418MB/s
work_mem=1468006kB and temp_buffers=8MB, peak of 61MB/s
Also, at peak write, the server alomost ran of memory: the cache almost
goes to 0 and it starts swapping.
This query is a bit extreme in terms of sorting. Maybe I should try to
benchmark while counting all the records of my biggest table like Mark
Kirkwood suggested. I'll do some more tests and post the results back to
the mailing list.
Thanks!
Charles
On Fri, Jul 14, 2017 at 9:13 PM, Igor Neyman <[email protected]> wrote:
> *From:* Charles Nadeau [mailto:[email protected]]
> *Sent:* Friday, July 14, 2017 11:35 AM
> *To:* Igor Neyman <[email protected]>
> *Cc:* Jeff Janes <[email protected]>; [email protected]
> *Subject:* Re: [PERFORM] Very poor read performance, query independent
>
>
>
> Igor,
>
>
>
> Initially temp_buffer was left to its default value (8MB). Watching the
> content of the directory that stores the temporary files, I found that I
> need at most 21GB of temporary files space. Should I set temp_buffer to
> 21GB?
>
> Here is the explain you requested with work_mem set to 6GB:
>
> flows=# set work_mem='6GB';
>
> SET
>
> flows=# explain (analyze, buffers) SELECT DISTINCT
>
> srcaddr,
>
> dstaddr,
>
> dstport,
>
> COUNT(*) AS conversation,
>
> SUM(doctets) / 1024 / 1024 AS mbytes
>
> FROM
>
> flowscompact,
>
> mynetworks
>
> WHERE
>
> mynetworks.ipaddr >>= flowscompact.srcaddr
>
> AND dstaddr IN
>
> (
>
> SELECT
>
> dstaddr
>
> FROM
>
> dstexterne
>
> )
>
> GROUP BY
>
> srcaddr,
>
> dstaddr,
>
> dstport
>
> ORDER BY
>
> mbytes DESC LIMIT 50;
>
>
> QUERY PLAN
>
>
> ------------------------------------------------------------
> ------------------------------------------------------------
> ------------------------------------------------------------------------
>
> Limit (cost=48135680.07..48135680.22 rows=50 width=52) (actual
> time=2227678.196..2227678.223 rows=50 loops=1)
>
> Buffers: shared hit=728798038 read=82974833, temp read=381154
> written=381154
>
> -> Unique (cost=48135680.07..48143613.62 rows=2644514 width=52)
> (actual time=2227678.194..2227678.217 rows=50 loops=1)
>
> Buffers: shared hit=728798038 read=82974833, temp read=381154
> written=381154
>
> -> Sort (cost=48135680.07..48137002.33 rows=2644514 width=52)
> (actual time=2227678.192..2227678.202 rows=50 loops=1)
>
> Sort Key: (((sum(flows.doctets) / '1024'::numeric) /
> '1024'::numeric)) DESC, flows.srcaddr, flows.dstaddr, flows.dstport,
> (count(*))
>
> Sort Method: quicksort Memory: 654395kB
>
> Buffers: shared hit=728798038 read=82974833, temp
> read=381154 written=381154
>
> -> GroupAggregate (cost=48059426.65..48079260.50
> rows=2644514 width=52) (actual time=2167909.030..2211446.192 rows=5859671
> loops=1)
>
> Group Key: flows.srcaddr, flows.dstaddr, flows.dstport
>
> Buffers: shared hit=728798038 read=82974833, temp
> read=381154 written=381154
>
> -> Sort (cost=48059426.65..48060748.90
> rows=2644514 width=20) (actual time=2167896.815..2189107.205 rows=91745640
> loops=1)
>
> Sort Key: flows.srcaddr, flows.dstaddr,
> flows.dstport
>
> Sort Method: external merge Disk: 3049216kB
>
> Buffers: shared hit=728798038 read=82974833,
> temp read=381154 written=381154
>
> -> Gather (cost=30060688.07..48003007.07
> rows=2644514 width=20) (actual time=1268989.000..1991357.232 rows=91745640
> loops=1)
>
> Workers Planned: 12
>
> Workers Launched: 12
>
> Buffers: shared hit=728798037
> read=82974833
>
> -> Hash Semi Join
> (cost=30059688.07..47951761.31 rows=220376 width=20) (actual
> time=1268845.181..2007864.725 rows=7057357 loops=13)
>
> Hash Cond: (flows.dstaddr =
> flows_1.dstaddr)
>
> Buffers: shared hit=728795193
> read=82974833
>
> -> Nested Loop
> (cost=0.03..17891246.86 rows=220376 width=20) (actual
> time=0.207..723790.283 rows=37910370 loops=13)
>
> Buffers: shared hit=590692229
> read=14991777
>
> -> Parallel Seq Scan on
> flows (cost=0.00..16018049.14 rows=55094048 width=20) (actual
> time=0.152..566179.117 rows=45371630 loops=13)
>
> Buffers: shared
> hit=860990 read=14991777
>
> -> Index Only Scan using
> mynetworks_ipaddr_idx on mynetworks (cost=0.03..0.03 rows=1 width=8)
> (actual time=0.002..0.002 rows=1 loops=589831190)
>
> Index Cond: (ipaddr >>=
> (flows.srcaddr)::ip4r)
>
> Heap Fetches: 0
>
> Buffers: shared
> hit=589831203
>
> -> Hash
> (cost=30059641.47..30059641.47 rows=13305 width=4) (actual
> time=1268811.101..1268811.101 rows=3803508 loops=13)
>
> Buckets: 4194304 (originally
> 16384) Batches: 1 (originally 1) Memory Usage: 166486kB
>
> Buffers: shared hit=138102964
> read=67983056
>
> -> HashAggregate
> (cost=30059561.64..30059601.56 rows=13305 width=4) (actual
> time=1265248.165..1267432.083 rows=3803508 loops=13)
>
> Group Key:
> flows_1.dstaddr
>
> Buffers: shared
> hit=138102964 read=67983056
>
> -> Nested Loop Anti
> Join (cost=0.00..29729327.92 rows=660467447 width=4) (actual
> time=0.389..1201072.707 rows=125838232 loops=13)
>
> Join Filter:
> (mynetworks_1.ipaddr >> (flows_1.dstaddr)::ip4r)
>
> Rows Removed by
> Join Filter: 503353617
>
> Buffers: shared
> hit=138102964 read=67983056
>
> -> Seq Scan on
> flows flows_1 (cost=0.00..17836152.73 rows=661128576 width=4) (actual
> time=0.322..343152.274 rows=589831190 loops=13)
>
> Buffers:
> shared hit=138102915 read=67983056
>
> -> Materialize
> (cost=0.00..1.02 rows=4 width=8) (actual time=0.000..0.000 rows=2
> loops=7667805470)
>
> Buffers:
> shared hit=13
>
> -> Seq
> Scan on mynetworks mynetworks_1 (cost=0.00..1.01 rows=4 width=8) (actual
> time=0.006..0.007 rows=4 loops=13)
>
>
> Buffers: shared hit=13
>
> Planning time: 0.941 ms
>
> Execution time: 2228345.171 ms
>
> (48 rows)
>
>
>
> With a work_mem at 6GB, I noticed that for the first 20 minutes the query
> was running, the i/o wait was much lower, hovering aroun 3% then it jumped
> 45% until almost the end of the query.
>
>
>
> flowscompact and dstexterne are actually views. I use views to simplify
> query writing and to "abstract" queries that are use often in other
> queries. flowscompact is a view built on table flows (having about 590
> million rows), it only keeps the most often used fields.
>
> flows=# \d+ flowscompact;
>
> View "public.flowscompact"
>
> Column | Type | Modifiers | Storage | Description
>
> -----------+--------------------------+-----------+---------+-------------
>
> flow_id | bigint | | plain |
>
> sysuptime | bigint | | plain |
>
> exaddr | ip4 | | plain |
>
> dpkts | integer | | plain |
>
> doctets | bigint | | plain |
>
> first | bigint | | plain |
>
> last | bigint | | plain |
>
> srcaddr | ip4 | | plain |
>
> dstaddr | ip4 | | plain |
>
> srcport | integer | | plain |
>
> dstport | integer | | plain |
>
> prot | smallint | | plain |
>
> tos | smallint | | plain |
>
> tcp_flags | smallint | | plain |
>
> timestamp | timestamp with time zone | | plain |
>
> View definition:
>
> SELECT flowstimestamp.flow_id,
>
> flowstimestamp.sysuptime,
>
> flowstimestamp.exaddr,
>
> flowstimestamp.dpkts,
>
> flowstimestamp.doctets,
>
> flowstimestamp.first,
>
> flowstimestamp.last,
>
> flowstimestamp.srcaddr,
>
> flowstimestamp.dstaddr,
>
> flowstimestamp.srcport,
>
> flowstimestamp.dstport,
>
> flowstimestamp.prot,
>
> flowstimestamp.tos,
>
> flowstimestamp.tcp_flags,
>
> flowstimestamp."timestamp"
>
> FROM flowstimestamp;
>
> mynetworks is a table having one column and 4 rows; it contains a list of
> our network networks:
>
> flows=# select * from mynetworks;
>
> ipaddr
>
> ----------------
>
> 192.168.0.0/24
>
> 10.112.12.0/30
>
> 10.112.12.4/30
>
> 10.112.12.8/30
>
> (4 row)
>
> flows=# \d+ mynetworks;
>
> Table "public.mynetworks"
>
> Column | Type | Modifiers | Storage | Stats target | Description
>
> --------+------+-----------+---------+--------------+-------------
>
> ipaddr | ip4r | | plain | |
>
> Indexes:
>
> "mynetworks_ipaddr_idx" gist (ipaddr)
>
> dstexterne is a view listing all the destination IPv4 addresses not inside
> our network; it has one column and 3.8 million rows.
>
> flows=# \d+ dstexterne;
>
> View "public.dstexterne"
>
> Column | Type | Modifiers | Storage | Description
>
> ---------+------+-----------+---------+-------------
>
> dstaddr | ip4 | | plain |
>
> View definition:
>
> SELECT DISTINCT flowscompact.dstaddr
>
> FROM flowscompact
>
> LEFT JOIN mynetworks ON mynetworks.ipaddr >>
> flowscompact.dstaddr::ip4r
>
> WHERE mynetworks.ipaddr IS NULL;
>
> Thanks!
>
>
>
> Charles
>
>
>
> Charles,
>
>
>
> Don’t change temp_buffers.
>
> Try to increase work_mem even more, say work_mem=’12GB’, because it’s
> still using disk for sorting (starting around 20th minute as you noticed).
>
> See if this:
>
> “Sort Method: external merge Disk: 3049216kB”
>
> goes away.
>
> Igor
>
>
>
--
Charles Nadeau Ph.D.
http://charlesnadeau.blogspot.com/
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 19:24 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-11 11:02 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 00:39 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-12 10:04 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 14:31 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-12 16:39 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-14 15:34 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
@ 2017-07-17 20:56 ` Claudio Freire <[email protected]>
2017-07-18 09:20 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
1 sibling, 1 reply; 52+ messages in thread
From: Claudio Freire @ 2017-07-17 20:56 UTC (permalink / raw)
To: Charles Nadeau <[email protected]>; +Cc: Igor Neyman <[email protected]>; Jeff Janes <[email protected]>; pgsql-performance
On Fri, Jul 14, 2017 at 12:34 PM, Charles Nadeau
<[email protected]> wrote:
> Workers Planned: 12
> Workers Launched: 12
> Buffers: shared hit=728798037 read=82974833
> -> Hash Semi Join
> (cost=30059688.07..47951761.31 rows=220376 width=20) (actual
> time=1268845.181..2007864.725 rows=7057357 loops=13)
> Hash Cond: (flows.dstaddr =
> flows_1.dstaddr)
> Buffers: shared hit=728795193
> read=82974833
> -> Nested Loop
> (cost=0.03..17891246.86 rows=220376 width=20) (actual time=0.207..723790.283
> rows=37910370 loops=13)
> Buffers: shared hit=590692229
> read=14991777
> -> Parallel Seq Scan on flows
> (cost=0.00..16018049.14 rows=55094048 width=20) (actual
> time=0.152..566179.117 rows=45371630 loops=13)
> Buffers: shared
> hit=860990 read=14991777
> -> Index Only Scan using
> mynetworks_ipaddr_idx on mynetworks (cost=0.03..0.03 rows=1 width=8)
> (actual time=0.002..0.002 rows=1 loops=589831190)
> Index Cond: (ipaddr >>=
> (flows.srcaddr)::ip4r)
> Heap Fetches: 0
> Buffers: shared
> hit=589831203
12 workers on a parallel sequential scan on a RAID-10 volume of
rotating disks may not be a good idea.
Have you measured average request size and average wait times with iostat?
Run "iostat -x -m -d 60" while running the query and copy a few
relevant lines (or attach the whole thing). I suspect 12 parallel
sequential scans are degrading your array's performance to random I/O
performance, and that explains the 10MB/s very well (a rotating disk
will give you about 3-4MB/s at random I/O, and you've got 2 mirrors on
that array).
You could try setting the max_parallel_workers_per_gather to 2, which
should be the optimum allocation for your I/O layout.
You might also want to test switching to the deadline scheduler. While
the controller may get more aggregate thoughput rearranging your I/O
requests, high I/O latency will severly reduce postgres' ability to
saturate the I/O system itself, and deadlines tends to minimize
latency. I've had good results in the past using deadline, but take
this suggestion with a grain of salt, YMMV.
--
Sent via pgsql-performance mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 19:24 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-11 11:02 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 00:39 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-12 10:04 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 14:31 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-12 16:39 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-14 15:34 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-17 20:56 ` Re: Very poor read performance, query independent Claudio Freire <[email protected]>
@ 2017-07-18 09:20 ` Charles Nadeau <[email protected]>
2017-07-18 16:01 ` Re: Very poor read performance, query independent Claudio Freire <[email protected]>
2017-07-19 01:08 ` Re: Very poor read performance, query independent Scott Marlowe <[email protected]>
0 siblings, 2 replies; 52+ messages in thread
From: Charles Nadeau @ 2017-07-18 09:20 UTC (permalink / raw)
To: Claudio Freire <[email protected]>; +Cc: Igor Neyman <[email protected]>; Jeff Janes <[email protected]>; pgsql-performance
Claudio,
Find attached the iostat measured while redoing the query above
(iostat1.txt). sda holds my temp directory (noop i/o scheduler), sdb the
swap partition (cfq i/o scheduler) only and sdc (5 disks RAID0, noop i/o
scheduler) holds the data. I didn't pay attention to the load caused by 12
parallel scans as I thought the RAID card would be smart enough to
re-arrange the read requests optimally regardless of the load. At one
moment during the query, there is a write storm to the swap drive (a bit
like this case:
https://www.postgresql.org/message-id/AANLkTi%3Diw4fC2RgTxhw0aGpyXANhOT%3DXBnjLU1_v6PdA%40mail.gmail...).
I can hardly explain it as there is plenty of memory on this server. The
execution time of the query was 4801.1s (about 1h20min).
I reduced max_parallel_workers_per_gather to 2 and max_parallel_workers to
3, restarted postgresql then ran the query again while running iostat again
(iostat2.txt): The query ran much faster, 1992.8s (about 33min) instead of
4801.1s (about 1h20min) and the swap storm is gone! You were right about
the max_parallel_workers_per_gather!!
For the last test, I changed the scheduler on sdc to deadline (iostat3.txt)
keeping max_parallel_workers_per_gather=2 and max_parallel_workers=3 then
restarted postgresql. The execution time is almost the same: 1938.7s vs
1992.8s for the noop scheduler.
Thanks a lot for the suggestion, I'll keep my number of worker low to make
sure I maximize my array usage.
Charles
On Mon, Jul 17, 2017 at 10:56 PM, Claudio Freire <[email protected]>
wrote:
> On Fri, Jul 14, 2017 at 12:34 PM, Charles Nadeau
> <[email protected]> wrote:
> > Workers Planned: 12
> > Workers Launched: 12
> > Buffers: shared hit=728798037
> read=82974833
> > -> Hash Semi Join
> > (cost=30059688.07..47951761.31 rows=220376 width=20) (actual
> > time=1268845.181..2007864.725 rows=7057357 loops=13)
> > Hash Cond: (flows.dstaddr =
> > flows_1.dstaddr)
> > Buffers: shared hit=728795193
> > read=82974833
> > -> Nested Loop
> > (cost=0.03..17891246.86 rows=220376 width=20) (actual
> time=0.207..723790.283
> > rows=37910370 loops=13)
> > Buffers: shared
> hit=590692229
> > read=14991777
> > -> Parallel Seq Scan on
> flows
> > (cost=0.00..16018049.14 rows=55094048 width=20) (actual
> > time=0.152..566179.117 rows=45371630 loops=13)
> > Buffers: shared
> > hit=860990 read=14991777
> > -> Index Only Scan using
> > mynetworks_ipaddr_idx on mynetworks (cost=0.03..0.03 rows=1 width=8)
> > (actual time=0.002..0.002 rows=1 loops=589831190)
> > Index Cond: (ipaddr
> >>=
> > (flows.srcaddr)::ip4r)
> > Heap Fetches: 0
> > Buffers: shared
> > hit=589831203
>
> 12 workers on a parallel sequential scan on a RAID-10 volume of
> rotating disks may not be a good idea.
>
> Have you measured average request size and average wait times with iostat?
>
> Run "iostat -x -m -d 60" while running the query and copy a few
> relevant lines (or attach the whole thing). I suspect 12 parallel
> sequential scans are degrading your array's performance to random I/O
> performance, and that explains the 10MB/s very well (a rotating disk
> will give you about 3-4MB/s at random I/O, and you've got 2 mirrors on
> that array).
>
> You could try setting the max_parallel_workers_per_gather to 2, which
> should be the optimum allocation for your I/O layout.
>
> You might also want to test switching to the deadline scheduler. While
> the controller may get more aggregate thoughput rearranging your I/O
> requests, high I/O latency will severly reduce postgres' ability to
> saturate the I/O system itself, and deadlines tends to minimize
> latency. I've had good results in the past using deadline, but take
> this suggestion with a grain of salt, YMMV.
>
--
Charles Nadeau Ph.D.
http://charlesnadeau.blogspot.com/
charles@hpdl380g6:~$ iostat -x -m -d 60
Linux 4.4.0-86-generic (hpdl380g6) 2017-07-18 _x86_64_ (16 CPU)
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 1.45 15.24 3.85 4.91 0.68 0.47 269.64 3.15 351.06 501.22 233.27 4.84 4.24
sdb 1.52 635.12 67.38 10.80 0.27 2.54 73.58 7.66 97.90 28.43 531.24 2.26 17.65
sdc 0.08 0.01 47.90 0.01 15.62 0.00 667.83 0.97 20.30 20.30 6.89 3.05 14.62
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.30 0.03 4.95 0.00 0.09 35.83 0.00 0.56 8.00 0.51 0.13 0.07
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.07 0.23 169.95 0.40 5.66 0.01 68.11 0.85 4.98 5.00 0.00 4.02 68.43
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.42 0.00 5.02 0.00 0.09 35.88 0.00 0.19 0.00 0.19 0.03 0.01
sdb 0.13 0.00 0.07 0.00 0.00 0.00 24.00 0.00 7.00 7.00 0.00 7.00 0.05
sdc 0.12 0.00 231.63 0.00 7.20 0.00 63.68 1.07 4.61 4.61 0.00 3.64 84.21
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.18 0.00 4.63 0.00 0.09 37.73 0.00 0.03 0.00 0.03 0.03 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 206.27 0.00 6.09 0.00 60.43 1.13 5.47 5.47 0.00 3.91 80.63
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.38 0.00 4.73 0.00 0.09 37.44 0.00 0.03 0.00 0.03 0.03 0.01
sdb 0.08 0.00 0.02 0.00 0.00 0.00 48.00 0.00 16.00 16.00 0.00 16.00 0.03
sdc 0.02 0.00 234.78 0.00 3.84 0.00 33.52 0.80 3.41 3.41 0.00 2.84 66.75
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.13 0.00 4.62 0.00 0.09 37.72 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 36.75 0.00 9.57 0.00 533.07 0.45 12.24 12.24 0.00 3.84 14.11
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.15 0.00 4.95 0.00 0.09 35.80 0.00 0.78 0.00 0.78 0.03 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.02 0.00 172.67 0.00 9.22 0.00 109.34 0.96 5.56 5.56 0.00 3.88 67.07
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.48 0.08 5.08 0.00 0.09 36.13 0.00 0.17 5.60 0.08 0.12 0.06
sdb 0.00 0.00 0.02 0.00 0.00 0.00 8.00 0.00 16.00 16.00 0.00 16.00 0.03
sdc 0.00 0.00 149.77 0.00 5.01 0.00 68.48 0.82 5.51 5.51 0.00 3.83 57.33
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.02 0.00 4.70 0.00 0.08 37.02 0.00 0.13 0.00 0.13 0.04 0.02
sdb 0.07 0.00 0.02 0.00 0.00 0.00 40.00 0.00 16.00 16.00 0.00 16.00 0.03
sdc 0.00 0.00 227.00 0.00 6.38 0.00 57.55 2.75 12.13 12.13 0.00 2.62 59.59
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.25 0.00 4.90 0.00 0.09 36.24 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 154.33 0.00 5.60 0.00 74.25 1.39 9.01 9.01 0.00 3.35 51.77
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.92 0.00 4.68 0.00 0.08 36.93 0.01 2.38 0.00 2.38 0.07 0.03
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 182.73 0.00 6.07 0.00 68.04 1.92 10.53 10.53 0.00 2.95 53.95
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.57 0.00 4.98 0.00 0.09 36.28 0.01 2.42 0.00 2.42 0.07 0.03
sdb 0.00 0.00 0.02 0.00 0.00 0.00 8.00 0.00 12.00 12.00 0.00 12.00 0.02
sdc 0.07 0.00 314.02 0.00 5.62 0.00 36.65 2.26 7.18 7.18 0.00 2.52 79.09
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.85 0.03 4.60 0.00 0.08 37.44 0.00 0.04 6.00 0.00 0.04 0.02
sdb 0.05 0.00 0.02 0.00 0.00 0.00 32.00 0.00 12.00 12.00 0.00 12.00 0.02
sdc 0.03 0.00 176.68 0.00 5.48 0.00 63.51 1.04 5.90 5.90 0.00 3.18 56.23
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 119.87 21.48 107.68 7.72 1.48 0.19 29.61 0.38 3.25 3.42 0.90 2.59 29.92
sdb 4.00 0.00 7.55 0.00 0.05 0.00 12.24 0.04 5.70 5.70 0.00 2.89 2.18
sdc 0.00 0.05 264.52 0.15 7.43 0.00 57.51 1.43 5.42 5.42 0.00 2.86 75.73
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.43 0.02 4.50 0.00 0.08 37.17 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.05 0.00 0.02 0.00 0.00 0.00 32.00 0.00 8.00 8.00 0.00 8.00 0.01
sdc 0.00 0.00 247.80 0.00 8.15 0.00 67.36 1.60 6.45 6.45 0.00 2.76 68.44
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.35 0.00 4.82 0.00 0.09 36.87 0.00 0.19 0.00 0.19 0.03 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.08 0.00 278.37 0.00 9.29 0.00 68.31 2.22 7.97 7.97 0.00 2.32 64.48
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 10.97 0.00 4.12 0.00 0.06 31.32 0.00 0.02 0.00 0.02 0.02 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 480.53 0.00 5.97 0.00 25.46 2.20 4.59 4.59 0.00 1.84 88.33
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.30 0.12 4.97 0.00 0.09 36.12 0.00 0.18 8.00 0.00 0.18 0.09
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 486.63 0.00 5.11 0.00 21.51 2.32 4.77 4.77 0.00 1.82 88.69
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.33 0.00 4.88 0.00 0.09 36.40 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 599.40 0.00 5.79 0.00 19.77 1.79 2.99 2.99 0.00 1.42 84.92
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.93 0.00 4.72 0.00 0.08 36.72 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 545.27 0.00 4.89 0.00 18.35 1.37 2.51 2.51 0.00 1.64 89.15
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.20 0.02 4.90 0.00 0.09 35.99 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.07 0.00 63.77 0.00 34.92 0.00 1121.49 1.15 17.96 17.96 0.00 2.08 13.25
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.12 0.00 4.62 0.00 0.08 37.66 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.15 0.00 199.00 0.00 33.88 0.00 348.71 3.12 15.68 15.68 0.00 1.07 21.35
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.45 0.10 4.98 0.00 0.09 35.93 0.00 0.16 8.00 0.00 0.16 0.08
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.10 0.00 401.32 0.00 23.90 0.00 121.97 5.42 13.52 13.52 0.00 1.10 44.27
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 17.05 0.03 5.25 0.00 0.09 35.36 0.00 0.11 6.00 0.08 0.11 0.06
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.02 0.00 242.53 0.00 26.70 0.00 225.44 4.31 17.78 17.78 0.00 1.05 25.40
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.15 0.00 5.00 0.00 0.09 35.41 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.37 0.00 225.52 0.00 173.64 0.00 1576.91 7.19 31.91 31.91 0.00 2.02 45.58
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.20 0.00 4.62 0.00 0.09 37.83 0.00 0.35 0.00 0.35 0.03 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.80 0.00 239.58 0.00 236.94 0.00 2025.38 6.87 28.66 28.66 0.00 2.27 54.33
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.45 0.17 5.12 0.01 0.09 36.57 0.01 1.06 7.20 0.86 0.28 0.15
sdb 0.00 0.00 0.05 0.00 0.00 0.00 8.00 0.00 16.00 16.00 0.00 5.33 0.03
sdc 1.25 0.00 304.82 0.00 300.68 0.00 2020.22 10.87 35.62 35.62 0.00 2.40 73.05
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.42 0.02 4.88 0.00 0.09 37.44 0.00 0.04 8.00 0.01 0.04 0.02
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 1.32 0.00 351.78 0.00 348.00 0.00 2025.96 13.64 38.83 38.83 0.00 2.44 85.77
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.27 0.00 4.97 0.00 0.09 35.87 0.00 0.01 0.00 0.01 0.01 0.01
sdb 0.00 0.00 0.02 0.00 0.00 0.00 8.00 0.00 8.00 8.00 0.00 8.00 0.01
sdc 1.10 0.00 337.22 0.00 333.62 0.00 2026.17 11.89 35.24 35.24 0.00 2.38 80.41
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.43 0.05 5.12 0.00 0.09 35.02 0.00 0.17 5.33 0.12 0.09 0.05
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 1.58 0.00 363.97 0.00 360.07 0.00 2026.08 15.16 41.67 41.67 0.00 2.44 88.81
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.20 0.00 4.75 0.00 0.09 37.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.87 0.00 320.37 0.00 316.34 0.00 2022.27 10.54 32.91 32.91 0.00 2.29 73.52
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.17 0.00 3.93 0.00 0.08 40.03 0.01 1.69 0.00 1.69 0.03 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.78 1.47 4.80 0.02 0.09 33.51 0.01 2.13 9.00 0.03 0.35 0.22
sdb 0.18 0.00 0.07 0.00 0.00 0.00 30.00 0.00 31.00 31.00 0.00 31.00 0.21
sdc 0.00 0.00 0.35 0.00 0.00 0.00 27.43 0.00 6.29 6.29 0.00 6.29 0.22
charles@hpdl380g6:~$ iostat -x -m -d 60
Linux 4.4.0-86-generic (hpdl380g6) 2017-07-18 _x86_64_ (16 CPU)
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 1.49 15.22 3.98 4.92 0.71 0.48 273.71 3.25 357.20 501.44 240.57 4.92 4.38
sdb 1.57 655.83 69.58 11.15 0.28 2.62 73.58 7.91 97.91 28.43 531.24 2.26 18.23
sdc 0.08 0.01 42.10 0.01 14.16 0.00 688.62 0.90 21.34 21.34 7.59 3.18 13.40
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.82 0.47 5.20 0.00 0.09 33.36 0.01 1.64 13.29 0.59 0.15 0.09
sdb 0.73 0.00 0.25 0.00 0.00 0.00 31.47 0.00 2.40 2.40 0.00 1.60 0.04
sdc 0.00 0.20 196.78 0.37 6.34 0.00 65.87 1.04 5.25 5.26 0.00 4.14 81.68
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.28 0.00 4.77 0.00 0.09 37.01 0.00 0.45 0.00 0.45 0.06 0.03
sdb 0.13 0.00 0.05 0.00 0.00 0.00 29.33 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.13 0.00 229.25 0.00 6.99 0.00 62.45 1.06 4.61 4.61 0.00 3.68 84.37
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.02 0.00 4.82 0.00 0.09 36.43 0.01 2.19 0.00 2.19 0.06 0.03
sdb 0.10 0.00 0.02 0.00 0.00 0.00 56.00 0.00 4.00 4.00 0.00 4.00 0.01
sdc 0.00 0.00 213.52 0.00 4.89 0.00 46.92 0.94 4.38 4.38 0.00 3.81 81.39
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.58 0.13 5.25 0.00 0.09 34.65 0.00 0.62 9.00 0.41 0.25 0.13
sdb 0.20 0.00 0.23 0.00 0.00 0.00 14.86 0.00 2.29 2.29 0.00 2.29 0.05
sdc 0.02 0.00 179.37 0.00 5.10 0.00 58.18 0.62 3.47 3.47 0.00 2.87 51.51
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 10.75 0.00 3.72 0.00 0.06 34.08 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.20 0.00 0.05 0.00 0.00 0.00 40.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 69.87 0.00 8.79 0.00 257.73 0.55 7.87 7.87 0.00 4.12 28.81
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.32 0.00 5.03 0.00 0.09 35.76 0.00 0.21 0.00 0.21 0.03 0.01
sdb 0.60 0.00 0.12 0.00 0.00 0.00 49.14 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.03 0.00 163.32 0.00 8.92 0.00 111.88 0.96 5.85 5.85 0.00 3.85 62.90
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.33 0.00 4.90 0.00 0.09 36.30 0.00 0.63 0.00 0.63 0.08 0.04
sdb 0.22 0.00 0.08 0.00 0.00 0.00 28.80 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 160.37 0.00 5.22 0.00 66.63 1.22 7.58 7.58 0.00 3.61 57.81
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.25 0.00 4.90 0.00 0.09 36.30 0.00 0.35 0.00 0.35 0.05 0.03
sdb 0.13 0.00 0.05 0.00 0.00 0.00 29.33 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 222.75 0.00 6.29 0.00 57.81 2.71 12.15 12.15 0.00 2.65 59.14
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.25 0.00 4.78 0.00 0.09 36.85 0.00 0.17 0.00 0.17 0.03 0.01
sdb 0.17 0.00 0.03 0.00 0.00 0.00 48.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 153.22 0.00 5.67 0.00 75.85 1.34 8.72 8.72 0.00 3.35 51.35
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.20 0.08 4.95 0.00 0.09 35.44 0.00 0.12 7.20 0.00 0.12 0.06
sdb 0.32 0.00 0.10 0.00 0.00 0.00 33.33 0.00 2.67 2.67 0.00 2.67 0.03
sdc 0.00 0.00 194.87 0.00 5.50 0.00 57.84 1.68 8.61 8.61 0.00 3.06 59.62
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.25 0.02 4.80 0.00 0.09 36.82 0.01 1.13 8.00 1.11 0.11 0.05
sdb 0.58 0.00 0.17 0.00 0.00 0.00 36.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.02 0.00 286.10 0.00 5.93 0.00 42.43 2.37 8.27 8.27 0.00 2.60 74.31
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.28 0.02 4.90 0.00 0.09 36.18 0.00 0.28 0.00 0.29 0.04 0.02
sdb 0.12 0.00 0.02 0.00 0.00 0.00 64.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.02 0.00 174.37 0.00 4.51 0.00 52.94 0.90 5.14 5.14 0.00 3.60 62.70
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.33 0.03 4.75 0.01 0.09 41.56 0.00 0.78 12.00 0.70 0.11 0.05
sdb 0.17 0.00 0.03 0.00 0.00 0.00 48.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 248.05 0.00 7.15 0.00 59.02 1.42 5.72 5.72 0.00 3.07 76.12
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.28 0.03 4.77 0.00 0.09 38.31 0.01 1.68 10.00 1.62 0.10 0.05
sdb 0.10 0.00 0.07 0.00 0.00 0.00 20.00 0.00 1.00 1.00 0.00 1.00 0.01
sdc 0.00 0.00 277.58 0.00 7.64 0.00 56.35 1.71 6.15 6.15 0.00 2.77 76.85
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.02 16.22 0.23 4.90 0.02 0.09 43.69 0.00 0.34 7.43 0.00 0.34 0.17
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.12 0.00 238.58 0.00 9.08 0.00 77.96 2.29 9.61 9.61 0.00 2.37 56.60
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.42 0.00 4.95 0.00 0.09 36.23 0.00 0.03 0.00 0.03 0.03 0.01
sdb 0.18 0.00 0.05 0.00 0.00 0.00 37.33 0.00 5.33 5.33 0.00 5.33 0.03
sdc 0.00 0.00 445.87 0.00 6.28 0.00 28.86 1.78 3.99 3.99 0.00 2.04 90.78
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.17 0.00 4.88 0.00 0.09 36.15 0.00 0.27 0.00 0.27 0.05 0.03
sdb 0.02 0.00 0.02 0.00 0.00 0.00 16.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 504.27 0.00 5.31 0.00 21.55 3.37 6.68 6.68 0.00 1.80 90.79
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.87 0.00 4.62 0.00 0.08 37.23 0.00 0.01 0.00 0.01 0.01 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 580.73 0.00 5.57 0.00 19.66 1.59 2.74 2.74 0.00 1.48 85.95
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.12 0.03 4.82 0.00 0.09 38.21 0.00 0.29 42.00 0.00 0.15 0.07
sdb 0.05 0.00 0.03 0.00 0.00 0.00 20.00 0.00 6.00 6.00 0.00 6.00 0.02
sdc 0.00 0.00 590.58 0.00 5.39 0.00 18.70 1.59 2.69 2.69 0.00 1.54 90.67
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 17.03 0.05 5.07 0.00 0.09 36.43 0.00 0.07 6.67 0.00 0.07 0.03
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.05 0.00 162.92 0.00 27.80 0.00 349.48 1.15 7.07 7.07 0.00 1.88 30.67
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.75 0.02 5.05 0.00 0.09 36.32 0.00 0.09 4.00 0.08 0.09 0.05
sdb 0.15 0.00 0.05 0.00 0.00 0.00 32.00 0.00 6.67 6.67 0.00 6.67 0.03
sdc 0.18 0.00 119.80 0.00 33.20 0.00 567.62 1.75 14.64 14.64 0.00 1.65 19.83
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 12.55 0.00 4.13 0.00 0.07 34.16 0.00 0.58 0.00 0.58 0.06 0.03
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.12 0.00 396.90 0.00 21.63 0.00 111.61 3.09 7.79 7.79 0.00 1.37 54.50
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 20.68 0.00 5.90 0.00 0.11 37.47 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.02 0.00 0.02 0.00 0.00 0.00 16.00 0.00 12.00 12.00 0.00 12.00 0.02
sdc 0.08 0.00 469.32 0.00 24.58 0.00 107.25 7.45 15.87 15.87 0.00 0.84 39.57
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 9.73 0.00 3.62 0.00 0.06 31.67 0.00 1.22 0.00 1.22 0.11 0.04
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.18 0.00 307.52 0.00 63.52 0.00 423.05 8.39 27.28 27.28 0.00 0.86 26.41
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.65 0.17 5.17 0.02 0.09 42.23 0.00 0.74 9.20 0.46 0.23 0.12
sdb 0.12 0.00 0.02 0.00 0.00 0.00 64.00 0.00 12.00 12.00 0.00 12.00 0.02
sdc 0.77 0.00 231.62 0.00 228.86 0.00 2023.61 7.72 33.29 33.29 0.00 2.34 54.13
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.72 0.00 4.93 0.00 0.09 36.84 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.07 0.00 0.03 0.00 0.00 0.00 24.00 0.00 8.00 8.00 0.00 8.00 0.03
sdc 0.98 0.00 244.73 0.00 242.63 0.00 2030.40 7.49 30.58 30.58 0.00 2.31 56.65
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.18 0.00 4.88 0.00 0.09 36.26 0.01 1.17 0.00 1.17 0.03 0.01
sdb 0.03 0.00 0.05 0.00 0.00 0.00 13.33 0.00 5.33 5.33 0.00 4.00 0.02
sdc 1.62 0.00 340.48 0.00 335.98 0.00 2020.93 12.83 37.72 37.72 0.00 2.42 82.37
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.77 0.00 4.65 0.00 0.08 36.79 0.00 0.52 0.00 0.52 0.03 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 1.68 0.00 342.80 0.00 339.21 0.00 2026.56 12.70 37.04 37.04 0.00 2.43 83.17
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.65 0.00 5.00 0.00 0.09 36.35 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.02 0.00 0.02 0.00 0.00 0.00 16.00 0.00 12.00 12.00 0.00 12.00 0.02
sdc 1.67 0.00 356.23 0.00 350.65 0.00 2015.93 14.08 39.54 39.54 0.00 2.39 85.03
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.75 0.23 4.75 0.03 0.08 46.07 0.01 1.57 8.86 1.21 0.37 0.19
sdb 0.02 0.00 0.02 0.00 0.00 0.00 16.00 0.00 16.00 16.00 0.00 16.00 0.03
sdc 1.15 0.00 351.48 0.00 347.52 0.00 2024.92 12.89 36.67 36.67 0.00 2.37 83.19
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.53 0.00 4.97 0.00 0.09 36.13 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.75 0.00 0.30 0.00 0.00 0.00 28.00 0.00 2.89 2.89 0.00 2.89 0.09
sdc 0.65 0.00 179.88 0.00 178.26 0.00 2029.51 5.47 30.43 30.43 0.00 2.32 41.75
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.58 0.00 4.25 0.00 0.08 38.09 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.60 0.00 4.40 0.00 0.08 37.21 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
charles@hpdl380g6:~$ iostat -x -m -d 60
Linux 4.4.0-86-generic (hpdl380g6) 2017-07-18 _x86_64_ (16 CPU)
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 1.19 15.29 3.36 4.96 0.66 0.51 289.56 3.00 353.20 499.26 254.22 4.61 3.83
sdb 1.41 557.27 59.71 9.44 0.24 2.23 73.06 6.71 97.05 27.92 534.28 2.23 15.40
sdc 0.06 0.01 37.03 0.01 13.02 0.00 720.10 0.81 21.77 21.77 9.30 3.17 11.73
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 11.50 3.68 4.25 0.05 0.07 30.86 0.04 5.18 9.25 1.66 1.09 0.87
sdb 9.57 0.00 21.02 0.00 0.12 0.00 11.64 0.03 1.37 1.37 0.00 1.17 2.45
sdc 0.07 1.62 87.48 1.13 2.71 0.02 63.17 0.44 4.93 4.99 0.00 4.23 37.51
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 19.55 0.03 5.47 0.00 0.10 37.89 0.00 0.21 6.00 0.17 0.06 0.03
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.02 0.02 193.13 0.13 6.54 0.00 69.34 0.99 5.11 5.11 0.00 4.08 78.92
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.22 0.07 4.80 0.00 0.09 36.38 0.00 0.34 6.00 0.26 0.12 0.06
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.12 0.00 221.63 0.00 6.61 0.00 61.04 1.03 4.62 4.62 0.00 3.70 81.92
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 11.98 4.18 4.13 0.03 0.07 22.89 0.04 4.64 7.63 1.61 0.62 0.51
sdb 0.03 0.00 0.05 0.00 0.00 0.00 13.33 0.00 16.00 16.00 0.00 8.00 0.04
sdc 0.00 0.00 226.45 0.00 3.80 0.00 34.33 0.87 3.83 3.83 0.00 3.51 79.53
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.10 0.00 4.72 0.00 0.09 37.00 0.00 0.03 0.00 0.03 0.03 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.02 0.00 114.75 0.00 4.22 0.00 75.37 0.41 3.55 3.55 0.00 2.82 32.41
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.20 0.00 4.57 0.00 0.09 38.13 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 5.60 0.00 5.60 0.00 2048.00 0.18 32.05 32.05 0.00 2.51 1.41
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.28 0.00 4.68 0.00 0.09 37.58 0.00 0.01 0.00 0.01 0.01 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 143.55 0.00 6.30 0.00 89.85 0.78 5.41 5.41 0.00 4.01 57.63
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.38 0.05 4.92 0.00 0.09 36.62 0.00 0.07 6.67 0.00 0.07 0.03
sdb 0.95 0.00 0.88 0.00 0.01 0.00 16.60 0.00 4.60 4.60 0.00 4.15 0.37
sdc 0.08 0.00 121.30 0.00 6.96 0.00 117.49 0.75 6.16 6.16 0.00 4.03 48.85
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.18 0.00 4.90 0.00 0.09 36.16 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 129.38 0.00 4.47 0.00 70.71 1.01 7.82 7.82 0.00 3.64 47.09
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.23 0.00 4.67 0.00 0.09 37.54 0.01 1.14 0.00 1.14 0.03 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 197.12 0.00 5.50 0.00 57.12 2.18 11.08 11.08 0.00 2.66 52.47
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.28 0.00 4.75 0.00 0.09 37.16 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 123.97 0.00 4.46 0.00 73.63 1.01 8.17 8.17 0.00 3.60 44.64
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.27 0.00 4.73 0.00 0.09 37.18 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 160.92 0.00 5.58 0.00 71.04 1.99 12.37 12.37 0.00 2.76 44.43
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.58 0.00 5.03 0.00 0.09 36.00 0.00 0.77 0.00 0.77 0.03 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 275.52 0.00 4.90 0.00 36.43 1.32 4.80 4.80 0.00 2.85 78.45
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.12 0.00 4.85 0.00 0.09 36.26 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.03 0.00 154.33 0.00 5.27 0.00 69.89 1.72 11.14 11.14 0.00 2.78 42.91
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.23 0.00 4.80 0.00 0.09 36.72 0.00 0.57 0.00 0.57 0.07 0.03
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 198.68 0.00 5.02 0.00 51.78 1.03 5.21 5.21 0.00 3.47 68.93
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.27 0.00 4.87 0.00 0.09 36.44 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 278.92 0.00 5.47 0.00 40.14 1.60 5.73 5.73 0.00 2.72 75.80
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.20 0.00 4.75 0.00 0.09 36.97 0.00 0.08 0.00 0.08 0.03 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 164.75 0.00 7.00 0.00 87.07 0.95 5.77 5.77 0.00 3.04 50.04
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.22 0.00 4.83 0.00 0.09 36.94 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 212.92 0.00 5.69 0.00 54.73 2.07 9.71 9.71 0.00 2.29 48.76
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.52 0.00 4.48 0.00 0.08 37.53 0.00 0.21 0.00 0.21 0.03 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.02 0.00 409.77 0.00 8.30 0.00 41.47 1.75 4.28 4.28 0.00 2.02 82.83
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.10 0.03 4.73 0.00 0.08 35.05 0.00 0.53 6.00 0.49 0.11 0.05
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 499.55 0.00 5.28 0.00 21.65 3.26 6.52 6.52 0.00 1.78 89.04
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.17 0.00 4.88 0.00 0.08 34.48 0.00 0.25 0.00 0.25 0.03 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 539.70 0.00 5.18 0.00 19.66 1.42 2.63 2.63 0.00 1.56 84.17
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 17.63 0.00 5.07 0.00 0.09 37.42 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 604.53 0.00 5.61 0.00 19.00 1.84 3.04 3.04 0.00 1.45 87.87
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.25 0.03 4.80 0.00 0.09 36.55 0.00 0.06 8.00 0.00 0.06 0.03
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 253.27 0.00 12.74 0.00 103.03 0.83 3.29 3.29 0.00 1.85 46.86
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 30.00 9.88 49.55 4.24 3.81 0.06 147.23 30.67 512.55 511.82 521.06 10.90 58.65
sdb 1.31 11581.46 6.95 137.86 0.03 45.68 646.56 69.06 474.75 65.02 495.40 3.98 57.62
sdc 0.53 0.00 27.42 0.00 19.54 0.00 1459.84 3.48 126.91 126.91 0.00 18.20 49.89
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 25.82 1.61 64.91 1.06 8.34 0.01 259.24 56.68 810.89 809.60 889.94 14.01 92.45
sdb 3.39 18155.76 27.35 294.55 0.12 71.70 456.97 158.73 492.62 205.87 519.24 2.99 96.12
sdc 0.88 0.00 28.30 0.00 17.38 0.00 1257.54 3.94 139.16 139.16 0.00 26.14 73.96
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 28.34 7.11 72.85 2.30 9.09 0.04 248.86 65.05 883.90 878.77 1046.59 15.16 113.96
sdb 5.59 20757.70 31.61 337.69 0.15 82.26 456.98 201.34 542.84 233.68 571.78 3.08 113.80
sdc 0.92 0.00 25.53 0.00 17.98 0.00 1442.16 4.30 168.36 168.36 0.00 31.06 79.30
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 18.94 2.17 68.16 1.23 9.78 0.01 289.19 51.19 736.34 735.41 788.11 14.35 99.56
sdb 1.44 17642.69 20.33 285.22 0.09 72.14 484.09 199.75 656.89 479.91 669.51 3.27 99.78
sdc 0.96 0.00 27.05 0.00 17.62 0.00 1333.71 3.84 142.10 142.10 0.00 26.13 70.69
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 34.81 0.53 59.38 0.79 8.98 0.01 305.84 54.65 887.48 873.79 1918.43 15.36 92.42
sdb 3.83 16364.59 32.06 259.40 0.14 65.75 462.97 162.40 557.08 255.41 594.36 3.17 92.40
sdc 1.04 0.00 23.15 0.00 15.56 0.00 1376.79 3.68 158.93 158.93 0.00 27.97 64.75
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 38.04 1.03 138.97 1.00 17.55 0.01 256.92 70.25 487.79 478.46 1783.75 6.87 96.16
sdb 5.88 16247.37 46.64 309.44 0.21 64.82 374.00 160.17 449.56 103.93 501.65 2.69 95.89
sdc 0.92 0.00 21.85 0.00 14.27 0.00 1337.21 3.81 174.32 174.32 0.00 25.65 56.04
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 87.16 6.01 144.64 3.02 12.88 0.04 179.13 44.17 306.55 308.78 199.54 6.41 94.58
sdb 19.42 13537.32 57.67 281.56 0.30 54.66 331.83 109.99 324.35 51.45 380.25 2.58 87.69
sdc 0.81 0.00 116.85 0.00 14.83 0.00 259.96 2.82 24.10 24.10 0.00 6.83 79.81
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 35.47 1.08 79.00 1.21 10.52 0.01 268.79 65.35 722.25 719.59 895.29 12.84 102.98
sdb 4.63 18098.84 45.12 373.82 0.19 72.61 355.91 193.68 462.20 100.15 505.89 2.42 101.49
sdc 1.03 0.00 26.84 0.00 16.93 0.00 1291.56 3.64 135.45 135.45 0.00 23.77 63.82
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 64.93 1.57 83.11 1.81 9.05 0.01 218.46 61.38 722.98 721.80 777.35 12.09 102.69
sdb 6.88 17618.66 41.79 331.23 0.19 69.80 384.29 158.30 422.96 110.46 462.39 2.68 99.96
sdc 0.88 0.00 24.74 0.00 13.35 0.00 1104.93 2.97 119.95 119.95 0.00 23.91 59.16
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 74.98 13.69 139.76 4.65 10.80 0.07 154.27 19.27 144.73 146.79 82.88 3.95 57.06
sdb 29.31 6576.10 110.89 94.84 0.55 27.18 275.97 43.53 214.17 22.40 438.39 2.55 52.56
sdc 0.56 0.00 146.95 0.00 10.70 0.00 149.14 1.64 11.15 11.15 0.00 4.52 66.46
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 4.87 14.15 37.29 5.07 1.12 0.08 58.17 0.24 5.70 6.32 1.12 1.17 4.96
sdb 13.62 0.00 246.84 0.00 1.02 0.00 8.44 6.69 27.19 27.19 0.00 2.30 56.86
sdc 0.02 0.00 133.77 0.00 1.37 0.00 21.03 0.49 3.63 3.63 0.00 3.52 47.06
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 18.25 0.82 5.20 0.01 0.10 37.27 0.01 1.00 7.35 0.00 0.30 0.18
sdb 0.15 0.00 1.00 0.00 0.00 0.00 9.20 0.00 2.00 2.00 0.00 1.87 0.19
sdc 0.00 0.00 191.27 0.00 10.50 0.00 112.44 0.90 4.69 4.69 0.00 2.87 54.93
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.28 0.00 4.88 0.00 0.09 36.53 0.00 0.19 0.00 0.19 0.03 0.01
sdb 1.58 0.00 139.63 0.00 0.55 0.00 8.09 3.74 26.69 26.69 0.00 2.20 30.69
sdc 0.00 0.00 86.55 0.00 8.18 0.00 193.51 0.52 6.03 6.03 0.00 3.32 28.74
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 11.80 0.02 4.20 0.00 0.07 32.28 0.00 0.32 8.00 0.29 0.13 0.05
sdb 3.23 0.00 447.23 0.00 1.76 0.00 8.06 13.00 29.04 29.04 0.00 2.24 100.00
sdc 0.00 0.00 1.32 0.00 0.01 0.00 16.00 0.01 4.46 4.46 0.00 4.46 0.59
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 1.37 16.38 8.88 4.73 0.27 0.09 53.52 0.08 5.89 7.34 3.18 1.41 1.91
sdb 4.05 0.00 448.40 0.00 1.77 0.00 8.07 13.02 29.07 29.07 0.00 2.23 100.00
sdc 0.00 0.00 0.40 0.00 0.00 0.00 16.00 0.01 21.67 21.67 0.00 21.67 0.87
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.15 0.00 5.15 0.00 0.09 34.69 0.00 0.00 0.00 0.00 0.00 0.00
sdb 3.10 0.00 447.02 0.00 1.76 0.00 8.06 12.98 29.03 29.03 0.00 2.24 100.00
sdc 0.00 0.00 1.95 0.00 0.02 0.00 16.00 0.01 4.58 4.58 0.00 4.58 0.89
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.05 0.00 4.77 0.00 0.09 36.56 0.00 0.00 0.00 0.00 0.00 0.00
sdb 1.40 0.00 444.00 0.00 1.74 0.00 8.03 12.99 29.26 29.26 0.00 2.25 100.00
sdc 0.00 0.00 0.50 0.00 0.00 0.00 16.00 0.00 5.33 5.33 0.00 5.33 0.27
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.93 0.00 4.75 0.00 0.08 36.13 0.00 0.20 0.00 0.20 0.03 0.01
sdb 0.68 0.00 444.13 0.00 1.74 0.00 8.01 12.98 29.23 29.23 0.00 2.25 100.00
sdc 0.00 0.00 0.28 0.00 0.00 0.00 16.00 0.00 7.76 7.76 0.00 7.76 0.22
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.90 0.02 4.70 0.00 0.08 37.06 0.00 0.06 16.00 0.00 0.06 0.03
sdb 2.63 0.00 444.48 0.00 1.75 0.00 8.05 13.01 29.27 29.27 0.00 2.25 100.00
sdc 0.00 0.00 0.22 0.00 0.00 0.00 19.69 0.00 5.54 5.54 0.00 5.54 0.12
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.28 0.60 4.85 0.00 0.09 33.17 0.01 1.44 8.44 0.58 0.20 0.11
sdb 3.03 0.00 445.87 0.00 1.75 0.00 8.05 13.01 29.17 29.17 0.00 2.24 100.00
sdc 0.00 0.00 0.47 0.00 0.00 0.00 16.00 0.00 5.57 5.57 0.00 5.57 0.26
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.20 0.02 4.95 0.01 0.09 39.11 0.00 0.05 16.00 0.00 0.05 0.03
sdb 2.55 0.00 444.58 0.00 1.75 0.00 8.05 12.99 29.22 29.22 0.00 2.25 100.00
sdc 0.00 0.00 0.82 0.00 0.01 0.00 16.00 0.00 5.55 5.55 0.00 5.47 0.45
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.22 0.00 4.85 0.00 0.09 36.62 0.01 1.90 0.00 1.90 0.08 0.04
sdb 2.52 0.00 445.68 0.00 1.75 0.00 8.05 12.82 28.77 28.77 0.00 2.24 100.00
sdc 0.00 0.00 27.53 0.00 1.78 0.00 132.67 0.14 5.23 5.23 0.00 3.44 9.47
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.30 0.00 4.98 0.00 0.09 35.93 0.00 0.00 0.00 0.00 0.00 0.00
sdb 3.23 0.00 427.80 0.00 1.68 0.00 8.06 10.93 25.55 25.55 0.00 2.34 100.00
sdc 0.00 0.00 222.67 0.00 2.32 0.00 21.34 0.77 3.45 3.45 0.00 3.33 74.06
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.33 0.00 4.85 0.00 0.09 36.81 0.00 0.00 0.00 0.00 0.00 0.00
sdb 1.02 0.00 123.90 0.00 0.49 0.00 8.07 1.56 12.66 12.66 0.00 3.42 42.38
sdc 0.03 0.00 117.07 0.00 16.07 0.00 281.14 0.82 6.97 6.97 0.00 2.53 29.58
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.32 0.00 4.92 0.00 0.09 36.80 0.00 0.11 0.00 0.11 0.03 0.01
sdb 1.77 0.00 178.93 0.00 0.71 0.00 8.08 4.49 25.02 25.02 0.00 2.42 43.27
sdc 0.00 0.00 96.32 0.00 5.55 0.00 117.91 0.52 5.40 5.40 0.00 3.52 33.93
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.27 0.00 4.87 0.00 0.09 36.68 0.00 0.00 0.00 0.00 0.00 0.00
sdb 1.90 0.00 298.08 0.00 1.17 0.00 8.05 7.83 26.32 26.32 0.00 2.30 68.68
sdc 0.00 0.00 68.77 0.00 4.26 0.00 127.02 0.38 5.46 5.46 0.00 3.53 24.29
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.87 0.02 4.48 0.00 0.08 38.43 0.01 2.00 4.00 1.99 0.04 0.02
sdb 1.77 0.00 154.85 0.00 0.61 0.00 8.09 4.31 27.75 27.75 0.00 2.19 33.95
sdc 0.00 0.00 137.35 0.00 4.49 0.00 66.89 0.74 5.36 5.36 0.00 3.79 52.01
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.78 0.00 4.58 0.00 0.08 36.60 0.00 0.44 0.00 0.44 0.06 0.03
sdb 5.83 0.00 454.80 0.00 1.80 0.00 8.10 13.03 28.64 28.64 0.00 2.20 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.82 0.00 4.43 0.00 0.08 37.50 0.00 0.24 0.00 0.24 0.03 0.01
sdb 5.35 0.00 444.33 0.00 1.76 0.00 8.10 13.02 29.31 29.31 0.00 2.25 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.02 0.00 4.73 0.00 0.08 36.03 0.01 1.48 0.00 1.48 0.04 0.02
sdb 5.58 0.00 448.57 0.00 1.77 0.00 8.10 13.05 29.07 29.07 0.00 2.23 100.00
sdc 0.00 0.00 0.02 0.00 0.00 0.00 8.00 0.00 12.00 12.00 0.00 12.00 0.02
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.00 0.00 4.57 0.00 0.08 35.18 0.00 0.39 0.00 0.39 0.07 0.03
sdb 5.88 0.00 445.73 0.00 1.76 0.00 8.11 13.02 29.20 29.20 0.00 2.24 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 13.88 0.00 4.27 0.00 0.07 34.97 0.00 0.00 0.00 0.00 0.00 0.00
sdb 5.62 0.00 451.23 0.00 1.78 0.00 8.10 13.04 28.92 28.92 0.00 2.22 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 13.48 0.00 4.15 0.00 0.07 35.08 0.00 0.02 0.00 0.02 0.02 0.01
sdb 6.35 0.00 446.07 0.00 1.77 0.00 8.11 13.03 29.22 29.22 0.00 2.24 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.50 15.88 0.68 4.63 0.01 0.08 35.13 0.01 1.48 10.05 0.22 0.39 0.21
sdb 5.98 0.00 446.48 0.00 1.77 0.00 8.11 13.04 29.20 29.20 0.00 2.24 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.97 0.00 4.70 0.00 0.08 36.77 0.00 0.43 0.00 0.43 0.09 0.04
sdb 4.98 0.00 447.57 0.00 1.77 0.00 8.09 13.06 29.16 29.16 0.00 2.23 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.02 0.00 4.58 0.00 0.08 36.97 0.00 0.20 0.00 0.20 0.03 0.01
sdb 5.92 0.00 447.28 0.00 1.77 0.00 8.11 13.03 29.15 29.15 0.00 2.24 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.63 0.00 4.52 0.00 0.08 36.55 0.00 0.21 0.00 0.21 0.03 0.01
sdb 7.13 0.00 452.10 0.00 1.79 0.00 8.13 13.06 28.89 28.89 0.00 2.21 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.85 0.00 4.42 0.00 0.08 37.55 0.00 0.21 0.00 0.21 0.03 0.01
sdb 7.50 0.00 449.48 0.00 1.79 0.00 8.13 13.06 29.05 29.05 0.00 2.22 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.75 0.00 4.48 0.00 0.08 37.06 0.01 2.63 0.00 2.63 0.10 0.05
sdb 6.20 0.00 445.37 0.00 1.76 0.00 8.11 13.06 29.32 29.32 0.00 2.25 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.02 16.02 0.13 4.82 0.02 0.08 43.10 0.00 0.39 8.50 0.17 0.26 0.13
sdb 7.63 0.00 451.18 0.00 1.79 0.00 8.14 13.10 29.06 29.06 0.00 2.22 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.98 0.00 4.63 0.00 0.08 36.52 0.00 0.00 0.00 0.00 0.00 0.00
sdb 7.83 0.00 449.08 0.00 1.78 0.00 8.14 13.08 29.12 29.12 0.00 2.23 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.73 0.00 4.32 0.00 0.08 38.02 0.01 2.10 0.00 2.10 0.06 0.03
sdb 8.17 0.00 449.77 0.00 1.79 0.00 8.15 13.08 29.08 29.08 0.00 2.22 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.90 0.18 4.60 0.02 0.08 44.74 0.00 0.81 8.73 0.49 0.31 0.15
sdb 6.12 0.00 449.95 0.00 1.78 0.00 8.11 13.07 29.06 29.06 0.00 2.22 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.72 0.17 4.60 0.02 0.08 42.57 0.00 0.74 9.20 0.43 0.27 0.13
sdb 7.28 0.00 457.88 0.00 1.82 0.00 8.13 13.14 28.70 28.70 0.00 2.18 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.07 1.13 4.55 0.11 0.08 69.79 0.01 1.50 7.53 0.00 0.90 0.51
sdb 8.93 0.00 456.77 0.00 1.82 0.00 8.16 13.14 28.72 28.72 0.00 2.19 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.90 0.00 4.38 0.00 0.08 38.02 0.01 1.67 0.00 1.67 0.03 0.01
sdb 6.33 0.00 455.58 0.00 1.80 0.00 8.11 13.04 28.67 28.67 0.00 2.19 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.78 0.00 4.68 0.00 0.08 35.76 0.00 0.00 0.00 0.00 0.00 0.00
sdb 4.88 0.00 456.78 0.00 1.80 0.00 8.09 13.05 28.56 28.56 0.00 2.19 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.02 0.03 4.13 0.00 0.08 38.14 0.00 0.11 12.00 0.02 0.06 0.03
sdb 4.38 0.00 453.62 0.00 1.79 0.00 8.08 12.91 28.45 28.45 0.00 2.20 100.00
sdc 0.00 0.00 0.18 0.00 0.11 0.00 1241.45 0.00 15.27 15.27 0.00 7.64 0.14
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 11.55 0.00 4.13 0.00 0.06 31.90 0.00 0.45 0.00 0.45 0.06 0.03
sdb 3.60 0.00 441.13 0.00 1.74 0.00 8.07 11.69 26.53 26.53 0.00 2.27 100.00
sdc 0.22 0.00 48.63 0.00 48.46 0.00 2040.59 1.40 28.84 28.84 0.00 2.26 11.01
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.02 21.08 0.15 6.10 0.02 0.11 43.95 0.00 0.32 6.22 0.17 0.14 0.09
sdb 3.22 0.00 381.57 0.00 1.50 0.00 8.07 7.57 19.86 19.86 0.00 2.62 99.98
sdc 1.22 0.00 253.88 0.00 248.78 0.00 2006.82 8.70 34.26 34.26 0.00 2.32 58.81
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 10.93 0.00 4.27 0.00 0.06 30.56 0.00 0.00 0.00 0.00 0.00 0.00
sdb 3.25 0.00 185.57 0.00 0.74 0.00 8.14 0.89 4.81 4.81 0.00 4.72 87.66
sdc 1.32 0.00 380.37 0.00 370.70 0.00 1995.97 16.96 44.55 44.55 0.00 2.46 93.59
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.25 0.02 5.02 0.00 0.09 35.55 0.01 1.30 8.00 1.28 0.08 0.04
sdb 0.15 0.00 0.53 0.00 0.00 0.00 10.25 0.01 10.25 10.25 0.00 2.12 0.11
sdc 2.33 0.00 389.65 0.00 377.44 0.00 1983.80 21.09 54.15 54.15 0.00 2.54 99.10
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.42 0.73 5.03 0.02 0.09 38.50 0.01 1.82 6.27 1.17 0.58 0.33
sdb 0.17 0.00 0.03 0.00 0.00 0.00 48.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 2.02 0.00 388.88 0.00 377.60 0.00 1988.59 21.07 54.16 54.16 0.00 2.54 98.92
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.45 0.00 5.00 0.00 0.09 36.08 0.00 0.03 0.00 0.03 0.03 0.01
sdb 0.18 0.00 0.12 0.00 0.00 0.00 20.57 0.00 9.71 9.71 0.00 3.43 0.04
sdc 1.55 0.00 398.50 0.00 385.25 0.00 1979.89 21.65 54.35 54.35 0.00 2.48 98.95
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.02 16.25 2.65 4.77 0.18 0.09 72.83 0.02 2.44 6.82 0.00 1.75 1.30
sdb 1.90 0.00 7.32 0.00 0.04 0.00 10.08 0.08 10.55 10.55 0.00 1.59 1.16
sdc 1.00 0.00 263.72 0.00 255.83 0.00 1986.78 14.13 53.63 53.63 0.00 2.53 66.79
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.93 0.00 4.63 0.00 0.08 36.52 0.01 1.42 0.00 1.42 0.07 0.03
sdb 0.18 0.00 0.18 0.00 0.00 0.00 16.00 0.00 1.82 1.82 0.00 1.82 0.03
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.85 0.00 4.27 0.00 0.08 38.72 0.00 0.06 0.00 0.06 0.03 0.01
sdb 0.17 0.00 0.18 0.00 0.00 0.00 15.27 0.00 1.45 1.45 0.00 1.45 0.03
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
--
Sent via pgsql-performance mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
Attachments:
[text/plain] iostat3.txt (15.9K, 3-iostat3.txt)
download | inline:
charles@hpdl380g6:~$ iostat -x -m -d 60
Linux 4.4.0-86-generic (hpdl380g6) 2017-07-18 _x86_64_ (16 CPU)
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 1.45 15.24 3.85 4.91 0.68 0.47 269.64 3.15 351.06 501.22 233.27 4.84 4.24
sdb 1.52 635.12 67.38 10.80 0.27 2.54 73.58 7.66 97.90 28.43 531.24 2.26 17.65
sdc 0.08 0.01 47.90 0.01 15.62 0.00 667.83 0.97 20.30 20.30 6.89 3.05 14.62
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.30 0.03 4.95 0.00 0.09 35.83 0.00 0.56 8.00 0.51 0.13 0.07
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.07 0.23 169.95 0.40 5.66 0.01 68.11 0.85 4.98 5.00 0.00 4.02 68.43
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.42 0.00 5.02 0.00 0.09 35.88 0.00 0.19 0.00 0.19 0.03 0.01
sdb 0.13 0.00 0.07 0.00 0.00 0.00 24.00 0.00 7.00 7.00 0.00 7.00 0.05
sdc 0.12 0.00 231.63 0.00 7.20 0.00 63.68 1.07 4.61 4.61 0.00 3.64 84.21
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.18 0.00 4.63 0.00 0.09 37.73 0.00 0.03 0.00 0.03 0.03 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 206.27 0.00 6.09 0.00 60.43 1.13 5.47 5.47 0.00 3.91 80.63
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.38 0.00 4.73 0.00 0.09 37.44 0.00 0.03 0.00 0.03 0.03 0.01
sdb 0.08 0.00 0.02 0.00 0.00 0.00 48.00 0.00 16.00 16.00 0.00 16.00 0.03
sdc 0.02 0.00 234.78 0.00 3.84 0.00 33.52 0.80 3.41 3.41 0.00 2.84 66.75
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.13 0.00 4.62 0.00 0.09 37.72 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 36.75 0.00 9.57 0.00 533.07 0.45 12.24 12.24 0.00 3.84 14.11
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.15 0.00 4.95 0.00 0.09 35.80 0.00 0.78 0.00 0.78 0.03 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.02 0.00 172.67 0.00 9.22 0.00 109.34 0.96 5.56 5.56 0.00 3.88 67.07
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.48 0.08 5.08 0.00 0.09 36.13 0.00 0.17 5.60 0.08 0.12 0.06
sdb 0.00 0.00 0.02 0.00 0.00 0.00 8.00 0.00 16.00 16.00 0.00 16.00 0.03
sdc 0.00 0.00 149.77 0.00 5.01 0.00 68.48 0.82 5.51 5.51 0.00 3.83 57.33
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.02 0.00 4.70 0.00 0.08 37.02 0.00 0.13 0.00 0.13 0.04 0.02
sdb 0.07 0.00 0.02 0.00 0.00 0.00 40.00 0.00 16.00 16.00 0.00 16.00 0.03
sdc 0.00 0.00 227.00 0.00 6.38 0.00 57.55 2.75 12.13 12.13 0.00 2.62 59.59
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.25 0.00 4.90 0.00 0.09 36.24 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 154.33 0.00 5.60 0.00 74.25 1.39 9.01 9.01 0.00 3.35 51.77
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.92 0.00 4.68 0.00 0.08 36.93 0.01 2.38 0.00 2.38 0.07 0.03
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 182.73 0.00 6.07 0.00 68.04 1.92 10.53 10.53 0.00 2.95 53.95
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.57 0.00 4.98 0.00 0.09 36.28 0.01 2.42 0.00 2.42 0.07 0.03
sdb 0.00 0.00 0.02 0.00 0.00 0.00 8.00 0.00 12.00 12.00 0.00 12.00 0.02
sdc 0.07 0.00 314.02 0.00 5.62 0.00 36.65 2.26 7.18 7.18 0.00 2.52 79.09
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.85 0.03 4.60 0.00 0.08 37.44 0.00 0.04 6.00 0.00 0.04 0.02
sdb 0.05 0.00 0.02 0.00 0.00 0.00 32.00 0.00 12.00 12.00 0.00 12.00 0.02
sdc 0.03 0.00 176.68 0.00 5.48 0.00 63.51 1.04 5.90 5.90 0.00 3.18 56.23
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 119.87 21.48 107.68 7.72 1.48 0.19 29.61 0.38 3.25 3.42 0.90 2.59 29.92
sdb 4.00 0.00 7.55 0.00 0.05 0.00 12.24 0.04 5.70 5.70 0.00 2.89 2.18
sdc 0.00 0.05 264.52 0.15 7.43 0.00 57.51 1.43 5.42 5.42 0.00 2.86 75.73
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.43 0.02 4.50 0.00 0.08 37.17 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.05 0.00 0.02 0.00 0.00 0.00 32.00 0.00 8.00 8.00 0.00 8.00 0.01
sdc 0.00 0.00 247.80 0.00 8.15 0.00 67.36 1.60 6.45 6.45 0.00 2.76 68.44
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.35 0.00 4.82 0.00 0.09 36.87 0.00 0.19 0.00 0.19 0.03 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.08 0.00 278.37 0.00 9.29 0.00 68.31 2.22 7.97 7.97 0.00 2.32 64.48
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 10.97 0.00 4.12 0.00 0.06 31.32 0.00 0.02 0.00 0.02 0.02 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 480.53 0.00 5.97 0.00 25.46 2.20 4.59 4.59 0.00 1.84 88.33
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.30 0.12 4.97 0.00 0.09 36.12 0.00 0.18 8.00 0.00 0.18 0.09
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 486.63 0.00 5.11 0.00 21.51 2.32 4.77 4.77 0.00 1.82 88.69
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.33 0.00 4.88 0.00 0.09 36.40 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 599.40 0.00 5.79 0.00 19.77 1.79 2.99 2.99 0.00 1.42 84.92
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.93 0.00 4.72 0.00 0.08 36.72 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 545.27 0.00 4.89 0.00 18.35 1.37 2.51 2.51 0.00 1.64 89.15
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.20 0.02 4.90 0.00 0.09 35.99 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.07 0.00 63.77 0.00 34.92 0.00 1121.49 1.15 17.96 17.96 0.00 2.08 13.25
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.12 0.00 4.62 0.00 0.08 37.66 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.15 0.00 199.00 0.00 33.88 0.00 348.71 3.12 15.68 15.68 0.00 1.07 21.35
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.45 0.10 4.98 0.00 0.09 35.93 0.00 0.16 8.00 0.00 0.16 0.08
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.10 0.00 401.32 0.00 23.90 0.00 121.97 5.42 13.52 13.52 0.00 1.10 44.27
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 17.05 0.03 5.25 0.00 0.09 35.36 0.00 0.11 6.00 0.08 0.11 0.06
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.02 0.00 242.53 0.00 26.70 0.00 225.44 4.31 17.78 17.78 0.00 1.05 25.40
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.15 0.00 5.00 0.00 0.09 35.41 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.37 0.00 225.52 0.00 173.64 0.00 1576.91 7.19 31.91 31.91 0.00 2.02 45.58
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.20 0.00 4.62 0.00 0.09 37.83 0.00 0.35 0.00 0.35 0.03 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.80 0.00 239.58 0.00 236.94 0.00 2025.38 6.87 28.66 28.66 0.00 2.27 54.33
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.45 0.17 5.12 0.01 0.09 36.57 0.01 1.06 7.20 0.86 0.28 0.15
sdb 0.00 0.00 0.05 0.00 0.00 0.00 8.00 0.00 16.00 16.00 0.00 5.33 0.03
sdc 1.25 0.00 304.82 0.00 300.68 0.00 2020.22 10.87 35.62 35.62 0.00 2.40 73.05
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.42 0.02 4.88 0.00 0.09 37.44 0.00 0.04 8.00 0.01 0.04 0.02
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 1.32 0.00 351.78 0.00 348.00 0.00 2025.96 13.64 38.83 38.83 0.00 2.44 85.77
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.27 0.00 4.97 0.00 0.09 35.87 0.00 0.01 0.00 0.01 0.01 0.01
sdb 0.00 0.00 0.02 0.00 0.00 0.00 8.00 0.00 8.00 8.00 0.00 8.00 0.01
sdc 1.10 0.00 337.22 0.00 333.62 0.00 2026.17 11.89 35.24 35.24 0.00 2.38 80.41
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.43 0.05 5.12 0.00 0.09 35.02 0.00 0.17 5.33 0.12 0.09 0.05
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 1.58 0.00 363.97 0.00 360.07 0.00 2026.08 15.16 41.67 41.67 0.00 2.44 88.81
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.20 0.00 4.75 0.00 0.09 37.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.87 0.00 320.37 0.00 316.34 0.00 2022.27 10.54 32.91 32.91 0.00 2.29 73.52
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.17 0.00 3.93 0.00 0.08 40.03 0.01 1.69 0.00 1.69 0.03 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.78 1.47 4.80 0.02 0.09 33.51 0.01 2.13 9.00 0.03 0.35 0.22
sdb 0.18 0.00 0.07 0.00 0.00 0.00 30.00 0.00 31.00 31.00 0.00 31.00 0.21
sdc 0.00 0.00 0.35 0.00 0.00 0.00 27.43 0.00 6.29 6.29 0.00 6.29 0.22
[text/plain] iostat2.txt (16.3K, 4-iostat2.txt)
download | inline:
charles@hpdl380g6:~$ iostat -x -m -d 60
Linux 4.4.0-86-generic (hpdl380g6) 2017-07-18 _x86_64_ (16 CPU)
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 1.49 15.22 3.98 4.92 0.71 0.48 273.71 3.25 357.20 501.44 240.57 4.92 4.38
sdb 1.57 655.83 69.58 11.15 0.28 2.62 73.58 7.91 97.91 28.43 531.24 2.26 18.23
sdc 0.08 0.01 42.10 0.01 14.16 0.00 688.62 0.90 21.34 21.34 7.59 3.18 13.40
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.82 0.47 5.20 0.00 0.09 33.36 0.01 1.64 13.29 0.59 0.15 0.09
sdb 0.73 0.00 0.25 0.00 0.00 0.00 31.47 0.00 2.40 2.40 0.00 1.60 0.04
sdc 0.00 0.20 196.78 0.37 6.34 0.00 65.87 1.04 5.25 5.26 0.00 4.14 81.68
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.28 0.00 4.77 0.00 0.09 37.01 0.00 0.45 0.00 0.45 0.06 0.03
sdb 0.13 0.00 0.05 0.00 0.00 0.00 29.33 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.13 0.00 229.25 0.00 6.99 0.00 62.45 1.06 4.61 4.61 0.00 3.68 84.37
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.02 0.00 4.82 0.00 0.09 36.43 0.01 2.19 0.00 2.19 0.06 0.03
sdb 0.10 0.00 0.02 0.00 0.00 0.00 56.00 0.00 4.00 4.00 0.00 4.00 0.01
sdc 0.00 0.00 213.52 0.00 4.89 0.00 46.92 0.94 4.38 4.38 0.00 3.81 81.39
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.58 0.13 5.25 0.00 0.09 34.65 0.00 0.62 9.00 0.41 0.25 0.13
sdb 0.20 0.00 0.23 0.00 0.00 0.00 14.86 0.00 2.29 2.29 0.00 2.29 0.05
sdc 0.02 0.00 179.37 0.00 5.10 0.00 58.18 0.62 3.47 3.47 0.00 2.87 51.51
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 10.75 0.00 3.72 0.00 0.06 34.08 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.20 0.00 0.05 0.00 0.00 0.00 40.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 69.87 0.00 8.79 0.00 257.73 0.55 7.87 7.87 0.00 4.12 28.81
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.32 0.00 5.03 0.00 0.09 35.76 0.00 0.21 0.00 0.21 0.03 0.01
sdb 0.60 0.00 0.12 0.00 0.00 0.00 49.14 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.03 0.00 163.32 0.00 8.92 0.00 111.88 0.96 5.85 5.85 0.00 3.85 62.90
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.33 0.00 4.90 0.00 0.09 36.30 0.00 0.63 0.00 0.63 0.08 0.04
sdb 0.22 0.00 0.08 0.00 0.00 0.00 28.80 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 160.37 0.00 5.22 0.00 66.63 1.22 7.58 7.58 0.00 3.61 57.81
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.25 0.00 4.90 0.00 0.09 36.30 0.00 0.35 0.00 0.35 0.05 0.03
sdb 0.13 0.00 0.05 0.00 0.00 0.00 29.33 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 222.75 0.00 6.29 0.00 57.81 2.71 12.15 12.15 0.00 2.65 59.14
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.25 0.00 4.78 0.00 0.09 36.85 0.00 0.17 0.00 0.17 0.03 0.01
sdb 0.17 0.00 0.03 0.00 0.00 0.00 48.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 153.22 0.00 5.67 0.00 75.85 1.34 8.72 8.72 0.00 3.35 51.35
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.20 0.08 4.95 0.00 0.09 35.44 0.00 0.12 7.20 0.00 0.12 0.06
sdb 0.32 0.00 0.10 0.00 0.00 0.00 33.33 0.00 2.67 2.67 0.00 2.67 0.03
sdc 0.00 0.00 194.87 0.00 5.50 0.00 57.84 1.68 8.61 8.61 0.00 3.06 59.62
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.25 0.02 4.80 0.00 0.09 36.82 0.01 1.13 8.00 1.11 0.11 0.05
sdb 0.58 0.00 0.17 0.00 0.00 0.00 36.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.02 0.00 286.10 0.00 5.93 0.00 42.43 2.37 8.27 8.27 0.00 2.60 74.31
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.28 0.02 4.90 0.00 0.09 36.18 0.00 0.28 0.00 0.29 0.04 0.02
sdb 0.12 0.00 0.02 0.00 0.00 0.00 64.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.02 0.00 174.37 0.00 4.51 0.00 52.94 0.90 5.14 5.14 0.00 3.60 62.70
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.33 0.03 4.75 0.01 0.09 41.56 0.00 0.78 12.00 0.70 0.11 0.05
sdb 0.17 0.00 0.03 0.00 0.00 0.00 48.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 248.05 0.00 7.15 0.00 59.02 1.42 5.72 5.72 0.00 3.07 76.12
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.28 0.03 4.77 0.00 0.09 38.31 0.01 1.68 10.00 1.62 0.10 0.05
sdb 0.10 0.00 0.07 0.00 0.00 0.00 20.00 0.00 1.00 1.00 0.00 1.00 0.01
sdc 0.00 0.00 277.58 0.00 7.64 0.00 56.35 1.71 6.15 6.15 0.00 2.77 76.85
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.02 16.22 0.23 4.90 0.02 0.09 43.69 0.00 0.34 7.43 0.00 0.34 0.17
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.12 0.00 238.58 0.00 9.08 0.00 77.96 2.29 9.61 9.61 0.00 2.37 56.60
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.42 0.00 4.95 0.00 0.09 36.23 0.00 0.03 0.00 0.03 0.03 0.01
sdb 0.18 0.00 0.05 0.00 0.00 0.00 37.33 0.00 5.33 5.33 0.00 5.33 0.03
sdc 0.00 0.00 445.87 0.00 6.28 0.00 28.86 1.78 3.99 3.99 0.00 2.04 90.78
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.17 0.00 4.88 0.00 0.09 36.15 0.00 0.27 0.00 0.27 0.05 0.03
sdb 0.02 0.00 0.02 0.00 0.00 0.00 16.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 504.27 0.00 5.31 0.00 21.55 3.37 6.68 6.68 0.00 1.80 90.79
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.87 0.00 4.62 0.00 0.08 37.23 0.00 0.01 0.00 0.01 0.01 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 580.73 0.00 5.57 0.00 19.66 1.59 2.74 2.74 0.00 1.48 85.95
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.12 0.03 4.82 0.00 0.09 38.21 0.00 0.29 42.00 0.00 0.15 0.07
sdb 0.05 0.00 0.03 0.00 0.00 0.00 20.00 0.00 6.00 6.00 0.00 6.00 0.02
sdc 0.00 0.00 590.58 0.00 5.39 0.00 18.70 1.59 2.69 2.69 0.00 1.54 90.67
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 17.03 0.05 5.07 0.00 0.09 36.43 0.00 0.07 6.67 0.00 0.07 0.03
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.05 0.00 162.92 0.00 27.80 0.00 349.48 1.15 7.07 7.07 0.00 1.88 30.67
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.75 0.02 5.05 0.00 0.09 36.32 0.00 0.09 4.00 0.08 0.09 0.05
sdb 0.15 0.00 0.05 0.00 0.00 0.00 32.00 0.00 6.67 6.67 0.00 6.67 0.03
sdc 0.18 0.00 119.80 0.00 33.20 0.00 567.62 1.75 14.64 14.64 0.00 1.65 19.83
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 12.55 0.00 4.13 0.00 0.07 34.16 0.00 0.58 0.00 0.58 0.06 0.03
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.12 0.00 396.90 0.00 21.63 0.00 111.61 3.09 7.79 7.79 0.00 1.37 54.50
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 20.68 0.00 5.90 0.00 0.11 37.47 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.02 0.00 0.02 0.00 0.00 0.00 16.00 0.00 12.00 12.00 0.00 12.00 0.02
sdc 0.08 0.00 469.32 0.00 24.58 0.00 107.25 7.45 15.87 15.87 0.00 0.84 39.57
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 9.73 0.00 3.62 0.00 0.06 31.67 0.00 1.22 0.00 1.22 0.11 0.04
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.18 0.00 307.52 0.00 63.52 0.00 423.05 8.39 27.28 27.28 0.00 0.86 26.41
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.65 0.17 5.17 0.02 0.09 42.23 0.00 0.74 9.20 0.46 0.23 0.12
sdb 0.12 0.00 0.02 0.00 0.00 0.00 64.00 0.00 12.00 12.00 0.00 12.00 0.02
sdc 0.77 0.00 231.62 0.00 228.86 0.00 2023.61 7.72 33.29 33.29 0.00 2.34 54.13
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.72 0.00 4.93 0.00 0.09 36.84 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.07 0.00 0.03 0.00 0.00 0.00 24.00 0.00 8.00 8.00 0.00 8.00 0.03
sdc 0.98 0.00 244.73 0.00 242.63 0.00 2030.40 7.49 30.58 30.58 0.00 2.31 56.65
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.18 0.00 4.88 0.00 0.09 36.26 0.01 1.17 0.00 1.17 0.03 0.01
sdb 0.03 0.00 0.05 0.00 0.00 0.00 13.33 0.00 5.33 5.33 0.00 4.00 0.02
sdc 1.62 0.00 340.48 0.00 335.98 0.00 2020.93 12.83 37.72 37.72 0.00 2.42 82.37
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.77 0.00 4.65 0.00 0.08 36.79 0.00 0.52 0.00 0.52 0.03 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 1.68 0.00 342.80 0.00 339.21 0.00 2026.56 12.70 37.04 37.04 0.00 2.43 83.17
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.65 0.00 5.00 0.00 0.09 36.35 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.02 0.00 0.02 0.00 0.00 0.00 16.00 0.00 12.00 12.00 0.00 12.00 0.02
sdc 1.67 0.00 356.23 0.00 350.65 0.00 2015.93 14.08 39.54 39.54 0.00 2.39 85.03
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.75 0.23 4.75 0.03 0.08 46.07 0.01 1.57 8.86 1.21 0.37 0.19
sdb 0.02 0.00 0.02 0.00 0.00 0.00 16.00 0.00 16.00 16.00 0.00 16.00 0.03
sdc 1.15 0.00 351.48 0.00 347.52 0.00 2024.92 12.89 36.67 36.67 0.00 2.37 83.19
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.53 0.00 4.97 0.00 0.09 36.13 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.75 0.00 0.30 0.00 0.00 0.00 28.00 0.00 2.89 2.89 0.00 2.89 0.09
sdc 0.65 0.00 179.88 0.00 178.26 0.00 2029.51 5.47 30.43 30.43 0.00 2.32 41.75
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.58 0.00 4.25 0.00 0.08 38.09 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.60 0.00 4.40 0.00 0.08 37.21 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
[text/plain] iostat1.txt (38.8K, 5-iostat1.txt)
download | inline:
charles@hpdl380g6:~$ iostat -x -m -d 60
Linux 4.4.0-86-generic (hpdl380g6) 2017-07-18 _x86_64_ (16 CPU)
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 1.19 15.29 3.36 4.96 0.66 0.51 289.56 3.00 353.20 499.26 254.22 4.61 3.83
sdb 1.41 557.27 59.71 9.44 0.24 2.23 73.06 6.71 97.05 27.92 534.28 2.23 15.40
sdc 0.06 0.01 37.03 0.01 13.02 0.00 720.10 0.81 21.77 21.77 9.30 3.17 11.73
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 11.50 3.68 4.25 0.05 0.07 30.86 0.04 5.18 9.25 1.66 1.09 0.87
sdb 9.57 0.00 21.02 0.00 0.12 0.00 11.64 0.03 1.37 1.37 0.00 1.17 2.45
sdc 0.07 1.62 87.48 1.13 2.71 0.02 63.17 0.44 4.93 4.99 0.00 4.23 37.51
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 19.55 0.03 5.47 0.00 0.10 37.89 0.00 0.21 6.00 0.17 0.06 0.03
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.02 0.02 193.13 0.13 6.54 0.00 69.34 0.99 5.11 5.11 0.00 4.08 78.92
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.22 0.07 4.80 0.00 0.09 36.38 0.00 0.34 6.00 0.26 0.12 0.06
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.12 0.00 221.63 0.00 6.61 0.00 61.04 1.03 4.62 4.62 0.00 3.70 81.92
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 11.98 4.18 4.13 0.03 0.07 22.89 0.04 4.64 7.63 1.61 0.62 0.51
sdb 0.03 0.00 0.05 0.00 0.00 0.00 13.33 0.00 16.00 16.00 0.00 8.00 0.04
sdc 0.00 0.00 226.45 0.00 3.80 0.00 34.33 0.87 3.83 3.83 0.00 3.51 79.53
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.10 0.00 4.72 0.00 0.09 37.00 0.00 0.03 0.00 0.03 0.03 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.02 0.00 114.75 0.00 4.22 0.00 75.37 0.41 3.55 3.55 0.00 2.82 32.41
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.20 0.00 4.57 0.00 0.09 38.13 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 5.60 0.00 5.60 0.00 2048.00 0.18 32.05 32.05 0.00 2.51 1.41
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.28 0.00 4.68 0.00 0.09 37.58 0.00 0.01 0.00 0.01 0.01 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 143.55 0.00 6.30 0.00 89.85 0.78 5.41 5.41 0.00 4.01 57.63
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.38 0.05 4.92 0.00 0.09 36.62 0.00 0.07 6.67 0.00 0.07 0.03
sdb 0.95 0.00 0.88 0.00 0.01 0.00 16.60 0.00 4.60 4.60 0.00 4.15 0.37
sdc 0.08 0.00 121.30 0.00 6.96 0.00 117.49 0.75 6.16 6.16 0.00 4.03 48.85
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.18 0.00 4.90 0.00 0.09 36.16 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 129.38 0.00 4.47 0.00 70.71 1.01 7.82 7.82 0.00 3.64 47.09
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.23 0.00 4.67 0.00 0.09 37.54 0.01 1.14 0.00 1.14 0.03 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 197.12 0.00 5.50 0.00 57.12 2.18 11.08 11.08 0.00 2.66 52.47
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.28 0.00 4.75 0.00 0.09 37.16 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 123.97 0.00 4.46 0.00 73.63 1.01 8.17 8.17 0.00 3.60 44.64
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.27 0.00 4.73 0.00 0.09 37.18 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 160.92 0.00 5.58 0.00 71.04 1.99 12.37 12.37 0.00 2.76 44.43
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.58 0.00 5.03 0.00 0.09 36.00 0.00 0.77 0.00 0.77 0.03 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 275.52 0.00 4.90 0.00 36.43 1.32 4.80 4.80 0.00 2.85 78.45
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.12 0.00 4.85 0.00 0.09 36.26 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.03 0.00 154.33 0.00 5.27 0.00 69.89 1.72 11.14 11.14 0.00 2.78 42.91
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.23 0.00 4.80 0.00 0.09 36.72 0.00 0.57 0.00 0.57 0.07 0.03
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 198.68 0.00 5.02 0.00 51.78 1.03 5.21 5.21 0.00 3.47 68.93
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.27 0.00 4.87 0.00 0.09 36.44 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 278.92 0.00 5.47 0.00 40.14 1.60 5.73 5.73 0.00 2.72 75.80
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.20 0.00 4.75 0.00 0.09 36.97 0.00 0.08 0.00 0.08 0.03 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 164.75 0.00 7.00 0.00 87.07 0.95 5.77 5.77 0.00 3.04 50.04
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.22 0.00 4.83 0.00 0.09 36.94 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 212.92 0.00 5.69 0.00 54.73 2.07 9.71 9.71 0.00 2.29 48.76
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.52 0.00 4.48 0.00 0.08 37.53 0.00 0.21 0.00 0.21 0.03 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.02 0.00 409.77 0.00 8.30 0.00 41.47 1.75 4.28 4.28 0.00 2.02 82.83
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.10 0.03 4.73 0.00 0.08 35.05 0.00 0.53 6.00 0.49 0.11 0.05
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 499.55 0.00 5.28 0.00 21.65 3.26 6.52 6.52 0.00 1.78 89.04
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.17 0.00 4.88 0.00 0.08 34.48 0.00 0.25 0.00 0.25 0.03 0.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 539.70 0.00 5.18 0.00 19.66 1.42 2.63 2.63 0.00 1.56 84.17
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 17.63 0.00 5.07 0.00 0.09 37.42 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 604.53 0.00 5.61 0.00 19.00 1.84 3.04 3.04 0.00 1.45 87.87
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.25 0.03 4.80 0.00 0.09 36.55 0.00 0.06 8.00 0.00 0.06 0.03
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 253.27 0.00 12.74 0.00 103.03 0.83 3.29 3.29 0.00 1.85 46.86
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 30.00 9.88 49.55 4.24 3.81 0.06 147.23 30.67 512.55 511.82 521.06 10.90 58.65
sdb 1.31 11581.46 6.95 137.86 0.03 45.68 646.56 69.06 474.75 65.02 495.40 3.98 57.62
sdc 0.53 0.00 27.42 0.00 19.54 0.00 1459.84 3.48 126.91 126.91 0.00 18.20 49.89
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 25.82 1.61 64.91 1.06 8.34 0.01 259.24 56.68 810.89 809.60 889.94 14.01 92.45
sdb 3.39 18155.76 27.35 294.55 0.12 71.70 456.97 158.73 492.62 205.87 519.24 2.99 96.12
sdc 0.88 0.00 28.30 0.00 17.38 0.00 1257.54 3.94 139.16 139.16 0.00 26.14 73.96
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 28.34 7.11 72.85 2.30 9.09 0.04 248.86 65.05 883.90 878.77 1046.59 15.16 113.96
sdb 5.59 20757.70 31.61 337.69 0.15 82.26 456.98 201.34 542.84 233.68 571.78 3.08 113.80
sdc 0.92 0.00 25.53 0.00 17.98 0.00 1442.16 4.30 168.36 168.36 0.00 31.06 79.30
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 18.94 2.17 68.16 1.23 9.78 0.01 289.19 51.19 736.34 735.41 788.11 14.35 99.56
sdb 1.44 17642.69 20.33 285.22 0.09 72.14 484.09 199.75 656.89 479.91 669.51 3.27 99.78
sdc 0.96 0.00 27.05 0.00 17.62 0.00 1333.71 3.84 142.10 142.10 0.00 26.13 70.69
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 34.81 0.53 59.38 0.79 8.98 0.01 305.84 54.65 887.48 873.79 1918.43 15.36 92.42
sdb 3.83 16364.59 32.06 259.40 0.14 65.75 462.97 162.40 557.08 255.41 594.36 3.17 92.40
sdc 1.04 0.00 23.15 0.00 15.56 0.00 1376.79 3.68 158.93 158.93 0.00 27.97 64.75
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 38.04 1.03 138.97 1.00 17.55 0.01 256.92 70.25 487.79 478.46 1783.75 6.87 96.16
sdb 5.88 16247.37 46.64 309.44 0.21 64.82 374.00 160.17 449.56 103.93 501.65 2.69 95.89
sdc 0.92 0.00 21.85 0.00 14.27 0.00 1337.21 3.81 174.32 174.32 0.00 25.65 56.04
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 87.16 6.01 144.64 3.02 12.88 0.04 179.13 44.17 306.55 308.78 199.54 6.41 94.58
sdb 19.42 13537.32 57.67 281.56 0.30 54.66 331.83 109.99 324.35 51.45 380.25 2.58 87.69
sdc 0.81 0.00 116.85 0.00 14.83 0.00 259.96 2.82 24.10 24.10 0.00 6.83 79.81
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 35.47 1.08 79.00 1.21 10.52 0.01 268.79 65.35 722.25 719.59 895.29 12.84 102.98
sdb 4.63 18098.84 45.12 373.82 0.19 72.61 355.91 193.68 462.20 100.15 505.89 2.42 101.49
sdc 1.03 0.00 26.84 0.00 16.93 0.00 1291.56 3.64 135.45 135.45 0.00 23.77 63.82
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 64.93 1.57 83.11 1.81 9.05 0.01 218.46 61.38 722.98 721.80 777.35 12.09 102.69
sdb 6.88 17618.66 41.79 331.23 0.19 69.80 384.29 158.30 422.96 110.46 462.39 2.68 99.96
sdc 0.88 0.00 24.74 0.00 13.35 0.00 1104.93 2.97 119.95 119.95 0.00 23.91 59.16
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 74.98 13.69 139.76 4.65 10.80 0.07 154.27 19.27 144.73 146.79 82.88 3.95 57.06
sdb 29.31 6576.10 110.89 94.84 0.55 27.18 275.97 43.53 214.17 22.40 438.39 2.55 52.56
sdc 0.56 0.00 146.95 0.00 10.70 0.00 149.14 1.64 11.15 11.15 0.00 4.52 66.46
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 4.87 14.15 37.29 5.07 1.12 0.08 58.17 0.24 5.70 6.32 1.12 1.17 4.96
sdb 13.62 0.00 246.84 0.00 1.02 0.00 8.44 6.69 27.19 27.19 0.00 2.30 56.86
sdc 0.02 0.00 133.77 0.00 1.37 0.00 21.03 0.49 3.63 3.63 0.00 3.52 47.06
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 18.25 0.82 5.20 0.01 0.10 37.27 0.01 1.00 7.35 0.00 0.30 0.18
sdb 0.15 0.00 1.00 0.00 0.00 0.00 9.20 0.00 2.00 2.00 0.00 1.87 0.19
sdc 0.00 0.00 191.27 0.00 10.50 0.00 112.44 0.90 4.69 4.69 0.00 2.87 54.93
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.28 0.00 4.88 0.00 0.09 36.53 0.00 0.19 0.00 0.19 0.03 0.01
sdb 1.58 0.00 139.63 0.00 0.55 0.00 8.09 3.74 26.69 26.69 0.00 2.20 30.69
sdc 0.00 0.00 86.55 0.00 8.18 0.00 193.51 0.52 6.03 6.03 0.00 3.32 28.74
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 11.80 0.02 4.20 0.00 0.07 32.28 0.00 0.32 8.00 0.29 0.13 0.05
sdb 3.23 0.00 447.23 0.00 1.76 0.00 8.06 13.00 29.04 29.04 0.00 2.24 100.00
sdc 0.00 0.00 1.32 0.00 0.01 0.00 16.00 0.01 4.46 4.46 0.00 4.46 0.59
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 1.37 16.38 8.88 4.73 0.27 0.09 53.52 0.08 5.89 7.34 3.18 1.41 1.91
sdb 4.05 0.00 448.40 0.00 1.77 0.00 8.07 13.02 29.07 29.07 0.00 2.23 100.00
sdc 0.00 0.00 0.40 0.00 0.00 0.00 16.00 0.01 21.67 21.67 0.00 21.67 0.87
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.15 0.00 5.15 0.00 0.09 34.69 0.00 0.00 0.00 0.00 0.00 0.00
sdb 3.10 0.00 447.02 0.00 1.76 0.00 8.06 12.98 29.03 29.03 0.00 2.24 100.00
sdc 0.00 0.00 1.95 0.00 0.02 0.00 16.00 0.01 4.58 4.58 0.00 4.58 0.89
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.05 0.00 4.77 0.00 0.09 36.56 0.00 0.00 0.00 0.00 0.00 0.00
sdb 1.40 0.00 444.00 0.00 1.74 0.00 8.03 12.99 29.26 29.26 0.00 2.25 100.00
sdc 0.00 0.00 0.50 0.00 0.00 0.00 16.00 0.00 5.33 5.33 0.00 5.33 0.27
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.93 0.00 4.75 0.00 0.08 36.13 0.00 0.20 0.00 0.20 0.03 0.01
sdb 0.68 0.00 444.13 0.00 1.74 0.00 8.01 12.98 29.23 29.23 0.00 2.25 100.00
sdc 0.00 0.00 0.28 0.00 0.00 0.00 16.00 0.00 7.76 7.76 0.00 7.76 0.22
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.90 0.02 4.70 0.00 0.08 37.06 0.00 0.06 16.00 0.00 0.06 0.03
sdb 2.63 0.00 444.48 0.00 1.75 0.00 8.05 13.01 29.27 29.27 0.00 2.25 100.00
sdc 0.00 0.00 0.22 0.00 0.00 0.00 19.69 0.00 5.54 5.54 0.00 5.54 0.12
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.28 0.60 4.85 0.00 0.09 33.17 0.01 1.44 8.44 0.58 0.20 0.11
sdb 3.03 0.00 445.87 0.00 1.75 0.00 8.05 13.01 29.17 29.17 0.00 2.24 100.00
sdc 0.00 0.00 0.47 0.00 0.00 0.00 16.00 0.00 5.57 5.57 0.00 5.57 0.26
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.20 0.02 4.95 0.01 0.09 39.11 0.00 0.05 16.00 0.00 0.05 0.03
sdb 2.55 0.00 444.58 0.00 1.75 0.00 8.05 12.99 29.22 29.22 0.00 2.25 100.00
sdc 0.00 0.00 0.82 0.00 0.01 0.00 16.00 0.00 5.55 5.55 0.00 5.47 0.45
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.22 0.00 4.85 0.00 0.09 36.62 0.01 1.90 0.00 1.90 0.08 0.04
sdb 2.52 0.00 445.68 0.00 1.75 0.00 8.05 12.82 28.77 28.77 0.00 2.24 100.00
sdc 0.00 0.00 27.53 0.00 1.78 0.00 132.67 0.14 5.23 5.23 0.00 3.44 9.47
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.30 0.00 4.98 0.00 0.09 35.93 0.00 0.00 0.00 0.00 0.00 0.00
sdb 3.23 0.00 427.80 0.00 1.68 0.00 8.06 10.93 25.55 25.55 0.00 2.34 100.00
sdc 0.00 0.00 222.67 0.00 2.32 0.00 21.34 0.77 3.45 3.45 0.00 3.33 74.06
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.33 0.00 4.85 0.00 0.09 36.81 0.00 0.00 0.00 0.00 0.00 0.00
sdb 1.02 0.00 123.90 0.00 0.49 0.00 8.07 1.56 12.66 12.66 0.00 3.42 42.38
sdc 0.03 0.00 117.07 0.00 16.07 0.00 281.14 0.82 6.97 6.97 0.00 2.53 29.58
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.32 0.00 4.92 0.00 0.09 36.80 0.00 0.11 0.00 0.11 0.03 0.01
sdb 1.77 0.00 178.93 0.00 0.71 0.00 8.08 4.49 25.02 25.02 0.00 2.42 43.27
sdc 0.00 0.00 96.32 0.00 5.55 0.00 117.91 0.52 5.40 5.40 0.00 3.52 33.93
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.27 0.00 4.87 0.00 0.09 36.68 0.00 0.00 0.00 0.00 0.00 0.00
sdb 1.90 0.00 298.08 0.00 1.17 0.00 8.05 7.83 26.32 26.32 0.00 2.30 68.68
sdc 0.00 0.00 68.77 0.00 4.26 0.00 127.02 0.38 5.46 5.46 0.00 3.53 24.29
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.87 0.02 4.48 0.00 0.08 38.43 0.01 2.00 4.00 1.99 0.04 0.02
sdb 1.77 0.00 154.85 0.00 0.61 0.00 8.09 4.31 27.75 27.75 0.00 2.19 33.95
sdc 0.00 0.00 137.35 0.00 4.49 0.00 66.89 0.74 5.36 5.36 0.00 3.79 52.01
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.78 0.00 4.58 0.00 0.08 36.60 0.00 0.44 0.00 0.44 0.06 0.03
sdb 5.83 0.00 454.80 0.00 1.80 0.00 8.10 13.03 28.64 28.64 0.00 2.20 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.82 0.00 4.43 0.00 0.08 37.50 0.00 0.24 0.00 0.24 0.03 0.01
sdb 5.35 0.00 444.33 0.00 1.76 0.00 8.10 13.02 29.31 29.31 0.00 2.25 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.02 0.00 4.73 0.00 0.08 36.03 0.01 1.48 0.00 1.48 0.04 0.02
sdb 5.58 0.00 448.57 0.00 1.77 0.00 8.10 13.05 29.07 29.07 0.00 2.23 100.00
sdc 0.00 0.00 0.02 0.00 0.00 0.00 8.00 0.00 12.00 12.00 0.00 12.00 0.02
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.00 0.00 4.57 0.00 0.08 35.18 0.00 0.39 0.00 0.39 0.07 0.03
sdb 5.88 0.00 445.73 0.00 1.76 0.00 8.11 13.02 29.20 29.20 0.00 2.24 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 13.88 0.00 4.27 0.00 0.07 34.97 0.00 0.00 0.00 0.00 0.00 0.00
sdb 5.62 0.00 451.23 0.00 1.78 0.00 8.10 13.04 28.92 28.92 0.00 2.22 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 13.48 0.00 4.15 0.00 0.07 35.08 0.00 0.02 0.00 0.02 0.02 0.01
sdb 6.35 0.00 446.07 0.00 1.77 0.00 8.11 13.03 29.22 29.22 0.00 2.24 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.50 15.88 0.68 4.63 0.01 0.08 35.13 0.01 1.48 10.05 0.22 0.39 0.21
sdb 5.98 0.00 446.48 0.00 1.77 0.00 8.11 13.04 29.20 29.20 0.00 2.24 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.97 0.00 4.70 0.00 0.08 36.77 0.00 0.43 0.00 0.43 0.09 0.04
sdb 4.98 0.00 447.57 0.00 1.77 0.00 8.09 13.06 29.16 29.16 0.00 2.23 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.02 0.00 4.58 0.00 0.08 36.97 0.00 0.20 0.00 0.20 0.03 0.01
sdb 5.92 0.00 447.28 0.00 1.77 0.00 8.11 13.03 29.15 29.15 0.00 2.24 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.63 0.00 4.52 0.00 0.08 36.55 0.00 0.21 0.00 0.21 0.03 0.01
sdb 7.13 0.00 452.10 0.00 1.79 0.00 8.13 13.06 28.89 28.89 0.00 2.21 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.85 0.00 4.42 0.00 0.08 37.55 0.00 0.21 0.00 0.21 0.03 0.01
sdb 7.50 0.00 449.48 0.00 1.79 0.00 8.13 13.06 29.05 29.05 0.00 2.22 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.75 0.00 4.48 0.00 0.08 37.06 0.01 2.63 0.00 2.63 0.10 0.05
sdb 6.20 0.00 445.37 0.00 1.76 0.00 8.11 13.06 29.32 29.32 0.00 2.25 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.02 16.02 0.13 4.82 0.02 0.08 43.10 0.00 0.39 8.50 0.17 0.26 0.13
sdb 7.63 0.00 451.18 0.00 1.79 0.00 8.14 13.10 29.06 29.06 0.00 2.22 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.98 0.00 4.63 0.00 0.08 36.52 0.00 0.00 0.00 0.00 0.00 0.00
sdb 7.83 0.00 449.08 0.00 1.78 0.00 8.14 13.08 29.12 29.12 0.00 2.23 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.73 0.00 4.32 0.00 0.08 38.02 0.01 2.10 0.00 2.10 0.06 0.03
sdb 8.17 0.00 449.77 0.00 1.79 0.00 8.15 13.08 29.08 29.08 0.00 2.22 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.90 0.18 4.60 0.02 0.08 44.74 0.00 0.81 8.73 0.49 0.31 0.15
sdb 6.12 0.00 449.95 0.00 1.78 0.00 8.11 13.07 29.06 29.06 0.00 2.22 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.72 0.17 4.60 0.02 0.08 42.57 0.00 0.74 9.20 0.43 0.27 0.13
sdb 7.28 0.00 457.88 0.00 1.82 0.00 8.13 13.14 28.70 28.70 0.00 2.18 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.07 1.13 4.55 0.11 0.08 69.79 0.01 1.50 7.53 0.00 0.90 0.51
sdb 8.93 0.00 456.77 0.00 1.82 0.00 8.16 13.14 28.72 28.72 0.00 2.19 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.90 0.00 4.38 0.00 0.08 38.02 0.01 1.67 0.00 1.67 0.03 0.01
sdb 6.33 0.00 455.58 0.00 1.80 0.00 8.11 13.04 28.67 28.67 0.00 2.19 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.78 0.00 4.68 0.00 0.08 35.76 0.00 0.00 0.00 0.00 0.00 0.00
sdb 4.88 0.00 456.78 0.00 1.80 0.00 8.09 13.05 28.56 28.56 0.00 2.19 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.02 0.03 4.13 0.00 0.08 38.14 0.00 0.11 12.00 0.02 0.06 0.03
sdb 4.38 0.00 453.62 0.00 1.79 0.00 8.08 12.91 28.45 28.45 0.00 2.20 100.00
sdc 0.00 0.00 0.18 0.00 0.11 0.00 1241.45 0.00 15.27 15.27 0.00 7.64 0.14
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 11.55 0.00 4.13 0.00 0.06 31.90 0.00 0.45 0.00 0.45 0.06 0.03
sdb 3.60 0.00 441.13 0.00 1.74 0.00 8.07 11.69 26.53 26.53 0.00 2.27 100.00
sdc 0.22 0.00 48.63 0.00 48.46 0.00 2040.59 1.40 28.84 28.84 0.00 2.26 11.01
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.02 21.08 0.15 6.10 0.02 0.11 43.95 0.00 0.32 6.22 0.17 0.14 0.09
sdb 3.22 0.00 381.57 0.00 1.50 0.00 8.07 7.57 19.86 19.86 0.00 2.62 99.98
sdc 1.22 0.00 253.88 0.00 248.78 0.00 2006.82 8.70 34.26 34.26 0.00 2.32 58.81
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 10.93 0.00 4.27 0.00 0.06 30.56 0.00 0.00 0.00 0.00 0.00 0.00
sdb 3.25 0.00 185.57 0.00 0.74 0.00 8.14 0.89 4.81 4.81 0.00 4.72 87.66
sdc 1.32 0.00 380.37 0.00 370.70 0.00 1995.97 16.96 44.55 44.55 0.00 2.46 93.59
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.25 0.02 5.02 0.00 0.09 35.55 0.01 1.30 8.00 1.28 0.08 0.04
sdb 0.15 0.00 0.53 0.00 0.00 0.00 10.25 0.01 10.25 10.25 0.00 2.12 0.11
sdc 2.33 0.00 389.65 0.00 377.44 0.00 1983.80 21.09 54.15 54.15 0.00 2.54 99.10
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.42 0.73 5.03 0.02 0.09 38.50 0.01 1.82 6.27 1.17 0.58 0.33
sdb 0.17 0.00 0.03 0.00 0.00 0.00 48.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 2.02 0.00 388.88 0.00 377.60 0.00 1988.59 21.07 54.16 54.16 0.00 2.54 98.92
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 16.45 0.00 5.00 0.00 0.09 36.08 0.00 0.03 0.00 0.03 0.03 0.01
sdb 0.18 0.00 0.12 0.00 0.00 0.00 20.57 0.00 9.71 9.71 0.00 3.43 0.04
sdc 1.55 0.00 398.50 0.00 385.25 0.00 1979.89 21.65 54.35 54.35 0.00 2.48 98.95
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.02 16.25 2.65 4.77 0.18 0.09 72.83 0.02 2.44 6.82 0.00 1.75 1.30
sdb 1.90 0.00 7.32 0.00 0.04 0.00 10.08 0.08 10.55 10.55 0.00 1.59 1.16
sdc 1.00 0.00 263.72 0.00 255.83 0.00 1986.78 14.13 53.63 53.63 0.00 2.53 66.79
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.93 0.00 4.63 0.00 0.08 36.52 0.01 1.42 0.00 1.42 0.07 0.03
sdb 0.18 0.00 0.18 0.00 0.00 0.00 16.00 0.00 1.82 1.82 0.00 1.82 0.03
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.85 0.00 4.27 0.00 0.08 38.72 0.00 0.06 0.00 0.06 0.03 0.01
sdb 0.17 0.00 0.18 0.00 0.00 0.00 15.27 0.00 1.45 1.45 0.00 1.45 0.03
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 19:24 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-11 11:02 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 00:39 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-12 10:04 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 14:31 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-12 16:39 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-14 15:34 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-17 20:56 ` Re: Very poor read performance, query independent Claudio Freire <[email protected]>
2017-07-18 09:20 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
@ 2017-07-18 16:01 ` Claudio Freire <[email protected]>
2017-07-18 17:13 ` Re: Very poor read performance, query independent Claudio Freire <[email protected]>
1 sibling, 1 reply; 52+ messages in thread
From: Claudio Freire @ 2017-07-18 16:01 UTC (permalink / raw)
To: Charles Nadeau <[email protected]>; +Cc: Igor Neyman <[email protected]>; Jeff Janes <[email protected]>; pgsql-performance
On Tue, Jul 18, 2017 at 6:20 AM, Charles Nadeau
<[email protected]> wrote:
> Claudio,
>
> At one moment
> during the query, there is a write storm to the swap drive (a bit like this
> case:
> https://www.postgresql.org/message-id/AANLkTi%3Diw4fC2RgTxhw0aGpyXANhOT%3DXBnjLU1_v6PdA%40mail.gmail...).
> I can hardly explain it as there is plenty of memory on this server.
That sounds a lot like NUMA zone_reclaim issues:
https://www.postgresql.org/message-id/[email protected]
--
Sent via pgsql-performance mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 19:24 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-11 11:02 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 00:39 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-12 10:04 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 14:31 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-12 16:39 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-14 15:34 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-17 20:56 ` Re: Very poor read performance, query independent Claudio Freire <[email protected]>
2017-07-18 09:20 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-18 16:01 ` Re: Very poor read performance, query independent Claudio Freire <[email protected]>
@ 2017-07-18 17:13 ` Claudio Freire <[email protected]>
2017-07-18 18:01 ` Re: Very poor read performance, query independent Justin Pryzby <[email protected]>
0 siblings, 1 reply; 52+ messages in thread
From: Claudio Freire @ 2017-07-18 17:13 UTC (permalink / raw)
To: Charles Nadeau <[email protected]>; +Cc: Igor Neyman <[email protected]>; Jeff Janes <[email protected]>; pgsql-performance
On Tue, Jul 18, 2017 at 1:01 PM, Claudio Freire <[email protected]> wrote:
> On Tue, Jul 18, 2017 at 6:20 AM, Charles Nadeau
> <[email protected]> wrote:
>> Claudio,
>>
>> At one moment
>> during the query, there is a write storm to the swap drive (a bit like this
>> case:
>> https://www.postgresql.org/message-id/AANLkTi%3Diw4fC2RgTxhw0aGpyXANhOT%3DXBnjLU1_v6PdA%40mail.gmail...).
>> I can hardly explain it as there is plenty of memory on this server.
>
> That sounds a lot like NUMA zone_reclaim issues:
>
> https://www.postgresql.org/message-id/[email protected]
I realize you have zone_reclaim_mode set to 0. Still, the symptoms are
eerily similar.
--
Sent via pgsql-performance mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 19:24 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-11 11:02 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 00:39 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-12 10:04 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 14:31 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-12 16:39 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-14 15:34 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-17 20:56 ` Re: Very poor read performance, query independent Claudio Freire <[email protected]>
2017-07-18 09:20 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-18 16:01 ` Re: Very poor read performance, query independent Claudio Freire <[email protected]>
2017-07-18 17:13 ` Re: Very poor read performance, query independent Claudio Freire <[email protected]>
@ 2017-07-18 18:01 ` Justin Pryzby <[email protected]>
2017-07-19 11:48 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
0 siblings, 1 reply; 52+ messages in thread
From: Justin Pryzby @ 2017-07-18 18:01 UTC (permalink / raw)
To: Claudio Freire <[email protected]>; +Cc: Charles Nadeau <[email protected]>; Igor Neyman <[email protected]>; Jeff Janes <[email protected]>; pgsql-performance
On Tue, Jul 18, 2017 at 02:13:58PM -0300, Claudio Freire wrote:
> On Tue, Jul 18, 2017 at 1:01 PM, Claudio Freire <[email protected]> wrote:
> > On Tue, Jul 18, 2017 at 6:20 AM, Charles Nadeau
> > <[email protected]> wrote:
> >> Claudio,
> >>
> >> At one moment
> >> during the query, there is a write storm to the swap drive (a bit like this
> >> case:
> >> https://www.postgresql.org/message-id/AANLkTi%3Diw4fC2RgTxhw0aGpyXANhOT%3DXBnjLU1_v6PdA%40mail.gmail...).
> >> I can hardly explain it as there is plenty of memory on this server.
> >
> > That sounds a lot like NUMA zone_reclaim issues:
> >
> > https://www.postgresql.org/message-id/[email protected]
>
> I realize you have zone_reclaim_mode set to 0. Still, the symptoms are
> eerily similar.
Did you look at disabling KSM and/or THP ?
sudo sh -c 'echo 2 >/sys/kernel/mm/ksm/run'
https://www.postgresql.org/message-id/20170524155855.GH31097%40telsasoft.com
https://www.postgresql.org/message-id/[email protected]...
https://www.postgresql.org/message-id/CAHyXU0y9hviyKWvQZxX5UWfH9M2LYvwvAOPQ_DUPva2b71t12g%40mail.gma...
https://www.postgresql.org/message-id/[email protected]
https://www.postgresql.org/message-id/CAE_gQfW3dBiELcOppYN6v%3D8%2B%2BpEeywD7iXGw-OT3doB8SXO4_A%40ma...
https://www.postgresql.org/message-id/flat/1436268563235-5856914.post%40n5.nabble.com#1436268563235-...
https://www.postgresql.org/message-id/CAL_0b1tJOZCx3Lo3Eve1RqGaT%[email protected]...
https://www.postgresql.org/message-id/[email protected]
https://www.postgresql.org/message-id/1415981309.90631.YahooMailNeo%40web133205.mail.ir2.yahoo.com
https://www.postgresql.org/message-id/CAHyXU0yXYpCXN4%3D81ZDRQu-oGzrcq2qNAXDpyz4oiQPPAGk4ew%40mail.g...
https://www.pythian.com/blog/performance-tuning-hugepages-in-linux/
http://structureddata.org/2012/06/18/linux-6-transparent-huge-pages-and-hadoop-workloads/
Justin
--
Sent via pgsql-performance mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 19:24 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-11 11:02 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 00:39 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-12 10:04 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 14:31 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-12 16:39 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-14 15:34 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-17 20:56 ` Re: Very poor read performance, query independent Claudio Freire <[email protected]>
2017-07-18 09:20 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-18 16:01 ` Re: Very poor read performance, query independent Claudio Freire <[email protected]>
2017-07-18 17:13 ` Re: Very poor read performance, query independent Claudio Freire <[email protected]>
2017-07-18 18:01 ` Re: Very poor read performance, query independent Justin Pryzby <[email protected]>
@ 2017-07-19 11:48 ` Charles Nadeau <[email protected]>
0 siblings, 0 replies; 52+ messages in thread
From: Charles Nadeau @ 2017-07-19 11:48 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Claudio Freire <[email protected]>; Igor Neyman <[email protected]>; Jeff Janes <[email protected]>; pgsql-performance
Justin,
Thanks for the extensive reading list, very educative.
After reading
https://blog.jcole.us/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/
I was thinking that it could be a NUMA/THP-related problem.
Turning off THP solved the "swap storm" problem. Some queries are even 40%
faster with THP off.
Then also turning off KSM improved performance by another 5%
I was seriously worried about this issue as we received today another
server with 144GB of RAM.
I will try to post a little summary of all the suggestion I received via
this thread later this week/early next week.
Thanks!
Charles
On Tue, Jul 18, 2017 at 8:01 PM, Justin Pryzby <[email protected]> wrote:
> On Tue, Jul 18, 2017 at 02:13:58PM -0300, Claudio Freire wrote:
> > On Tue, Jul 18, 2017 at 1:01 PM, Claudio Freire <[email protected]>
> wrote:
> > > On Tue, Jul 18, 2017 at 6:20 AM, Charles Nadeau
> > > <[email protected]> wrote:
> > >> Claudio,
> > >>
> > >> At one moment
> > >> during the query, there is a write storm to the swap drive (a bit
> like this
> > >> case:
> > >> https://www.postgresql.org/message-id/AANLkTi%
> 3Diw4fC2RgTxhw0aGpyXANhOT%3DXBnjLU1_v6PdA%40mail.gmail.com).
> > >> I can hardly explain it as there is plenty of memory on this server.
> > >
> > > That sounds a lot like NUMA zone_reclaim issues:
> > >
> > > https://www.postgresql.org/message-id/[email protected]
> >
> > I realize you have zone_reclaim_mode set to 0. Still, the symptoms are
> > eerily similar.
>
> Did you look at disabling KSM and/or THP ?
>
> sudo sh -c 'echo 2 >/sys/kernel/mm/ksm/run'
>
> https://www.postgresql.org/message-id/20170524155855.
> GH31097%40telsasoft.com
> https://www.postgresql.org/message-id/CANQNgOrD02f8mR3Y8Pi=
> [email protected]
> https://www.postgresql.org/message-id/CAHyXU0y9hviyKWvQZxX5UWfH9M2LY
> vwvAOPQ_DUPva2b71t12g%40mail.gmail.com
> https://www.postgresql.org/message-id/20130716195834.
> [email protected]
> https://www.postgresql.org/message-id/CAE_gQfW3dBiELcOppYN6v%3D8%2B%
> 2BpEeywD7iXGw-OT3doB8SXO4_A%40mail.gmail.com
> https://www.postgresql.org/message-id/flat/1436268563235-
> 5856914.post%40n5.nabble.com#[email protected]
> https://www.postgresql.org/message-id/CAL_0b1tJOZCx3Lo3Eve1RqGaT%2BJJ_
> [email protected]
> https://www.postgresql.org/message-id/[email protected]
> https://www.postgresql.org/message-id/1415981309.90631.
> YahooMailNeo%40web133205.mail.ir2.yahoo.com
> https://www.postgresql.org/message-id/CAHyXU0yXYpCXN4%3D81ZDRQu-
> oGzrcq2qNAXDpyz4oiQPPAGk4ew%40mail.gmail.com
> https://www.pythian.com/blog/performance-tuning-hugepages-in-linux/
> http://structureddata.org/2012/06/18/linux-6-transparent-huge-pages-and-
> hadoop-workloads/
>
> Justin
>
--
Charles Nadeau Ph.D.
http://charlesnadeau.blogspot.com/
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 19:24 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-11 11:02 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 00:39 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-12 10:04 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 14:31 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-12 16:39 ` Re: Very poor read performance, query independent Igor Neyman <[email protected]>
2017-07-14 15:34 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-17 20:56 ` Re: Very poor read performance, query independent Claudio Freire <[email protected]>
2017-07-18 09:20 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
@ 2017-07-19 01:08 ` Scott Marlowe <[email protected]>
1 sibling, 0 replies; 52+ messages in thread
From: Scott Marlowe @ 2017-07-19 01:08 UTC (permalink / raw)
To: Charles Nadeau <[email protected]>; +Cc: Claudio Freire <[email protected]>; Igor Neyman <[email protected]>; Jeff Janes <[email protected]>; pgsql-performance
On Tue, Jul 18, 2017 at 3:20 AM, Charles Nadeau
<[email protected]> wrote:
> Claudio,
>
> Find attached the iostat measured while redoing the query above
> (iostat1.txt). sda holds my temp directory (noop i/o scheduler), sdb the
> swap partition (cfq i/o scheduler) only and sdc (5 disks RAID0, noop i/o
> scheduler) holds the data. I didn't pay attention to the load caused by 12
> parallel scans as I thought the RAID card would be smart enough to
> re-arrange the read requests optimally regardless of the load. At one moment
> during the query, there is a write storm to the swap drive (a bit like this
> case:
> https://www.postgresql.org/message-id/AANLkTi%3Diw4fC2RgTxhw0aGpyXANhOT%3DXBnjLU1_v6PdA%40mail.gmail...).
My experience from that case (and few more) has led me to believe
that Linux database servers with plenty of memory should have their
swaps turned off. The Linux kernel works hard to swap out little used
memory to make more space for caching active data.
Problem is that whatever decides to swap stuff out gets stupid when
presented with 512GB RAM and starts swapping out things like sys v
shared_buffers etc.
Here's the thing, either your memory is big enough to buffer your
whole data set, so nothing should get swapped out to make room for
caching.
OR your dataset is much bigger than memory. In which case, making more
room gets very little if it comes at the cost of waiting for stuff you
need to get read back in.
Linux servers should also have zone reclaim turned off, and THP disabled.
Try running "sudo swapoff -a" and see if it gets rid of your swap storms.
--
Sent via pgsql-performance mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 19:24 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-11 11:02 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-12 00:39 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-12 10:04 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
@ 2017-07-12 22:27 ` Jeff Janes <[email protected]>
1 sibling, 0 replies; 52+ messages in thread
From: Jeff Janes @ 2017-07-12 22:27 UTC (permalink / raw)
To: Charles Nadeau <[email protected]>; +Cc: pgsql-performance
On Wed, Jul 12, 2017 at 3:04 AM, Charles Nadeau <[email protected]>
wrote:
> Jeff,
>
> Here are the 2 EXPLAINs for one of my simplest query:
>
It looks like dstexterne and flowcompact are both views over flow. Can you
share the definition of those views?
I think the iowait > 12.5% is due to the parallel query execution. But
then the question is, why is it only 25% when you have 10 fold parallelism?
It certainly looks like you are doing more than 4MB/s there, so maybe
something is wrong with the instrumentation, or how you are interpreting
it.
Although it is still less than perhaps it could do. To put a baseline on
what you can expect out of parallel seq scans, can you do something like:
explain (analyze, buffers) select avg(doctets) from flow;
Cheers,
Jeff
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
@ 2017-07-11 23:15 ` Merlin Moncure <[email protected]>
2017-07-11 23:42 ` Re: Very poor read performance, query independent Joshua D. Drake <[email protected]>
4 siblings, 1 reply; 52+ messages in thread
From: Merlin Moncure @ 2017-07-11 23:15 UTC (permalink / raw)
To: Charles Nadeau <[email protected]>; +Cc: pgsql-performance
On Mon, Jul 10, 2017 at 9:03 AM, Charles Nadeau
<[email protected]> wrote:
> I’m running PostgreSQL 9.6.3 on Ubuntu 16.10 (kernel 4.4.0-85-generic).
> Hardware is:
>
> *2x Intel Xeon E5550
>
> *72GB RAM
>
> *Hardware RAID10 (4 x 146GB SAS 10k) P410i controller with 1GB FBWC (80%
> read/20% write) for Postgresql data only:
>
> The problem I have is very poor read. When I benchmark my array with fio I
> get random reads of about 200MB/s and 1100IOPS and sequential reads of about
> 286MB/s and 21000IPS. But when I watch my queries using pg_activity, I get
> at best 4MB/s. Also using dstat I can see that iowait time is at about 25%.
> This problem is not query-dependent.
Stop right there. 1100 iops * 8kb = ~8mb/sec raw which might
reasonably translate to 4mb/sec to the client. 200mb/sec random
read/sec on spinning media is simply not plausible; your benchmark is
lying to you. Random reads on spinning media are absolutely going to
be storage bound.
merlin
--
Sent via pgsql-performance mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-11 23:15 ` Re: Very poor read performance, query independent Merlin Moncure <[email protected]>
@ 2017-07-11 23:42 ` Joshua D. Drake <[email protected]>
2017-07-12 01:03 ` Re: Very poor read performance, query independent Jeff Janes <[email protected]>
2017-07-12 07:30 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
0 siblings, 2 replies; 52+ messages in thread
From: Joshua D. Drake @ 2017-07-11 23:42 UTC (permalink / raw)
To: Merlin Moncure <[email protected]>; Charles Nadeau <[email protected]>; +Cc: pgsql-performance
On 07/11/2017 04:15 PM, Merlin Moncure wrote:
> On Mon, Jul 10, 2017 at 9:03 AM, Charles Nadeau
> <[email protected]> wrote:
>> I’m running PostgreSQL 9.6.3 on Ubuntu 16.10 (kernel 4.4.0-85-generic).
>> Hardware is:
>>
>> *2x Intel Xeon E5550
>>
>> *72GB RAM
>>
>> *Hardware RAID10 (4 x 146GB SAS 10k) P410i controller with 1GB FBWC (80%
>> read/20% write) for Postgresql data only:
>>
>> The problem I have is very poor read. When I benchmark my array with fio I
>> get random reads of about 200MB/s and 1100IOPS and sequential reads of about
>> 286MB/s and 21000IPS. But when I watch my queries using pg_activity, I get
>> at best 4MB/s. Also using dstat I can see that iowait time is at about 25%.
>> This problem is not query-dependent.
>
> Stop right there. 1100 iops * 8kb = ~8mb/sec raw which might
> reasonably translate to 4mb/sec to the client. 200mb/sec random
> read/sec on spinning media is simply not plausible;
Sure it is, if he had more than 4 disks ;) but he also isn't going to
get 1100 IOPS from 4 10k disks. The average 10k disk is going to get
around 130 IOPS . If he only has 4 then there is no way he is getting
1100 IOPS.
Using the above specs (4x146GB) the best he can reasonably hope for from
the drives themselves is about 50MB/s add in the 1GB FWBC and that is
how he is getting those high numbers for IOPS but that is because of
caching.
He may need to adjust his readahead as well as his kernel scheduler. At
a minimum he should be able to saturate the drives without issue.
JD
--
Command Prompt, Inc. || http://the.postgres.company/ || @cmdpromptinc
PostgreSQL Centered full stack support, consulting and development.
Advocate: @amplifypostgres || Learn: https://pgconf.us
***** Unless otherwise stated, opinions are my own. *****
--
Sent via pgsql-performance mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-11 23:15 ` Re: Very poor read performance, query independent Merlin Moncure <[email protected]>
2017-07-11 23:42 ` Re: Very poor read performance, query independent Joshua D. Drake <[email protected]>
@ 2017-07-12 01:03 ` Jeff Janes <[email protected]>
1 sibling, 0 replies; 52+ messages in thread
From: Jeff Janes @ 2017-07-12 01:03 UTC (permalink / raw)
To: Joshua D. Drake <[email protected]>; +Cc: Merlin Moncure <[email protected]>; Charles Nadeau <[email protected]>; pgsql-performance
On Tue, Jul 11, 2017 at 4:42 PM, Joshua D. Drake <[email protected]>
wrote:
> On 07/11/2017 04:15 PM, Merlin Moncure wrote:
>
>> On Mon, Jul 10, 2017 at 9:03 AM, Charles Nadeau
>> <[email protected]> wrote:
>>
>>> I’m running PostgreSQL 9.6.3 on Ubuntu 16.10 (kernel 4.4.0-85-generic).
>>> Hardware is:
>>>
>>> *2x Intel Xeon E5550
>>>
>>> *72GB RAM
>>>
>>> *Hardware RAID10 (4 x 146GB SAS 10k) P410i controller with 1GB FBWC (80%
>>> read/20% write) for Postgresql data only:
>>>
>>> The problem I have is very poor read. When I benchmark my array with fio
>>> I
>>> get random reads of about 200MB/s and 1100IOPS and sequential reads of
>>> about
>>> 286MB/s and 21000IPS. But when I watch my queries using pg_activity, I
>>> get
>>> at best 4MB/s. Also using dstat I can see that iowait time is at about
>>> 25%.
>>> This problem is not query-dependent.
>>>
>>
>> Stop right there. 1100 iops * 8kb = ~8mb/sec raw which might
>> reasonably translate to 4mb/sec to the client. 200mb/sec random
>> read/sec on spinning media is simply not plausible;
>>
>
> Sure it is, if he had more than 4 disks ;)
Or more to the point here, if each random read is 4MB long. Which makes it
more like sequential reads, randomly-piecewise, rather than random reads.
> but he also isn't going to get 1100 IOPS from 4 10k disks. The average 10k
> disk is going to get around 130 IOPS . If he only has 4 then there is no
> way he is getting 1100 IOPS.
>
I wouldn't be sure. He is using an iodepth of 256 in his benchmark. It
wouldn't be all that outrageous for a disk to be able to find 3 or 4
sectors per revolution it can read, when it has that many to choose from.
Cheers,
Jeff
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-11 23:15 ` Re: Very poor read performance, query independent Merlin Moncure <[email protected]>
2017-07-11 23:42 ` Re: Very poor read performance, query independent Joshua D. Drake <[email protected]>
@ 2017-07-12 07:30 ` Charles Nadeau <[email protected]>
2017-07-12 14:11 ` Re: Very poor read performance, query independent bricklen <[email protected]>
1 sibling, 1 reply; 52+ messages in thread
From: Charles Nadeau @ 2017-07-12 07:30 UTC (permalink / raw)
To: Joshua D. Drake <[email protected]>; +Cc: Merlin Moncure <[email protected]>; pgsql-performance
Joshua,
I use noop as the scheduler because it is better to let the RAID controller
re-arrange the IO operation before they reach the disk. Read ahead is set
to 128:
charles@hpdl380g6:~$ cat /sys/block/sdc/queue/read_ahead_kb
128
charles@hpdl380g6:~$ cat /sys/block/sdc/queue/scheduler
[noop] deadline cfq
Thanks!
Charles
On Wed, Jul 12, 2017 at 1:42 AM, Joshua D. Drake <[email protected]>
wrote:
> On 07/11/2017 04:15 PM, Merlin Moncure wrote:
>
>> On Mon, Jul 10, 2017 at 9:03 AM, Charles Nadeau
>> <[email protected]> wrote:
>>
>>> I’m running PostgreSQL 9.6.3 on Ubuntu 16.10 (kernel 4.4.0-85-generic).
>>> Hardware is:
>>>
>>> *2x Intel Xeon E5550
>>>
>>> *72GB RAM
>>>
>>> *Hardware RAID10 (4 x 146GB SAS 10k) P410i controller with 1GB FBWC (80%
>>> read/20% write) for Postgresql data only:
>>>
>>> The problem I have is very poor read. When I benchmark my array with fio
>>> I
>>> get random reads of about 200MB/s and 1100IOPS and sequential reads of
>>> about
>>> 286MB/s and 21000IPS. But when I watch my queries using pg_activity, I
>>> get
>>> at best 4MB/s. Also using dstat I can see that iowait time is at about
>>> 25%.
>>> This problem is not query-dependent.
>>>
>>
>> Stop right there. 1100 iops * 8kb = ~8mb/sec raw which might
>> reasonably translate to 4mb/sec to the client. 200mb/sec random
>> read/sec on spinning media is simply not plausible;
>>
>
> Sure it is, if he had more than 4 disks ;) but he also isn't going to get
> 1100 IOPS from 4 10k disks. The average 10k disk is going to get around 130
> IOPS . If he only has 4 then there is no way he is getting 1100 IOPS.
>
> Using the above specs (4x146GB) the best he can reasonably hope for from
> the drives themselves is about 50MB/s add in the 1GB FWBC and that is how
> he is getting those high numbers for IOPS but that is because of caching.
>
> He may need to adjust his readahead as well as his kernel scheduler. At a
> minimum he should be able to saturate the drives without issue.
>
> JD
>
>
>
> --
> Command Prompt, Inc. || http://the.postgres.company/ || @cmdpromptinc
>
> PostgreSQL Centered full stack support, consulting and development.
> Advocate: @amplifypostgres || Learn: https://pgconf.us
> ***** Unless otherwise stated, opinions are my own. *****
>
--
Charles Nadeau Ph.D.
http://charlesnadeau.blogspot.com/
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-11 23:15 ` Re: Very poor read performance, query independent Merlin Moncure <[email protected]>
2017-07-11 23:42 ` Re: Very poor read performance, query independent Joshua D. Drake <[email protected]>
2017-07-12 07:30 ` Re: Very poor read performance, query independent Charles Nadeau <[email protected]>
@ 2017-07-12 14:11 ` bricklen <[email protected]>
0 siblings, 0 replies; 52+ messages in thread
From: bricklen @ 2017-07-12 14:11 UTC (permalink / raw)
To: Charles Nadeau <[email protected]>; +Cc: pgsql-performance
On Wed, Jul 12, 2017 at 12:30 AM, Charles Nadeau <[email protected]>
wrote:
>
> I use noop as the scheduler because it is better to let the RAID
> controller re-arrange the IO operation before they reach the disk. Read
> ahead is set to 128:
>
> charles@hpdl380g6:~$ cat /sys/block/sdc/queue/read_ahead_kb
> 128
> charles@hpdl380g6:~$ cat /sys/block/sdc/queue/scheduler
> [noop] deadline cfq
>
>
>
Perhaps pg_test_fsync (
https://www.postgresql.org/docs/9.6/static/pgtestfsync.html) and
pg_test_timing will help shed some light here, or at the very least give
some numbers to compare against.
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: Very poor read performance, query independent
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
@ 2017-07-25 09:36 ` Charles Nadeau <[email protected]>
4 siblings, 0 replies; 52+ messages in thread
From: Charles Nadeau @ 2017-07-25 09:36 UTC (permalink / raw)
To: pgsql-performance
All,
Here is a list of what I did based of the suggestions made after my initial
post:
*Reduce max_parallel_workers to 4: Values higher makes the workers wait for
data as the RAID0 array can't deliver high enough IOPS.
*Reduce random_page_cost to 1: Forcing the use of index makes queries
faster despite low random throughput.
*Increase shared_buffer to 66GB and effective_cache_size to 53GB: With the
new server having 144GB of RAM, increasing shared_buffer allows Postgresql
to keep a lot of data in memory reducing the need to go to disk.
*Reduce min_parallel_relation_size to 512kB to have more workers when doing
sequential parallel scan
*Increased the /sys/block/sd[ac]/queue/read_ahead_kb to 16384 for my arrays
using HDD
*Reused old SSDs (that are compatible with my RAID controller, to my
surprise) to put my most used index and tables.
Thanks to everybody who made suggestions. I now know more about Postgresql
tuning.
Charles
On Mon, Jul 10, 2017 at 4:03 PM, Charles Nadeau <[email protected]>
wrote:
> I’m running PostgreSQL 9.6.3 on Ubuntu 16.10 (kernel 4.4.0-85-generic).
> Hardware is:
>
> *2x Intel Xeon E5550
>
> *72GB RAM
>
> *Hardware RAID10 (4 x 146GB SAS 10k) P410i controller with 1GB FBWC (80%
> read/20% write) for Postgresql data only:
>
> Logical Drive: 3
>
> Size: 273.4 GB
>
> Fault Tolerance: 1+0
>
> Heads: 255
>
> Sectors Per Track: 32
>
> Cylinders: 65535
>
> Strip Size: 128 KB
>
> Full Stripe Size: 256 KB
>
> Status: OK
>
> Caching: Enabled
>
> Unique Identifier: 600508B1001037383941424344450A00
>
> Disk Name: /dev/sdc
>
> Mount Points: /mnt/data 273.4 GB
>
> OS Status: LOCKED
>
> Logical Drive Label: A00A194750123456789ABCDE516F
>
> Mirror Group 0:
>
> physicaldrive 2I:1:5 (port 2I:box 1:bay 5, SAS, 146 GB, OK)
>
> physicaldrive 2I:1:6 (port 2I:box 1:bay 6, SAS, 146 GB, OK)
>
> Mirror Group 1:
>
> physicaldrive 2I:1:7 (port 2I:box 1:bay 7, SAS, 146 GB, OK)
>
> physicaldrive 2I:1:8 (port 2I:box 1:bay 8, SAS, 146 GB, OK)
>
> Drive Type: Data
>
> Formatted with ext4 with: sudo mkfs.ext4 -E stride=32,stripe_width=64 -v
> /dev/sdc1.
>
> Mounted in /etc/fstab with this line: "UUID=99fef4ae-51dc-4365-9210-0b153b1cbbd0
> /mnt/data ext4 rw,nodiratime,user_xattr,noatime,nobarrier,errors=remount-ro
> 0 1"
>
> Postgresql is the only application running on this server.
>
>
> Postgresql is used as a mini data warehouse to generate reports and do
> statistical analysis. It is used by at most 2 users and fresh data is added
> every 10 days. The database has 16 tables: one is 224GB big and the rest
> are between 16kB and 470MB big.
>
>
> My configuration is:
>
>
> name | current_setting | source
>
> ---------------------------------+--------------------------
> ----------------------+----------------------
>
> application_name | psql | client
>
> autovacuum_vacuum_scale_factor | 0 | configuration file
>
> autovacuum_vacuum_threshold | 2000 | configuration file
>
> checkpoint_completion_target | 0.9 | configuration file
>
> checkpoint_timeout | 30min | configuration file
>
> client_encoding | UTF8 | client
>
> client_min_messages | log | configuration file
>
> cluster_name | 9.6/main | configuration file
>
> cpu_index_tuple_cost | 0.001 | configuration file
>
> cpu_operator_cost | 0.0005 | configuration file
>
> cpu_tuple_cost | 0.003 | configuration file
>
> DateStyle | ISO, YMD | configuration file
>
> default_statistics_target | 100 | configuration file
>
> default_text_search_config | pg_catalog.english | configuration file
>
> dynamic_shared_memory_type | posix | configuration file
>
> effective_cache_size | 22GB | configuration file
>
> effective_io_concurrency | 4 | configuration file
>
> external_pid_file | /var/run/postgresql/9.6-main.pid | configuration file
>
> lc_messages | C | configuration file
>
> lc_monetary | en_CA.UTF-8 | configuration file
>
> lc_numeric | en_CA.UTF-8 | configuration file
>
> lc_time | en_CA.UTF-8 | configuration file
>
> listen_addresses | * | configuration file
>
> lock_timeout | 100s | configuration file
>
> log_autovacuum_min_duration | 0 | configuration file
>
> log_checkpoints | on | configuration file
>
> log_connections | on | configuration file
>
> log_destination | csvlog | configuration file
>
> log_directory | /mnt/bigzilla/data/toburn/hp/postgresql/pg_log |
> configuration file
>
> log_disconnections | on | configuration file
>
> log_error_verbosity | default | configuration file
>
> log_file_mode | 0600 | configuration file
>
> log_filename | postgresql-%Y-%m-%d_%H%M%S.log | configuration file
>
> log_line_prefix | user=%u,db=%d,app=%aclient=%h | configuration file
>
> log_lock_waits | on | configuration file
>
> log_min_duration_statement | 0 | configuration file
>
> log_min_error_statement | debug1 | configuration file
>
> log_min_messages | debug1 | configuration file
>
> log_rotation_size | 1GB | configuration file
>
> log_temp_files | 0 | configuration file
>
> log_timezone | localtime | configuration file
>
> logging_collector | on | configuration file
>
> maintenance_work_mem | 3GB | configuration file
>
> max_connections | 10 | configuration file
>
> max_locks_per_transaction | 256 | configuration file
>
> max_parallel_workers_per_gather | 14 | configuration file
>
> max_stack_depth | 2MB | environment variable
>
> max_wal_size | 4GB | configuration file
>
> max_worker_processes | 14 | configuration file
>
> min_wal_size | 2GB | configuration file
>
> parallel_setup_cost | 1000 | configuration file
>
> parallel_tuple_cost | 0.012 | configuration file
>
> port | 5432 | configuration file
>
> random_page_cost | 22 | configuration file
>
> seq_page_cost | 1 | configuration file
>
> shared_buffers | 34GB | configuration file
>
> shared_preload_libraries | pg_stat_statements | configuration file
>
> ssl | on | configuration file
>
> ssl_cert_file | /etc/ssl/certs/ssl-cert-snakeoil.pem | configuration file
>
> ssl_key_file | /etc/ssl/private/ssl-cert-snakeoil.key | configuration file
>
> statement_timeout | 1000000s | configuration file
>
> stats_temp_directory | /var/run/postgresql/9.6-main.pg_stat_tmp |
> configuration file
>
> superuser_reserved_connections | 1 | configuration file
>
> syslog_facility | local1 | configuration file
>
> syslog_ident | postgres | configuration file
>
> syslog_sequence_numbers | on | configuration file
>
> temp_file_limit | 80GB | configuration file
>
> TimeZone | localtime | configuration file
>
> track_activities | on | configuration file
>
> track_counts | on | configuration file
>
> track_functions | all | configuration file
>
> unix_socket_directories | /var/run/postgresql | configuration file
>
> vacuum_cost_delay | 1ms | configuration file
>
> vacuum_cost_limit | 5000 | configuration file
>
> vacuum_cost_page_dirty | 200 | configuration file
>
> vacuum_cost_page_hit | 10 | configuration file
>
> vacuum_cost_page_miss | 100 | configuration file
>
> wal_buffers | 16MB | configuration file
>
> wal_compression | on | configuration file
>
> wal_sync_method | fdatasync | configuration file
>
> work_mem | 1468006kB | configuration file
>
>
> The part of /etc/sysctl.conf I modified is:
>
> vm.swappiness = 1
>
> vm.dirty_background_bytes = 134217728
>
> vm.dirty_bytes = 1073741824
>
> vm.overcommit_ratio = 100
>
> vm.zone_reclaim_mode = 0
>
> kernel.numa_balancing = 0
>
> kernel.sched_autogroup_enabled = 0
>
> kernel.sched_migration_cost_ns = 5000000
>
>
> The problem I have is very poor read. When I benchmark my array with fio I
> get random reads of about 200MB/s and 1100IOPS and sequential reads of
> about 286MB/s and 21000IPS. But when I watch my queries using pg_activity,
> I get at best 4MB/s. Also using dstat I can see that iowait time is at
> about 25%. This problem is not query-dependent.
>
> I backed up the database, I reformated the array making sure it is well
> aligned then restored the database and got the same result.
>
> Where should I target my troubleshooting at this stage? I reformatted my
> drive, I tuned my postgresql.conf and OS as much as I could. The hardware
> doesn’t seem to have any issues, I am really puzzled.
>
> Thanks!
>
>
> Charles
>
> --
> Charles Nadeau Ph.D.
>
--
Charles Nadeau Ph.D.
http://charlesnadeau.blogspot.com/
^ permalink raw reply [nested|flat] 52+ messages in thread
end of thread, other threads:[~2017-08-19 06:51 UTC | newest]
Thread overview: 52+ messages (download: mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2017-07-10 14:03 Very poor read performance, query independent Charles Nadeau <[email protected]>
2017-07-10 15:25 ` Rick Otten <[email protected]>
2017-07-11 10:39 ` Charles Nadeau <[email protected]>
2017-07-12 13:38 ` Charles Nadeau <[email protected]>
2017-07-12 14:10 ` Rick Otten <[email protected]>
2017-07-10 15:35 ` Andreas Kretschmer <[email protected]>
2017-07-10 15:48 ` Charles Nadeau <[email protected]>
2017-07-10 18:35 ` Igor Neyman <[email protected]>
2017-07-11 10:42 ` Charles Nadeau <[email protected]>
2017-07-11 14:34 ` Igor Neyman <[email protected]>
2017-07-11 15:16 ` Igor Neyman <[email protected]>
2017-07-12 07:21 ` Charles Nadeau <[email protected]>
2017-07-12 13:57 ` Igor Neyman <[email protected]>
2017-07-11 15:25 ` Charles Nadeau <[email protected]>
2017-07-11 15:46 ` Igor Neyman <[email protected]>
2017-07-11 12:46 ` Charles Nadeau <[email protected]>
2017-07-12 02:11 ` Mark Kirkwood <[email protected]>
2017-07-14 14:34 ` Charles Nadeau <[email protected]>
2017-07-14 23:09 ` Mark Kirkwood <[email protected]>
2017-07-14 23:57 ` Mark Kirkwood <[email protected]>
2017-07-15 16:12 ` Charles Nadeau <[email protected]>
2017-07-15 23:48 ` Mark Kirkwood <[email protected]>
2017-07-20 12:50 ` Charles Nadeau <[email protected]>
2017-08-19 06:51 ` Mark Kirkwood <[email protected]>
2017-07-15 17:53 ` Charles Nadeau <[email protected]>
2017-07-15 17:58 ` Scott Marlowe <[email protected]>
2017-07-16 09:22 ` Charles Nadeau <[email protected]>
2017-07-10 19:24 ` Jeff Janes <[email protected]>
2017-07-11 11:02 ` Charles Nadeau <[email protected]>
2017-07-12 00:39 ` Jeff Janes <[email protected]>
2017-07-12 10:04 ` Charles Nadeau <[email protected]>
2017-07-12 14:31 ` Igor Neyman <[email protected]>
2017-07-12 16:39 ` Igor Neyman <[email protected]>
2017-07-14 15:34 ` Charles Nadeau <[email protected]>
2017-07-14 19:13 ` Igor Neyman <[email protected]>
2017-07-14 20:18 ` Igor Neyman <[email protected]>
2017-07-17 11:22 ` Charles Nadeau <[email protected]>
2017-07-16 09:20 ` Charles Nadeau <[email protected]>
2017-07-17 20:56 ` Claudio Freire <[email protected]>
2017-07-18 09:20 ` Charles Nadeau <[email protected]>
2017-07-18 16:01 ` Claudio Freire <[email protected]>
2017-07-18 17:13 ` Claudio Freire <[email protected]>
2017-07-18 18:01 ` Justin Pryzby <[email protected]>
2017-07-19 11:48 ` Charles Nadeau <[email protected]>
2017-07-19 01:08 ` Scott Marlowe <[email protected]>
2017-07-12 22:27 ` Jeff Janes <[email protected]>
2017-07-11 23:15 ` Merlin Moncure <[email protected]>
2017-07-11 23:42 ` Joshua D. Drake <[email protected]>
2017-07-12 01:03 ` Jeff Janes <[email protected]>
2017-07-12 07:30 ` Charles Nadeau <[email protected]>
2017-07-12 14:11 ` bricklen <[email protected]>
2017-07-25 09:36 ` Charles Nadeau <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox