I have an (almost) dedicated system which I'm using for some database stuff. My problem is: A Java application is running pretty slow (taking 2-3 days to complete a task), and I don't know why.
The system is a Xubuntu 12.04.1 LTS, 3.2.0-32-generic x86_64, mySQL 5.5.28, Phenom X2 550, 16 GB RAM and an Intel SSD 520 (utilizing a SandForce processor known for high IOPS).
First, the CPU load is never really at 100%. The htop output looks like this most of the time:
1 : 63.9% sys: 12.9% low: 0.0% Tasks: 99, 164 thr; 2 running
2 : 18.5% sys: 21.2% low: 0.7% Load average: 1.14 1.23 1.22
Mem[|||||||||||||||||13490/16049MB] Uptime: 1 day, 06:32:13
Swp:8188M used:195M Time: 18:33:04
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
16168 micha 20 0 5163M 1595M 4500 S 75.0 9.9 11h36:10 java -jar ./build
16169 micha 20 0 5163M 1595M 4500 R 75.0 9.9 11h31:34 java -jar ./build
5968 mysql 20 0 11.9G 10.7G 3992 S 31.0 68.4 13h21:10 /usr/sbin/mysqld
6142 mysql 20 0 11.9G 10.7G 3992 S 31.0 68.4 10h44:50 /usr/sbin/mysqld
Sometimes also mysqld has a higher CPU usage than java, but it never get's close to 100% on both cores.
The database I'm working on is rather big, with a total of 24 GB of InnoDB tables. Every table has a PRIMARY key and I also used EXPLAIN to get my indexes right. The java tool does rather simple stuff, but tons of it. The tables involved have around 10-20M rows and the java application is doing a magnitude more of queries (SELECT, UPDATE and INSERT). I'm using prepared statements.
These are my settings for InnoDB:
mysql> show variables like 'innodb%'
-> ;
+---------------------------------+------------------------+
| Variable_name | Value |
+---------------------------------+------------------------+
| innodb_adaptive_flushing | ON |
| innodb_adaptive_hash_index | ON |
| innodb_additional_mem_pool_size | 8388608 |
| innodb_autoextend_increment | 8 |
| innodb_autoinc_lock_mode | 1 |
| innodb_buffer_pool_instances | 1 |
| innodb_buffer_pool_size | 10737418240 |
| innodb_change_buffering | all |
| innodb_checksums | ON |
| innodb_commit_concurrency | 0 |
| innodb_concurrency_tickets | 500 |
| innodb_data_file_path | ibdata1:10M:autoextend |
| innodb_data_home_dir | |
| innodb_doublewrite | ON |
| innodb_fast_shutdown | 1 |
| innodb_file_format | Antelope |
| innodb_file_format_check | ON |
| innodb_file_format_max | Antelope |
| innodb_file_per_table | OFF |
| innodb_flush_log_at_trx_commit | 0 |
| innodb_flush_method | |
| innodb_force_load_corrupted | OFF |
| innodb_force_recovery | 0 |
| innodb_io_capacity | 200 |
| innodb_large_prefix | OFF |
| innodb_lock_wait_timeout | 50 |
| innodb_locks_unsafe_for_binlog | OFF |
| innodb_log_buffer_size | 8388608 |
| innodb_log_file_size | 5242880 |
| innodb_log_files_in_group | 2 |
| innodb_log_group_home_dir | ./ |
| innodb_max_dirty_pages_pct | 75 |
| innodb_max_purge_lag | 0 |
| innodb_mirrored_log_groups | 1 |
| innodb_old_blocks_pct | 37 |
| innodb_old_blocks_time | 0 |
| innodb_open_files | 300 |
| innodb_purge_batch_size | 20 |
| innodb_purge_threads | 0 |
| innodb_random_read_ahead | OFF |
| innodb_read_ahead_threshold | 56 |
| innodb_read_io_threads | 4 |
| innodb_replication_delay | 0 |
| innodb_rollback_on_timeout | OFF |
| innodb_rollback_segments | 128 |
| innodb_spin_wait_delay | 6 |
| innodb_stats_method | nulls_equal |
| innodb_stats_on_metadata | ON |
| innodb_stats_sample_pages | 8 |
| innodb_strict_mode | OFF |
| innodb_support_xa | ON |
| innodb_sync_spin_loops | 30 |
| innodb_table_locks | ON |
| innodb_thread_concurrency | 0 |
| innodb_thread_sleep_delay | 10000 |
| innodb_use_native_aio | OFF |
| innodb_use_sys_malloc | ON |
| innodb_version | 1.1.8 |
| innodb_write_io_threads | 4 |
+---------------------------------+------------------------+
59 rows in set (0.01 sec)
Finally, when looking at iotop, the disk hovers at around 1% usage, so I think mySQL actually works only in/with system memory most of the time and does not thrash the disk. I think this excludes RAM and SSD as bottleneck and only leaves the CPU.
But before investing in a new CPU, I would like to ask for other/more opinions. Is the CPU really my bottleneck? If yes, why is it well below 100% usage?