Below is a not so brief summary from my on staff server technician on how he would go about finding an issue like what you are reporting. I hope this helps you track it down.
Check historical load average using the sar command:
for day in $(find /var/log/sa -type f -name 'sa*');do
sar -q $day
done
This will give you the load average taken every hour of the day for the past month (assuming that you have sar installed). If you do not have sar installed, install it with your package manager.
Look for trends. If your load sky rockets at a specific time of the day every day for a short period then look for cron jobs, including those that are part of web application code, that run at
these times.
If you don't have sar then you may want to go straight to the 'top' command while the server is running slowly. Watch the cpu usage, load, and iowait to see where the bottleneck is. A high load
with a low iowait suggests that there is a process is simply require a lot of cpu, while high load coupled with high iowait suggests that something disk reading/writing is causing the bottleneck.
If the load is high, but there is little to no iowait then look at the process list to see what is using the most CPU. If the process name does not make clear what file it is then you can get the pid
of the process (far left side of top) an run the command 'lsof -p ' to get a list of files opened by the process, or look around at /proc/ to see things like the file 'cmdline', which will tell
you what command started the process.
If iowait is high then you should start by looking at disk usage. If you are on a single disk, raid 0, or raid 1 system then having a disk more than about 80% full will make a major impact. If
running raid 5 or 10, especially raid 10 with 6 or more disks in the array, you can get up to around 95-97% before it is a major problem in most cases. A full disk causes the load to rise due to
instructions getting queued becuase the currently executing instrunctions are waiting on some disk IO to finish while the disks are taking a long time running seeks to find empty space.
If your disk usage is fine then you should run a program called iotop to see what process is using the most disk io. Remember that bandwidth isn't necessarily the problem. It causes a lot more wait
often times to do a lot of random reads and writes than it does to write more bytes in sequence at once. Normally the offending process will be doing more iops and the Mbps written isn't that important.
Common iowait culprits:
mysql queries that write temp tables to disk or do huge full table scans without indexes. You can see these temp tables being written to disk as they happen by running "ls -lah /tmp|grep '#'|grep mysql".
This will even tell you the size of the temp table that is being written. To use this method you have to run the command over and over again very quickly. Mysql can write and remove even large temp tables (200M+)
in less than a second, so after running the command in your terminal once you can use the up arrow followed by enter to replay that last command or do something like "watch -n.2 'ls -lah /tmp|grep '#'|grep mysql".
If you do find that temp tables are the culprit, or want to further investigate queries that could be the issue, then mysqladmin is your best friend. The command 'mysqladmin proc' will show the mysql
process list including the queries themselves. Normally you will have to use the -h, -u, and -p arguments for host, user, and password with it. This is another command that should be run over and over
as the queries come and go very quickly.
Another issue I have often found is iowait generated by having too many inodes (files) in one directory. This is common when there is a crontab that generates output to stdout. In this case the output gets
emailed to the crontab user's email address. On a stock Ubuntu or CentOS installation this would be something like a single file in /var/mail/, however if email aliases are set up, as with a cPanel server
then the email is sent to that users actual dovecot/courier mail box, with Maildir using a separate file for each message. I have seen servers where there was a crontab running several jobs once per minute that generated
an email on each run. This cron user's email account had millions of files in one directory. It was crippling a 24 core VPS hosting server with an 8 disk raid 10 array. Make sure that any cron jobs end with stdout
and stderr being redirected to either /dev/null, or a log.
Logs that rotate on a schedule without clearing dated logs are a common issue as well. Check anywhere that you store logs and make sure that there aren't any building up into the 10's of thousands. You may be able to use
repquota -a to view how many inodes are owned by each user to aid in this. This depends on whether or not you have quotas turned on or not in the file system.
If none of these things shine a light on the problem then we should look at network wait.
99% of the time that network wait is an issue at the same time every day it is a backup running and being shipped off somewhere.
Either from that server or by the hosting company
php -f shell/log.php -- statusmytopto make sure database is the problem, check queries and query time. also I/O could be a problem too, with bad drives