A server in a live environment is always susceptible to malware attacks, code vulnerabilities, hacking, rootkits and what not else !
Analyzing your server for these back-doors and pin-pointing the exact issue always poise a big problem to the server owners or their administrators. Thankfully Linux can provide you with many results, logs and statistics which can help you in a great way. Lets look at some of the things which you can do when you feel like your system is compromised..
- Unintended processes running can at-times increase the server load. Using the command
# top -c, check the current server load and analyze if you can find any unfamiliar processes or process paths running.
- Check for the process tree in your server and see if you can spot any unusual process/paths. Use the command
# pstree -p. If you find an unsual process get the PID of it and alanyze using lsof command, which will lead to current working directory ( cwd ) of the process.
Just for an example, consider you find lots of Perl process running, the root perl process has got the PID ‘9905’. Use the command # lsof PID to check more about this.
=================
=================
Checking this shows you something, an invalid/malicious Perl script is running from /tmp, which should not happen at all. Analyze the files in /tmp and make sure you clean up the unwanted ones.
- Use # ls -al /tmp ( followed by grep arguments ) to check for the files in /tmp.
- Another recently noted process in
# pstree -p, is lots of hosts commands being carried out. Capture the PID of one of the process with the name ‘hosts’ and run an lsof on it. Probably you might see something like :
================
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
host 7909 nobody cwd DIR 0,32 4096 117770255 /home/domain/public_html/blog/wp-content/uploads
================
The host command, which is used for resolving DNS, is being executed and some scripts is coercing it to do HTTP requests, which is seen as the issue here. Blocking the scripts found in the location ( cwd ) will help in this case. Analyze if it was uploaded via any unsecured plugins ( mostly happens with WordPress )
- Check the IP’s making connections to you server. Use the command given below to find the number of tcp/udp flow :
# netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
If you find the connections from an IP is not on the normal side, check for what it is trying to access :
# netstat -plan | grep IP
And block ’em in the firewall if it looks invalid.
- If you are using a cPanel server, you can find what exactly it is trying to access if its to the web-server.
# grep -i -r IP /usr/local/apache/domlogs*
There are many occasions in which the IP would be trying to brute-force attack the logins such as WP-login.php. In those cases, you can find that IP is ‘POST’ ing the logins to the login.php page.
- Changing the username of WordPress from the default ‘admin’ to a stronger one can also help. These sort of attacks can increase the server load as well as compromise the server security, if the brute-force attack is successful.
- Find out the most accessed domain for a particular day by giving the following command :
# grep -r '14/Dec/2014' /usr/local/apache/domlogs* |awk {'print $1'}|cut -d: -f1|sort -n|uniq -c|sort -n
^ Date is to be given in this format.
- Find out the most accessed IP for the domain which you get from the above result :
# grep -r '14/Dec/2014' /usr/local/apache/domlogs/domain.tld |awk {'print $1'}|cut -d: -f1|sort -n|uniq -c|sort -n
- Checking the root history (
cat /root/.bash_history) can help you to see if your server is root hacked. Also check # last command to see if you can spot any unfamiliar IPs, which tried to login to your box.
- You can use rkhunter to scan for possible rootkits and local exploits in your linux based box. It also performs checks to see if commands/paths have been altered , if the system startup files have been modified, and various checks on the network interfaces, including checks for listening applications.
Once rkhunter is installed, you can scan the system using the command # rkhunter --check and find if it returns any negative result.
So, let the Sherlock Holmes investigation start !