Linux Troubleshooting

Bootup Issues won’t Boot

Fix grub:
=> check if grub exists: windows installed or wiped from MBR etc – press either Esc or shift after bios boot to see if grub exits.

=> grub prompt: may be corrupt or something missing in the config lines, check another instance on a different computer and see if typing those commands can manually boot the system.

=> Misconfigured prompt: try an older entry in the grub menu to see if it’s an error or new kernel

=> Check /etc/defualt/grub for correct syntax or errors. Run update-grub after file change to update /boot/grub/grub.cfg

=> Repair grub with rescue disk – Ubuntu install disk has a rescue mode option by default that can reinstall grub etc.

=> Can’t mount file system – Check root and partitions uuid labels or the disk label

Disk full can’t write to disk

=> Track down largest directories with:

 du -ckx | sort -n > /tmp/durpt.log 

Other DU examples:

du -sh
du -ckhx | sort -n
du -ah | grep M |sort -n
du -ckxh /var/ | sort -n |grep M
du -a /var | sort -n -r | head -n 10
du -hsx * | sort -rh | head -10

Some find commands to use for disk space usage:

find / -xdev -type f -size +50M
find / -xdev -type f -size +50M -exec ls -alh {} \; | sort -nk 5
find /usr -type f -printf "%s %p\n" | sort -rn | head -n 10

Then tail the log file and check the largest sizes to investigate.

=> Check /var/log for oversized log files and rotate or truncate if possible.

=> Check /tmp for large files .swp files caused by another process or user leaving a log file opened with vim. Use lsof to find the pid and kill that process which in turn will free the disk space. This is often when df -h and du -h provide different results.

=> Out of INODES – As per above if du and df show different results you could be out of inodes. df -i to check free inodes or inodes usage. Check for large amount if small files and see if they can be moved. tar them etc

=> Error that file system is read-only – first see if a remount with rw will work with:

 sudo mount -o remount,rw /home 

Could be due to error on boot and the system is protecting itself, check /var/log/dmesg for mount errors.

=> Check file system with fsck – unmount the system first and run fsck -y -C /dev/sda3 as an example. Also can superblocks check with mke2fs commands.

Website is down

=> Check ports are open e.g. port 80 443 or special app ports 8000 for splunk Check the port is available with either telnet ot nmap and if a firewall/network issue is preventing the access. Then check the netstat command to see if the service is running correctly on the web server itself.

telnet webserver 80
nmap -v -p 80 webserver
netstat -lnp | grep :80

=> Test the webserver via curl

 curl http://webserver.com 

Check for http status code with curl:

curl -w "%{http_code}\n" http://www.example.net

HTTP STATUS CODES

List of most important HTTP Status Codes, before you get into the list, you must know the 4 important categories:

  • Success Codes (2xx)
  • Redirection Codes (3xx)
  • Client Side Error Codes (4xx)
  • Server Side Error Codes (5xx)

1xx: Information:
100     Continue
101     Switching Protocol

Successful responses:
200     OK
201     Created
202     Accepted
203     Non-Authoritative Information
204     No Content
205     Reset Content
206     Partial Content

Redirection messages:
300     Multiple Choice
301     Moved Permanently
302     Found
303     See Other
304     Not Modified
305     Use Proxy
306     unused
307     Temporary Redirect
308     Permanent Redirect

Client error responses:
400     Bad Request
401     Unauthorized
402     Payment Required
403     Forbidden
404     Not Found
405     Method Not Allowed
406     Not Acceptable
407     Proxy Authentication Required
408     Request Timeout
409     Conflict
410     Gone
411     Length Required
412     Precondition Failed
413     Request Entity Too Large
414     Request-URI Too Long
415     Unsupported Media Type
416     Requested Range Not Satisfiable
417     Expectation Failed

Server error responses:
500     Internal Server Error
501     Not Implemented
502     Bad Gateway
503     Service Unavailable
504     Gateway Timeout
505     HTTP Version Not Supported

WebServer Configuration Tests:

Apache2 - apache2ctl configtest
Nginx    - nginx -t

GENERAL

sudo dd if=/dev/mem | cat | strings

cat /proc/meminfo

cat /proc/cpuinfo

lspci

lsusb

fdisk -l

Kill a process ps -A | grep ProgramName kill 7207

List all files that are currently open on the system with “lsof”. This command will allow you to see all the files that are currently open on your system. Limiting the directory or coupling this command with grep is often useful for finding files that are still open restricting the ability to unmount a device. Lsof will also ouput the process id or PID. You can then kill the process using the kill command above.

 lsof 

Keep an eye on something for awhile – watch

The watch command will repeat a command at a set interval (default 2 seconds) and output the response. This is useful for watching directories that change, watching hard drives fill up when a lot of data is being transfered, or using it with lsusb to watch for USB device being plugged in.

watch ls
watch df -h

Find where a binary is stored and its libraries Often times when running a cron command you want to include the absolute path to the command. Sometimes I run scheduled PHP tasks. This can be acomplished by using the ëwhereisë command.

whereis php5

See if you have kernel boot issues dmesg | less

For more logs just cd into the /var/log directory and start using, cat, less, tail, grep, find or any other tool to view and search.

SSLScan – check ssl servers

SSLScan queries SSL services, such as HTTPS, in order to determine the ciphers that are supported. SSLScan is designed to be easy, lean and fast. The output includes prefered ciphers of the SSL service, the certificate and is in Text and XML formats.

Install:

sudo apt-get install sslscan

Syntax and Example:

sslscan --no-failed ausinfotech.net

If I want to know whether the server still supports SSLv2, I can check the target like this:

sslscan --no-failed --ssl2 ausinfotech.net

Check mail Servers:

sslscan --no-failed --starttls --mail.example.com:2

For more details see:
https://github.com/rbsec/sslscan

Generating puppet password hashes

Puppet needs user passwords in configuration files to be encrypted in the format the local system expects. For Linux and most unix-like system, that means, you have to put the sha1 sum of the password into the configuration file.

Example:
user { 'root':
ensure => 'present',
password => '$1$HTQx9U32$T6.lLkYxCp3F/nGc4DCYM/',
}

To generate the password use Python with:
python -c 'import crypt; print crypt.crypt("password", "$6$salt")'

Obviously don’t use password as your password!