Guide to DevSecOps

Appearance

Ubuntu System Monitoring and Log Management Guide ​

This guide covers comprehensive system monitoring and log management techniques for Ubuntu systems, including log rotation, network traffic analysis, resource monitoring, and automated alerts.

Table of Contents ​

  1. System Logs Overview
  2. Managing and Rotating Logs
  3. Monitoring Network Traffic
  4. System Resource Monitoring
  5. User Login and Session Monitoring
  6. Automated Monitoring with Cron Jobs
  7. Advanced Monitoring Solutions

System Logs Overview ​

Ubuntu stores most system logs in the /var/log directory. Here are the key log files:

  • /var/log/syslog: General system logs
  • /var/log/auth.log: Authentication and authorization logs
  • /var/log/kern.log: Kernel messages
  • /var/log/dmesg: Boot messages
  • /var/log/apache2/: Apache web server logs (if installed)
  • /var/log/mysql/: MySQL database logs (if installed)

Viewing Logs ​

Use these commands to view logs:

bash
# View logs in real-time
tail -f /var/log/syslog

# Search logs for specific terms
grep "error" /var/log/syslog

# View last 100 lines of a log
tail -n 100 /var/log/auth.log

# Use journalctl to view systemd logs
journalctl -xe

Managing and Rotating Logs ​

Ubuntu uses logrotate to manage log rotation, preventing logs from consuming excessive disk space.

Configuring Logrotate ​

The main configuration file is /etc/logrotate.conf with additional configs in /etc/logrotate.d/.

To create a custom log rotation policy:

  1. Create a new file in /etc/logrotate.d/:
bash
sudo nano /etc/logrotate.d/custom-app
  1. Add the rotation configuration:
/var/log/custom-app/*.log {
    weekly
    rotate 4
    compress
    delaycompress
    missingok
    notifempty
    create 640 root adm
}
  1. Test your configuration:
bash
sudo logrotate -d /etc/logrotate.d/custom-app

Manual Log Rotation ​

Force log rotation manually:

bash
sudo logrotate -f /etc/logrotate.conf

Monitoring Network Traffic ​

Monitor network packets and connections using various tools.

Using netstat and ss ​

bash
# Show all active connections
ss -tuln

# Display all TCP connections
netstat -tuln

Using tcpdump ​

Capture and analyze packets:

bash
# Capture packets on interface eth0
sudo tcpdump -i eth0

# Capture packets for specific host/port
sudo tcpdump host 192.168.1.1 and port 80

Using iptraf ​

For real-time network statistics:

bash
sudo apt install iptraf-ng
sudo iptraf-ng

System Resource Monitoring ​

Monitoring CPU and Memory ​

bash
# Display system resource usage
top
htop  # More user-friendly alternative (install with: sudo apt install htop)

# Get CPU info
mpstat -P ALL

# Check memory usage
free -h

Disk Space Monitoring ​

bash
# Show disk usage summary
df -h

# Display directory sizes
du -sh /path/to/directory

# Find large files
find / -type f -size +100M -exec ls -lh {} \; 2>/dev/null

I/O Monitoring ​

bash
# Monitor I/O operations
iostat -x 2

# Show I/O statistics by process
iotop  # Install with: sudo apt install iotop

User Login and Session Monitoring ​

Track user logins and active sessions:

bash
# Show currently logged-in users
who
w

# Display last logins
last
lastlog

# Check failed login attempts
faillog
sudo lastb

Log User Activities ​

To enable detailed user command logging, edit the .bashrc file:

bash
sudo nano /etc/bash.bashrc

Add the following:

bash
export PROMPT_COMMAND='if [ "$(id -u)" -ne 0 ]; then echo "$(date +"%Y-%m-%d.%H:%M:%S") $(pwd) $(history 1)" >> ~/.bash_history.log; fi'

Automated Monitoring with Cron Jobs ​

Setting Up Cron Jobs for Monitoring ​

Edit the crontab:

bash
crontab -e

Add monitoring tasks:

# Check disk space every hour
0 * * * * df -h | awk '{ if($5 > "80%") print $0 }' | mail -s "Disk Space Alert" admin@example.com

# Monitor system load every 15 minutes
*/15 * * * * uptime | awk '{ if($10 > 4.0) print "High load: "$0 }' | mail -s "Load Alert" admin@example.com

# Log user logins daily
0 0 * * * last -n 50 > /var/log/user_logins_$(date +\%Y\%m\%d).log

Custom Monitoring Scripts ​

Create a disk space monitoring script:

bash
#!/bin/bash
# Save as /usr/local/bin/monitor-disk.sh

THRESHOLD=80
ADMIN_EMAIL="admin@example.com"

df -h | grep -vE "^Filesystem|tmpfs|cdrom|loop" | awk '{ print $5 " " $1 }' | while read -r output;
do
  usage=$(echo "$output" | awk '{ print $1 }' | cut -d'%' -f1)
  partition=$(echo "$output" | awk '{ print $2 }')

  if [ "$usage" -ge $THRESHOLD ]; then
    echo "ALERT: Partition $partition is at $usage%" | mail -s "Disk Usage Alert" $ADMIN_EMAIL
  fi
done

Make it executable and add to crontab:

bash
sudo chmod +x /usr/local/bin/monitor-disk.sh
0 * * * * /usr/local/bin/monitor-disk.sh

Advanced Monitoring Solutions ​

For more comprehensive monitoring, consider these tools:

Setting Up Prometheus with Node Exporter ​

bash
# Install Node Exporter
wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz
tar xvfz node_exporter-*.tar.gz
sudo mv node_exporter-*/node_exporter /usr/local/bin/

# Create systemd service
sudo nano /etc/systemd/system/node_exporter.service

Add the following:

ini
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Start and enable the service:

bash
sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter

Setting Up Grafana ​

For visualizing metrics:

bash
sudo apt-get install -y apt-transport-https software-properties-common
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install grafana
sudo systemctl start grafana-server
sudo systemctl enable grafana-server

Visit http://your-server-ip:3000 to access Grafana (default credentials: admin/admin).

ELK Stack for Log Management ​

For advanced log aggregation and analysis, consider setting up the ELK stack (Elasticsearch, Logstash, Kibana).

Conclusion ​

This guide covered essential techniques for monitoring an Ubuntu system, managing logs, tracking network traffic, monitoring system resources, tracking user sessions, and setting up automated monitoring. By implementing these practices, you can maintain system health, detect issues early, and ensure optimal performance.

For production environments, consider implementing a comprehensive monitoring solution like Prometheus with Grafana or the ELK stack for more advanced monitoring capabilities.