System Administrator Full Documentation

This documentation is an extensive guide designed for Linux system administrators. It covers complete workflows including server setup, networking, security, email systems, web hosting, MySQL, and storage management.

Code examples are formatted in VS Code dark theme. Every command includes comments, explanations, and real-world usage scenarios.

System Overview

This section introduces fundamental tools used by system administrators to inspect, monitor, and understand the state of a Linux server. These commands are the first step when diagnosing issues or performing audits.

# Show system information
uname -a # Explanation: Shows kernel version, architecture, and OS type. # Used when checking compatibility or diagnosing kernel issues.

# Distribution details
cat /etc/os-release # Example output: NAME="Ubuntu" VERSION="22.04 LTS"

# System uptime
uptime # Example: "14:22 up 5 days, load average: 0.15, 0.20, 0.30"

# Hardware info
lscpu lsmem lsblk lspci lsusb # These help identify RAM, CPU cores, PCI devices, or attached USB hardware.

# Disk usage summary
df -h # Use this to ensure partitions are not full. A full disk stops MySQL, Postfix, or Apache.

# Memory usage
free -h # Shows available and used RAM. High swap usage indicates low memory.

# List all services
systemctl list-units --type=service # Useful to confirm which services failed.

Users and Groups

This section covers user management, group administration, and best practices for secure account handling. System administrators use these commands daily when onboarding users, assigning roles, or securing directories.

# Create a new user (with home directory)
sudo adduser johndoe # Prompts for password and creates /home/johndoe # Use this for normal user accounts.

# Create a system user (no shell, no home)
sudo useradd -r -s /usr/sbin/nologin mysqluser # Used for system services like MySQL, Postfix, web apps.

# Change user shell
sudo chsh -s /bin/bash johndoe

# Assign user to a group
sudo usermod -aG sudo johndoe # Adds johndoe to sudoers without removing other groups.

# List all users
cut -d: -f1 /etc/passwd

# List groups
cut -d: -f1 /etc/group

# Delete a user (keep home folder)
sudo userdel johndoe

# Delete a user with home folder
sudo userdel -r johndoe

# Switch to another user account
su - johndoe

Permissions and Ownership

Permissions determine who can read, write, or execute files and directories. Misconfiguration can cause service failures (Apache, MySQL, Postfix). Understanding these is essential for security.

# Viewing permissions
ls -l # Example: -rw-r--r-- 1 root root 1200 file.txt # Meaning: owner=rw, group=r, others=r

# Change owner
sudo chown johndoe:johndoe file.txt

# Recursive ownership change
sudo chown -R www-data:www-data /var/www/html # Required for Apache document root.

# Change permissions
chmod 755 script.sh chmod 644 config.php # 755 = rwx r-x r-x (execute allowed) # 644 = rw- r-- r-- (common for configs)

# Give execute permission
chmod +x run.sh

# Remove write permissions from others
chmod o-w file.txt

Sudoers Configuration

Sudo allows controlled privilege escalation. Editing sudoers incorrectly can lock you out entirely, so always use visudo, which checks syntax.

# Edit sudoers safely
sudo visudo

# Allow user full sudo access
johndoe ALL=(ALL) ALL

# Allow user to run specific command
johndoe ALL=(ALL) NOPASSWD: /usr/bin/systemctl restart apache2

# Allow group sudo privileges
%support ALL=(ALL) ALL

Processes, Logs, and System Monitoring

Process management is essential for diagnosing high CPU, memory leaks, or services failing. Linux provides many tools for system observation.

# List running processes
ps aux

# Real-time process view
top
htop # install: sudo apt install htop

# Detailed kernel messages
dmesg | less

# Follow system logs live
sudo journalctl -f

# Service status
sudo systemctl status apache2
sudo systemctl status mysql

# Restart failed services
sudo systemctl restart postfix

# Enable services on boot
sudo systemctl enable dovecot

# Show failed units
systemctl --failed

Disk Management

Disk operations include creating partitions, formatting, mounting, or preparing disks for LVM or RAID. These commands must be used carefully because mistakes can destroy data.

# List disks and partitions
lsblk fdisk -l

# Create or edit a partition
sudo fdisk /dev/sdb # Commands: n = new partition, d = delete, w = write changes

# Format a disk
sudo mkfs.ext4 /dev/sdb1 sudo mkfs.xfs /dev/sdb1

# Mount a filesystem
sudo mount /dev/sdb1 /mnt/data

# List mounted filesystems
mount | grep /mnt

# Unmount a disk
sudo umount /mnt/data

Logical Volume Management (LVM)

LVM allows flexible disk management: resizing storage without unmounting, adding new disks easily, or creating snapshots. Essential for servers.

# 1. Create Physical Volume
sudo pvcreate /dev/sdb1

# 2. Create Volume Group
sudo vgcreate datavg /dev/sdb1

# 3. Create Logical Volume
sudo lvcreate -L 50G -n datalv datavg

# 4. Format LV
sudo mkfs.ext4 /dev/datavg/datalv

# 5. Mount LV
sudo mkdir /data
sudo mount /dev/datavg/datalv /data

# Extend LV and filesystem (online)
sudo lvextend -L +20G /dev/datavg/datalv sudo resize2fs /dev/datavg/datalv # Now LV grows by 20 GB without downtime.

RAID (mdadm)

RAID increases redundancy or speed. RAID1 mirrors disks, RAID5 provides parity, RAID0 improves performance but no redundancy.

# Install RAID tools
sudo apt install mdadm -y

# Create RAID1 (mirror)
sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1

# Create RAID5
sudo mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sd{b1,c1,d1}

# Check RAID status
cat /proc/mdstat

# Save RAID config
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

fstab & Automounting

The /etc/fstab file ensures filesystems mount automatically at boot. Mistakes here can prevent the server from booting.

# Find UUID of a disk
sudo blkid

# Sample fstab entry
# /dev/sdb1 mounted to /data with ext4 filesystem
UUID=xxxx-xxxx /data ext4 defaults 0 2

# Test fstab for errors
sudo mount -a # If no output, fstab is correct.

Backups & Archiving

Backups are essential for server reliability. Administrators use tar, zip, rsync, or mysqldump to protect data.

# Create archive
tar -czvf backup.tar.gz /var/www/html

# Extract archive
tar -xzvf backup.tar.gz

# Sync directories (incremental backup)
rsync -avz /var/www/ /backup/www/

Cron Jobs

Cron automates repetitive tasks like backups, updates, log rotation, and certificate renewal.

# Edit cron jobs
crontab -e

# Cron format:
# minute hour day month weekday command
# Daily backup at 1 AM
0 1 * * * /usr/bin/rsync -av /data /backup/data

# Weekly apt updates
0 4 * * 0 sudo apt update && sudo apt upgrade -y

IP Configuration & Routing

System administrators frequently configure network interfaces, diagnose connectivity issues, or modify routing tables. Below are the foundational commands with full explanations.

# View all IP addresses
ip addr show # Shows interface names, MAC addresses, assigned IPs, and link status.

# Assign temporary IP to interface (lost after reboot)
sudo ip addr add 192.168.1.50/24 dev enp0s3

# Delete assigned IP
sudo ip addr del 192.168.1.50/24 dev enp0s3

# Bring interface up or down
sudo ip link set enp0s3 up sudo ip link set enp0s3 down

# Display routing table
ip route # Shows default gateway and route priorities.

# Add new route
sudo ip route add 10.0.0.0/24 via 192.168.1.1 dev enp0s3

# Delete route
sudo ip route del 10.0.0.0/24

# Test connectivity
ping -c 4 google.com ping -c 4 8.8.8.8 # DNS-independent test

# Trace path to a remote server
traceroute google.com

DNS Tools

DNS resolution is crucial for communication between services. Administrators need to test DNS records, MX mail routing, or troubleshoot name failures.

# DNS lookup
dig example.com

# Query specific record types
dig MX example.com dig A example.com dig TXT example.com

# Reverse lookup
dig -x 8.8.8.8

# Query using specific DNS server
dig @8.8.8.8 example.com

# Simplified DNS query
nslookup example.com

Netplan Configuration

Netplan controls permanent network configuration in Ubuntu Server. It processes YAML configuration files and applies them to systemd-networkd or NetworkManager.

# Edit main netplan file
sudo nano /etc/netplan/50-cloud-init.yaml

# Sample static IP configuration
network:
  version: 2
  renderer: networkd
  ethernets:
    enp0s3:
      dhcp4: no
      addresses: [192.168.1.10/24]
      gateway4: 192.168.1.1
      nameservers:
        addresses: [8.8.8.8, 1.1.1.1]

# Apply configuration
sudo netplan apply

# Validate configuration syntax
sudo netplan try

Network Bonding (Interface Teaming)

Bonding provides redundancy or increased bandwidth by combining multiple network interfaces into a single logical interface.

# Install bonding tools
sudo apt install ifenslave

# Sample Netplan bonding configuration
network:
  version: 2
  renderer: networkd
  bonds:
    bond0:
      interfaces: [enp0s3, enp0s8]
      parameters:
        mode: active-backup
        primary: enp0s3
      dhcp4: yes

# Apply bonding
sudo netplan apply

# Check bonding status
cat /proc/net/bonding/bond0

DHCP Server & Client

DHCP automatically assigns IP addresses. System administrators configure DHCP servers for internal networks or troubleshoot client issues.

DHCP Server (isc-dhcp-server)

# Install DHCP server
sudo apt install isc-dhcp-server -y

# Edit DHCP configuration
sudo nano /etc/dhcp/dhcpd.conf

# Example DHCP pool
subnet 192.168.1.0 netmask 255.255.255.0 {
  range 192.168.1.100 192.168.1.200;
  option routers 192.168.1.1;
  option domain-name-servers 8.8.8.8, 1.1.1.1;
}

# Enable interface
sudo nano /etc/default/isc-dhcp-server INTERFACESv4="enp0s3"

# Restart service
sudo systemctl restart isc-dhcp-server
sudo systemctl status isc-dhcp-server

DHCP Client Tools

# Renew DHCP lease
sudo dhclient -r sudo dhclient

# View DHCP leases
cat /var/lib/dhcp/dhclient.leases

SSH Hardening

SSH is the primary admin access method. Hardening prevents unauthorized access and brute-force attacks.

# Check SSH status
sudo systemctl status ssh

# Edit SSH config
sudo nano /etc/ssh/sshd_config

# Recommended security settings
Port 2222 PermitRootLogin no PasswordAuthentication no PubkeyAuthentication yes AllowUsers johndoe sysadmin

# Restart SSH
sudo systemctl restart ssh

UFW Firewall

UFW provides simple syntax for firewall rule configuration. Essential for restricting access to services.

# Default policies
sudo ufw default deny incoming sudo ufw default allow outgoing

# Allow essential ports
sudo ufw allow 22/tcp # SSH
sudo ufw allow 80/tcp # HTTP
sudo ufw allow 443/tcp # HTTPS
sudo ufw allow 25/tcp # SMTP
sudo ufw allow 587/tcp # Submission
sudo ufw allow 993/tcp # IMAP SSL

# Deny specific IP
sudo ufw deny from 10.0.0.55

# Enable firewall
sudo ufw enable

# Check rules
sudo ufw status numbered

Fail2ban Intrusion Prevention

Fail2ban blocks repeated unauthorized login attempts. It is vital for SSH, Postfix, and Apache brute-force protection.

# Install fail2ban
sudo apt install fail2ban -y

# Create local jail override file
sudo nano /etc/fail2ban/jail.local

# Recommended configuration
[sshd]
enabled = true
port = 2222
filter = sshd
logpath = /var/log/auth.log
maxretry = 5

# Restart fail2ban
sudo systemctl restart fail2ban

# View banned IPs
sudo fail2ban-client status sshd

OpenSSL Certificate Management

OpenSSL creates private keys, CSRs, and self-signed certificates. Used for HTTPS, SMTP, IMAP, and internal secure communication.

# Generate a private key
openssl genrsa -out server.key 2048

# Generate a certificate signing request
openssl req -new -key server.key -out server.csr
# Fill in domain details: CN = example.com

# Self-signed certificate (valid 1 year)
openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

# View certificate details
openssl x509 -in server.crt -noout -text

Apache Web Server

Apache hosts websites, APIs, and applications. System administrators configure virtual hosts, SSL certificates, and document roots.

# Install Apache
sudo apt update
sudo apt install apache2 -y

# Verify Apache status
sudo systemctl status apache2

# Enable Apache modules
sudo a2enmod rewrite sudo a2enmod ssl sudo systemctl restart apache2

# Create a new site configuration
sudo nano /etc/apache2/sites-available/example.conf

# Sample Virtual Host
<VirtualHost *:80>
  ServerName example.com
  ServerAlias www.example.com
  DocumentRoot /var/www/example
  ErrorLog ${APACHE_LOG_DIR}/example-error.log
  CustomLog ${APACHE_LOG_DIR}/example-access.log combined
</VirtualHost>

# Enable site
sudo a2ensite example.conf
sudo systemctl reload apache2

# Disable default site
sudo a2dissite 000-default.conf

PHP Installation & Modules

PHP is required by many applications including Roundcube, WordPress, Laravel, moodle, etc.

# Install PHP and common modules
sudo apt install php php-cli php-mysql php-curl php-gd php-xml php-mbstring php-intl php-zip libapache2-mod-php -y

# Check PHP version
php -v

# Restart Apache to load modules
sudo systemctl restart apache2

SSL Certificates with Certbot (Let's Encrypt)

SSL encrypts communication for HTTPS, IMAP, SMTP, and Roundcube. Certbot provides free, automated certificates.

# Install Certbot
sudo apt install certbot python3-certbot-apache -y

# Request an SSL certificate
sudo certbot --apache -d example.com -d www.example.com

# Certificate location
/etc/letsencrypt/live/example.com/

# Test renewal
sudo certbot renew --dry-run

MySQL Installation

MySQL powers databases for PHP applications. Correct configuration ensures security and performance.

# Install MySQL
sudo apt install mysql-server -y

# Secure installation
sudo mysql_secure_installation

# Log in as root
sudo mysql

# Create database
CREATE DATABASE exampledb;

# Create user with password
CREATE USER 'exampleuser'@'localhost' IDENTIFIED BY 'StrongPass123!';

# Grant privileges
GRANT ALL PRIVILEGES ON exampledb.* TO 'exampleuser'@'localhost';
FLUSH PRIVILEGES;

MySQL Backups

Backups protect against corruption, accidental deletion, and hardware failures.

# Export database
mysqldump -u root -p exampledb > exampledb.sql

# Import database
mysql -u root -p exampledb < exampledb.sql

# Scheduled backup via cron
0 2 * * * mysqldump exampledb > /backup/exampledb-$(date +\%F).sql

Postfix (SMTP Server)

Postfix is the SMTP server used to send outgoing email. Below is the full installation, configuration, and explanation for a production mail server.

# Install Postfix
sudo apt install postfix -y

# Select configuration type:
# Internet Site

# Configure main Postfix settings
sudo nano /etc/postfix/main.cf

# Recommended settings
myhostname = mail.example.com
mydomain = example.com
myorigin = /etc/mailname
inet_interfaces = all
mydestination = \$myhostname, localhost.\$mydomain, localhost
home_mailbox = Maildir/
smtpd_tls_cert_file=/etc/letsencrypt/live/example.com/fullchain.pem
smtpd_tls_key_file=/etc/letsencrypt/live/example.com/privkey.pem
smtpd_use_tls=yes
smtp_tls_security_level = may

# Restart Postfix
sudo systemctl restart postfix
sudo systemctl status postfix

Dovecot (IMAP/POP3 Authentication)

Dovecot allows users to receive email using IMAP/POP3. It works together with Postfix to store and retrieve messages.

# Install Dovecot
sudo apt install dovecot-core dovecot-imapd dovecot-pop3d -y

# Use Maildir format
sudo nano /etc/dovecot/conf.d/10-mail.conf

mail_location = maildir:~/Maildir

# Enable SSL for IMAP
sudo nano /etc/dovecot/conf.d/10-ssl.conf

ssl = yes
ssl_cert =
ssl_key =

# Restart Dovecot
sudo systemctl restart dovecot
sudo systemctl status dovecot

OpenDKIM

OpenDKIM digitally signs outgoing mail so receiving servers trust your mail. Prevents spam and spoofing alerts.

# Install OpenDKIM
sudo apt install opendkim opendkim-tools -y

# Configure OpenDKIM
sudo nano /etc/opendkim.conf

Syslog yes
UMask 002
Mode sv
KeyTable /etc/opendkim/key.table
SigningTable /etc/opendkim/signing.table
ExternalIgnoreList /etc/opendkim/trusted.hosts
InternalHosts /etc/opendkim/trusted.hosts

# Create signing directory
sudo mkdir -p /etc/opendkim/keys/example.com

# Generate DKIM key
sudo opendkim-genkey -D /etc/opendkim/keys/example.com/ -d example.com -s mail
sudo chown opendkim:opendkim /etc/opendkim/keys/example.com/*

# Add key to KeyTable
mail._domainkey.example.com example.com:mail:/etc/opendkim/keys/example.com/mail.private

# Add signing entry
example.com mail._domainkey.example.com

# Restart service
sudo systemctl restart opendkim opendkim.service

OpenDMARC

OpenDMARC evaluates incoming email authenticity using SPF and DKIM. Helps prevent spoofing and phishing.

# Install OpenDMARC
sudo apt install opendmarc -y

# Configure OpenDMARC
sudo nano /etc/opendmarc.conf

AuthservID mail.example.com
TrustedAuthservIDs mail.example.com
Socket inet:8893@localhost

# Restart OpenDMARC
sudo systemctl restart opendmarc

Roundcube Webmail

Roundcube provides a full-featured webmail client for IMAP.

# Install Roundcube
sudo apt install roundcube roundcube-core roundcube-mysql -y

# Apache integration
sudo a2enconf roundcube
sudo systemctl reload apache2

# Set Roundcube config
sudo nano /etc/roundcube/config.inc.php

DNS Records for Email (MX, SPF, DKIM, DMARC)

Correct DNS configuration is mandatory for mail delivery. Most issues come from missing or incorrect DNS records.

# MX Record
example.com. 3600 IN MX 10 mail.example.com.

# SPF Record
example.com. IN TXT "v=spf1 mx a ip4:YOUR_SERVER_IP ~all"

# DKIM Record
mail._domainkey.example.com. IN TXT "v=DKIM1; k=rsa; p=PUBLIC_KEY_HERE"

# DMARC Record
_dmarc.example.com. IN TXT "v=DMARC1; p=quarantine; rua=mailto:admin@example.com"

Testing Email Delivery

These tests verify that mail successfully leaves the SMTP server and arrives at its destination.

# Send simple test email via Postfix
echo "Test body" | mail -s "Test Subject" user@example.com

# Debug email
echo -e "Subject: Test\nHello world" | sendmail -v user@example.com

# Test SMTP port
telnet mail.example.com 25

Email Troubleshooting & Logs

Logs provide clues when messages fail to send or authenticate.

# View mail log
sudo tail -f /var/log/mail.log

# Inspect queued messages
sudo postqueue -p

# View specific message
sudo postcat -q QUEUE_ID

# Check Dovecot authentication
sudo journalctl -u dovecot -n 50

# Check Postfix errors
sudo journalctl -u postfix -n 50

System Administration

Networking

Security

Web Stack

Database

Email Stack