Deep Dive into Linux SSH: From Protocol to Practice#

SSH (Secure Shell) is essential for every sysadmin and developer, yet many only know basic connection commands. Let’s explore the technical details that make SSH one of the most important tools in your arsenal.

The Three-Layer Protocol Architecture#

SSH isn’t just encrypted telnet. The SSH-2 protocol consists of three distinct layers:

Transport Layer: Handles server authentication, key exchange, encryption, and integrity protection. That “fingerprint verification” on first connect? That’s the transport layer protecting you from man-in-the-middle attacks.

Authentication Layer: Supports multiple methods - password, public key, GSSAPI, keyboard-interactive. Public key authentication is the gold standard: secure and automation-friendly.

Connection Layer: Multiplexes multiple logical channels over the encrypted tunnel. This enables remote commands, shell sessions, port forwarding, SFTP - all over a single connection.

Public Key Authentication: How It Actually Works#

The magic of public key auth lies in challenge-response:

  1. Client sends auth request with public key identifier
  2. Server checks ~/.ssh/authorized_keys for a match
  3. If found, server generates random challenge encrypted with the public key
  4. Client decrypts with private key, proving ownership without ever transmitting it

This is why private key leakage is catastrophic - it’s equivalent to having passwords for every authorized server.

# Generate ED25519 key (faster and more secure than RSA)
ssh-keygen -t ed25519 -C "your_email@example.com"

# Generate RSA key (better compatibility with old systems)
ssh-keygen -t rsa -b 4096 -C "your_email@example.com"

# Copy public key to remote server
ssh-copy-id -i ~/.ssh/id_ed25519.pub user@hostname

# Manual key installation (when ssh-copy-id isn't available)
cat ~/.ssh/id_ed25519.pub | ssh user@host "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"

Port Forwarding: Three Modes Demystified#

SSH port forwarding is powerful but often misunderstood. Here’s the breakdown:

Local Forwarding (-L): Map remote service to local port. Access internal databases through a bastion:

# Forward local 3306 to internal database through jump server
ssh -L 3306:db.internal:3306 user@jump-server

# Now connect locally
mysql -h 127.0.0.1 -P 3306 -u db_user -p

Remote Forwarding (-R): Expose local service to remote network. Let colleagues access your dev server:

# Make local port 3000 accessible on remote:8080
ssh -R 8080:localhost:3000 user@remote-server

# Others can now hit http://remote-server:8080

Dynamic Forwarding (-D): Create a SOCKS proxy. Route any SOCKS-capable traffic through the tunnel:

# Local SOCKS5 proxy on port 1080
ssh -D 1080 user@remote-server

# Configure browser: socks5://127.0.0.1:1080
# All traffic exits through remote server

The SSH Config File: Your Secret Weapon#

~/.ssh/config is overlooked but incredibly powerful. Proper configuration saves time and reduces errors:

# ~/.ssh/config

# Global defaults
Host *
    AddKeysToAgent yes
    IdentityFile ~/.ssh/id_ed25519
    ServerAliveInterval 60
    ServerAliveCountMax 3

# Production servers
Host prod-*
    User deploy
    ForwardAgent no
    StrictHostKeyChecking yes

# Development server with port forwarding
Host dev-server
    HostName 192.168.1.100
    User developer
    Port 2222
    LocalForward 3306 localhost:3306

# Jump host setup
Host jump
    HostName jump.company.com
    User admin

# Internal servers through jump
Host internal-*
    ProxyJump jump
    User deploy

Now ssh dev-server just works - no memorizing flags or addresses.

Security Hardening#

SSH misconfiguration causes countless breaches. Here’s the essential checklist:

Server side /etc/ssh/sshd_config:

# Force key-based auth only
PasswordAuthentication no
PubkeyAuthentication yes

# Disable root login
PermitRootLogin no

# Restrict allowed users
AllowUsers deploy admin

# Change default port (reduces brute force noise)
Port 2222

# Strong ciphers only
Ciphers aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com

Client security:

# Private key must be 600
chmod 600 ~/.ssh/id_ed25519

# Use ssh-agent (avoid keeping decrypted key in memory)
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519

# List loaded keys
ssh-add -l

Performance Optimization#

SSH can feel sluggish on high-latency networks. These settings help:

# Enable compression (good for slow links, text data)
ssh -C user@host

# Connection multiplexing (reuse existing tunnel)
# Add to ~/.ssh/config:
Host *
    ControlMaster auto
    ControlPath ~/.ssh/sockets/%r@%h-%p
    ControlPersist 600

# Disable unused features
ssh -o ForwardX11=no -o ForwardAgent=no user@host

Advanced Techniques# 

# Execute local script remotely without uploading
cat local-script.sh | ssh user@host 'bash -s'

# Persistent reverse tunnel (autossh is more reliable)
autossh -M 0 -f -N -R 2222:localhost:22 user@jump-server

# Multi-hop connection
ssh -J jump1,jump2 user@final-destination

# Debug connection issues
ssh -vvv user@host

# Generate signed certificate (enterprise setup)
ssh-keygen -s ca_key -I user@company -V +52w id_ed25519.pub

Troubleshooting Common Issues#

Connection refused: Check if sshd is running, verify port, check firewall rules.

Permission denied (publickey): Verify authorized_keys format (one key per line), permissions (600 for files, 700 for .ssh directory), and ownership. Use -v for details.

Host key verification failed: Server fingerprint changed. Could be server reinstall or MITM attack. After confirming safety: ssh-keygen -R hostname.

SSH appears simple but has tremendous depth. Master public key auth, learn the config file, and understand the three forwarding modes - these are the pillars of SSH expertise. The protocol’s elegance lies in providing strong security without sacrificing usability.

Related Tools: