SSH Key Setup & Hardening: The Complete Guide

Every server exposed to the internet gets probed thousands of times a day by automated bots scanning for weak passwords. SSH key authentication eliminates that attack surface entirely — as long as you set it up correctly. This guide covers the full process from generating your key pair to hardening the server so that brute force attacks become irrelevant.

Why SSH Keys Are Better Than Passwords

Passwords can be guessed. Attackers run dictionary attacks — lists of common passwords — against port 22 around the clock on every public-facing server. No matter how complex your password is, it's still susceptible to a brute force attack given enough time.

SSH keys use asymmetric cryptography. You have a private key (kept secret on your machine) and a public key (uploaded to the server). The math behind Ed25519 keys means knowing the public key does not help derive the private key. Authentication happens through a cryptographic challenge-response that never transmits the private key over the network.

Step 1 — Generate the Key Pair

Which Algorithm?

Algorithm	Bits	Status
ed25519	256	✅ Recommended — fast, compact, modern
ecdsa	256/384/521	⚠️ Acceptable, but NIST curves have lingering controversy
rsa	4096	⚠️ Works but slower, larger keys, legacy
dsa	1024	❌ Deprecated — do not use

Generate the Key

ssh-keygen -t ed25519 -f ~/.ssh/my_key -C "your_email@example.com"

Flag	Value	Meaning
-t	ed25519	Algorithm type
-f	~/.ssh/my_key	File path for the private key
-C	"your_email@example.com"	Label — identification only, not used for auth

The -f flag lets you use a custom name instead of the default id_ed25519. This is useful when you have multiple keys for different servers.

The Passphrase

Enter passphrase (empty for no passphrase):
Enter same passphrase again:

Use a strong passphrase. This unlocks your local private key. Without a passphrase, anyone with file access to your private key can use it immediately. With a passphrase, they need to crack it first.

A good passphrase is long (20+ characters) but memorable. Think of a sentence: correct horse battery staple. Store it in your password manager.

What Was Created?

ls -la ~/.ssh/my_key*
# -rw------- 1 user user  464 Apr 26 04:30  ~/.ssh/my_key        ← private key (keep secret!)
# -rw-r--r-- 1 user user   93 Apr 26 04:30  ~/.ssh/my_key.pub   ← public key (upload this everywhere)

cat ~/.ssh/my_key.pub
# ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGPxk3H7aLwP4mV2x8Lw8ZvY3K2mH5sT9nL6qR2wX4yZ your_email@example.com

Step 2 — Copy the Public Key to the Server

Method 1: ssh-copy-id (Recommended)

ssh-copy-id -i ~/.ssh/my_key.pub user@your-server.com

ssh-copy-id handles everything automatically: creates ~/.ssh/, sets correct permissions, and appends your public key to ~/.ssh/authorized_keys. You'll still need your server password for this one-time step.

Method 2: Manual (When ssh-copy-id Isn't Available)

cat ~/.ssh/my_key.pub | ssh user@your-server.com "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

Command Part	Purpose
cat ~/.ssh/my_key.pub	Outputs the public key content
| ssh user@server	Pipes it over SSH to the remote server
mkdir -p ~/.ssh	Creates .ssh directory (-p = no error if exists)
chmod 700 ~/.ssh	Only you can read/write/execute the .ssh folder
cat >> ~/.ssh/authorized_keys	Appends the public key to the authorized keys file
chmod 600 ~/.ssh/authorized_keys	Only you can read/write the authorized keys file

Method 3: Overwrite (Clean Slate)

ssh user@your-server.com "mkdir -p ~/.ssh && chmod 700 ~/.ssh && echo '$(cat ~/.ssh/my_key.pub)' > ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

⚠️ Warning: This overwrites authorized_keys instead of appending. Use only if you have a single key and want a clean file.

Step 3 — Test Key-Based Login

Before disabling passwords, verify that key login works:

ssh -i ~/.ssh/my_key user@your-server.com

Add -v for verbose output if something goes wrong:

ssh -i ~/.ssh/my_key -v user@your-server.com

Look for these lines:

debug1: Authentications that can continue: publickey
debug1: Server accepts public key: /home/user/.ssh/my_key.pub

Using SSH Agent

The SSH agent holds your decrypted private key in memory so you only enter your passphrase once per session:

# Start the agent
eval "$(ssh-agent -s)"

# Add your key (enter passphrase once)
ssh-add ~/.ssh/my_key

# Now connecting won't ask for the passphrase
ssh user@your-server.com

Step 4 — Server-Side SSH Hardening

Edit /etc/ssh/sshd_config (Debian/Ubuntu) or /etc/sshd_config (RHEL/CentOS):

sudo nano /etc/ssh/sshd_config

⚠️ Keep your current SSH session open while testing. If you lock yourself out, the existing session stays active so you can fix it. Always run sudo sshd -t before restarting.

4.1 Disable Password Authentication

PasswordAuthentication no
PubkeyAuthentication yes

This tells the server: only accept key-based login. Reject any password attempt.

4.2 Disable Root Login

PermitRootLogin no

If you need root privileges, log in as a regular user then use sudo.

4.3 Change the Default Port

Port 2222

Port 22 is scanned constantly. Changing it dramatically reduces noise in your logs. Some networks (corporate firewalls) block non-standard ports — consider port 443 as a compromise.

4.4 Disable Empty Passwords

PermitEmptyPasswords no

4.5 Allow Specific Users Only

AllowUsers alice bob

Creates a whitelist. Anyone not in the list is rejected immediately.

4.6 Set Idle Timeout

ClientAliveInterval 300
ClientAliveCountMax 2

Sends a keepalive every 5 minutes and disconnects after 2 failed keepalives. Prevents stale sessions from staying open indefinitely.

4.7 Restrict Key Algorithms and Ciphers

HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key

Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com
KexAlgorithms curve25519-sha256
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com

Setting	Purpose
HostKey	DSA and ECDSA host keys removed — only Ed25519 and RSA offered
Ciphers	Removes old ciphers like aes128-cbc, 3des-cbc
KexAlgorithms	Removes weak key exchange like diffie-hellman-group14-sha1
MACs	Uses -etm (encrypt-then-mac) variants which are stronger

4.8 Restart SSH

sudo systemctl restart sshd

On older systems without systemd:

sudo service ssh restart

Step 5 — Client-Side SSH Config

On your local machine, create ~/.ssh/config for convenient connections:

Host myserver
    HostName your-server.com
    User alice
    Port 2222
    IdentityFile ~/.ssh/my_key
    AddKeysToAgent yes
    ServerAliveInterval 60
    ServerAliveCountMax 3

Setting	Value	What it does
HostName	your-server.com	The actual domain/IP to connect to
User	alice	Username to log in as
Port	2222	SSH port (non-default)
IdentityFile	~/.ssh/my_key	Which private key to use
AddKeysToAgent	yes	Automatically add key to agent on connect
ServerAliveInterval	60	Send keepalive every 60 seconds
ServerAliveCountMax	3	Disconnect after 3 failed keepalives

Now connect with just:

ssh myserver

Step 6 — Permissions (The #1 Reason SSH Breaks)

SSH is extremely picky about file permissions. Wrong permissions = silent auth failure.

Local Machine

chmod 700 ~/.ssh
chmod 600 ~/.ssh/my_key
chmod 644 ~/.ssh/my_key.pub
chmod 600 ~/.ssh/config

File	Permission	Why
~/.ssh/	700	Only you can read/write/execute
Private key (my_key)	600	Only you can read/write
Public key (my_key.pub)	644	World-readable (it's public info)
~/.ssh/config	600	Only you can read/write

Server Side

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

Step 7 — Install Fail2ban

Fail2ban monitors log files for repeated failed login attempts and bans the source IP automatically.

sudo apt update && sudo apt install fail2ban -y

Create a local override file so updates don't overwrite your settings:

sudo nano /etc/fail2ban/jail.local

[sshd]
enabled  = true
port     = 2222
maxretry = 3
bantime  = 3600
findtime = 600
logpath  = /var/log/auth.log

Setting	Value	Meaning
port	2222	Your actual SSH port
maxretry	3	Ban after 3 failed attempts
bantime	3600	Ban for 1 hour (seconds)
findtime	600	Within a 10-minute window
logpath	/var/log/auth.log	Where SSH login attempts are logged

sudo systemctl enable fail2ban
sudo systemctl start fail2ban

sudo fail2ban-client status

sudo fail2ban-client status sshd

⚠️ The PAM Password Expiry Gotcha

Even with SSH keys configured, your server's system password still matters. PAM (Pluggable Authentication Modules) checks password expiry independently of SSH pubkey auth. If a user's password expires, PAM will block login — even if the SSH key is perfectly valid.

This is a real trap: you set up key-only SSH, disable password auth, then months later your password expires and you're locked out with a valid key.

Fix — Set password to never expire for key-only accounts

# Check password expiry for a user
sudo chage -l username

# Set password to never expire
sudo chage -E -1 username

# Or set max days to unlimited
sudo chage -M -1 username

For service accounts or users who only use SSH keys, it's safe to disable password expiry. The SSH key is their real authentication method.

Why this matters for STIG/Compliance

If you're following security standards like STIGs (used in government/military environments), password age requirements are mandatory. But if you're using key-only SSH, you can exclude those users from password policy enforcement — just ensure the key auth path is properly locked down first.

What NOT to Do — Telnet

You'll occasionally see jokes online suggesting telnet as an alternative to SSH. Do not do this. Telnet transmits everything — including your username and password — in clear text, readable by anyone on the network. SSH encrypts the entire session.

🤡 liseklucian™ on TikTok: "Why SSH when you can use telnet? ;-)"
Answer: Because telnet sends your password in plaintext across the internet for anyone to sniff.

Additional Hardening — Restrict SSH by IP

If you have a static IP (home ISP or office), restrict SSH to your IP only:

# In /etc/ssh/sshd_config
AllowUsers alice@203.0.113.45 bob@198.51.100.0/24

This means only alice can log in from 203.0.113.45, and bob can log in from the 198.51.100.0/24 subnet. All other IPs are rejected before even seeing the auth prompt.

⚠️ Warning: If your ISP gives you a dynamic IP that changes, you'll lock yourself out. For dynamic IPs, consider a VPN (WireGuard) or allow a small range instead.

Additional Hardening — Disable Unused Services

Every running service is an attack surface. Check what's running:

# See all listening services
sudo ss -tlnp

# See what services are enabled at boot
systemctl list-unit-files --type=service --state=enabled

Common culprits to disable on a personal VPS:

# Disable if not needed
sudo systemctl disable --now nginx   # web server
sudo systemctl disable --now mysql   # database
sudo systemctl disable --now redis   # cache
sudo systemctl disable --now postfix # mail

Only keep running what you actually use. A minimal server has fewer things to exploit.

Additional Hardening — Kernel Hardening with sysctl

Kernel parameters control network and memory behavior. Edit /etc/sysctl.conf or drop files in /etc/sysctl.d/:

# Ignore ICMP redirects (prevents man-in-the-middle via ICMP)
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0

# Don't forward packets unless this is a router
net.ipv4.ip_forward = 0

# Enable TCP SYN cookies (prevents SYN flood attacks)
net.ipv4.tcp_syncookies = 1

# Restrict kernel dmesg output (hides kernel messages from non-root)
kernel.dmesg_restrict = 1

# Disable core dumps (prevents leaking sensitive data in crashes)
kernel.core_uses_pid = 1
fs.suid_dumpable = 0

Apply without reboot:

sudo sysctl -p

Additional Hardening — Lock Down sudo

Control which users can run privileged commands:

# Edit sudoers safely (validates before saving)
sudo visudo

# Example: alice can run all commands without password
alice ALL=(ALL) NOPASSWD: ALL

# Better: alice can only restart nginx
alice ALL=(ALL) NOPASSWD: /bin/systemctl restart nginx

Use NOPASSWD only for specific commands you need non-interactively (like service restarts in scripts). For interactive sessions, require the password.

Troubleshooting

"Connection refused" or "Connection timed out"

• Is the SSH service running? sudo systemctl status sshd
• Is the port correct? Check the Port setting in sshd_config
• Is a firewall blocking it? sudo ufw status or sudo iptables -L

"Permission denied (publickey)"

• Did you add the correct public key to ~/.ssh/authorized_keys on the server?
• Are permissions correct? (chmod 600 ~/.ssh/authorized_keys)
• Are you using the correct private key? Try ssh -v -i ~/.ssh/my_key ...
• Are you connecting as the correct user?

"Too many authentication failures"

The server rejected you because too many keys were offered. Fix by being explicit:

Host myserver
    IdentityFile ~/.ssh/my_key
    IdentitiesOnly yes

"Bad configuration options" after editing sshd_config

• Run sshd -t — it shows exactly which line has the error
• Make sure you're editing sshd_config (server-side), not ssh_config (client-side)

Locked yourself out

• Use the server console/KVM/IPMI/iDRAC — physical console bypasses network
• Ask your hosting provider for web-based VNC or serial console access
• This is why you always test sshd -t and keep an active session open

Quick Reference Checklist

Quick Reference Table

Setting	Recommended	Location
Key algorithm	ed25519	Client key generation
SSH port	2222 (custom)	/etc/ssh/sshd_config
Password auth	no	/etc/ssh/sshd_config
Root login	no	/etc/ssh/sshd_config
Empty passwords	no	/etc/ssh/sshd_config
Allowed users	specific usernames	/etc/ssh/sshd_config
Idle timeout	300s	/etc/ssh/sshd_config
~/.ssh/ perms	700	Server & client
authorized_keys perms	600	Server
Private key perms	600	Client
fail2ban	enabled	/etc/fail2ban/jail.local