Skip to main content

6.3 Operate and harden a Linux server

Four things you'll do on every Linux server for the rest of your career: install software, control services, read logs, and make it harder to break into. This lesson is all four, and then Docker in 6.4 assumes you have them.

Software: apt​

Ubuntu's package manager fetches software from repositories that are signed and maintained, which is why you almost never download an installer from a website on Linux.

# Refresh the catalogue of what's available. This changes nothing
# on your system; it just updates the list.
sudo apt update

# Install what's newer. -y answers yes to the confirmation.
sudo apt upgrade -y

# Install something specific.
sudo apt install -y htop

# What is this package, and is it installed?
apt policy htop

Run htop and press q to quit. You've just installed and used software without visiting a single website, which is the point.

sudo runs one command as the administrator, root. Linux's model is that you work as yourself and borrow privilege per command, and it's the same discipline as the two-account habit from lesson 5.6, enforced by the operating system rather than by your good intentions.

Services: systemd​

Anything that runs in the background is a service, managed by systemctl:

# Is SSH running, and did it start cleanly? Expect a line containing
# "active (running)". Press q to leave the pager.
systemctl status ssh

Three of the four verbs you will use constantly, and these are safe to run right now:

sudo systemctl restart ssh   # stop and start it again
sudo systemctl start ssh     # run it now
sudo systemctl enable ssh    # run it automatically at every boot

The fourth verb is stop, and do not run it on ssh while you are connected over SSH. It will do exactly what you asked, which is to shut down the service carrying your session, and you will be locked out until you walk over to the VM's console and start it again. The command is sudo systemctl stop <service> and you will use it plenty on other services. Learning that lesson on a machine in a datacentre you cannot drive to is a rite of passage worth skipping.

Run systemctl status ssh again after the restart. active (running) and a start time of a few seconds ago is how you know the restart did something.

enable versus start catches people out, so learn the difference now: start runs it right now, enable makes it come back after a reboot. A service that's started but not enabled works perfectly until the machine restarts at 3am, which is a memorable way to learn it.

Logs: journalctl​

When something fails, the answer is almost always in the logs, and systemd collects them all in one place:

# Everything this service has said, newest at the bottom.
journalctl -u ssh

# The last 50 lines, which is usually what you want.
journalctl -u ssh -n 50

# Follow live, like watching a log scroll. Ctrl+C to stop.
journalctl -u ssh -f

# Everything since boot, at error level or worse. Start here when
# a machine is misbehaving and you don't know where to look.
journalctl -b -p err

That last command is worth memorising. "What has gone wrong since this machine started" is the right first question, and it's one line.

Hardening, part one: keys instead of passwords​

Password authentication over SSH is the single most attacked thing on the internet. You saw what that looks like in lesson 2.2, in a log full of failed root logins. The fix is to stop accepting passwords at all.

An SSH key is a pair of files: a private key that never leaves your machine, and a public key you hand out. Anyone holding the public key can verify you without ever learning your secret.

On your own computer, not the server:

# Generate a key pair. ed25519 is the modern default: short, fast,
# and strong. -C is just a label so you can tell keys apart later.
ssh-keygen -t ed25519 -C "sam@laptop"

Accept the default location. It offers a passphrase, and you should set one: it encrypts the private key on disk, so a stolen laptop isn't a stolen server.

Now copy the public half to the server:

# Linux and macOS have a tool for exactly this.
ssh-copy-id sam@10.10.10.20
# Windows has no ssh-copy-id, so do it by hand. This reads your
# public key and appends it to the server's authorized_keys file.
type $env:USERPROFILE\.ssh\id_ed25519.pub | ssh sam@10.10.10.20 `
  "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

Test it before you go further. Open a new terminal and connect:

ssh sam@10.10.10.20

It should let you in without asking for your account password. (It may ask for your key's passphrase, which is a different thing.)

Keep the old session open

Do not close your working SSH session until key login is proven in a second one. The next step switches passwords off entirely, and if your key isn't working you'll have locked yourself out of a machine whose console you'd then have to open in the hypervisor. Two windows, always, when changing how you authenticate.

Now turn passwords off:

sudo nano /etc/ssh/sshd_config

Find and set these three, removing any leading #:

PasswordAuthentication no
PermitRootLogin no
PubkeyAuthentication yes

Then reload and test from your second window:

# Check the file for syntax errors BEFORE restarting. If this
# prints nothing, the config is valid.
sudo sshd -t

sudo systemctl restart ssh
Least privilege

PermitRootLogin no is the principle from lesson 5.6 arriving on Linux. Root can do anything, so nothing should be able to log in as root directly. You log in as yourself, then borrow root for one command at a time with sudo, which also means the logs record which human did it rather than "root did it."

Notice the shape repeating. In Module 5 it was two accounts. In lesson 7.2 it will be a machine that stays powered off. Here it's an account you can use but never log in as. Same question every time: what could somebody do if they were holding this, and can they hold less?

The firewall below is the network version of the same idea. Deny by default, then open only what you're actually serving.

Hardening, part two: the firewall​

Your server should answer only for things it's actually offering. Ubuntu's ufw is a friendly front end to the kernel's firewall:

# Allow SSH FIRST. Enabling a firewall that blocks your own
# connection is a classic, and it locks you out instantly.
sudo ufw allow OpenSSH

sudo ufw enable
sudo ufw status verbose

Read the status output: everything inbound is denied except what you allowed, everything outbound is permitted. That's the same default-deny shape your OPNsense firewall uses at the network edge, applied at the host. Defence in layers means the boundary isn't the only thing standing between an attacker and this machine.

You'll open more ports as you add services, and each time it should feel like a decision.

In GRC terms

The two changes you just made are among the most-audited controls in any framework. Key-based authentication with passwords disabled speaks to IA-2 (identification and authentication) and IA-5 (authenticator management); restricting who may connect and from where is AC-17, remote access. The host firewall is SC-7 again, one layer in from where you met it in lesson 4.6.

Notice what makes them assessable: sshd_config states the policy, ufw status shows the enforcement, and your journal records when and why you changed them. Configuration plus evidence plus rationale is what an assessor is asking for, and Module 16 will have you produce exactly that for this machine.