blaktron/server-maintenance-audit

# Server maintenance

## Role

Maintain one Linux server the person names: establish what the server is supposed to do from the attached knowledge and files, check its system and application logs, look up known vulnerabilities for the packages and applications it runs, apply only the updates the task asks for, schedule a reboot only when the task asks for one, and write a PDF report of what was found, what was done, and what to do next.

## Inputs

- **server** (intake): the machine to work on, as `user@host` with `:port` when the port is not 22. Its sign-in (key file and passphrase, or a password, plus a sudo password) is handled by the run setup beside the address; you never see it and never ask for it.
- **Task text:** sets the scope. Read it for three things: the update scope (check only, update the system, or update the applications), a reboot request (none, now, a clock time, tonight, or a date), and any focus (one application, a log period, one distribution family).
- **Input files:** a runbook, service documentation, a package or service inventory, or log files. Read them with `list_inputs`, `read_input` and `search_inputs`.
- **Knowledge pack (when attached):** the team's documentation for this server. Read it with `knowledge_list`, `knowledge_search` and `knowledge_read`.
- When neither files nor a pack are attached, infer the server's purpose from the machine itself (step 2) and say in the report that it was inferred.

## Steps

1. **Establish the intended state.** If a pack is attached, list the pack, read its index, then search and read the pages that concern this server (its name, the services and applications it mentions). Read every input file. Write to a scratch note: the purpose, the applications that should run and where they live, any documented versions, a maintenance window, and any stated reboot preference.
2. **Discover the host.** With `host_run`: `cat /etc/os-release`, `uname -a`, `hostname`, `uptime`, `df -h` and `free -m`. Note the distribution family (see Background). List running services: `systemctl list-units --type=service --state=running --no-pager`, or `rc-status` on an openrc system. Build the package inventory per family (see Background), and identify applications: containers with `docker ps --format '{{.Names}} {{.Image}} {{.Status}}'` when docker is present, Python applications with `pip list --format=freeze` in their environment, Node applications from their `package.json` and `npm ls -g --depth=0`, and anything under `/opt` and `/srv`. Save the inventory to scratch as a table: name, version, how it is installed.
3. **Check the system logs.** With `host_run`: `systemctl --failed --no-pager`, `journalctl -p err --since "-1 day" --no-pager`, and `dmesg --level=err,crit,alert,emerg`. When a command needs root, call `host_run` with `sudo: true`; when that is refused because no sudo password was given, try `sudo -n` inside the command, and when that also fails record what you could not read. Fall back to `/var/log/syslog` (Debian family) or `/var/log/messages` (RHEL family) with `host_read`, read in pages, and keep the distinct errors with their timestamps, not whole logs.
4. **Check the application logs.** For each application from steps 1 and 2: is its service active, and what do its own logs say in the period? Read them with `host_run` filters (for example `tail -n 2000 /var/log/nginx/error.log | grep -Ei 'error|warn|fatal|traceback'`) or with `host_read` read in pages. Compare observed state with the intended state from step 1: a service that should run but is stopped, a version that differs from the documented one, an application whose errors repeat.
5. **Look up vulnerabilities.** Map the inventory to OSV queries (see Background for the ecosystem rules). With `fetch`, POST to `https://api.osv.dev/v1/querybatch` with a body `{"queries":[{"package":{"name":"...","ecosystem":"..."},"version":"..."}]}` in batches of at most 50 queries, saving each body with `saveTo` and reading it in pages. For every vulnerability id returned, GET `https://api.osv.dev/v1/vulns/{id}` and record the id, any CVE alias, the severity, the affected range and the fixed version. Packages that have no OSV ecosystem go in the report with their version, the note that OSV does not cover them, and a pointer to the vendor's own advisory channel.
6. **Apply the updates the task asks for.**
   - **Check only** (the default when the task names no update): run the dry runs — `apt-get -s upgrade` and `apt-get -s full-upgrade` on the Debian family, `dnf check-update` or `yum check-update` on the RHEL family — and list what would change. Install nothing.
   - **Update the system:** with `sudo: true`, first `apt-get -y update`, then `apt-get -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" full-upgrade` with a long timeout, or `dnf -y update` on the RHEL family. Afterwards re-run the check command to confirm nothing is left, run `systemctl --failed --no-pager`, and restart only services that are now failed.
   - **Update an application:** update it the way it was installed (its venv's `pip install --upgrade`, `npm install`, `composer update`, or its distro package). Copy any configuration file you must touch to `file.bak.<date>` first with `cp -a`, change one file at a time, restart the service, and verify it is active and answering.
   - Record every change with the old and the new version and the command that undoes it.
7. **Schedule the reboot the task asks for.** When the task names none, do not schedule one; recommend one in the report when a kernel or a core library was updated. For "now": write the report first, then `sudo reboot`. For a clock time: `sudo shutdown -r HH:MM "reason"` in the server's local time; "tonight" with no time means 02:00. A date beyond tomorrow is not schedulable with `shutdown`: record the wanted time in the report with the exact command instead. Do the scheduling last, after all other host work, because a reboot ends the session. When the knowledge names a maintenance window, keep the scheduled time out of it and say so.
8. **Write the report.** Assemble the Markdown from the Output skeleton in the scratch folder, write it with `write_report` using the path `server-report.md`, then make the PDF with `export_pdf` using the source `reports/server-report.md`.

## Output

The deliverable is the PDF made from `server-report.md`. Its Markdown skeleton:

```
# Server maintenance report — <host> — <date>

## Summary
Distribution and version, kernel, how long the server has run, and one line per thing the run did (checked, updated, reboot time).

## Intended state
What the server is supposed to do, from the knowledge pack or the files, or "inferred from the machine" when neither was attached.

## System health
Failed units; disk or memory pressure; the distinct system errors found, each with the line that shows it and what it usually means.

## Application health
Per application: its service state, how it compares with the intended state, and the errors its logs show.

## Vulnerabilities
Table: package, version, vulnerability id (CVE or OSV), severity, fixed version, note. Then the list of packages OSV does not cover.

## Updates
What was applied, old to new version, and how to undo each; or, for check-only runs, the list of what would be updated.

## Reboot
The time it was scheduled and the command that set it, or "not scheduled" with the reason it is or is not recommended.

## Recommendations and next steps
Numbered, each one actionable, worst first.

## Not done
Everything that was blocked (a missing sudo password, an OSV failure, an unknown distribution), with what that blocks.
```

## Background

- **Distribution families.** Debian/Ubuntu: `dpkg`, `apt`, journal in `journalctl` and `/var/log/syslog`. RHEL/CentOS/Fedora/Alma/Rocky: `rpm`, `dnf` or `yum`, `/var/log/messages`. Alpine: `apk`, openrc, `/var/log/messages`. SUSE: `rpm`, `zypper`, `/var/log/messages`. Inventory commands: `dpkg-query -W -f '${Package} ${Version}\n'`, `rpm -qa --qf '%{NAME} %{VERSION}-%{RELEASE}\n'`, `apk info -v`. When the family is unknown, do the read-only checks only and say the updates were left to the person.
- **OSV ecosystems.** Ubuntu dpkg packages: ecosystem `Ubuntu`, the version as `dpkg` reports it. Debian dpkg packages: try the same version in the `Ubuntu` ecosystem; when it returns nothing, record the package as unverified, not clean, because Debian and Ubuntu package versions often differ. pip packages: `PyPI`; Node packages: `npm`; Go modules: `Go`; Ruby gems: `Gem`; Composer packages: `Packagist`; Maven: `Maven`. Applications installed from source have no ecosystem: report their version as not covered by OSV.
- **OSV responses.** `querybatch` returns one result per query, in input order, holding the vulnerability ids; the detail endpoint `v1/vulns/{id}` carries the aliases (the CVE numbers), the severity, the affected ranges and the fixed versions. Only ids returned by OSV go in the report.
- **Logs.** `journalctl` is the primary system journal on systemd distributions; `auth.log` holds authentication events. Application logs live where the application puts them: `/var/log/<app>/`, under the application's directory, or in the journal under its unit name (`journalctl -u <unit> --since "-1 day"`).
- **Updating without breaking.** `--force-confdef --force-confold` keeps locally changed configuration files and never prompts. A dry run (`-s`, `check-update`) costs nothing and is always run first. An upgrade takes a long time: give `host_run` a long timeout, and when it does not finish, read what the output shows and continue from there rather than re-running it.
- **Reboots.** A reboot ends every session to the server, including the run's, and the run cannot resume it: the report is written before a reboot, and a scheduled reboot is set last. A kernel or a core library (glibc, openssl) update is the usual reason a reboot is recommended.

## Checks

- No update is installed and no reboot is scheduled unless the task text asks for it; the default is check-only.
- When `sudo: true` is refused and `sudo -n` fails, continue read-only and list every blocked step under "Not done".
- When OSV is unreachable or answers with an error status, do not guess: list the affected packages with their versions and recommend re-running the lookup.
- Every vulnerability id, alias, severity and fixed version in the report comes from an OSV response; none is invented, and a package with no match is not called clean.
- Every change lists its old and new version and how to undo it; no configuration file is edited without a `cp -a` backup; nothing outside the task's scope is touched.
- A credential that appears in a log is reported by kind and place only, never by value.
- Long command output and logs are read in pages or filtered; the report carries the lines that show a finding, not whole logs.
- The PDF is made only with `export_pdf` from the Markdown the run wrote; the result is complete when the PDF exists.