nixcfg/docs/content/internal/deployment.md

# Infrastructure Deployment

The whole cluster infrastructure is build using [NixOS](https://nixos.org/).
The configuration repository is hosted at {{ config.repo_url }} and is deployed using [colmena](https://github.com/zhaofengli/colmena).

## Building the configuration
To build the configuration, as system with [Nix](https://nix.dev/install-nix) installed is required.

To activate the environment, run `nix develop` inside the configuration folder.
This will fetch all required build dependecies and makes them available in the environment.

Building the whole configuration is as easy as running:
```
colmana build --verbose --show-trace
```
*Go grap a coffee, this can take a while*

## Deploying
> Note: Deployment requires SSH access as the `root` user to all machines.

To deploy a configuration change or updates to the cluster, run the following command:
```
colmena apply switch
```

### Using the manager as a SSH jump host
SSH access to the nodes is limited.
Therefore it the manager system can be used as a jump host.
To do so, add the following lines to your local `~/.ssh/config` file (before the the `Host *` entry):
```
Host 10.32.47.1??
  IdentitiesOnly yes
  ProxyJump root@10.32.47.10
```

## Updating
Updating all systems can be done by running the following command in the configuration repository:
```
nix flake update
```

This will update all dependencies including the NixOS operation system.

After doing the update, the changed config (with the updated dependencies) must be [deployed](#deploying).

## Gather node information
The configuration repository relies on some information gathered from the machines itself.
After bootstrapping a machine, these information need to be gathered from the machines into the configuration repository.

To gather there data, run the following command:
```
./gather.sh
```

## Secret management
The config repository contains several secrets which are secured by [sops](https://github.com/getsops/sops) and the according [Nix integration](https://github.com/Mic92/sops-nix).

To edit a config file, run the following command:
```
sops <path/to/secrets/file>
```

This requires the editor to have its PGP-key fingerprint be part of the `adminKeys` list in `sops.nix`.

Altering the list requires one of the previous members to [update the keys](#update-keys).

### Update keys
Whenever a key, either the SSH key of a machine or the PGP key of an administrator, changes, the secret files need updating.
To do so, run the following command:
```
find \( -name "secrets.yaml" -or -path "*/secrets/**" -type f \) -exec sops updatekeys {} \;
```

## Bootstrapping a node
Compute nodes can be bootstrapped using PXE boot.
The manager will provide a touchless boot image which will install the node with the current deployment automatically.
Booting the node from PXE (network boot) is enough to activate the bootstrapping process.

After bootstrapping a node, make sure to [gather the node data](#gather-node-information) and [update the secret keys](#update-keys).