Pipeline

This guide explains how to set up a GitLab pipeline for your Brezel instance. The pipeline will build and deploy your instance to one- or multiple-available deployment targets:

• An Ubuntu/Debian virtual server, using SSH over Tailscale
• An Ubuntu/Debian virtual server, using direct SSH or a customer VPN
• Docker
• Kubernetes

Virtual Server over Tailscale (preferred)

This is the preferred deployment method for customer servers. Each deployment job creates its own ephemeral Tailscale node, connects to the server with normal OpenSSH, and is automatically removed from the tailnet shortly after the job.

The pipeline uses the official Tailscale container as a GitLab service with userspace networking and a SOCKS5 proxy. It does not require /dev/net/tun, privileged containers, additional Linux capabilities, or changes to the GitLab Docker runner.

Prerequisites

• Complete Server Access with Tailscale for the target server.
• The server has tag:server and tag:<SLUG>-server.
• The tailnet policy grants tag:<SLUG>-ci access to tag:<SLUG>-server on TCP port 22.
• The shared tag:ci identity can query tag:dns on TCP and UDP port 53.
• The brezel user can authenticate with a normal OpenSSH key.

Tailscale provides the network path only. The deployment key still controls SSH authentication.

The slug, hostname, and tags are different values

For a deployment with the slug foo, use:

Purpose	Value	Description
Slug	foo	used in the tailscale ACL
Server’s Tailscale hostname	foo-prod	The name of the target machine in tailscale. Normally the same as <SLUG>
Common server tag	tag:server
Server-specific tag	tag:foo-server
Common CI tag	tag:ci
Server-specific CI tag	tag:foo-ci
GitLab PROD_HOST	foo-prod
GitLab TS_CI_TAGS	tag:ci,tag:foo-ci

There is no tag:foo or tag:foo-prod in this naming model. The hostname identifies the machine, while tags provide the identities used by Tailscale grants.

1. Create the deployment SSH key

Generate a dedicated key pair on a trusted machine:

ssh-keygen -t ed25519 -C "<SLUG>@GitLab" -f gitlab_tailscale

Leave the passphrase empty because the non-interactive CI job cannot enter it. Append the public key to the brezel user’s authorized keys on the server:

cat gitlab_tailscale.pub | ssh brezel@<TAILSCALE_HOSTNAME> 'cat >> ~/.ssh/authorized_keys'

Verify the login before configuring GitLab:

ssh -i gitlab_tailscale brezel@<TAILSCALE_HOSTNAME>

2. Record the SSH host key

The Tailscale pipeline requires host-key verification. From a trusted machine that can reach the server, collect its public host key:

ssh-keyscan -t ed25519 <TAILSCALE_HOSTNAME> > gitlab_known_hosts

Verify its fingerprint against the server before using it. Obtain the server-side fingerprint through the provider console or an existing, independently verified administration session:

ssh-keygen -lf gitlab_known_hosts
sudo ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub

The fingerprints must match. Do not replace this with StrictHostKeyChecking no.

3. Create a customer-specific OAuth client

In the Tailscale admin console, open Settings > Trust credentials, create an OAuth client, and grant it:

• write access to Auth keys
• tag:ci
• tag:<SLUG>-ci

Create a separate OAuth client for each customer/tag scope. Do not authorize a shared client for every tag:<SLUG>-ci tag.

The pipeline passes the OAuth client secret directly to Tailscale’s supported auth-key flow. For every deployment job, Tailscale creates a pre-approved ephemeral node with both CI tags and a unique hostname based on the GitLab project and job IDs.

This is preferable to storing a reusable auth key:

• the OAuth client remains narrowly scoped to two tags
• each job receives a fresh node identity
• no Tailscale state is persisted by the runner
• interrupted jobs still disappear automatically as ephemeral nodes

Copy the generated client secret immediately. Tailscale does not show it again.

4. Configure GitLab variables

In the Brezel repository, open Settings > CI/CD > Variables and configure:

Key	Example value	GitLab settings
TS_OAUTH_CLIENT_SECRET	tskey-client-...	Masked and hidden, protected, expansion disabled
TS_CI_TAGS	tag:ci,tag:<SLUG>-ci	Protected
PRIVATE_KEY	Contents of gitlab_tailscale	File variable, protected
SSH_KNOWN_HOSTS	Contents of gitlab_known_hosts	File variable, protected
PROD_HOST	<TAILSCALE_HOSTNAME>	Protected
PROD_PORT	22	Protected
PROD_USER	brezel	Protected
PROD_PATH	/var/www/vhosts/<INSTANCE>	Protected
PROD_API_URL	https://<API_DOMAIN>	Protected
PROD_SYSTEM	<SYSTEM>	Protected

TS_CI_TAGS must contain exactly the shared CI tag followed by the server-specific CI tag, for example:

tag:ci,tag:wrs-scan-ci

Use the server’s Tailscale/MagicDNS hostname for PROD_HOST, for example wrs-scan-prod. The userspace SOCKS5 proxy resolves that name through Tailscale, so the deployment container does not need a kernel route or custom system resolver.

For staging deployments, also configure:

Key	Example value
STAGING_HOST	Staging server’s Tailscale hostname
STAGING_PORT	22
STAGING_USER	brezel
STAGING_PATH	Staging deployment path
STAGING_API_URL	Staging API URL
STAGING_SYSTEM	Staging system name

If staging uses a separate host, append its independently verified host key to the same gitlab_known_hosts file before creating SSH_KNOWN_HOSTS:

ssh-keyscan -t ed25519 <STAGING_TAILSCALE_HOSTNAME> >> gitlab_known_hosts

The same customer-specific OAuth client can be used only when the staging server belongs to the same customer/tag scope and carries a destination tag allowed by tag:<SLUG>-ci.

Protect the main branch and, when used, the staging branch under Settings > Repository > Protected branches. The Tailscale deployment jobs intentionally do not run for unprotected refs.

Protect the deployment credentials

Run deployment jobs only from protected branches. Mask and hide the OAuth secret, disable variable expansion for it, and never expose protected variables to untrusted merge-request or fork pipelines.

5. Enable the Tailscale pipeline

Create .gitlab-ci.yml from the provided example if necessary:

cp .gitlab-ci.yml.example .gitlab-ci.yml

Include the Tailscale variant:

.gitlab-ci.yml

include:
  - local: 'gitlab/.tailscale.gitlab-ci.yml'

Do not include gitlab/.ssh.gitlab-ci.yml separately. The Tailscale variant imports its shared build and deployment definitions itself and disables the direct and legacy VPN jobs.

Commit and push .gitlab-ci.yml. A production pipeline on the protected main branch runs:

1. build frontend
2. deploy over tailscale

The deployment job uses one ephemeral Tailscale node and deploys the backend and matching frontend artifact while holding an environment-specific deployment lock. The backend checks out the exact CI_COMMIT_SHA associated with the pipeline instead of whichever commit happens to be the current branch head.

The template waits for Tailscale’s health endpoint and verifies a real SSH connection before deploying.

Troubleshooting Tailscale jobs

Tailscale does not become ready

• Confirm TS_OAUTH_CLIENT_SECRET is available to the protected branch.
• Confirm the OAuth client has Auth keys write permission.
• Confirm it is authorized for both tags listed in TS_CI_TAGS.
• Confirm the tag names exist under tagOwners in the tailnet policy.

Tailscale is ready, but SSH fails

• Confirm the CI-to-server grant permits tag:<SLUG>-ci to reach tag:<SLUG>-server on TCP port 22.
• Confirm PROD_HOST is the server’s Tailscale/MagicDNS hostname.
• Confirm PROD_PORT, PROD_USER, and the deployment private key are correct.
• Confirm SSH_KNOWN_HOSTS contains the host key under the same hostname used by PROD_HOST.

Host-key verification fails

Re-check the server’s host-key fingerprint. Update SSH_KNOWN_HOSTS only after confirming that the server was intentionally rebuilt or its SSH host keys were deliberately rotated.

Virtual Server using direct SSH or a customer VPN

This section assumes you have access to a brezel user via SSH on a virtual server.

1. Generate an SSH keypair

On your local Linux/Unix machine, generate an RSA key pair:

ssh-keygen -b 4096 -C "example@GitLab"

This will output something like:

Generating public/private rsa key pair.
Enter file in which to save the key (/users/saul/.ssh/id_rsa):

Enter the path of the key. The name could be gitlab_rsa.

Enter passphrase (empty for no passphrase):

No passphrase. Hit enter.

Enter same passphrase again:

Hit enter. It will then output something like this:

Your identification has been saved in /users/saul/.ssh/gitlab_rsa.
Your public key has been saved in /users/saul/.ssh/gitlab_rsa.pub.
The key fingerprint is:
SHA256:hcsvjIXxM8ZS16X3IHIoGFwPSHr5Cjw35HOIZMEIW1c example@GitLab
The key's randomart image is:
+---[RSA 4096]----+
|...oo+Eoo     .  |
| o...oo+ + o o   |
|.   + * + * = o  |
|   + = X = o o o |
|    = O S       .|
|     + @ +       |
|      o o .      |
|         .       |
|                 |
+----[SHA256]-----+

Authorize the brezel user to access the virtual server via the private key. Login:

ssh brezel@brezel.example.io

Copy the contents of gitlab_rsa.pub and append it to ~/.ssh/authorized_keys:

echo "<PUBLIC KEY>" >> ~/.ssh/authorized_keys

2. Configure GitLab

In GitLab, go to the instance repository, to Settings > CI/CD > Variables and click on Add variable.

Key	Value	Protect variable
PRIVATE_KEY	Contents of your gitlab_rsa private key	✅
PROD_HOST	brezel.example.io	✅
PROD_PORT	22 (222 on Hetzner managed servers)	✅
PROD_USER	brezel	✅
PROD_PATH	/var/www/vhosts/brezel.example.io	✅
VITE_APP_API_URL	https://api.example.brezel.io	✅
VITE_APP_SYSTEM	example	✅

Make sure to add an empty newline at the end of PRIVATE_KEY!

Otherwise the pipline will fail as it pipes the value directly into a key file which needs a newline at the end.

Ensure that your pipelines can see the variables

They are protected by default, so either protect your branch as well or unprotect the variables (not recommended).

Legacy alternative: customer VPN

Use this only when the server cannot be onboarded to Tailscale. If your server is behind a traditional VPN, add the VPN connection information to the pipeline. A connection is established using Gluetun You do this by adding special environment variables to the GitLab CI/CD settings:

Key	Value	Description
USE_VPN	”true” or “false”	If your Pipeline should use a VPN during the deployment steps. Set to “true” in order to enable the below variables and use the VPN
VPN_TYPE	wireguard or openvpn	The VPN type to use
WIREGUARD_CONFIG_CONTENT	WireGuard raw configuration content (if using WireGuard)	The raw content of your wireguard config file
OPENVPN_USER	OpenVPN username (if using OpenVPN)	OpenVPN username
OPENVPN_PASSWORD	OpenVPN password (if using OpenVPN)	OpenVPN password
OPENVPN_CONFIG_CONTENT	OpenVPN raw configuration content (if using OpenVPN)	The raw content of your .ovpn file

3. Configure your .gitlab-ci.yml

In your repository, there should be a .gitlab-ci.yml.example file. Create the .gitlab-ci.yml file:

cp .gitlab-ci.yml.example .gitlab-ci.yml

Add the file to git:

git add .gitlab-ci.yml

In your .gitlab-ci.yml, uncomment the SSH deployment line:

include:
# - local: 'gitlab/.Docker.gitlab-ci.yml'  # Uncomment this for Docker deployment
  - local: 'gitlab/.ssh.gitlab-ci.yml'     # Uncomment this for SSH deployment

Special case: Plesk

If your deployment target is a server managed by Plesk, then PHP is under a different path. Go to gitlab/.ssh.gitlab-ci.yml and change all occurrences of php in the script: sections to

/opt/plesk/php/7.4/bin/php ...

Keep in mind to also change the paths of composer. If you downloaded composer to ~/bin/composer, the lines should be

/opt/plesk/php/7.4/bin/php ~/bin/composer ...

Kubernetes Deployment with Terraform (default)

This section explains how to deploy the Brezel instance to a Kubernetes cluster using Terraform.

Prerequisites

Before you begin, ensure you have the following:

1. You have pushed the package-lock.json and composer.lock files to your repository. This is important for the pipeline to work correctly.
2. A Kubernetes cluster with sufficient resources
3. A Scaleway account (if you’re using Scaleway as your cloud provider)
4. Required environment variables or secrets for your Brezel instance

Terraform Configuration

The project includes Terraform configurations in the terraform directory. These configurations use the brezel-instance/kubernetes//brezel-instance module to deploy Brezel to a Kubernetes cluster. The deployed Brezel instance will be available under

• {branch_slug}.{system}.{base_domain} for the web application
• api.{branch_slug}.{system}.{base_domain} for the API
• brotcast.{branch_slug}.{system}.{base_domain} for the Brotcast service
• pma.{branch_slug}.{system}.{base_domain} for PhpMyAdmin (if enabled)

E.g., for the system “acme” in the Git branch “staging” and the base domain “review.brezel.cloud”, the URLs would be:

• staging.acme.review.brezel.cloud for the web application
• api.staging.acme.review.brezel.cloud for the API
• brotcast.staging.acme.review.brezel.cloud for the Brotcast service
• pma.staging.acme.review.brezel.cloud for PhpMyAdmin (if enabled)

The Docker images will be pulled from

• {registry_image}:{branch_slug}

Key Files

• gitlab-ci.yml: The GitLab CI/CD configuration file
• main.tf: Contains the main Terraform configuration for deploying Brezel
• variables.tf: Defines the variables used in the Terraform configuration
• backend.tf: Configures the Terraform backend for state storage
• scaleway.auto.tfvars: Contains variables specific to Scaleway (if you’re using Scaleway)

Deployment Steps

1. Build and Push Docker Images

In your .gitlab-ci.yml, uncomment the Terraform deployment section in the .gitlab-ci.yml file so that it looks like this:

include:
  - local: 'gitlab/.terraform.gitlab-ci.yml'

variables:
  SYSTEM: "example"  # Replace it with your system name

2. Configure Variables

You can configure variables in the GitLab CI/CD settings for your project (Settings > CI/CD > Variables).

Required variables:

• TF_VAR_vapid_public_key: The VAPID public key used for push notifications.
• TF_VAR_vapid_private_key: The VAPID private key used for push notifications.

You must generate them locally via php bakery init, copy them from your .env file, and copy them to the GitLab CI/CD variables.

Optional variables:

• TF_VAR_namespace: Override the namespace for the Kubernetes resources. If not set, it will be generated using the review template review-{system}-{branch_slug}.
• TF_VAR_host: Override the base hostname for the application. If not set, it will be generated using the review template {branch_slug}.{system}.{base_domain}.
• TF_VAR_registry_image: The container registry URL where the Docker images are stored. This is filled automatically by the GitLab pipeline.
• TF_VAR_base_domain: The base domain for the application. Projects under https://gitlab.kiwis-and-brownies.de/kibro/basedonbrezel/ have this set automatically.
• TF_VAR_scw_rdb_instance_id: The Scaleway RDB instance ID for the database. Projects under https://gitlab.kiwis-and-brownies.de/kibro/basedonbrezel/ have this set automatically.
• TF_VAR_app_env: The environment (e.g., production, staging). Default is staging.
• TF_VAR_secure: Whether to issue a certificate for the domain. Default is true.
• TF_VAR_mail_host: The SMTP server used for sending emails.
• TF_VAR_mail_port: The SMTP server port.
• TF_VAR_mail_username: The SMTP username.
• TF_VAR_mail_password: The SMTP password.
• TF_VAR_mail_encryption: The encryption method used for the SMTP server (e.g., ssl, tls). Default is tls.
• TF_VAR_mail_from_address: The email address used for sending emails.
• TF_VAR_mail_from_name: The name used for sending emails.
• TF_VAR_vapid_public_key: The VAPID public key used for push notifications.
• TF_VAR_vapid_private_key: The VAPID private key used for push notifications.

You can also set these in variable files (e.g., terraform.tfvars or scaleway.auto.tfvars), where you don’t need to set the TF_VAR_ prefix for variable names.

Tip

In production, you typically don’t want a hostname like main.example.review.brezel.cloud. In the GitLab CI/CD settings, configure these variables to apply to your specific production environment:

• TF_VAR_host: example.com
• TF_VAR_namespace: example

Deployment from Local Machine

3. Initialize Terraform

Set required environment variables for initialization. You can do this in your shell or in a .env file under the terraform directory. The required variables for Terraform initialization are:

• GITLAB_USERNAME: GitLab username
• GITLAB_ACCESS_TOKEN: GitLab password or personal access token

Then, navigate to the terraform directory and initialize Terraform:

cd terraform
./init.sh  # or terraform init if you're not using the provided script

4. Plan the Deployment

Generate and review the Terraform execution plan:

terraform plan -out=tfplan

5. Apply the Configuration

Apply the Terraform configuration to create or update the infrastructure:

terraform apply tfplan