Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Kennel

Kennel is the deployment platform for ScottyLabs. When you push code to a repository, kennel builds it with Nix, deploys it, and routes traffic to it.

What it does

  • Builds your project with Nix when you push to any branch
  • Deploys services as systemd units and static sites via Caddy
  • Provisions per-deployment databases (PostgreSQL), caches (Valkey), and object storage (Garage)
  • Resolves secrets from OpenBao via secretspec
  • Generates HTTPS URLs for every deployment, including PR previews
  • Tears down deployments and their resources when branches are deleted or PRs are closed

How it works

Your project’s devenv.nix declares what it needs to run: processes, databases, secrets, static sites. Kennel evaluates this configuration, builds the Nix packages, and deploys everything with isolated resources per branch.

Every deployment gets a URL at {project}-{branch}.scottylabs.net. Production deployments can also have custom domains.

Getting started

See the enabling features and deploying a project guides.

Enabling Features

In the ScottyLabs governance repository, add your project to its team’s TOML file and list its features. Governance provisions everything those features imply:

  • kennel provisions the webhook that connects your repository to kennel for builds and deployments
  • sentry creates a Sentry project and writes its DSN to Vault as SENTRY_DSN on the prod profile
  • posthog creates a PostHog project and writes its key and host to Vault as POSTHOG_KEY and POSTHOG_HOST on the prod profile
  • cdn provisions a per-project public-read Garage bucket and writes CDN_S3_ENDPOINT, CDN_S3_BUCKET, CDN_ACCESS_KEY_ID, CDN_SECRET_ACCESS_KEY, and CDN_PUBLIC_URL to Vault on the prod profile
  • oidc_client provisions prod, staging, and dev Keycloak OIDC clients and writes OIDC_CLIENT_ID, OIDC_CLIENT_SECRET, KEYCLOAK_URL, KEYCLOAK_REALM, OAUTH_RELAY_URL, PROJECT_GROUP, and PROJECT_ADMIN_GROUP to Vault for each profile
  • admin_client provisions a Keycloak service-account client with the view-users, manage-users, and view-identity-providers roles, written to Vault as KEYCLOAK_ADMIN_CLIENT_ID and KEYCLOAK_ADMIN_CLIENT_SECRET

OIDC

For OIDC authentication, note the distinction between the OAuth relay (scottylabs.ricochet.enable in devenv) and the Keycloak OIDC client (oidc_client in governance). The OAuth relay is used to relay between the standardized authorized redirect URI used by the OIDC client.

  • prod uses the standard <name> OIDC client, preview and staging share <name>-staging, and dev uses <name>-dev
  • prod, staging, and preview all share the https://oauth.scottylabs.org/oauth2/callback (Ricochet) relay, while dev uses the http://localhost:8090/oauth2/callback (local) relay that requires scottylabs.ricochet.enable to be enabled in devenv.

Your service receives the client credentials, Keycloak connection settings, and project group paths as env vars, declared in your secretspec.toml:

[profiles.default]
OIDC_CLIENT_ID = { description = "Keycloak OIDC client ID" }
OIDC_CLIENT_SECRET = { description = "Keycloak OIDC client secret" }
KEYCLOAK_URL = { description = "Keycloak base URL" }
KEYCLOAK_REALM = { description = "Keycloak realm" }
OAUTH_RELAY_URL = { description = "OAuth relay callback URL" }
PROJECT_GROUP = { description = "Keycloak project group path" }
PROJECT_ADMIN_GROUP = { description = "Keycloak project admin group path" }

# Declare all profiles, even if you only use default
[profiles.prod]

[profiles.staging]

[profiles.preview]

[profiles.dev]

Your service builds its own callback URL from APP_URL and the relay forwards the authorization code there after login. Set it with scottylabs.ricochet.appUrl in local development; in deployments kennel injects APP_URL from the domain (see runtime environment).

The OIDC client also adds the user’s group memberships to the token as a groups claim, in full-path form like /projects/<slug>. Match it against PROJECT_GROUP and PROJECT_ADMIN_GROUP to grant enhanced permissions.

Sentry

If your service reports errors to Sentry, add sentry to its features in governance. It creates a Sentry project and writes the project DSN to Vault as SENTRY_DSN on the prod profile, so declare it there:

[profiles.prod]
SENTRY_DSN = { description = "Sentry project DSN" }

Read SENTRY_DSN at startup and pass it to the Sentry SDK to turn on error reporting, following Sentry’s setup guide. Without a DSN, in local development and previews, the SDK stays inert.

PostHog

If your service sends product analytics to PostHog, add posthog to its features in governance. It creates a PostHog project and writes the project key and host to Vault as POSTHOG_KEY and POSTHOG_HOST on the prod profile, so declare them there:

[profiles.prod]
POSTHOG_KEY = { description = "PostHog project API key" }
POSTHOG_HOST = { description = "PostHog instance host" }

Read them at startup and pass them to the PostHog SDK to turn on analytics. Without a key, in local development and previews, the SDK stays inert.

CDN

If your service serves static assets from object storage, add cdn to its features in governance. It provisions a per-project Garage bucket with website mode enabled and a scoped access key, then writes the S3 credentials and public URL to Vault on the prod profile, so declare them there:

[profiles.prod]
CDN_S3_ENDPOINT = { description = "Garage S3 endpoint" }
CDN_S3_BUCKET = { description = "CDN bucket name" }
CDN_ACCESS_KEY_ID = { description = "CDN bucket access key ID" }
CDN_SECRET_ACCESS_KEY = { description = "CDN bucket secret access key" }
CDN_PUBLIC_URL = { description = "Public base URL for the bucket" }

Upload assets with the S3 credentials; they are publicly readable under CDN_PUBLIC_URL (https://cdn.scottylabs.org/<repo>/).

Documentation

Documentation is controlled separately by docs (boolean, default true), which aggregates the repository’s ./docs directory into the documentation hub.

Deploying a Project

This guide walks through setting up a ScottyLabs project for deployment with kennel.

Prerequisites

  • A project registed in governance with an associated Codeberg repository
  • devenv and direnv installed locally

1. Import the shared module

Add the ScottyLabs devenv input to your devenv.yaml:

secretspec:
  enable: true
  provider: vault://secrets2.scottylabs.org/secret
  profile: dev

inputs:
  nixpkgs:
    url: github:nixos/nixpkgs/nixos-unstable
  scottylabs:
    url: git+https://codeberg.org/ScottyLabs/devenv
    inputs:
      nixpkgs:
        follows: nixpkgs
  rust-overlay:
    url: github:oxalica/rust-overlay
    inputs:
      nixpkgs:
        follows: nixpkgs
  treefmt-nix:
    url: github:numtide/treefmt-nix
  git-hooks:
    url: github:cachix/git-hooks.nix
    inputs:
      nixpkgs:
        follows: nixpkgs

This nixpkgs pin must match the one in flake.nix below, so the shell and the build resolve identical packages.

Import it in your devenv.nix:

{ pkgs, config, inputs, ... }:
{
  imports = [ inputs.scottylabs.devenvModules.default ];

  scottylabs = {
    enable = true;
    project.name = "my-project";
  };
}

The secretspec block resolves your project’s secrets from OpenBao into the shell. See the Secrets guide to declare and manage them.

2. Set up direnv and .gitignore

Create an .envrc to automatically activate the devenv environment when you enter the project directory:

eval "$(devenv direnvrc)"
use devenv

Then allow it:

direnv allow

If this command fails with a secret-related error, make sure you have run the one-time OpenBao setup.

Add a .gitignore for generated and local-only files:

# Nix / devenv
.devenv/
.devenv.flake.nix
.pre-commit-config.yaml
result
result-*

# AI
.mcp.json
.claude

# direnv
.direnv/

# Rust
target/
.cargo/

# Secrets
.env

# OS
.DS_Store
rustc-ice-*.txt

Add any project-specific entries as needed (e.g., sites/docs/book/ for mdbook output, node_modules/ for JS projects).

3. Declare what to deploy

Add kennel options to your devenv.nix to tell kennel what your project produces.

For a backend service:

scottylabs.kennel.services.api = {
  customDomain = "api.my-project.scottylabs.org";
};

In production kennel runs the flake package, so the scottylabs.kennel.services key and the flake package must share the same name (here, api).

For a static site:

scottylabs.kennel.sites.app = {
  spa = true;
};

Set spa = true for single-page apps so Caddy serves index.html for unmatched routes. It defaults to false, which serves files directly, suitable for pre-rendered sites like mdbook docs.

Note that custom domains that are not already in use must first have their Cloudflare Zone IDs registered with kennel in the infrastructure repository.

Health checks

Every backend service must expose a GET /api/health endpoint that returns HTTP 200 once it is ready to accept traffic. After starting a service, kennel polls this endpoint every 2 seconds for up to 60 seconds, holding back traffic until it returns 200. If it has not passed by then, the deployment is marked failed and the public domain is never routed to it. Static sites have no health endpoint and are exempt.

Flake packages

Your flake.nix must expose these packages, and their names must match the keys in scottylabs.kennel.services and scottylabs.kennel.sites. Build them with the shared build helpers:

inputs = {
  nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
  scottylabs = {
    url = "git+https://codeberg.org/ScottyLabs/devenv";
    inputs.nixpkgs.follows = "nixpkgs";
  };
};

outputs = { nixpkgs, scottylabs, ... }:
  let
    forAllSystems = nixpkgs.lib.genAttrs [ "x86_64-linux" "aarch64-linux" ];
  in {
    packages = forAllSystems (system:
      let
        pkgs = nixpkgs.legacyPackages.${system};
        helpers = scottylabs.mkLib pkgs;
      in {
        api = helpers.buildRustService {
          src = ./.;
          pname = "api";
          buildArgs.cargoExtraArgs = "-p api";
        };
        docs = helpers.buildMdbook { src = ./sites/docs; };
      }
    );
  };

This must be the same nixpkgs pinned in devenv.yaml, so nix build and devenv shell resolve identical packages.

A Deno or JavaScript front-end uses buildDenoTask the same way.

A browser front-end (Vite, Svelte) also needs a root deno.json that excludes its directory, so the deno check hook skips it instead of failing on browser code. Replace sites/web with your front-end’s path:

{
  "exclude": ["sites/web"]
}

For a SvelteKit app, svelte-check type-checks it (enable scottylabs.deno.svelte.enable).

Kennel builds each package with nix build .#packages.{system}.{name}.

Runtime environment

Kennel injects these variables into every backend service it deploys:

  • PORT: the port your service must bind to. Kennel allocates it and routes the public domain to it through Caddy, so read it at startup instead of hardcoding a port.
  • COMMIT_HASH: the full Git commit SHA of the running build.
  • APP_URL: the public URL of this deployment, its custom domain when configured or the generated domain otherwise.

Resolved secrets from your secretspec.toml are injected alongside these.

Kennel runs each service from a working directory that contains your secretspec.toml, so the secretspec SDK’s runtime load() finds it automatically. The filesystem is read-only apart from a private /tmp, so keep persistent state in a provisioned database or object store.

4. Enable infrastructure

If your project needs a database:

scottylabs.postgres.enable = true;

This gives you a local PostgreSQL instance in development and a provisioned per-deployment database in production. Your app reads DATABASE_URL from the environment in both cases.

sqlite, garage (S3-compatible object storage), and valkey (Redis-compatible key-value store) are also available and documented in the devenv options. Aside from garage, these databases are configured with unix socket auth rather than TCP/password auth.

When developing the OAuth flow locally, enable the relay so the IdP redirects back through it the way it does in production:

scottylabs.ricochet.enable = true;
scottylabs.ricochet.appUrl = "http://localhost:3000";

Governance already seeds the correct OAUTH_RELAY_URL for profiles.dev, so enable only runs Ricochet locally. appUrl is your service’s local URL, the port your dev server listens on, exported as APP_URL so the relay can return to it. Set it only for development: deployed environments receive APP_URL from kennel automatically (see Runtime environment), and it is never declared in secretspec.toml.

5. Development scripts

scripts.<name>.exec defines a command available in your shell whenever the devenv environment is active. Use it for project-specific development tasks such as database migrations or code generation:

scripts = {
  migrate.exec = "sea-orm-cli migrate up -d crates/migration";
  generate-api.exec = "cd app && deno task generate-api";
};

Running migrate in the shell then executes that command. Scripts are a development convenience only. Kennel never runs them, in builds or deployments.

6. Push

Kennel deploys three long-lived branches (main, staging, or dev). To get a preview, open a pull request and kennel builds and deploys it. Pushing any other branch is ignored.

Your deployment will be available at:

  • my-project-main.scottylabs.net for the main branch
  • my-project-pr-42.scottylabs.net for PR #42

As it works, kennel reports progress back to Forgejo as commit statuses on the pushed commit. The kennel/build status moves from pending to success or failure as the build runs, and kennel/deploy reports success once the deployment passes its health check. A failed status means the commit never reached a healthy deployment; check the build log to see why.

Disabling preview deployments

Some services should never run more than one instance at a time. A Discord bot, for example, would respond twice if a preview ran alongside production. Disable preview deployments for the whole project:

scottylabs.kennel.previewDeployments = false;

It defaults to true. When disabled, opening or updating a pull request still builds and reports kennel/build, but kennel does not deploy the preview.

Continuous Integration

ScottyLabs projects share one reusable Forgejo Actions workflow from the devenv repo. It runs the git hooks your devenv.nix enables and builds the project, matching what runs on commit.

Adding CI to a project

Create .forgejo/workflows/ci.yml:

name: CI

on:
  push:
    branches: [main, staging, dev]
  pull_request:

jobs:
  check:
    uses: ScottyLabs/devenv/.forgejo/workflows/ci.yml@main
    secrets: inherit

It installs Lix and devenv, then:

  • runs every git hook your modules enable (formatting, linting, type-checking, tests, commit message checks) with devenv shell -- pre-commit run --all-files
  • builds every service and site declared in scottylabs.kennel.services and scottylabs.kennel.sites with nix build, the same set kennel deploys

Caching

  • Cachix (scottylabs) as a substituter for pulls and a push target for Nix store paths.
  • sccache for Rust compilation, backed by the shared S3 bucket.
  • The Rust target cache (Swatinem/rust-cache) and the Deno and uv caches under .devenv/state.
  • devenv’s Nix eval cache.

Secrets

secrets: inherit passes the caller’s org-level credentials (the CACHIX_AUTH_TOKEN and sccache S3 keys) to the workflow so it can reach the shared caches. A project’s own secrets never enter CI, because the ci secretspec profile marks them optional and the workflow resolves nothing from OpenBao.

Secrets

Kennel resolves secrets from OpenBao via secretspec. Secrets are injected as environment variables and never written to disk or stored in the database.

Declaring secrets

Create a secretspec.toml in your project root. The [project] table requires a name and revision = "1.0", and secrets are declared per profile:

[project]
name = "my-project"
revision = "1.0"

[profiles.ci.defaults]
required = false

[profiles.default]
JWT_SECRET = { description = "JWT signing key", required = true }
STRIPE_KEY = { description = "Stripe API key", required = true }

# Declare all profiles, even if you only use default
# default secrets can only be substituted into existing profiles
[profiles.prod]

[profiles.staging]

[profiles.preview]
STRIPE_KEY = { description = "Stripe test key", required = false }

[profiles.dev]

[profiles.default] holds the secrets shared across every profile; named profiles inherit from it and may override individual entries, for example making STRIPE_KEY optional in preview. Declare a [profiles.<name>] header for dev (used locally) and for every environment you deploy to (see the branch-to-profile mapping); a section may be left empty to inherit default unchanged.

The [profiles.ci.defaults] block makes every inherited secret optional for CI. The shared CI workflow sets SECRETSPEC_PROFILE=ci and SECRETSPEC_PROVIDER=env so that devenv shell does not require an OpenBao token. CI checks run without project secret values and must not depend on them. The workflow’s only secrets are the org-level Cachix and sccache credentials it forwards for binary caching.

Configuring the provider

Point secretspec at OpenBao in your devenv.yaml. Copy this block once per project:

secretspec:
  enable: true
  provider: vault://secrets2.scottylabs.org/secret
  profile: dev

enable turns on resolution, provider is the default backend for every secret, and profile selects which profile to load locally (kennel chooses the profile per branch when it deploys). The shared ScottyLabs config (scottylabs.enable = true) supplies the bao and secretspec CLIs, sets BAO_ADDR, and exports every resolved secret into your shell.

Per-developer secrets

Some secrets differ per developer and cannot be shared, for example each person’s own DISCORD_TOKEN while developing a bot. Source those from a gitignored .env instead of OpenBao: define a local alias in a [providers] table and give the secret a provider chain in the dev profile.

[providers]
local = "dotenv://.env"

[profiles.prod]
DISCORD_TOKEN = { description = "Discord bot token" }

[profiles.dev]
DISCORD_TOKEN = { providers = ["local"] }

prod resolves DISCORD_TOKEN from OpenBao (the default provider) while dev reads it from your .env. Chains are tried in order, so providers = ["vault", "local"] would try OpenBao first and fall back to .env; every alias you name must be defined in this committed [providers] table. Keep .env gitignored.

Local development

Log in to OpenBao and configure Cachix once per machine:

nix run git+https://codeberg.org/ScottyLabs/devenv#login

A project shell resolves secrets as it loads, so it won’t build without a token. On a fresh checkout you don’t have one and can’t reach bao from inside the shell yet, hence the standalone command above. It also stores your Cachix auth token so devenv pulls from and pushes to the shared binary cache.

The token is periodic and renews on each shell entry, so it stays valid as long as you open a project shell at least once every 90 days. You only log in again on a new machine, or after 90 days without using it.

Secrets then load into your shell automatically each time direnv reloads.

Managing secrets

Set a secret for the default (dev) profile:

secretspec set JWT_SECRET

Set a secret for a specific profile:

secretspec set -P prod STRIPE_KEY
secretspec set -P preview STRIPE_KEY

Verify all required secrets are present:

secretspec check
secretspec check -P prod

Production

Kennel authenticates to OpenBao with a service token provided via VAULT_TOKEN in its environment file. It resolves secrets for each deployment using the profile matching the branch:

BranchProfile
mainprod
stagingstaging
devdev
pr-*preview

If a required secret cannot be resolved, the deployment fails. Deployed services also receive PORT, COMMIT_HASH, and APP_URL (see Deploying a Project).

Runtime loading

Use the typed secretspec SDKs to load secrets in your application code. Environment variables work but lose type safety and expose every secret to every process. The SDKs generate typed accessors from your secretspec.toml at build time.

Runtime load() reads secretspec.toml from disk, not only at build time. Kennel makes it available to deployed services automatically (see Runtime environment).

See the SDK overview for the full list of supported languages.

Preview Deployments

Every pull request and feature branch gets its own isolated deployment with its own database, cache, and URL.

Lifecycle

  1. PR opened: kennel builds the PR branch, provisions resources, and deploys. The deployment is available at {project}-pr-{number}.scottylabs.net.
  2. Push to PR: kennel rebuilds. Unchanged services (same nix store path) are skipped. Changed services are redeployed.
  3. PR closed: kennel tears down all deployments for the branch, deprovisions resources (drops the database, flushes the cache, deletes the storage bucket), and removes the Caddy route.

Status comments

After each successful PR deploy, kennel posts (and on subsequent deploys edits) a sticky comment on the pull request listing every service URL for that branch. When the PR closes and deployments are torn down, the comment is updated to reflect the teardown. Comments are identified by an HTML marker in the body so kennel can update the same comment instead of creating duplicates.

This requires the operator to configure services.kennel.forgejo.apiTokenFile with a Forgejo API token that has the write:issue scope. See the NixOS Module reference.

Resource isolation

Each PR deployment gets:

  • Its own PostgreSQL database (kennel_{project}_{branch})
  • Its own Valkey DB number
  • Its own Garage S3 bucket and API key

Connection strings are injected as environment variables. Your application code does not need to know whether it is running in production or a PR preview.

Expiry

PR deployments that have had no activity for 7 days are hibernated: the process is stopped but the database is kept. After 30 days, the deployment and its resources are fully torn down.

URLs

PR URLs follow the flat scheme {project}-pr-{number}.scottylabs.net, covered by a single wildcard DNS record. No per-deployment DNS management is needed.

Architecture

Kennel runs as a single binary with four main responsibilities: receiving webhooks, building Nix packages, deploying them, and reconciling desired state against actual state.

Request flow

Git push -> Webhook -> Build (nix) -> Deploy (systemd + Caddy) -> Live
  1. Forgejo sends a webhook to kennel’s /webhook endpoint.
  2. Kennel parses the repository name from the payload, verifies the HMAC signature, ensures a build record for the commit, and records a deploy request for the branch.
  3. The build worker runs the build in a per-project/branch systemd unit, as a dynamic user with no daemon credentials. The unit clones the repo, runs devenv build scottylabs.kennel.config to discover declared services and sites, and runs nix build for each package, streaming output to journald under the unit and on to Loki. The daemon collects the result, pushes artifacts to cachix, and stores the log in the builds.log column.
  4. For each pending deploy request whose build has finished, the reconciler reuses the build’s artifacts to provision resources (database, cache, storage), resolve secrets from OpenBao, start a systemd transient unit for services, and add a Caddy route. Requests for a preview branch are skipped when the project disables previewDeployments.
  5. Caddy serves traffic over HTTPS with on-demand TLS.

Delegation

Kennel delegates process supervision to systemd and HTTP routing to Caddy, keeping the core focused on build orchestration and resource provisioning.

Systemd transient units are created via D-Bus using the zbus crate. Units are placed in the kennel.slice cgroup for aggregate accounting, with CPUAccounting, MemoryAccounting, IOAccounting, and TasksAccounting enabled so per-deployment resource usage is queryable from cgroup metrics by anything scraping systemd_unit_* or systemd_slice_* (e.g. prometheus-systemd-exporter filtered to kennel-* units). Each unit runs sandboxed as a DynamicUser with no daemon credentials, a read-only filesystem (ProtectSystem=strict, ProtectHome=yes) except a private /tmp (PrivateTmp=yes), and NoNewPrivileges. Transient units survive kennel crashes since they are independent of the kennel process.

Caddy routes are managed via the admin API. Each deployment gets a route identified by @id for individual add/remove operations. Caddy handles TLS certificate provisioning, HTTP/3, static file serving, reverse proxying, and SPA fallback.

Nix-store files carry epoch modification times, so a frozen Last-Modified strands proxied clients on a stale shell. Reverse-proxy routes mark text/html responses Cache-Control: no-store. Static-site routes need none, since Caddy’s file server drops epoch validators itself.

HTTP API

Kennel exposes a small set of HTTP endpoints alongside the webhook receiver:

MethodPathPurpose
POST/webhookGit push and pull request events from Forgejo, HMAC-verified.
GET/metricsPrometheus exposition: kennel_builds{status=...}, kennel_deployments, kennel_projects gauges.
GET/deployments/:id/healthJSON: active, active_state, sub_state, result, active_enter_usec, n_restarts, and app_healthy (a live probe of the service’s /api/health) from the unit’s D-Bus properties.
GET/internal/caddy/check-domainUsed by Caddy’s on-demand TLS to validate a hostname is a registered deployment before acquiring a cert.

All endpoints other than /webhook are unauthenticated and read-only. Caddy’s services.kennel.domain virtualhost reverse-proxies these to the kennel API server, which only listens on localhost; the trust boundary is the host firewall plus tailnet, not application-level auth.

Routes are mounted in http.rs; per-resource handlers live under handlers/.

Reconciliation

A single reconciliation loop handles all deployment convergence. It runs on startup, when signaled by a webhook or build completion, and on a periodic 30-second timer.

The reconciler compares desired state (deployment rows in the database) against actual state (systemd units and Caddy routes) and converges:

  • A pending deploy request gets deployed once its build finishes.
  • A deployment row with no running unit gets its unit started.
  • A running unit with no deployment row gets stopped.
  • All Caddy routes are re-added on each pass since Caddy config is ephemeral.
  • The custom domains of all live deployments are written to customDomainsFile, if configured.

State

Kennel stores state in SQLite with four tables:

  • projects – registered repositories with webhook secrets
  • builds – per-commit build queue and history (queued, building, built, failed, cancelled), plus the captured per-phase log of subprocess output
  • deploy_requests – per-branch deploy intents naming the commit each branch wants, with status pending/deployed/skipped/failed
  • deployments – active deployments with store paths, domains, unit names, and ports

Runtime process state (running, stopped, failed) is owned by systemd and queried via D-Bus. Routing state is owned by Caddy and queried via the admin API. Kennel’s database only tracks intent plus the historical build artifacts (logs) systemd doesn’t keep.

Crate structure

  • kennel – main binary. HTTP router lives in src/http.rs, request handlers under src/handlers/{webhook,metrics,deployments,caddy}.rs. Build orchestration in src/build.rs, deploy in src/deploy.rs, reconciliation in src/reconcile.rs. Systemd, Caddy, and OpenBao clients each have their own module.
  • kennel-config – shared types, constants, environment enum
  • kennel-provision – resource provisioning trait and implementations (PostgreSQL, Valkey, Garage)
  • entity – SeaORM generated entities
  • migration – SQLite schema migrations

devenv Options

These options are provided by the shared ScottyLabs devenv module. Import it in your devenv.nix:

imports = [ inputs.scottylabs.devenvModules.default ];

scottylabs.enable

Enable the shared ScottyLabs development configuration. Required for all other scottylabs.* options to take effect. Also installs always-on hooks: two that refresh and stage flake.lock (nix flake update) and devenv.lock (devenv update) on every commit, and one that rejects commits carrying AI tool co-author trailers.

Type: boolean

Default:

false

Declared by:

scottylabs.claude.enable

Enable Claude Code integration. Generates the .mcp.json configuration with the devenv MCP server.

Type: boolean

Default:

true

Declared by:

scottylabs.conventionalCommits.enable

Enforce Conventional Commits on git commit via the commitizen git hook. Commit messages that do not match the conventional format are rejected at commit time.

Type: boolean

Default:

true

Declared by:

scottylabs.deno.enable

Enable the Deno/JavaScript development toolchain. Adds Deno, oxlint (denying the correctness, suspicious, pedantic, and perf categories), oxfmt, and tsgolint on PATH for oxlint --type-aware. Runs oxlint, deno check, and deno test on every commit.

Type: boolean

Default:

false

Declared by:

scottylabs.deno.react.enable

Add the react and jsx-a11y plugins to oxlint.

Type: boolean

Default:

false

Declared by:

scottylabs.deno.svelte.enable

Add the svelte-check pre-commit hook.

Type: boolean

Default:

false

Declared by:

scottylabs.deno.svelte.dir

The SvelteKit app directory, relative to the project root. When svelte.enable is set, deno install and svelte-kit sync run here on shell entry so svelte-check has node_modules and the generated .svelte-kit types available.

Type: string

Default:

"."

Declared by:

scottylabs.garage.enable

Enable a local Garage S3 instance for development. Creates a bucket named after scottylabs.project.name and exports S3_ENDPOINT, S3_REGION, S3_ACCESS_KEY, S3_SECRET_KEY, and S3_BUCKET into the shell environment.

Type: boolean

Default:

false

Declared by:

scottylabs.garage.accessKey

S3 access key for the project bucket

Type: string

Default:

config.scottylabs.project.name

Declared by:

scottylabs.garage.secretKey

S3 secret key for the project bucket

Type: string

Default:

"${config.scottylabs.project.name}admin"

Declared by:

scottylabs.kennel.config

The generated kennel.json derivation that the kennel builder evaluates at build time

Type: package (read only)

Declared by:

scottylabs.kennel.previewDeployments

Deploy a preview environment for each open pull request. Disable for singleton services such as bots, where a preview would run a second instance alongside production. When disabled, pull request commits still build and report kennel/build, but kennel does not deploy the preview.

Type: boolean

Default:

true

Declared by:

scottylabs.kennel.services

Backend services deployed by kennel. Each key must match a devenv process name. Kennel builds the corresponding flake package and deploys it as a systemd transient unit.

Type: attribute set of (submodule)

Default:

{ }

Declared by:

scottylabs.kennel.services.<name>.customDomain

Custom domain for this service

Type: null or string

Default:

null

Declared by:

scottylabs.kennel.sites

Static sites deployed by kennel. Each key names a site. Kennel builds the corresponding flake package and serves it via Caddy’s file server.

Type: attribute set of (submodule)

Default:

{ }

Declared by:

scottylabs.kennel.sites.<name>.customDomain

Custom domain for this static site

Type: null or string

Default:

null

Declared by:

scottylabs.kennel.sites.<name>.spa

Serve index.html for all routes

Type: boolean

Default:

false

Declared by:

scottylabs.postgres.enable

Enable a local PostgreSQL 18 instance with Unix socket access. Creates an initial database named after scottylabs.project.name and exports DATABASE_URL into the shell environment.

Type: boolean

Default:

false

Declared by:

scottylabs.postgres.extensions

PostgreSQL extensions as a function of the extensions set

Type: function that evaluates to a(n) list of package

Default:

e: [ e.pg_uuidv7 ]

Declared by:

scottylabs.project.name

Used for database naming, log filtering, and secrets path resolution

Type: string

Declared by:

scottylabs.python.enable

Enable the Python development toolchain. Manages a virtual environment with uv (running uv sync on shell entry once a pyproject.toml exists), adds ruff (lint pre-commit hook, formatting via treefmt’s ruff-format) and ty (type-check pre-commit hook) on PATH.

Type: boolean

Default:

false

Declared by:

scottylabs.ricochet.enable

Run a local OAuth relay on 127.0.0.1:8090 for development, the loopback callback the dev Keycloak client is registered against. Enable it alongside oidc_client to complete an OAuth flow locally the way production does: the IdP redirects to the relay, which forwards the authorization code on to your service’s own callback.

Type: boolean

Default:

false

Declared by:

scottylabs.ricochet.appUrl

Public URL the service is reached at in local development, exported as APP_URL so it can build OAuth redirect targets and absolute links. This is the development value only; deployed environments receive APP_URL from kennel, derived from the deployment domain (see Deploying a Project), so it is never declared in secretspec.toml.

Type: string

Declared by:

scottylabs.rust.enable

Enable the Rust development toolchain. Configures nightly Rust with cranelift (fast debug-mode codegen), clippy, rustfmt, wild/lld linker, and sccache backed by the org’s S3 cache. Sets CARGO_TARGET_DIR to .devenv/state/target. Clippy and cargo test run on every commit.

Type: boolean

Default:

false

Declared by:

scottylabs.rust.cranelift.excludePackages

Crate names forced to the LLVM backend instead of cranelift. Some crates use features that cranelift does not support (FFI symbol emission, linker sections).

Type: list of string

Default:

[
  "aws-lc-sys"
  "aws-lc-rs"
  "rustls"
]

Declared by:

scottylabs.rust.nativeBuildInputs

Extra packages for crates that link C libraries (e.g. [ pkgs.pkg-config pkgs.openssl ]). Prefer rustls over native-tls for TLS (e.g. reqwest = { default-features = false, features = ["rustls-tls"] }) to avoid needing OpenSSL.

Type: list of package

Default:

[ ]

Declared by:

scottylabs.secrets.enable

When scottylabs.enable = true, the openbao (bao) and secretspec CLIs are added to the shell, BAO_ADDR is set for OpenBao authentication, and every secret secretspec resolves is exported into the shell environment. Resolution is enabled per project through the secretspec block in devenv.yaml (see Secrets).

Type: boolean

Default:

true

Declared by:

scottylabs.sqlite.enable

Enable SQLite for local development. Adds the sqlite package and exports DATABASE_PATH pointing to a database file in the devenv state directory.

Type: boolean

Default:

false

Declared by:

scottylabs.valkey.enable

Enable a local Valkey instance for development. Layers services.redis.package = pkgs.valkey under the hood, so the upstream services.redis devenv module drives the process while the binary is the wire-compatible Valkey fork. Adds pkgs.valkey to the shell so valkey-cli is on the path.

Type: boolean

Default:

false

Declared by:

Build Helpers

These builders come from the shared ScottyLabs devenv flake as mkLib, a function applied to a pkgs set (the same shape as crane.mkLib). Add the flake as an input, then call a helper per system:

# flake.nix inputs
scottylabs = {
  url = "git+https://codeberg.org/ScottyLabs/devenv";
  inputs.nixpkgs.follows = "nixpkgs";
};

# in outputs
pkgs = nixpkgs.legacyPackages.${system};
kennel = (scottylabs.mkLib pkgs).buildRustService { ... };

Each helper takes the consumer’s pkgs, so packages build against your nixpkgs pin rather than the shared flake’s.

buildRustService

Builds a Rust crate with crane, caching dependencies separately from the build.

src

Crate or workspace root. Passed through craneLib.cleanCargoSource, so only Cargo-relevant files enter the build.

Type: path, required

pname

Package name. When unset, it is read from the pname field of Cargo.toml. A workspace has a virtual manifest with no package name, so a workspace must set this explicitly.

Type: nullOr str, default: null

version

Package version. When unset, it is read from the version field of Cargo.toml.

Type: nullOr str, default: null

paths

Scopes src to just these workspace-member paths plus Cargo.toml/Cargo.lock, so changes elsewhere in the workspace don’t invalidate the build.

Type: nullOr (listOf str), default: null

nativeBuildInputs

Build-time tools, applied to both the dependency build and the final build. For example pkg-config or makeWrapper.

Type: listOf package, default: [ ]

buildInputs

Libraries to link against, applied to both the dependency build and the final build. For example openssl.

Type: listOf package, default: [ ]

buildArgs

Build-only options forwarded to crane’s buildPackage, kept separate from the arguments above so changing them never rebuilds the cached dependencies. doCheck defaults to false. Common keys are cargoExtraArgs to select a workspace member such as "-p kennel", and postInstall to wrap the binary.

Type: attrs, default: { }

(scottylabs.mkLib pkgs).buildRustService {
  src = ./.;
  pname = "kennel";
  version = "0.1.0";
  buildArgs.cargoExtraArgs = "-p kennel";
}

buildDenoTask

Builds a Deno project that has npm dependencies, running a deno task and copying its output directory. On Linux it patches the prebuilt native addons and drops the musl variants, which cannot be patched against a glibc host.

src

Directory holding deno.lock and deno.json.

Type: path, required

pname

Package name.

Type: str, required

version

Package version.

Type: str, default: "0.1.0"

task

The deno task to run.

Type: str, default: "build"

output

Directory the task emits, copied verbatim to the result.

Type: str, default: "dist"

entrypoint

Passed to deno install --entrypoint so dependency resolution ignores dependencies not reachable from it, such as test-only dependencies.

Type: nullOr str, default: null

compile

Provision denort so deno compile runs inside the offline build sandbox. Enable when the task compiles a binary instead of emitting a dist directory.

Type: bool, default: false

(scottylabs.mkLib pkgs).buildDenoTask {
  src = ./sites/web;
  pname = "link-shortener-web";
  version = "0.1.0";
}

buildPythonService

Builds a uv project into a virtual environment with uv2nix, resolving every dependency from uv.lock as a Nix derivation. The result is a relocatable venv exposing each [project.scripts] entry under bin/, consumed as ${pkg}/bin/<script>.

src

Project root holding pyproject.toml and uv.lock. The package name is read from pyproject.toml.

Type: path, required

python

The Python interpreter the environment is built against.

Type: package, default: pkgs.python3

sourcePreference

Whether dependencies come from prebuilt wheels ("wheel") or are built from source ("sdist"). The build-system overlay is matched to this choice automatically.

Type: str, default: "wheel"

selectDeps

Selects the locked dependency closure to install from the loaded workspace. Override to install an optional group, for example ws: ws.deps.optionals.prod.

Type: function, default: ws: ws.deps.default

overrides

An overlay applied last over the Python package set, to supply system libraries or build tools uv2nix cannot infer for a package built from source (for example adding pkgs.openssl to buildInputs).

Type: function, default: _final: _prev: { }

(scottylabs.mkLib pkgs).buildPythonService {
  src = ./.;
}

buildMdbook

Builds an mdBook site to the result.

src

Directory holding book.toml.

Type: path, required

name

Derivation name.

Type: str, default: "docs"

(scottylabs.mkLib pkgs).buildMdbook {
  src = ./sites/docs;
  name = "kennel-docs";
}

buildOptionsDoc

Renders a module set’s option declarations to a CommonMark reference with nixpkgs’ nixosOptionsDoc, producing a single markdown file listing every option with its description, type, default, and a link to the declaring module. Both options reference pages in these docs are generated with it. Only declarations are evaluated, never the merged config, so an option whose default reads other config must set defaultText. Every documented option must have a description, or the build fails.

module

Module (or import path) declaring the options.

Type: module, required

subtree

Selects the options to document from the evaluated options tree.

Type: function, required

root

Repo root. Declaration paths under it become source links relative to it.

Type: path, required

repoUrl

Source browser base URL the declaration links point to.

Type: str, required

(scottylabs.mkLib pkgs).buildOptionsDoc {
  module = ./nixos;
  subtree = options: options.services.kennel;
  root = ./.;
  repoUrl = "https://codeberg.org/ScottyLabs/kennel/src/branch/main";
}

NixOS Module

The kennel NixOS module configures the kennel service, Caddy, systemd integration, and resource provisioning on a NixOS host.

{ kennel, ... }:
{
  imports = [ kennel.nixosModules.default ];

  services.kennel = {
    enable = true;
    package = kennel.packages.x86_64-linux.kennel;
    devenvPackage = kennel.packages.x86_64-linux.devenv;
    webhookSecretFile = config.age.secrets.kennel-webhook.path;
    environmentFile = config.age.secrets.kennel.path;

    domains = {
      ephemeral = "scottylabs.net";
      cloudflare.zones."scottylabs.org" = "<zone-id>";
    };

    resources.postgres = {
      enable = true;
      socketDir = "/run/postgresql";
    };

    secrets = {
      enable = true;
      vaultEndpoint = "https://secrets2.scottylabs.org";
    };
  };
}

Options

services.kennel.enable

Whether to enable Kennel deployment platform.

Type: boolean

Default:

false

Example:

true

Declared by:

services.kennel.package

The Kennel package to use

Type: package

Default:

pkgs.kennel

Declared by:

services.kennel.api.host

API server bind address

Type: string

Default:

"0.0.0.0"

Declared by:

services.kennel.api.port

API server port

Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default:

3000

Declared by:

services.kennel.builder.cachix.enable

Whether to enable pushing build artifacts to a Cachix binary cache.

Type: boolean

Default:

false

Example:

true

Declared by:

services.kennel.builder.cachix.cacheName

Cachix cache name

Type: null or string

Default:

null

Declared by:

services.kennel.builder.maxConcurrentBuilds

Maximum number of concurrent nix builds

Type: signed integer

Default:

2

Declared by:

services.kennel.builder.workDir

Build working directory

Type: absolute path

Default:

"/var/lib/kennel/builds"

Declared by:

services.kennel.caddy.adminUrl

Caddy admin API URL

Type: string

Default:

"http://localhost:2019"

Declared by:

services.kennel.customDomainsFile

Writes the custom domains of all live deployments to this path, one per line, refreshed on each reconciliation pass. Unset disables it.

Type: null or string

Default:

null

Example:

"/run/kennel/custom-domains"

Declared by:

services.kennel.devenvPackage

The devenv package. The build worker uses devenv build to evaluate project kennel configs from their devenv.nix.

Type: package

Declared by:

services.kennel.domain

Public domain for the kennel API and webhook endpoint. The module configures a Caddy virtualhost with automatic TLS for this domain, reverse-proxying to the API server.

Type: string

Default:

"kennel.scottylabs.org"

Declared by:

services.kennel.domains.cloudflare.publicIp

Public IPv4 used as the content of the A records that kennel creates for custom domains. Required to enable DNS automation.

Type: null or string

Default:

null

Declared by:

services.kennel.domains.cloudflare.zones

Map of domain names to Cloudflare zone IDs. When this is non-empty, publicIp is set, and the CLOUDFLARE_API_TOKEN environment variable is provided (typically via the environmentFile secret), kennel automatically manages A records for any custom domain whose suffix matches one of the configured zones. The most specific zone wins for nested domains.

The token must have Zone:DNS:Edit permission on the zones listed.

Records are upserted on deploy and on each reconciliation pass (so they self-heal if pruned externally). Records are deleted only when kennel tears the deployment down, which happens in three cases:

  1. The branch backing the deployment is deleted from the source repo (push event with deleted=true on that ref).
  2. The deployment is associated with a pull request and that pull request is closed.
  3. The deployment is on a dev or preview branch and exceeds DEPLOYMENT_EXPIRY_DAYS since its last update during a reconciliation pass.

Production deployments are not subject to expiry, so a record for a production custom domain stays in place until the project’s main branch is deleted or the deployment row is removed manually.

Type: attribute set of string

Default:

{ }

Declared by:

services.kennel.domains.ephemeral

Base domain for auto-generated deployment URLs. A wildcard DNS record should point *.{domain} to the kennel server.

Type: string

Default:

"scottylabs.net"

Declared by:

services.kennel.environmentFile

Path to an environment file containing secrets like VAULT_TOKEN, CACHIX_AUTH_TOKEN, and GARAGE_ADMIN_TOKEN. Loaded by systemd before the service starts.

Type: null or absolute path

Default:

null

Declared by:

services.kennel.forgejo.apiTokenFile

Path to a file containing a Forgejo API token with the write:issue scope. Kennel uses it to post and update a sticky comment on each pull request listing its deployment URLs, and to mark the comment torn down when the pull request is closed.

Type: absolute path

Declared by:

services.kennel.forgejo.apiUrl

Forgejo API base URL used to post PR deployment comments

Type: string

Default:

"https://codeberg.org/api/v1"

Declared by:

services.kennel.grafanaUrl

Base URL of Grafana for Logs Drilldown links in commit statuses

Type: null or string

Default:

"https://grafana.scottylabs.org"

Declared by:

services.kennel.group

Group to run Kennel as

Type: string

Default:

"kennel"

Declared by:

services.kennel.resources.garage.enable

Whether to enable Garage S3 resource provisioning. Kennel creates a bucket and API key per deployment. Requires GARAGE_ADMIN_TOKEN in the environment file.

Type: boolean

Default:

false

Declared by:

services.kennel.resources.garage.adminEndpoint

Garage admin API endpoint

Type: string

Default:

"http://localhost:3903"

Declared by:

services.kennel.resources.garage.s3Endpoint

Garage S3 API endpoint

Type: string

Default:

"http://localhost:3900"

Declared by:

services.kennel.resources.postgres.enable

Whether to enable PostgreSQL resource provisioning. Kennel creates a database per deployment using the specified socket directory for peer authentication.

Type: boolean

Default:

false

Declared by:

services.kennel.resources.postgres.socketDir

PostgreSQL Unix socket directory

Type: absolute path

Default:

"/run/postgresql"

Declared by:

services.kennel.resources.valkey.enable

Whether to enable Valkey resource provisioning. Kennel allocates a DB number per deployment from the shared instance.

Type: boolean

Default:

false

Declared by:

services.kennel.resources.valkey.socketPath

Valkey Unix socket path

Type: absolute path

Default:

"/run/valkey/valkey.sock"

Declared by:

services.kennel.secrets.enable

Whether to enable secretspec/OpenBao secret resolution at deploy time.

Type: boolean

Default:

false

Example:

true

Declared by:

services.kennel.secrets.vaultEndpoint

secretspec provider URI for OpenBao/Vault

Type: string

Default:

"vault://secrets2.scottylabs.org/secret?auth=approle"

Declared by:

services.kennel.user

User to run Kennel as

Type: string

Default:

"kennel"

Declared by:

services.kennel.webhookSecretFile

Path to a file containing the HMAC secret used to verify all incoming webhooks. This is a single secret shared across all projects, provisioned by governance.

Type: absolute path

Declared by:

What the module configures

  • A systemd service for kennel with Delegate=yes for cgroup v2 access
  • A polkit rule allowing the kennel user to create transient systemd units via D-Bus
  • A kennel.slice for all managed deployment units
  • A Caddy virtualhost for the kennel domain with automatic TLS, plus the admin API for dynamic route management
  • tmpfiles rules for /var/lib/kennel subdirectories
  • Firewall rules for ports 80 and 443
  • Cachix binary cache substituter