# Deployment runbook

How to put this application on a server, keep it running, back it up and get
it back. Written for someone who has never seen the project.

This runbook ends where **`docs/go-live.md`** begins. This document installs
the system and leaves you at an empty, working panel with one branch and one
user. `docs/go-live.md` then brings a real business centre into it.

---

## Contents

1. [What you are deploying](#1-what-you-are-deploying)
2. [Requirements](#2-requirements)
3. [Six things about this build you must know first](#3-six-things-about-this-build-you-must-know-first)
4. [The names used in this runbook](#4-the-names-used-in-this-runbook)
5. [The database](#5-the-database)
6. [First install](#6-first-install)
7. [The web server](#7-the-web-server)
8. [The queue worker](#8-the-queue-worker)
9. [The scheduler](#9-the-scheduler)
10. [Every deploy after the first](#10-every-deploy-after-the-first)
11. [Post-deploy checks](#11-post-deploy-checks)
12. [Backups](#12-backups)
13. [Restore](#13-restore)
14. [Continuous integration](#14-continuous-integration)
15. [Handing off to go-live](#15-handing-off-to-go-live)
16. [When something is wrong](#16-when-something-is-wrong)

---

## 1. What you are deploying

A Laravel 13 application with a single Filament 5 admin panel at `/admin`.
There is no public site and no API: `/` shows a placeholder page, `/up` is a
health check, and everything else lives behind sign-in at `/admin`.

It runs a UAE business centre — units, leases, invoices, receipts, cheques,
VAT, purchases, payroll, assets, tickets and room bookings — for one company
across several branches.

Four processes make up a working install:

| Process | What it is | What happens without it |
|---|---|---|
| PHP-FPM behind nginx | serves the panel | nothing is reachable |
| `queue:work` | one supervised worker | every go-live import says *Import started* and then nothing happens; payment-reminder emails are never sent |
| `schedule:run` | one cron line, every minute | leases never expire, no expiry alert is ever sent, overdue tickets are never flagged, **and nobody is invoiced** |
| MySQL 8.4 | the database | — |

---

## 2. Requirements

### The host

A single Linux server. Nothing here needs more than one machine. Debian 12 or
Ubuntu 24.04 is assumed; adjust package names for another distribution.

### PHP 8.3

PHP 8.3 is required by `composer.json` (`"php": "^8.3"`). These extensions are
required — the list is `composer check-platform-reqs` plus `pdo_mysql`, which
the database connection needs at runtime:

```
sudo apt install php8.3-fpm php8.3-cli php8.3-mysql php8.3-mbstring \
    php8.3-xml php8.3-intl php8.3-zip php8.3-gd php8.3-curl php8.3-bcmath
```

Those packages provide: `ctype dom exif fileinfo filter gd hash iconv intl
json libxml mbstring openssl pcre pdo_mysql session tokenizer xmlreader zip
zlib`. `gd` is what spatie/laravel-medialibrary uses for image conversions;
`zip` is a hard requirement in `composer.json`.

Verify after installing:

```
php -v
php -m
```

### MySQL 8.4

```
mysql --version
```

MySQL, not MariaDB and not sqlite. The schema uses stored generated columns
(for example `period_closes.posted_period`) and the application uses
`SELECT ... FOR UPDATE` in the lease, deposit and ticket services.

### Node 20.19+ or 22.12+

**This is the single most common way this deploy fails.** `package.json` pins
`vite@^8`, and vite 8 builds on rolldown, which declares:

```
engines: ^20.19.0 || >=22.12.0
```

Note what that range excludes: everything below 20.19, **all of Node 21**, and
22.0 through 22.11. Check before you start:

```
node --version
```

Below the floor, a clean `npm ci && npm run build` **fails**, and the useful
part is that **neither error mentions the Node version**. Both were observed
on 22.11.0 with vite 8.3.0:

```
Error: Cannot find native binding. npm has a bug related to optional
dependencies (https://github.com/npm/cli/issues/4828). Please try `npm i`
again after removing both package-lock.json and node_modules directory.
```

That message is a misdirection. `@rolldown/binding-win32-x64-msvc` *is* in
`package-lock.json`; it simply was not installed. **Do not follow its advice**
— deleting `package-lock.json` throws away the pinned dependency set to work
around something that is not the cause. Installing the binding by hand gets
you a second, equally silent failure:

```
TypeError: The "paths[1]" argument must be of type string.
Received an instance of Array
```

So `deploy.sh` refuses below the floor and says why, rather than letting you
debug a dependency tree that is not broken.

To upgrade:

```
curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
sudo apt install -y nodejs
node --version
```

> **One trap worth recording, because it cost a wrong correction to this very
> section.** On 2026-10-02 this page was rewritten to say the opposite — that
> 22.11 merely warns and builds fine — on the strength of running
> `npm run build` on 22.11 and watching it succeed.
>
> That test was invalid. `node_modules` had been installed earlier, under
> conditions that worked, so a successful build proved nothing about a fresh
> install. The failure only appears once `npm ci` has replaced the tree — and
> `npm ci && npm run build` is what a deploy actually runs.
>
> The original text was substantially right and has been restored. The lesson
> is not "trust the version number" — it is **test the whole operation, not
> the convenient half of it.** The two halves disagreed, and the half that
> matched the deploy was the one not run.
>
> One thing from the wrong version was a real fix and is kept: the old check
> refused every Node 20 release, though 20.19+ is supported. The floor now
> matches the engines range exactly.

### Composer 2

```
composer --version
```

### Outbound network access

Only `composer install` and `npm ci` need it, to reach packagist and the npm
registry. Nothing else in the build or in the running application fetches
anything from the internet.

The build used to download the Instrument Sans web font from
`fonts.bunny.net`, and the panel used to link the same host from every admin
page, so a locked-down build host failed at `npm run build` — which is fatal,
see §3.2 — and a browser behind a restricted network got no typeface at all.
The family is now checked into the repository at
`public/fonts/instrument-sans` (SIL Open Font License 1.1; `OFL.txt` sits
beside the files) and served by the application. Both the build and the
browser are offline-clean.

---

## 3. Six things about this build you must know first

### 3.1 `APP_KEY` is not rotatable. Rotating it destroys every stored IBAN.

`app/Models/Employee.php` casts `iban` with Laravel's `encrypted` cast:

```php
'iban' => 'encrypted',
```

Employee bank details are encrypted with `APP_KEY` before they reach the
database. There is no second copy and no key history. If `APP_KEY` changes,
every stored IBAN becomes ciphertext nobody can read — not a decryption
error you can work around, a permanent loss of the payroll bank details for
every employee on file.

So:

- Run `php artisan key:generate` **exactly once**, on the first install, and
  never again.
- Copy `APP_KEY` out of `.env` and store it where you store passwords — not
  only in the database backup, which does not contain it.
- Never run `php artisan key:generate` as part of a deploy. `deploy.sh` does
  not, and refuses to run at all if `APP_KEY` is empty.
- A restore of the database onto a server with a **different** `APP_KEY` will
  look completely healthy until somebody opens an employee record. Restore the
  key with the data — §13 covers this.

If the key has to change for a real reason (it leaked), every IBAN must be
read with the old key and re-written with the new one before the old key is
discarded. There is no command in this application that does that. Treat it
as a project, not a step.

### 3.2 The panel will not load at all without compiled assets

`app/Providers/Filament/AdminPanelProvider.php` registers the panel with
`->viteTheme('resources/css/filament/admin/theme.css')`. Filament resolves
that through Laravel's Vite helper on **every** admin page, and the helper
reads `public/build/manifest.json`.

`/public/build` is in `.gitignore`. It is never in a checkout and never in a
`git pull`. So a deploy that skips `npm ci && npm run build` does not degrade
to an unstyled page — it throws `Illuminate\Foundation\ViteException: Unable
to locate file in Vite manifest` and serves a 500 on every admin page,
including the login page.

`deploy.sh` builds the assets and then refuses to finish if
`public/build/manifest.json` is not there.

(Filament's own CSS and JS under `public/css/filament`, `public/js/filament`
and `public/fonts/filament` are a separate thing: they are committed to the
repository, and `composer install` republishes them through the
`filament:upgrade` hook. You do not have to do anything about those.)

### 3.3 Everything is in UAE time, and the server clock does not decide that

`config/app.php` sets `'timezone' => env('APP_TIMEZONE', 'Asia/Dubai')`, and
its comment says why: the platform serves one business operating in the UAE,
which is UTC+4 all year with no daylight saving. Every `now()`, every date
cast, every Filament date picker and the scheduler all work in UAE wall-clock
time. A scheduled `dailyAt('06:45')` means 06:45 in Dubai, whatever the
server's own zone is.

`config/database.php` separately pins the MySQL **session** time zone to
`+04:00` (`'timezone' => env('DB_TIMEZONE', '+04:00')`). The comment there
explains the trap: some values reach SQL already carrying a `+04:00` offset —
the booking calendar's FullCalendar feed sends `+04:00`, not `Z` — and MySQL
converts those using the session zone. A fresh server's session zone defaults
to `SYSTEM`, which is commonly UTC, and every such comparison would shift by
four hours in silence. Pinning it removes the dependency on the server's
configuration entirely.

**What this means for you:** leave `APP_TIMEZONE` unset and `DB_TIMEZONE` at
`+04:00`. A server in London or a MySQL instance running UTC is fine and needs
no change — that is the point of pinning both. What is *not* fine is
overriding either of them to match the server: a server in another zone that
sets `APP_TIMEZONE=Europe/London` would run the 06:45 lease sweep at 10:45
Dubai time, date invoices by London's midnight, and give the VAT return the
wrong period boundary at the turn of a month.

Set the server clock to UTC and let the application do the converting:

```
sudo timedatectl set-timezone UTC
timedatectl
```

### 3.4 Document scans are not in the database

`config/filesystems.php` defines a private `documents` disk rooted at
`storage/app/documents`. Passports, Emirates IDs, visas and trade licences
live there as files. They are deliberately **not** under `storage/app/public`
(so `storage:link` cannot expose them) and **not** under `storage/app/private`
(so a signed temporary URL cannot reach them either). A scan leaves the server
only through `DocumentFileController`, which checks the branch scope and
`DocumentPolicy::view()`.

A backup that covers only MySQL does not contain a single scan. §12 covers
both.

### 3.5 Imports and reminder emails are queued; the scheduler is separate

`QUEUE_CONNECTION=database`. Two things go through the queue:

- **Every go-live import** (branches, customers, units, live leases, lease
  schedules, opening invoices, opening advances, opening vendor bills). These
  are Filament import jobs, run in a job batch.
- **Payment reminder emails** (`app/Jobs/SendPaymentReminder.php`), which
  staff trigger from the Payment Reminders page.

With no worker running, an import says *Import started* and then nothing at
all happens — no rows, no failed-rows file, every readiness check stuck at
zero. Nothing is lost; the job is sitting in the `jobs` table. `docs/go-live.md`
§2 says the same thing, and §8 below makes the worker a supervised service so
nobody has to remember.

The **eight scheduled commands are not queued.** They do their work inline in
the scheduler's own process. They need cron, not the worker.

### 3.6 `DemoSeeder` must never run on production — and now it cannot

`database/seeders/DemoSeeder.php` creates a fictional business centre:
branches, customers, units, leases, invoices, receipts, cheques, journals,
employees, payroll runs, assets, tickets and bookings. It is development
scaffolding.

`DatabaseSeeder` used to call it, which is the seeder these all reach by
default:

```
php artisan db:seed            # runs DatabaseSeeder
php artisan migrate --seed     # the same
php artisan migrate:fresh --seed
```

so any of them would have written demo records into a production ledger. The
only thing standing in the way was this page telling you to name each seeder
with `--class=`.

**There are two real guards now** (`tests/Feature/DemoSeederGuardTest.php`
proves both):

- `DatabaseSeeder` calls `DemoSeeder` only under `local` and `testing`. The
  five it always calls are the real ones — roles, unit types, alert rules,
  the chart of accounts and asset categories.
- `DemoSeeder::run()` refuses outright outside `local` and `testing`, so
  `db:seed --class=DemoSeeder` typed by hand on a production server throws
  instead of seeding.

§6 still names each seeder explicitly, and `deploy.sh` does not seed at all.

---

## 4. The names used in this runbook

Every command below can be pasted as it stands if you use these names. If you
choose others, change them in the same places — `.env`, the systemd unit, the
nginx server block and the cron line.

| Thing | Value used here |
|---|---|
| Public URL | `https://crm.example.ae` |
| Application directory | `/var/www/crm` |
| Unix user that owns the code and runs the worker | `deploy` |
| Web server user | `www-data` |
| MySQL database | `crm` |
| MySQL user | `crm` |
| Backup directory | `/var/backups/crm` |
| systemd unit for the worker | `crm-queue.service` |

Create the user and the directory:

```
sudo adduser --system --group --home /var/www/crm --shell /bin/bash deploy
sudo mkdir -p /var/www/crm
sudo chown deploy:www-data /var/www/crm
```

---

## 5. The database

Create the schema as `utf8mb4` / `utf8mb4_unicode_ci`. The migrations assume
it, and a restore into a differently-collated schema is a slow-burning
problem.

```
sudo mysql -e "CREATE DATABASE crm CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;"
sudo mysql -e "CREATE USER 'crm'@'127.0.0.1' IDENTIFIED BY 'put-a-long-random-password-here';"
sudo mysql -e "GRANT ALL PRIVILEGES ON crm.* TO 'crm'@'127.0.0.1';"
sudo mysql -e "FLUSH PRIVILEGES;"
```

Replace `put-a-long-random-password-here` with a password you generate, for
example with `openssl rand -base64 32`, and put the same value in `.env` as
`DB_PASSWORD`. `ALL PRIVILEGES` on that one schema is what the application
needs: migrations create and alter tables, add generated columns and add
foreign keys.

Store the password in the same place as `APP_KEY`.

---

## 6. First install

Do these in order. Steps 1–5 must be done before anything can serve a request.

### Step 1 — get the code

```
sudo -u deploy git clone https://your-git-host/crm.git /var/www/crm
cd /var/www/crm
```

If this repository has no remote — see §14, it may not — copy the working
tree to the server instead and initialise a repository there:

```
rsync -a --exclude vendor --exclude node_modules --exclude .env \
    --exclude public/build /path/to/crm/ deploy@your-server:/var/www/crm/
```

`deploy.sh --no-pull` deploys what is already on disk, so a remote is a
convenience, not a requirement.

### Step 2 — write `.env`

```
cd /var/www/crm
cp .env.example.production .env
chmod 640 .env
chown deploy:www-data .env
nano .env
```

Edit every line marked `EDIT` in that file. Each setting there carries a
sentence saying what it is for. The ones that will bite you if you skip them:

- `APP_DEBUG=false` — `true` prints stack traces, file paths and `.env`
  contents to whoever triggered the error.
- `APP_URL` — every link in every email and PDF is built from it.
- `MAIL_MAILER` — the development default is `log`, which writes messages to
  `storage/logs` and sends nothing to anyone. Set `smtp` and fill in the host,
  port, username and password, or staff and customers receive nothing.
- `DB_*` — the values from §5.

**Do not copy `.env.example`.** That file is the development one:
`APP_ENV=local`, `APP_DEBUG=true`, `DB_CONNECTION=sqlite`, `MAIL_MAILER=log`.
It is used by CI and by developers, and it is wrong on a server in four
separate ways.

`.env` must exist before step 3: `composer install` runs `php artisan
filament:upgrade` through its `post-autoload-dump` hook, and artisan needs an
environment file to boot.

### Step 3 — install dependencies and generate the key

```
composer install --no-dev --optimize-autoloader --no-interaction
php artisan key:generate
```

`key:generate` writes `APP_KEY` into `.env`. **Run it once, here, and never
again** — §3.1. Copy the value it produced into your password store now,
before you go any further.

### Step 4 — build the front-end

```
npm ci
npm run build
ls -l public/build/manifest.json
```

That last line must print a file. If it does not, stop: the panel cannot
serve a page (§3.2).

### Step 5 — create the schema

```
php artisan migrate --force
```

`--force` skips the interactive confirmation that `APP_ENV=production`
triggers. This is the right flag here; it is not a way around a warning.

### Step 6 — seed the reference data

Run these **by class**, one at a time. Never `php artisan db:seed` on a
production database — it would run `DemoSeeder` (§3.6).

```
php artisan db:seed --class=RolesAndPermissionsSeeder --force
php artisan db:seed --class=ChartOfAccountsSeeder --force
php artisan db:seed --class=AlertRuleSeeder --force
php artisan db:seed --class=UnitTypeSeeder --force
php artisan db:seed --class=AssetCategorySeeder --force
```

What each one is for:

| Seeder | What it creates | Why you need it on day one |
|---|---|---|
| `RolesAndPermissionsSeeder` | the six roles (`super_admin`, `head_office`, `branch_manager`, `sales`, `accounts`, `receptionist`) and the `view_all_branches` permission | nobody can be given a role, and therefore nobody can sign in to the panel, until this has run |
| `ChartOfAccountsSeeder` | the 33 ledger accounts, including 3900 Opening Balance Clearing and 5800 Bad Debts Written Off | every posting the system makes names an account by code; without it the first invoice fails |
| `AlertRuleSeeder` | the 18 expiry-alert horizons | `alerts:scan` has nothing to match against and warns about nothing |
| `UnitTypeSeeder` | the six unit types | the unit importer matches `unit_type` against these by slug or name and refuses every row without them |
| `AssetCategorySeeder` | the six asset categories | the asset register cannot classify anything |

All five are `firstOrCreate`-shaped and safe to run again; running them a
second time changes nothing.

The spec for this milestone names the first three as the ones that matter.
The other two are here because the go-live importers and the asset register
need them on the same first day, and adding them costs nothing.

### Step 7 — create the first branch and the first user

The panel cannot be opened until both exist. After that, every other member
of staff is created in the panel under **Setup → Staff**, not on the server.

**The branch.** There is no command for this one, and it is the only tinker
this install needs: the panel cannot be opened without a user, a user cannot
be created without a branch, and the branch therefore has to come first. Run
it once:

```
php artisan tinker --execute '$branch = App\Models\Branch::firstOrCreate(["code" => "HO"], ["name" => "Head Office", "emirate" => "dubai", "is_active" => true]); echo "branch ".$branch->id." ".$branch->code."\n";'
```

`emirate` must be one of the enum's backing values, all lower case with
underscores: `abu_dhabi`, `dubai`, `sharjah`, `ajman`, `umm_al_quwain`,
`ras_al_khaimah`, `fujairah`. `"Dubai"` throws
`ValueError: "Dubai" is not a valid backing value for enum App\Enums\Emirate`.

**The user.** `crm:create-admin` does this, and nothing about it goes into
your shell history:

```
php artisan crm:create-admin
```

It asks for the name, the email address and the branch (by code or by id),
then asks for the password **twice at a hidden prompt** — nothing is echoed
as you type it and there is deliberately no `--password` option, because an
option would be kept in the shell's history file and visible in `ps` to
every other account on the server while the command ran. Answer the first
three ahead of time if you prefer:

```
php artisan crm:create-admin --name="First Administrator" --email=admin@example.ae --branch=HO
```

It creates the user, assigns `head_office` and attaches the branch, and
prints:

```
Created user #1 admin@example.ae with the head_office role, attached to branch HO.
```

`head_office` is the right first role: `User::canAccessPanel()` returns true
for anyone who either holds `view_all_branches` (which `super_admin` and
`head_office` do) or is attached to a branch. The command does both, which
also makes the branch switcher behave.

It refuses, clearly and without writing anything, if the roles have not been
seeded (step 6), if there is no branch yet, if the branch you named does not
exist, if the email is already taken, or if the two passwords do not match.
So it is safe to run again after a typo.

**Everyone else is created in the panel, not on the server.** This is the
last thing on this install that needs a shell. Sign in as the first
administrator and go to **Setup → Staff**: *New staff member* asks for a
name, an email address, one role, the branches they work at, and a password
typed twice. Only `head_office` and `super_admin` can open that screen.

The rules it enforces are worth knowing before you meet them. Everybody
except `head_office` and `super_admin` needs at least one branch, or they
would be refused at the login page. Nobody can deactivate their own account
or change their own role. And the last active `head_office` or `super_admin`
cannot be deactivated or demoted, so the system can never be left with
nobody able to administer it.

There is no CSV import for staff, deliberately: a password is never a
column, a file or a command-line argument here.

**A leaver is deactivated, never deleted.** Switch *Active* off on their
record. They are refused at the login page immediately, and so is any
session they already had open — the panel re-checks on every request. The
row stays because the activity log, journal entries and the `prepared_by` /
`approved_by` columns reference it.

**One head-office user is not enough for go-live.** `ManualJournalService`
refuses to let anyone post a manual journal they prepared themselves, so the
opening trial balance needs two people — `docs/go-live.md` §4 explains this,
and a single operator working alone can never complete a go-live. Create the
second one under **Setup → Staff** before go-live starts.

### Step 8 — cache and set permissions

```
php artisan optimize
sudo chown -R deploy:www-data /var/www/crm/storage /var/www/crm/bootstrap/cache
sudo chmod -R 775 /var/www/crm/storage /var/www/crm/bootstrap/cache
```

`storage/app/documents` is created the first time a scan is uploaded; the
ownership above covers it.

### Step 9 — the web server, the worker and the scheduler

§7, §8 and §9. Then run the checks in §11.

---

## 7. The web server

nginx in front of PHP-FPM. The document root is `public/`, never the
application directory — `.env`, `storage/` and `vendor/` sit one level above
it and must never be reachable over HTTP.

`/etc/nginx/sites-available/crm`:

```nginx
server {
    listen 80;
    server_name crm.example.ae;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    server_name crm.example.ae;

    root /var/www/crm/public;
    index index.php;

    ssl_certificate     /etc/letsencrypt/live/crm.example.ae/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/crm.example.ae/privkey.pem;

    charset utf-8;
    client_max_body_size 32M;

    add_header X-Content-Type-Options "nosniff" always;
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header Referrer-Policy "same-origin" always;

    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }

    location ~ \.php$ {
        fastcgi_pass unix:/run/php/php8.3-fpm.sock;
        fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
        include fastcgi_params;
        fastcgi_read_timeout 120;
    }

    location ~ /\.(?!well-known).* {
        deny all;
    }

    location = /favicon.ico { access_log off; log_not_found off; }
    location = /robots.txt  { access_log off; log_not_found off; }

    error_page 404 /index.php;
}
```

```
sudo ln -s /etc/nginx/sites-available/crm /etc/nginx/sites-enabled/crm
sudo nginx -t
sudo systemctl reload nginx
```

`client_max_body_size 32M` is there for the go-live CSV uploads and document
scans. Keep PHP's own limits at least as large:

```
sudo sed -i 's/^upload_max_filesize = .*/upload_max_filesize = 32M/' /etc/php/8.3/fpm/php.ini
sudo sed -i 's/^post_max_size = .*/post_max_size = 32M/' /etc/php/8.3/fpm/php.ini
sudo systemctl restart php8.3-fpm
```

---

## 8. The queue worker

One worker, supervised by systemd, started at boot, restarted if it dies.

`/etc/systemd/system/crm-queue.service`:

```ini
[Unit]
Description=CRM queue worker
After=network.target mysql.service
Requires=mysql.service

[Service]
Type=simple
User=deploy
Group=www-data
WorkingDirectory=/var/www/crm

# --queue=default matches config/queue.php's database connection, which
#   takes its queue name from DB_QUEUE (default: "default"). This is the
#   same command docs/go-live.md §2 tells the operator to run by hand.
# --sleep=3    seconds to wait when the jobs table is empty.
# --tries=3    a job that throws is retried twice more, then written to
#              failed_jobs. Check it with `php artisan queue:failed`.
# --timeout=120 kills a job that hangs. It MUST stay below the connection's
#              retry_after (180 seconds, config/queue.php), or the queue
#              would hand the same job to a second worker while the first is
#              still running it.
# --max-time=3600 makes the worker exit once an hour so systemd restarts it,
#              which returns any memory a long import leaked.
ExecStart=/usr/bin/php /var/www/crm/artisan queue:work \
    --queue=default \
    --sleep=3 \
    --tries=3 \
    --timeout=120 \
    --max-time=3600

# A worker that exits after --max-time has "succeeded", so restart on any
# exit, not only on failure. RestartSec keeps a crash loop from spinning.
Restart=always
RestartSec=5

# Stop politely: queue:work traps SIGTERM and finishes the job in hand.
KillSignal=SIGTERM
TimeoutStopSec=130

StandardOutput=append:/var/log/crm-queue.log
StandardError=append:/var/log/crm-queue.log

[Install]
WantedBy=multi-user.target
```

Install it:

```
sudo touch /var/log/crm-queue.log
sudo chown deploy:deploy /var/log/crm-queue.log
sudo systemctl daemon-reload
sudo systemctl enable --now crm-queue.service
sudo systemctl status crm-queue.service
```

Let the deploy user restart it without a password, so `deploy.sh` can:

```
echo 'deploy ALL=(root) NOPASSWD: /usr/bin/systemctl restart crm-queue.service, /usr/bin/systemctl is-active crm-queue.service, /usr/bin/systemctl reload php8.3-fpm.service' | sudo tee /etc/sudoers.d/crm-deploy
sudo chmod 440 /etc/sudoers.d/crm-deploy
sudo visudo -c
```

Check that `/usr/bin/systemctl` is where your distribution puts it — `command
-v systemctl` — and use whatever it prints. sudoers matches the resolved path,
so a rule naming `/bin/systemctl` on a host where it lives at
`/usr/bin/systemctl` silently fails and `deploy.sh` will prompt for a
password.

Rotate the log:

`/etc/logrotate.d/crm-queue`:

```
/var/log/crm-queue.log {
    weekly
    rotate 8
    compress
    missingok
    notifempty
    copytruncate
}
```

### Checking on it

```
sudo systemctl status crm-queue.service
php artisan queue:monitor default
php artisan queue:failed
```

`queue:monitor` prints the number of pending and delayed jobs. A healthy idle
system looks like this:

```
  Queue name ................................. Size / Status
  [database] default ................................ [0] OK
  Pending jobs .......................................... 0
  Delayed jobs .......................................... 0
```

A number that climbs and never falls means the worker is not running or is
stuck. A job that gave up three times is in `failed_jobs`; retry it with
`php artisan queue:retry all` after you have fixed whatever made it fail.

### One worker or more

One is right for a single business centre. Filament's importers run one job
per chunk of CSV rows, so a large go-live file is already several jobs — but
they touch the same leases, invoices and ledger accounts, and running them in
parallel buys little and risks lock contention. If you do add a second worker,
copy the unit to `crm-queue-2.service`; do not add `--max-jobs` or raise
`--timeout` above 180.

---

## 9. The scheduler

One cron line. Add it to the `deploy` user's crontab:

```
sudo crontab -u deploy -e
```

and add exactly this line:

```
* * * * * cd /var/www/crm && /usr/bin/php artisan schedule:run >> /var/log/crm-schedule.log 2>&1
```

```
sudo touch /var/log/crm-schedule.log
sudo chown deploy:deploy /var/log/crm-schedule.log
sudo crontab -u deploy -l
```

It runs every minute; Laravel decides internally which of the eight commands
is due. The times are UAE wall-clock time, because the scheduler falls back to
`config('app.timezone')` — see §3.3. Check what it thinks is scheduled:

```
php artisan schedule:list
```

which prints:

```
  45 6 * * *  php artisan leases:update-statuses ....... Next Due: ...
  0  7 * * *  php artisan alerts:scan .................. Next Due: ...
  5  0 * * *  php artisan documents:ensure-series ...... Next Due: ...
  30 6 * * *  php artisan bookings:expire-holds ........ Next Due: ...
  15 7 * * *  php artisan reminders:suggest ............ Next Due: ...
  30 7 * * *  php artisan followups:notify ............. Next Due: ...
  45 7 * * *  php artisan billing:run --all-branches ... Next Due: ...
  0  * * * *  php artisan tickets:notify-overdue ....... Next Due: ...
```

Those cron expressions are Dubai time: `45 6 * * *` is 06:45 in the UAE.

The table below is not prose: `tests/Feature/ScheduleTest.php` parses it
and compares it to the schedule the application actually registers, in
both directions. A command moved, dropped or added without this table
following it fails the suite.

### What each one does, and what breaks without it

None of the eight needs the queue worker — they do their work inline. Only
`alerts:scan` sends email, and it sends it synchronously.

| Command | When | What it does | If the scheduler never runs |
|---|---|---|---|
| `documents:ensure-series` | 00:05 daily | For every branch, inserts the missing `document_sequences` rows for the current year (invoice, quote, receipt and the rest), each starting at 1. | Numbering still works, but the first document of a new year for each branch has to create its own sequence row under contention — the deadlock this command exists to prevent. Also affects any branch created before the auto-create hook shipped. |
| `bookings:expire-holds` | 06:30 daily | Cancels `Tentative` room bookings older than `config('bookings.tentative_hold_hours')`, releases their credit hours and zeroes the charge. | Abandoned holds sit on the room calendar for ever. The slots look booked and staff cannot offer them to anyone. |
| `leases:update-statuses` | 06:45 daily | Moves every `Active` lease past its notice window to `Expiring` and every lease past its end date to `Expired`, and recomputes the status of each attached unit. | An expired lease stays Active for ever and its unit stays Occupied for ever. Availability lists, the dashboard and any new allocation on that unit are wrong, and nothing else in the application fixes it. |
| `alerts:scan` | 07:00 daily | Sweeps customer and employee documents, trade licences, cheques due for deposit, asset warranties and service dates against the seeded alert horizons; sends `ExpiringItemNotification` to the eligible users and records the send so it is not repeated. | Nobody is ever warned about an expiring trade licence, visa, passport or warranty, or a cheque due for deposit. Compliance items lapse silently and the alert bell stays empty. **This is the only scheduled command that sends mail**, so it is the one that proves `MAIL_MAILER` is configured. |
| `reminders:suggest` | 07:15 daily | Sends a database-only notice to each branch's billing users listing the receivables worth chasing today. | Billing staff get no daily nudge. Nothing goes stale — the Payment Reminders page computes its list live — but overdue invoices quietly age. |
| `followups:notify` | 07:30 daily | Sends a database-only notice to each sales owner whose open enquiries are due or overdue for follow-up. | No daily prompt. The My Follow-ups page still works; follow-ups just age into overdue unnoticed. |
| `billing:run --all-branches` | 07:45 daily | Drafts one invoice per customer, per active branch, for every unbilled lease instalment and room charge due today or earlier. **Drafts only** — see below. | Nobody is invoiced. Instalments fall due and age with no document behind them, the ageing report has nothing to show and no tenant is ever asked for money. |
| `tickets:notify-overdue` | hourly | Stamps `tickets.overdue_notified_at` and notifies the assignee, or the branch managers when nobody eligible is assigned. | `overdue_notified_at` stays null on every ticket and no SLA breach is ever surfaced. Tickets go overdue in silence. |

Seven of the eight take no arguments, and all eight are safe to run by hand
at any time. Catching up after an outage is one command each:

```
cd /var/www/crm
php artisan documents:ensure-series
php artisan bookings:expire-holds
php artisan leases:update-statuses
php artisan alerts:scan
php artisan reminders:suggest
php artisan followups:notify
php artisan billing:run --all-branches
php artisan tickets:notify-overdue
```

Running them twice in one day does not double anything: the three
notification commands carry same-day guards, `tickets:notify-overdue` stamps
each ticket once and only once, `billing:run` never drafts an instalment a
draft line already claims, and the other three re-read their candidates and
change nothing the second time. So a week of missed runs produces one notice
per subject, not seven, and one draft per instalment, not seven.

### The billing run drafts. It never issues.

`php artisan billing:run {branch}`, or `billing:run --all-branches`, drafts
one invoice per customer for every unbilled lease instalment and room charge
due up to a date. Exactly one of the two is required: a bare `billing:run`
refuses rather than guessing that you meant every branch.

It is scheduled daily at 07:45 Dubai (`routes/console.php`), after the 06:45
lease sweep and before the counter opens, so the day's drafts are waiting
when billing staff sign in. There is also a **Draft due invoices** button in
the header of Billing → Invoices, which does the same thing for the branch
the user is in; it is open to the billing roles, exactly as **New invoice**
is.

What it produces are **drafts**: no number is assigned, nothing is posted to
the ledger and nothing reaches a tenant. Somebody reviews each draft and
issues it. `--issue` is refused by the command, deliberately, and the
scheduled entry does not pass it.

Two things make the daily run safe:

- An instalment already sitting on an invoice line is never drafted again,
  so a second run the same day drafts nothing rather than double-billing.
- A branch whose run throws is reported and the run carries on to the next
  branch; the command exits non-zero and names every branch that failed.
  Each branch's run is its own transaction, so a failure rolls back only
  itself.

`--all-branches` covers the **active** branches. A deactivated branch is a
centre the business has closed and is left out; it is still billable by name
if you need to.

Read the log the first few mornings:

```
grep billing /var/log/crm-schedule.log
```

---

## 10. Every deploy after the first

From the application directory:

```
cd /var/www/crm
./deploy.sh
```

**Seeders are not part of a deploy, and two of them sometimes need to be.**
`deploy.sh` runs migrations, never seeders, which is right: a seeder that ran
on every deploy would be a standing risk to live data. But the reference data
some seeders own does grow between releases, and an install that seeded at §6
and never again is missing whatever a later release added.

Two seeders own reference data that grows:

- **`AlertRuleSeeder`.** A release that adds an alert type adds its horizons
  there. Without the new rows that alert watches nothing, silently, for ever.
  It is `updateOrCreate`-shaped and touches nothing but `alert_rules`.
- **`ChartOfAccountsSeeder`.** A release that posts to a new account adds it
  there. It is `firstOrCreate`-shaped, so re-running it never overwrites an
  account your accountant renamed, and the only other thing it does is
  re-apply the system-controlled flag to the codes that must carry it.

Both are safe to re-run at any time and change nothing when there is nothing
new:

```
php artisan db:seed --class=AlertRuleSeeder --force
php artisan db:seed --class=ChartOfAccountsSeeder --force
```

Run them after any release whose notes mention a new alert or a new account.
If you are unsure, run them - the cost is a query per row and the alternative
is a compliance alert that never fires or a posting that cannot resolve its
account. On a current install
`php artisan tinker --execute 'echo App\Models\AlertRule::count();'` should
print 18, and
`php artisan tinker --execute 'echo App\Models\Account::where("code","5800")->exists();'`
should print 1.

That second check names the account the release added rather than counting
the chart, deliberately. A count is a floor - an accountant adds their own
accounts, so it can only ever say "at least 33" - and a floor cannot fail
informatively: an install *missing* the new account still passes it. Check
for the account the release notes name, and the check stays true when the
next one is added.

New accounts also ship with a data migration, so `deploy.sh` installs them
without your help; re-running the chart seeder is the belt to that braces.

`deploy.sh` is committed at the repository root. It is safe to run twice —
running it again on an unchanged checkout reinstalls the same dependencies,
rebuilds the same assets, prints `Nothing to migrate.` and restarts the
worker. It does what §6 does, minus the parts that must happen only once:

1. Refuses to run if `.env` is missing or `APP_KEY` is empty.
2. Refuses to run unless Node satisfies `^20.19.0 || >=22.12.0` (§2).
3. `git pull --ff-only` on the current branch (skip with `--no-pull`).
4. `composer install --no-dev --optimize-autoloader`.
5. `npm ci && npm run build`, then checks `public/build/manifest.json` exists
   and aborts if it does not.
6. `php artisan migrate --force`.
7. `php artisan optimize:clear` then `php artisan optimize` — config, routes,
   views and events, in that order. Clearing first means a stale config cache
   from the previous release can never be read while the new one is written.
8. `php artisan storage:link` (idempotent, and irrelevant to document scans).
9. `php artisan queue:restart` and `systemctl restart crm-queue.service`.
10. `systemctl reload php8.3-fpm` so the new code leaves opcache.

It never writes `.env`, never generates a key and never seeds.

```
./deploy.sh --no-pull    # deploy what is already checked out
```

### A note on `config:cache`

Once configuration is cached, `env()` returns `null` everywhere outside
`config/`. That is a Laravel rule, not a choice this project made, and this
application already respects it. If you ever add an `env()` call to
application code it will work in development and return `null` on the server.

### Zero-downtime

This sequence has a short window — between `migrate` and `optimize` — where
new code is running against a cache that has not been rebuilt. For a
single-tenant internal panel that is a few seconds at a quiet hour, and it is
not worth the complexity of a release-directory symlink scheme. If you want
the window gone, put the site into maintenance mode around the deploy:

```
php artisan down --retry=60
./deploy.sh
php artisan up
```

`php artisan down` returns 503 to everyone, including staff.

---

## 11. Post-deploy checks

Run all six. They take a minute.

> **One expected failure before the first branch exists.** The 07:45
> `billing:run --all-branches` exits non-zero with "There are no active
> branches to bill." until §7's Head Office branch has been created. That is
> deliberate rather than a fault: every other scheduled command treats
> nothing-to-do as success, but for a running business every branch being
> inactive means billing has silently stopped, which should be loud. If you
> complete §7 during setup you will never see it; if you deploy and leave it
> overnight, this is the line in cron's output and it needs no action.
> Verified on a fresh install: the other seven scheduled commands all exit 0
> against an empty database.

**1. Debug mode is off.**

```
php artisan about --only=environment
```

`Debug Mode` must read `OFF` and `Environment` must read `production`. If
`Debug Mode` says `ON`, fix `.env` and run `php artisan optimize` again before
anyone touches the site.

**2. The login page loads.**

```
curl -s -o /dev/null -w '%{http_code}\n' https://crm.example.ae/up
curl -s -o /dev/null -w '%{http_code}\n' https://crm.example.ae/admin/login
```

Both must print `200`. A `500` on `/admin/login` with `/up` at `200` is
almost always the missing Vite manifest — §3.2.

Then open it in a browser and check it is styled: the product name, the mark
and the teal primary colour. An unstyled page means the built CSS is not
being served.

**3. The schedule lists in UAE time.**

```
php artisan schedule:list
```

Eight commands, `45 6 * * *` for `leases:update-statuses` and `45 7 * * *`
for `billing:run --all-branches`. That is 06:45 and 07:45 Dubai. If your
server clock is UTC, "Next Due" will still be right — the scheduler
converts.

**4. A queued job completes.**

```
sudo systemctl is-active crm-queue.service
php artisan queue:monitor default
```

`active`, and `[0] OK`. The real test is the first go-live import: upload a
small branches CSV on Setup → Go-live and watch rows appear within a few
seconds. If `queue:monitor` climbs and stays up, the worker is not consuming.

**5. A document scan downloads.**

Upload a scan against any document record in the panel, then click its
download link. The file must come back as an attachment. If it 404s, check
that `storage/app/documents` exists and is writable by both `deploy` and
`www-data`.

**6. Mail actually leaves.**

```
php artisan alerts:scan
```

This is the only scheduled command that sends email. With `MAIL_MAILER=log`
it writes to `storage/logs/laravel-*.log` and sends nothing — which is the
symptom to look for if staff say the alerts never arrive.

---

## 12. Backups

Two things must be in the backup, and a database dump is only one of them:

1. **MySQL** — everything except the scans.
2. **`storage/app/documents`** — the scans (§3.4). Not in the database, not in
   the repository.

And two things must be stored somewhere else entirely, because the backup
cannot restore without them:

3. **`APP_KEY`** — without it the restored IBANs are unreadable for ever
   (§3.1).
4. **`DB_PASSWORD`** and the SMTP credentials.

`.env` holds 3 and 4 and is gitignored, so back it up too — to your password
store or an encrypted location, **not** next to the database dump.

### Credentials for the backup

Put the database password in a file `mysqldump` reads, rather than on its
command line, where `ps` would show it to every user on the box:

```
sudo tee /root/.crm-backup.cnf > /dev/null <<'EOF'
[client]
host=127.0.0.1
user=crm
password=put-the-same-password-you-set-in-section-5-here
EOF
sudo chmod 600 /root/.crm-backup.cnf
```

### The nightly script

Create it:

```
sudo mkdir -p /var/backups/crm
sudo chmod 700 /var/backups/crm
sudo tee /usr/local/bin/crm-backup.sh > /dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

APP_DIR=/var/www/crm
BACKUP_DIR=/var/backups/crm
STAMP="$(date +%Y%m%d-%H%M%S)"
KEEP_DAYS=30

mkdir -p "$BACKUP_DIR"

# --defaults-extra-file must be the FIRST argument mysqldump sees.
# --single-transaction takes a consistent snapshot without locking writers.
# --routines --triggers --events carry anything MySQL holds outside tables.
# --set-gtid-purged=OFF keeps the dump loadable into a server that is not a
#   replica of this one, which is exactly what a restore rehearsal is.
mysqldump \
    --defaults-extra-file=/root/.crm-backup.cnf \
    --single-transaction \
    --quick \
    --routines --triggers --events \
    --set-gtid-purged=OFF \
    --default-character-set=utf8mb4 \
    crm | gzip -9 > "$BACKUP_DIR/db-$STAMP.sql.gz"

# The scans. -C means the archive contains "documents/...", not the whole
# absolute path, so it unpacks anywhere.
tar -czf "$BACKUP_DIR/documents-$STAMP.tar.gz" -C "$APP_DIR/storage/app" documents

sha256sum "$BACKUP_DIR/db-$STAMP.sql.gz" "$BACKUP_DIR/documents-$STAMP.tar.gz" \
    > "$BACKUP_DIR/checksums-$STAMP.txt"

find "$BACKUP_DIR" -type f -mtime +$KEEP_DAYS -delete

echo "Backed up to $BACKUP_DIR (stamp $STAMP)"
ls -l "$BACKUP_DIR" | tail -n 5
EOF
sudo chmod 750 /usr/local/bin/crm-backup.sh
```

Run it once by hand before you trust the cron line:

```
sudo /usr/local/bin/crm-backup.sh
```

Nightly at 01:30 Dubai time — the server clock is UTC (§3.3), so that is
21:30 UTC. Before `documents:ensure-series` at 00:05 Dubai and well clear of
the morning sweeps:

```
sudo crontab -e
```

```
30 21 * * * /usr/local/bin/crm-backup.sh >> /var/log/crm-backup.log 2>&1
```

**Copy the files off this machine.** A backup on the same disk as the database
is not a backup. `rsync` to another host, or push to object storage:

```
30 22 * * * rsync -a --delete /var/backups/crm/ backup@elsewhere:/srv/crm-backups/
```

---

## 13. Restore

### Rehearse it before you need it

> Rehearsed locally on 2026-09-27 against a freshly installed database
> carrying a full demo dataset, with the exact `mysqldump` flags above:
> the schema comparison printed nothing, all 82 tables came back, and
> `journal_lines` still balanced to the fil (`SUM(debit) - SUM(credit) = 0`)
> after the round trip. The procedure below is known to work as written; what
> has never been exercised is the cron that runs it on a server, and the
> rsync that copies it off the box.

Restore into a **scratch** database with a name nothing else uses. Never
restore over the live one to "test" it.

```
mysql --host=127.0.0.1 --user=root -e "DROP DATABASE IF EXISTS crm_restore_check; CREATE DATABASE crm_restore_check CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;"
gunzip -c /var/backups/crm/db-20260923-013000.sql.gz > /tmp/restore.sql
mysql --host=127.0.0.1 --user=root --default-character-set=utf8mb4 crm_restore_check -e "source /tmp/restore.sql"
```

(Substitute the timestamp of the backup you are restoring; `ls
/var/backups/crm` lists them.)

Verify the restored schema matches the source. This prints **nothing at all**
when they are identical:

```
mysql --host=127.0.0.1 --user=root --table -e "
SELECT COUNT(*) AS column_definitions_that_differ FROM (
  SELECT table_name, column_name, column_type, is_nullable, column_default, extra, collation_name, character_set_name FROM information_schema.columns WHERE table_schema='crm'
  UNION ALL
  SELECT table_name, column_name, column_type, is_nullable, column_default, extra, collation_name, character_set_name FROM information_schema.columns WHERE table_schema='crm_restore_check'
) c GROUP BY table_name, column_name, column_type, is_nullable, column_default, extra, collation_name, character_set_name HAVING COUNT(*) <> 2;
SELECT COUNT(*) AS index_definitions_that_differ FROM (
  SELECT table_name, index_name, seq_in_index, column_name, non_unique FROM information_schema.statistics WHERE table_schema='crm'
  UNION ALL
  SELECT table_name, index_name, seq_in_index, column_name, non_unique FROM information_schema.statistics WHERE table_schema='crm_restore_check'
) i GROUP BY table_name, index_name, seq_in_index, column_name, non_unique HAVING COUNT(*) <> 2;
SELECT COUNT(*) AS foreign_keys_that_differ FROM (
  SELECT constraint_name, table_name, referenced_table_name FROM information_schema.referential_constraints WHERE constraint_schema='crm'
  UNION ALL
  SELECT constraint_name, table_name, referenced_table_name FROM information_schema.referential_constraints WHERE constraint_schema='crm_restore_check'
) f GROUP BY constraint_name, table_name, referenced_table_name HAVING COUNT(*) <> 2;"
```

Do **not** verify by diffing two `mysqldump` outputs. A restored schema is
functionally identical but mysqldump renders it differently: it repeats
`CHARACTER SET utf8mb4` on each column where the source omitted it, because
the restore marked the column charset explicit. The query above compares what
the server actually holds and is not fooled by that.

Then check the row counts and that migrations are complete:

```
mysql --host=127.0.0.1 --user=root --table -e "
SELECT 'migrations' t, COUNT(*) n FROM crm_restore_check.migrations
UNION ALL SELECT 'accounts', COUNT(*) FROM crm_restore_check.accounts
UNION ALL SELECT 'branches', COUNT(*) FROM crm_restore_check.branches
UNION ALL SELECT 'users', COUNT(*) FROM crm_restore_check.users
UNION ALL SELECT 'invoices', COUNT(*) FROM crm_restore_check.invoices
UNION ALL SELECT 'leases', COUNT(*) FROM crm_restore_check.leases;"
```

Unpack the scans into a scratch directory and compare a file:

```
mkdir -p /tmp/crm-restore-check
tar -xzf /var/backups/crm/documents-20260923-013000.tar.gz -C /tmp/crm-restore-check
find /tmp/crm-restore-check -type f | head
sha256sum /tmp/crm-restore-check/documents/1/passport.pdf /var/www/crm/storage/app/documents/1/passport.pdf
```

Then clean up:

```
mysql --host=127.0.0.1 --user=root -e "DROP DATABASE crm_restore_check;"
rm -rf /tmp/crm-restore-check /tmp/restore.sql
```

**This rehearsal has been performed**, on 23 September 2026, against MySQL
8.4.3 with a scratch source database (`crm_deploy_check`) and a scratch target
(`crm_restore_check`). What it showed:

- 82 tables in, 82 tables out.
- The `information_schema` query above returned nothing: not one column
  definition, index or foreign key differed.
- Hashing every `INSERT` line of a fresh dump of each database gave the same
  digest on both sides — `4321c3394db5908e…` — so every row round-tripped
  exactly.
- `php artisan migrate --force` against the restored database printed
  `Nothing to migrate.`, and the application booted against it: the seeded
  accounts, the branch and the user were all there and
  `canAccessPanel()` was true.
- The scans' archive unpacked to a file with the same SHA-256 as the original.

A raw `diff` of the two dumps was the one thing that was *not* clean, for the
cosmetic reason given above. That is why the verification here is the
`information_schema` query and not a dump diff.

### The real thing

To bring the system back on a new server:

1. Install the requirements (§2) and create the database and user (§5).
2. Deploy the code: §6 steps 1, 3 and 4 — clone, `composer install`,
   `npm ci && npm run build`. **Skip step 5 (`migrate`) and step 6
   (the seeders).** The dump already contains the schema, the migration
   history and the reference data. Running the seeders on top of a restore
   would be harmless for the five listed, but `migrate` on an empty schema
   before the restore is not — restore first, then check.
3. Put back the **original `.env`**, with the **original `APP_KEY`**. A new
   key here makes every employee IBAN unreadable and you will not find out
   until somebody opens a payroll record (§3.1).
4. Load the dump:

   ```
   gunzip -c /var/backups/crm/db-20260923-013000.sql.gz > /tmp/restore.sql
   mysql --host=127.0.0.1 --user=crm --password --default-character-set=utf8mb4 crm -e "source /tmp/restore.sql"
   rm /tmp/restore.sql
   ```

5. Unpack the scans:

   ```
   tar -xzf /var/backups/crm/documents-20260923-013000.tar.gz -C /var/www/crm/storage/app
   sudo chown -R deploy:www-data /var/www/crm/storage/app/documents
   ```

6. Confirm the schema is current, then cache and start:

   ```
   cd /var/www/crm
   php artisan migrate --force        # expect "Nothing to migrate."
   php artisan optimize
   sudo systemctl start crm-queue.service
   ```

   If `migrate` reports pending migrations, the backup predates the code you
   just deployed. That is fine — it applies them — but read what it is about
   to run first.

7. Run every check in §11, and one more: open an employee record and confirm
   the IBAN is readable. That is the only thing that proves you restored the
   right `APP_KEY`.

---

## 14. Continuous integration

`.github/workflows/tests.yml` runs the suite on every push and pull request:
Ubuntu, PHP 8.3, a MySQL 8.4 service, `composer install`, `npm ci && npm run
build`, then `php artisan test --compact`.

Two details in it are not decoration:

- **Node is pinned to `22.12`,** not `22`. See §2.
- **The MySQL service is published on host port 3307,** because `phpunit.xml`
  hard-codes `DB_PORT=3307` and `DB_DATABASE=crm_testing`. Matching the file
  rather than overriding it keeps CI and a developer machine on the same
  configuration.

The workflow builds the front-end before running the tests for the same reason
§3.2 gives: a feature test that renders a panel page reads the Vite manifest.

### If this repository has no remote

At the time this was written, `git remote -v` in this repository printed
nothing. **There is no remote, so nothing is running this workflow.** The file
is committed anyway, so that it works the day one is added.

To give it one:

```
cd /var/www/crm          # or your working copy
git remote -v            # confirm it is empty
gh repo create your-org/crm --private --source=. --remote=origin
git push -u origin HEAD
```

or, with a repository you created in the GitHub UI:

```
git remote add origin git@github.com:your-org/crm.git
git push -u origin HEAD
```

The workflow runs on the first push. Until then, run the suite by hand before
every deploy:

```
php artisan test --compact
```

and note the memory rule this project already has: the suite must be pointed
at a test database explicitly (`DB_DATABASE=crm_testing`), because there is no
`.env.testing` and `--env=testing` would otherwise wipe the development
database.

### What has never been tested, and what was checked instead

**This suite has only ever run on Windows.** CI would be the first time it
runs on Linux, and CI has never run. That matters because Linux is
case-sensitive about filenames and Windows is not: a view, a class or a path
referred to in the wrong case works on a developer machine and is a fatal
error on the server.

That class of defect was audited statically on 2026-09-26, because it can be
found without a Linux box:

| Checked | How | Result |
|---|---|---|
| PSR-4 class/file case | `composer dump-autoload --optimize --strict-psr` | clean, 11,077 classes |
| Blade view names | every `view()`, `$view`, `@include`, `@extends` compared case-exactly against the files on disk | 26 references, all exact |
| Literal paths | every `base_path`/`app_path`/`public_path`/`storage_path`/`database_path` string | all exact |
| Seeder names in this runbook | each `--class=` compared to `database/seeders` | all exact |
| Line endings | `deploy.sh` carriage returns, and every tracked file's index EOL | zero CRs; `.gitattributes` pins `eol=lf` repository-wide |

The line-ending one is worth naming: a `deploy.sh` saved with CRLF on Windows
fails on Linux with `bad interpreter: /bin/bash^M`, which reads like a broken
shell rather than a broken file. `.gitattributes` already prevents it, and the
file is LF in both the index and the working tree.

**None of this replaces running the suite on Linux.** It rules out the static,
findable half — the half that would otherwise turn the first deploy into a
debugging session. Runtime differences (file permissions, the `storage`
symlink, locale, timezone data, case-sensitivity inside data rather than
code) are still unproven until CI or the first deploy actually runs.

---

## 15. Handing off to go-live

At this point you have a working, empty system: schema, reference data, one
branch, one head-office user, a running worker and a running scheduler.

**`docs/go-live.md` takes over from here.** It covers bringing a real business
centre in: the branch's go-live date, the eight imports in order, the opening
trial balance, the readiness checks and completion.

Three things in this runbook exist because that document needs them:

- The **queue worker** must be running for the whole go-live. `docs/go-live.md`
  §2 says to start one by hand with `php artisan queue:work --queue=default`;
  §8 above makes it a supervised service instead, which is the same worker on
  the same queue, always running. If you followed §8, you do not need to start
  one by hand.
- **Two users** are needed, not one. A manual journal is prepared by one person
  and posted by another, and `ManualJournalService` refuses to let anyone post
  their own. A lone operator can never bring 3900 to zero and can never
  complete a go-live (`docs/go-live.md` §4).
- **`ChartOfAccountsSeeder` must have run**, because the opening balances post
  against account 3900 Opening Balance Clearing.

Nothing in this runbook changes anything `docs/go-live.md` says.

---

## 16. When something is wrong

| Symptom | Almost certainly |
|---|---|
| Every admin page 500s, `/up` is fine | `public/build/manifest.json` is missing. Run `npm ci && npm run build` (§3.2). Check `storage/logs` for `ViteException`. |
| `npm ci` or `npm run build` fails, mentioning a native binding or `paths[1]` | Node is outside `^20.19.0 || >=22.12.0` (§2). Neither error names the Node version, and the native-binding one wrongly tells you to delete `package-lock.json` — do not. Upgrade Node. |
| Stack traces in the browser | `APP_DEBUG=true`. Fix `.env`, then `php artisan optimize`. |
| An import says "Import started" and nothing happens | The queue worker is not running. `sudo systemctl status crm-queue.service`, then `php artisan queue:monitor default`. |
| Leases never expire; units stay Occupied | The scheduler is not running. `sudo crontab -u deploy -l`, then `php artisan schedule:list`. |
| Nobody receives expiry alerts or reminders | `MAIL_MAILER=log`. Everything is in `storage/logs` and nothing was sent (§6 step 2). |
| Nobody is ever invoiced | The scheduler is not running, so the 07:45 `billing:run --all-branches` never fires (§9). Check `grep billing /var/log/crm-schedule.log`, or draft by hand from Billing → Invoices → **Draft due invoices**. |
| Drafts appear but nothing reaches a tenant | That is by design. `billing:run` drafts; issuing is a separate human step (§9). |
| Signed in, then signed straight out again | The site is on plain HTTP with `APP_ENV=production`. The session cookie is marked Secure by default, so the browser never sends it back. Serve it over HTTPS (§7). |
| Times are four hours out | Something has overridden `APP_TIMEZONE` or `DB_TIMEZONE`. Both must stay at their defaults (§3.3). |
| An employee's IBAN is unreadable | `APP_KEY` has changed. Put the original back. There is no other recovery (§3.1). |
| A config change has no effect | The config cache is stale. `php artisan optimize:clear && php artisan optimize`. |
| A document scan 404s | `storage/app/documents` is missing, unreadable by `www-data`, or the file is genuinely gone. Authorisation failures give 403, not 404. |
| A job is in `failed_jobs` | `php artisan queue:failed` to see it, fix the cause, `php artisan queue:retry all`. |

Logs:

```
tail -f /var/www/crm/storage/logs/laravel-$(date +%Y-%m-%d).log
tail -f /var/log/crm-queue.log
tail -f /var/log/crm-schedule.log
sudo journalctl -u crm-queue -n 100 -f
sudo tail -f /var/log/nginx/error.log
```