# Sehat Sahoolat Production Deployment Guide

This document is the current production deployment runbook for the Sehat
Sahoolat v2 monorepo. It covers server specifications, DNS, environment
variables, provisioning, deployment, health checks, rollback, backups, and
mobile release configuration.

The production stack is designed for one application host plus managed
database/storage services:

- Public site: `https://sehatsahoolat.com`
- Patient portal: `https://app.sehatsahoolat.com`
- Doctor portal: `https://doctor.sehatsahoolat.com`
- Admin portal: `https://admin.sehatsahoolat.com`
- Backend API: `https://api.sehatsahoolat.com/api/v2`

## 1. Production Architecture

```text
Internet / Cloudflare
        |
        | HTTPS 443
        v
Ubuntu application server
  - nginx TLS reverse proxy
  - NestJS backend container
  - Next.js public-site container
  - Next.js patient-portal container
  - Next.js doctor-portal container
  - Next.js admin-portal container
  - Redis container
  - Whisper sidecar container
        |
        | TLS Postgres connection
        v
Managed PostgreSQL 16 with pgvector

Object storage:
  - DigitalOcean Spaces or S3-compatible storage for uploads/backups
```

Production compose file:

```bash
deploy/docker-compose.prod.yml
```

Provisioning script:

```bash
deploy/scripts/provision.sh
```

Deployment script:

```bash
deploy/scripts/deploy.sh
```

## 2. Server Specifications

### 2.1 Recommended Production Server

Use this for the first live launch.

| Layer | Recommended spec |
| --- | --- |
| Provider | DigitalOcean, AWS Lightsail/EC2, Hetzner, or equivalent VPS |
| Region | Closest stable region to Pakistan users; DigitalOcean BLR1 is a good fit |
| OS | Ubuntu 24.04 LTS x64 |
| CPU | 4 vCPU minimum |
| RAM | 8 GB minimum |
| Disk | 160 GB SSD minimum |
| Swap | 4 GB |
| Network | Public IPv4, 1 Gbps preferred |
| Docker | Docker Engine with Compose plugin |
| Timezone | `Asia/Karachi` |
| Open ports | `22`, `80`, `443` only |

Expected workload for this shape:

- Public marketing site traffic
- Patient, doctor, and admin portals
- API traffic for normal telehealth workflows
- Redis cache/session/rate-limit store
- CPU-based Whisper sidecar with `small` model
- Small to medium launch traffic

### 2.2 Minimum Staging Server

Use only for staging, demos, or internal QA.

| Layer | Minimum staging spec |
| --- | --- |
| CPU | 2 vCPU |
| RAM | 4 GB |
| Disk | 80 GB SSD |
| Swap | 4 GB |
| Whisper | `base` or `small`, CPU, light usage only |

Do not use this for production if real consult traffic or transcription is
expected.

### 2.3 Scale-Up Production Server

Move to this once usage grows or transcription becomes heavy.

| Layer | Scale-up spec |
| --- | --- |
| App server | 8 vCPU, 16 GB RAM, 240+ GB SSD |
| Database | Managed PostgreSQL 4 vCPU, 8 GB RAM or better |
| Whisper | Separate 4 vCPU, 8 GB worker host, or GPU worker if transcription load grows |
| Redis | Managed Redis or dedicated Redis container with 512 MB to 1 GB max memory |
| Storage | S3-compatible bucket with lifecycle rules and backups |

### 2.4 Managed PostgreSQL Specification

Use managed PostgreSQL in production rather than a local database container.

| Setting | Value |
| --- | --- |
| Version | PostgreSQL 16 |
| Extensions | `pgvector`, `pgcrypto`, `uuid-ossp` |
| Minimum size | 2 vCPU, 4 GB RAM |
| Backups | Daily backups plus point-in-time recovery if provider supports it |
| SSL | Required |
| Connection pool | Transaction pool, 25 connections to start |
| Database name | `sehat_sahoolat` |
| Port | Provider default, commonly `25060` on DigitalOcean managed PG |

### 2.5 Redis Specification

The current production compose runs Redis as an internal container.

| Setting | Value |
| --- | --- |
| Image | `redis:7-alpine` |
| Network exposure | Internal Docker network only |
| Max memory | `256mb` initially |
| Eviction policy | `allkeys-lru` |
| Password | Required via `REDIS_PASSWORD` |

For higher traffic, move Redis to a managed Redis instance or increase max
memory to `512mb` or `1gb`.

### 2.6 Whisper Specification

The compose stack runs a faster-whisper sidecar.

| Setting | Launch value |
| --- | --- |
| Model | `small` |
| Device | `cpu` |
| Compute type | `int8` |
| Cache volume | `whisper_cache` |

If doctors frequently use live transcription, move Whisper to a separate host
or GPU worker. Do not let transcription compete with the main API under heavy
load.

## 3. Required DNS Records

Point these records to the application server public IP.

| Type | Host | Target |
| --- | --- | --- |
| A | `sehatsahoolat.com` | App server IPv4 |
| A | `app.sehatsahoolat.com` | App server IPv4 |
| A | `doctor.sehatsahoolat.com` | App server IPv4 |
| A | `admin.sehatsahoolat.com` | App server IPv4 |
| A | `api.sehatsahoolat.com` | App server IPv4 |

Recommended DNS settings:

- TTL: `300` during deployment, `Auto` or `3600` after stable launch
- Cloudflare SSL mode: `Full (strict)`
- Always use HTTPS: enabled
- Minimum TLS: `1.2`
- Bypass cache for `/.well-known/acme-challenge/*`

## 4. Required Server Software

Install these through `deploy/scripts/provision.sh` or manually:

| Software | Purpose |
| --- | --- |
| Ubuntu 24.04 LTS | Base OS |
| Docker Engine | Container runtime |
| Docker Compose plugin | Production orchestration |
| nginx container | TLS reverse proxy |
| UFW | Firewall |
| fail2ban | SSH protection |
| acme.sh | Let's Encrypt certificates |
| postgresql-client-16 | Database smoke checks and manual SQL access |
| s3cmd | Spaces/S3 backup operations |
| git | Pull deployments |
| openssl | Secret generation |

## 5. Firewall and Security Baseline

Only these inbound ports should be open:

| Port | Protocol | Purpose |
| --- | --- | --- |
| 22 | TCP | SSH |
| 80 | TCP | HTTP redirect and ACME challenge |
| 443 | TCP | HTTPS application traffic |

Production baseline:

- SSH key authentication only
- Disable SSH password login
- Disable root password login
- Enable unattended security upgrades
- Enable UFW
- Enable fail2ban
- Store production secrets only in `/etc/sehat/.env`
- Set `/etc/sehat/.env` permissions to `0640`
- Keep backend, Redis, and Whisper off public host ports

## 6. Production Environment File

Production secrets live on the server at:

```bash
/etc/sehat/.env
```

Never commit this file.

Minimum required production variables:

```bash
NODE_ENV=production
TZ=Asia/Karachi

DB_HOST=<managed-postgres-host>
DB_PORT=25060
DB_USERNAME=<managed-postgres-user>
DB_PASSWORD=<managed-postgres-password>
DB_DATABASE=sehat_sahoolat
DB_SSL=true

REDIS_PASSWORD=<generated-secret>

JWT_SECRET=<openssl-rand-hex-32>
JWT_EXPIRES_IN=12h
JWT_REFRESH_SECRET=<openssl-rand-hex-32>
JWT_REFRESH_EXPIRES_IN=7d
ENCRYPTION_KEY=<openssl-rand-hex-32>

API_PUBLIC_URL=https://api.sehatsahoolat.com/api/v2
FRONTEND_URL=https://app.sehatsahoolat.com,https://doctor.sehatsahoolat.com,https://admin.sehatsahoolat.com,https://sehatsahoolat.com
PATIENT_PORTAL_URL=https://app.sehatsahoolat.com
DOCTOR_PORTAL_URL=https://doctor.sehatsahoolat.com
ADMIN_PORTAL_URL=https://admin.sehatsahoolat.com
PUBLIC_SITE_URL=https://sehatsahoolat.com

PAYFAST_API_PUBLIC_URL=https://api.sehatsahoolat.com/api/v2
PAYFAST_FRONTEND_SUCCESS_URL=https://app.sehatsahoolat.com/dashboard?payment=success
PAYFAST_FRONTEND_CANCEL_URL=https://app.sehatsahoolat.com/packages?payment=cancelled
PAYFAST_MERCHANT_ID=<payfast-merchant-id>
PAYFAST_SECURED_KEY=<payfast-secured-key>
PAYFAST_STORE_ID=<payfast-store-id>
PAYFAST_SECRET_WORD=<payfast-secret-word-if-enabled>
PAYFAST_MERCHANT_NAME=Sehat Sahoolat
PAYFAST_CURRENCY_CODE=USD

OPENAI_API_KEY=<openai-key>
OPENAI_MODEL=gpt-4o-mini

AGORA_APP_ID=<agora-app-id>
AGORA_APP_CERTIFICATE=<agora-app-certificate>

SMTP_HOST=<smtp-host>
SMTP_PORT=465
SMTP_USER=<smtp-user>
SMTP_PASS=<smtp-password>
SMTP_FROM_EMAIL=no-reply@sehatsahoolat.com
SMTP_FROM_NAME=Sehat Sahoolat

DO_SPACES_ENDPOINT=<spaces-endpoint>
DO_SPACES_REGION=<spaces-region>
DO_SPACES_BUCKET=<spaces-bucket>
DO_SPACES_KEY=<spaces-key>
DO_SPACES_SECRET=<spaces-secret>

WHISPER_MODEL=small
WHISPER_DEVICE=cpu
WHISPER_COMPUTE_TYPE=int8
```

Generate secrets:

```bash
openssl rand -hex 32
openssl rand -hex 24
```

`JWT_SECRET`, `JWT_REFRESH_SECRET`, `ENCRYPTION_KEY`, and `REDIS_PASSWORD`
must be unique for production. Do not reuse development values.

## 7. One-Time Server Provisioning

SSH into the new server as root:

```bash
ssh root@<server-ip>
```

Run provisioning:

```bash
curl -fsSL https://raw.githubusercontent.com/mujtabatariq18/sehat-sahoolat/main/deploy/scripts/provision.sh -o /tmp/provision.sh
bash /tmp/provision.sh
```

Switch to the application user:

```bash
su - sehat
```

Clone the repository:

```bash
git clone https://github.com/mujtabatariq18/sehat-sahoolat.git
cd sehat-sahoolat
```

Fill `/etc/sehat/.env` before starting the stack:

```bash
sudo nano /etc/sehat/.env
```

Confirm required values are present:

```bash
grep -E '^(DB_HOST|DB_PASSWORD|JWT_SECRET|JWT_REFRESH_SECRET|ENCRYPTION_KEY|REDIS_PASSWORD|API_PUBLIC_URL|FRONTEND_URL)=' /etc/sehat/.env
```

## 8. First Production Deploy

From the server repo directory:

```bash
cd ~/sehat-sahoolat
bash deploy/scripts/deploy.sh --issue-certs
```

This will:

1. Verify `/etc/sehat/.env`.
2. Pull the latest `main`.
3. Issue TLS certificates.
4. Build all production images.
5. Run TypeORM migrations.
6. Start/recreate containers.
7. Probe backend health.
8. Probe the public portal hosts.

Expected containers:

```bash
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
```

You should see:

- `sehat-nginx`
- `sehat-backend`
- `sehat-patient`
- `sehat-doctor`
- `sehat-admin`
- `sehat-public`
- `sehat-redis`
- `sehat-whisper`

## 9. Production Compose Validation

Before deploying a new production compose change, validate it:

```bash
docker compose --env-file /etc/sehat/.env -f deploy/docker-compose.prod.yml config --quiet
```

Build only:

```bash
docker compose --env-file /etc/sehat/.env -f deploy/docker-compose.prod.yml build
```

Start or update the full stack:

```bash
docker compose --env-file /etc/sehat/.env -f deploy/docker-compose.prod.yml up -d --remove-orphans
```

## 10. Post-Deploy Health Checks

Run these after every deployment.

Backend should return `401` for unauthenticated `/auth/me`:

```bash
curl -i https://api.sehatsahoolat.com/api/v2/auth/me
```

Public pages should return `200`, `301`, or `302`:

```bash
curl -I https://sehatsahoolat.com
curl -I https://app.sehatsahoolat.com
curl -I https://doctor.sehatsahoolat.com
curl -I https://admin.sehatsahoolat.com
```

Internal backend health from inside the container:

```bash
docker exec sehat-backend curl -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3000/api/v2/auth/me
```

Logs:

```bash
docker compose --env-file /etc/sehat/.env -f deploy/docker-compose.prod.yml logs -f backend
docker compose --env-file /etc/sehat/.env -f deploy/docker-compose.prod.yml logs -f nginx
```

Database smoke check:

```bash
PGPASSWORD="$(grep '^DB_PASSWORD=' /etc/sehat/.env | cut -d= -f2-)" \
psql -h "$(grep '^DB_HOST=' /etc/sehat/.env | cut -d= -f2-)" \
  -p "$(grep '^DB_PORT=' /etc/sehat/.env | cut -d= -f2-)" \
  -U "$(grep '^DB_USERNAME=' /etc/sehat/.env | cut -d= -f2-)" \
  -d "$(grep '^DB_DATABASE=' /etc/sehat/.env | cut -d= -f2-)" \
  -c "select now();"
```

## 11. Normal Deployment Flow

Local machine:

```bash
git checkout main
git pull
git merge <tested-branch>
git push origin main
```

Server:

```bash
ssh sehat@<server-ip>
cd ~/sehat-sahoolat
bash deploy/scripts/deploy.sh
```

Deploy one service only:

```bash
bash deploy/scripts/deploy.sh --service backend
bash deploy/scripts/deploy.sh --service patient-portal
bash deploy/scripts/deploy.sh --service doctor-portal
bash deploy/scripts/deploy.sh --service admin-portal
bash deploy/scripts/deploy.sh --service public-site
```

Deploy code already on disk without pulling:

```bash
bash deploy/scripts/deploy.sh --no-pull
```

Restart without rebuilding:

```bash
bash deploy/scripts/deploy.sh --no-pull --skip-build
```

## 12. Database Migration Rules

Production deploy runs migrations before restarting containers. If migrations
fail, the deploy script aborts before rolling the stack.

Manual migration command:

```bash
docker compose --env-file /etc/sehat/.env -f deploy/docker-compose.prod.yml run --rm --no-deps --entrypoint='' backend \
  node ./node_modules/typeorm/cli.js migration:run -d ./dist/src/data-source.js
```

Manual migration rollback:

```bash
docker compose --env-file /etc/sehat/.env -f deploy/docker-compose.prod.yml run --rm --no-deps --entrypoint='' backend \
  node ./node_modules/typeorm/cli.js migration:revert -d ./dist/src/data-source.js
```

Migration safety rules:

- Take a fresh database backup before destructive migrations.
- Never run development seeders in production.
- Do not manually edit production tables unless there is a written rollback
  plan.
- Prefer additive schema changes for zero-downtime deploys.

## 13. Backup Requirements

Minimum production backup policy:

| Asset | Backup method | Retention |
| --- | --- | --- |
| Managed Postgres | Provider backup plus encrypted dump to Spaces/S3 | Daily 7, weekly 5, monthly 12 |
| Uploads volume | Sync to Spaces/S3 | Daily |
| `/etc/sehat/.env` | Password manager secure note | Every secret rotation |
| TLS certificates | Re-issuable through acme.sh | No manual backup required |

Recommended cron:

```cron
15 2 * * * /home/sehat/sehat-sahoolat/deploy/scripts/backup.sh >> /var/log/sehat-backup.log 2>&1
```

Manual database dump:

```bash
mkdir -p ~/backups
PGPASSWORD="$(grep '^DB_PASSWORD=' /etc/sehat/.env | cut -d= -f2-)" \
pg_dump \
  -h "$(grep '^DB_HOST=' /etc/sehat/.env | cut -d= -f2-)" \
  -p "$(grep '^DB_PORT=' /etc/sehat/.env | cut -d= -f2-)" \
  -U "$(grep '^DB_USERNAME=' /etc/sehat/.env | cut -d= -f2-)" \
  -d "$(grep '^DB_DATABASE=' /etc/sehat/.env | cut -d= -f2-)" \
  -Fc \
  -f ~/backups/sehat_sahoolat_$(date +%Y%m%d_%H%M%S).dump
```

## 14. Rollback

Rollback application code:

```bash
cd ~/sehat-sahoolat
git log --oneline -10
git checkout <known-good-sha>
bash deploy/scripts/deploy.sh --no-pull
```

Rollback a single service:

```bash
git checkout <known-good-sha>
bash deploy/scripts/deploy.sh --no-pull --service backend
```

If the failed deploy included a migration:

1. Check the migration file.
2. Confirm whether `migration:revert` is safe.
3. Take a database dump.
4. Run `migration:revert`.
5. Redeploy the known-good code.

Never roll back code blindly after a destructive migration.

## 15. Mobile App Production Configuration

For Android production builds, compile with production URLs:

```bash
cd mobile
flutter build appbundle --release \
  --dart-define=BACKEND_URL=https://api.sehatsahoolat.com/api/v2 \
  --dart-define=CHECKOUT_GATEWAY=payfast \
  --dart-define=PATIENT_PORTAL_URL=https://app.sehatsahoolat.com \
  --dart-define=DOCTOR_PORTAL_URL=https://doctor.sehatsahoolat.com \
  --dart-define=ADMIN_PORTAL_URL=https://admin.sehatsahoolat.com \
  --dart-define=MOBILE_CALL_BASE_URL=https://mobile-call.sehatsahoolat.com
```

For local emulator QA against Docker:

```bash
cd mobile
flutter test integration_test/packages_flow_test.dart -d emulator-5554 \
  --dart-define=BACKEND_URL=http://10.0.2.2:3000/api/v2 \
  --dart-define=CHECKOUT_GATEWAY=mock \
  --dart-define=E2E_PATIENT_EMAIL=patient3@sehat.local \
  --dart-define=E2E_PATIENT_PASSWORD=PatientPass1!
```

If host port `3000` is already taken locally, expose the backend on another
host port and update the emulator URL, for example:

```bash
docker compose run -d --name sehat-backend-3005 -p 3005:3000 backend
flutter test integration_test/packages_flow_test.dart -d emulator-5554 \
  --dart-define=BACKEND_URL=http://10.0.2.2:3005/api/v2 \
  --dart-define=CHECKOUT_GATEWAY=mock
```

## 16. Admin Go-Live Checklist

Before announcing production:

- Create the first admin account.
- Enable admin MFA.
- Configure PayFast production credentials.
- Configure SMTP and send a test email.
- Configure Agora production credentials.
- Confirm packages are active and priced correctly.
- Confirm patient signup/login.
- Confirm doctor invite/login.
- Confirm admin login/MFA.
- Confirm a package checkout handoff.
- Confirm prescription and appointment flows.
- Confirm public site loads at the apex domain.
- Confirm backup job succeeds.
- Confirm logs do not expose secrets.

## 17. Monitoring and Alerts

Minimum alerts:

| Metric | Threshold |
| --- | --- |
| CPU | > 80% for 10 minutes |
| RAM | > 85% for 5 minutes |
| Disk | > 80% |
| Backend uptime | API health check fails for 2 minutes |
| TLS certificate | Expires within 14 days |
| Database CPU | > 80% for 10 minutes |
| Database storage | > 80% |
| Backup job | No successful backup in 28 hours |

Useful commands:

```bash
docker stats
docker system df
df -h
free -h
docker compose --env-file /etc/sehat/.env -f deploy/docker-compose.prod.yml ps
```

## 18. Common Failures

| Symptom | Likely cause | Fix |
| --- | --- | --- |
| `502 Bad Gateway` | Upstream container crashed | Check `docker compose logs backend` or the relevant portal |
| Backend starts then exits | Missing production env var | Check `/etc/sehat/.env` and backend logs |
| API returns CORS errors | `FRONTEND_URL` missing a portal URL | Add all four origins and redeploy backend |
| PayFast callback points to localhost | PayFast public URLs missing | Set `PAYFAST_API_PUBLIC_URL` and frontend success/cancel URLs |
| Public site 502 | Missing `public-site` container or nginx upstream | Confirm compose includes `public-site` and `public_site` upstream |
| Mobile app cannot connect | Wrong `BACKEND_URL` dart define | Rebuild with production API URL |
| Login device limit errors | Too many active sessions for package | Revoke old sessions or adjust package device limits |
| Cert issue fails | DNS or Cloudflare ACME cache issue | Verify A records and bypass cache for ACME path |
| Disk fills up | Old images/build cache/logs | Run `docker image prune -f` after confirming containers are healthy |

## 19. Production File Map

```text
deploy/
  docker-compose.prod.yml        Production container stack
  dockerfiles/
    Dockerfile.backend           Backend production image
    Dockerfile.nextjs            Shared Next.js app image
    Dockerfile.whisper           Whisper sidecar image
  nginx/
    nginx.conf                   nginx worker and rate-limit config
    conf.d/
      00-security-headers.conf   Shared security headers
      10-upstreams.conf          Docker upstreams
      20-redirects.conf          HTTP to HTTPS redirect
      30-patient.conf            Patient portal host
      31-doctor.conf             Doctor portal host
      32-admin.conf              Admin portal host
      33-public.conf             Public site host
      40-api.conf                Backend API host
      proxy-headers.conf         Shared proxy headers
  scripts/
    provision.sh                 One-time server bootstrap
    deploy.sh                    Production deployment runner

docs/
  PRODUCTION_DEPLOYMENT_GUIDE.md This guide
```

## 20. Final Launch Verification

Run this final checklist after the first production deploy:

```bash
curl -I https://sehatsahoolat.com
curl -I https://app.sehatsahoolat.com
curl -I https://doctor.sehatsahoolat.com
curl -I https://admin.sehatsahoolat.com
curl -i https://api.sehatsahoolat.com/api/v2/auth/me
docker compose --env-file /etc/sehat/.env -f deploy/docker-compose.prod.yml ps
docker compose --env-file /etc/sehat/.env -f deploy/docker-compose.prod.yml logs --tail=100 backend
```

The API `/auth/me` endpoint should normally return `401` without a token. That
is a healthy unauthenticated response, not a deployment failure.