SwiftDeploy: Building an Observable, Policy-Driven Deployment Engine with OPA

Introduction As part of the HNG Internship DevOps Track Stage 4B, I extended my Stage 4A project — SwiftDeploy — into a fully observable, policy-aware deployment platform. In Stage 4A, SwiftDeploy could: - generate infrastructure files from a declarative manifest - deploy containers using Docker Compose - manage deployment modes (stable/canary) - configure Nginx automatically Stage 4B transformed it into something much closer to a real production deployment system by adding: - Prometheus instrumentation - Open Policy Agent (OPA) policy enforcement - live operational dashboards - deployment safety gates - audit logging and reporting - chaos engineering validation The result is a deployment tool that not only deploys services, but also decides whether deployments are safe enough to proceed. The Core Philosophy: One Manifest, Everything Else Generated SwiftDeploy is built around a single principle: manifest.yaml is the only file you should ever edit manually. Everything else is generated from it. Here is the manifest structure: services: name: app image: swift-deploy-1-node:latest port: 3000 version: "1.0.0" mode: stablenginx: image: nginx:latest port: 8080 proxy_timeout: 30network: name: swiftdeploy-net driver_type: bridge From this manifest, the CLI generates: - generated/nginx.conf - generated/docker-compose.yml - OPA runtime configuration This design provides: - consistency - reproducibility - environment portability - infrastructure-as-code discipline The grader can delete all generated files and rerun: ./swiftdeploy init and the entire stack regenerates correctly. Architecture Overview The system architecture consists of four major components: User ↓Nginx Reverse Proxy ↓Flask API Service ↓Prometheus Metrics ↓SwiftDeploy CLI ↓OPA Policy Engine The deployment stack includes: - Flask application container - Nginx reverse proxy - Open Policy Agent (OPA) - internal Docker network - named log volumes The SwiftDeploy CLI The heart of the project is the swiftdeploy executable. It is a Python-based CLI tool that manages the entire deployment lifecycle. Supported Commands CommandPurposeinitGenerate config files from templatesvalidateRun pre-flight validation checksdeployStart the stackpromote canarySwitch deployment into canary modepromote stableReturn deployment to stable modestatusLive metrics dashboardauditGenerate audit reportteardownDestroy containers and networks The API Service The API service is a Flask application that supports both stable and canary deployment modes. Deployment mode is controlled through the MODE environment variable. Endpoints Root Endpoint GET / Returns: - deployment mode - version - timestamp Example: { "message": "Welcome to SwiftDeploy", "mode": "stable", "version": "1.0.0"} Health Endpoint GET /healthz Returns: - health status - application uptime Chaos Endpoint POST /chaos Available only in canary mode. Supports: { "mode": "slow", "duration": 3 } { "mode": "error", "rate": 0.5 } { "mode": "recover" } This endpoint was used to simulate: - degraded latency - random failures - recovery workflows Instrumentation: The /metrics Endpoint One of the biggest upgrades in Stage 4B was observability. I instrumented the Flask service using the prometheus_client library. The service now exposes: GET /metrics in Prometheus text format. Metrics Collected Request Throughput http_requests_total Labels: method path status_code Example: http_requests_total{method="GET",path="/",status_code="200"} 152 Request Latency http_request_duration_seconds Histogram used for: - latency analysis - P99 calculation Application Uptime app_uptime_seconds Tracks process uptime. Deployment Mode app_mode Values: - 0 = stable - 1 = canary Chaos State chaos_active Values: - 0 = none - 1 = slow - 2 = error Why Metrics Matter Without metrics: - deployments are blind - failures become invisible - canary safety cannot be enforced Metrics became the foundation for: - policy decisions - dashboards - auditing - promotion safety Open Policy Agent (OPA): The Brain of SwiftDeploy The most important design principle in Stage 4B was: The CLI must never make allow/deny decisions itself. All decision-making lives entirely inside OPA. SwiftDeploy only: - gathers data - sends context to OPA - acts on the response This separation makes the system: - modular - secure - maintainable - extensible OPA Policy Domains I separated policies into independent domains. Each policy: - answers one question - owns its own logic - operates independently Infrastructure Policy Runs before deployment. Blocks deployment when: - disk free space is below 10GB - CPU load exceeds 2.0 Rego Example package infradefault allow = falseallow { input.disk_free_gb >= data.thresholds.disk_free_gb input.cpu_load <= data.thresholds.cpu_load} Canary Safety Policy Runs before promotion. Blocks promotion when: - error rate exceeds 1% - P99 latency exceeds 500ms Rego Example package canarydefault allow = falseallow { input.error_rate <= data.thresholds.error_rate input.p99_latency_ms <= data.thresholds.p99_latency_ms} Policy Thresholds Thresholds are stored separately in: policies/data.json Example: { "thresholds": { "disk_free_gb": 10, "cpu_load": 2.0, "error_rate": 0.01, "p99_latency_ms": 500 }} This prevents: - hardcoded values - duplicated configuration - policy coupling OPA Isolation The OPA container runs on an internal Docker network. It is intentionally NOT exposed through Nginx. Only the CLI can access OPA directly via: http://localhost:8181 This prevents external users from: - querying policies - bypassing deployment logic - inspecting internal rules This mirrors real production security architecture. Pre-Deploy Policy Enforcement Before deployment, SwiftDeploy collects: - CPU load - available disk space Example payload: { "disk_free_gb": 8.5, "cpu_load": 2.4} OPA evaluates the payload. If policies fail: Deployment blocked:Infrastructure policy violation The deployment never proceeds. Canary Safety Enforcement Before promotion, SwiftDeploy: - scrapes /metrics - calculates error rate - calculates P99 latency - submits metrics to OPA If the canary is unhealthy: - promotion is blocked - rollout is prevented This introduces production-grade deployment safety. The Status Dashboard The status command provides a live operational dashboard. ./swiftdeploy status The dashboard: - refreshes continuously - scrapes live metrics - calculates request rate - calculates P99 latency - evaluates policy compliance - appends results to history.jsonl Example output: SwiftDeploy Status Dashboard==================================================Mode: canaryChaos: errorError Rate: 52%P99 Latency: 430msPolicy Compliance:✓ Infrastructure policy: PASSING✗ Canary safety policy: FAILING Chaos Engineering This was one of the most interesting parts of the project. I intentionally injected: - high error rates - slow responses Example: curl -X POST http://localhost:8080/chaos -d '{"mode":"error","rate":0.9}' Immediately: - metrics reflected failures - policies began failing promotions were blocked This validated that:metrics were accurate policies were functional safety gates worked correctly Audit Logging Every: - deploy - promote - status scrape - policy violation is appended to: history.jsonl Example entry: { "timestamp": "2026-05-06T12:00:00", "mode": "canary", "error_rate": 0.52} Audit Report Generation Running: ./swiftdeploy audit generates: audit_report.md The report includes: - deployment timeline - mode changes - chaos injections - policy violations Example: | Timestamp | Policy | Details ||-----------|--------|---------|| 2026-05-06T00:47:10Z | Canary Safety | error_rate=50% | Challenges Faced a. Python Virtual Environment Issues Ubuntu’s externally-managed Python environment caused repeated package installation failures. The solution was: - recreating the virtual environment - installing dependencies inside the venv only b. Nginx Validation Problems Generated Nginx configs initially failed validation due to unresolved upstream references. Fix: - validate only inside container context - avoid host-side upstream resolution c. Metrics Parsing - Calculating: - error rate - P99 latency from Prometheus text format required careful parsing and aggregation. d. OPA Failure Handling The CLI had to gracefully handle: - OPA downtime - connection failures - malformed responses The system never crashes when OPA becomes unavailable. Lessons Learned Declarative Systems Scale Better A single source of truth drastically reduces configuration drift. Observability Is Mandatory Without metrics: - policy enforcement becomes impossible - deployments become blind - Policy Engines Should Be Isolated - Keeping OPA internal-only mirrors real enterprise architectures. Chaos Engineering Builds Confidence Breaking the system intentionally proved that: - metrics were accurate - policies were effective - safety mechanisms worked Automation Must Be Explainable Every policy response included human-readable reasoning. This made debugging and operational decisions much easier. Final Thoughts Stage 4B transformed SwiftDeploy from a deployment generator into a lightweight deployment platform with: - observability - governance - auditing - deployment safety The project demonstrated how: - metrics - policy engines - infrastructure generation - deployment orchestration can work together to create reliable deployment systems. Most importantly, it reinforced a key DevOps principle: Safe automation is more valuable than fast automation.