Mountaingreen.at — Platform Migration & Data Recovery

Problem

Mountaingreen operated a growing affiliate/MLM platform for cannabis plant leasing, serving hundreds of users managing orders, commissions, and KYC compliance. The infrastructure was ancient:

• MySQL 8 monolith on a single EC2 instance with host Nginx (manual SSL, snowflake configs)
• No IaC — infrastructure in someone’s head; SSH into production to troubleshoot
• Data integrity crisis — A failed MySQL->PostgreSQL migration 2 years prior left 2,347 orders orphaned in the old database; commission calculations broken; Stripe/bank reconciliation impossible
• No auto-scaling — Single EC2 host; any spike meant downtime or manual intervention

The business was growing (500+ active users), but the infrastructure couldn’t scale. The team needed:

1. Recover lost data (or face audit liability)
2. Migrate to PostgreSQL + ECS Fargate (serverless, auto-scaling)
3. Implement GitOps (automated deployments, zero manual SSH)
4. Keep the platform live during cutover

Data Preparation

The data recovery was a forensic challenge. We exported MySQL (2GB, 8 years of transactions) and compared against the partial PostgreSQL migration:

Identified inconsistencies:

• 2,347 orders in MySQL but missing from PostgreSQL (old migration script failed silently)
• 412 duplicate user records (emails differing by whitespace)
• Commission payouts calculated as of Sep 2023 but never exported; no record after cutover
• Foreign key mismatches (38 orphaned affiliates, 156 orders without users)

Recovery steps:

1. Wrote CSV reconciliation tool: export MySQL -> map to PostgreSQL IDs -> generate SQL patches
2. Manual reconstruction: For 2,347 lost orders, cross-referenced email archives + Stripe API to recover:
   • Order date (from email timestamps)
   • Order value (from Stripe transaction history)
   • User ID (from email sender)
   • Commission metadata (calculated retroactively)
3. Re-ran commission calculations for 18 months (Mar 2024–Aug 2025) using recovered data; verified against bank deposits

Result: All data recovered and reconciled. PostgreSQL now source-of-truth.

Models Evaluated

Architecture

High-level flow:

Internet (users)
  v
ALB (AWS, HTTPS via ACM, rate-limiting via WAF)
  ├─ /api/* -> ECS Fargate service: backend
  │           (Node.js Express, stateless, JWT auth, scales 2–20 tasks)
  └─ /*      -> ECS Fargate service: frontend
             (React SPA on nginx:alpine, CDN-cached via CloudFront)
             │
             ├─ RDS PostgreSQL 16 (+ read replicas for reports)
             ├─ ElastiCache Redis (rate-limit store, session locks)
             └─ S3 (user uploads: KYC documents, etc.)

Backend (Node.js Express):

• Modular OOP design: src/modules/<domain> (e.g., orders, users, commissions, support)
• Stateless: JWT tokens (no sessions on server), all state in PostgreSQL/Redis
• Health check: /healthz (ALB uses this for auto-scaling decisions)
• Graceful shutdown on SIGTERM (100ms timeout for in-flight requests, then close)

Frontend (React + Vite):

• SPA: served from nginx:alpine container
• nginx config: SPA routing (all 404s -> index.html), cache headers (immutable for /dist/, no-cache for HTML)
• Vite build: code-splitting, tree-shaking, <200KB JS bundle

Database (PostgreSQL 16):

• Schema: users, orders, affiliates, commissions, support_tickets, transactions, audit_log
• Foreign keys enforced; trigger-based commission recalc (on order status change)
• Read replicas: run heavy analytical queries (reports, reconciliation) without hitting primary
• Automated backups: daily snapshots to S3 (30-day retention)

Cache (Redis):

• Rate-limiting: key = rate:${userID}:${endpoint}, TTL = 60s
• Session locks: distributed lock on order updates (prevent race conditions)
• Ephemeral: if Redis unavailable, fallback to in-memory map (loses cross-process state, but service stays up)

Infrastructure (Terraform):

• VPC with public/private subnets, NAT gateway, security groups (ALB allows 80/443, tasks allow 3001/5173)
• ALB: target groups for backend (port 3001) and frontend (port 80); health checks every 30s
• Auto-scaling policy: backend scales when CPU > 70% or requests/task > 1000; frontend fixed at 2 tasks
• RDS: Multi-AZ, automated failover, encryption at rest
• Secrets Manager: DB credentials, API keys (rotated quarterly)

CI/CD (GitHub Actions):

On: push to main
  1. Build backend Docker image
  2. Run integration tests (PostgreSQL + Redis in containers)
  3. Push image to ECR
  4. Update ECS task definition
  5. Deploy to Fargate (rolling update, 0 downtime)
  6. Smoke tests against production (login, place order, check commission)
  7. Post Slack notification

Production & Scale

Deployment timeline:

• Feb 2026: Data recovery completed; Terraform infrastructure stood up in staging
• Mar 2026: Canary deployment (5% of traffic to Fargate) — ran parallel with EC2 for 2 weeks
• Apr 2026: Full cutover to Fargate; EC2 decommissioned
• May–Aug 2026: Stability & optimization (Redis tuning, CloudFront caching, auto-scaling tweaks)

Results:

• Deployment time: 1.5h manual (SSH, docker pull, restart) -> 8min automated (GitHub Actions)
• Data integrity: 100% (all 2,347 lost orders recovered & reconciled)
• Uptime: 99.8% (6h downtime = intentional DB failover testing; no unplanned incidents)
• Scaling: Auto-scales from 2 to 20 backend tasks within 3 minutes during traffic spikes
• Cost: 42% reduction (Fargate spot instances vs EC2 always-on)

Monitoring:

• CloudWatch dashboards: latency (P50/P95/P99), error rates, DB connections, Redis hit ratio
• Datadog APM: distributed tracing (request spans across backend/DB/Redis)
• PagerDuty: on-call rotation; automated alerts for error rate >1%, latency P95 >2s
• Weekly review: analyze slow queries (RDS Performance Insights), optimize indexes

Challenges & resolutions:

1. Race condition on order updates: Multiple API calls could increment commission twice
   • Solution: Distributed lock (Redis with key expiry); serialized critical sections
2. PostgreSQL connection pool exhaustion: During traffic spikes, connection pool drained
   • Solution: Tuned pgBouncer (transaction pooling mode); added connection monitoring
3. Cold start on Fargate: New tasks took 8s to be healthy (Node.js startup)
   • Solution: Pre-warming tasks (maintain min 2 always running); health check grace period = 30s

Future work:

• Migrate frontend to Next.js SSR (for SEO + performance)
• Add Redis Cluster (from standalone instance) for higher throughput
• Implement CQRS pattern (read replicas for real-time analytics)

Result: Mountaingreen now runs on modern, scalable infrastructure. Data is recovered and auditable. Deployments are automated (0 downtime). The team can focus on features instead of ops toil.