Guide to DevSecOps

Appearance

System Design: CRM System (Customer Relationship Management) ​

1. Business Requirements ​

Functional Requirements ​

  • Manage customer records (CRUD: create, read, update, delete)
  • Track interactions (calls, emails, meetings, notes)
  • Manage sales pipeline (leads, opportunities, deals)
  • Assign tasks and reminders to users
  • User authentication and role-based access control
  • Search and filter customers and activities
  • Import/export customer data (CSV, Excel)
  • Generate reports and dashboards (sales, activity, performance)
  • Integration with email/calendar (e.g., Google, Outlook)
  • Audit logging for changes

Non-Functional Requirements ​

  • 99.9% availability (highly available, fault-tolerant)
  • Low latency for UI and API (<100ms p99 in-region)
  • Scalability to support thousands of users and millions of records
  • Strong security (encryption, RBAC, input validation, HTTPS)
  • Data consistency for customer and sales data
  • GDPR compliance (data privacy, deletion on request)
  • Observability (logging, monitoring, alerting)

Out of Scope ​

  • Built-in telephony/VoIP
  • Marketing automation (email campaigns, SMS blasts)
  • AI-based lead scoring
  • Mobile app (web only for MVP)
  • Third-party marketplace for plugins

2. Estimation & Back-of-the-Envelope Calculations ​

  • Users: 5,000 active users
  • Customer Records: 2M customers
  • Interactions: 10M/year (~27,000/day)
  • Sales Opportunities: 500,000/year
  • API QPS: 200 QPS (peak), 50 QPS (avg)
  • Storage: 2M customers × 2KB + 10M interactions × 1KB ≈ 12GB/year
  • Reports: 10,000/month
  • Availability: 99.9% = <9h downtime/year

2a. Estimation & Back-of-the-Envelope Calculations (Large Scale) ​

  • Users: 5,000 active users
  • Customer Records: 60M customers
  • Interactions: 300M/year (~820,000/day)
  • Sales Opportunities: 15M/year
  • API QPS: 2,000 QPS (peak), 500 QPS (avg)
  • Storage: 60M customers × 2KB + 300M interactions × 1KB ≈ 660GB/year
  • Reports: 100,000/month
  • Availability: 99.9% = <9h downtime/year

3. High-Level Design ​

Key Design Decisions ​

  • Database Choice: Use a scalable, highly available relational DB (e.g., PostgreSQL with read replicas, or managed cloud DB like Azure SQL, Amazon Aurora, Google Cloud SQL). For search: Elasticsearch or managed search service.
  • API: RESTful API (optionally GraphQL for flexible queries)
  • Frontend: SPA (React, Angular, Vue)
  • Authentication: OAuth2, SSO (SAML, OIDC)
  • Caching: Redis for session and hot data
  • File Storage: Cloud object storage for attachments (e.g., S3, Azure Blob)

Component Diagram ​

mermaid
flowchart TD
    User[User / Browser] -->|UI| FE[Frontend SPA]
    FE --> API[API Gateway]
    API --> Auth[Auth Service]
    API --> App[CRM App Service]
    App --> DB[(Relational DB)]
    App --> Search[Search Service]
    App --> Cache[Redis Cache]
    App --> Storage[Object Storage]
    App --> Audit[Audit/Event Log]
    App --> Analytics[Reporting Service]

Data Flow Diagram ​

mermaid
sequenceDiagram
    participant U as User
    participant FE as Frontend
    participant API as API Gateway
    participant S as CRM Service
    participant DB as Relational DB
    participant C as Cache
    participant SR as Search
    participant ST as Storage
    U->>FE: Interact with UI
    FE->>API: API Request (CRUD, search, etc.)
    API->>S: Validate & Auth
    S->>C: Check cache (hot data)
    alt Cache miss
        S->>DB: Query/update data
        S->>C: Update cache
    end
    S->>SR: Search (if needed)
    S->>ST: File upload/download (if needed)
    S->>API: Return response

4. Conceptual Design ​

Database ​

  • Primary DB: Relational DB (PostgreSQL/Aurora/Cloud SQL)
    • Tables: customers, users, interactions, opportunities, tasks, audit_logs
    • Use read replicas for scaling reads
    • Partitioning for large tables (e.g., by customer region)
    • Strong consistency for transactional operations
  • Search: Elasticsearch for full-text search on customers/interactions
  • Cache: Redis for sessions, hot customer data, rate limiting
  • File Storage: Cloud object storage for attachments

API & Backend ​

  • RESTful API (OpenAPI/Swagger)
  • Authentication: OAuth2, SSO (SAML/OIDC)
  • RBAC: Roles (admin, sales, support, etc.)
  • Input validation, rate limiting, HTTPS
  • Audit logging for all changes
  • Reporting service for analytics

Frontend ​

  • SPA (React/Angular/Vue)
  • Responsive design
  • Auth flows (login, SSO)
  • Data visualization (charts, dashboards)

Integrations ​

  • Email/calendar integration via provider APIs
  • Import/export via CSV/Excel

Security ​

  • Encryption at rest and in transit
  • Secure password storage (bcrypt/argon2)
  • Regular security reviews

Observability ​

  • Centralized logging, metrics, distributed tracing
  • Alerting on error rates, latency, availability

5. Bottlenecks & Refinement ​

Potential Bottlenecks ​

  • DB Write Load: High volume of interactions/updates. Mitigate with batching, partitioning, and read replicas.
  • Search Latency: Large datasets may slow search. Use indexing, tune queries, and scale search nodes.
  • Cache Eviction: High churn may cause cache misses. Tune eviction policy and cache size.
  • Reporting Load: Complex reports may impact DB. Use read replicas and pre-aggregation.
  • File Storage: Large attachments may slow uploads/downloads. Use CDN and async processing.

Refinement Strategies ​

  • Use multi-region DB replication for failover
  • Pre-warm cache for frequently accessed data
  • Use async/batch processing for heavy operations
  • Monitor and autoscale API, DB, and search layers
  • Regularly review security and compliance

To create a POC for the CRM application using Vue (frontend) and NestJS (backend), here’s a high-level plan and the components that will be generated:

Backend (NestJS): ​

Project structure with modules for Customers, Users, Interactions, Auth, and Opportunities.

  • RESTful API endpoints for CRUD operations.
  • In-memory or SQLite DB for demo.
  • Simple JWT-based authentication (simulating OAuth2/SSO).
  • Basic RBAC (admin, sales, support).
  • Audit logging (console or file).
  • CORS enabled for frontend.

Frontend (Vue): ​

Project structure with views for Login, Dashboard, Customers, Interactions, Opportunities, and Reports.

  • Vuex or Pinia for state management.
  • Axios for API calls.
  • Auth flow (login, logout, role-based navigation).
  • Simple data tables and forms for CRUD.
  • Basic charts for reports (e.g., Chart.js).
  • Responsive layout.

Interactivity Simulation: ​

  • Users can log in, view dashboards, manage customers, add interactions, and view reports.
  • Role-based UI (admin sees all, sales/support see limited).
  • Demo data for customers, interactions, and opportunities.
sh
docker-compose up -d

docker exec -i crm-db psql -U crm -d crm < api/schema.sql
docker exec -i crm-db psql -U crm -d crm < api/seed.sql

Access the app

Frontend: http://localhost:8080 API: http://localhost:3000 Postgres: localhost:5432 (user: crm, pass: crm, db: crm) Redis: localhost:6379 Elasticsearch: localhost:9200 MinIO: http://localhost:9000 (user: minio, pass: minio123)