Guide to DevSecOps

Appearance

Video Streaming App System Design ​

1. Business Requirements ​

Functional Requirements ​

  • User registration and authentication (viewers, creators, admins)
  • Video upload, transcoding, and storage
  • Video streaming (adaptive bitrate, live and on-demand)
  • Search and browse videos (by category, tags, popularity)
  • Playlists, likes, comments, and sharing
  • Real-time notifications and urgent alerts (live events, new uploads, system issues)
  • Analytics and reporting (views, engagement, trends)
  • Mobile-ready responsive UI and API
  • Role-based access control

Non-Functional Requirements ​

  • 99.9% availability (max ~8.76 hours downtime/year)
  • Scalability to support millions of users and concurrent streams
  • Fast response and low-latency streaming (<2s startup, <300ms API)
  • Secure data storage and access control (DRM, HTTPS)
  • Audit logging and monitoring
  • Backup and disaster recovery
  • GDPR/data privacy compliance
  • Mobile responsiveness

Out of Scope ​

  • Built-in ad serving/monetization (unless specified)
  • Video editing tools
  • Integration with external streaming platforms

2. Estimation & Back-of-the-Envelope Calculations ​

  • Users: 10M
  • Videos: 10M (average 100MB each)
  • Daily streams: 50M
  • Peak concurrent viewers: ~500,000
  • Data size:
    • Videos: 10M × 100MB = 1PB (object storage)
    • Metadata: 10M × 2KB ≈ 20GB
    • User data: 10M × 2KB ≈ 20GB
    • Comments/likes: 1B × 0.2KB ≈ 200GB
    • Logs/analytics: 1B × 0.2KB ≈ 200GB
    • Total DB size: ~500GB (excluding logs, backups, video storage)
  • Availability:
    • 99.9% = 8.76 hours/year downtime max
    • Use distributed storage, multi-AZ deployment, health checks, auto-scaling

3. High Level Design (Mermaid Diagrams) ​

Component Diagram ​

mermaid
flowchart LR
  User[User (Web/Mobile)]
  LB[Load Balancer]
  App[Application Server]
  DB[(Database)]
  Cache[Cache (Redis)]
  Storage[Object Storage (Videos)]
  Transcoder[Transcoding Service]
  CDN[CDN]
  Alert[Alert/Notification Service]
  Analytics[Analytics Engine]

  User --> LB --> App
  App --> DB
  App --> Cache
  App --> Storage
  App --> Transcoder
  Transcoder --> Storage
  Storage --> CDN
  App --> Alert
  App --> Analytics
  Analytics --> DB

Data Flow Diagram ​

mermaid
sequenceDiagram
  participant U as User
  participant A as App Server
  participant D as Database
  participant C as Cache
  participant S as Storage
  participant T as Transcoder
  participant N as CDN
  participant L as Alert Service

  U->>A: Upload Video
  A->>S: Store Raw Video
  S-->>A: Success/Fail
  A->>T: Trigger Transcoding
  T->>S: Store Transcoded Video
  S-->>T: Success/Fail
  S->>N: Distribute to CDN
  U->>A: Stream Video
  A->>C: Check Metadata/Session
  C-->>A: Hit/Miss
  A->>D: Log View/Engagement
  A->>L: Send Urgent Alert (if needed)
  N-->>U: Deliver Video Stream

Key Design Decisions ​

  • Database: NoSQL (e.g., Cassandra, MongoDB) for video metadata, comments, likes; Relational DB (e.g., PostgreSQL) for user/auth data
  • Object Storage: For videos (e.g., AWS S3, Azure Blob)
  • Transcoding: Dedicated service for adaptive bitrate and format conversion
  • CDN: For global, low-latency video delivery
  • Cache: Redis for sessions, trending videos, and metadata
  • Alerting/Notifications: Email/SMS/push via third-party service (e.g., Twilio, Firebase)
  • Analytics: Batch or streaming (e.g., Kafka + Spark, or managed cloud analytics)
  • Deployment: Cloud-based, multi-AZ, managed services for high availability
  • API: REST/GraphQL for mobile and web clients

4. Conceptual Design ​

Entities ​

  • User: id, name, email, password_hash, role, registration_date, status
  • Video: id, user_id, title, description, url, status, upload_date, views, likes, tags, categories
  • TranscodeJob: id, video_id, status, formats, started_at, completed_at
  • Comment: id, video_id, user_id, content, created_at
  • Like: id, video_id, user_id, created_at
  • Playlist: id, user_id, name, video_ids, created_at
  • Alert: id, user_id, video_id, type (urgent/live/new), message, created_at, status
  • AuditLog: id, user_id, action, entity, entity_id, timestamp

Key Flows ​

  • Video Upload/Transcoding:
    1. User uploads video
    2. App stores raw video, triggers transcoding
    3. Transcoder stores multiple formats, updates metadata
    4. CDN distributes transcoded video
  • Streaming:
    1. User requests video
    2. App checks cache, fetches metadata
    3. CDN delivers video stream
    4. App logs view, triggers alert if needed
  • Alerts:
    • System triggers urgent alerts for live events, new uploads, or system issues
  • Analytics:
    • Periodic jobs aggregate views, engagement, and trends

Security ​

  • Role-based access control (RBAC)
  • Input validation, rate limiting
  • Encrypted connections (HTTPS, DRM)
  • Regular backups and audit logs
  • GDPR/data privacy compliance

5. Bottlenecks and Refinement ​

Potential Bottlenecks ​

  • Transcoding throughput:
    • Use scalable, distributed transcoding service
  • CDN delivery:
    • Use global CDN with edge caching
  • Database write throughput:
    • Use NoSQL DB with sharding and replication
  • Object storage bandwidth:
    • Use multi-region, scalable object storage
  • Alert delivery:
    • Use async queues for urgent notifications
  • Single region failure:
    • Deploy across multiple availability zones/regions

Refinement ​

  • Monitor system metrics and auto-scale transcoder/app servers
  • Regularly test failover and backup restores
  • Optimize queries and indexes for frequent operations
  • Consider sharding if user/video volume grows significantly

This design provides a scalable, highly available, and mobile-ready video streaming system with robust urgent alerts, analytics, and operational best practices.