Scalable Video Transcoding Microservices Platform on AWS

1. Project Overview

The platform aims to provide a flexible and scalable video transcoding service consisting of two independent microservices:

• Auth Service: Responsible for user authentication and authorization, exposing RESTful APIs.

• Transcoding Service: Pulls video transcoding tasks from a queue and executes the transcoding process.

The overall architecture routes traffic via an AWS Application Load Balancer (ALB), uses TLS certificates issued by AWS Certificate Manager (ACM) to enforce HTTPS encryption, and serves static assets stored in S3 with CloudFront CDN caching.

---

2. Service Decomposition and Traffic Routing

• Both services are deployed on separate EC2 instances with Auto Scaling Groups to support flexible scaling.

• ALB routes requests based on path rules:

  • Requests to /api/auth* are forwarded to the Auth Service target group.

  • Requests to /api/transcode* are forwarded to the Transcoding Service target group.

• All service APIs are prefixed with /api to simplify ALB routing logic.

---

3. Static Assets Management and CDN

• Static files such as CSS and JavaScript are stored under the /static path in an S3 bucket.

• CloudFront is configured as the CDN to accelerate and improve the reliability of static resource delivery.

• CloudFront configuration includes:

  • Enforcing HTTPS with an ACM-issued SSL certificate.

  • Cache policies optimized for performance.

• Frontend HTML references static assets using the CloudFront domain, for example:

<link rel="stylesheet" href="https://<cloudfront_domain>/static/styles.css" />

4. Lambda Event-Driven Design

• Uploading videos to a specific S3 directory (e.g., /uploads) triggers a Lambda function.

• Initially, the Lambda logs the event to facilitate future extension of the transcoding trigger workflow.

Example Lambda code:

exports.handler = async (event) => {
  console.log("Received S3 event:", JSON.stringify(event, null, 2));
  // Future: Add transcoding task scheduling or notification logic here
};

---

5. Infrastructure as Code (IaC)

• Terraform is used to manage all AWS resources, ensuring repeatability and version control of the environment.

• Managed resources include:

  • EC2 instances and security group configurations

  • ALB, target groups, and listener rules

  • SQS queue setup

  • S3 bucket and access permissions

  • CloudFront distribution configuration

---

6. Transcoding Service Auto Scaling

• Auto Scaling Groups monitor EC2 CPU usage and dynamically adjust the number of transcoding instances.

• EC2 user data script automates installation of dependencies and application deployment:

#!/bin/bash
sudo apt-get update
sudo apt-get install -y nodejs ffmpeg awscli
# Download and start the transcoding service, e.g., using pm2

---

7. Load Testing and Monitoring

• Tools like stress-ng are used to generate CPU load on the transcoding service for testing auto scaling and system stability.

• Monitoring scripts continuously track resource usage and log scaling events.

---

8. Challenges and Learnings

• Precise ALB path rule configuration was required to ensure correct traffic distribution.

• Fine-tuning S3 and CloudFront caching behavior was necessary to avoid delays in static resource updates.

• Designing Lambda event handlers required balancing extensibility with error handling.

• Writing Terraform code deepened understanding and practical skills in Infrastructure as Code.

---

9. Future Directions

• Plan to introduce containerization (Docker, ECS) and serverless architectures (Lambda + API Gateway) to reduce operational costs.

• Further enhance automated testing and deployment pipelines to improve system stability.

• Explore advanced monitoring and alerting solutions for early anomaly detection.

---