I Got AWS Credits. So I Built Something for the Community.

Introduction

When a cloud provider hands you a grant of credits, the instinct is usually to spin up a vanity cluster or run benchmark scripts until the balance hits zero. I chose a different path. The open-source ecosystem has a persistent infrastructure deficit: maintainers need reliable CI/CD compute, but self-hosted runners demand patching, network hardening, and isolation that most volunteer teams cannot sustain.

We built a public, auto-scaling CI/CD runner pool on AWS. The system ingests repository webhooks, provisions isolated Fargate tasks per pipeline run, executes build steps, archives artifacts, and tears down the environment within seconds of completion. It is not a toy. It is a production-grade platform designed to handle untrusted code, enforce zero-trust boundaries, and respect strict budget guardrails. The credits funded the initial architecture, but the design decisions were driven by the same constraints we face in commercial platform engineering: security, observability, and deterministic cost.

Why This Matters

Open-source maintainers routinely hit free-tier CI limits or abandon security hardening because managing runner infrastructure competes directly with shipping features. Commercial teams face the same problem internally: shared CI pools leak secrets, suffer from noisy-neighbor effects, and become operational liabilities when base images drift.

This architecture solves three concrete production problems:

1. Isolation at scale: Every build runs in a fresh microVM with dedicated IAM roles, preventing cross-tenant data leakage.
2. Deterministic cost control: The control plane enforces hard budget limits, automatically throttles concurrency, and shuts down idle components before credits expire.
3. Operational simplicity: By treating runners as immutable, ephemeral resources, we eliminate patch cycles, drift, and manual maintenance.

Platform engineers care about this pattern because it demonstrates how to build multi-tenant compute layers that are secure by default, observable out of the box, and economically sustainable. The same principles apply to internal CI, sandboxed testing environments, and serverless data pipelines.

How It Works

The system separates the control plane from the data plane. The control plane handles webhook ingestion, job scheduling, state tracking, and cost enforcement. The data plane consists of ephemeral Fargate tasks that execute build steps in isolated network segments.

flowchart TD
    subgraph External
        Dev[OSS Developer]
        Repo[GitHub/GitLab Repository]
    end

    subgraph Control_Plane
        GW[API Gateway]
        Validator[Webhook Validator Lambda]
        Queue[SQS Job Queue]
        Scheduler[Orchestrator Lambda]
        State[DynamoDB Job State]
        Guardrail[Budget Guardrail Lambda]
    end

    subgraph Data_Plane
        ECS[ECS Service Fargate]
        Runner[Runner MicroVM]
        ECR[ECR Base Images]
        Artifacts[S3 Build Artifacts]
        Logs[CloudWatch Logs]
    end

    subgraph Governance
        Billing[CloudWatch Billing Metrics]
        VPC[VPC Private Subnets]
        IAM[IAM Roles for Tasks]
    end

    Dev -->|Push Event| Repo
    Repo -->|Webhook| GW
    GW --> Validator
    Validator -->|Validated Payload| Queue
    Queue --> Scheduler
    Scheduler -->|Reads| State
    Scheduler -->|Provisions| ECS
    ECS -->|Pulls| ECR
    ECS -->|Executes| Runner
    Runner -->|Writes| Artifacts
    Runner -->|Streams| Logs
    Billing --> Guardrail
    Guardrail -->|Throttles| Scheduler
    VPC -.->|Network Isolation| ECS
    IAM -.->|Least Privilege| Runner

The workflow follows a strict sequence:

1. A repository push triggers a webhook to API Gateway.
2. The Webhook Validator Lambda verifies the signature, extracts the repository ID, and publishes a job message to SQS.
3. The Orchestrator Lambda consumes the message, checks DynamoDB for existing runs, and provisions an ECS Fargate task.
4. The Fargate task mounts a read-only base image, attaches a scoped IAM role, and connects to a private subnet with restricted security groups.
5. The runner clones the repository, executes the pipeline configuration, uploads artifacts to a tagged S3 bucket, and streams logs to CloudWatch.
6. Upon completion or timeout, the task self-destructs. The orchestrator updates the state machine and releases the concurrency slot.
7. A separate Budget Guardrail Lambda monitors CloudWatch billing metrics. If spend approaches the credit threshold, it reduces the SQS visibility timeout and pauses the orchestrator to prevent runaway costs.

This design ensures that no long-lived infrastructure persists between builds. Every component is stateless except the DynamoDB state store, which tracks job lifecycle and audit trails.

Core Concepts

The architecture relies on four foundational principles:

Ephemeral MicroVM Isolation Fargate tasks run on Firecracker microVMs. Each task gets dedicated CPU and memory allocations, isolated kernel namespaces, and separate network stacks. We treat these as disposable execution contexts. If a build crashes or compromises the environment, the microVM is terminated and never reused.

Least-Privilege IAM Roles for Tasks Every runner receives a dynamically scoped IAM role. The role grants read access to the ECR repository, write access to a specific S3 prefix, and CloudWatch log publishing permissions. We attach IAM policy boundaries at the task definition level to prevent privilege escalation through misconfigured pipeline scripts.

Event-Driven State Machine Jobs transition through explicit states: PENDING, PROVISIONING, RUNNING, ARCHIVING, COMPLETED, or FAILED. DynamoDB conditional writes enforce state transitions and prevent duplicate provisioning. The orchestrator uses idempotency keys derived from the webhook payload to handle retries safely.

Cost-Aware Concurrency Control The system tracks active tasks against a concurrency budget. The guardrail lambda evaluates CloudWatch billing metrics every 60 seconds. If the projected hourly burn rate exceeds 80% of the remaining credit pool, the orchestrator scales down by increasing SQS message delay and rejecting new jobs until spend stabilizes.

Examples & Code Walkthrough

Below is the Terraform module that defines the runner task. It enforces ARM64 architecture, strict memory/CPU boundaries, and attaches a scoped IAM role with a policy boundary.

resource "aws_ecs_task_definition" "runner" {
  family                   = "oss-ci-runner"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  execution_role_arn       = aws_iam_role.ecs_execution.arn
  task_role_arn            = aws_iam_role.runner_task.arn

  cpu    = "1024"
  memory = "2048"

  runtime_platform {
    cpu_architecture        = "ARM64"
    operating_system_family = "LINUX"
  }

  container_definitions = jsonencode([
    {
      name      = "runner"
      image     = "${var.aws_account_id}.dkr.ecr.${var.aws_region}.amazonaws.com/oss-runner:latest"
      essential = true

      logConfiguration = {
        logDriver = "awslogs"
        options = {
          "awslogs-group"         = "/ecs/oss-runner"
          "awslogs-region"        = var.aws_region
          "awslogs-stream-prefix" = "ecs"
        }
      }

      environment = [
        { name = "PIPELINE_ID", value = "${aws_cloudwatch_log_group.runner.name}" },
        { name = "ARTIFACT_BUCKET", value = aws_s3_bucket.build_artifacts.id }
      ]

      healthCheck = {
        command     = ["CMD-SHELL", "pgrep -f runner-agent || exit 1"]
        interval    = 30
        retries     = 3
        startPeriod = 60
        timeout     = 10
      }
    }
  ])
}

resource "aws_iam_role" "runner_task" {
  name = "runner-task-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow