Serverless Cold Start Performance Optimization

Based on Multi-Request Processing and Adaptive Hierarchical Scaling

Presented byAkshay Kumar A
Reg. No PRN23CS022
GuideAnison Abraham
Authors Yu Liu · Li Fu · Chenhao Sun
Published IEEE Access, September 2024
Overview

Presentation Agenda

01 Abstract
02 Problem Statement
03 Objectives
04 Introduction
05 Literature Review
06 Methodology
07 Experimental Setup
08 Result & Discussion
09 Conclusion
10 References
Section 01

Abstract

  • Background Study: Serverless computing offers high elasticity, cost-efficiency, and low operational overhead — but the platform-managed lifecycle introduces hidden performance costs.
  • Specific Problem: The critical cold start problem severely degrades user experience; traditional single-concurrency models worsen it under traffic spikes, and multi-concurrency alone cannot proactively scale.
  • Performance: Reduces the cold start rate by 18.4%–25.6% and average response time by 8.9%–27.5% via a hierarchical startup scheme combining single-instance multi-concurrency with an adaptive scaling threshold.
  • Future Opportunities: AI-driven reinforcement learning to auto-tune scaling parameters and extension to multi-node serverless workflows.
Section 02: Problem Statement

The Cold Start Bottleneck

  • On-Demand Reclaiming: Platforms release inactive function instances to save resources, but new requests trigger costly recreate-and-initialize processes.
  • Single-Concurrency Defect: Traditional models execute one request per instance, leading to frequent instance creation and heavy cold start cycles under traffic spikes.
  • Multi-Concurrency Limitations: While single-instance multi-concurrency improves reuse, scaling remains passive (lagging behind demand), leaving systems prone to sudden resource overloading or underutilization.
Section 03

Research Goals

  • Balance Performance & Cost: Minimize cold start delays while maintaining high resource utilization efficiency.
  • Proactive Instance Management: Introduce fine-grained instance control to perceive traffic changes ahead of time and trigger proactive scaling.
  • Asynchronous Closed-Loop Control: Build an off-the-critical-path system where request distribution is completely decoupled from instance scaling decisions.
Section 04: Introduction

Understanding the Serverless Domain

  • Paradigm Shift: Splits monolithic applications into fine-grained, event-driven functions.
  • Dynamic Operations: Features automatic elastic scaling and pay-as-you-go pricing.
  • The Infrastructure Gap: The platform manages resources automatically, which introduces a fundamental trade-off: maintaining expensive idle standby instances vs sacrificing performance to cold starts.
Section 05

Literature Review

Category Key Mechanisms Limitations Author
Runtime & Container Optimization Optimize runtimes, lightweight booting, secure runtimes Bound to OS/runtime; limits cross-language generality Du et al. (Catalyzer), Oakes et al. (SOCK), Li et al. (RunD)
Prewarming & Caching Warm up instances in advance based on heuristics Substantial idle resource overhead and waste Romero et al. (Faa$T), Fuerst et al. (FaasCache)
Resource Pooling Share/reuse isolated resources across functions High architectural and implementation complexity Stojkovic et al. (MXFaaS), Shillaker et al. (Faasm)
Scheduling & Load Balancing Optimize request routing and queue buffers Cannot eliminate scale-out latency spikes Lin et al., Ustiugov et al. (REAP)
Section 06 · Methodology 1 / 3

Core Concept of Hierarchical Startup

  • Dual-Parameter Guardrails: Governed by a Concurrency Limit (max requests per instance) and a Scaling Threshold (the instance-pool trigger for scaling out).
  • Scaling Threshold Synergy: Spawns a new instance before the active ones are fully saturated, seamlessly routing oncoming traffic to hot instances.
  • Dynamic Equilibrium: Quickly scales out during load spikes and safely reclaims idle containers after quiet periods.
Inst 1 Inst 2 Inst 3 Scaling Threshold Concurrency Limit Active Inst New Spawn Traffic Re-routed REQUEST LOAD Server 1 Server 2 Server 3 Server 4
Section 06 · Methodology 2 / 3

System Architecture & Controller

  • Modular Architecture: Comprises Controller, Scheduler, Hierarchical Scaling Component, and Monitoring Module.
  • Scheduler Logic: Evaluates active instance concurrency; routes requests to unsaturated instances to maximize reuse.
  • Proactive Scaling: Triggers instance startup when queue/concurrency hits scaling threshold.
API Gateway (Requests) Scheduler Async Queue Hierarchical Scaling Instances Hot Inst Cold Inst Scheduler Submodule Instance 1 80% Usage Instance 2 40% Usage Route to Unsaturated Message Queue Threshold Hierarchical Scaling Instance 1 (Hot) Initializing... Instance 2 (Hot)
Section 06 · Methodology 3 / 3

Asynchronous Processing & Queue Isolation

  • Parallel Loops: Decouples Request Reception (rapid client ACK), Request Consumption (batch pulling and queue classification), and State Monitoring (elastic scaling execution).
  • Fine-Grained Queue Isolation: Separates workloads into dedicated queues (compute-intensive vs I/O-intensive) with independent resource pools to preserve QoS.
  • Batch Scaling Execution: Determines scaling decisions at the batch level to drastically reduce container creation overhead.
Message Queue Loop 1: Request Reception Client API Gateway Rapid ACK Loop 2: Request Consumption Scheduler Inst 1 Inst 2 Poll Batch Loop 3: State Monitoring & Scaling Monitor Scale Engine
CPU Queue CPU Pool I/O Queue I/O Pool Time (s) Requests Batch Scaleout Decision Batch Scalein Decision Inst 1 Inst 2 Inst 3 Inst 4 Inst 5 Inst 6
Section 07

Experimental Configuration

  • Deployment: Alibaba Cloud's Function Compute
  • Workloads:
    • Compute-intensive (web services)
    • I/O-intensive (file processing)
    • Mixed (AI inference)
  • Evaluation Groups:
    • Group 1: Traditional Serverless (Benchmark)
    • Group 2: Single-Instance Multi-Concurrency
    • Group 3: Hierarchical Startup (Threshold = 1)
  • Stress Test: Up to 300 max concurrency, 10% incremental steps, 1-min intervals over 10 minutes
Section 08 · Results 1 / 2

Quantitative Performance

18–26%
Cold Start Rate Reduction
(vs Group 2)
5,932
Avg TPS (Scenario 1)
vs 4,312 (G2) · 1,911 (G1)
50ms
Avg Response Time
vs 69ms (G2) · 157ms (G1)
  • P99 Response Time: Drops to 127ms (from 1.90s in Group 1), ensuring tail-latency stability even under peak concurrency.
Section 08 · Results 2 / 2

Scenario Analysis

  • Compute-Intensive (Scenario 1): Highest optimization gains. Multi-core CPUs are fully utilized through concurrent scaling and thread execution.
  • I/O-Intensive (Scenario 2): Showcases structural limitations. Response times deteriorate slightly due to disk I/O bottlenecks and contention — blind concurrency expansion can be counterproductive.
  • Mixed Workloads (Scenario 3): Performance gains fall between compute-intensive and I/O-intensive, balancing parallel CPU execution and I/O wait boundaries.
Section 09

Key Takeaways & Future Directions

  • Major Contributions: Validated the theoretical design of a dynamic, asynchronous hierarchical scaling controller.
  • Systemic Advantages: Combines the low-cost footprint of on-demand scaling with the performance speed of prewarmed environments.
  • Future Horizons: Focus on AI-driven reinforcement learning to auto-tune parameters (limit and threshold) and expand the optimization model to multi-node serverless workflows.
Section 10

Key References

  • Du et al. (Catalyzer) — Sub-millisecond startup for serverless with initialization-less booting (ASPLOS 2020).
  • Oakes et al. (SOCK) — Rapid task provisioning with serverless-optimized containers (USENIX ATC 2018).
  • Romero et al. (Faa$T) — A transparent autoscaling cache for serverless applications (SoCC 2021).
  • Zhou et al. (AQUATOPE) — QoS-and-uncertainty-aware resource management for multi-stage workflows (ASPLOS 2022).

Thank You!

Any Questions?