ANUJ
DATA×AI×ENGINEERING
INITIALIZING SYSTEM
ANUJ MUNDU
RETURN TO SELECTED WORK
SOURCE CODE
PROJECT 13•AI ENGINEERING•AI / ML

Deep RL Job Scheduling — Makespan Minimization

Optimizing single-machine job scheduling to minimize makespan using Proximal Policy Optimization (PPO)

Python 3.8+GymnasiumStable-Baselines3PPOReinforcement LearningNumPyMatplotlibOptimization
Policy
PPO
Proximal Policy Optimization
Baseline Comparison
vs SJF / FCFS
Benchmarked against classic scheduling heuristics
Environment
Gymnasium
Custom discrete-action queue simulator
Visualization
Gantt Charts
Automated schedule timeline rendering
// INTERACTIVE SYSTEM TELEMETRY & DIAGNOSTIC LAB
RUNTIME ENGINE & LATENCY BENCHMARK COMPARATOR

Empirical benchmark comparing INT8 Post-Training Quantized ONNX against vanilla TorchScript C++ tracing.

INFERENCE BATCH SIZE:
P95 Latency
24.8ms
Deterministic SLA
Throughput
40.3 FPS
Video streaming limit
RAM Footprint
14.2 MB
Model weight & graph
CPU Usage
38%
8-Core Edge node
Target: Sub-30ms budget on edge hardware✓ 3.1x Faster Than TorchScript

01 // SYSTEM OVERVIEW

Job scheduling is an NP-hard combinatorial optimization challenge ubiquitous in cloud compute clusters, manufacturing pipelines, and CPU execution queues. This project trains a Deep Reinforcement Learning agent using Proximal Policy Optimization (PPO) in a custom Gymnasium environment to dynamically dispatch queued jobs and minimize total makespan.

02 // THE PROBLEM & ENGINEERING SIGNIFICANCE

The Core Challenge

Traditional heuristics like FCFS (First-Come, First-Served) and SJF (Shortest Job First) perform well on static queues but fail to optimize complex non-linear arrival sequences where future job distributions are uncertain.

Why This Matters

Minimizing makespan directly reduces computing cluster operational costs and maximizes resource utilization.

Key Constraints:
  • •Designing a state representation that encodes both current machine state and variable-length queue properties.
  • •Preventing the RL policy from converging to suboptimal myopic actions.
  • •Benchmarking against theoretical and empirical scheduling baselines.

03 // DATA PIPELINE & PREPROCESSING

Input Format: Synthetic and empirical job queues parameterized by job size distributions, processing durations, and release timesSample Volume: Thousands of simulated scheduling episodes across varying queue sizes
Transformation Steps:
  • Observation space normalization (job duration divided by max duration bound)
  • Masking invalid actions for finished or empty queue slots
  • Reward shaping to penalize idle processor time and total elapsed makespan
Cleaning Strategy: Deterministic seeding across evaluation episodes for rigorous baseline comparability.

04 // SYSTEM ARCHITECTURE & DATA FLOW

Custom JobSchedulingEnv (Gymnasium) <-> Stable-Baselines3 PPO Agent <-> Policy Network (MlpPolicy) <-> Baseline Comparator (FCFS/SJF) <-> Gantt Chart Visualizer.

STEP 01Gymnasium · NumPy
Custom Gymnasium Env

Simulates job queue arrival, machine state, and step reward calculation.

STEP 02Stable-Baselines3 · PyTorch
PPO Policy Network

Multi-layer perceptron policy learning dynamic job selection probabilities.

STEP 03Python Algorithms
Heuristic Benchmarker

Runs parallel episodes with FCFS and SJF algorithms for direct performance comparison.

STEP 04Matplotlib
Gantt Renderer

Plots comparative execution timelines for qualitative schedule validation.

05 // MODEL ENGINEERING & HYPERPARAMETERS

Base Architecture: Proximal Policy Optimization (PPO) with Actor-Critic Architecture

Trained over 200,000 timesteps with vectorized environments.

Hyperparameters & Training Dynamics:
  • • Learning Rate: 3e-4
  • • n_steps: 2048
  • • batch_size: 64
  • • gamma: 0.99
  • • clip_range: 0.2
Loss Function: PPO Clipped Surrogate Objective + Value Function MSE - Entropy Bonus
Trade-off Rationale: Balanced exploration vs exploitation with entropy coefficient to prevent premature convergence on pure SJF.

06 // FAILURE ANALYSIS & ZERO-TRUST SAFEGUARDS

OBSERVED FAILURE MODES UNDER STRESS
  • • Out-of-distribution job sizes exceeding the maximum normalized observation ceiling.
  • • Rapid bursts of identically-sized jobs where all actions have equal expected return.
Mitigation & Fallback: Dynamic feature scaling in the environment observation wrapper.

07 // PRODUCTION DEPLOYMENT SPECS

Serving Framework
Python Policy Inference Module
Containerization
Reproducible Python virtual environment with pinned dependencies
P95 SLA
1.4ms per dispatch decision
Throughput
700 actions/sec

08 // ARCHITECTURAL DECISIONS & TRADE-OFFS

Employed Stable-Baselines3 PPO rather than DQN.
Why: PPO's actor-critic framework and clipped surrogate objective provide superior training stability on scheduling policy spaces.
Alternative Discarded: DQN (which exhibited high value overestimation on scheduling horizons).
Engineered automated Gantt chart visualizers (`plot_gantt.py`).
Why: Enables instant visual verification of processor utilization and idle gaps across algorithms.
Alternative Discarded: Text-only metrics tables.

09 // PLANNED IMPROVEMENTS & NEXT REVISIONS

  • →Expand environment to multi-machine heterogeneous cluster scheduling.
  • →Incorporate attention-based Graph Neural Networks (GNNs) for variable-length job queue representations.
BACK TO ALL PROJECTS
INSPECT ON GITHUB