rl-lunar-lander
🚀 Deep Q-Learning — Lunar Lander
Training a reinforcement learning agent to land a spacecraft between two flags using a Deep Q-Network (DQN) with experience replay and a target network. The agent starts by crashing randomly and, through trial and error, learns a control policy that reliably solves the environment.
▶️ The trained agent landing successfully. If the video doesn’t play inline on GitHub, open
assets/lunar_lander.mp4.
Result
Environment solved in 534 episodes (~13 minutes of training) — reaching an average score of 200+ points over 100 consecutive episodes, the threshold OpenAI Gym defines as “solved.”
The cyan line is the raw score per episode; the magenta line is the 100-episode moving average. The agent goes from ~-200 (crashing) to consistently landing above 200.
How it works
The agent is a Deep Q-Network that approximates the action-value function Q(s, a) — the expected return of taking action a in state s.
| Component | Detail |
|---|---|
| State space | 8 continuous values (position, velocity, angle, angular velocity, leg-contact flags) |
| Action space | 4 discrete actions (do nothing, fire left/main/right engine) |
| Network | Input(8) → Dense(64, relu) → Dense(64, relu) → Dense(4, linear) |
| Optimizer | Adam, learning rate 1e-3 |
| Discount (Îł) | 0.995 |
| Replay buffer | 100,000 transitions |
| Exploration | ε-greedy with decaying ε |
Three techniques make training stable:
- Experience replay — transitions
(s, a, r, s′, done)are stored in a buffer and sampled in random minibatches, breaking the correlation between consecutive steps. - Target network — a slowly-updated copy of the Q-network provides the TD targets, so the network isn’t chasing a moving objective.
- Soft target updates — after each learning step the target weights are nudged toward the online weights.
The loss is the mean-squared error between the predicted Q(s, a) and the Bellman target:
y = r if the episode terminates
y = r + γ · max Q_target(s′, a′) otherwise
Tech stack
- Python, TensorFlow / Keras — Q-network and training loop
- OpenAI Gym (
LunarLander-v2) — physics environment - NumPy — array ops and experience sampling
- PIL / imageio — rendering the agent’s landing video
Running it
pip install tensorflow gym[box2d] numpy pillow imageio pyvirtualdisplay
jupyter notebook C3_W3_A1_Assignment.ipynb
Run the cells top to bottom. Training the agent from scratch takes ~13 minutes on CPU.
Portfolio page
lunar-lander.html is a self-contained case-study page (no build step, no dependencies) for embedding this project in a personal portfolio. Open it in a browser or host it with GitHub Pages.
Note
This project is based on the Deep Q-Learning programming assignment from DeepLearning.AI’s Machine Learning Specialization (Course 3, Week 3). It is shared here as a completed portfolio piece to demonstrate applied reinforcement learning. If you are currently taking the course, please complete the assignment yourself first — the Coursera Honor Code asks that you don’t copy others’ solutions.