Hands-On Reinforcement Learning – TD 1

This repository contains my individual work for the Hands-On Reinforcement Learning project. The project explores reinforcement learning (RL) techniques applied to the CartPole and Panda-Gym robotic arm environments. The goal is to implement and evaluate RL models using both custom PyTorch implementations and high-level libraries like Stable-Baselines3.

---

1. REINFORCE on CartPole

Implementation

• File: reinforce_cartpole.ipynb
  The REINFORCE (Vanilla Policy Gradient) algorithm was implemented using PyTorch. The model learns an optimal policy for solving the CartPole-v1 environment by updating the policy network using gradients computed from episode returns.

Training Results

• The model was trained for 500 episodes, showing a steady increase in total rewards. The goal (total reward = 500) was reached consistently after 400 episodes, confirming successful learning.
• Training Plot:
  
  (Figure: Total rewards increase per episode, indicating successful learning.)

Model Saving

• The trained model is saved as: reinforce_cartpole.pth.

Evaluation

• File: evaluate_reinforce_cartpole.ipynb
  The model was evaluated over 100 episodes, with the success criterion being a total reward of 500.

• Evaluation Results:

  • 100% of the episodes reached a total reward of 500, demonstrating the model’s reliability.

• Evaluation Plot:
  
  (Figure: The model consistently reaches a total reward of 500 over 100 evaluation episodes.)

• Example Video:
  REINFORCE CartPole Evaluation Video

---

2. A2C with Stable-Baselines3 on CartPole

Implementation

• File: a2c_sb3_cartpole.ipynb
  Implemented Advantage Actor-Critic (A2C) using Stable-Baselines3, which combines value-based and policy-based RL methods.

Training Results

• The model was trained for 500,000 timesteps, reaching a total reward of 500 consistently after 400 episodes. It continued training for 1,400 episodes, confirming stable convergence similar to the REINFORCE approach.
• Training Plot:
  
  (Figure: A2C training performance over time.)

Evaluation

• The trained model was evaluated, achieving 100% success, with all episodes reaching a total reward of 500.
• Evaluation Plot:
  
  (Figure: A2C model consistently achieves perfect performance over 100 episodes.)

Model Upload

• The trained A2C model is available on Hugging Face Hub:
  A2C CartPole Model

---

3. Tracking with Weights & Biases (W&B) on CartPole

Training with W&B

• File: a2c_sb3_cartpole.ipynb
  The A2C training process was tracked using Weights & Biases (W&B) to monitor performance metrics.
• W&B Run:
  W&B Run for A2C CartPole

Training Analysis

• Observations:
  • The training curve indicates that the A2C model stabilizes after 1,300 episodes.
  • The model exhibits strong and consistent performance.
• Training Plot:
  

Model Upload

• The trained A2C model (tracked with W&B) is available on Hugging Face Hub:
  A2C CartPole (W&B) Model

Evaluation

• Evaluation Results:
  • 100% of episodes reached a total reward of 500, confirming the model’s reliability.
• Evaluation Plot:
  
  (Figure: Evaluation results tracked using W&B.)
• Example Video:
  W&B Evaluation Video
  The A2C model stabilizes the balancing process more efficiently due to its superior performance compared to the REINFORCE approach.

---

4. Full Workflow with Panda-Gym

Implementation

• File: a2c_sb3_panda_reach.ipynb
  Used Stable-Baselines3 to train an A2C model on the PandaReachJointsDense-v3 environment, controlling a robotic arm to reach a target in 3D space.
• Training Duration: 500,000 timesteps
• Integrated Weights & Biases for tracking.

Training Results

• W&B Run for Panda-Gym:
  Panda-Gym W&B Run
• Observations:
  • The training curve shows consistent improvement over time.
  • The model successfully learns to reach the target efficiently.
  • It stabilizes after 2,500 episodes, with minor fluctuations in rewards.
• Training Plot:
  
  (Figure: The robotic arm’s learning progress over 500,000 timesteps.)

Model Upload and Evaluation

• The trained model is available on Hugging Face Hub:
  A2C Panda-Reach Model

Evaluation

• Evaluation Results:
  • The total reward across all episodes ranged between 0 and -1, indicating stable control.
  • 100% of episodes met the success criteria.
• Evaluation Plot:
  
  (Figure: The robotic arm’s performance in the PandaReachJointsDense-v3 environment.)
• Example Video:
  Panda-Gym Evaluation Video

---

Conclusion

This project successfully applied reinforcement learning techniques to control both a CartPole system and a Panda-Gym robotic arm using REINFORCE and A2C algorithms. The experiments demonstrated that:

• REINFORCE efficiently learned an optimal policy for CartPole but required more episodes to stabilize.
• A2C (Stable-Baselines3) improved training stability and efficiency, reaching optimal performance faster.
• Weights & Biases (W&B) was valuable for tracking and analyzing training performance in real-time.
• The Panda-Gym experiment showed that A2C effectively trained the robotic arm to reach targets in 3D space.

These results confirm the effectiveness of policy-gradient-based RL methods for solving control and robotics problems, highlighting the advantages of actor-critic approaches in stabilizing learning. Future work could explore more advanced RL algorithms (e.g., PPO, SAC) and extend experiments to more complex robotic tasks.

---