a67f474fc0
- Pre-training: diffusion model on offline D4RL data (200 epochs) - Fine-tuning: PPO fine-tune with online environment interaction - Dev test: 2 epochs only for quick verification, not full training
110 lines
3.3 KiB
Markdown
110 lines
3.3 KiB
Markdown
# DPPO Experiment Plan
|
|
|
|
## Current Status
|
|
|
|
### Setup Complete ✅
|
|
- Installation successful on HoReKa with Python 3.10 venv
|
|
- SLURM scripts created for automated job submission
|
|
- All dependencies installed including PyTorch, d4rl, dm-control
|
|
|
|
### Initial Testing
|
|
✅ **DPPO Confirmed Working on HoReKa with WandB**
|
|
- Successfully completed dev test (Job ID 3445117)
|
|
- Quick verification: 2 epochs only (not full training), loss reduction 0.2494→0.2010
|
|
- WandB logging working: https://wandb.ai/dominik_roth/gym-hopper-medium-v2-pretrain/runs/rztwqutf
|
|
- Model checkpoints and logging fully functional
|
|
- Ready for full 200-epoch production runs
|
|
|
|
## Experiments To Run
|
|
|
|
### 1. Reproduce Paper Results - Gym Tasks
|
|
|
|
**Pre-training Phase** (Train diffusion model on offline D4RL datasets):
|
|
- hopper-medium-v2 → diffusion model trained on offline data (200 epochs)
|
|
- walker2d-medium-v2 → diffusion model trained on offline data (200 epochs)
|
|
- halfcheetah-medium-v2 → diffusion model trained on offline data (200 epochs)
|
|
|
|
**Fine-tuning Phase** (PPO fine-tune diffusion model with online interaction):
|
|
- hopper-v2 → fine-tune pre-trained hopper model with PPO + online env
|
|
- walker2d-v2 → fine-tune pre-trained walker2d model with PPO + online env
|
|
- halfcheetah-v2 → fine-tune pre-trained halfcheetah model with PPO + online env
|
|
|
|
**Settings**: Paper hyperparameters, 3 seeds each
|
|
|
|
### 2. Additional Environments (Future)
|
|
|
|
**Robomimic Suite**:
|
|
- lift, can, square, transport
|
|
|
|
**D3IL Suite**:
|
|
- avoid_m1, avoid_m2, avoid_m3
|
|
|
|
**Furniture-Bench Suite**:
|
|
- one_leg, lamp, round_table (low/med difficulty)
|
|
|
|
## Running Experiments
|
|
|
|
### Quick Development Test
|
|
```bash
|
|
./submit_job.sh dev
|
|
```
|
|
|
|
### Gym Pre-training
|
|
```bash
|
|
./submit_job.sh gym hopper pretrain
|
|
./submit_job.sh gym walker2d pretrain
|
|
./submit_job.sh gym halfcheetah pretrain
|
|
```
|
|
|
|
### Gym Fine-tuning (after pre-training completes)
|
|
```bash
|
|
./submit_job.sh gym hopper finetune
|
|
./submit_job.sh gym walker2d finetune
|
|
./submit_job.sh gym halfcheetah finetune
|
|
```
|
|
|
|
### Manual SLURM Submission
|
|
```bash
|
|
# With environment variables
|
|
TASK=hopper MODE=pretrain sbatch slurm/run_dppo_gym.sh
|
|
```
|
|
|
|
## Job Tracking
|
|
|
|
| Job ID | Type | Task | Mode | Status | Duration | Results |
|
|
|--------|------|------|------|---------|----------|---------|
|
|
| 3445117 | dev test | hopper | pretrain | ✅ SUCCESS | 2m17s | [WandB](https://wandb.ai/dominik_roth/gym-hopper-medium-v2-pretrain/runs/rztwqutf) |
|
|
|
|
## Configuration Notes
|
|
|
|
### WandB Setup Required
|
|
```bash
|
|
export WANDB_API_KEY=<your_api_key>
|
|
export WANDB_ENTITY=<your_username>
|
|
```
|
|
|
|
### Resource Requirements
|
|
- **Dev jobs**: 30min, 24GB RAM, 8 CPUs, dev_accelerated
|
|
- **Production**: 8h, 32GB RAM, 40 CPUs, accelerated
|
|
|
|
## Issues Encountered
|
|
|
|
No issues with the DPPO repository - installation and setup completed successfully.
|
|
|
|
## Next Steps
|
|
|
|
### Immediate Tasks (To Verify All Environments Work)
|
|
|
|
1. **Test remaining Gym environments**:
|
|
- [ ] walker2d-medium-v2 (2 epochs dev test)
|
|
- [ ] halfcheetah-medium-v2 (2 epochs dev test)
|
|
|
|
2. **Test other environment types**:
|
|
- [ ] Robomimic: can task (basic test)
|
|
- [ ] D3IL: avoid_m1 (basic test)
|
|
|
|
3. **Full production runs** (after confirming all work):
|
|
- [ ] Full pre-training: hopper, walker2d, halfcheetah (200 epochs each)
|
|
- [ ] Fine-tuning experiments
|
|
|
|
**Status**: Only hopper-medium-v2 confirmed working. Need to verify other environments before production runs. |