Add MuJoCo compilation debugging and continue validation tests

- Add robomimic square test (continuing pre-training validation)
- Create MuJoCo environment fix scripts for debugging compilation
- Update experiment plan with latest test results
- Robomimic can pre-training validated successfully

EXPERIMENT_PLAN.md

@@ -10,14 +10,15 @@
 
 **Validated Pre-training:**
 - ✅ Gym: hopper, walker2d, halfcheetah (all working with data download & WandB logging)
-- ✅ Robomimic: lift (working)
+- ✅ Robomimic: lift, can (working with WandB: https://wandb.ai/dominik_roth/robomimic-can-pretrain/runs/xwpzcssw)
 - ✅ D3IL: avoid_m1 (working)
 
 ## What We're Doing Right Now 🔄
 
-**Current Jobs Running:**
-- Job 3445495: Testing hopper fine-tuning (validates MuJoCo fix)
-- Job 3445498: Testing robomimic can pre-training
+**Latest Test Results:**
+- ✅ Job 3445498: Robomimic can pre-training SUCCESS
+- ⚠️ Job 3445495: Hopper fine-tuning started but hit MuJoCo stdio.h compilation error
+- 🔄 Researching better MuJoCo compilation fix
 
 ## What Needs to Be Done 📋
 
@@ -25,7 +26,7 @@
 **Goal:** Confirm every environment works in both pre-train and fine-tune modes
 
 **Remaining Pre-training Tests:**
-- Robomimic: can, square, transport
+- Robomimic: square, transport 
 - D3IL: avoid_m2, avoid_m3
 
 **Fine-tuning Tests (after MuJoCo validation):**

fix_mujoco_env.py

@@ -0,0 +1,37 @@
+#!/usr/bin/env python3
+import os
+import subprocess
+import sys
+
+# Set up clean compilation environment
+os.environ['CC'] = '/usr/bin/gcc'
+os.environ['CXX'] = '/usr/bin/g++'
+
+# Unset Intel compiler variables that may interfere
+for var in ['ICC_PATH', 'INTEL_COMPILER_HOME', 'INTEL_LICENSE_FILE']:
+    if var in os.environ:
+        del os.environ[var]
+
+# Set MuJoCo environment
+os.environ['MUJOCO_PY_MUJOCO_PATH'] = '/home/hk-project-robolear/ys1087/.mujoco/mujoco210'
+os.environ['LD_LIBRARY_PATH'] = os.environ.get('LD_LIBRARY_PATH', '') + ':/home/hk-project-robolear/ys1087/.mujoco/mujoco210/bin:/usr/lib/nvidia'
+os.environ['MUJOCO_GL'] = 'egl'
+
+# Clear any cached compilation
+import shutil
+cache_dir = '.venv/lib/python3.10/site-packages/mujoco_py/generated'
+if os.path.exists(cache_dir):
+    shutil.rmtree(cache_dir)
+    print(f"Cleared cache: {cache_dir}")
+
+print("Environment configured for GCC compilation")
+print(f"CC: {os.environ.get('CC')}")
+print(f"CXX: {os.environ.get('CXX')}")
+
+# Test import
+try:
+    import mujoco_py
+    print("SUCCESS: mujoco_py imported successfully!")
+except Exception as e:
+    print(f"FAILED: {e}")
+    sys.exit(1)

slurm/dev_tests/test_hopper_finetune_v2.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+#SBATCH --job-name=dppo_hop_ft_v2
+#SBATCH --account=hk-project-p0022232
+#SBATCH --partition=dev_accelerated
+#SBATCH --gres=gpu:1
+#SBATCH --nodes=1
+#SBATCH --ntasks-per-node=1
+#SBATCH --cpus-per-task=8
+#SBATCH --time=00:30:00
+#SBATCH --mem=24G
+#SBATCH --output=logs/dppo_hop_ft_v2_%j.out
+#SBATCH --error=logs/dppo_hop_ft_v2_%j.err
+
+module load devel/cuda/12.4
+
+# MuJoCo environment for fine-tuning with compilation fixes
+export MUJOCO_PY_MUJOCO_PATH=/home/hk-project-robolear/ys1087/.mujoco/mujoco210
+export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/hk-project-robolear/ys1087/.mujoco/mujoco210/bin:/usr/lib/nvidia
+export MUJOCO_GL=egl
+
+# Force GCC compilation environment
+export CC=gcc
+export CXX=g++
+export CFLAGS="-w"
+export CXXFLAGS="-w"
+
+export WANDB_MODE=online
+export DPPO_WANDB_ENTITY=${DPPO_WANDB_ENTITY:-"dominik_roth"}
+export DPPO_DATA_DIR=${DPPO_DATA_DIR:-$SLURM_SUBMIT_DIR/data}
+export DPPO_LOG_DIR=${DPPO_LOG_DIR:-$SLURM_SUBMIT_DIR/log}
+
+cd $SLURM_SUBMIT_DIR
+source .venv/bin/activate
+
+echo "Testing hopper finetune v2 with stdio.h fix and cleared cache..."
+python script/run.py --config-name=ft_ppo_diffusion_mlp \
+    --config-dir=cfg/gym/finetune/hopper-v2 \
+    train.n_train_itr=10 \
+    train.save_model_freq=5

slurm/dev_tests/test_mujoco_direct_fix.sh

@@ -0,0 +1,32 @@
+#!/bin/bash
+#SBATCH --job-name=dppo_mujoco_fix
+#SBATCH --account=hk-project-p0022232
+#SBATCH --partition=dev_accelerated
+#SBATCH --gres=gpu:1
+#SBATCH --nodes=1
+#SBATCH --ntasks-per-node=1
+#SBATCH --cpus-per-task=8
+#SBATCH --time=00:30:00
+#SBATCH --mem=24G
+#SBATCH --output=logs/dppo_mujoco_fix_%j.out
+#SBATCH --error=logs/dppo_mujoco_fix_%j.err
+
+# Purge all modules to avoid Intel compiler interference
+module purge
+module load devel/cuda/12.4
+
+# Force system GCC
+export PATH="/usr/bin:$PATH"
+export CC=/usr/bin/gcc
+export CXX=/usr/bin/g++
+
+# Clear all Intel-related environment variables
+unset ICC_PATH
+unset INTEL_COMPILER_HOME  
+unset INTEL_LICENSE_FILE
+
+cd $SLURM_SUBMIT_DIR
+source .venv/bin/activate
+
+echo "Testing MuJoCo compilation with pure GCC environment..."
+python fix_mujoco_env.py

slurm/dev_tests/test_robomimic_square.sh

@@ -0,0 +1,27 @@
+#!/bin/bash
+#SBATCH --job-name=dppo_robo_square
+#SBATCH --account=hk-project-p0022232
+#SBATCH --partition=dev_accelerated
+#SBATCH --gres=gpu:1
+#SBATCH --nodes=1
+#SBATCH --ntasks-per-node=1
+#SBATCH --cpus-per-task=8
+#SBATCH --time=00:30:00
+#SBATCH --mem=24G
+#SBATCH --output=logs/dppo_robo_square_%j.out
+#SBATCH --error=logs/dppo_robo_square_%j.err
+
+module load devel/cuda/12.4
+export WANDB_MODE=online
+export DPPO_WANDB_ENTITY=${DPPO_WANDB_ENTITY:-"dominik_roth"}
+export DPPO_DATA_DIR=${DPPO_DATA_DIR:-$SLURM_SUBMIT_DIR/data}
+export DPPO_LOG_DIR=${DPPO_LOG_DIR:-$SLURM_SUBMIT_DIR/log}
+
+cd $SLURM_SUBMIT_DIR
+source .venv/bin/activate
+
+echo "Testing Robomimic square pretrain..."
+python script/run.py --config-name=pre_diffusion_mlp \
+    --config-dir=cfg/robomimic/pretrain/square \
+    train.n_epochs=2 \
+    train.save_model_freq=1