EUREKA — task successfully completed!

Goal: figure out how to prevent regression to mean / mode collapse

What I did:
- watched sergey levine lecture on IL to get some background and inspiration

• discretized action space since it is tractable with 2dof (only 4 bins for current task)
• tried to fix data by removing idle franes
• tried to fix training weights using inverse frequency
• collected new, simpler dataset with reduced sampling rate and more balanced action distribution
• FIXED THE SIM EVALUATION PROBLEM!
  • turned out to be a missing model.eval() ......

What worked:
- Balancing the dataset (trimming idle frames), discretizing action space, and ensuring the model was in eval mode got it to a functional state!
- Found good (>90% task success rate) hyperparameters to be:
- action chunk length: 4, action history: 4, actions discretized into 4 bins (forward, left, right, stop)
- Unfrozen resnet backbone with 1e-5 LR
- 3e-4 LR for action head
- training for ~10 epochs with 20000 samples
- Got more stable training with lower learning rates (the ones listed above)
- randomly sampling from the softmax of the output improves task completion rate noticably. It results in smoother motion and in some cases gives the model a kick in the right direction forcing it out of local minima
-

Demo of multiple successful task completions

What failed:
- Leaving model in training mode during sim evaluation caused BatchNorm layers to not work correctly, resulting in unexpected behavior
- model still has certain failure modes in simulation. Eg. approaching wrong color cube, looking at green cube then not approaching, etc

Key learning:
- Multimodal data is an issue for behaviour cloning on continuous action spaces in general (sergey levine lecture)
- expressive contiuous distributions solutions:
- mixture of gaussians: output X number of gaussian distributions from the model, then sample from them at inference time. (How do you optimize this / what is the training objective? Sergey said you take negative log of the MoG formula? Need to understand better)
- diffusion models: work by learning to remove noise from an input. This can be used in policy learning as well, by keeping state the same (image / observations) and noising the action to create training data, then training the model to denoise the action.
- High dimension discretization solutions:
-

    - if you bin all dimensions you get exponentially many bins (eg. cannot bin every possible state of a 6dof arm tractably) Solution is to discretize each dimension individually
    - Can be done using autoregressive discretization. We have a sequence model output each action dimension 
    - ![Pasted image 20251115092110](media/pasted-image-20251115092110.png)
    - This works because we are only binning one dimension at a time (so if we have an action dimension $a_{t,0}$ from -1 to 1 we can put it in 10 bins for example) As such, the discretization problem becomes tractable. This also makes sense since dimension $a_{t,i+1}$ is predicted GIVEN $a_{t,i}$ , so we are only predicting "how likely is dimension i+1 to fall into this bin GIVEN that dimension i has value x".
    - So at each "time step" of the sequence model, we output a probability distribution of each action dimension. 
    - Notice how at training time, we input the last action dimension (ground truth), but during inference, we would sample the distribution (possibly just picking the largest bin, or using random sampling) This is similar to how GPT / LLM style models work as well. its **auto-regressive**, since it uses its past output as input to the next time step
    - ![Pasted image 20251115093652](media/pasted-image-20251115093652.png)
    - This way, you train your entire policy while also discretizing the action space.
    - Why not just discretize each dimension individually beforehand? (eg. bin each dimension into 10 bins and call it a day)?
        - From what i understand right now, if we bin each action dimension and concatenate them, you end up with $\text{bin\_num} \times \text{action\_dimension}$ classes, which doesn't actually correspond to any specific action state, it would just be a flat list of possible values for each dimension, which doesnt make sense as an action distribution. 
    - You could instead train separate models to predict each dimension, and then bin the output there, but then you lose the dependencies between dimensions.

Next session:
- start doing sim to real work
- start with domain randomization
- then try to run model on hardware
- then collect data on hardware

Time spent: 11 hours