LeWAM · 2026-04-12 · 10 hours
Goal: Train a single task pick and place policy using LeWAM
Work Done:
- significantly improved dataloading speed by preprocessing before training
- av1 -> h264 encoding, long -> short gop length
- pre crop from 640 x 480 -> 256x256
- improved time per batch 8sec -> 0.05 sec
- trained LeWAM v0.3 single task policies, with and without video pretraining (single task pretraining)
Successes:
- Training is much faster despite using a much larger model, ~16k steps per 24 hours @ batch size 40
- gradient checkpointing + video preprocessing helps a lot
- video-only pretraining significantly improves video prediction quality at the same step count
- action-video gradient alignement is also better after training video only first
- policy is able to sometimes grasp the cube, with 1/20 chance it fully completes the task
Failures:
- policy is still unable to consistently complete the full pick and place task 19/20 times
- Mainly grasping issues. Potentially mismatch between the teleoperation commands and the recorded join state
- when teleoperating, need to squeeze gripper more than the actual gripper goes in order to apply force.
Next steps:
- video only pre train on full community dataset
- collect data on an easier task? rubiks cube is not easy to pick up with so101 standard gripper
- fine tune smolvla on the same data to see if its a data problem
Time spent: 10 hours