Devlog
behind-the-scenes on my projects
ACT policy trainingTry training a lerobot act policy Work Done: Got familiarized with lerobot policy training parameters Created automated scripts for training and uploading models on vast.ai instanc…
Restarting Zima in SwitzerlandTest zima hardware to see if it still works after being shipped to europe. Come up with next actionable steps for policy implementation Work Done: connected to zima using lerobot i…
Teleoperation tuning and dataset collectionTest dataset recording, replay, upload, visualization and teleoperation smoothness Work Done: Tested and tuned full dataset collection pipeline, from teleoperation to online upload…Fixing electrical and firmware issuesResume zima project, fix motor driver, arm issues, and plan next steps Work Done: installed new motor driver after short circuiting the last one debugged intermittent joint failure…
LeRobot dataset collection testingCreate lerobot teleoperation interface and test dataset collection Work Done: Tested zimarobot lerobot interface implemented and tested lerobot teleoperation ps3 controller interfa…LeRobot interface implementationImplement Zima LeRobot interface and continue VLA readings and implementation planning Work Done: installed lerobot and finalized communication architecture design looked for obser…Exams and data collection improvementsLock in on finals and improve Zima data collection system to include all relevant manipulation data (arm delta commands, joint state, multi camera, etc.) What I did: explored huggi…Inverse kinematics tuningGet a functional / smooth arm control system working for zima What I did: read http://www.incompleteideas.net/IncIdeas/BitterLesson.html (Sutton, 2019) + Era of experience paper in…
Zima arm kinematics testingPolish and test arm kinematics on hardware What I did: Added joint and workspace limits Further tested ik solver in simulation Completed ros2 connected arm controller Sucessfully T…
Zima arm kinematics (pt. 2)Complete second iteration of zima arm kinematic solver What I did: wrote a kd tree and gradient descent based ik solver for zima's arm What worked: Gained a much better understandi…Zima arm kinematicsStart setting up forward and inverse kinematics solving for zima arm What I did: used http://www.pabr.org/sixlinux/sixpair.c to set bluetooth host mac of ps3 controller to raspi ma…Zima arm validationTest zima arm ability to pick up various objects, + clean up internal wiring What I did: watched a recent robotic learning phd defence from CMU https://youtu.be/bWVt5uX57TU Tested…
Finishing the Zima arm reworkHave a fully functioning gripper capable of lifting medium sized objects (~0.5kg max, 10cmx10cmx10cm) What I did: Finished designing, printing, assembling, and testing zima's new a…
MATH 307 project: SVD image compressionN/A What I did: Created an image compression script using the singular value decomposition of an image for Applied Linear Algebra course (UBCO MATH 307) What worked: What failed: N…
Completed Zima gripper redesignFinish designing new zima gripper and start 3d printing What I did: Finished designing new gripper with camera mount and compliant fingers finished designing new wrist to hand join…
Smaller model experimentTrain a smaller model with same dataset to see if overfitting is reduced What I did: Trained a smaller (5M param) model on the 200M word wikipedia dataset Learned a bit more about…Devlog CI/CD + internal electronics revamp planningGet devlog to auto upload to eryk.ca + start on KiCAD electrical schematic + pcb design What I did: created new ci/cd and markdown parsing code to eryk.ca repo downloaded KiCAD and…
Larger model trainingTrain larger model on larger dataset for longer, trying to acheive more coherent output What I did: Updated wikipedia scraper algorithm to use priority queue instead of greedy sear…
GPT training — first coherent sentences!Debug model output and train semi coherent model What I did: Debugged model training loop implemented automated, dockerized training setup Trained first coherent model on Vast.ai S…
GPT architectureBuild and test GPT architecture (not training yet) What I did: Finished attention head design finished implemeing full GPT architecture Implemented basic tokenizer and model infere…Starting the transformer implementationBegin implementing transformer paper + gpt1 in pytorch What I did: Revisted attention is all you need learned about layer normalization implemented attention head from scratch in p…
Real-world robustness improvementsMake model reliably execute search and approach strategy in real life What I did: Collected more real world demonstrations evaluated model in real world again optimized data loadin…Real-world policy evaluationevaluate the policy in real life and iterate What I did: reduced max wheel speed and compromised on camera resolution fps tradeoff, reducing motion blur tried on device inference f…Real-world generalizationCollect more real world data and continue model fine tuning What I did: Collected 20 more episodes of real world data, with varying scene setups and starting positions Further fine…Quick bugfixFix model inference jitter bug What I did: debugged model inference, removing inference jitter turned out to be teleopcontroller sending competing motor commands at control rate Wh…
3D printing and real-world task completion!Print a backing for zima's body and begin collecting real world data What I did: Reinstalled Freecad and moved cad models into the zima repo Set up 3d printer again Collected 10 ep…
Sim-to-real prepbegin testing sim model performance on real hardware, and set up pipeline for real world data collection What I did: ported the nnconroller class from the mujoco simulator to a ros…
macOS compatibilityGet simulation, model inferencing, and training working on macbook What I did: Refactored mujoco teleoperation interface to use flask web server mainly to circumvent macos quirks,…
Evaluating the model in a more realistic sim environmentCreate a more realistic sim environment (prepping for sim2real) and see if model can generalize What I did: create a simulated bedroom environment in mujoco created simulated rubik…
EUREKA — task successfully completed!figure out how to prevent regression to mean / mode collapse What I did: watched sergey levine lecture on IL to get some background and inspiration https://www.youtube.com/watch?v=…
Model improvementsIterate on model architecture and training recipe What I did: Recorded new (cleaner?) dataset, with more consistent action completion strategy (turn clockwise ONLY while searching…
Model evaluation and improvementTest model in simulation, iterate and improve What I did: Wrote a model adapter for the mujoco simulation to evaluate performance explored why model was under performing tested var…
Dataset loadingOne sentence What I did: Got torch dataset and dataloader working with hdf5 data transformed data into resnet format Finished first iteration of model design Wrote training and tes…Building the model and collecting dataStart building pytorch action model and training pipeline. Collect dataset What I did: Collected 10 minutes of demonstrations (~70 demonstrations) Created the ActionResNet model (r…
MuJoCo setupSet up a full teleloperated simulation in mujoco with data collection What I did: created a wheeled / tank robot model in MJCF debugged weird physics added camera functionality to…Hardware improvementsInstall cameras and clean up internals What I did: Reorganized / cleaned up circuit board layout inside main enclosure Installed 2 cameras (1 wrist and 1 forward mounted) Wrote bas…
Sim tryouts and selectionimplement a basic tracked robot in PyBullet What I did: Read start of PyBullet quickstart quide Gave up on PyBullet because it was insanely slow despite being "lightweight" Made a…Running the hardware againConnect to on board computer remotely, control tracks and arm using ros2 commands What I did: turned on the robot fixed networking issues manually published ros2 messages from zima…Brushing up on PyTorch basicstraing a basic CNN on MNIST dataset using pytorch What I did: Following intoductory CNN MNIST tutorial What worked: Model training success (98% accuracy and test set) What failed:…
Laying the hardware groundworkHave a fully functional hardware prototype What I did: CAD modelled and 3D printed frame Hacked together tracks + motor encoders for tracked base CAD modelled and printed 5DOF arm…
Restarting Zima in SwitzerlandTest zima hardware to see if it still works after being shipped to europe. Come up with next actionable steps for policy implementation Work Done: connected to zima using lerobot i…Teleoperation tuning and dataset collectionTest dataset recording, replay, upload, visualization and teleoperation smoothness Work Done: Tested and tuned full dataset collection pipeline, from teleoperation to online upload…Fixing electrical and firmware issuesResume zima project, fix motor driver, arm issues, and plan next steps Work Done: installed new motor driver after short circuiting the last one debugged intermittent joint failure…LeRobot dataset collection testingCreate lerobot teleoperation interface and test dataset collection Work Done: Tested zimarobot lerobot interface implemented and tested lerobot teleoperation ps3 controller interfa…LeRobot interface implementationImplement Zima LeRobot interface and continue VLA readings and implementation planning Work Done: installed lerobot and finalized communication architecture design looked for obser…Exams and data collection improvementsLock in on finals and improve Zima data collection system to include all relevant manipulation data (arm delta commands, joint state, multi camera, etc.) What I did: explored huggi…Inverse kinematics tuningGet a functional / smooth arm control system working for zima What I did: read http://www.incompleteideas.net/IncIdeas/BitterLesson.html (Sutton, 2019) + Era of experience paper in…Zima arm kinematics testingPolish and test arm kinematics on hardware What I did: Added joint and workspace limits Further tested ik solver in simulation Completed ros2 connected arm controller Sucessfully T…Zima arm kinematics (pt. 2)Complete second iteration of zima arm kinematic solver What I did: wrote a kd tree and gradient descent based ik solver for zima's arm What worked: Gained a much better understandi…Zima arm kinematicsStart setting up forward and inverse kinematics solving for zima arm What I did: used http://www.pabr.org/sixlinux/sixpair.c to set bluetooth host mac of ps3 controller to raspi ma…Zima arm validationTest zima arm ability to pick up various objects, + clean up internal wiring What I did: watched a recent robotic learning phd defence from CMU https://youtu.be/bWVt5uX57TU Tested…Finishing the Zima arm reworkHave a fully functioning gripper capable of lifting medium sized objects (~0.5kg max, 10cmx10cmx10cm) What I did: Finished designing, printing, assembling, and testing zima's new a…MATH 307 project: SVD image compressionN/A What I did: Created an image compression script using the singular value decomposition of an image for Applied Linear Algebra course (UBCO MATH 307) What worked: What failed: N…Completed Zima gripper redesignFinish designing new zima gripper and start 3d printing What I did: Finished designing new gripper with camera mount and compliant fingers finished designing new wrist to hand join…Smaller model experimentTrain a smaller model with same dataset to see if overfitting is reduced What I did: Trained a smaller (5M param) model on the 200M word wikipedia dataset Learned a bit more about…Devlog CI/CD + internal electronics revamp planningGet devlog to auto upload to eryk.ca + start on KiCAD electrical schematic + pcb design What I did: created new ci/cd and markdown parsing code to eryk.ca repo downloaded KiCAD and…Larger model trainingTrain larger model on larger dataset for longer, trying to acheive more coherent output What I did: Updated wikipedia scraper algorithm to use priority queue instead of greedy sear…GPT training — first coherent sentences!Debug model output and train semi coherent model What I did: Debugged model training loop implemented automated, dockerized training setup Trained first coherent model on Vast.ai S…GPT architectureBuild and test GPT architecture (not training yet) What I did: Finished attention head design finished implemeing full GPT architecture Implemented basic tokenizer and model infere…Starting the transformer implementationBegin implementing transformer paper + gpt1 in pytorch What I did: Revisted attention is all you need learned about layer normalization implemented attention head from scratch in p…Real-world robustness improvementsMake model reliably execute search and approach strategy in real life What I did: Collected more real world demonstrations evaluated model in real world again optimized data loadin…Real-world policy evaluationevaluate the policy in real life and iterate What I did: reduced max wheel speed and compromised on camera resolution fps tradeoff, reducing motion blur tried on device inference f…Real-world generalizationCollect more real world data and continue model fine tuning What I did: Collected 20 more episodes of real world data, with varying scene setups and starting positions Further fine…Quick bugfixFix model inference jitter bug What I did: debugged model inference, removing inference jitter turned out to be teleopcontroller sending competing motor commands at control rate Wh…3D printing and real-world task completion!Print a backing for zima's body and begin collecting real world data What I did: Reinstalled Freecad and moved cad models into the zima repo Set up 3d printer again Collected 10 ep…Sim-to-real prepbegin testing sim model performance on real hardware, and set up pipeline for real world data collection What I did: ported the nnconroller class from the mujoco simulator to a ros…macOS compatibilityGet simulation, model inferencing, and training working on macbook What I did: Refactored mujoco teleoperation interface to use flask web server mainly to circumvent macos quirks,…Evaluating the model in a more realistic sim environmentCreate a more realistic sim environment (prepping for sim2real) and see if model can generalize What I did: create a simulated bedroom environment in mujoco created simulated rubik…EUREKA — task successfully completed!figure out how to prevent regression to mean / mode collapse What I did: watched sergey levine lecture on IL to get some background and inspiration https://www.youtube.com/watch?v=…Model improvementsIterate on model architecture and training recipe What I did: Recorded new (cleaner?) dataset, with more consistent action completion strategy (turn clockwise ONLY while searching…Model evaluation and improvementTest model in simulation, iterate and improve What I did: Wrote a model adapter for the mujoco simulation to evaluate performance explored why model was under performing tested var…Dataset loadingOne sentence What I did: Got torch dataset and dataloader working with hdf5 data transformed data into resnet format Finished first iteration of model design Wrote training and tes…Building the model and collecting dataStart building pytorch action model and training pipeline. Collect dataset What I did: Collected 10 minutes of demonstrations (~70 demonstrations) Created the ActionResNet model (r…MuJoCo setupSet up a full teleloperated simulation in mujoco with data collection What I did: created a wheeled / tank robot model in MJCF debugged weird physics added camera functionality to…Hardware improvementsInstall cameras and clean up internals What I did: Reorganized / cleaned up circuit board layout inside main enclosure Installed 2 cameras (1 wrist and 1 forward mounted) Wrote bas…Sim tryouts and selectionimplement a basic tracked robot in PyBullet What I did: Read start of PyBullet quickstart quide Gave up on PyBullet because it was insanely slow despite being "lightweight" Made a…Running the hardware againConnect to on board computer remotely, control tracks and arm using ros2 commands What I did: turned on the robot fixed networking issues manually published ros2 messages from zima…Brushing up on PyTorch basicstraing a basic CNN on MNIST dataset using pytorch What I did: Following intoductory CNN MNIST tutorial What worked: Model training success (98% accuracy and test set) What failed:…Laying the hardware groundworkHave a fully functional hardware prototype What I did: CAD modelled and 3D printed frame Hacked together tracks + motor encoders for tracked base CAD modelled and printed 5DOF arm…