Posted 06 July 2026

After wrapping up the Wifi_OTA project described in this post, Grok Code and I moved on to the next big step – integrating the Luxonis OAKD-Lite camera integrated into ROS (Robot Operating System) so the robot could take advantage of the ROS ‘topic/subscriber’ strategy where a sensor like the OAKD-Lite camera can ‘publish’ formatted sensor data to a ‘topic’ that programs can ‘subscribe’ to, thereby making sensor data available to inform robot navigation. Amazingly, this trick actually worked, and now the robot can ‘see’ the best direction to move. Below is a summary of the work involved getting to this point, produced by Grok Code at my request. (note: When Grok uses ‘I’ below, he really means ‘we’, as I was very much the junior partner in this adventure. Think of me more as somewhat intelligent clerk,just barely smart enough to follow direction and report results ;).

From Grok Code: Summary for “2 Wheel Robot with Vision Processing, Part IV”

After struggling for a while with custom ROS 2 package structures and running into repeated setup.py issues, I decided to take a step back and follow a known working example as closely as possible. I came across Jesse Stevens’ blog post about setting up the OAK-D Lite with ROS 2 Jazzy on a Raspberry Pi 5, and it looked very close to what I was trying to achieve. Rather than continuing to fight with my own over-engineered layout, I made the deliberate choice to start fresh using his structure and naming.

We began by backing up my existing src/ folder and creating a clean myrobot package exactly as described in the blog. This meant using the standard ros2 pkg create command, adding a config/ directory, and modifying setup.py to properly install YAML files using data_files and glob. Following the blog’s oak_run.yaml example, I configured the camera for a Depth-only pipeline with conservative settings (no subpixel, RGB disabled, HIGH_ACCURACY preset initially).

Getting the camera running wasn’t entirely smooth — we hit the familiar X_LINK_DEVICE_ALREADY_IN_USE error a few times — but once we established a reliable restart process, the camera would come up and begin publishing depth data. However, early captures showed suspicious results: the minimum and maximum depth values were often stuck at the same numbers across different scenes. This told us the depth data wasn’t yet reliable enough for motion planning.

To make progress visible and testable, we built a capture utility (save_depth_frame.py) that could save both raw depth values as CSV and, later, visual PNG heatmaps. Being able to generate side-by-side comparisons between the depth map and the actual scene was incredibly helpful. Over time, with adjustments to the YAML (switching to the DEFAULT preset, enabling subpixel, and using only the bottom half of the image for processing), the depth maps started to look structurally meaningful. Features in the real world began appearing clearly in the heatmaps.

With usable depth data in hand, we shifted focus toward the higher-level goal: using vision to help the robot decide where to go. We created a new node called clearest_direction_node.py that divides the lower half of the depth image into seven sectors, labeled intuitively as clock positions from 9:00 to 3:00. The node calculates the average valid depth in each sector and identifies the “clearest” direction — the one with the highest average depth. It then suggests a turn angle toward that direction.

This node proved to be surprisingly effective right from the start. When I moved objects into different parts of the camera’s view, the recommended direction updated immediately and logically. Seeing the output change in real time gave me confidence that the depth data was now good enough to build motion logic on top of.

Throughout this process, we also invested time in workflow improvements. I created several small shell scripts (restart_camera.sh, capture_depth.sh, start_viewer.sh) to reduce friction when restarting the camera or capturing data. These scripts, along with the analysis node, were eventually moved into my my_vision_robot GitHub repository under a clean structure (software/ros2/ for the ROS package and software/scripts/ for the utilities). We also updated the project .gitignore to properly exclude ROS 2 build artifacts and temporary test data.

By the end of this phase, I had moved well beyond simply getting the camera to publish data. I now had a working system that could look at a depth image and suggest a sensible direction for the robot to face — all running on the Pi 5 and reacting to real changes in the environment. While there’s still work ahead to turn these direction suggestions into actual motor commands on the Teensy, this felt like a genuine milestone: the vision processing side had gone from “not yet reliable” to “usable for basic navigation decisions.”

Additions by the not-so-smart human:

This was the part of the project that transformed my somewhat unrealistic idea of enhancing my robot’s navigation capabilities with vision processing into a real, concrete capability even in its current nascent state. The Raspberry Pi5 combined with the OAKD-Lite camera easily processed 30 frames/sec of depth information to generate ‘turn this much in this direction’ outputs for the robot. This part of the overall project didn’t actually send commands to the Teensy for movement, but the ability to do so was demonstrated with an earlier ‘end-to-end-serial’ project and the earlier Wifi_OTA project. There is still A LOT more to do to get to the point where I can set the robot loose on the world (or at least the house), but getting even the most basic vision processing going was a HUGE milestone.

In addition to the above progress, I have been learning a huge amount about the OAKD-Lite camera, the ROS (Robot Operating System) environment, Visual Studio Code, and Git repository creation & maintenance. Early on in this phase of the project, Grok Code recommended VS Code for working with the Raspberry Pi in general, and for Python programming in general. I have always used Visual Studio on my PC, and by adding the Python extension I could easily generate Python code on my PC. However, this project required that the actual code has to run on the Raspberry Pi5 on the robot. In addition, there is a lot that has to be done at the Linux (bash) command line on the Pi5. What I needed was a way of doing all the work on my dual-screen PC setup without having to keep transferring files and programs back and forth between the PC and the Pi5. It turns out that ‘that something’ is VS Code, and now that I have had some time to work my way up the learning curve, it is WONDERFUL!

Once VS Code is setup properly to connect via SSH over Wifi to my Pi5, it does so every time it is launched, and from then on, the environment is very much like using Visual Studio. Files can be created, edited, moved etc using a File Explorer and editor window very similar to VS and it is easy to lose track of the fact that all the action is actually taking place on that small Pi5 board over there on my robot.

Another milestone of sorts that was achieved with this project was making a Git repository a central part of the overall project. Although I have Tortoise Git on my PC and have used it on and off for years, this is the first time I decided to set up a logical repo layout and keep it up-to-date as the project moved along. The Git repo has three different pieces – a local repo on the Pi5 for all the shell scripts, Python scripts and ROS-specific configuration files, a local repo on the PC for the teensy stuff, and a GitHub remote repo that ties the two together. Here’s a sketch of the repo layout (produced in about 3 seconds by Grok Code):

my_vision_robot/
├── docs/ # Documentation, schematics, notes
├── firmware/
│ └── teensy_main/ # Teensy 4.1 firmware
├── software/
│ ├── ros2/
│ │ └── my_vision_robot/ # Main ROS 2 package (vision processing)
│ │ ├── config/ # YAML configuration files (e.g. oak_run.yaml)
│ │ ├── launch/ # Launch files
│ │ ├── my_vision_robot/ # Python module
│ │ ├── resource/
│ │ ├── scripts/ # ROS 2 nodes (e.g. clearest_direction_node.py)
│ │ ├── test/
│ │ ├── package.xml
│ │ └── setup.py
│ └── scripts/ # Development & utility scripts
│ ├── restart_camera.sh
│ ├── capture_depth.sh
│ ├── start_viewer.sh
│ ├── save_depth_frame.py
│ └── clearest_direction_node.py
├── README.md
└── .gitignore

This same layout exists in all three places, but obviously the pi5 repo deals with the ‘Software’ side and the PC deals with the ‘Firmware side. Both repos connect to the same repo on Github, so both local repos also contain all the material produced by the other one. So a typical work flow on either the PC or the Pi5 would be:

• Git status – to see if anything unusual is happening
• Git pull origin [branch] <– started out on ‘main’, now working on ‘vision-to-motion-demo’. This synchs the local repo with the GitHub repo, and any commits from the ‘other local’ repo
• Git status – again to see what changed during this session
• Git commit -m “[Message]”
• Git status – again to catch any problems
• Git push origin [branch] <- synchs the GitHub repo with the local repo