Tag Archives: OAK-D Lite

Moving Vision Processing to My 4-Wheel Robot, Part 2

Leave a reply

Posted 03 August 2026,

Part 1 in this series laid out the strategy for adding vision processing capabilities to my 4-wheel robot. This post continues that effort.

The first step was to replace the Teensy 3.6 main controller with a Teensy 4.1. As part of this process, I also replaced the glued-on ‘Gnd’ and ‘+5V’ bussbars with a bit more finished product, as you can see in the following photos:

The next step is moving the old 4-wheel firmware to the ‘new-improved’ 4-wheel hardware. I started this by porting “C:\Users\Frank\Documents\Arduino\WallE3_Git\WallE3_Git\WallE3_Git.ino” to “C:\Users\Frank\Documents\Robot_Projects\VisionEnhaned4WheelRobot_V1\VisionEnhaned4WheelRobot_V1.ino”. This took a while, and I’m not sure I wouldn’t have been better off just cloning the entire WallE3_Git.ino file, but I “got it done’.

05 August 2026 Update:

It took a while, but I now have a working 4-wheel drive robot program running again on the 4-wheel chassis with a Teensy 4.1 main controller The only small ‘gotcha’ I encountered during the Teensy 3.5 -> Teensy 4.1 controller switch was the control lines for the right motor driver. It turns out the ‘speed’ (PWM) pin used on the Teensy 3.5 isn’t PWM-capable on the Teensy 4.1, so this required some minor pin swapping, as shown below:

Using the manual control features built accessible via the ‘CheckforUserInput() function, I was able to perform left/right 10 degree turns and speed up/slow down, all from my keyboard. I don’t have the second deck connected up yet, but that is next on the agenda.

Upgraded 4-wheel robot demonstrating basic motor

06 August 2026 Update:

4-wheel robot with Raspberry Pi5 and OAKD-Lite camera mounted on 2nd deck

At this point I have a complete 4-wheel drive robot assembled with a Raspberry Pi5 and OAKD-Lite camera, and I can send it basic (left/right/faster/slower/forward/reverse) movement commands via a serial port. In previous work on the 2-wheel robot I demonstrated the ability to send vision-processing-derived movement commands to the main processor from the Pi5 via a serial port.

It occurs to me that with the above capability, The robot doesn’t really need anything else for navigation and obstacle avoidance. The OAKD-Lite can provide visual depth information to the Pi5, and the Pi5 can send basic movement commands to the robot. In addition, it seems to me that I might be able to eliminate the entire ‘homing-to-charger’ software and hardware blocks, as the camera should be able to tell the Pi5 where the charger dock is located in the field of view, and the Pi5 should be able to tell the robot which way to turn and by how much, and when to speed up and slow down. This could make the robot a LOT simpler (as Elon Musk said about Tesla FSD “Once you have vision solved, you no longer need anything else’).

08 August 2026 Update:

The robot is alive! With Grok’s help I now have a python script running on the Pi5 that can direct the robot to turn left or right depending on image processing by the OAKD-Lite camera.

First, the Python shell script ‘restart_camera.sh’ is executed to get the OAKD-Lite camera ready:

Once the camera is ready, then the demo program ‘clearest_direction_node.py’ is executed on the Pi5 (with ‘VisionEnhanced4WheelRobot.ino already running on the Teensy 4.1)

Based on a rudimentary analysis of the depth (distance) distribution in the camera field of view, the ‘clearest_direction_node.py’ Python script sends left/right/stop commands to the Teensy 4.1 from the Pi5’s serial port to the Teensy 4.1’s ‘Serial1’ port. The commands use the already-existing robot manual motion control command structure shown below:

Here’s a short video showing the action

Here’s another short video showing the robot actually making 10-deg turns left & right in response to a simulated obstacle in the field of view:

And here is some of the telemetry output from the above run:

At this point in the project, we have demonstrated that visual ‘depth’ information can be used to guide the robot, and we have separately demonstrated that the Teensy firmware can be updated ‘Over The Air’ (OTA) via the wifi connection to the Pi5. I believe the next step should be to add wifi OTA to the ‘clearest_direction_node.py’ script in order to remove the requirement for physically attaching a USB cable to the Teensy 4.1 to update the firmware.

08 August 2026 Update:

Adding Wifi OTA to the current ‘clearest_direction_node.py’ configuration turned out to be relatively easy, as this ability had already been developed as a stand-alone demo back in June 2026. All we had to do was update the ‘board.txt’ file to point to the new location of ‘PostBuild_OTA.py’ on my PC, and make sure ‘Wifi_OTA.py’ was running on the Pi5 so it would look for changes in “/home/pi/my_vision_robot/firmware/latest.hex” and trigger the update.

09 August 2026 Update:

Thinking while drifting off to sleep last night, I realized that the addition of the Pi5 to the robot means that all the navigation and obstacle avoidance tasks (including possibly navigating to and connecting with the charger module) will be handled by the Pi5, which significantly simplifies the Teensy 4.1 code. All the Teensy code has to do now is respond to motion commands from the Pi5, monitor the various hardware sensors and update the rear-facing LED array.

Teensy Responsibilities:

  • Run the motors in response to motion commands from Pi5 on Serial1
  • Update the ‘state-of-charge’ (and left/right steering) LEDs
  • Monitor the output from three different 1NA169 current sensors (I_chg, I_tot, I_upper-deck)
  • Monitor the output from two different Vbatt voltage sensors (V_batt, V_upper-deck)
  • Monitor the ‘charger connected’ output from the TPS5100 battery pack
  • Monitor the rear-facing VL53L1X LIDAR for rear collision avoidance
  • Run the speaker
  • Home in on and connect to the charger module (may be done by the Pi5 instead)

Pi5 Responsibilities:

  • Manage all navigation and obstacle avoidance tasks
  • Possibly manage charger module homing (April tag?)
  • Create map (SLAM)?
  • Respond to rear-facing obstacle alerts from Teensy
  • Pass robot telemetry data on Serial1 to PC
  • Pass Pi5-software-related telemetry data to VS Code terminal

10 August 2026 Update:

Good progress on the vision-enhanced robot project today. Yesterday we installed two additional cases in the switch block that accepts manual control commands for the robot, to allow the user to command left/right turns of an arbitrary number of degrees at an arbitrary turn rate. The command format consists of the letter ‘L’ (left/CCW) or ‘R’ (Right/CW) followed by the number of degrees to turn (and optionally a specified turn rate in deg/sec), but the commands failed. Today (while waiting in the cellphone lot for my wife) I figured out what was wrong with the teensy command parsing code. Now I can type “L30.5” at a VS Code prompt that gets transmitted via SSH over Wifi to the Pi5, then via the dedicated serial to the Teensy. Then the Teensy calls ‘SpinTurn(30.5, rate = 30) to actually effect the turn. Here’s a short video showing a left (CCW) turn of approximately 90 deg at a 90 deg/sec rate, followed by another left turn of 90 deg at the default 30 deg/sec rate, followed by a right turn of 180 deg at the default 30 deg/sec rate:

We also implemented a replacement for my old telemetry display using the Visual Studio serial port monitor. In theory I could maintain that feature as I still have the HC-05 Bluetooth serial module installed on the robot, but I would rather put everything over wifi for better range & reliability. What we did instead was to have WallE_5.py log everything that comes to it on its serial port (Teeny ‘Serial1’) to a ‘telemetry.log’ file, which allows me to start up a new Pi5 terminal session in its own window and run

to display whatever telemetry information the teensy is sending to ‘Serial1’. Here’s a short sampling of the telemetry data setup:

15 August 2026 Update:

I discovered that the Teensy 4.1 was unresponsive – yuk! After some research and troubleshooting, I realized that I had fallen into a classsic Teensy 3.5->4.1 transition trap; Teensy 3.5 is 5V tolerant on (almost) all pins, while the 4.5 is not. I didn’t really look hard at the circuit before making the 3.5->4.1 change, but almost all the peripheral hardware produces 5V signals on the MCU pins – oops!

So, I have spent the last few days going backwards – from Teensy 4.1 to Teensy 3.5. I took the opportunity to improve wire/pin labelling while doing the reverse transition. Here is the updated schematic:

And here are some photos showing the wire labelling:

I created a small test program, originally to just test the rear-facing VL53L1X LIDAR, but later expanded to test all sensors. Here’s the code:

And here’s the output showing all three current sensors, the battery voltage, and the rear-facing VL53L1X LIDAR output (variations in the rear distanced are due to my waving my hand in front of the sensor).

I wasn’t quite sure what to make of the zero readings for Bot_I, but then I realized that when the USB cable is plugged into the Teensy 3.5, it provides all the current needed for the first deck peripherals, meaning the current through the Bot_I INA169 sensor is essentially zero.

Next I ported the MPU-6050 support code from the VisionEnhanced4WheelRobot project into my little demo program and put some code in ‘loop()’ to add the IMU heading output to the sensor line. This didn’t work at first, but with Grok’s help I traced the problem to a disconnected power/ground lead from the main power distribution bus to the MPU6050. Once this was connected, the demo program started working right away:

I also installed and tested a front-facing red laser diode and exercised the on-board speaker. At this point I believe I have exercised/tested all the sensors, the laser diode and the speaker. I haven’t run the motors with this program as I believe that will happen when I get back to the ‘mainline’ VisionEnhanced4WheelRobot.ino program.

16 August 2026 Update:

I back-ported the code from my VL53L1X_Demo program back to my VisionEnhanced4WheelRobot program and after the normal number of mistakes, got it going. Here’s a small bit of output from loop(), with me rotating the robot back and forth to show variation in the ‘Hdg’ and ‘RearCm’ values:

Then I fired up the VS Code connection to the pi5, started WallE_5.py supervisor program and verified that I could send motor commands in the WallE_5.py window and have the robot respond appropriately – yay. I also verified that ‘tail -f ./logs/telemetry.log’ in a separate VS Code terminal window produced the same telemetry output as above.

17 August 2026 Update:

I finally have the ‘VisionEnhanced4WheelRobot’ program running again with a Teensy 3.5. The last little bit was undoing the change in motor control pin assignments for the right-hand motor driver. Going from Teenys 3.5 to 4.1 I had to change the pin assignments to place the ‘speed’ pin on a PWM-able output pin, and going back I didn’t initially undo this pin assignment – all fixed now.

At this point the robot firmware is pretty much complete – it correctly monitors all three current sensors, the battery voltage, the current heading, and the rear distance reading. I think the next step is to work on the python side to add more navigation and obstacle-avoidance ‘smarts’.

Stay tuned!

Frank

2 Wheel Robot with Vision Processing, Part III

Leave a reply

Posted 28 June 2026

When I work on significant projects like the vision-enhanced robot project, I generally hold two different but related mental maps for the overall project.

One map describes the physical and/or logical entities that are needed for the overall project to succeed, such as wheel odometers, battery packs, power regulation/distribution, the OAKD-Lite camera, the pi5, wheel motors and drivers, etc.

The second mental map describes the software (here the term ‘software’ includes both the pi5 software and related teensy firmware) pieces needed to give the robot the ability to do what we want, primarily the enhanced navigation possible with vision processing via the OAKD-Lite camera and associated software. This mental map (at least for me) seems to be project oriented, where each project addresses different, mostly independent software capability implementations (Wifi_OTA, with its precursor projects ‘SerialPassthroughDemo’, Wifi_OTA_Demo, etc).

I tend to work on a big project like this from the top (system-view) and bottom (small sub-projects that will be later integrated into the overall project) at the same time. I have learned over the years that creating (and later modifying as needed) a clear top-down system architecture is absolutely crucial to improving the chances of getting someplace that looks like where you wanted to go. This systems architecture is the ‘aspirational view’ (to borrow a modern Elon-ism) of the project’s long-term goal. In the case of the vision-enhanced robot (VER), the goal is similar to the one for my 4-wheel robot, i.e. “Autonomous navigation around our home”. However, instead of ‘wall-following’ as the 4-wheel robot did, this project will utilize vision processing at the primary navigation technology.

Over the last few weeks, Grok Code and I have been working at the bottom of the systems architecture with things like getting OTA updates for Teensy firmware working, both via the PC -> Bluetooth -> HC-05 -> Teensy Serial channel and most recently, via the PC -> Wifi -> Pi5 -> Teensy Serial channel. After (mostly) getting the Wifi_OTA capability going, I decided it was time to stop and make sure we had a Git Repo structure consistent with the top-down view. After the requisite amount of fumbling around, we (me and Grok Code) came up with the following structure.

my_vision_robot/
├── .gitignore
├── README.md
├── docs/ ← Overall project documentation
│
├── hardware/ ← Schematics, BOMs, mechanical (future)
│
├── shared/ ← Code used by multiple projects
│ ├── firmware/ ← Shared Teensy code/libraries
│ └── software/ ← Shared Pi5 Python modules
│
├── software/
│ ├── SerialPassthroughDemo/
│ │ ├── README.md
│ │ ├── pi5/
│ │ ├── teensy/
│ │ └── docs/
│ │
│ ├── Wifi_OTA/
│ │ ├── Wifi_OTA.py
│ │ ├── README.md
│ │ ├── pi5/ (if needed later)
│ │ ├── teensy/
│ │ └── docs/
│ │
│ └── Vision_Navigation/ ← Future main vision project
│ ├── pi5/
│ ├── teensy/
│ └── docs/
│
├── tests/ ← System/integration tests
└── tools/ ← General one-off utilities

Now the challenge is to move all my sub-project files from our current quite-messy repo structure into this one. Standby!

07 July 2026 Update: Wrapping up the Wifi_OTA Demo project

This little project had more than its share of bumps and bruises, but I think Grok Code and I have finally got it done. The project has four major parts; three on my PC and one on the pi5.

  • The Wifi_OTA Visual Studio/Visual Micro project on my PC
  • A ‘board.txt’ file in the same folder as the Wifi_OTA project. This file tells VS what to do after the build step
  • A small Python script that copies the .HEX output from the compile to the ‘latest.hex’ file on the pi5
  • A Python program on the pi5 that watches for updates to ‘latest.hex’ and when one is detected, passes that along the the teensy on its Serial1 port

Here is the Wifi_OTA.ino file: All it does is blink the built-in LED a couple of times and then waits for a ‘U’ character to start the flash update process.

Here is the ‘board.txt’ file that calls the PostBuild.py program when an F5 debug compile is completed:

And here is the Python script that copies the .HEX output from the compiler over to ‘latest.hex’ on the pi5:

Here’s a short video showing the Wifi_OTA update process. The video starts just after I pressed F5 to start the debug compile on my PC. After about two seconds, the built-in LED on the Teensy 4.1 (lower-left foreground) goes OFF when the HEX file transfer starts. The file transfer takes about 30 sec, and then a few seconds after the file transfer finishes, the built-in LED on the Teensy 4.1 blinks twice and then stays ON, confirming that the update was successful.

A significant part of this little sub-project was getting the Git repository set up and running, on my PC, on the pi5, and on GitHub. The same folder structure is used in all three locations, but the pi5 side only updates entries in the ‘pi5’ subfolders, and the PC side only updates the Teensy firmware. When I do an update either on the PC or the pi5, I follow the same steps each time:

  • git pull origin main <<— synchs the local repo with the master on Github
  • git add -A <<– adds anything new to the local repo
  • git commit -m “put my update description here”
  • git push origin main <<– synchs the Github repo with the updated local repo.

    At this point I believe I have the basic infrastructure in place to proceed with the real project of adding vision-processing-based navigation capabilities to the robot
  • The OAKD-Lite camera is installed and confirmed working
  • The pi5 is installed and connected to the OAKD-Lite camera and via Serial1 to the Teensy4.1.
  • Teensy firmware can be updated via the new Wifi_OTA update channel
  • The Hall-effect wheel encoders are installed and confirmed working.
  • The next big step is to integrate the ROS (Robot Operating System) with the OAKD-Lite camera and the Teensy.

Stay Tuned!

2 Wheel Robot with Vision Processing, Part II

Leave a reply

This is the second of (hopefully) many posts on my project to add modern vision processing to my autonomous wall-following robot. The first post is here.

Lots of changes since my first post. I discovered that my 8.4-to-5V LDO regulator board wouldn’t reliably drive the Raspberry Pi5, so I had Grok look around for other options. He found a step-down converter module at Pololu advertising 85+% efficiency and a much cleaner output. I have it on order so we’ll see.

I also struggled to get the Luxonis OAK-D Lite stereo camera working on my Pi5/Ubuntu 24.04LTS setup. Previously we had gotten it to work with the Pi5 running the Raspberry OS, but getting it to work with the Ubuntu OS was a lot more challenging. This effort also ran afoul of Grok’s complete inability to realize that it is in a hole and to stop digging. We went through dozens of Python scripts designed to get the camera to connect and show some data (I was connecting to the RPi5 via SSH from my windows box, so showing images wasn’t possible), and they all failed due to one subtle problem or another. After several days of getting nowhere I finally called a halt, took a couple of days off, and came back determined to start over from first principals. Instead of using Grok, I started from scratch with some web searches to find other successful implementations of The OAK-D Lite camera. At the Luxonis ‘Documentation’ site I found this page describing a viewer for the OAK-D Lite (and other) cameras. The OAK Viewer is available for windows and *nix OS, so I decided to start by trying to get images from the camera using the Windows version to bypass all the annoyances associated with peripheral handling in Linux. This turned out to be pretty much plug-and-play, and immediately the OAK-D camera showed up in the list of available devices. When I clicked on ‘Connect’ instead of connecting immediately the app immediately started measuring the available bandwidth of the USB connection as shown below.

After several USB connect/disconnect cycles, I got this display:

Oops! I hadn’t even considered the USB cable/connector bandwidth issue – and neither had Grok. For convenience I had plugged the camera cable into my USB hub, which is definitely not ‘super speed’ whatever the heck that is. After some cable and connector switching, I found that a heavy-duty Type-C cable connected directly to a Type-C connector on my Dell XP15-9530 laptop allowed the bandwidth check to succeed, and now I got some images showing up on my Windows 11 display – yay!

OAK-D Lite images. Depth pseudo-color on left, raw RGB image on right

So the moral of this story is – the Grok path was never going to work because Grok never considered that cable/usb connector bandwidth might be an issue. By going back to ‘first principals’ and taking the simplest possible path to a working camera/display configuration with a known-good Windows 11 app, I was able to immediately identify a completely unknown (to me and to Grok) – but fatal – stumbling block – USB cable/connector bandwidth. Grok has no sense of time, so every iteration was just like a puppy chasing a dog – willing to chase that ball an infinite number of times without ever thinking about the fact that ‘chasing the ball’ and ‘progress toward the goal’ aren’t necessarily the same thing. It took a mere mortal like me to say “whoa – this isn’t getting us anywhere – maybe a different approach?”

Now that I had demonstrated that the OAK-D Lite camera and the proper cable/connector combination worked – at least on Windows 11, I had a ‘known-good baseline’ that I could always retreat to, I started working on getting the OAK-D Lite camera working on the RPi5/Ubuntu camera with the same OAK Viewer application (but in the Linux flavor).

This turned out to be another maze to navigate. The Luxonis site has detailed instructions for the Linux version of the viewer, but it involves installing from a *.deb package, which unfortunately is targeted at the amd64 64-bit chip ecology – but the RPi5 uses arm64 – a different animal entirely. When I tried to install the ‘viewer.deb’ package, I got the following errors:

The following packages have unmet dependencies: oak-viewer:amd64 : Depends: libgtk-3-0:amd64 but it is not installable Depends: libnotify4:amd64 but it is not installable Depends: libnss3:amd64 but it is not installable Depends: libatspi2.0-0:amd64 but it is not installable Depends: libdrm2:amd64 but it is not installable Depends: libgbm1:amd64 but it is not installable Depends: libxcb-dri3-0:amd64 but it is not installable Recommends: pulseaudio:amd64 or libasound2:amd64 but it is not installable

It was at this point that I re-engaged Grok and started to make real progress. Grok immediately identified the Pi5/Ubuntu-compatible DepthAI Python library as the way to go and guided me through the installation process. Fortunately, this had a happy ending, even though there were several ‘gotchas’ along the road. However, since I knew for a fact that the hardware (and USB cable) were ‘known good’ elements due to my Windows 11 work, I was pretty sure any detours were software-only. After working my way through the various twists and turns with Grok’s help, we got to here – success!

Initial images captured by the OAK-D Lite camera running on my RPi5 with the Linux Ubuntu OS

Getting from my easy Windows 11 camera demo to the RPi5/Ubuntu camera demo would have been improbable if not impossible for me to do without Grok’s help. I might have gotten there, but it would have involved days/weeks of web searches and forum posts at the very least. I believe this is where Grok really shines – a definite problem with a definite end, with very few (none in my case) outside corrupting factors like the USB bandwidth/cable issue.

Interestingly, after getting the Pi5/Ubuntu/OAK-D Lite combination working, I asked Grok to help me find an easier way to take screen shots on the Pi5, and Grok obliged by offering the ‘Flameshot’ app as a substitute for the built-in Gnome keystroke shortcuts. And then we went down another rabbit-hole, and I ended up wasting an hour or so trying to get Flameshot and Gnome to work and play well together, only to wind up removing Flameshot and learning how to better use the Gnome built-in shortcuts.

So Grok is definitely a mixed blessing, and I cannot imagine how a younger less-experienced engineer would do without the (literally) lifetime’s worth of experience I have in troubleshooting hardware/software systems. When I was that young less-experienced engineer half a century ago I was trying to troubleshoot a RF EMI problem with a small electronics device made by Motorola. Eventually my supervisor suggested that I travel to Motorola and work with their engineers to figure out the problem. I did, and over the space of two days a very experienced Motorola engineer taught me the ‘divide and conquer’ method of troubleshooting that I use to this day. When Grok inevitably goes down a rabbit-hole with this young engineer in tow, who’s going to be there to throw them a life-line?