Tag Archives: Arduino

Basketball Shot Make/Miss Recorder

Leave a reply

Posted 27 November 2020

After my recent right shoulder surgery, I have been looking for ways to get back into BBall shooting practice. However, the combination of Covid-19 lockdowns and the oncoming winter weather has drastically reduced my options for shooting practice. Not to be dissuaded from BBall practice, I figured out a way of moving my practice area from my driveway outside the garage inside the garage. As it turns out, our earth-sheltered house boasts an attached earth-sheltered two-car garage with a 12′ interior ceiling and no vertical support beams (I insisted on this in the design, and paid the extra cost, knowing from prior experience that interior support columns in a two-car garage was a bad thing.

Still not quite perfect, as even a 12′ ceiling isn’t quite enough room for a 10′ high BBall rim and enough headroom for practical shooting practice. So, I decided to mount my new practice rim at 9′ vs 10′, leaving 3′ of headroom. Obviously this isn’t ideal, but I figured a lot of practice on a 9′ rim was much better than no practice on a 10′ rim ;-).

As I started to shoot in my new practice gym, I realized I also now needed a way to record hits and misses so I could keep a running ‘made shots’ percentage. My short-term memory degradation due to being an old fart has made it almost impossible to do this in my head, and besides, what could be better than being able to combine my love of BBall, my love of programming, and my love for hardware/software projects? That’s at least a trifecta if not a quadfecta (and that is actually a word!). So, I programmed up a Windows program in C# to display ‘Hits’, ‘Attempts’, and ‘Hit Percentage’, and used an Arduino microcontroller to interface to two push-buttons – one for a hit, and the other for a miss. The arduino talks to the Windows program via a serial port over the USB cable, and the Windows program displays everything in a readily visible way. My idea was to use foot switches for the input so it would be easy to provided the needed hit/miss inputs without having to break out of a shooting rhythm. So, I jumped up on Thingiverse and found a neat, rugged footswitch by ‘vandenmar’. I needed two switches, so I modified vandenmar’s design by adding an additional cable routing trench out to the other side of the switch (so the switch cable can be routed in from the left or right-hand side of the switch).

My very own indoor basketball court (with a few minor restrictions…)
Nice, rugged footswitch design by vandenmar on Thingiverse
The 12 mm switch used here is available from Adafruit

Here’s a photo of the Arduino switch contoller

After getting all the pieces together, this is how it looks in my garage. The red footswitch is for ‘makes’ and the blue one is for ‘misses’.

red/blue footswitches, with cable to UNO and laptop in the background

This setup worked pretty well, but it still wasn’t quite ready for prime time. The UNO & breadboard were, well, breadboards, and not very mechanically robust. I was afraid I would spend more time repairing the breadboard than I would actually shooting, so I decided I needed to move from a breadboard to a more finished project. In addition, the Windows program I was using to display stats needed some work, so both sides (Windows and Arduino) needed some TLC.

I happened to have a Sparkfun Pro Micro module laying around in my parts bin, so I decided I would change out the UNO for the much smaller Pro Micro. With only two pushbutton switches and a Red/Green LED, the Pro Micro has plenty of I/O for the project. Here’s the Pro Micro version of the hardware layout.

So then I opened up my trusty Open SCAD ’rounded box with lid’ script and printed up a translucent enclosure, shown below

Sparkfun Pro Micro translucent enclosure. Photo above shows the ‘Make’ green LED illuminated

Next I updated the Windows program to show Makes, Misses, Percentage makes, and number of shots made in a row, along with the longest run of made shots for this session, as shown below

And here is the ‘final’ setup in my garage:

The two footswitches are shown at bottom left, with the Pro Micro enclosure hanging down from the laptop.

Here’s the schematic for the Sparkfun Pro Micro setup

And here is a short video showing the system in action.

At 0:43 into the above video you can see the green ‘Make’ LED illuminate in the background when I pressed the red ‘Make’ footswitch, and at 1:25 you can see me pressing the blue ‘Miss’ footswitch.

Stay tuned!

Frank

Left Side Wall Tracking Success With VL53L0X Array, Part II

3 Replies

Posted 10 October 2020

After the left-wall tracking success described previously in this post, I made some more adjustments and also set up a ‘ tracking sandbox’ in my lab to test Wall-E2’s ability to detect & respond to upcoming obstacles. Here’s a short video showing Wall-E2 in action

Tracking run demonstrating obstacle avoidance maneuvers

Here’s the raw output from the run:

And here is an Excel plot of just the movement sections of the above, highlighting the avoidance maneuvers.

left-side wall distances are shown in mm, while the front distance is shown in cm. Note 1-2 sec gaps during turns

Comparing the Excel plot to to the video, the front distance plot shows a monotonically decreasing value and then a large jump after each obstacle avoidance turn. It appears that the robot acquires and tracks the 30cm offset target successfully on the first wall, but doesn’t do as well on the second one. It was much more successful on the third wall. The plot for the last wall is only about 2 seconds long.

All in all, this looks like a pretty successful run for Wall-E2. It tracked three different walls (the fourth wall was too short to track) and successfully avoided obstacles three times – woo hoo!

12 October 2020 Update:

On the above ‘sandbox’ run, I noticed that at the end of the third leg at about 14 seconds into the run, the ‘spin turn’ at the white foam core wall wasn’t a ‘step turn’, but a ‘backup and turn’ triggered by the front distance going below the front obstacle limit of 20 cm, rather than the tracking obstacle clearance limit of 30 cm. Here are two output lines that illustrate the difference

and

In the video, these events are at about 7 & 14 seconds respectively. From this I came to the conclusion that at least the front distance wasn’t getting updated enough to keep the robot from getting too close to the obstacle before it realized there was a problem. At the time, the update rate for the system was set at 5Hz or 200 mSec. If the robot is travelling at 50 cm/sec, it means that it will travel 10 cm between distance updates – ouch!

So, I changed the timer interrupt timeout value for a 10Hz rate, and ran the ‘sandbox’ run again. This time when I looked at the output I could see that each leg terminated with something like

and it was clear that the updates were happening about every 100 mSec. Here’s the output:

and a short video:

And an Excel plot showing the left wall and forward distances progressing through the run.

Note that the front distance is shown in cm, while the left wall distances are shown in mm

At this point, I’m pretty happy with Wall-E2’s new-found wall tracking superpowers, at least for the left wall case. Now I need to port the V7 left-side-only code back into the main program and also port it to the right wall case.

Stay tuned!

Frank

Replacing HC-SRO4 Ultrasonic Sensors with VL53L0X Arrays

7 Replies

Posted 20 May 2020

In a recent post, I described the issue I had discovered with the HC-SR04 ultrasonic ‘Ping’ distance sensors – namely that they don’t provide reliable distance information for off-perpendicular orientations, making them unusable for determining when Wall-E2, my autonomous wall-following robot, is oriented parallel to the nearest wall.

In the same post, I described my discovery of the STMicroelectronics VL53L0X infrared laser ‘Time-of-Flight’ distance sensor, and the thought that this might be a replacement for the HC-SR04.  After running some basic experiments with one device, I became convinced that I should be able to use an array of three sensors oriented at 30 degree angles to each other to provide an accurate relative orientation to a nearby wall.

So, this post is intended to document my attempt to integrate two 3-sensor arrays into Wall-E2, starting with just the right side.  If the initial results look promising, then I’ll add them to the left side as well.

Physical layout:

The VL53L0X sensors can be found in several different packages.  The one I’m trying to use is the GY530/VL53L0X module available from multiple vendors.  Here’s a shot from eBay

GY530/VL53L0X test setup with Arduino. Note size relative to U.S. Dime coin

The basic idea is to mount three of the above sensors in an array oriented to cover about 60 degrees.  So, I designed and printed a bracket that would fit in the same place as the HC-SR04 ‘ping’ sensor, as shown below:

TinkerCad design for triple sensor mounting bracket

And here is the finished bracket with the center sensor installed (I haven’t received the others yet)

VL53L0X module mounted at the center position on triple-sensor bracket

System Integration:

The VL53L0X chip has a default I2C address of 0x29.  While this address can be changed in software to allow multiple V53L0X modules to be used, the chips don’t remember the last address used, and so must be reprogrammed at startup each time.  The procedure requires the use of the chip’s XSHUT input, which means that a digital I/O line must be dedicated to each of the six modules planned for this configuration, in addition to the power, ground and I2C SCL/SDA lines.  This doesn’t pose an insurmountable obstacle, as there are still plenty of unused I/O lines on the Arduino Mega 2560 controlling the robot, but since I want to mount the sensor arrays on the robot’s second deck in place of the existing HC-SR04 ‘ping’ sensors, those six additional wires will have to be added through the multiple-pin connector pair I installed some time ago. Again, not a deal-breaker, but a PITA all the same.

So, I started thinking about some alternate ideas for managing the sensor arrays.  I already have a separate micro-controller (a Teensy 3.5) managing the IR sensor array used for homing to charging stations, so I started thinking I could mount a second Teensy 3.2 on the second deck, and cut down on the requirement for intra-deck wiring.  Something like the following:

The Teensy would manage startup I2C address assignments for each of the six VL53L0X sensor modules via a secondary I2C bus, and would communicate distance and steering values to the Arduino Mega 2560 main controller via the system I2C bus.

22 May 2020 Update:

After some fumbling around and some quality time with the great folks on the Teensy forum, I managed to get a 3-sensor array working on a Teensy 3.5’s Wire1 I2C bus (SCL1/SDA1), using the Adafruit VL53L0X library.  The code as it stands capitalizes on the little-known fact that when a Teensy 3.x is the compile target, there are  three more ‘TwoWire’ objects availble – Wire1, Wire2, and Wire3.  So, I was able to use the following code to instantiate and configure three VL53L0X objects:

January 18 2022 Note:

The ‘magic’ that creates the ‘Wire1’, ‘Wire2’, and ‘Wire3’ objects occurs in WireKinetis.h/cpp in the Arduino\hardware\teensy\avr\libraries\Wire folder.

Here’s some sample output:

And the physical setup:

Triple VL53L0X LIDAR array. The two outside sensors are angled at 30 degrees. Teensy 3.5 in background

The next big questions are whether or not this 3-sensor array can be used to determine when Wall_E2 is oriented parallel to the nearest wall, and whether or not the steering value proposed in my previous post, namely

where the ‘l’, ‘r’ and ‘c’ subscripts denote the left/forward, right/rearward, and center sensor measured distances respectively.

23 May Update:

I’ve been improving my understanding of my triple-VL53L0X setup, and I think I’m to the point where I can try out the steering idea.  Here’s the experimental setup:

I have a wood barrier set up with a 40 cm offset from the center of the compass rose. Before getting started, I checked each sensor’s measured distance from the barrier by manually placing each sensor at the center of the compass rose, aligned as closely as possible with the perpendicular to the barrier.  When I did this, the measured offsets were within a few mm of the actual distance as measured by the tape measure.

The plan is to rotate the sensor bracket from -30 degrees to +30 degrees relative to the normal perpendicular orientation, while recording the left, center, and right measured distances.  Then I should be able to determine if the steering idea is going to work.  In this experiment, I manually rotated the array from 0 (i.e. the center sensor aligned with the perpendicular to the barrier) to -30 degrees, and then in 10 degree steps from -30 to 30 degrees.  The rotation was paused for a few seconds at each orientation.  As shown in the plot below, the Steering Value looks very symmetric, and surprisingly linear – yay!

Steering values resulting from manualy rotating the triple VL53L0X array from -30 to 30 degrees

However, the above experiment exposed a significant problem with my array design.  The 30 degree angular offset between the center sensor and the outer two sensors is too large; when the center sensor is oriented 30 degrees off perpendicular, the line-of-sight distance from the ‘off’ sensor to the wall is very near the range limit for the sensors, with the 40 cm nominal offset chosen for the test.  I could reduce the problem by reducing the initial offset, but in the intended application the initial offset might be more than 40 cm, not less.

So, back to TinkerCad for yet another bracket rev.  Isn’t having a 3D printer (or two, in my case) WONDERFUL!  The new version has a 20 deg  relative angle rather than 30, so when the center sensor is at 30 degrees relative to the wall, the outside one will be at 50 deg.  Hopefully this will still allow good steering while not going ‘off the wall’ literally.

Revised sensor carrier with 20-deg offsets vs 30-deg

04 June 2020 Update:

In order to make accurate measurements of the triple-VL53L0X array performance, I needed a way to accurately and consistently rotate the array with respect to a nearby wall, while recording the distance measurements from each array element.  A few years ago I had created a rotary table program to run a stepper motor for this purpose, but I hadn’t documented it very well, so I had to take some time away from this project to rebuild the tool I needed.  The result was a nice, well-documented (this time) rotary scan table controlled by a Teensy 3.2, and a companion measurement program.  The rotary scan table program can be synchronized with the measurement program via control pins, and the current step  # and relative pointing angle can be retrieved from the scan table via I2C.

Here’s the test setup:

Test setup for triple VL53L0X angle sweeps. Board in background extends about .5m left and right of center.

And here’s a short video showing one scan (-20 to +20 deg)

I ran two scans – one from -30 to +30 deg, and another from -20 to +20 deg.  Unfortunately, my test ‘wall’ isn’t quite long enough for the -30/+30 scan, and in addition the left-most sensor started picking up clutter from my PC in the -30 position. In any case, both scans showed a predictable ‘steering’ value transition from positive to negative at about +10 deg.

-30 to +30 deg scan. Note the steering value crosses zero at about +10 deg

-20 to +20 deg scan. Note the steering value crosses zero at about +10 deg

09 June 2020 Update:

After this first series of scans, I discovered that my scan setup was not producing consistent results, and so all the data taken to this point was suspect.  So, back to the drawing board.

Based on a comment on an earlier experiment made by john kvam of ST Microlectronics, regarding the 27 degree cone coverage of the photo beam from the LIDAR unit, I thought at least part of the problem was that the beam might be picking up the ‘floor’ (actually my desk surface in these first experiments).  As a way of illuminating (literally) the issue, I created a new stepper motor mount assembly with holes for mounting three laser diodes – one straight ahead, one tilted down at 14 degrees, and one tilted up at 14 degrees. The combination of all three allows me to visualize the extents of the VL53L0X beam coverage on the target surface, as shown in the following photos.

The second photo shows the situation with the ‘wall’ moved away until the beam extent indicator dots are just captured, with the distance measured to be about 220 mm.  The following short video shows how the beam coverage changes as the relative angle between the center sensor and the wall changes.

As shown in the video, the beam extents are fully intercepted by the 6″ wall only during the center portion of the scan, so the off-axis results start to get a little suspect after about 50 degrees off bore-sight.  However, for +/- 30 to 40 degrees, the beam extents are fully on the ‘wall’, so those measurements should be OK.  However, the actual measurements have a couple of serious problems, as follows:

  • As shown in the above Excel plots, the ‘steering’ value, calculated as (Front-Rear)/Center crosses the zero line well to the right of center, even though the sensors themselves are clearly symmetric with the ‘wall’ at 0 degrees
  • The scans aren’t repeatable; when I placed a pencil mark on the ‘wall’ at the center laser dot, ran a scan, and then looked at the laser dot placement after the ‘return to start position’ movement, the laser dot was well to the left of the pencil mark. After each scan, the start position kept moving left, farther and farther from the original start position.

So, I started over with the rotary table program; After some more research on the DRV8825, I became convinced that some or all of the problem was due to micro-stepping errors – or more accurately, to the user (me) not correctly adjusting the DRV8825 current-limiting potentiometer for proper micro-stepping operation.  According to Pololu’s description of DRV8825 operation, if the current limit isn’t set properly, then micro-stepping may not operate properly.  To investigate this more thoroughly, I revised my rotary table program to use full steps rather than micro-steps by setting the microstepping parameter to ‘1’.  Then I carefully set the scan parameters so the scan would traverse from -30 to +30 steps (not degrees). When I did this, the scan was completely repeatable, with the ‘return to starting position’ maneuver always returning to the same place, as shown in the following short video

The above experiment was conducted with the DRV8825 current limit set for about 1A (VREF = 0.5V).  According to information obtained from a Pololu support post on a related subject, I came to believe this current limit should be much lower – around 240-250 mA for proper microstepping operation, so I re-adjusted the current limiting pot for VREF = 0.126V.

After making this adjustment, I redid the full-step experiment and confirmed that I hadn’t screwed anything up with the current limit change – Yay!

Then I changed the micro-stepping parameter to 2, for ‘half-step’ operation, and re-ran the above experiment with the same parameters.  The ‘2’ setting for micro-stepping should enable  micro-stepping operation.  As shown in the following video, the scan performed flawlessly,  covering the -54 to 54 degree span using 60 micro-steps per scan step instead of the 30 previously, and returning precisely to the starting position – double Yay!

Next I tried a microstepping value of 8, with the same (positive) results.  Then I tried a stepping value of 32, the value I started with originally.  This also worked fine.

So, at this point I’m convinced that microstepping is working fine, with the current limit set to about 240 mA as noted above.  This seems to fully address the second bullet above, but as shown in the plot below taken with microstepping = 32 and with data shown from -30 to +54 degrees), I still have a problem with the ‘steering’ value not being synchronized with the actual pointing angle of the array sensor.

Next I manually acquired sensor data from -30 to +30 degrees by rotating the de-energized stepper motor shaft by hand and recording the data from all three sensors.  After recording them in Excel and plotting the result, I got the following chart.

This looks pretty good, except for two potentially serious problems:

  1. The ‘steering’ value, defined as (Front Distance – Rear Distance)/Center Distance, is again skewed to the right of center, this time about 8 degrees.  Not a killer, but definitely unwanted.
  2. Rotation beyond 30 degrees left of center is not possible without getting nonsense data from the Front (left-hand) sensor, but I can rotate 60 degrees to the right while still getting good data, and this is with much more ‘wall’ available to the left than to the right, as shown in the following photo

-30 deg relative to center

+30 deg relative to center

-60 deg relative to center

+60 deg relative to center

After finishing this, I received a reply to a question I had asked on the Pololu support forum about micro-stepping and current limiting with the Pololu DRV8825.  The Pololu guy recommended that the current limit be set to the coil current rating (350 mA for the Adafruit NEMA 17 stepper motor I’m using), or 0.175V on VREF.  So, I changed VREF from 0.122V to 0.175V and re-verified proper micro-stepping performance.  Here’s a plot using the new setup, micro-stepping set to 32, -54 to +54 deg sweep, 18 steps of 6 deg each.

and a short video showing the sweep action.

In the video, note that at the end, the red wire compass heading pointer and the laser dots return precisely to their starting points – yay!

So, at this point everything is working nicely, except I still can’t figure out why the steering value zero doesn’t occur when the sensor array is oriented parallel to the ‘wall’.  I’m hoping John, the ST Micro guy, can shed some light (pun intended) on this 😉

11 June 2020 Update.

I think I figured out why the steering value zero didn’t occur when the sensor array was oriented parallel to the wall, and it was, as is usually the case, a failure of the gray matter between my ears ;-).

The problem was the way I was collecting the data with the scanner program that runs the rotary table program.  The scanner program wasn’t properly synchronizing the sensor measurements with the rotary table pointing angle.  Once I corrected this software error, I got the following plot (the test setup for this plot still has the left & right sensors physically reversed).

As can be seen in the above plot, the steering value y-axis crossover now occurs very close to zero degrees, where the sensor array is oriented parallel to the wall.

12 June 2020 Update

After numerous additional steering value vs angle scans, I’m reaching the conclusion that the VL53L0X sensors have the same sort of weakness as the ‘ping’ sensors – they aren’t particularly accurate off-perpendicular.  To verify this, I removed the sensors from my 3-sensor array bracket and very carefully measured the distance from each sensor position to the target ‘wall’ using a tape measure and a right-angle draftsman’s triangle, with the results shown in the following Excel plot.  Measuring the off-perpendicular distances turned out to be surprisingly difficult due to the very small baseline presented by the individual sensor mounting surfaces and the width of the tape measure tape.  I sure wish I had a better way to make these measurements – oh, wait – that’s what I was trying to do with the VL53L0X sensors! ;-).

With just the physical measurements, the steering value crosses the y-axis very close to zero degrees relative to parallel, plus/minus measurement error as expected.  One would expect even better accuracy when using a sensor that can measure distances with milometer accuracy, but that doesn’t seem to be the case.  In the following Excel plot, the above tape measure values are compared to an automatic scan using the VL53L0X sensors. As can be seen, there are significant differences between what the sensors report and the actual distance, and these errors aren’t constant with angular orientation (otherwise they could be compensated out).

For example, in the above plot the difference between the solid green (rear sensor distance) and dashed green (rear sensor mounting surface tape measure distance) is about 80 mm at -30 degrees, but only about 30 mm at +30 degrees.  The difference between the measurements for the blue (front) sensor and the tape measure numbers is even more dramatic.

So it is clear that my idea of orienting the sensors at angles to each other is fundamentally flawed, as this arrangement exacerbates the relative angle between the ‘outside’ sensor orientations and the target wall when the robot isn’t oriented parallel to the wall.  For instance, with the robot oriented at 30 degrees to the wall, one of the sensors will have an orientation of at least 50 degrees relative to the wall.

So my next idea is to try a three sensor array again, but this time they will all be oriented at the same angle, as shown below:

The idea here is that this will reduce the maximum relative angle between any sensor and the target wall to just the robot’s orientation.

3-sensor linear array

I attached the three VL53L0X sensors to the linear array mount and ran an automatic scan from -30 to +30 degrees, with much better results than with the angled-off array, as shown in the Excel plot below:

VL53L0X sensors attached to the linear array mount. Note the nomeclature change

Triple sensor linear array automatic scan. Left/Right curve are very symmetric

So, the linear array performance is much better than the previous ‘angled-off’ arrangement, probably because the off-perpendicular ‘look’ angle of the outside sensors is now never more than the scan angle; at -30 and +30 degrees, the look angle is still only -30 and +30 degrees.  Its clear that keeping the off-perpendicular angle as low as possible provides significantly greater accuracy.  As an aside, the measured perpendicular distance from the sensor surface to the target ‘wall’ was almost exactly 250 mm, and the scan values at 0 degrees were 275, 238, and 266 mm for the left, center, and right sensors respectively.   so the absolute accuracy isn’t great, but I suspect most of that error can be calibrated out.

13 June 2020 Update:

So, now that I have a decently-performing VL53L0X sensor array, the next step is to verify that I can actually use it to orient the robot parallel to the nearest wall, and to track the wall at a specified offset using a PID engine.

My plan is to create a short Arduino/Teensy program that will combine the automated scan program and the motor driver program to drive the stepper motor to maintain the steering value at zero, using a PID engine.  On the actual robot, this algorithm will be used to track the wall at a desired offset.  For my test program, I plan to skew the ‘wall’ with respect to the stepper motor and verify that the array orientation changes to maintain a wall-parallel orientation.

14 June 2020 Update:

I got the tracking program running last night, and was able to demonstrate effective ‘parallel tracking’ with my new triple VL53L0X linear sensor array, as shown in the following video

If anyone is interested in the tracking code, here it is:  It’s pretty rough and has lots of bugs, but it shows the method.

At this point, I’m pretty confident I can use the linear array arrangement and a PID engine to track a nearby wall at a specified distance, so the next step will be to replace the ‘ping’ ultrasonic sonar sensor on Wall-E2, my autonomous wall-following robot, with the 3-sensor array, and see if I can successfully integrate it into the ‘real world’ environment.

Stay tuned!

Frank

I2C between an Arduino Mega and a Teensy 3.x

3 Replies

posted 18 March 2020

This post describes my efforts to troubleshoot an I2C communications problem between an Arduino Mega control board and a Teensy 3.2 IR beam demodulation module.

Background:

My Wall-E2 autonomous wall-following robot homes in on a modulated IR beacon to connect to a charging station. The IR beacon modulation is decoded by a dedicated Teensy 3.2 and provides left/right and combined steering values to an Arduino Mega main controller over an I2C link.

During my recent work to update the robot after some enhancements, I discovered that the main controller was no longer receiving steering information from the Teensy, even though both seemed to be operating properly.  Initial efforts to troubleshoot the problem did not bear fruit, so I was forced to back up and start over from scratch.

Troubleshooting:

Initially I thought the problem was a loose connection, as the system was working before.  However, I am now pretty sure that I have eliminated all the obvious culprits, so I am left with the non-obvious ones.

To start with, I resurrected an old example project to demonstrate I2C master/slave operation between two Teensy modules.

Teensy 3.2 Slave (left) and Teensy 3.5 Master (right)

Here’s the Teensy ‘Master’ code:

And the Teensy ‘Slave’ code:

With this configuration I got the following output:

Master:

Slave:

So this seems to be working OK.

Then I added in a third Teensy module (T3.5) running my newly developed I2C-Sniffer code, so that I could ‘sniff’ the I2C traffic between the two Teensy’s.  This will give me a ‘known-good’ baseline for when I move back to the non-working Arduino Mega Master and  Teensy Slave condition that is causing me problems.

Teensy 3.2 Slave (left), Teensy 3.5 I2C Sniffer (middle), Teensy 3.5 Master (right)

Here’s the Teensy ‘Sniffer’ code:

With this setup with the master sending data to the slave, I got the following outputs:

Master

Slave:

I2C Sniffer:

In the other direction, with the slave sending data to the master, I got the following:

Slave:

Master:

I2C Sniffer:

Where the HEX sequence 4B D8 A9 AD converted to a IEEE float = 2.83984e+07

So, the Teensy-to-Teensy I2C connection is clearly working in both directions, and the Teensy I2C Sniffer is successfully capturing and decoding the I2C traffic between the two modules – cool!

So, the next step is to replace the ‘Master’ Teensy with an Arduino Mega and repeat the process.  Hopefully this will allow me to figure out why the slave-to-master data transfer isn’t working

21 March 2020 Update:

Based on the above results, I formed a hypothesis that the problem with sending I2C data from a Teensy slave to an Arduino Mega master might be due to the Mega not being able to properly interpret 0-3.3V transitions from the Teensy.  So, I decided to construct a low-to-high level converter to make sure that SDA data from the Teensy was presented to the Mega as 0-5V transitions rather than the Teensy’s ‘raw’ 0-3.3V ones.  To do this I used the circuit shown below, except with a 1K instead of 10K resistor for R2 to clean up the transitions at 100KHz.

Bidirectional 3.3-5V level shifter using 2N7000 MosFet

Here’s a scope photo of the 3.3V input to and the 5V output from the 2N7000-based level shifter. The top trace is the Teensy 0-3.3V output, and the bottom trace is the 0-5V level-shifted version.  Both traces are 1V/cm, and the horizontal time scale is 20 uSec/cm.

Teensy slave, Mega master: Top: 0-3.3V input. Bottom: 0-5V output. Both traces are 1V/cm vertical, 20 uSec cm horizontally

To verify that the above circuit worked properly, I used it in the SDA line (the SCL line doesn’t require any level shifting as it is from the 5V Mega to the 3.3V Teensy) with a Teensy slave and a Teensy master.  This is OK, as Teensy 3.x’s can handle 5V inputs, and it worked properly in both directions.  Here’s a shot of the SDA line

Teensy slave, Teensy master: Top: 0-3.3V input. Bottom: 0-5V output. Both traces are 1V/cm vertical, 20 uSec cm horizontally

However, after replacing the Teensy master with the Mega one, I had the same problem as before – I can successfully transfer data from Mega master to Teensy slave, but not in the other direction. There is still a problem somewhere, and I no longer think it’s due to level-shifting problems.  From the above scope photos, it is clear that only a few bytes are transmitted by the Teensy slave each time it services the OnRequest() interrupt, while that same function transmits MUCH more data when servicing the same request from the Teensy 3.5 master.

Stay tuned!

23 March 2020 Update:

OK, back to the basics:  I downloaded the code for a very basic Arduino-Arduino I2C tutorial, and verified that it worked OK.  I also took a scope shot of SDA/SCL activity during the two-way data transfers, as shown below:

Basic Arduino-Arduino tutorial layout

I2C line activity for the ‘Basic Arduino – Arduino I2C Tutorial’

So now that I have a working baseline for the Arduino-Arduino I2C case, I plan to incrementally modify it toward duplicating the non-working Arduino-Arduino I2C case until it breaks, and then I’ll know that the last incremental modification is the culprit. At the moment, I’m leaning toward the use of the I2CDev & SBWIRE libraries as they are the only significant difference between the working and non-working setups.  We’ll see….

24 March 2020 Update:

I created a new Arduino project called ‘I2C_Master_Tut_Mod1’ initially identical to ‘I2C_Master_Tutorial’ and verified that it worked properly with the unmodified ‘I2C_Slave_Tutorial’ project, and then started modifying it.

  • Added #include <PrintEx.h> //allows printf-style printout syntax
    StreamEx mySerial = Serial; //added 03/18/18 for printf-style printing, and changed all occurrences of ‘Serial’ to ‘mySerial’ and modified print statements to ‘printf’ format. All worked fine.
  • Added #include <I2C_Anything.h> and changed loop() to write a float value to the slave.  However, this caused a compile error.  When I started tracking it down I realized that I had previously modified ‘I2C_Anything.h’ to #include SBWIRE.h rather than the original #include Wire.h.  So, this could be the culprit, assuming that there is something about SBWIRE that causes Arduino I2C slaves to misbehave.
  • Copied I2C_Anything.h/cpp from the Libraries folder to my new ‘I2C_Master_Tut_Mod1’ folder, and un-modified it to reference back Wire.h vs SBWIRE.h.

25 March 2020 Update:

As it turns out, I was never able to get the ‘I2C_Master_Tut_Mod1’ project working with #include <I2C_Anything.h> , no matter what I did, including copying the #include <I2C_Anything.h> to the local folder and including it as a resource int the VS2019 project, build cleans, removing the _vm folder created by Visual Micro, etc.  To make it even stranger, I created a new Arduino project in VS2019 called ‘I2C_Master_Tut_Mod2’, copied the ‘I2C_Anything.h’ file to the local folder, and it compiles without error!  So now I have two almost completely identical Arduino projects, one of which compiles fine and the other of which blows a bunch of errors about twi.c.  I’m currently working with Tim Leek at VisualMicro to sort out what I did wrong.

27 March 2020 Update:

Ok, I now have the Arduino Mega master to Arduino Uno slave setup working, with I2C_Anything.  I can transfer a float value in either direction with no problem. The master and slave code sets and the corresponding outputs are shown below.

Master Sketch:

Slave Sketch:

Master  & Slave Output:

So now I have a working Arduino-Arduino I2C master/slave setup, using I2C_Anything.h to transfer float values back and forth, and a working Teensy-Teensy I2C master/slave setup, using I2C_Anything.h to transfer float values back and forth. 

The next step was to replace the Arduino Uno I2C slave with the Teensy 3.2 I2C slave and see if I can get that combination working.  This turned out to be surprisingly easy – basically plug-and-pray (oops, I meant ‘play’).

Now that I have a simple Arduiono Mega Master – Teensy 3.2 Slave setup working, I can start to explore why my 4WD robot setup doesn’t work.  Some possibilities:

  • It could be that the SBWire version of the Wire library has some hidden bugs in the slave-related code.  The SBWire library eliminates the well-known and well-hated ‘hang’ bug in the Arduino wire library.
  • It could be that some other library I am using, like the RTC library or the Adafruit FRAM library is interfering with the Arduino-Teensy interrupts needed for master-slave communication.
  • It could be that the IR demod program running on the slave Teensy is somehow interfering with master-slave communications (this is very unlikely as this same program had been working flawlessly for several months).
  • Something else…

To start with, I copied the current ‘I2C_Master_Tut2’ VS2019 project to a new one – ‘I2C_Master_Tut3’ before making any modifications.  I plan to keep a ‘breadcrumb trail’ of incrementally modified projects so that I can go backwards in case I get lost at some point (and, in my fairly long 50+ years as a practicing engineer, I have learned that ‘getting lost’ is part and parcel of any non-trivial troubleshooting project).

Once I verified that ‘Tut3’ properly communicated with the Teensy slave, I started making modifications.

  • Replaced #include <Wire.h> with #include SBWire.h.  This failed in compile with a linker error, but succeeded after I did a ‘Build -> Clean’.  After uploading the new ‘Tut3’ version to the Mega, I found that master-slave communications still worked properly.  This is actually a welcome development, as that now eliminates SBWire as the culprit for lost Mega-Teensy I2C capability on the robot.
  • Replaced ‘#include “SBWire” with #include “MPU6050_6Axis_MotionApps_V6_12.h” and include “I2Cdev.h” (this includes SBWire.h).  These two libraries are required for interfacing to the MPU6050 IMU module.

29 March 2020 Update:

So I created a new copy of the I2C master, ‘I2C_Master_Tut4’ and started adding things from the original non-working robot program.

  • Added all the remaining #include’s.  Still works fine with the Teensy 3.2 slave
  • Added all the #defines. Still works fine
  • Added all the IRHOMING parameters.  OK
  • Added all ENUMs, Battery constants, distance measurement support constants & array declarations,  motor parameters, wheel direction constants and variables, and heading based turn parameters.  No problem
  • Added pin assignments.  No problem.
  • Added all the other pre-setup stuff.  No problem
  • Added ‘ Serial1.begin(9600); //03/04/16 bugfix’. This isn’t actually used anymore in the robot project, but it is there, so it could be the culprit.  Nope –  master/slave comms still OK.
  • Added RTC initialization and  support function.  Still no problem
  • Added FRAM initialization.  No problem.
  • Added MPU6050 initialization.  No problem
  • Added PID distance array and incremental variance initialization.  No problem
  • Added I/O pin initialization.  No problem
  • Added the rest of Setup(), and all the support functions required for the POST check to run.  Compiles and runs fine.  Of course, no peripherals are attached, so not much happens.

At this point I have all the pre-setup and setup code incorporated into the master/slave example, and everything is still happily plugging away.  So obviously the culprit hasn’t yet been identified.  However, before going any farther, I think I’ll drop ‘Tut4’ into the safe-deposit box and create a ‘Tut5’ to continue on into the loop() function.

OK, so I have a ‘Tut5’ project now, with everything up to loop() incorporated from the ‘FourWD_WallE2_V2’ robot project.  Now the question is, ‘What next?’.  I need to figure out what is causing I2C comms between the Mega master and the Teensy 3.2 slave to fail, so I need a way to faithfully replicate/simulate/emulate the actual robot code for this function.

The current robot algorithm as pertaining to the Teensy IR Demod module

  • Each time through the loop, the current operating mode is determined.
    • If IsIRBeamAvail() returns TRUE, the IR HOMING mode is activated
      • IsIRBeamAvail() gets three float values from the Teensy 3.2, and returns TRUE if the total of the first two values (Fin1 & Fin2) is above a set threshold.  It is this function that is failing to communicate with the Teensy.  More specifically, it is this function that is not successfully acquiring the three float values, and subsequently always returns FALSE.

So, it may be that all I have to do is to call IsIRBeamAvail() by itself, and modify the slave code to send back three floats as expected.  If this works, then I’ll have to start suspecting the robot’s Teensy or the I2C wiring, or something else entirely.

2 April 2020 Update:

After some additional fumbling around with I2C_Anything and the PrintEx library printf() formats, I now have a working ‘IsIRBeamAvail()’ function in ‘Tut5’, as follows:

Which, when connected to my basic  Teensy 3.2 I2C slave program produces the following (correct) output:

At this point we have a working robot code emulation that communicates successfully with a basic Teensy 3.2 slave.  So, the available culprits have been reduced significantly to

  • Something about the Teensy IR demod code
  • The I2C Wiring
  • A hardware problem at either the robot’s Mega controller or the robot’s Teensy 3.2 controller
  • Something else.

For the next step, I modified the Teensy IR Demod code to emulate the IR detector response and loaded this code into the Teensy 3.2 slave connected to the Mega master.  After some initial mis-steps, it started working nicely, with the following (correct) output from the Mega master:

So this eliminates the ‘Something about the Teensy IR demod code’ possibility from the above list.

Next, I replaced the direct SCL/SDA jumpers with the daisy-chain wiring from the robot, and miracle of miracles, I2C comms failed – YAY!!.  Now hopefully I can figure out (and fix) the problem.

  • First, I un-replaced the daisy-chain wiring to confirm that I2C comms were still working, and they were.
  • Unplugged the daisy-chain from the FRAM, 6050IMU, and RTC modules – now working OK
  • Plugged back in to FRAM unit (now have FRAM and Teensy 3.2 in chain)- – still OK
  • Plugged back in to 6050 IMU (now have IMU, FRAM, and Teensy 3.2 in chain)- failed
  • Plugged back in to RTC (now have RTC, FRAM, and Teensy 3.2 in chain)- failed

3 April 2020 Update:

The next step in the saga was to load the ‘Tut5’ code onto the robot’s Mega 2560 controller, with the slave code still running on the original (off-robot) Teensy 3.2.  Setting things up in this fashion allowed the sensors to receive power from the robot’s Mega controller as in normal operation.

In the photo above, the Teensy 3.2 slave is shown to the left of the robot.  The I2C ‘daisy-chain’ cable starts at the Mega controller (just to the right of the front left wheel), and goes through three ‘hops’ to the Teensy. With this setup, the I2C comms code still ran fine, with direct I2C jumper wires or with the daisy-chain wiring setup (with no connections to the other I2C sensors) as shown above.

The next step was to connect the robot’s Mega controller to the robot’s Teensy 3.2 running the unmodified IR Demodulation code, with the I2C daisy-chain cable, but without anything else connected.  This actually worked!  So now I have a working robot system again, so something about the connections with the other sensors must be killing the I2C link to the Teensy.  Here’s a scope photo of the SDA line showing the data activity. The vertical scale is 1V/cm, showing the HIGH value is about 3.8V, so the Mega must be comfortable with that value as a HIGH logic level

In the above photo, notice the business card for Probe Master, Inc (www.probemaster.com).  These folks were kind enough to replace my ten year old scope probe with a new upgraded version at no cost – thank you Probe Master!

At this point I have the ‘Tut5’ program running on the Mega, and the unmodified IR Demod code running on the robot’s Teensy 3.2, and the Mega appears to be acquiring valid demod data from the Teensy.  To further validate the data, I fired up my charging station with it’s square-wave modulated IR beam, and was pleased to see that the acquired data from the Teensy varied appropriately as I manually rotated the robot, as shown in the following Excel chart.

So, after all this work, the whole thing came down to a bad I2C ‘daisy-chain’ cable. This particular cable has been on the robot for a while, and was the soldering graduation exercise of my grandson Danny  when he was here a couple of summers ago.  It wasn’t the best job I’ve ever seen, but it was pretty good for a 15 year old ;-).  In any case, I took the opportunity to build a new cable with smaller diameter wire so things would fit a little bit better into the Pololu pins, sockets, and 2-pin header sleeves as shown in the following photo.

New I2C ‘daisy-chain’ cable.  Run starts from the Mega connector at bottom left, then to the RTC, IMU and FRAM modules in that order, then last to the Teensy 3.2 IR Demod module.

05 April 2020 Epilogue:

Well, there was one last ‘gotcha’ in all this.  When I loaded the original ‘FourWD-WallE2_V2.ino’ program back onto the Mega, it still refused to acquire valid data from the Teensy IR Demod module.  So, I compared the ‘Tut5’ code to the ‘_V2’ code, and noticed three significant differences:

  • In the ‘_V2’ code, the ‘Fin1/2’ variables had at some time been changed from ‘long’ to ‘int’. While this sounds reasonable, it isn’t – because a Teensy ‘int’ is 4 bytes – the same as a Mega ‘long’.
  • In the ‘_V2’ code, the  Wire.requestFrom() call asks for 32 bytes, but the Teensy only sends 12.
  • In the ‘_V2’ code there is a loop that waits for either a timeout or receiving an entire 32-byte buffer from the Teensy.  Apparently the Teensy (and/or the Mega) is slow enough so that Wire.Available() never reports a non-zero buffer size, so the loop times out.

Replacing the line

with the line

did the trick, but I have no idea why.

Replacing the debug printout lines

with the line

Produced lines like this one from the ‘_V2’ program

Which appear to be the correct values for the given robot orientation with respect to the charging station.

With no other changed except removing power from the charging station, the output became

showing definitively that the ‘_V2’ program correctly identified and decoded the square-wave modulated IR beam.

So, after a two-week off-road trip to I2C comms hell and back, I think I finally have that particular issue put to bed.  It wasn’t exactly what I wanted to do, but it was at least interesting, and more important – consumed a lot of coronavirus quarantine time ;-).

Stay tuned!

Frank

Arduino SPI Data Exchange Between Two Arduinos in a Master/Slave Configuration

2 Replies

posted 13 March 2020,

While updating my four wheel autonomous wall-following robot (aka Wall-E2), I ran into a roadblock when I couldn’t get my Teensy 3.2 IR demodulator/tracker module to communicate with the Mega 2560 main microcontroller over I2C.  This worked fine when I last tested it, but now it seems to have taken a vacation.  It’s been a while, and I have added some functionality since I originally installed and tested IR homing to the charging station, but it still should all work, right?

In any case, after trying (and failing) to get the I2C bus connection working, I thought I might try an alternate solution and just use SPI between the main controller (Mega 2560) and the IR homing controller (Teensy 3.2).  This would have the advantage of making the Teensy independent of the I2C bus, and also give me the chance to play with a part of the Arduino ecosystem that I haven’t used before.

As usual, I started this process with a lot of Googling, and quickly ran across Australian Nick Gammon’s  “SPI – Serial Peripheral Interface – for Arduino“  .  This post was way more than I ever wanted to know about SPI, but it sure is complete!

Then I moved on to trying to get SPI working for myself.  As noted above, my intended application is to transfer steering values from my IR detector/demodulator/homing module to the main Mega 2560 microcontroller. The Mega uses the steering data to adjust wheel speeds to home in on a charging station, so Wall-E2 can continue to roam our house autonomously.

It turns out that SPI between two Arduinos isn’t entirely straightforward, at least not at first.  I went through a bunch of iterations, and even more passes through the available documentation.  In the end though, I think I wrassled it into a reasonable facsimile of a working solution, as shown below.  The program repeatedly transfers a fixed string (“Hello World!”) from the UNO (master) to the Mega 2560 (slave), and then transfers the string version of a float value (3.159) from the slave to the master.

The Circuit:

The circuit and layout is just about as basic as it gets.  The UNO (master) connections are the default pinouts:

UNO (master) connected to Mega 2560 (slave) using default SPI pins on both ends

 

The Master:

The Slave:

The above configuration produced the following output:

Master:

 

Slave:

Now that I’ve had my fun and games with SPI, I’m still not at all sure I want to use it to connect the Mega to the Teensy IR Demodulator/Homing module.  While it is almost certain to work (eventually), it has some drawbacks

  • It requires 4 additional wires
  • It requires that I add an ISR and other code to the Teensy module, and companion code to the Mega
  • More things to go wrong.

So, I think I’ll try again to get the original I2C based code working, as it worked before, so it should work again.  If I can’t get it working after another good try, I’ll go with door #2 (SPI)

Stay tuned!

Frank

 

Wall-E2 Motor Controller Study Part II

3 Replies

Posted 07 December 2019 (Pearl Harbor Day)

Back in May of this year I posted about different motor driver possibilities for my 2-motor and 4-motor robots, and showed some results for two drivers (an Adafruit Featherwing and an Adafruit DRV8871).  However, I got kind of sidetracked after this post when I discovered the RFI/EMI problem with the MPU6050 IMU.  At the time, I blamed the RFI/EMI problem on the non-linear nature of the newer drivers, and went back to using the L298N linear driver.

After quite a bit of experimentation and work, it finally turned out that most (if not all) the RFI/EMI problem was the Pololu 20D metal-geared motors themselves, and properly suppressing the noise at the motor terminals with bypass capacitors solved the problem.  So I have decided to repeat some of my initial motor driver testing.

Adafruit DRV8871 Single Channel Motor Driver Testing:

I removed the L298N driver I have been using up until now on my 2-motor robot, and replaced it with two DRV8871 Single Channel Motor Drivers. The hookup with the DRV8871’s is actually significantly simpler than the L298N, requiring only two control lines per channel instead of three.  After the normal number of errors, I got it running and started some long-term tests with the motors running continuously (with varying speeds and directions) while polling the MPU6050 every 200 mSec for yaw data.

This test ran for over 5 hours without problems, and then the GY-521 (generic MPU6050) module stopped responding to requests for data.  This is the second MPU6050 module that  has behaved this way – running for an indefinite amount of time and then refusing to respond.

The good news is, the DRV8871 motor drivers worked flawlessly the entire time, and are efficient enough so the T0-3 size chip container was just barely warm to the touch.

I have run  several long-term tests now with the DRV8871 drivers and two different MPU6050 modules, and the longest error-free run has been about 5 hours as shown above.  However, I have also had the test terminate in less than 5 minutes, so this is clearly not reliable enough for prime time. Also, adding my 2-stage power supply filter back into the system did not seem to effectively suppress errors, so no I have no clue what is going on.

11 December 2019 Update:

I ran the system overnight with the same test program, except I commented out the motor run commands so the motors themselves did not run. The test ran for over 11 hours and was still running fine when I terminated it.  So, the motors definitely have to be running for the problem to occur.

15 December 2019 Update:

Hoping to maybe eliminate a variable, I changed from the UNO controller to a Teensy 3.2. As usual, this ‘small change’ took a LOT more time than I thought it would.

There aren’t any good example programs for interfacing a Teensy 3.x with the MPU6050, especially using the I2CDEV/MPU6050/DMP libraries. I had a huge problem trying to track down compile issues with the I2CDev libraries, but in the end it came down to figuring out a way to get I2CDev to use i2c_t3.h instead of Wire.h. I solved this by copying I2CDev.h/cpp from my Libraries folder to my local project/solution folder, and editing the code to define the I2C implementation as I2CDEV_ARDUINO_WIRE and then replacing “#include <Wire.h>” with “#include <i2c_t3.h> in the appropriate section as shown below. I’m sure there are more elegant ways of doing this (maybe adding a ‘I2CDEV_TEENSY_3.X’ section at the top?)

After making the above change, the project started compiling OK, and I was able to connect to the MPU6050 and pull off yaw values using the DMP.

The next problem occurred when I tried to run a test program with the Adafruit DRV8871 drivers.  The two control lines alternate between being used as digital outputs (for direction control) and analog/PWM outputs (for speed control). Unfortunately, once a Teensy 3.2 line has been written to with ‘analogWrite()’, it can no longer be used as a digital output without first running ‘pinMode([pin],OUTPUT)’ on it. This particular little ‘gotcha’ appears to have been there since around 2016, but got lost in the shuffle as it is still there in the latest TeensyDuino libraries.

After fixing that problem, I was successful in both getting yaw values from the MPU6050 using Jeff’s I2CDev libraries, and in driving my Pololu D20 metal-geared motors via the Adafruit DRV8871 motor driver modules.

After getting everything working, I took the time to fork Jeff Rowberg’s I2CDev library, edit I2CDev.h/cpp appropriately and create a pull request for Jeff.  The changes were merged into the Github master distro pretty quickly, so in theory at least, all a new Teensy 3.2 user has to do is grab Jeff’s I2CDevLib stuff and go.

Here’s my Teensy 3.2 program for testing the MPU6050/DMP and the Adafruit DRV8871 motor drivers:

IMU Motor Noise Troubleshooting, Part II

1 Reply

Posted 13 November 2019,

After a month-long diversion to help Homer Creutz investigate some of the many issues associated with the Invensense MPU6050 IMU chip, I’m now back to figuring out how to keep my robots’ motor & driver noise from corrupting the yaw values from my MPU6050 IMU module.  In my last post on this issue, I worked through a number of theories for what might be causing the problem, and eventually decided the issue was more associated with the motor drivers than with the motors themselves.  At the conclusion of this prior study, I was able to demonstrate effective turn control using MPU6050 yaw values, even with the motors running.

However, I have since determined that I basically had gotten lucky; the problem still persists, even after changing back from switch-mode to linear motor drivers.  After a lot more work, I am now convinced that the basic problem is very high frequency voltage spikes on the order of 2-3 V p-p being conducted onto the Mega 2560 microcontroller board via the motor control connections from the Mega to the motor driver.  The result is intermittent and erratic behavior from the Mega and/or the MPU6050 module.

As a potential solution to the problem, I spent some time investigating whether or not I could use a ESP32 module to replace the Mega and it’s accompanying wireless module  (HC-05 BT module on the 2-motor robot, and a Wixel wireless serial extender on the 4-motor model). The idea was that since the ESP32 module is much smaller and has an integrated/shielded wireless module, it might be more immune to the conducted noise issue.  As it turned out though, getting the ESP32 to work the way I needed it to was next to impossible, and so I abandoned ship after a few weeks.

Anyway, now I’m back to working the motor noise problem again.  When I left the problem the last time, I thought that one possible solution to the noise conduction problem would be to use an optical isolator module such as this one, to isolate the motor power circuits from the Mega circuits.

So, I set up an 6-channel optical isolation bank between the Mega 2560 controller and the 2-motor robot’s motor controller, as shown below:

6-channel opto-isolator setup with 2-motor robot and Mega 2560 controller

However, when I tried this trick, I still got lots of high-frequency transients on the control and power lines, and the computed yaw values from the MPU6050 soon became invalid.  After poking around a bit with a scope, I realized that while this cheap 4-channel optical isolator is great for voltage difference isolation and low frequency isolation, the high-frequency stuff I was seeing was going around the optical isolation and capacitively coupling from the output side back to the controller side – bummer!

After getting back on the project, the first thing I did was to refresh my memory on a prior project to test a Sparkfun MPU9250 (basically the same as the MPU6050 but with a magnetometer included) interfaced to a Teensy 3.2 microcontroller.  The Teensy is much faster and physically much smaller than the Mega controller currently on my robots, so I thought it might be less susceptible to motor noise – worth a shot anyway.  So, I got the Teensy/MPU9250 configuration working again, and then did the absolute minimum to get the Teensy to drive just one motor on my 2-motor robot.  Surprisingly, the Teensy/MPU9250 combination went Tango Uniform as soon as the motor started up – wow!

So, now I wasn’t any closer to solving the problem, but I did have some additional information.  Now I knew that the problem wasn’t unique to the Mega 2560/MPU6050 combination – it also happened in the same manner with the Teensy 3.2/MPU9250. This indicated to me that the issue really was high-frequency noise spikes being conducted down the motor control wires and back into the microcontroller circuits.

So, I needed a way to effectively block these transients from reaching the microcontroller.  As noted above, I tried the cheap eBay optical isolator module, but although it clearly isolated the DC circuits, the high-frequency transients still made it through to the controller board. I needed an optical isolator setup with an ‘air gap’ big enough so that there would be no chance for the transient energy to jump the gap.  And, because to a man with a 3D printer, everything looks like a 3D fabrication problem, I figured I could whip something up using IR LED/IR phototransistor pairs, something like the following:

This model is just a small solid piece of plastic with 4 holes, sized so that a 3-mm LED / phototransistor will slide in, with a step to stop if from sliding all the way through.  There is no metal at all, other than the LED/phototransistor leads, so (hopefully) no conductive or capacitive path.

After the required three or four quick and dirty model iterations, I had a model that I could use for a very basic experiment.  In fact, I realized my little 4-hole model was way overkill, as all I really needed was one channel for the motor speed PWM signal – the other two inputs to the motor controller could be tied HIGH or LOW as necessary on the motor controller side for purposes of this experiment.

The setup is shown below:

Teensy 3.2 controlling one robot motor via homebrew optical isolator

Closeup comparing the 3D-printed optical isolator with the commercial 4-channel module

After getting the 1-channel optical isolator working, I used it to control one of the two motors on the 2-motor robot, and found that I could run this setup indefinitely with no data corruption – yay!  Here’s an Excel plot of a 35+ minute run with the robot motor running.  As can be seen from the plot, the MPU9250 responded properly the entire time. The five ‘spikes’ are caused by occasional manual sensor rotation to confirm proper operation

35-minute run showing reliable yaw data acquisition. The three ‘spikes’ are caused by manual sensor rotation to confirm proper operation

For the next step, I plan to expand my homebrew opto-isolator to 2 channels so I can control both robot motors.  If that is successful, I’ll add 4 more channels so I can control both motors completely.  If this all works, then I’ll need two complete 6-channel opto-isolators to control all 4 motors on the 4-motor robot.

15 November 2019 Update:

I added two more optical isolator channels so I could control both the speed and the direction of one motor on my two-motor robot.  Then I modified my Teensy 3.2 test program to run the motor at increasing and then decreasing speeds, both forward and backward, forever.  The motor speed increments and direction changes occur in the same section of code that acquires a new yaw value, every 200 mSec or so.  Here’s the setup

This setup puts about 30 mm separation between the two circuits, with the only connection being via my homebrew optoisolator.

I ran the system for over two hours with the motor running in both directions and with the speed varied across the entire range.  The motor, the program, and the yaw retrieval process worked perfectly the entire time, as shown in the following Excel plot and short video clip.

 

This clearly demonstrates that my homebrew optical isolator works as it should, and effectively isolates motor/driver related transients from the Teensy board.  What it doesn’t do is demonstrate the same capability with the Teensy connected to the same power supply; right now the Teensy is powered from my laptop via USB, and the motor is powered from the robot battery supply.  The next step will be to figure out how to suppress these transients to the point where both the Teensy and the motor/driver can live on the same circuit.

16 November 2019 Update

Success!  I now have the Teensy controller and the Robot running from the same power supply – yay!!  I accomplished this by constructing a two-stage RC filter between the 8.5V battery supply voltage and the 4-6V Teensy power input, as shown below, and modifying the USB cable to cut the red power wire.

Two-stage power supply filter for Teensy

2-stage power supply filter

With this setup, I ran for almost 30 minutes with no problems, as shown in the following Excel plot (the ‘glitches’ at the start and finish are manual sensor rotations to confirm valid data retrieval.

 

Stay Tuned!

17 November 2019 Update

According to this link, Thomas Huxley once said “The great tragedy of science—the slaying of a beautiful hypothesis by an ugly fact.”  Well, my ‘beautiful hypothesis’ has been well and truly slain by an ugly fact! As a last experiment to verify my hypothesis about the need for both an optical isolator and a power supply filter to adequately address the two-wheel robot’s motor noise transients, I bypassed the optical isolator as shown in the following photo, and ran the setup for several minutes.  Unfortunately (or fortunately depending on one’s point of view), the setup did not cooperate with my hypothesis – the setup ran fine, with no missed yaw value acquisitions, as shown in the following Excel plot (the ‘glitches’ in the plot are due to me manually rotating the sensor to verify proper operation).  So, apparently the power supply filter is required, but the optical isolator is not – Yay! (I think).

 

 

 

So then I removed the Female-Female jumpers and wired the motor control lines directly to the Teensy – no problem.  Then I added the three control lines for the second motor, and changed the Teensy program to control both motors.  This ran fine, so it is now clear that all that is required for motor transient suppression (at least for the Teensy setup) is a good power supply filter.  It’s even possible that my 2-stage power supply filter circuit is overkill for the application, and one stage would do fine.  To test this theory, I eliminated the first stage of the filter entirely, and tried again.  This works as well, so it looks like only a single-stage power supply filter is required for reliable operation, at least with the two motor robot and the Teensy microcontroller.  However, one fly in the ointment is that eliminating the first stage causes the power dissipation of the 5V Zener diode to increase to well over its nominal max Pd of 1W.  With a 10-ohm series resistor dropping 3.5V, the current through the zener is 3.5/10 = 0.25A (minus the negligible Teensy current), so Pd = 0.25 * 5 = 2.5W – oops!  I need to increase the series resistor by a factor of 2.5 to get Pd down to 1W.  So, I increased the series resistor to 20 ohms and with the actual power supply voltage of 8V, the zener Pd falls well below 1W.

For a final confirmation test, I eliminated the power supply filter entirely, bringing the setup back to to the original baseline, with the Teensy controlling both robot motors with no power supply filtering or optical isolator.  As expected, this caused the Teensy to lose synch with the IMU within a few minutes.

19 November 2019 Update:

After testing with the Teensy/MPU9250 combination, I decided to try and go back to the Arduino series of microcontrollers.  The reasons for doing this are:

  • The Teensy doesn’t have sufficient I/O channels to control all the required peripherals on the 4-wheel robot.  I does have enough I/O to control the 2-wheel robot, but that’s not what I’m after.
  • Remote programming of the Teensy is not possible without the use of another controller, like a Raspberry Pi Zero W or something like that.  However, this is easy to accomplish with the Arduino UNO or Mega 2560.

So, I started with an Arduino UNO that has been modified by removing T1 so board power is isolated from the USB cable power lead, and hooked up the motor control lines directly to the 2-wheel robot.  This arrangement failed after a few minutes, with or without power supply filtering.

So then I re-introduced my homebrew optical isolator, and found that the UNO will run indefinitely while running one of the two motors via the optical isolator and the power supply filter.  So at this point a reasonable hypothesis is that the Teensy is a bit more robust with respect to EMI/RFI effects than the UNO/Mega controllers, but the combination of the two-stage power filter and my homebrew optical isolator effectively suppresses motor and motor driver EMI/RFI (at least for one motor).

30 November 2019 Update:

I now have my two-motor robot running reliably with an Arduino UNO running my DF Robots MPU6050 module, with both motors running. As far as I have been able to determine, this requires both a power supply filter between Vbatt and the UNO Vin, and a 6-channel optoisolator between the UNO and the motor driver module.  Some photos and schematics are shown below:

6-channel homebrew optoisolator and power filter mounted on an Adafruit ‘Perma-Proto’ half-size breadboard

Optoisolator mounted on two-motor robot. Note DF Robots MPU6050 module in top left of photo

6-channel optoisolator schematic

 

03 December 2019 Update:  The Final Chapter (I hope)

After going through the entire process described above, proving to my satisfaction that the ‘final cure’ to the motor noise problem was to opto-isolate the motor driver control signals and thoroughly filter the power to the UNO, I was once again rudely smacked in the face by reality when my finely tuned setup refused to cooperate. As I prepared to start ‘field’ testing again, the yaw value corruption problem once again reared its ugly head.  This was beyond depressing – I have now spent upwards of 5 months trying to kill this particular alligator, and it keeps coming back.

Based on my philosophy that if I’m having an apparently insoluble problem, there’s someone (possibly many someones) out there in the i-verse that has had (and solved!) that particular problem already, I started over on internet research, googling for ‘motor noise problem’ and similar terms.  One of the hits I got was for an application note on the Pololu site dealing with just this problem (don’t know why I didn’t see it before, but…). In any case, Pololu’s solution was to install one or three bypass caps on the motor body itself.  Since I had already tried everything else, I thought ‘what the heck -it can’t get any worse than it is already’, and installed the three-capacitor arrangement, using non-polar 1 uF caps from each terminal to the motor body, and a 0.01 uF cap across the terminals.

Pololu D20 metal-geared motor with bypass caps installed. Blue caps are 1uF, orange cap is 0.01uF

Lo and Behold! It worked!  I was able to run for 90 minutes without a problem with all three elements in place; the motor bypass capacitors, the power supply filter, and the 6-channel optoisolator.  So, the next step was to bypass the optoisolator and see if that was a necessary component – and it continued to work without problems – yay!  Next, I bypassed the power supply filter circuit, and everything STILL continued to work great – double yay!

Two-motor robot with power supply filter and optoisolator bypassed

So, I wound up back at the beginning of my five-month circular journey, having learned a lot and had a lot of fun, but having wasted half a year.  So, if you are using metal-geared motors like the Pololu D20/D25 models, for Pete’s sake install bypass capacitors before doing anything else!

Stay tuned,

Frank

 

MPU6050 FIFO Buffer Management Study

12 Replies

Posted 13 October 2019

I have been attempting to use the Invensense MPU6050 6-axis IMU for some time now in both my two and four-wheel robots for improved wall-following ability. By measuring the relative heading change during turns, I could get the robot to accurately acquire and maintain a specified distance from the currently tracked wall.  I say ‘attempting’, as I have experienced somewhat mixed results in getting reliable results from the IMU.  A large part of the problem, as I described in this post, wasn’t the IMU at all, but rather the sensitivity of the Arduino I2C bus to RFI/EMI caused by the motor drivers.  However, even after solving this problem, my programs would still occasionally ‘lose synch’ with the IMU’s FIFO buffer and start returning garbage for heading values – not good!

In addition to the I2C issue, there are several factors that make the MPU6050 harder do work with:

  • Invensense, the company that makes the MPU6050 chip, appears to have been purchased by TDK, and their technical support forums don’t appear to be supported any longer.
  • Apparently there are significant pieces of the MPU6050 firmware that aren’t freely available as human-readable code, so much of the MPU6050 magic is just that – magic.  In particular, the details of how the MPU6050 handles its internal FIFO are (at least to me) completely unknown, except by reverse-engineering.
  • While there is a lot of MPU6050-related information and code floating around out there in the i-verse, that is as much a trial as a blessing; it has been very difficult for me to wade through everything and to try to sort out the wheat from the chaff.
  • Jeff Rowberg’s wonderful I2CDevLib contains support for the MPU6050, with examples.  While it is fairly easy to get started using Jeff’s examples, it was difficult for me to understand how the examples work so I would know how to modify them for my application without running off into the bushes.
  • Almost all the example code out there assumes an interrupt based IMU management scheme.  For my wall-following robot application, the interrupt scheme was overkill and then some, so I wanted to use a polling arrangement, which is very poorly documented. Eventually I developed a working polling algorithm (described here) , which I now use in my robots.
  • Invensense (now TDK) has released several updates to the MPU6050 firmware, and it is difficult (at least for me) to figure out what the differences are and whether or not those differences are worthwhile for my application.  There is some information in the header files provided by Jeff Rowberg, but if there is any sort of formal change history, I haven’t found it.

Despite all this, the MPU6050 is a wonderful device, and it’s EVERYWHERE – you can get MPU6050 modules from Adafruit, Sparkfun, DFRobots, and GY-521 Chinese knockoffs from eBay.  The GY-521 modules have some reputed issues with quality control, but at about $1-2 per module, it’s hard to go wrong.

In my continued attempts to understand how the MPU6050 works, and the details of some of the latest example code provided on Jeff Rowberg’s Github site, I posted an issue related to a particular part of the example code that defied my ability to understand, as shown below:

This code obviously works, but I get a headache everytime I try to figure out how it makes sense.  One of the replies I got from this post was from Homer Creutz, who I knew to be one of the very best experts on all things MPU6050.  The gist of Homer’s reply was “yeah, it looks kinda clumsy, but it does work”. But then Homer went on to suggest an alternative using the modulus (%) operator that piqued my interest, as I had used this technique in my four-wheel robot code.  In subsequent email conversations Homer went WAY out of his way to thoroughly answer my stupid questions about the MPU6050, especially about the issue with FIFO overflow.  He sent me a link to this video explaining how circular buffers work, and the following graphic illustration (slightly edited for clarity) of the problem of MPU6050 overflow and multiple-byte packets

The combination of the video and the above simple graphic finally allowed me to understand why properly managing FIFO overflow is so critical for successful MPU6050 implementations. Ironically, FIFO overflow management is more of an issue in my low-speed, high mass, low-dynamics environment than at the other end of the scale.  In high-dynamics applications, FIFO overflow is almost never an issue because the application sucks data out of the FIFO as fast as it is put in, in order to provide the best possible stabilization and control.  However, in low-dynamics applications like my wall-following robots, there is no need for IMU information more than a few times per second, meaning that the FIFO will almost certainly overflow if it isn’t proactively drained even if the information isn’t really needed.

Homer also sent me a couple of untested alternatives for FIFO management to replace the ‘while() within an if()’ algorithm, so I decided to test them and report the results back to Homer. After all, Homer was going WAY out of his way to answer my ignorant questions, so the least I could do was to be his lab tech.  So, I started with Jeff Rowberg’s MPU6050_DMP6_using_DMP_V6.12 example (the one with the ‘while() within an if()’ snippet) and modified it to deliberately cause FIFO overflows. After a bit of debugging and some very slight changes to Homer’s code, I got the following output from a run using a 100 mSec delay at the start of loop():

As can be seen from the above, about 20 interrupt cycles are skipped in each loop() iteration, causing the 1024-byte FIFO contents to expand by either 280 or 252 bytes each time, until it overflowed and was reset.  The example code handled FIFO overflows properly resetting the FIFO each time so that the retrieved yaw values continued to make sense.

The next step was to replace the example code with Homer’s proposed setup using the modulus operator for FIFO management. The first section below shows the example code loop() function before Homer’s modifications:

And the following shows the same loop() function after implementing Homer’s suggested code:

This resulted in the following output:

Showing that FIFO overflow was indeed handled properly.  The FIFO overflowed after the 3rd time through the loop, (the returned count was capped at the 1024-byte physical length of the FIFO), and Homer’s code correctly removed the 16 corrupted bytes at the beginning of the FIFO, plus one more 28-byte packet.  The subsequent mpu.getFIFOBytes() call retrieved an entire valid packet, which was then process to produce a valid yaw value.  Of course, since only one extra packet was removed, the FIFO overflowed again when the next 336 bytes were loaded during the 100 mSec delay at the start of the next loop() iteration.

When the code snippet to retrieve all available packets

was uncommented from the above program, I got an almost perfect output as shown below:

There were 4 bad values starting at 15250 mSec, as shown below

I’m not real sure what happened here.  A FIFO count of 308 is nowhere near the overflow condition, and it is an even multiple of 28 (the packet size), so everything should have gone swimmingly.  However the displayed value of 2.04 degrees at time 8859 mSec is clearly incorrect, as I was manually (and slowly) rotating the MPU at the time.

Another issue with all of this is the FIFO count associated with the number of interrupts shown in the output.  22 interrupts should produce 22*28 = 616 bytes, but mpu.GetFIFOCount() only returns 308 – exactly half the expected value.  So, either the packet size is actually 14 bytes (not very likely, as mpu.GetPacketSize() returns 28) or the IMU is only loading half a packet on each interrupt.  I added digitalWrite() statements to the ISR so I could directly monitor interrupt occurrences with my O’scope, and the interrupts happened exactly as expected, at approximately 4.58 mSec intervals (about 220Hz).  The 100 mSec delay at the top of loop() should produce approximately 100/4.58 = ~22 interrupts each iteration, and that is what is reported in the output.  So, why the x2 error in the reported FIFO count?

I ran another test, and this one responded perfectly for as long as I let it run (about 22 seconds). During the run I manually rotated the robot (and its attached IMU) back and forth, as shown in the following Excel plot

14 October 2019 Update:

There is clearly something not-quite-right with the way the MPU6050 reports the current length of the FIFO contents.  Here are the results from a recent run:

Aside from the fact that 22 x 28 = 616 not the reported 308, there is also the problem that after one more interrupt (23 vs 22), the reported FIFO content length didn’t go up by 14 (half the expected 28, but…) but instead by 12 bytes – what the heck?  This clearly implies that MPU interrupts aren’t entirely synchronous with actually filling the FIFO – like some sort of race condition?  In other words, the number that is reported by mpu.getFIFOCount() isn’t always an integer multiple of the packet size!  This is contrary to Homer’s assumption that the only way for mpu.getFIFOCount() to retrieve a non-integral multiple was for the FIFO to overflow. This clearly is not happening here, but I’m still getting non-integer multiple results.  Here’s another snippet from the same run:

In the above snippet, it can be seen that a 22 interrupt interval can sometimes result in 316 bytes being reported rather than the expected (ignoring for the moment the issue of a 2x error), and the ‘success’ of removing 36 bytes still resulted in a ‘bad’ yaw value computation (-179.17) 308.  In the very next loop iteration, 22 interrupts resulted in 328 bytes being reported. This time removing the excess allowed a valid yaw computation (-30.46).  So, a 22 interrupt loop interval can result in 308 (the ‘normal’ result), 316, or 328.

15 October 2019 Update:

I changed the loop() delay time from 100 mSec to 200 mSec, and (with no other changes) re-ran the test, with the following output:

The above results showed the expected 42-43 interrupt count between loop() iterations, and the expected (ignoring for the moment the 2x error factor) FIFO contents byte count of 588-616.  However, there were a couple of anomalous occurrences on two consecutive loop() iterations at 19094 and 19312 mSec.  The first one reported a FIFO contents count of 600 instead of 616 and (even after error correction) a clearly erroneous yaw value of -179.20, and the second one reported 604 bytes and (after error correction) an apparently valid yaw value of 61.54.

After an email exchange with Homer, I tried replacing this line

with this one:

In the following section of Homer’s algorithm

After this change, I re-ran the test at the 100 mSec loop() iteration delay setting with and without the above code change.  In both cases, I still got errors trapped,  as shown below (The test conditions for each run below are noted at the top of the text file)

So then, also at Homer’s suggestion, I instrumented the code to get the FIFO count rapidly several times after an error detection to see if the first mpu.getFIFOCount() occurred while data was actually being loaded into the FIFO and therefore got an erroneous count. So, I changed Homer’s code correction section to the following:

and re-ran the 100 mSec loop() iteration delay test.  What I got was this:

Wow!  the FIFO count changed!  The first mpu.getFIFOCount() at the top of the detection section got 334, and the next 3 calls all got 336!  So the first mpu.getFIFOCount() call must have hit the mpu 26/28 of the way through the packet load!

So, the MPU6050 packet load scheme isn’t atomic and there is, in fact, some sort of a race condition.  I think we have all been assuming that the MPU6050 loads the FIFO with a complete packet and then triggers an interrupt, while it appears that it is actually happening the other way around; the interrupt is triggered and then the packet is loaded into the FIFO. Most of the time this doesn’t cause a problem, but if you ‘get lucky’, the register associated with the mpu.getFIFOCount() call is read before the entire packet is loaded

16 October 2019 Update:

Homer asked me to change the code to determine exactly how long it takes to “clear the error”, which I take to mean “how long would a program have to wait for the MPU6050 to finish loading the rest of the packet into the FIFO”.  Homer sent me some sample code, which I modified slightly to produce the output Homer was looking for, as shown below:

When I ran this code, I got the following output:

 

21 October 2019 Update:

After several more email exchanges with Homer, he believes that he has now come up with pretty ‘bullet-proof’ way of handling MPU6050 errors, with the following subroutine

The idea behind this subroutine is to ensure that any overflow condition is detected and managed properly. The routine is completely independent of interrupts, so it can be used in a program using interrupts or polling.

Homer also sent me some test results using the program, with a variable loop delay time designed to be just below and then just above the delay required to overflow the buffer. This demonstrated that his subroutine properly handles FIFO overflow conditions, and returns valid packet data whenever possible.

In the above output, the first column is the loop delay in mSec, then the ‘Flag’ value returned by the subroutine, and then the ypr (yaw,pitch, roll) values extracted from the buffer filled by the subroutine.  As is shown, loop delay values above about 177 mSec start returning ‘2’ Flag values, indicating the routine detected (and recovered from) an overflow condition.

I replicated this experiment on my end, but discovered that for my installation, I had to use a loop delay almost exactly twice the value used by Homer (360-370 vs 177-178). The implication is that either my MPU6050 is loading the FIFO at one-half the rate of Homer’s unit, or my IMU has a buffer size twice the one being used by Homer.  Curioser and Curioser ;-).

Here’s my code and results:

Summary of results to date:

 

  • Homer has clearly created an effective algorithm for detecting and recovering from FIFO overflow events, and the subroutine that implements his algorithm can be used in an interrupt-driven or polling configuration.  I personally like the polling arrangement because it requires one less connecting wire, and removes the need for an ISR.
  • Both Homer and I have demonstrated that the algorithm works as designed, but the loop delay required to just trigger FIFO overflows in my configuration is almost exactly twice the delay needed for Homer’s. This needs to be explained.
  • There is still the problem of a factor of 2 error between the expected return from mpu.getFIFOCount() and the number calculated by multiplying the number of interrupts times the expected packet length. In my configuration using an interrupt-driven arrangement, a 22-interrupt loop delay resulted in a FIFO count of 308.  But, 22 x the expected packet size of 28 yields 616, not 308!  This also needs to be explained.

23 October 2019 Update:

To further investigate the ‘factor of 2 error’ problem, I went back and re-ran the initial experiment that produced the problem, just to verify that it was still there.  Here’s the entire program to recreate the results, and a partial printout of the results:

Significant points to note from the above output:

  • The time (in mSec) between adjacent output lines is about 111 mSece on average (110.5762 mSec according to Excel), and the reported number of interrupts is either 20 or 21 in almost every case (average is 20.0833333 according to Excel).  O’Scope observations confirm this is the case, as the output from the interrupt monitoring pin shows almost exactly 5 mSec between interrupts and an almost exactly 200 Hz interrupt frequency.  So, an interrupt count of 20 or 21 is reasonable, and cannot be the reason for the ‘factor of 2’ error in the FIFO buffer count.  However, there is an apparent ‘off by 2’ problem with the interrupt count, as the reported FIFO counts are consistent with interrupt counts of 22 & 23 rather than 20 or 21 as shown
  • Every so often a 22 interrupt span produces a FIFO count of 336 instead of 308 – a 28 byte difference.  In fact, over a run time of about 6.5 minutes (388,671 mSec), this phenomenon occurred 264 times, about 0.07% of the time.

The inference I draw from the above two points is that the MPU6050 chip isn’t actually loading an entire 28-byte packet during each interrupt cycle, but is in fact loading only 14 bytes each time.  With an artificially imposed 100 mSec (about 22 interrupt cycles) loop delay, the MPU loads 22 * 14 = 308 bytes.

An alternative explanation is that the MPU6050 loads complete packets into the FIFO every other interrupt. Under this scenario, it takes 11 cycles (22 interrupts) to load 11*28 = 308 bytes, and 12 cycles (24 interrupts) to load 12 * 28 = 336 bytes.

Another (and maybe even more reasonable) possibility is that the MPU6050 loads packet data into the FIFO one byte at a time, at an average rate of 14 bytes per 5 mSec, or 2.8 KBS. To the outside world, this might not be noticeable unless the application was trying to retrieve packets at the full 200 Hz.  At rates of 100Hz or less, the MPU would still have loaded at least one complete packet every time one was requested.

29 October 2019 Update:

As someone once said as both a benediction and a curse – “May you live in interesting times”.  In our ongoing investigation into the depths of MPU6050 behavior, we now have solved one mystery solved, only to encounter another one.

As I have noted several times above, there appears to be a factor of 2 mismatch between the number of interrupts counted from ISR activations and the number of bytes reported by mpu.getFIFOCount().  Either the interrupt count is off, or the FIFO count is off, and there doesn’t seem to be any other explanation.  Well, now I have determined that there is apparently a third option – that the MPU6050 only loads a packet into the FIFO on every other interrupt! This doesn’t make a whole lot of sense to me, but I now have what I believe to be irrefutable proof that this is exactly what is happening.

I set up a very simple program to get the FIFO byte count as rapidly as possible.  In order to avoid slowing the system down, I stored the results in a 1000-entry array during the retrieval process, and then printed out all 1000 entries at the end. Then I plotted the results in Excel as shown in the following figure:

The stairsteps in the above plot are (almost) uniformly the expected packet size of 28 bytes;  the MPU6050 is clearly loading entire 28-byte packets into the FIFO each time, contrary to one of the possibilities I had considered to explain the factor of 2 inconsistency between the interrupt count and the FIFO count.

However, when I started looking at the time interval between FIFO loads, I got the following plot:

As the above plot clearly shows, the MPU6050 loads a new packet into the FIFO every 10 mSec or so (I think it is exactly 10 mSec, with the differences explained by the lack of resolution in the time axis).  But wait – the MPU6050 produces an interrupt on the Arduino interrupt pin every 5 mSec – not every 10 mSec!  So, the mystery of the ‘factor of 2’ error is now solved.  The MPU6050 loads a new packet into the FIFO every other interrupt – not every interrupt as expected. So, the interrupt count is too high by a factor of 2 when compared to the actual FIFO count.

Unfortunately (or fortunately depending on one’s point of view), that simply begs the question – why is the MPU6050 producing interrupts on a 5 mSec schedule when it only changes the FIFO count on a 10 mSec schedule?  Who knows?

Changing the subject slightly, I got another ‘overflow proof’ version of GetCurrentFIFOPacket() from Homer Creutz to try.  I set up a small program to test it.   Since the idea was that GetCurrentFIFOPacket() would return a valid packet no matter how long it had been since the last time it was called, I set up a program to iterate through a sequence of delay times from 100 mSec to 500 mSec. For each loop delay setting I called GetCurrentFIFOPacket() multiple times and printed out the extracted yaw value and the return status from the function.

As the plot below shows, everything works swimmingly up to a loop delay of 350 mSec. Unfortunately, the wheels came off with a 400 mSec loop delay, and the packet values were all invalid after that – bummer!

 

06 November 2019 Update:

Holy cow!  Homer and I started this marathon project back in mid October, and we don’t seem to be any closer to nirvana than we were before.  What I can say though, is that I have learned a lot more about the MPU6050 and Jeff Rowberg’s driver software ;-).

One of the major issues we encountered with the polling method (vs interrupt-driven) is that, without the synchronization with MPU6050’s internal processes provided by the interrupt model, we can’t count on (no pun intended) the FIFO count returned by mpu.getFIFOCount() being accurate.  Depending on the timing of the call, the return value could be wildly inaccurate.  However, we discovered that two back-to-back calls to mpu.getFIFOCount() always resulted in an accurate count, although there was still a very small probability that a 3rd call would be required.  So, I created a small routine called ‘getPollingFIFOCount()’ to wrap this construct, as follows:

This function simply loops until two adjacent calls to mpu.getFIFOCount() return the same value. As always, however, there is a backup counter to force the function to exit if it gets hung up for any reason.  In the calling program I have a line like:

to set the backup loop counter value.

Another major issue with the MPU6050 is that it can overflow its packet FIFO buffer, and there doesn’t appear to be any way to prevent this, other than removing packets from the FIFO at the same rate or higher than they are loaded.  This may not be a problem for ‘high dynamics’ applications like quadcopters or balancing robots that need continuous attitude information, but for ‘slow dynamics’ applications like my wall-following robot where yaw information is only needed a few times per second, overflow becomes a practical certainty.  In an interrupt -driven environment, it might be reasonable to simply retrieve and then discard DMP packets as fast as they arrive, and then only process the latest packet when the application needs an update.  However, for a polling strategy, doing this may or may not work depending on what else is going on. So, for polling we need a way of ensuring we can retrieve a valid packet from the DMP FIFO, whether or not the FIFO has overflowed. Doing so would be trivial if the FIFO length was an integral multiple of packetSize, but it isn’t – yuk!!  So, now when the FIFO overflows, the first packet in the FIFO is guaranteed to be corrupted. The good news is, the last complete packet (i.e. the most recent information)  in the FIFO is always valid, but getting to that last good packet is non-trivial.  To summarize:

  • FIFO overflow is almost certain to happen in a ‘low dynamics’ polling-based program
  • When FIFO overflow occurs, the first packet is always corrupted, but the last one is still valid
  • The last (valid) packet isn’t the last [packetSize] bytes, due to the non-modular size of the FIFO
  • The MPU6050 DMP may start loading another packet at any time, but there will always be 10 mSec or so between packet loads
  • Any last-valid-packet retrieval algorithm must work for any loop delay time.

So, the ‘last-valid-packet-retrieval’ algorithm is something like this:

  1. Get the current packet count, using the ‘getPollingFIFOCount()’ routine above
  2. Extract [packetSize] chunks until there are less than 2* [packetSize]  bytes left. This ensures there is exactly one valid packet remaining in the FIFO.
  3. Extract one more [packetSize] chunk and return it as the result.

Note that implementation of this algorithm will require several ‘while’ loops, so there must also be provision for forcibly terminating all such loops in case some edge condition causes the normal exit condition to never be reached.  Homer and I have been creating and testing versions of this function for the last couple of weeks, without quite getting there yet.  Either they don’t handle all the edge conditions, or they are too slow as the loop delays get larger.

While I was testing my most recent concoction, I ran into a third major problem with the MPU6050 API.  I wanted to be able to remove up to 35 [packetSize] chunks (980 bytes) in one go, and I expected the mpu.getFIFOBytes() API call to manage the required chunking for me.  When I tried this trick, the getFIFOBytes() function crashed repeatedly.  Eventually I figured out that the reason it was crashing wa that it’s ‘length’ parameter is declared as a ‘uint8_t’ object and of course it couldn’t handle a value greater than 255 without choking.  I thought that was a little odd, but that maybe changing the declaration from ‘uint8_t’ to ‘uint16_t’ in MPU6050.h/cpp would solve the problem.  Nope – It turns out that, due to the way that the Arduino I2C bus operates, there is a limitation on how many bytes can be transferred across the bus in one operation, and this limit is currently set at 32 bytes.  As a result of this fundamental limitation, all the I2CDev functions that use the I2C bus also have the same underlying limitation and all of them have their length parameters declared as ‘uint8_t’. This reminds me of the old pre-scientific myth about the earth resting on the back of a turtle.  When the myth was challenged, the defender says “its no use – it’s turtles all the way down”.  In our case “it’s no use – it’s uint8_t all the way down!”.

So, I had to figure out a way around this problem, so I decided to create yet another wrapper function, this one cleverly called ‘getManyFIFOBytes()’. The idea for this would be to pull one [packetSize] chunk off the FIFO at a time using the normal ‘mpu.getFIFOBytes()’ call and place the result in a destination buffer large enough to hold the entire result.  Since it has been a (long, long) while since I last played with pointer gymnastics, I decided to write a short test program to figure out a reasonable technique.  Here is the program, and some results:

As the output shows, the ‘getManyFIFOBytes(uint16_t* buffer, uint16_t len)’ function can take an arbitrary ‘uint16_t’ length parameter and fill the destination buffer with [packetSize] (28 bytes in this case) chunks, followed by the non-modular remainder.  Although this test was done with a simulated receive buffer and simulated packet contents, I believe it will work using the actual contents of the MPU6050 FIFO and repeated calls to ‘mpu.getFIFOBytes()’ to retrieve the ‘chunks’ and any non-modular remainder bytes.

Having convinced myself that my two helper functions actually did what I wanted, I revised my latest MPU6050 test program (MPU6050_Overflow8.ino) to test the whole thing out. The test program steps through a series of loop delays from 50 to 550 msec, and takes 25 yaw measurements at each loop delay setting.  The Excel plot and a snippet of the output log from the program is shown below

  • There were no invalid packets in the entire run, so the attempt to avoid invalid packet retrieval was a success.
  • The actual loop delay per measurement pass varied quite a bit from the desired loop delay setting.  For instance the average measured loop delay for the 25 yaw measurement passes at the 200 mSec loop delay setting was actually 274.08 mSec, almost 50% higher than desired.  At the 150 mSec loop delay setting, the average loop delay was 223.88 mSec, and at the 100 mSec setting it was 133.24 mSec.  So, if the application needs 5 measurements/sec, the allowable loop delay between passes needs to be between 100 and 150 mSec.

12 November 2019 Update:

Based on some comments and data from Homer’s experiments, it appears that a FIFO reset can be done in less than 10 mSec.  This means that a packet retrieval algorithm based on a mpu.resetFIFO() call will miss at most one 10 mSec FIFO load interval, which is insignificant compared to the typical polling interval (200 mSec in my case).  So I decided to try a ‘brute force’ approach to ‘GetCurrentFIFOPacket()’ as follows:

When I first tested this algorithm, I ran into an occasional glitch where the FIFO would somehow fail to reset, resulting in a FIFO count > 28. So I added an outer loop to allow multiple shots at getting a clean FIFO reset.

Shown below are some Excel plots from some long runs

 

The first plot above shows a long run (over 17 minutes) of valid yaw data (the perturbations in the yaw plot are due to occasional manual rotations of the sensor to verify that the sensor was still responding).  The interesting thing about this plot is the yellow curve, showing the ‘outer loop’ count. The only way this value can be greater than 1 is if the first mpu.resetFIFO() call fails to actually clear the FIFO, which appears to happen an average of about once per minute, or about once every 6,000 10 mSec MPU6050 FIFO cycles.

The second plot is a closeup of the first 171.42 seconds of the overall plot, showing the detail of the FIFO clear failures, occurring about once per minute.

So, it is clear that the MPU6050 has some significant behavioral quirks, especially when used in a non-interrupt-driven environment.  That being said, I believe the ‘brute force’ algorithm shown above is a reliable way of interfacing with the MPU6050 in a polling environment, and obviates the need for a separate interrupt line, and the associated ISR software.

This will probably be the last update on this subject, as I now think Homer and I have pretty much beat the MPU6050 FIFO issue to death.

Stay tuned!

Frank

 

 

 

 

 

 

 

 

Heading-based Turns Using MPU6050 and Polling vs Interrupts

3 Replies

Posted 06 October 2019

In previous posts I have described my efforts to integrate heading-based wall tracking into my two-wheel and four-wheel robots.  I installed a MPU6050 module into Wall-E2, my primary four-wheel robot, some time ago but was never able to make heading-based turns work for one reason or another.  In conjunction with some other experiments, I installed an MPU6050 module on my two-wheel robot so that I could investigate the issues with heading-based turns and heading-based wall tracking with a simpler hardware configuration.

With the two-wheel robot I was able to demonstrate successful heading-based wall tracking, but I was unable to port the capability to my four-wheel configuration.  Not only that, but for some reason I started having problems getting reliable yaw/heading values from my two-wheel robot configuration.  This post describes the steps I took to troubleshoot the problem, ultimately arriving at a stable polling-only (no interrupt line required) yaw/heading value retrieval algorithm suitable for both the two-wheel and four-wheel robot configurations.

Back to Basics:

As I always do when faced with a complex problem with conflicting results, I decided to simplify the problem as much as possible.  In this case that meant reducing the hardware configuration to just a MPU6050 module and an Arduino Mega controller, as shown below:

Arduino Mega and MPU6050

On the software side, I started with the simplest possible Arduino sketch – Jeff Rowberg’s ‘MPU6050_DMP6.ino’ example, included in his latest I2CDevLib library and described in this post.

After getting everything running properly in this very basic configuration using an interrupt-driven algorithm, I moved on to working with the polling-driven arrangement, to confirm that polling was a viable strategy.  To do this I modified the hardware to disconnect the interrupt line from the MPU6050 to the controller board, and modified the software as described in this post to use a polling arrangement vs interrupts.

After confirming that this simple example worked properly and seemed stable, it was time to work my way back into the two-wheel robot hardware and software configuration (again!).  To do this I started with the basic controller/MPU6050 only hardware configuration, but running my two-wheel robot software program, modified to eliminate everything but the ‘RollingTurn()’ function that uses heading information from the MPU6050 to initiate and terminate turns.  After some false starts and blind alleys, I finally arrived at a stable software configuration demonstrating consistent heading-based turn performance using polling only – no interrupts!  the code is shown below:

In the above code, the only relevant functions are the ‘GetIMUHeading()’ and ‘RollingTurn()’ functions, as shown below:

When the ‘RollingTurn()’ function is called, it waits for mpu.dmpPacketAvailable() to return TRUE, and then it calls GetIMUHeadingDeg(), which updates a global variable (subtly named ‘global_yawval’.  This value is then used to determine turn completion.

GetIMUHeadingDeg() reads bytes from the FIFO and computes a yaw value using the retreived quaternion data.

After getting everything going to my satisfaction, I added code to setup() for a 30-degree turn to the right followed by a 30-degree turn to the left, followed by an infinite loop of yaw value readouts.  The output from one test run is shown below.

Shown below are the yaw values plotted against time in Excel

The next step will be to port the updated software back into my two-wheel robot to confirm that heading-based turns can be accomplished automatically (this is something that I had going before, but…).

Stay tuned!

Frank

 

 

 

 

 

Polling vs Interrupt with MPU6050 (GY-521) and Arduino

2 Replies

Posted 04 October 2019,

In my last post I described my Arduino Mega test program to interface with the popular Invensense MPU6050 IMU and it’s GY-521 clone.  In this post I describe the interface configuration for using a polling strategy rather than relying on the IMU6050’s interrupt signal.  A polling strategy can be very useful as it is much simpler, and saves a pin connection from the MPU6050 to the controller; all that is required is +V, GND, SDA & SCL, as shown below:

With this wiring setup, the control program is shown below:

In the above program, the interrupt service routine (ISR) and the accompanying ‘attachInterrupt()’ setup function have been removed as they are no longer needed for the polling arrangement.  Instead, the program calls ‘mpu.dmpPacketAvailable()’ every time through the loop, and if a packet is available, GetIMUHeadingDeg() is called to read the packet and return a yaw value.  The rest of the code in the loop() function is the place holder for the ‘other stuff’ that the program does when it isn’t paying attention to the IMU.

In this test program, I have set this section up to execute every 100 Msec, but in my robot programs I usually set it up for a 200 Msec interval; 5 cycles/sec is plenty fast enough for a wheeled robot that uses only the IMU yaw information for turn management.

So far, this arrangement seems very stable; I have been running it now for several hours without a hiccup.

Stay tuned,

Frank