I've been tearing my hair out for the last couple of days trying to work out why our automated SPI test was failing on the Raspberry Pi I use for testing.
I was pretty sure it wasn't a problem with the interface board. I'd done a manual continuity check and was as sure as I could be that the Pi's expansion header was connected to the right pins of our SPI bus.
I didn't think it was a problem with our software. We've done quite a bit of testing with SPI, and that part of our library hadn't changed since the last time the test was passing.
The original tests were run on a rev 1.0 Raspberry Pi, and we're now using a rev 2.0 board. I doubted that was the cause. Although GPIO has changed slightly, and the Rev 2.0 uses a different I2C bus, the SPI pins are unchanged.
I was mystified, and decided to bring out the big guns. I don't have an oscilloscope these days (though it is on my shopping list for next year). Instead, when I need to debug I2C or SPI, I use the Open Workbench Logic Sniffer from Seeed studios.
The OWLS is an amazing piece of hardware. It's about $50 from Seeed studios, and you need to buy one or two probe cables to go with it, but it has most of the features of bus analysers costing several times more.
Here it is connected to the board under test. The OWLS is the red board at the bottom of the picture.

Open Logic Workbench Sniffer debugging an SPI board
The board uses the SUMP protocol, and there is a great SUMP client called OLS. A few minutes of debugging found the source of the problem - I'd got the SI and SO wires reversed when I plugged jump leads from our header to the prototype circuit. I had checked and rechecked, but I guess my brain was seeing what I expected to see, not what was actually there.
Once I'd got that corrected, the SPI test passed with flying colours. Here's a screenshot from the OLS client showing the SPI bus traffic when the software is sending data to the board.

- click to see full-size
I'm taking a couple of lessons away from this.
- The OWLS is the first thing to use when I'm puzzled by a problem with SPI or I2C.
- It's really easy to get things wrong when you're connecting jump wires. That's why we link our production boards with IDC cables.
The story has a happy ending, too: here's the output from our loop-back test:

Passing SPI test