A vehicle approaches the entrance to Duxbury Beach. A sensor in the road detects it. An RFID reader picks up a tag. A camera takes a picture. The software has to put those pieces together and decide which vehicle just entered the beach. On paper, that's a few lines of logic.

In practice, the reader might pick up the tag before the vehicle is anywhere near the camera. It might read a tag from a vehicle that isn't entering at all. A ranger might park right in front of the equipment for an hour. And the whole system runs on a solar panel and a cellular modem at the edge of the ocean. None of those are traditional software bugs. They're what happens when software meets the real world.

Fix it or throw it away?

I didn't start this project from scratch. I inherited it — and the station I inherited identified vehicles correctly only about 20% of the time and barely lasted a day on its battery. The first real engineering decision was whether to keep fixing the existing code or start over. The existing code was too messy and inefficient to build on — fixing it would have taken longer than rewriting it — so I decided to throw it away and start over. Everything below describes the system I built after that decision, and the reasons it ended up working.

What the system actually was

The station at the entrance was a Raspberry Pi running a Python program I wrote and maintained. Magnetic vehicle loops buried in the entrance and exit lanes detected when a vehicle passed over them. A UHF RFID reader with four antennas covering the lanes read the tags on permitted vehicles. A camera captured an image of each vehicle. A cellular modem sent everything back to a server. And the whole station ran from a solar panel and a battery.

Every time a loop tripped, the Pi built a small event — which lane, which direction, the tags the reader had seen, the battery voltage, and the temperature at the station — and queued it to be sent. That design sounds simple. Each part of it ran into a different piece of the physical world.

One station, many boundaries Vehicle + tag speed, position, timing Magnetic loops RFID antennas Network camera Raspberry Pi correlate loop + tags schedule device power queue events Relay modem power Cellular modem limited bandwidth Server Solar panel + battery · one shared power budget for everything above
Each box is a place where software meets something it doesn't control: vehicle movement, radio signals, power, a mechanical relay, or a cellular network. Conceptual view, not a wiring diagram.

1. A valid reading isn't a valid event

The first assumption to fall was that an RFID read means a vehicle entered. It doesn't. A UHF reader sees any tag within range of its antennas: a car waiting in line, a car in the other lane, a truck parked nearby. So the RFID reader couldn't be what defined an event. The magnetic loop in the road did. An entry only existed when a vehicle physically crossed the loop; the RFID data was evidence attached to that event, not the event itself.

The reader kept a running list of every tag it had seen in the last ten seconds, and for each one it reported which antennas saw it, how strong the signal was, a direction, and how many times it had been read. When a loop tripped, the Pi pulled that list and sent all of it up with the event. I deliberately didn't try to decide on the Pi which tag was "the" vehicle. The device collected the evidence; the server made the call. That meant the filtering rules could change without anyone driving out to the beach to update a box on a pole. The alternative — filtering on the device and sending one "winning" tag — would have meant deciding with less information: the Pi had no permit list and no view of what had happened in the last few minutes, and a wrong choice made at the edge would be unrecoverable, because the other candidates would never reach the server. The cost of sending every candidate was a few more tag IDs per event over cellular, which was a small price for keeping the decision correctable.

A sensor reading tells you what the sensor saw. It doesn't tell you what happened.

2. Timing is a physical constraint, not just a timestamp

When a loop trips, the vehicle is in one specific place. The tag may have been read a few seconds earlier as the car approached, or it may not have been read yet. The camera has its own field of view, and the moment the car is centered in it isn't the moment the loop tripped. Getting the right picture and the right tag for the right vehicle meant choosing time windows: how long to wait after the loop trips before asking the reader for tags, and how far back that tag list should reach. Too short, and a slow car's tag gets missed. Too long, and the previous car's tag gets attached to this one.

There's no way to calculate those numbers from a spec sheet. The comment next to the reader delay in my code literally says to tinker with it once the machine is hooked up — and that's what it took: watching real vehicles at real speeds and adjusting until the events matched what actually happened at the gate. I also moved the camera from USB to a network camera, which the software triggered with a request over the network. That decoupled the camera from the Pi, but it also meant every capture had network latency of its own to account for in the timing.

Same vehicle, different moments time reader's tag window (seconds back from the trip) previous car's tag tag first read loop trips camera capture Illustrative only — the real windows came from watching vehicles in the field.
The tag read, the loop trip, and the camera capture are three different moments for the same vehicle. The software's job is to decide which readings belong together — and the right windows depend on vehicle speed and equipment placement, not on the code.

3. Humans don't behave like test cases

Then there were the rangers. A ranger sitting in a truck in front of the reader isn't doing anything wrong — that's where the job is. But to an RFID reader, a tagged vehicle parked within range is a tag that never goes away. It shows up in the tag list again and again, and it can end up attached to every vehicle that passes while it's there. No test plan written at a desk includes "a ranger parks here for an hour."

The fix was what I called triangulation, though it was really relative signal-strength comparison rather than calculating a position. The reader had four antennas, and comparing how strongly each one read the same tag gave the software a rough sense of where that tag was — in the lane crossing the loop, or parked off to the side in front of the reader. Reads that didn't fit a vehicle crossing the loop were filtered out. It was approximate by nature: signal strength shifts with reflections, tag orientation, and interference. But it didn't need to be precise. It only needed to tell a car driving through from a truck sitting still.

4. When every watt matters

A server in a data center has as much power as it asks for. A station on a beach running from a solar panel and a battery does not. Every component — the Pi, the RFID reader, the loop detectors, the camera, the modem — draws from the same finite budget, and a few cloudy days in a row shrink it fast. The station ran on a 100-watt solar panel charging a car battery. So the software managed power, not just data. The Pi switched the RFID reader, the loop detectors, and auxiliary equipment on only during operating hours, from 6 a.m. to 11 p.m., and cut them overnight when no one was entering the beach.

And every event carried the battery voltage and the station's temperature along with the vehicle data. That made the power system visible from the server: the data needed to see a battery trending down arrived with every event, instead of the first sign of trouble being a station that had gone silent. With those changes, the station could run for two days with no sun at all — up from the system I inherited, which barely lasted a single day.

The cloud doesn't care about your battery. Your hardware does.

5. Treat the network as a limited resource, not an assumption

The station's only link to the outside world was a cellular modem, with limited bandwidth and coverage that came and went. So the design assumed the network would fail. Events were tiny text payloads, not streams of data. They went into a queue on the Pi, and a separate background thread sent them, so a slow or dropped connection never blocked the loop that watched for vehicles. Requests used a 40-second timeout to tolerate a weak signal, and every network error was caught: a crash at the beach meant no data at all until someone drove out to restart it.

The same principle applied to startup. If the RFID reader was still initializing when the first car arrived, the event was still logged — with a marker saying the reader wasn't ready — rather than being dropped. Losing a vehicle entirely was always worse than recording one with incomplete information.

It's worth being honest about the delivery guarantees, because they were weaker than that sounds. In the version I deployed, the queue lived in memory, so anything waiting in it was lost if the Pi lost power. A request that failed was caught and logged so the program kept running — but that event wasn't retried. And success was judged by the request completing, not by the server confirming it had stored the event. For a first version on a single station, keeping the vehicle loop running mattered more than perfect delivery. If I were building it again, I'd write each event to local storage before sending, retry with backoff, give every event a unique ID so the server could safely discard duplicates, and delete an event locally only after the server acknowledged it.

6. The bug that wasn't in the code

The most memorable problem involved the cellular modem. The Pi turned the modem on through a relay: hold a control pin for just over a second, wait for the modem to boot, then bring up the cellular data connection, and repeat until the station could reach the internet. In the field, the modem wasn't coming on.

That raised a deceptively simple question: was this a software bug or a hardware bug? In a web application, you'd start with logs, application state, and network requests. Here, the problem could be anywhere along the chain from a line of Python to a physical action. Was the program sending the signal? Was the relay switching? Was the modem getting what it needed to start? The software was doing exactly what it was written to do, and logs couldn't prove that. What settled it was an oscilloscope: I measured the bus and found the problem in the hardware — an underpowered relay switching the modem. The command was correct, but the modem didn't come on.

It's the same problem distributed systems have, just in physical form. A service can send a request and never know whether the operation actually happened. Here, the software knew it had asked for the modem to turn on. It didn't know whether the modem actually did. The only reliable answer was to check the outcome — can we reach the network? — rather than trust the command.

A correct software command doesn't guarantee a successful physical operation. Sometimes the answer isn't another log statement or a retry. Sometimes it's a relay.

What the field taught me that tests couldn't

  • Define events by what physically happened — a loop trip — and treat sensor readings as evidence attached to them.
  • Send evidence, decide centrally — so filtering rules can change without a trip to the hardware.
  • Find timing windows by observation — vehicle speed and equipment placement set them, not the code.
  • Make power and health observable — battery voltage and temperature with every event.
  • Assume the network will fail — queue locally, send in the background, never crash — and make the queue durable.
  • Verify outcomes, not commands — the software asking for something isn't proof it happened.

Integration testing doesn't end at the API

A unit test could verify how the program parsed the reader's output — and I wrote one. An integration test could verify that the server accepted an event. But no automated test would have told me that a ranger's parked truck would pollute the tag list, that the camera saw the car a moment after the loop did, or that a relay couldn't switch the modem reliably. Those assumptions only failed in the field. So that's where the debugging happened: the loops were buried in the ground at the entrance, real cars triggered them, and I spent many hours sitting in the back of my pickup truck with a laptop connected to the Pi — in the middle of the greenhead flies — watching each vehicle cross the loop and comparing what I saw with what the software recorded. That's how the timing windows got tuned and how I checked whether the filtering was attaching the right tag to the right vehicle.

The outcome

The rewrite took the station from identifying vehicles correctly about 20% of the time to about 80%, and from barely lasting a day on its battery to running for multiple days — two full days with no sun. None of that came from a cleverer algorithm. It came from defining events by what physically happened, sending the evidence to the server, tuning timing against real traffic, managing power deliberately, and finding the problems that only showed up at the beach.

That 80% is an operational estimate, not a formal study. I arrived at it three ways: watching vehicles in person, reconciling the end-of-day totals, and reviewing the camera images. It was never going to reach 100%, because the real world kept intervening. On one day the count was off after lightning struck the beach and the system logged a trigger that wasn't a vehicle. No amount of code turns a lightning strike into a car — but knowing why each miss happened was what made the remaining errors something to explain rather than something to fear.

What I took away

Building the Duxbury Beach system taught me that the hardest engineering problems aren't always solved with better algorithms or cleaner abstractions. Sometimes they're solved by comparing signal strength across four antennas, understanding a power budget, watching how people actually behave, or discovering that a relay doesn't have enough power to do its job. The software is only one part of the system. The real work is making all the parts work together — and that's just as true for cloud services, payment systems, and healthcare integrations as it is for a box on a pole at the beach.

The hardest bugs weren't in the code. They were in the assumptions we made about the world the code was supposed to represent.

Engineering leaders who want to see more of the systems I've designed and built can find my engineering portfolio at josephrounds.dev.

Joseph Rounds

Founder, Lighthouse Consulting

25+ years building enterprise software at McKesson (Fortune 10), Doctor On Demand, and IntelyCare. Now helping Boston-area businesses design and build custom software, AWS infrastructure, and AI integrations that fit how they actually operate.