Real Time Motion Plotting
by Alec Deitloff and Cameron Franke
With the rollout last summer of our Floor Plans feature, Verkada gave customers a more intuitive option for navigating and viewing systems with cameras across multiple locations. From a single bird’s-eye view, it allows users to access any individual live feed with just one click. That was an important step in and of itself, but we also knew it could be a foundation for other features to come — features that could make it even easier, and much faster, to understand what’s happening in a space. What if you could gather that information not with a few clicks, but one simple glance?
So a few months ago, we started experimenting with options for adding a live “heat map” to Floor Plan that would show any motion in real time. After about a week of tinkering, we had an ugly-but-viable proof of concept, and Rob Cromwell, our VP of Engineering, gave us the green light to make it happen. With Cameron leading the computer vision efforts and Alec handling the frontend, we recruited Jason Bao for backend support, and the project — code-named “Marauder’s Map” in honor of Harry Potter — was off and running.
Setting the Scene
The first step was calibration — and we realized quickly that process would require people. To ensure the system showed the location of motion accurately, users would have to match a series of points from a camera feed to the corresponding points on Floor Plan. They only had to do it once for each camera, but because some users have dozens and even hundreds of cameras, it had to be as quick and easy as possible.
At first, we tried calibrating with just eight pairs of points. But when we tested that by asking our coworkers to run through a set, we found the margin of error was too large. If even a couple of points were off, the results were pretty ugly. So we started adding more pairs, testing all the way up to 20 per camera. Eventually, we landed on 15, a number resilient enough to produce an accurate floor plan even when we breezed through calibration, but without asking for any more work than necessary from our users.
We also decided to give our users people, rather than objects or points on the floor, to locate in each space. There were two reasons for this: First, it improved the calibration results, because the human mind is particularly good at identifying where other people are. Second, it allowed us to leverage some of our existing features. Because Verkada runs in the cloud, we can power much of what we do with machine learning — and our cameras have already learned a lot about detecting people.
“Because Verkada runs in the cloud, we can power much of what we do with machine learning”
This decision did create its own challenge, however. If you ask someone to point to a person, their finger or cursor will probably go to the torso, not the feet — which is a problem when you’re trying to locate the spot on the floor where the person is standing. But here again, Verkada’s existing tools came in handy. By running our “pose extractor” neural network over the calibration images, we were able to map the joints of each person and draw a box around them before presenting them to users — so clicking anywhere in that box would give us the correct location on the floor.
Fine-Tuning Floor Plan
But even with calibration addressed, perspective was still a challenge. We needed to figure out how to correctly warp and stretch the 3D world our cameras saw and represent it on a single 2D grid.
Below, Figure 1 illustrates part of the problem: A single cell in a camera’s grid might cover a very small area if it’s located close to the lens, or a very large area if it’s far from the lens. Thus, a warning about, for example, “activity in cell 22” might indicate motion at any one of four or more workstations.
To address this, we decided to overlay a second visualization grid — which we now call the “master grid” (Figure 2). It still transforms each camera’s trapezoidal visual range onto the rectangular Floor Plan, but provides users with more fine-grain information about those far-away areas.
Building in Boundaries
But with those perspective issues largely resolved, we ran into another significant snag: Our cameras appeared to be seeing through walls! Because they didn’t have information on the visual boundaries of the space, they were “stretching out,” or mis-locating, the motion they did see.
As with calibration, we realized solving this challenge would require input from our users. Though it’s something we hope to automate in the future, in the short term, we needed people to tell the system where walls are located. This required a whole new round of infrastructure development, building out a tool where users provide this info. Then, we used a ray-casting algorithm to figure out what a camera could actually see, given the walls and other obstacles (Figure 3) and limit what it reports to that space.
Thinking Through Trade-Offs
Once we’d nailed down the functions of Motion Plotting on Floor Plans, it was time to fine-tune it for use in the real world. We wanted our “Marauder’s Map” to light up as close to real time as reasonably possible, in order to avoid lag time. If a user has a camera’s live feed pulled up on Floor Plan and someone walks in front of it, we want them to see that motion represented in both windows simultaneously, or close to it.
But with each camera reporting motion to the server roughly four times per second, we also had to pay close attention to trade-offs — especially given that a single Floor Plan might represent information from 20 cameras or more. If we updated Motion Plotting at every possible opportunity, we would tank performance. So we experimented, deciding exactly how “live” we wanted live to be and eventually landing on a rate that minimized traffic while still offering a useful, intuitive visualization.
The Future of Floor Plans
Now that Motion Plotting on Floor Plans has shipped to our customers, we’re looking to the future. It’s already exciting to be offering such a dense, rich representation of a space. But just as the early Floor Plans laid the foundation for this project, we believe Motion Plotting has now unlocked the next level. We’re looking forward to tying these features in with some of our other systems, such as identifying people, and making the most of Verkada’s cloud-based infrastructure to explore yet more complex processing. This is the start of years’ worth of projects — and only the beginning of what Floor Plans can become.
Interested in joining the Verkada team?
Check out open roles or email questions to recruiting@verkada.com
