Amazon Go – IoT to the 4th Degree

Amazon GoAmazon opened its check-out free Amazon Go convenience store to the general public last week. This high-tech low-touch store is a busy urban introvert’s dream. The opening triggered a second wave of media coverage that included descriptions of user experiences and a number of possible explanations about how Amazon is able to manage this high-tech customer experience. Amazon states on its website that this shopping experience is made possible by the same types of technologies used in self-driving cars. But I think it would be more accurate (but a less effective marketing message) to say that Amazon Go is enabled by the same technologies used in today’s high-tech warehouses.

How Does Amazon Go Do It?

Amazon isn’t providing many details about how it manages the checkout-free shopping process, but they do provide some insights into the technology elements at work, particularly the data capture tools. Amazon’s website states that Amazon Go utilizes computer vision technology and sensor fusion methods.

Customers must download a mobile application to their smartphones prior to becoming a store customer. This application facilitates payment and likely manages shopper identity and location while in the store. High-tech turnstiles manage the customer entry and exit processes. Sensors on product shelving monitor the movement of inventory as it is placed on and taken off the shelves. And perhaps most importantly, cameras capture images that can be used to monitor activity and make sense of data from other sensors used in the store. It is this pulling together, or “synchronization” of the data from various sources that appears to be the secret sauce in the Amazon Go recipe. I see similarities between the Amazon Go data synchronization process and PINC Solutions’ yard management system. PINC’s technology synchronizes RFID tag reads with location and time. A tag read alone simply confirms the presence of an inventory item. But the combination of tag read, location, and time allows the system to intelligently manage inventory, its location, and its change in status.  I suspect that time stamps play a critical role in Amazon’s data synchronization process as well.

Computer Vision and Analytics at Work

When I first read articles about Amazon Go, I could envision how data such as inventory sensor, shopper/smartphone location, and change in inventory location or status could be synchronized to match product movement, and ultimately sales, with individual shoppers. However, this would require substantial assumptions (just because a person is next to a sandwich when it was picked doesn’t mean they bought it) and would likely provide insufficient transaction accuracy. After all, a few bad experiences could undermine the marketing power of the Amazon Go concept.

To obtain some expert opinions about how Amazon Go likely achieves accuracy and reliability, I reached out to Bernd Stoeger at Knapp, and Brendan McKenna at Zebra Technologies. Knapp developed KiSoft Vision, an image recognition technology that is applied to process improvement in distribution and manufacturing operations. So I knew he would have first-hand insights about the application of computer vision technology. He stated that the computer vision, sensor, and location technology may be deployed at Amazon go with an overlap in capabilities and serve to double-check the results of the multiple sensors. This overlap can serve to improve upon reliability and accuracy. For example, the image analysis may be able to confirm the simultaneous presence of a shopper and a product pick to validate the association between customer and item. However, this is not a trivial process to automate. There is a great deal of intelligence that must be embedded in the image processing technology to recognize and distinguish items and make associations in near real-time within a dynamic environment.   The accurate and reliable association of an item to a shopper, and ultimately the purchase, is what Bernd views as a major complexity within the process.

Brendan McKenna at Zebra also provided me with his perspective on the likely technology in use at Amazon Go. Brendan drew upon his knowledge from the development of Zebra’s SmartLens technology. SmartLens uses multiple sensing technologies to make associations and ultimately deliver exceptional in-store inventory accuracy for retailers. The solution uses RFID, video, and micro-locationing. Zebra’s micro-locationing is a proprietary system that leverages ultrasonics. But Brendan from Zebra and Bernd from Knapp both stated that Amazon Go may leverage other micro-locationing sensors embedded in customers’ smartphones to obtain high levels of in store shopper location accuracy (better than GPS). Brendan informed me about the way in which multiple sensor technologies complement each other and improve upon reliability and accuracy. For example, video clearly requires line of sight, but Zebra’s ultrasonic micro-locationing doesn’t necessarily need line of sight to provide its data. He indicated that the analysis of the data streams and the correct association of those data streams require substantial programming logic and heuristics. At the same time, there is such a large magnitude of data that there is a need to filter out the unimportant to enable the analysis of that which is relevant.

I wish Amazon Go was located in Boston so I could easily visit the store and experience it for myself. Unfortunately, it is located in proximity to Amazon’s headquarters in Seattle, 3,000 miles from my home. But I haven’t given up hope yet. After all, Boston is still in the running for Amazon HQ2 – Amazon’s second headquarters in North America.