Amazon puts AI to work to ensure sale of good quality onions, tomatoes

E-commerce major Amazon is planning to use advanced computer vision, machine learning (ML) and artificial intelligence (AI) technologies to manage quality assurance for fruits, vegetables and other produce sold on its platform. 

In an interview with Mint, Rajeev Rastogi, vice president of machine learning (ML) at Amazon India, said the company has developed computer vision programs that recognize defects like cuts and scratches on tomatoes and onions to figure out when they have gone bad.

The system uses a mix of convolutional neural networks (CNNs), Visual Transformers (ViTs) algorithms. CNNs are deep learning algorithms that can take an image input and assign importance to various aspects of that image, while ViTs are specialized versions of transformer algorithms, which can weigh the significance of each part of data it gets.

Rajeev Rastogi, vice president of machine learning (ML) at Amazon India

“In our grocery business, produce quality is the single most important customer input and the number one driver of repeat purchase,” said Rastogi. “Currently quality is processed manually, which doesn’t really scale. It’s also very error prone, is costly and doesn’t have high repeatability. So, we developed a computer vision system for grading fresh produce quality by analysing images of produce,” he added.

The company uses computer vision techniques to detect cuts, cracks, pressure damages etc. The technology has been deployed for tomatoes and onions right now in Amazon’s store, but Rastogi said the company is in the process of building an artificial intelligence (AI) enabled machine, named Auto Grader, which will automatically grade produce moving on a conveyor belt. “It will improve the quality and consistency of the produce, while reducing the grading cost by almost 78% as compared to manual grading,” he said.

“In future, we also want to use infra-red sensors to detect attributes like sweetness and ripeness, which we cannot detect in RGB (red, green and blue) images that are captured by your traditional computer vision algorithms,” he added. He noted that infra-red can help avoid “destructive methods” of quality assurance, like how ripeness and sweetness often require a person to actually eat a fruit or vegetable. “A near infra-red image will be very different for a sweet versus not sweet product. Because the level of sugar will be different, and the near infra-red signature will be different.”

The system is deployed in stores in India right now, along with some pilot projects in Europe. Rastogi admitted that the system is at an early stage and will need more development and data. However, he said that the tech is at above 90% accuracy levels, in terms of precision in finding defects. “It varies from one produce to another. Like we have much better results for tomatoes, and we can do it with high precision. Our numbers are a bit lower for onions, and we are working on (improving) that.”

According to Rastogi, the level of difficulty in determining the quality of fresh produce can differ from one product to the next too. “Most defects that we are looking at (right now) can be visually detected by a human. Anything that can be visually detected, I think can be easily detected by computer vision as well,” he added.

In theory, Amazon could also use a system like this to classify and categorize products, so as to recognize the kind of product a seller is offering. For instance, if a seller is offering a particular type of produce — say varieties of mangoes — the system could categorize what that mango should look like and figure out in advance whether the right product is being delivered.

The company has also had to tailor the system to Indian produce quality. Since the quality of fresh produce in India isn’t the same as that in western countries, the approach to determining quality has to change as well. “A tomato in India is not the same as a tomato in the US. So, you can’t just say that you will have the same approach to dealing with tomatoes and onions in every country. The nature of the product could be quite different, because of farming practices, etc.”

Further, technology like this could also be used on the front end, where customers could just open the Amazon app and detect the quality of a vegetable or fruit by themselves. However, Rastogi noted that these are very advanced applications of AI, so he couldn’t provide a timeline for full scale rollout.

“They are what we call seed initiatives in my organization, which are early bets and are exploratory in nature. They have high science risk and technical risk associated with them. But if they worked out, they could be fairly impactful,” he said. Amazon is also working on an assistant, which is effectively a chatbot specialized for shopping use cases, and will assist customers in making buying decisions by asking questions a human operator would.