Quality estimation of food grains using Computer Vision!

Computer Vision is perhaps one of the hottest fields right now. It is finding applications in reading text, reading signs, recognizing objects, people, understanding and segmenting scenes, and so many other interesting things. One interesting area where Computer Vision has a huge potential is the field of Agriculture. Agricultural supply chains have so many redundancies that multiple parts need to be automated. Many of these parts can be automized using Computer Vision, like seed identification, crop disease identification, drone surveillance for yield estimation. These examples are pre-harvest problems, but even after the grains are harvested, there is an interesting application of Computer Vision with enormous potential.

After the grains are harvested, a farmer goes to an APMC mandi (common market place in India, where farmers come to sell their crops) to sell it to some trader. If all the steps are properly followed, a sample from various sacks brought by farmer is taken to a lab, where a technician would manually separate the sample into various categories defined by govt. parameters, and then he assigns a final quality to the produce.

Commodity Parameters for Wheat. Our focus is categories in A (Essential). Link: https://enam.gov.in/web/commodity/commodity-quality

This process of quality estimation is called assaying. Manual assaying of a 50g sample takes more than 15mins. And the number of samples which come at a big mandi during peak seasons count in thousands. Hence manually assaying all the samples is impractical, most of the times it is not done, and farmers don’t get to know the quality of their produce, because of which they aren’t able to bargain. While there are many other factors also like lack of storage for farmers’ produce, formation of cartels among traders, etc. yet a transparent and fast quality estimation system is required, because it will give at least some bargaining power to the farmers, and similarly it will give some assurance to the traders and they won’t have to physically examine each lot. This is why assaying has to be automized, and this is where Computer Vision is required.

I was introduced to this problem back in 2016 or so by Mr.Gaurav Agrawal, an IAS officer, and I took it up as my course project for Digital Image Processing course, which was being taken by my Bachelors advisor, Prof. Tanaya Guha. By the end of that project we were able to develop a POC in which we were able to distinguish between full grains and foreign matter (only two categories) using simple image processing techniques, in constrained lighting. After that I left it there, and graduated to join Advanced Technology Labs in Samsung-Bangalore in 2017.

However I was still curious about this problem, and about how much can I solve it. So two more people joined me (one of them being my flatmate so it was easy to convince him :p) and we pitched this problem in the C-Lab Entrepreneurial Challenge. They liked the idea, so we won (2 winners out of around 240 ideas!) and eventually we were converted into a 3-member team under Creative-Lab (C-Lab) at SRIB. We explored this idea for around 13 months. However, there are so many nuances to this problem that a single Software Center alone cannot solve it, and it requires a truly collaborative effort from community, government, as well as tech companies. Hence, SRIB permitted us to open up our work in the form of a research paper in BMVC’19 [Link here].

From here on, I will discuss the content of the paper which describes our approach. I will strictly stick to the paper because revealing anything extra would be a violation of my agreement with the company. Essentially, this blog can be seen as an easier description of that paper.

So, our setup is like this: Take a bright green A4 sheet, and spread the sample of food grains over it such that the grains may be touching each other, but not overlapping i.e. there is just a single layer of grains on the paper (actually best case would be that none of the grains are touching each other, but when you have 200–300 grains, and you try to spread them such that none is touching any other, you will have a very hard time doing that :P). Now, click its image using your smartphone, such that the sheet comes inside the screen completely. Then our engine will detect the sheet, apply perspective correction on it, and then it will feed it to our Computer Vision engine.

An image of the sheet is taken, then the sheet is detected in the image which is followed by perspective correction of the image.

So the final output of our network on a perspective corrected test image looks like this:

Output of Computer Vision engine on a sample image. Color coding: Navy Blue — Full, Red — Foreign Matter(Organic/Inorganic), Orange — Weevilled, Yellow — Broken/Immature/Shrivelled, Magenta — Other Grains, Cyan — Damaged.

Interesting results right?! As you might have guessed, the Vision engine essentially performs instance-segmentation of the food-grains, followed by classification. There is lot of Deep Learning and lot of classical image processing as well.

Overall pipeline of our approach. Images of samples of grains belonging to different categories spread in a non-touching manner are collected, and then U-Net for Foreground Background Separation (U-Net-FBS) is used to crop out particles from the green background. These particles are used to train a boundary aware CNN, which we call U-Net for Instance Segmentation (U-Net-IS) and the MobileNetV2 classifier, both of which together form the inference engine.

If you would like to know our training strategy or other details, please feel free to refer the paper! Or if you really want to know something and don’t want to read the full paper, just comment below I will answer! :D. Also, did I mention that the entire engine is running ‘on-the-phone’ i.e. the entire vision engine (including the neural networks) runs on the mobile phone itself. No need of cloud or internet connection for processing. While manual assaying takes around 15–20 mins, our system takes less than 100secs.

So can the current system be deployed in the field? Not yet! It still needs to overcome a number of challenges to be reliably used in the field. First challenge is estimation of weight. If you see the parameters list in first figure, the percentages of defects are by weight, while we are giving percentages by count. So incorporating the weights of the grains is a major problem. Another problem is the scalability. Presently our model is trained on Wheat from very few areas, but to make this solution truely scalable, samples has to be collected from all parts of India (or world), and also for all the different commodities. There is a third problem also that it doesn’t work properly in non-white lighting conditions, and in very dim lighting, but this issue is low priority, since arranging bright white LED light in the mandis is not too difficult.

So we are opening up our research to the community as this paper. People can take up our work (which is easy to replicate since we are using standard CNNs and standard algorithms) and train their own models for various commodities. Some other people may also try to solve some of the challenges we faced, and some others maybe try to convince government to launch a nation wide drive to collect labelled data across the nation to create a big dataset, which can then be opensourced for all agrotech startups to use. There are already few startups like Nebulaa Innovations, Intello-Labs, Agrix-Labs, etc which are working tirelessly in automating the assaying process.

📝 Read this story later in Journal.

👩‍💻 Wake up every Sunday morning to the week’s most noteworthy stories in Tech waiting in your inbox. Read the Noteworthy in Tech newsletter.