College of Engineering Unit:
Autonomously logging cafeteria food consumption provides the opportunity to reduce both malnutrition and food waste. According to the Agency for Healthcare Research and Quality, malnutrition in the United States affects more than 30% of hospitalized patients. As for food waste, in the United States, the USDA estimates that 31% of the available food supply went uneaten in 2010, which represents 133 billion pounds of food. Imagine an automated food computer vision system that notifies a nurse when a patient is at risk for malnutrition. Or imagine a system that uses food waste monitoring and makes suggestions to the head chef to alter their menu. In this project, I lay the foundation for this vision.
I first prepared a dataset of American cafeteria food images designed for machine learning by merging subsets of two publicly available international food datasets. I then trained a state-of-the-art object-detection model (YOLOR) on the American food dataset and achieved 91% overall accuracy (0.917 mAP@0.5). While some models only detect a singular type of food an image contains, my model localizes multiple foods across the image, and then classifies each of them. The results demonstrate aggregating images from disparate datasets as an effective method for developing a food detection model with high accuracy.
While working with publicly available online datasets worked for proof of concept, collecting our own data will be necessary for building a robust system. Students in my lab constructed a food detection device that will take pictures and record the weight of food trays at cafeterias. To process the newly collected data, I developed a server with Python3’s FastAPI to receive images and metadata from the device and store it in a MySQL database. The server not only allows for data collection needed for the development of more models, but it also detects foods in images taken by the device. Ultimately, the dataset, model, and data pipeline create a baseline for future work that we hope can be used to track nutritional intake in hospitals, schools, and prisons to help decrease malnutrition in vulnerable populations.
Project Communication Piece(s):
|Poster explaining Matthew's research.||1.18 MB|