MSU scientists use $1M grant to ‘get a grip’ on automated blackberry harvesting

Contact: Meg Henderson

STARKVILLE, Miss.-The labor-intensive work of harvesting delicate blackberries by hand is a must, but the development of advanced technologies by Mississippi State scientists could help automate the tedious process.

MAFES scientists at Mississippi State University, collaborating with Georgia Tech University and the University of Arkansas, have developed a robotic blackberry picker. (Photo by Anthony Gunderman, Georgia Tech University and University of Arkansas System Division of Agriculture)

Many agricultural crops are picked quickly by machines, and MSU Assistant Professor Xin Zhang, of the Department of Agricultural and Biological Engineering, is working with a university team to do the same for ripe blackberries-taking this high-value specialty crop from special handling to robotic harvesting.

In the university's Mississippi Agricultural and Forestry Experiment Station, Zhang and her team are developing a blackberry detection and localization system, the "eyes" and "brain" of a robotic harvester system powered by an innovative, artificial intelligence-driven deep learning approach.

Zhang is co-principal investigator on a $1 million multi-institutional effort funded by the USDA National Institute of Food and Agriculture National Robotics Initiative 3.0 (NRI-3.0) program in collaboration with the National Science Foundation.

As the MSU team develops this critical component of the automated harvester, partners at Georgia Institute of Technology are working on a soft touch robotic arm and gripper and a bipedal mobile platform to work hand-in-glove with the MSU-trained perception system. The prototype gripper is equipped with sensors located at the ends-like tiny fingertips-allowing it to grasp and pick the berry without squeezing and damaging it. University of Arkansas scientists are focused on post-harvesting fruit analysis.

The perception system designed by Zhang and her team is powered by YOLOv8 (You Only Look Once, version 8), a vision-based object detection model that identifies and locates objects of interest-in this case, ripe blackberries-quickly and accurately. This kind of technology is powerful enough to support robots, surveillance systems and self-driving cars.

The team trained a series of YOLO models to not only identify each blackberry on a bush but also detect its level of ripeness, from ripe (black in color), to ripening (red) and unripe (green). The process utilized over 1,000 images from plant canopies in various Arkansas commercial orchards, preparing the system to simultaneously single out ripe berries for harvest and keep track of others in preparation for the next round of picking.

"Our main focus is detecting the ripe berries for the robotic harvester, but we added the other two categories to provide a total berry count," Zhang said. "This way, the system not only identifies harvest-ready berries but also helps provide growers estimates of their total harvest."

During tests with multiple configurations and variants, the best-performing model was 94% accurate in identifying ripe berries, 91% for ripening berries and 88% for unripe berries. It also detected impressively high-resolution images in real time, clocking 21.5 milliseconds per image.

"The perception system identifies the berry and sends out 3D coordinates, including distance, to the robotic arm, which uses that feedback to reach out and pick the berry," Zhang said. "It is critical that our perception system communicates quickly and accurately with the arm and gripper system."

Aside from their contributions to the harvester, Zhang and her team are beginning to develop a mobile app based on their image detection system.

"The app is a separate project, but it would give growers a quick and easy way to forecast their total harvest at the beginning of the harvest season so they can swiftly adjust their marketing strategy," she said.

To learn more about the Mississippi Agricultural and Forestry Experiment Station, visit www.mafes.msstate.edu.

Mississippi State University is taking care of what matters. Learn more at www.msstate.edu.

Thursday, August 1, 2024 - 4:06 pm