Fruit-picking robots

The UK, like many north European countries, depends on seasonal migrant workers to harvest many of its crops. It is estimated that 63,000 parttime agricultural workers are needed in the UK annually. At the beginning of 2022, the government had allocated 40,000 seasonal agricultural worker licences, but the Russia/ Ukraine war dealt a severe blow to the size of the labour force, with many of 2021’s fruit pickers coming from these two countries.

The shortage of visas and delays in visa-processing have resulted in farmers leaving hundreds of tonnes of asparagus and millions of bunches of spring onions unpicked. Many other crops have been lost and ploughed back into the soil, harming the nation’s self-sufficiency in food production.

Growers and researchers have been working hard to find robotic solutions to help cover some of the jobs that seasonal workers do. The pace of development has quickened so rapidly that most companies and observers involved feel that human–robot cost parity will soon be reached. Within two, three or four years commercially viable robots could be picking and packing some of the UK’s summer crops.

Why strawberries?

Many companies developing horticultural robotics have focused on harvesting strawberries. It is a crop that is universally popular and always in demand. However, many things can affect business success, including diseases, or problems with packing, transport and harvesting. Robotics could help with all these issues and, in turn, be applied to other agricultural crops.

In northern Europe, most strawberry growers employ a ‘tabletop’ system, whereas in southern Europe, Australia and the US, farmers favour traditional on-the-ground growing. With robots, a standardised approach to how the crops are grown may develop in the future, but for now, developers are designing robots to cope with different farm set ups.

Tabletop systems usually have plants set 1.5 metres high. Pickers can conveniently view and pick fruit without having to kneel or bend to the floor. Plants are in lined growbags so that the plant doesn’t touch the soil – being exposed to wet and moist soil for too long can lead to fungal diseases. Strawberries are so-called because straw used to be put under the plants to keep them clean and avoid soil splash, which can blemish the fruit.

Whereas most domestic growers buy June-bearing strawberries that produce one large harvest in late spring or early summer, many commercial growers use ‘everbearers’. Everbearing strawberries, cultivated in polytunnels or large greenhouses, can produce fruit from the end of May to the end of October. The UK’s prototype machine-learning picking robots already ply their trade during these months, with some then transferring to Australia to further develop their picking skills.

How do they work?

Most companies developing robots have started their journeys from scratch. They are each developing their own ways of enabling their robots to see, grip, evaluate, transport, and pack fruit. The businesses know that if they can master strawberry picking, which is tricky, then they should be able to adapt their methodology to other crops.

Dogtooth Technologies is one of the most advanced robot fruit picking companies in the UK, with 70 of its fourth-generation robots already picking tabletop strawberries commercially. Dr Duncan Robertson, the company’s Co-Founder and CEO, says that the fifth-generation robot will achieve the same picking costs as humans. It will be demonstrated to customers next year with first deliveries in 2024/25. Human pickers can pick 15 to 30 kilograms an hour at a cost of roughly 50p a kilo and commercially viable robotic picking must be comparably cost effective. At the moment, picking robots are slower and not as dexterous compared to humans, but they can work longer hours and at night.

The first challenge for developers is to enable the robots to model their three-dimensional environments. Dr Robertson first worked with Mat Cook, one of Dogtooth’s three co-founders, at Microsoft Research in Cambridge helping to develop Kinect, the motion sensing add-on for the Xbox 360 gaming console. Dr Robertson says that although depth-sensing cameras work well in some applications, they perform poorly outdoors. Instead, Dogtooth’s robot uses vision systems based on simple RGB colour cameras. Dogtooth, like others, has used deep learning neural networks to train its vision systems using data provided by human annotators. These machine-learning algorithms enable its robots to interpret their environments using images as input.

By using a stereo camera comprising two cameras set a few centimetres apart, Dogtooth’s robot can judge depth from 2D images, and therefore create detailed geometric 3D models of its surroundings. Dr Robertson says: “Our robot takes images of the growing strawberry plant and can understand what each pixel in the image means. It can distinguish individual strawberries and stalks as well as background elements such as leaves and polytunnel infrastructure. It can tell if a strawberry is sufficiently ripe, or if not, how long it is likely to take to ripen.” This ability to judge the health and amounts of future fruit is an important advantage that robots have over human pickers.

Dogtooth deliberately doesn’t rely on cloud computing resources for real-time picking. This avoids picking interruptions caused by network failure. It also reduces the data needed, and cost, for individual robots. It uses its AWS Cloud resource for higher latency functions such as processing the day’s data to inform the grower about expected future yields. Dr Robertson continues: “And when it spots a ripe strawberry, our system then directs the robot’s end effector [the device that picks] to accurately grip its stalk and cut it.”

Picking the fruit

Every company developing fruit picking robots seems to have devised a different end effector. Some are cup-shaped and embrace the fruit before breaking or cutting the stalk, others grip the fruit using soft ‘fingers’. Some have pincer-like extensions that grip and cut the stalk so as to avoid touching easily bruised soft fruits.

Dogtooth’s end effector comprises a pair of slender grippers and resembles a pair of tweezers. A separate cutting edge extends forward and cuts the stalk above the grippers leaving them holding the single strawberry by its stalk.

Some robots drop picked berries directly into a container, but Dogtooth’s carries them to a quality control system a few inches away where, under controlled lighting, the strawberry is scanned to detect 16 types of defects such as bruising, mould and asymmetry before being placed into a punnet or waste chute. Berries are weighed and punnets are filled until they reach the required retail weight.

Flexible farm robots

In July 2022, Simon Pearson, Professor of Agri-Food Technology Research at the University of Lincoln, co-chaired the report Review of Automation in Agriculture with George Eustice, who was then the UK government’s Environment Secretary. The report found that there was an active and emerging cluster of agri-robotic expertise funded by both private and public sector investment that will be able to replace some of the human harvesting roles on farms.

Simon’s own university team has made good progress in several areas of fruit picking. It has teamed up with Saga Robotics to run five proof-of- concept Thorvald robotic vehicles autonomously transporting strawberries from field to warehouse.

Saga already run lightweight Thorvald robots in the US, Norway and the UK to apply UV-C light to tabletop strawberries. This controls the powdery mildew that they are prone to, without using chemical fungicides. Looking like two-metre-tall sunbed arches, the robots go up and down rows of polytunnel strawberries once a week applying UV-C autonomously. Saga has found that this treatment works equally well for grapes. The company currently uses a dozen robots all year round at the 35-hectare Clock House Farm in Kent doing jobs such as spraying, crop forecasting and logistics.

Dogtooth has improved its fifth-generation robots by adding lithium-ion batteries that double the robots’ shift times from 8 to 16 hours. The new robot has been designed to be operated by customer personnel rather than Dogtooth’s. The company has also added onboard lighting to enable night picking and improve imaging conditions at tricky times of day, such as dusk and dawn.

Picking at night prolongs the berries’ shelf-life. Strawberries rapidly lose their freshness and quality if they spend more than an hour in warmth and sunlight after being picked. Picking at night reduces the temperature and light stress on fruit and maintains better internal and external quality with regards to sugars, colour and firmness.

The University of Lincoln team has focused on using the flexible Thorvald wheeled robots to fetch fruit from pickers and deliver them to the warehouse. Seasonal agricultural workers not only pick but spend nearly 20% of their time taking the fruit from its growing location to storage or packing facilities. The researchers have been working on programming autonomous navigation and safe interaction with pickers and other obstacles within a farm environment.

Autonomous packers

Packing is another area of horticulture where there are labour shortages and retention issues. When fruit and vegetables arrive at packing plants, they need to be sorted manually, graded, weighed, and then placed into packaging. The jobs are invariably repetitive, and the industry is challenged by high turnover and recruiting expenses.

In 2022, Wootzano Ltd developed its robotic system, Avarai, and started working with a leading pack-house corporation that supplies fresh produce to UK retailers. Dr Atif Syed is the Founder and CEO of Wootzano, and also received a Royal Academy of Engineering Princess Royal Silver Medal in 2022, as well as being a member of the Academy’s Enterprise Hub. His invention Wootzkin is a patented electronic skin made of modified elastomer, which is used by the Avarai robot. It contains metal electrodes that can be stretched without causing damage to the sensors within, measuring force and pressure.

Avarai is a collaborative robot (cobot) system that works alongside humans. It can estimate weight, trim fruits to size, identify defects, measure quality, pick, and gently place into punnets or other packaging.

Dr Syed says the Avarai cobot: “has been designed so that the relationship between the robot hand and the robot eye – the camera – is dynamic. So you can place the product anywhere and as long as Avarai can see it and reach it, it will, in real-time. It took 20-plus patents using algorithms and software to enable that, but it’s resulted in reduced cost on the hardware that ensures our customers get a product that is comparable in cost to human labour. There’s no need to reinvent the wheel, but sometimes you have to redesign it to make it more cost effective!” Wootzano is currently working on a commercial pilot line for packing vine tomatoes. By the end of 2023, Atif expects the Avarai robotic system to be fully commercialised for table grapes and vine tomatoes, and then scale up substantially thereafter to pack other fruit and vegetables.

Delivering agricultural robots

Leaders in the agricultural robotics field say that the optimism and excitement created by their rapid progress must be balanced with the reality that widespread adoption is a few years down the line.

Dr Robertson adds: “The UK has enormously innovative growers and they are enthusiastic early adopters of new technology. But we need the UK’s horticultural industry to be thriving in order to have the money to invest in robotic automation. Without adequate supply of seasonal agricultural workers that is impossible.”

In 2021, nearly 8,000 tonnes of berries worth £36 million were left to waste because of a shortage of pickers. As a response, some of the UK’s bigger producers have been developing farms in Senegal, Spain and Portugal so that they can continue supplying the UK market.

Professor Pearson says: “One of the reasons I undertook the government’s automation review with George Eustice MP was to ensure that the science evidence in this field is correct. Science and engineering have got a really important role to play in making sure that the facts are presented clearly so that we carry on getting investment in robotics, that we don’t lose farms in the process, and don’t cause massive disruption in supplies to shops.”

In 2020, UK farms produced about 2.5 million tonnes of vegetables and 650,000 tonnes of fruit. Together, these two agricultural strands generated nearly £3 billion worth of business. Agricultural robots have the potential to help grow these production levels by stepping into jobs that people don’t want to do. Exactly when the picking robots will match human costs is difficult to pin down, but Professor Pearson adds: “I’m asked that all the time, but I think it will be in three or four years.

***

Thanks to Professor Marc Hanheide, Dr Grzegorz Cielniak and Dr Amir Ghalamzan for their help with In-field logistics.

***

This article has been adapted from "Fruit pickers with potential", which originally appeared in the print edition of Ingenia 93 (December 2022).