The candy robot Candytron shows how AI can act in the physical world

This RISE project, featuring the robot known as Candytron, isn’t really about candy robots. Its aim is to build expertise in composite AI systems and to test the limits of what AI agents can do in practice. This enables RISE to help companies and organizations understand and use the technology.

When an AI model does more than generate text or answer questions, and actually affects the world around it by calling tools, for example by controlling physical components that hand out candy, it is called agentic AI, or an AI agent. The same principle is used in coding agents, where a language model writes and runs code to solve a problem.

Candytron consists of different systems working together as an AI agent. This is something that will become more common and more present in everyday life, according to Anders Holst, Senior Research Scientist at RISE, who developed Candytron together with Niclas Finne and Joakim Eriksson.

“People have been talking about robots for a long time, but we are not far from flexible robots that are more dynamic in their behavior and in the tasks they solve. Large language models make it possible to use natural language to make clear what you want. If we had a smart oven, for example, you could instruct it to defrost a chicken. The manual is built into the appliance, which itself knows what to do based on spoken instructions,” says Anders Holst.

How the robot sees, hears and acts

Different types of candy are laid out on a large sheet of white paper divided into squares. You tell the robot the name of the candy you want, and the robot picks up the piece and gives it to you. Behind the scenes it is more complicated, as Candytron is made up of several AI-based subsystems (see the table below).

A camera placed above the table takes pictures of the candy and sends them to an image model trained to recognize 13 different kinds of candy. The model returns coordinates and classes for each piece, which are then translated into a table, for example showing that a toffee is in square B4. This information is then sent to a multilingual language model, which acts as the core of the system and is what calls for the candy to be moved when a person asks for a piece. The robot is now ready to interact and hand out candy.

A person speaks into a microphone and the speech is transcribed. The language model receives both the transcription and the list of available candy, determines what the person wants and sends a command to the robot arm to move the right candy to the pickup point. The language model then generates a reply, which is read aloud by a speech synthesizer.

The first version of the robot required visitors to press a button to start talking. The new version adds a camera that detects when someone stands in front of the robot. The robot then automatically greets them, asks for their name and the conversation begins. For turn-taking in the dialogue, the system must also detect when the user has finished speaking. Using the camera, the robot memorizes what a person looks like in a facial recognition program, so if they come back it knows who they are and the voice can say something like, “Good to see you again. Would you like the same candy as last time?”

Candytron har rätt åtta av tio gånger

Siffrorna nedan avser den första versionen av Candytron från 2025 års öppet hus, då ungefär 300 interaktioner loggades. En genomgång efteråt visade att roboten gjorde rätt i 79 % av fallen. Av felen berodde 12 procentenheter på att taligenkänningen hörde fel i den bullriga miljön, 1 procentenhet på att kameran inte identifierade godiset korrekt, och 8 procentenheter på att språkmodellen resonerade fel. Systemet har en viss självkorrigerande förmåga, så om ett godisnamn missförstås kan roboten be om ett förtydligande och ofta nås rätt svar till slut ändå.

Everything happens locally, nothing leaves the laptop

All processing takes place locally on a laptop, with no data sent to cloud services, which means that images, voice data and any preference profiles never leave the device. The system also tested facial recognition to be able to greet returning visitors by name and remember their previous choices. All visitors were informed of this, and all collected information was deleted after the event.

The question of local versus cloud-based operation is expected to become increasingly important as more consumer products begin using cameras and AI to recognize and remember users.

“This is a question that is becoming more and more important. And as AI is increasingly integrated into our household appliances, being able to store that information locally becomes all the more pressing,” says Anders Holst.

From candy robot to real world applications

Candytron is built around the Model Context Protocol (MCP), an interface for communication between different agents and services. This makes the language model at the core generic, meaning that it doesn’t know from the start that it is controlling a candy robot. Instead, the robot functionality is plugged in as a separate module. The same architecture can be used to control other devices, such as smart lamps or household appliances.

This type of setup, where natural voice control is connected to a physical device, has many concrete applications. Examples include store robots that can answer customer questions and navigate to the right shelf, or household appliances controlled by natural speech instead of menus.

Candytron may hand out candy, but what it really shows is how different systems can work together through a unified interface in physical agents that can interact with people in a natural way.

Want to know more about Candytron and how this type of composite AI system can work?

Contact Anders Holst, Niclas Finne or Joakim Eriksson for help testing agentic solutions together with your company.