Jump directly to content

Can evidence from the data center make it easier to trust AI services?

AI users want to know how their sensitive data is handled. The provider wants to protect its model and safeguard trade secrets. Is it possible to prove that what the provider says its AI does is actually true? RISE is investigating this in its research on verifiable AI.

Trust in AI is a hot-button issue. On the one hand, the use of generative AI is growing rapidly in society. On the other hand, concern and skepticism are also on the rise, according to research from the University of Gothenburg. As many as 62 percent of Swedes view AI as a risk rather than an opportunity, and 54 percent regard the technology as a threat to humanity.

A key aspect of the issue of trust concerns the level of transparency we have when using AI. Today, that transparency is extremely limited.

“It’s very difficult for someone outside the system to see and understand what an AI is doing. If you run the data center where the AI is running, you might be able to inspect what’s happening on your own machines, but as a user of the AI model, you have no insight,” says Simon Bouget, a senior researcher specializing in cybersecurity and connected systems at RISE.

Looking for more than just answers from AI

He is part of the RISE project “Verifiable AI,” which focuses specifically on creating transparency into what happens in our data centers. The project took shape when a startup whose goal was to build and sell tools to increase transparency in AI services contacted RISE. The company wanted help improving the security of its tools to ensure that they provide reliable information, that an AI service provider cannot deceive the transparency tools, and that the tools themselves cannot leak sensitive information.

The collaboration shed light on an issue that concerns all companies that develop, operate, or use AI services. How can they monitor what is actually happening behind the scenes when an AI model is running?

Typically, every question posed to AI is a request for a computation in a data center. In the context of “Verifiable AI,” the answer to this request must be supplemented with background information, such as the model’s identity and the amount of resources used in terms of energy and computation time. The hope is that the power imbalance between AI providers and regulatory authorities can be mitigated to some extent when processes, computations, or decisions can be traced and verified retrospectively.

As the technology matures and advances further, it will become a tool for Swedish industry to offer digital services with better privacy protection and stronger security guarantees.

Simon Bouget, senior researcher at RISE

Is it like opening the “black box”?

– To a certain extent. It’s not about examining how AI works from a purely technical standpoint and fully understanding how it reasons. But we’re opening a “black box” in the sense that when we send a prompt to an AI, we want more than just the result presented to us.

"We want certain metadata and strong assurances that this metadata is accurate," says Simon Bouget.

The use of AI involves several stakeholders. It is the model provider—who created and trained the model—who does not want that asset to be leaked or used without their approval and therefore keeps the models private and closed off from public scrutiny.

“On the other hand, you have the end users who send requests to the model and receive responses. End users sometimes share sensitive and private information and want their privacy to be respected. So both of these parties have concerns, and that’s why they need to be able to trust each other,” explains Simon Bouget.

In addition, the model provider and the end user need to trust the data center operator to be resilient against cyber threats and to protect both parties’ sensitive data.

– The parties involved don’t really trust each other these days. For the end user in particular, it’s a very unequal relationship. When we use digital services today, most of us click “accept terms of service” without reading through the terms. There’s no room for negotiation there—either you accept them or you can’t use the service. I wouldn’t call that trust; it’s more like blackmail,” says Simon Bouget.

A key concept in “Verifiable AI” is guarantees. This does not refer to clauses in a contract, but rather to technical guarantees in the form of concrete evidence and data that prove an AI model is operating as intended, without the data being manipulated or a different, cheaper, or simpler variant being used instead.

“Here, we want to use cryptography (mathematical methods for protecting and verifying information—Ed.) and traditional cybersecurity techniques in new ways, tailored to AI systems,” he says.

What can be verified today, and what is still the subject of research?

– To put it bluntly, nothing can be verified today, and that’s why we have this project at RISE. Many individual links already exist, but we need to build an entire chain. That’s the complicated part. The most obvious first step is to verify the identity of the model being used. This involves checking that a file has not been altered, which can be done using traditional digital signatures, integrity checks, and similar methods.

At the other end of the spectrum is a somewhat more ambitious goal: verifying what kind of calculations—or what type of task—the model is performing. Researchers at RISE are looking for ways to determine what a model is working on, in order to block tasks that should not be allowed. This could involve the development of biological weapons, for example.

“As the technology matures and develops further, it will become a tool for Swedish industry to offer digital services with better privacy protection and stronger security guarantees,” says Simon Bouget, adding:

– And if we choose to be optimistic, this type of research could become a building block for an international agreement in which all parties agree to use verifiable AI solutions to monitor one another.

Simon Bouget

Senior Researcher
+46 10 228 43 90 Read more about Simon

Contact Simon

CAPTCHA

* Mandatory 

By submitting the form, RISE will process your personal data.