Has AI Created a New Kind of Cyber Supervillain?

Has AI really introduced an entirely new breed of cyber threats? 

“We haven't seen convincing evidence of AI superhackers inventing entirely new forms of cyberattacks. But you could say that AI has given familiar threats superpowers. 

What has really changed is the speed and scale. Attackers still rely on well-known techniques such as phishing, stolen credentials and software vulnerabilities. But AI makes attacks faster, cheaper and more convincing. Phishing emails now arrive in flawless Swedish, and much of the work involved in identifying potential targets can be automated.” 

What does an AI-powered attack actually look like? 

“It might not look suspicious at all. Attackers can use legitimate tools, valid credentials and patterns of behaviour that closely resemble everyday work. 

A striking example is the 2024 incident at engineering firm Arup, where an employee was tricked into transferring the equivalent of $25 million following a video call with what appeared to be the company's CFO and colleagues. In reality, the people on the call were deepfakes. 

Many security systems are designed to detect unusual activity. But what happens when an attack looks perfectly normal?” 

If an organisation could do just one thing to improve its security, what should it be? 

“Make sure that no single email, phone call, video meeting or login is enough to gain access to sensitive systems or authorise large payments. 

In practice, that means using phishing-resistant authentication, such as passkeys, and always verifying sensitive requests through a separate channel. This applies to payments, password resets and changes to suppliers' bank details, for example. 

As AI makes fraud increasingly convincing, we can no longer rely on people being able to spot every deception.” 

How can we use AI to strengthen our defences without introducing new vulnerabilities? 

“Test it first! Try to trick the system, overwhelm it and feed it misleading information, just as an attacker would. 

Then treat your AI system like a new employee with privileged access. Give it only the permissions it needs, keep a record of what it does and require human approval for decisions that cannot be reversed. 

And don't forget to protect the AI system itself. Its accounts, access permissions and keys can all become targets.” 

So, which of your security procedures would hold up if your CEO called tomorrow, but it wasn't really your CEO? 

AI and Cybersecurity at RISE 

At RISE, research in AI, cybersecurity and resilient digital systems comes together. Our researchers work on everything from understanding and preventing cyber threats to developing methods for testing and strengthening the security of AI-based systems. 

Through research, collaboration with industry and the public sector, and testing and demonstration environments such as Cyber Range, RISE helps organisations turn knowledge of digital threats and vulnerabilities into practical security capabilities. This includes hands-on training and exercises in realistic environments, where organisations can put their resilience to the test and improve their readiness for cyberattacks. 

For more insights into AI-driven cyber threats and how to defend against them, read When AI is behind cyberattacks, the defence forces must keep up featuring Apostolos Pyrgelis and his colleagues at RISE.