Harder, Better, Faster, Stronger: Can Businesses Square The AI Security Circle?

Since the arrival in the commercial world of large-language-model chatbots and generative so-called “AI” tools, there has been a sense that normally pragmatic cybersecurity leaders have only just been managing to keep a lid on their true feelings. And these, it seems, are closer to hair-on-fire all-out panic than anything more measured or temperate.

Under sustained pressure from all sides – from business leaders, racing headlong into wider adoption of genAI as they fear ceding marketplace advantage to rival firms; from users, expecting ever-easier, ever-quicker, ever-more-streamlined interactions with the business; and from development teams and rank-and-file staff, champing at the bit to use LLMs to be able to do more work, faster – many CISOs, BISOs and other senior security leaders have often seemed to seek refuge in gallows humour. Wry smiles, mirthless chuckles and the occasional eye-roll have become a kind of shorthand. ‘What can we do?’ they seem to be saying. ‘We’ve warned them about all the dangers, but they refuse to listen.’

And yet, during a RANT roundtable held in Edinburgh at the end of April, and convened by the security vendor Check Point – a firm with a background in next-generation firewalls, but which has a well-developed AI dimension to its offering as well as an acute interest in helping clients secure their business data amid widening genAI adoption – there were signs that the worst fears of many specialists may be starting to recede. While nobody in the room was showing any signs of complacency, there was no widespread sense of fatalism or futility. Instead, leader after leader related experiences inside their businesses that suggested that boards are at least starting to listen to what their CISOs are telling them. And, while nobody felt certain that they were completely on top of the challenges posed by genAI tools, most businesses, regardless of size or industry vertical, had acknowledged that those challenges were real, and have begun to put systems and protocols in place to try to minimise the risks posed.

Around The World

The first indication of this new-found confidence came early in the discussion, with several security leaders describing corporate efforts to get ahead of the problems. A handful of attendees spoke about the working groups their companies had established, usually involving leaders from across the business who would meet to discuss all aspects of their internal AI transformation, including security.

   “We’re working to make sure the AI agents have the same login controls and the same contexts, the same prompting framework,” one CISO said. They said their firm’s AI working group had set itself the mission of “making sure that, every time [an AI agent] generates a piece of code, it’s secure regardless of whether the user asked it to be or not.” Their solution, so far, involves checking LLM chat metadata, and periodically sampling the outputs. The process, they said, was akin to “checking the AI’s homework.”

“That’s similar to what we’re doing,” another senior security leader said. “We give access to things like Copilot chat, but we don’t want Copilot to get its fingers into everything. We look at the prompts, and look at things that shouldn’t be going out – passwords, social-security numbers, HR information. So far we’re happy with what we’re seeing.”

Of course, there are limits to what degree controls and monitoring of this type will help. But it was telling that, when the counter-argument was put, the way the attendee chose to phrase their caveats was from a perspective of acknowledging inherent limitations, rather than worrying that the entire concept was structurally flawed and that all efforts to secure genAI tools were doomed to fail.

   “Our CEO is pushing for everyone to use AI, and we don’t have many controls. That’s a big problem,” they said. “The business is doing things that are sort-of controlled, but we don’t get any logs from it.”

And there was at least one glimmer of the old fatalism – at least, if we assume that the senior leader speaking did so with at least a gleam in their eye and more than a modicum of cynicism in what could be taken, at face value, as categoric certainty that there would be no problems. They said every employee was allowed to access Gemini, but every other AI tool was blocked.

   “You can do whatever you like with Gemini, and ostensibly it stays within our walled garden,” they said. “The contract says so. So I’m 100 per cent comfortable with that.”

Get Lucky

Summing up these positive early exchanges, RANT’s host for the event – professor of systems and software security and 2024’s UK Cyber Citizen of the Year, Rob Black – sounded relieved to hear that, as he put it, “we feel we’ve got strategies in place; we don’t have our heads in the sand. But,” he noted, “I’ve been in other conversations about it that haven’t felt as mature as this. Is anyone going, ‘Oh shit’?”

 “We do have a strategy, but we also say, ‘Oh shit’,” one leader replied drily.

“I’m grimacing,” another CISO said, with just such an expression on their face, “because we started off very tightly controlled, with AI governance and all of that. Now we’ve moved into a space of, ‘We need to open this up to the end user to develop their own use cases.’ We’re using Copilot, and the danger there is that someone develops a workflow that then becomes business-critical, but nobody understands how they created it. Did they put the right data into it? Where’s the governance around it? Nobody’s really checking that.”

“The crimes of the past are coming back to haunt you!” another leader said, also with what could well have been a grimace. They explained that they were referring to how genAI tools break a lot of established access-management processes and protocols. “Organisations that move faster are using AI because it’s the answer to everything,” they continued. “They need to sort a table, so they use AI! AI bots are in front of everything. In another organisation I know of, you can’t even talk to a human. All the barriers are gone.”

“As we were growing, the access provided to the agents was through old-fashioned mechanisms,” another CISO recalled. “You had access to a broad range of things they could change. The explosion of growth of AI hasn’t enabled us to catch up. The agents have too much access.”

“That’s where we are,” agreed Seamus McCorry, Check Point’s country manager. “You have to look at AI as an additional risk vector. A lot of organisations have run slightly quicker than they now wish they should have. In Check Point, we went slowly – it was controlled, even down to our large developer base. We have thousands of developers, but we didn’t let them use Claude until we had guidance. We’re a security company, so we’re paranoid – reputationally, it would be a disaster for a security company to find AI issues. So we probably went slower than a lot of people, and on that basis, we made sure there was no chance of leakage or breach.”

On/Off

With consensus and emerging practice suggesting that governance within organisations was finally getting on the right kind of trajectory, McCorry was interested to get into a related but different issue.

   “The next concern area is controls,” he said. “You’ve got Copilot running in emails, in HR – that’s a concern. Even within Check Point – a paranoid security company! – can I say all our intellectual property is secure? I think so, but it just needs one wrong configuration change and things could get exposed.”

While the responses to this line of the discussion remained broadly as positive as on the corporate strategic topic, there seemed to be a bit less confidence that their companies were as well prepared when it came to technical controls. Restricting access to the most trusted of employees, at least initially, appears necessary in many businesses.

 “We have four full licenses for Copilot,” one CISO explained. “The chief executive has one; their PA – who actually runs the company – has another. I’ve got a license, and our Red Team has one. These are just little steps, to try to make sure we’re not embarrassed in a few months’ time.”

“All our employees have access to Copilot, but we don’t always give them the full license,” another leader said. “We allow people to build their own Copilot agents, to let them do whatever they want. In terms of applications we specifically develop, our application security developer gives us code so we can see which applications have access to AI.”

Beyond technical controls, governance and license-related use restriction, another growing concern is around LLM “training”. Attendees touched on aspects of this, including a need to be aware of what data had been used to develop the LLM before it was deployed in the company, as well as ensuring that, during development and refinement of the model, customer data was not exposed to it, even though test data may not be sufficiently representative to optimise the tool.

Human After All

Further topics covered in a lively, wide-ranging and fast-moving discussion included corporate responsibility – or the lack of it (“no-one wants to own the risk, it’s too complex”); the difficulty of doing due diligence (“it’s nigh-on impossible”); and the fact that agentic systems will find ways to work around even the most elaborate controls if the user prompts suggest this is the optimum way it can achieve the tasks it has been given.

The leaders around the room also discussed the potential for automation of security tasks: whether they could see a point when some of the roles within the SOC could be handed over to AI agents, and if so, when that might happen. The consensus was that this is inevitable, if only because of the constraints currently being placed on responsiveness and capability by increasing incidents and limited staff numbers. “We’ve got 20,000 people in the company, and I have six people in incident handling,” one CISO said. “We’re relying on automation, though there’s a lot of trust-but-verify.”

Another perennial topic – the potential, and the limits, of the use of genAI tools to help businesses complete the onerous and time-consuming task of filling in third-party compliance questionnaires – was also aired. One novel twist to that discussion came when one leader mentioned that they had asked, in their questionnaire, whether the suppliers used AI and, if so, whether it was compliant with European Union regulations. And one supplier had come back with the response, No, we just use a bit of machine learning. “And suddenly, all those questions disappear. AI doesn’t exist! Funny old thing,” the CISO chuckled.

As the conversation began to wrap up, Black asked a question he said he is often reluctant to ask: what topics had not been covered already which attendees felt were vital to include? There was only one response, but it was made decisively and immediately.

   “Cost,” the CISO in question said. Their point was not so much about the costs involved in building defensive capabilities to protect businesses from the unintended consequences of genAI adoption – though those, too, are important. Rather, they explained, it is the unknown future costs that the adoption itself may be exposing businesses to, which they felt is potentially a huge problem that nobody as yet is seriously considering.

“At the moment, everything’s subsidised,” they pointed out, noting that the companies developing genAI tools are largely funded by venture-capital institutions. “What happens in a couple of years when that’s no longer the case? You’ll then have an inability to adopt – or something you’ve already adopted suddenly becomes so expensive that you can’t continue to use it. I don’t think anybody is really talking about that – and it’s huge.”