From Recovery To Resilience: Lessons From An Incident Response Leader

First Moves

It is often the case that an incident will come out of the blue, and at an inconvenient time. And the first lesson to be learned, perhaps, is: when you’re interviewing potential new staff, if they seem particularly committed to their work, they might well prove to be the best people you’ll ever hire. The speed of an initial response can depend on colleagues who recognise that something is wrong and know how to escalate it quickly. Without them, you may not realise what is happening until it’s too late.

 The early days of a major incident can be dominated by damage assessment and an all-hands-on-deck response, with the urgent priority of minimising the damage and limiting the downtime. Disruption to digital connections can affect both online services and physical operations. There will be particular difficulties responding when attackers have obtained privileged access. Activity undertaken with legitimate credentials can be harder to distinguish from authorised work, making the investigation and containment decisions more demanding. A wider point is that privileged access can allow an attacker to perform actions that the systems would ordinarily permit, so the existence of controls does not by itself guarantee that misuse will be detected.

Ransomware can affect both operational systems and recovery resources. The effects can be significant.The discussion also highlighted the practical challenges of switching to manual processes. It isn’t enough to have a fallback: staff need to know how to use it, including colleagues who have only ever worked with the digital system.

Building A Bridge

Reliable internal communications are an immediate priority, including a bridge and a shared way to record updates. Even when this work is given appropriate early priority, none of this is a trivial endeavour. One key consideration will be the decision to go paperless inside a business, often taken for sound commercial and practical reasons. This speeds up internal and customer-facing processes, but recovery records must remain accessible when normal digital systems and backups are unavailable.

Rebuilding a directory service is only one part of recovery. Deciding what comes next requires a shared view of priorities and dependencies. Without it, teams can find themselves chasing different objectives at an exhausting and stressful time. Reliance on digital systems brings another challenge, which becomes particularly apparent in the days immediately following an attack. Knowing a lot about individual products is different from understanding the services they are part of. As the security leader put it: “Unless you know where your connections are, you can’t say you’ve contained things.”

Leaders And Leadership

A challenge likely to be shared by staff across many different types of organisations is deciding who should lead the company during the response to an incident. Questions of responsibility and accountability, perhaps left pending during times when everything is going normally, will inevitably come to a head in the midst of a crisis. And answers arrived at in haste – as everyone tries their best to rescue an immediately desperate situation – may not always be the same ones that would have been taken after sober analysis under less pressure.

During a major incident, the security team can find itself fielding questions about the whole organisation’s security position. Clear responsibilities and an accurate record of decisions help keep the response focused. There may be people who see a route to recovery that will work within their own division or business area, and who believe they should be able to pursue that at pace, because it will help the company get its services back up and running as soon as possible. There is obvious merit to elements of this argument, but there are also some fairly self-evident pitfalls associated with following through on it. And, without clear, agreed recovery procedures, there is a risk that ongoing security efforts will be undermined by well-intentioned people seeking to speed up the restart of the services they provide and rely on.

The security leader’s advice was to agree who has authority during an incident before it happens. The business will understandably want services working again quickly, while the security response needs to contain the attack and establish that restoration can proceed safely. Those priorities need a common decision-making process, with clear security criteria for bringing systems back online.

The effect on the people involved can be profound, and lasting.

The pressure on response teams can be exhausting.

There will be a clear requirement to provide embattled staff with some relief.

Bringing In Reinforcements

In-house teams working extraordinary hours may need reinforcements to reduce the risk of burnout. Integrating that support is itself part of the response. Bringing in crisis-management experts can be reassuring for teams already under sustained pressure. But their arrival also means agreeing how their role fits with the existing response, and who is responsible for each decision.

External teams may be used to working in a particular way, with a particular level of decision-making autonomy, and to a different kind of working relationship with in-house security staff. Their experience can be valuable, but the way they are deployed needs to support a coherent response.

Advice on which risks to accept needs to reach senior decision-makers through an agreed process. Security expertise must remain part of those decisions, alongside the operational and commercial considerations. Experienced outside specialists with a clear remit can make a substantial difference to an exhausted in-house team, and the working relationships built during that period can last. Of course, this all comes at a cost. Insurance and retained support can form part of the preparation, but the most intensive level of external assistance cannot usually be kept on indefinitely. Longer-term recovery needs its own resourcing plan. Keeping track of names, time and costs, and writing things down from the point external support is engaged, helps ensure that senior leadership has an accurate account.

How The Business Fits Together

But perhaps the biggest learning-point from this leader’s perspective was the relationship between the way a business is structured and the way its services actually run. Organisational boundaries can create difficulties during an incident and its aftermath. A large organisation may have business units covering areas such as central services, finance, customer channels and operations. Its customer-facing services will often cross several of those boundaries.

They offered a typical example: order fulfilment. An order comes from the website, stock and finance are checked, then it passes through distribution and warehouse systems before it can be shipped. Each part of this process may sit within a different business unit, but the customer experiences one service. Business units can have different priorities and measures of success. The challenge is to ensure that those differences do not obscure the dependencies between them. This can make bringing systems back online more complicated. The business needs to understand the risks at those interfaces and decide how they will be managed. The security leader described the value of documenting systems, mapping the services they support, and overlaying the risk information. Looking at each system separately gives one picture; tracing the process across business-unit boundaries can reveal dependencies that are otherwise easy to miss. The lesson is to align recovery and risk management across those services, as well as within individual business units. A better understanding of dependencies can help limit disruption and inform recovery decisions.

Making that change requires more than a technically sound diagram. Leaders need a shared understanding of the problem, the responsibilities involved, and the practical implications for their own business areas. That is where leadership and co-operation become particularly important. Bringing people together around a common recovery objective helps teams move from separate priorities to a co-ordinated response. Clear sponsorship is also needed to carry the lessons into longer-term improvements. A fuller understanding of the digital interrelationships between different parts of the business gives teams a stronger basis for resilience planning. It also helps explain where departmental responsibilities need to connect, without assuming that a diagram alone will change how an organisation works.

The aim is to turn the lessons into practical changes that make the business better prepared for future disruption.

Looking To The Future

Some lessons will depend on the organisation involved. Others apply to more or less any modern business. Above all, the security leader argued, ensuring corporate leadership is prepared for the decisions and competing priorities of a post-attack period can help make the response as prompt and effective as possible. The first recommendation was to have an agreement in place with the CEO about incident authority and the security conditions for recovery. Containment, eradication and restoration need to be co-ordinated, with clear criteria for moving between them.

“Second point,” they continued, “is, obviously, have a plan, get it written down, and really work it hard. And equal second: make sure you’ve written down what your systems look like.”

Ultimately, the lessons are perhaps not too surprising.

  “It’s all the things that we all know from however long we’ve been in the industry,” they said. “Look at the basics.”

The ability to do all this, one of the CISOs around the table suggested, will depend on the nature and strength of the relationships within the company, too. Implicit in their point was that the vital relationships are not just those between key senior individuals, but between teams and business units as well. The security leader fully agreed.

  “And you don’t build those relationships during an incident,” Halcyon’s enterprise account executive, Luke Kinder, said. “You build them early. If you do, that makes them more indestructible.”

Responding to a question about the early hours of an incident, the security leader made a serious point about the speed of containment decisions. Isolating a critical identity service may be an option during an incident, but it has significant operational consequences. The appropriate action depends on the attack and the environment. Preparing those decisions in advance, including who can authorise them, is more useful than assuming one action will suit every incident.

The discussion also brought shared recognition of why this would be a particularly difficult decision for anyone to take. One CISO summed up the dilemma quite neatly.

“There’s a danger of if you act too hastily you cause damage – and if you act too slowly you cause damage,” they said.

Regardless, the challenge facing all organisations is becoming ever more starkly defined. As Kinder put it:

“I know it’s a cliche, but it’s not ‘if’, but ‘when’. That’s probably a fair assumption now.”

The security leader agreed.

“I think, at some point, when the industry catches up, there’ll have to be a new approach of some sort, towards allowing things,” they said.

“Rather than having a privileged user who can get into everything and drop what they like, everything has to be pre-approved. It’s only becoming more important with AI, because everything happens at lightning speed.”