The notion that AI-powered malware may one day pose a threat has become valid. While there aren’t any real examples in today’s attacks, the building blocks to create them are already available. From a cybersecurity perspective, it’s important to prepare for a future in which AI and/or machine learning augments both malware and attack tools.

Although cyber criminals and threat actors can use machine learning techniques for a variety of purposes, I’d like to focus on one potential use case – an attack tool that uses machine learning to guide its behavior and actions. Here are four important questions for us to consider:

What would AI-based malware look like?

An adversary can use a tool or piece of malware created using reinforcement learning – a machine learning technique used to train an agent based on what it observes from its environment – to automate parts of an attack chain, such as lateral movement, persistence, reconnaissance, privilege escalation, or data exfiltration. Such a tool could execute sequences of actions very quickly. In all likelihood, it would achieve goals faster than an organization could manually or automatically respond to alerts from an intrusion detection system.

Although one might imagine that bad threat actors could use a tool of this type  for attacking sophisticated targets, they would more likely use it against weak targets – organizations that have not implemented commonly recommended cybersecurity measures. Hardened targets are less likely to leave obvious security holes or vulnerabilities in their systems, and often require human-level creativity to craft attacks and remain stealthy. However, adversaries who wish to compromise a large number of targets in a short time would use a tool capable of automatically performing parts of an attack.

How would the attackers create such an agent?

In reinforcement learning, an agent receives observations from its environment as input, and outputs actions to perform in that environment. For instance, for an agent designed to play Pong, the observation (state) becomes an image of the screen, and actions move up and move down. Training occurs by letting agents perform actions in the environment over a large number of episodes. Agents receive rewards after performing relevant actions or achieving goals. In our Pong example, the agent would receive a positive reward when the ball gets intercepted and a negative reward if they lose a life. An episode in this case serves as one game played out until the agent hits a game over state.

Mapping this to our attack tool, the agent’s state would consist of observations derived from its host system. Its actions would consist of commands or subroutines to run. Examples of observations in this case might include information about services, registry entries, the file system, and information parsed from running system commands. Examples of actions may include service modifications, moving or copying files, launching executables, and altering registry entries. The system would issue rewards when the agent has performed the correct sequence of actions to achieve a goal. The agent would be trained against preconfigured systems that contain vulnerabilities, security holes, or misconfigurations typically encountered during red teaming or penetration testing operations. 

Is it feasible for such a tool to perform attacks?

Depending on the primitives available to the agent, attack tools created in this manner are only as good as the scenarios they are trained on. Given the current state of reinforcement learning, the tool may also need some domain knowledge built into it. Such a tool could not adapt to more challenging scenarios and could not discover novel attacks.

Building a tool that can generalize and discover novel attack scenarios requires creating an agent that’s able to learn directly from byte-level observations and generate arbitrary commands with only characters as building blocks. Such an agent would require very large action and state spaces. Solving these problems requires different, undiscovered innovations in the machine learning space.

Who could build such a tool?

Someone who knows how to implement machine learning models (using, for instance, pytorch), and who has familiarity with how attack techniques work, could design and build the attack tool I’ve described. The difficulty lies in the training. Reinforcement learning models often run for millions of steps before they converge on a good policy. Each step requires running commands on an actual machine (or virtual machine). And the machines need to be spun up and configured for each training episode. Considering these challenges, it could take weeks or even months to train an agent, even if the process is parallelized. Hand-coding logic may still prove more practical for simple attack scenarios.

Although tools that automate steps in an attack are useful for cyber criminals (such as those responsible for corporate ransomware attacks), highly resourced threat actors, such as nation-states, are more likely to develop one. The fact that it’s possible to develop tools like the one I’ve described means that we may see them in the not-too-distant future.

However, we need not fear AI-powered malware. In addition to the hurdles facing attackers that I’ve mentioned in this column, defenders will enjoy some advantages when these threats arrive at our security perimeters. For example, the speed at which AI-powered tools perform an attack may come at the expense of stealth – optimizing for the shortest number of steps to reach a goal may incorporate actions that could trigger security alarms. They would also, at least initially, rely on exploiting known vulnerabilities and misconfigurations that red teaming exercises or security audits might easily discover. As these tools become more widespread and sophisticated, we will need to develop appropriate defenses, which may lead to a paradigm shift in the way we approach cyber security.

Andrew Patel, researcher, Artificial Intelligence Center of Excellence, F-Secure

The post There’s no reason to fear AI-powered malware appeared first on SC Media.