How attackers and defenders use the same technology, and what organisations can do about it

For years, the quickest way to spot a phishing email was bad grammar. A strange greeting, a spelling mistake or an odd phrase told you that something was wrong. That clue is gone. Language models write clean, natural English, adapt a message to the reader’s job and send thousands of slightly different versions in a few minutes. Defenders use the same technology on their side to sort alerts, notice unusual behaviour and react faster. In many cases the fight is now between two sets of algorithms, while people decide what those algorithms are allowed to do.

What attackers are doing with AI

The biggest change is not a brand-new kind of attack. It is speed and volume. Generative AI allows less experienced criminals to run campaigns that used to need skilled operators.

Phishing shows this most clearly. Older campaigns used a handful of templates, and mail filters eventually learned to block them. Today an attacker can write a unique message for every target, in their own language, mentioning their role or a recent project. A LinkedIn profile and a company website are often all the research that is needed.

Deepfake fraud has become a real business risk. In early 2024, a finance employee at the engineering company Arup in Hong Kong joined a video call with what looked like the company’s CFO and several colleagues. Every other person on the call was fake. The employee made 15 transfers to five bank accounts, about 25.6 million US dollars in total, before the fraud came to light. The problem has grown since then. In IBM’s 2026 Cost of a Data Breach Report, one in four malicious breaches involved AI, 56% more than the year before, and deepfake impersonation was the most common type. Cloning a voice needs only a short recording, so a conference talk or a podcast interview is enough.

Then there is automation. In November 2025, Anthropic reported that a group it believes to be Chinese state-sponsored had used its Claude Code tool against around 30 organisations, including technology companies, banks and government agencies. According to the company, the AI did 80 to 90 percent of the hands-on work, such as scanning, writing exploit code and collecting passwords, while humans only stepped in at a few key decision points. Only a small number of the attacks succeeded, and the AI sometimes invented login details that did not work. Some researchers believe the level of autonomy was overstated. Even so, the case shows clearly where things are heading.

Verizon’s 2026 Data Breach Investigations Report points in the same direction. For the first time in almost twenty years, the exploitation of vulnerabilities (31%) overtook stolen credentials (13%) as the most common way into a network. Verizon suggests that AI may already be helping attackers find and use weaknesses faster.

What defenders are doing with AI

A security operations centre (SOC) can receive thousands of alerts a day, and most of them are harmless. Anyone who has spent a long shift in front of a SIEM dashboard knows how tiring this is, and tired analysts miss things. Machine learning helps in three main areas.

Detection. Behavioural models learn what is normal for each user and each system. If someone in the finance team downloads 40 GB at 3 a.m. from a country they have never logged in from, the system raises an alert, even if no known malware is involved.

Triage. Related alerts are grouped together and enriched with context, for example how important the affected server is or whether it appeared in an earlier incident. The analyst gets one short summary instead of fifty separate alerts. The decision still belongs to a human, but much of the repetitive first step disappears.

Response. Automated playbooks can isolate an infected laptop or lock a compromised account within seconds. This matters, because ransomware can encrypt a whole network in less than an hour.

The numbers support this. According to IBM, organisations that use AI and automation widely in their security operations saved 1.93 million US dollars per breach on average. Half of the breached organisations already use AI agents for threat hunting and response, but only 18% use them for vulnerability management. That is a gap, because vulnerabilities are exactly what attackers are now exploiting most.

Defensive AI has its limits too. Models produce false positives, lose accuracy when normal behaviour changes and can be fooled by attackers who understand how they work. Their output should be treated as evidence to check, not as a final answer.

When the AI itself is the target

Companies are rolling out chatbots, copilots and internal assistants very quickly, and every new tool gives attackers something new to aim at. The best-known problem is prompt injection: hidden instructions in a web page, email or document that an AI assistant reads and then follows. If the assistant has access to email or files, the damage can be serious. IBM found that prompt injection incidents cost organisations about 5.89 million US dollars on average. More than one in five organisations in the study reported an attack on their AI models or applications, and 92% of those had no proper access controls on these systems.

Shadow AI is another growing issue. The term describes employees using AI tools that IT has not approved, often pasting customer data or source code into them. Verizon found that 45% of employees now regularly use AI on company devices, compared with 15% a year earlier. In IBM’s data, the share of incidents involving shadow AI more than doubled to 43%. Poisoned training data and stolen models complete the list of risks.

Is it a fair fight?

Not really. An attacker only needs to succeed once and does not worry about false alarms or the law. A defender has to protect everything, and a wrong automated decision has a cost as well. Imagine the system locking the CEO’s account in the middle of an important presentation.

Speed is the second worry. The Anthropic case suggests that reconnaissance, exploitation and data theft can be joined into one automated chain that runs at machine speed. This leaves defenders less time to react, so the time needed to detect and respond matters more than ever.

Still, defenders have one advantage that attackers lack: they know their own environment. A model trained on a company’s own logs knows what normal activity looks like there. This only helps, however, if the data is clean, complete and actually used.

What organisations should do

Technology alone will not solve the problem. In my view, the following steps offer the best value:

1. Get the basics right. Patch internet-facing systems such as VPNs and firewalls quickly, because AI makes them easier to find. Use phishing-resistant MFA, for example FIDO2 security keys. They stop many AI-written phishing attempts, because there is no code for the victim to hand over.
2. Confirm through a second channel. Any request for a payment or a password change that arrives by phone or video should be checked by calling a number you already know. This should be a written rule, not friendly advice.
3. Keep a human in the loop. Automation can handle low-risk tasks, but actions that could stop the business need human approval.
4. Write an AI usage policy and offer approved tools. Staff need to know which tools they may use and which data must never go into them. If there is no safe official option, people will simply use their private accounts.
5. Treat AI agents like users. Give them their own identities and only the permissions they really need, and log what they do.
6. Test AI systems before launch. Penetration tests should cover prompt injection and data leakage.
7. Update awareness training. Use real deepfake examples. Advice like “look for spelling mistakes” no longer works.

AI will not make security teams unnecessary, and it will not make attackers unstoppable. What it does is speed up both sides. The advantage will go to organisations with better data, clearer processes and people who understand the tools they use. For now, the best defence is still a simple one: good security habits, applied every day, with AI as a helper and not as a replacement.

Contributed by GuestPosts.biz