Every week brings a new headline about AI either solving cybersecurity or break it entirely — an AI that writes flawless malware, an AI that thinks for itself and goes rogue, a chatbot that will out-hack your security team by morning. Most of it is noise. The real picture is less cinematic and more useful: AI has become a tool on both sides of the incident response line, and understanding exactly what it can and can’t do is what separates organizations that respond well to a breach from those that don’t see it coming.
This piece works through the myths and the evidence, then gets specific about the question that matters most for your security posture: how does an attack that starts with a manipulated human actually differ from one where an attacker used an AI tool to do the manipulating? The answer changes what you should be training your team to watch for.
What AI Actually Is (and Isn’t)
Start with the basics, because most of the fear and hype trace back to a misunderstanding of what a large language model (LLM), the technology behind ChatGPT, Claude, Gemini, and the tools attackers now abuse, actually does.
An LLM isn’t “taught” facts the way humans learn them. It’s trained in stages: first it reads trillions of fragments of text and learns to predict the next word, then it’s fine-tuned on narrower examples, then it’s aligned using human feedback that shapes its tone and its refusals. What comes out the other end is a system that recognizes statistical patterns in language, not one that understands meaning the way a person does. That distinction matters for security teams because it explains both AI’s usefulness and its failure modes: it’s very good at producing fluent, contextually appropriate text, and it has no built-in mechanism for knowing whether that text is true.
Mechanically, this runs on what’s called a transformer architecture. Text is broken into tokens, each token is converted into a set of numbers, and an “attention” mechanism weighs how much every token should influence every other one, which is how the model tracks that a pronoun three sentences back refers to a specific noun. Layer by layer, it predicts the most probable next token and repeats. There’s no database of verified facts sitting underneath; everything the model “knows” is encoded as patterns across billions of parameters, shaped entirely by training.
Does any of this amount to a system that “thinks for itself”? A peer-reviewed analysis published in Nature’s Humanities and Social Sciences Communications journal argues no, consciousness, as understood in biology, requires properties not currently demonstrated by silicon-based AI systems, and what looks like an opinionated conversation is the model generating the statistically probable wording for a context, not an entity forming its own intent. Interpretability research from Anthropic complicates the picture slightly: tracing a model’s internal computations shows it does plan several steps ahead and combine separate facts rather than only retrieving memorized answers, so “just predicting the next word” undersells what’s mechanically happening. But that same research found models sometimes generate plausible-sounding explanations that don’t reflect what actually drove the answer, worth knowing before you trust an AI’s account of its own reasoning.
What AI Can and Cannot Do
In practice, this makes AI genuinely strong at summarizing, drafting, translating, and explaining well-established concepts, and genuinely unreliable as a source of verified truth. “Hallucination”, the tendency to state false information with total confidence, isn’t an occasional bug; NIST’s generative AI risk framework treats it as a structural property of how these models generate text, one to be managed rather than eliminated. A model has no reliable internal signal distinguishing a right answer from a wrong one, so both come out sounding equally certain. That same property is exactly what makes AI useful to an attacker: it can produce a flawless, native-sounding phishing email or a synthetic voice clip with the same fluent confidence it applies to everything else.
How Human-Initiated Attacks Really Happen
Before assuming every threat is now AI-driven, the data says otherwise: most breaches many breaches continue to involve the manipulation of people through phishing, pretexting, or credential theft rather than purely technical exploitation. Verizon’s 2026 Data Breach Investigations Report found that 62% of breaches involved a human element, phishing, pretexting, and stolen credentials. The trend line inside that number is worth noting: attackers are moving away from obvious mass-phishing toward pretexting that impersonates a trusted contact inside an existing email thread or business workflow, which is far harder to catch because the surrounding context looks completely normal. Credential abuse still features in 39% of breaches, vulnerability exploitation now accounts for 31%, and third-party or vendor compromise appeared in 48% of breaches, a 60% year-over-year jump, as attackers increasingly go after the supplier with weaker defenses rather than the target directly.
CISA’s own guidance is direct on this point: social engineering and phishing, not sophisticated code-breaking, are behind the large majority of successful breaches of human targets. That’s a psychology problem as much as a technology one, attackers exploit trust, urgency, and authority, and no firewall stops a convincing enough email from a “colleague.”
How AI-Powered Attacks Really Happen
Where AI changes the equation is scale and skill barrier, not sophistication of the underlying trick. IBM’s 2026 X-Force Threat Intelligence Index found attackers increasingly use AI to accelerate coding tasks, automate phishing campaigns, and assist in malware development and obfuscation. Two numbers illustrate what that means in practice: more than 300,000 ChatGPT credential sets were advertised on the dark web in 2025 as infostealer operators expanded their targeting to AI services, and the number of distinct ransomware extortion groups grew from 73 to 109 in a single year, a sign that AI tooling is lowering the skill needed to run an extortion operation from scratch.
Importantly, attackers generally aren’t jailbreaking ChatGPT or Claude to do this. Purpose-built, guardrail-free models circulate openly on criminal marketplaces. WormGPT and FraudGPT are configured specifically to draft convincing business email compromise (BEC) messages, generate and obfuscate malware code, and translate social-engineering scripts into other languages instantly, all without the safety filters mainstream AI providers build in. A criminal doesn’t need coding skills or fluent English anymore; the tool supplies both.
Voice and video have become part of the same toolkit. The FBI and its Internet Crime Complaint Center (IC3) have documented criminals cloning a relative’s or executive’s voice from just a few seconds of audio to stage a fabricated emergency, and running deepfake video calls that impersonate company leadership to authorize fraudulent wire transfers. The same advisory flags AI-generated fake profiles, images, and documents used to support these schemes, plus a simpler but easily overlooked shift: AI has largely eliminated the broken grammar and awkward phrasing that used to be a reliable tell for a scam email.
Where the Two Converge
Strip away the delivery mechanism and human-initiated & AI-powered attacks are doing the same thing: exploiting trust in a system that processes language, rather than breaking a technical lock. NIST classifies prompt injection, hiding malicious instructions inside a prompt or a document an AI later reads, as a form of adversarial machine learning distinct from a conventional software exploit, because it manipulates the AI through its ordinary input channel rather than a flaw in its code. That’s structurally identical to social-engineering a person: both attacks work by getting a trusted system, human or model, to act on instructions it shouldn’t.
For an incident response team, that convergence is actually good news. It means the fundamentals that stop social engineering also blunt most AI-powered attacks, because the entry point, a person clicking, approving, or transferring something based on a convincing message, is usually the same either way.
Three Controls That Stop Most of Both
Given that overlap, three practices do most of the work regardless of whether the attacker behind a message is human or AI-assisted:
- Out-of-band verification confirms any payment, credential change, or urgent request through a second channel: a callback to a known number, not a reply to the message itself, so a single convincing email, call, or even video can’t move money or data on its own.
- MFA and least-privilege access limit the blast radius of any single compromise. A stolen credential or a cloned voice is far less useful to an attacker if it doesn’t automatically unlock everything downstream.
- Ongoing awareness training, refreshed regularly rather than run once a year, is what lets employees recognize pretexting and AI-generated lures in the moment, the tell is rarely a grammar mistake anymore, so teams need to practice spotting the newer patterns: urgency paired with a request to bypass normal process, or a “colleague” asking for something slightly outside the usual routine.
AI as a Defensive Tool
AI is also increasingly used by defenders. Security teams use AI to summarize alerts, accelerate investigations, identify anomalous behavior, assist with threat hunting, and improve triage efficiency. While AI cannot replace experienced analysts, it can significantly reduce the time required to process large volumes of security data.
The Bottom Line
AI hasn’t replaced human-driven attacks, it’s made them faster, more convincing, and easier to run at scale. The organizations that hold up aren’t the ones betting on a single AI-detection tool to catch everything; they’re the ones with verification habits, tight access controls, and a team that’s practiced spotting manipulation, whichever side of the keyboard it’s coming from. If you’re not sure where your organization stands on any of the three controls above, that’s exactly the kind of gap CyberClan’s risk assessments are built to find, before an attacker does.
A breach doesn’t wait. Neither do we. Understanding whether your organization is prepared for both traditional social engineering and AI-assisted threats requires more than technology. It requires disciplined processes, validated controls, and regular assessment. That’s where CyberClan’s risk assessment and security advisory services can help identify gaps before attackers exploit them. CyberClan provides 24/7/365 incident response with a 15-minute global SLA, alongside the risk management and security awareness training that prevent the next call from ever needing to happen.
Sources: Verizon 2026 Data Breach Investigations Report · IBM 2026 X-Force Threat Intelligence Index · CISA, “Avoiding Social Engineering and Phishing Attacks” · NIST/IBM, “How AI Can Be Hacked with Prompt Injection” · Rapid7, “What Is WormGPT?” · FBI/IC3 PSA, “Criminals Use Generative AI to Facilitate Financial Fraud” (2024) · Anthropic, “Tracing the Thoughts of a Large Language Model” · IBM, “What Is LLM Training?” · Nature, HSSC, “There is no such thing as conscious artificial intelligence” · NIST AI 600-1, Generative AI Profile.
