Artificial intelligence is no longer just a laboratory experiment. It’s quietly becoming part of everyday software, helping developers write code, helping analysts do research, and powering tools in banks, hospitals, and tech companies. In recent years, Large Language Models (LLMs) have evolved from a curiosity to the core infrastructure for many digital products.
But as companies rushed to develop smarter systems, one important issue was left behind: security. The way AI systems behave is very different from traditional software, and this difference is forcing the world of cybersecurity to rethink how protections actually work. As a result, a new discipline is emerging within the security community: AI penetration testing, often referred to as AI pentesting.
Why AI systems create new security risks
Most software behaves in predictable ways. You give it an input, the code follows a set of rules and produces an output. Security testing has always been based on this predictable structure.
Large language models don’t work this way.
They interpret language, guess intentions, and generate answers based on probabilities rather than strict logic. Sometimes this works great. In other cases, it opens doors that security teams never expected.
Risks security teams are already investigating include:
- Immediate injection attacks, where malicious inputs manipulate the behavior of the model
- Data leakwhere hidden training information is displayed in answers
- Model manipulationwhere attackers influence AI decisions through crafted prompts
- Insecure API actionsin which an AI assistant triggers unintended system commands
These issues become even more severe when AI systems connect to databases, APIs, or automated workflows.
When AI connects to real-world systems, the stakes are higher
Many modern AI applications do not work alone. They often act as an interface for complex systems in the background. Think about a typical AI-powered tool today. You can read company documents, access customer databases, start backend services, or send requests to an external API. Security researchers point out that the risk often lies not in the model itself, but in the way the model interacts with other systems. Even a seemingly innocuous request can result in the AI ​​Assistant receiving sensitive information or executing unintended commands.
The growing field of AI pentesting
To assess these risks, security experts are adapting traditional penetration testing techniques to AI environments.
AI pentesting examines how language models behave when exposed to adversarial input, unexpected prompts, or manipulated data sources. Instead of examining network ports or software vulnerabilities, testers analyze how AI systems interpret language and how that interpretation affects downstream systems.
The engineers exploring this space include This is Goela Principal Application Security Engineer whose work focuses on the interface between AI systems and modern application security.
Modern research examines what happens when large language models move from controlled environments into real-world software ecosystems. As AI interacts with APIs, data pipelines, and automated workflows, the number of possible sources of error quickly increases.
Research is starting to catch up
For a long time, most work on AI safety remained in academic circles. Researchers examined theoretical attacks or analyzed how machine learning systems could be manipulated.
Goel has contributed to this discussion through research on topics such as federated learning for secure AI models, securing AI systems in adversarial environments, and protecting autonomous systems. Some of this work has been presented at international conferences such as IEEE and Springer, reflecting the growing recognition of these challenges in both academic and industrial settings.
Creating security standards for AI applications
As more companies adopt AI tools, the need for common security policies becomes clear. Organizations like OWASP have begun publishing guidelines specifically for generating AI systems and large language models (LLMs).
Organizations like OWASP have begun publishing guidance specifically for generative AI systems and large language models (LLMs). Goel has also contributed to community efforts focused on defining security practices for AI-driven systems, including work related to OWASP’s agent security initiatives.
These guidelines represent an early attempt to provide structure to a rapidly evolving field. The goal of these projects is to help developers integrate security controls into AI applications before vulnerabilities become widespread.
Turn research into real security tools
Beyond research frameworks, security teams also need practical ways to test AI systems.
To address this gap, Goel’s recent work includes developing and testing methods to identify vulnerabilities such as prompt injection in AI models, an area that continues to receive attention as generative systems become more widespread. An interesting feature of this tool is Multi-agent testing approachin which different analysis agents evaluate each other’s behavior during the test. This setup helps mimic coordinated attack strategies that may occur in real-world scenarios.
A version of this framework has been presented at events such as BSides Chicago, where researchers and practitioners share approaches to assessing the resilience of AI systems in real-world conditions.
AI is also becoming part of defense
While AI introduces new security risks, it can also potentially help solve some of these risks. Security researchers are experimenting with machine learning systems that monitor behavior patterns, detect suspicious activity, and automate threat detection.
Training future safety engineers
Another important part of the AI ​​security ecosystem is education. Universities are expanding programs that combine cybersecurity with artificial intelligence, but many real-world security problems are still not fully covered in traditional courses.
Activities like these help bridge the gap between academic research and the practical skills engineers need in industry.
Why AI pentests will become more important in the future
With every major technological change, new security challenges have arisen. Web security became essential with the spread of the Internet in the 1990s. With the expansion of cloud computing, companies have been forced to review their infrastructure protections. AI seems to be in the same situation today. Large language models are integrated into everything from internal tools to customer applications. The greater their influence, the more important it becomes to test them carefully. AI pentests are still a young field, but are quickly gaining attention. As new research, security frameworks and testing tools emerge, the industry is beginning to lay the foundations needed to secure intelligent systems.
Daily Sparkz works with external contributors. All contributor content is reviewed by the Daily Sparkz editorial team.
