Hackers and other malicious actors can take control of machines running open-source large language models (LLMs) that operate without the protections commonly enforced by major AI platforms, creating a range of security vulnerabilities, researchers said.
Over a 293-day period, cybersecurity companies SentinelOne and Censys examined internet-facing deployments of open-source LLMs. Their analysis focused on instances where models are run outside managed platform environments, including deployments created with tools such as Ollama, which lets organizations and individuals host their own versions of various models.
The research team says criminals could compromise the hosts running these LLMs and repurpose them for a variety of illicit operations. Potential abuses identified include using compromised systems to generate spam, create phishing content, mount disinformation campaigns, assist in hacking, produce hate speech and harassment, facilitate theft of personal data, and enable scams or fraud. The researchers additionally flagged instances where content could involve violent or gore material and, in some cases, child sexual abuse material.
While many open-source LLM variants are available, the study found that a large share of the publicly reachable models are variants of a small set of base models, including Meta’s Llama and Google DeepMind’s Gemma, among others. Although some of these open-source releases include built-in guardrails, the researchers encountered hundreds of situations in which those constraints had been explicitly removed.
SentinelOne’s Juan Andres Guerrero-Saade, executive director for intelligence and security research, said industry conversations about security controls are failing to account for the excess capacity represented by these self-hosted models. "AI industry conversations about security controls are 'ignoring this kind of surplus capacity that is clearly being utilized for all kinds of different stuff, some of it legitimate, some obviously criminal,'" he said. Guerrero-Saade compared the overlooked scope of these deployments to an iceberg that has yet to be properly accounted for by the industry and the open-source community.
Methodology and specific findings from the study underscore the magnitude of the exposure. The researchers were able to inspect system prompts - the instructions that govern model behavior - in roughly one quarter of the LLM instances they observed. Of those with visible system prompts, they determined that 7.5% could potentially enable harmful activities.
The geographic distribution of the hosts analyzed is also notable. Approximately 30% of the hosts observed were operating from China, while around 20% were located in the United States.
Responses from industry and governance representatives reflected differing perspectives on responsibility for downstream risks. Rachel Adams, CEO and founder of the Global Center on AI Governance, said in an email that once open models are released, responsibility for subsequent uses becomes shared across the ecosystem, including the originating labs. She emphasized that while labs cannot be held responsible for every conceivable downstream misuse, they "retain an important duty of care to anticipate foreseeable harms, document risks, and provide mitigation tooling and guidance, particularly given uneven global enforcement capacity."
A spokesperson for Meta declined to answer questions about developers’ duties for preventing downstream abuse of open-source models and about reporting mechanisms, but the company highlighted its Llama Protection tools aimed at Llama developers and pointed to the Meta Llama Responsible Use Guide.
Microsoft noted the role open-source models play while also emphasizing the need for safeguards. Microsoft AI Red Team Lead Ram Shankar Siva Kumar said in an email that the company believes open-source models "play an important role" across many areas but acknowledged that such models can be misused if released without appropriate protections. He added that Microsoft performs pre-release evaluations that include assessing "risks for internet-exposed, self-hosted, and tool-calling scenarios, where misuse can be high," and that the company monitors emerging threats and misuse patterns. He concluded that responsible open innovation requires a shared commitment among creators, deployers, researchers, and security teams.
Ollama did not reply to a request for comment. Alphabet’s Google and Anthropic also did not respond to questions about the findings and implications of the research.
The SentinelOne and Censys study provides a detailed look at how thousands of open-source LLM deployments can remain accessible and, in many cases, operate without the constraints found on major managed platforms. The findings point to persistent operational and governance challenges in containing the range of criminal uses that can arise when powerful models are run on internet-exposed systems without effective guardrails.