Security: Who Actually Chooses Your Company’s AI? A New Study Says “Nobody’s Really Checking” — But Look Closer at Who They Asked

Author: Author: Berend Watchus. Independent non-profit AI & Cybersecurity Researcher. July 21, 2026. Publication for OSINT Team, online magazine.

https://arxiv.org/abs/2607.16660

[embed]How Do You Choose Your AI Component? An Interview Study of Secure AI Integration in Practice The increasing adoption of Large Language Models (LLMs) as AI components in modern software systems introduces distinct…arxiv.org

A new paper on arXiv, How Do You Choose Your AI Component? An Interview Study of Secure AI Integration in Practice (Tamanna et al., North Carolina State University), sets out to answer a question that has quietly become one of the most consequential in software engineering: when a developer drops a large language model into a production system, how do they decide which model — and does anyone stop to ask whether it’s safe?

What they set out to study

The researchers ran semi-structured interviews with 22 practitioners — software developers, AI/ML engineers, data scientists, and a handful of CEOs and architects — across finance, healthcare, cybersecurity, manufacturing, and education. Their three questions were straightforward: What drives model selection? What makes integration hard? And how do people think about, and defend against, the security risks?

What they found

The headline is blunt.

Model choice is driven almost entirely by functional concerns — performance, accuracy, cost, latency, tool-calling, context window — while security barely registers as a selection criterion.

Practitioners lean on the reputation of big vendors (OpenAI, Anthropic, Google) as a stand-in for actually evaluating a model’s security. Where safeguards exist, they’re mostly reactive and infrastructure-level: PII filtering, input/output sanitization, isolated environments, the occasional “LLM-as-a-judge.” Documentation and formal guidance are thin to nonexistent.

The authors package this into a memorable thesis: the industry is having a Back to the Future moment, repeating the same mistakes it spent two decades learning about traditional software dependencies —

grabbing whatever’s available and reusable, worrying about provenance and attack surface only after something breaks.

It’s a clean story, and much of it is probably true. But it’s worth reading the second half of the paper — the part about who they interviewed — before you let “the industry” stand in for the whole software world.

What they actually studied

Look at the recruitment. Participants came from Upwork (a freelance gig marketplace), plus posts on unspecified social media and Slack/Discord communities, plus snowball referrals from earlier interviewees.

That is a very particular fishing spot. It selects heavily for freelancers, contractors, and people at small, fast-moving shops — the kind of practitioner who can be recruited off a gig platform for a $40 Amazon voucher. Fifteen of the 22 were individual contributors. Even the CEOs and architects who look like they balance the sample toward the enterprise end are, on these channels, far more likely to be founders of five-to-fifty-person companies than officers of a global bank. A “CEO” title signals seniority, not scale.

And crucially, the paper never reports organization size — only sector and years of experience. Yet size is arguably the single best predictor of how an organization handles risk.

Why scale is the whole story

Anyone who has worked across the range — a one-person operation, a five-person team, twenty, a hundred, a thousand, fifty thousand — knows these are not points on a smooth curve. They’re different species, and what changes is not competence but who carries the consequence.

At the gig and small-shop end, the person choosing the model is the same person who eats the fallout, and the fallout is usually “client’s unhappy, redo it.” There’s no procurement gate, no legal team that says no, no compliance officer whose entire job is to be friction. “Where do I sign, I start tomorrow” isn’t recklessness — it’s a rational response to an environment with a small blast radius and no downstream auditor. Often the sensitive data simply isn’t there, so skipping enterprise controls is a correct judgment, not a lapse.

At the top-100 or top-250 end, model selection is barely the developer’s decision at all. It’s routed through vendor risk assessment, data-residency review, and non-negotiable contractual terms — “here are our contracts; with most of them we won’t even open a negotiation.” The safety is structural, a property of the institution rather than the individual. A less security-savvy engineer inside that machine still produces safer outcomes, because the machine won’t let the unsafe choice through.

The honest reading

The paper’s own limitations section concedes that purposive sampling “may introduce selection bias and limit representativeness” — so the authors would not disagree in principle. The sharper point is this: their finding and their sample are entangled. “Results over security” is, to a large degree, a faithful description of the gig-and-small-company end of the spectrum — which is precisely the end their recruitment favors. Generalizing it to “the industry” flattens a real, species-level difference in how organizations of different sizes deal with risk.

None of this makes the study worthless. As qualitative work, it does what it should: it surfaces the reasoning, habits, and blind spots of a real slice of practitioners, and its recommendations for adopters, providers, and researchers are sensible.

But the right way to read it is with one question held constantly in mind — which size of company did this quote come from? Read that way, the paper isn’t evidence that software as a whole is sleepwalking into an AI supply-chain crisis. It’s a sharp portrait of what AI integration looks like at the small and independent end, where most of the people are — and a reminder that the largest companies, the ones the framing implicitly indicts, almost certainly don’t look like this at all.